Documentation, It's MAD!

Let's face it, there's not many people who like documentation, except a good BA of course, but every single project needs it.

Now if the title and first sentence haven't put you off I'm going to let you into a few secrets.

1. A good set of design docs will save your project money
2. Post project documentation (actual designs, architecture maps, etc) will rarely, if ever, get read
3. Your documentation will be instantly out-of-date with release x.1

Still with me? OK, they're not really secrets, but they are issues we have to contend with. The motivation to write something detailed that will never get read and will be obsolete in 5 minutes, without the proper tools is pretty low. Especially if you have just had the go-live party the night before :)

So a typical Analytics (Oracle BI EE for the late comers) will need the following

a. Design of Datawarehouse - This includes Tables, fields, indices, schemas, tablespaces, parameters, views, Materialized views, etc
b. Design of DAC - EP's, Subjects, Table Groups, Tables, Indices (again!), Tasks
c. Design of ETL
d. Design of Analytics Repository (Oracle BI Repository)
e. Design of Webcat
f. Integration designs
g. Technical Architecture Map
h. Support Guide - What to do if.... scenarios

what did I miss?

(We are going to assume Analytics with Informatica ETL for this example)

Here's the catch. You have a reference to a physical table in at least four places - Db, DAC, Informatica, Physical layer. In your design you want to ensure referential integrity, i.e. don't refer to a table in the DAC that does not actually exist in the Db!. Another example where you have this is with tasks in the DAC that refer to Informatica Workflows.

My prefered approach is to create all these documents up front, i.e. in the design phase (say that last bit slowly with a deep voice), then update the documents at the end for what actually got built.

What tools do you have?

Most project sites that I go to will have MS Office, but you really need the professional version for Access and Visio Professional. You could try to publish the designs in Word or HTML, and you could even use Analytics to present the design!!
The Word vesion is probably the easiest (least technical) but requires most work, the Analytics version will need a place to be set up, with Webserver, db and Analytics server licence.


What do I use?

Glad you asked that because we have developed the MAD - Majendi Analytics Designer, system. This is a semi automated system that can document an existing Analytics aplication, from DB to Webcat. It produces MS Word documents which details all of the items in a) to e) above. I supplement this with Visio diagrams for star schemas (This also uses the design repository for the source tables).
As well as the Word documents templates being populated we also have a miniture Analytics system which details the meta data. The heart of the system is a set of Access databases which import metadata from txt files or directly using ADO (so it can work on any database) to conect to the databases. A side benefit is that building the system teaches you all the underlying data structures in the ETL and DAC.

How do you get the information you need?

You can query the repositories directly in the Oracle/SQL database.
You can use the documentation utilities of Analytics and the Web catalog manager
I also use the UDML file that you can get from Analytics, and parse this into an Access Db.

(If you want to know about UDML files I'll be posting a short note on that later).

The beauty about the MAD system is that I can document a whole complex system in a couple of days, even better is the ability to analyse an existing application to help with an upgrade, or with new releases - your documents can very quickly reflect the underlying new system.

So there you have it, Documentation may be boring but you can make it interesteing by using your technolgy skills to reduce the problem down to automated tasks, just like we did. :)