What makes a data warehouse die?

There have been a few different threads for me over the last couple of weeks all around what makes a reporting platform survive and thrive or die.

A data warehouse by any other name

(By reporting platform I mean data warehouses, analytical data marts, operational reporting, management reporting, etc)

A linkedin discussion over here, a tweet from Claudia musing if it’s the SLDC, a customer looking to consolidate its myriad of reporting platforms and a couple of projects I’m on getting close to go live and discussions around the post implementation support.

So for me the two major reasons a data warehouse dies is:

• a business owner wants to change technology for political reasons
• the processes in place to keep the reporting platform current, don’t, and cause its business value and relevance to degrade over time.

And of course the first one often happens due to the result of the second one.

So let’s explore this for a bit.

The project is only the beginning

Data warehouse and reporting projects are often treated as a one-off project, that are then moved into a typical application BAU process to keep it up and running. The BAU process is focused on keeping the lights on, not adding value over time.

But in reality reporting platform projects are just a starter, a way to focus the investment and resource required to define the initial requirements, find and install a hardware/software capability, acquire some data and produce some output.

These projects are always confined by time and dollars, which results in the project delivering an initial subset of the organisations full reporting needs.  In addition often the people who are articulating the requirements (often called “the business” don’t get me started on that one, opps I already have), don’t know whats possible, so find it hard to define requirements in any detail which would allow a final solution to be delivered under a project banner.

Change is constant

And of course the organisation will be constantly changing what and how it does things, which means the requirements for data and content will be constantly changing as well.

And this is the guts of it.  We end up putting in a BAU structure that is about managing and often minimising change, rather than embracing it as the key tenet of what should be happening.

But we make change hard

We make change hard in so many different ways:

• Require a project before any changes can be made
• Put in place restrictive SLDC’s, for example only releasing once a month
• Make changing the data warehouse a spare time task for a team
• Require detailed requirements that take months to gather before starting any work

And the natural reaction for somebody who needs a change to get the information and insight they require?  Eventually they give up and work around it themselves (enter Microsoft Access/Excel spreadmart stage left)

Its the size of your pipe that counts

So the first thing that needs to be done to enable the reporting platform to keep up with the demand for change, is to create some capacity.

I often refer to it as a pipe.  And what matters is the size of your pipe.

To me the pipe is how much resource (read people) you have allocated to making the changes.  It is also the mix of these resources and any bottlenecks you have in flushing a change through (4 BI/BA’s, 1 ETL dev and 1 report dev wouldn’t result in as large pipe a pipe as 2 BI/BA’s, 3 ETL devs and 1 report dev).

The pipe should always be full

So if the pipe is our people and we know our people are expensive it would make sense to get the most out of the cost of the pipe you can.  So the pipe needs to be as full as you can make it and as often as you can make it.

It also means your team are busy doing things (and good developers like being busy) rather than sitting around and waiting for work.  It means the people who want changes made know they are being made as fast as possible (with the current resourcing levels)

An its prioritising what goes through the pipe that counts

So we just let everybody ask for what they want and off we go making the changes right?  Well no, that would result in chaos and also mean we might keep the team busy making a change that is of low value while a high value change is not able to be delivered (because the pipe is only ever so big).

What we do is focus on the process of prioritising the changes, not stopping the changes.

Once there is a good process to prioritise the work in place, the team doing the changes are going as fast as they can, delivering as much value as they can.  The people (“the business 😉 who need access to new information and insight should be comfortable they are getting it as soon as possible and in the order of the value it will achieve.

And of course if they need it sooner the only conversation is whether to bump its priority or increase your pipe.

And the most important thing

The reporting platform is constantly evolving to meet the business needs, and won’t get replaced in a few years because its out of date and somebody wants to buy new toys.