Business decision-makers need data fast. Here’s how to deliver.

IT teams tasked with keeping the business informed have expended vast resources defining a methodical, painstaking BI system development life cycle that typically goes something like this:

• Gather requirements from business users
• Design a data model to support those requirements
• Scout out data sources
• Load all that data into a starlike schema
• Develop BI objects
• Roll out the final product to end users

Unfortunately, the quest to achieve perfection often results in anything but ideal results. We’ve focused on creating the perfect factory and assumed that once we had that, it would automatically produce the perfect product. The problem is, when it comes to BI, perfection is all about context. What was considered ideal back during the requirements process isn’t so great now. Is it the business user’s fault that the context changed? Clearly not, but the fact is, just as companies need to be Agile to succeed, the process by which we deliver BI Analytics also needs flexibility baked in.

BI Agility is achievable. BI perfection is not.

Below we outline five elements that together promote an Agile enterprise BI environment. We delve deeper into this topic in our full report, at informationweek.com/analytics/agilebi.

1. Agile Development

Typical data warehousing development cycles are black holes. We’ve seen it take months for business users to have their analytical needs transformed into operating reports that show meaningful data. We clearly need dramatic improvement.

At the highest level, common development methodologies such as Scrum and Extreme Programming are being applied to the problem. The basic underlying premise is the need for an Agile, iterative process that shortens development cycles and speeds time to market for BI requests. There’s no reason you can’t deliver value to end users in weeks–or even days or hours–rather than months. An Agile methodology can fundamentally change how your users perceive the value of BI services in a positive way.

2. Agile Project Management

Agile development requires Agile Project Management. In a conventional hierarchical process, planning is done at the beginning of the process and often results in unwieldy, 100,000-line work schedules. In contrast, Agile Project Management focuses on a continuous planning, execution, and feedback loop in which:

>> Planning is done at the beginning of each cycle, rather than once.

>> “Lessons learned” sessions are done at the end of each cycle, not just at the end of the project.

>> Scope can be changed during development–yes, Agile allows and, to some extent, even welcomes scope creep and manages it by reprioritizing deliverables.

Agile Project Management delivers great benefits to both IT and the business. Requirements are precise and clear. The risk of underdelivering is reduced, as each cycle delivers new sets of usable functionality. Quality becomes part of development as bugs are discovered and fixed early.

But what about the Project Management office? Can you really bypass the review cycles, scrutiny, and official sign-offs? At the end of the day, what really matters is the return on investment, which tends to be greater when agile practices are leveraged because the results are aligned more closely with business needs–something users can continually confirm. Agile project management takes the monolithic and structured documentation and approval approach and skins it to the bare essentials: Single-page charters and verbal sign-offs after demos are good enough.

3. Agile Infrastructure

The standard BI infrastructure goes something like this: Information flows from data sources to the operational data store and data warehouse via extract, transform, and load (ETL) and is provided to customers, in most cases, as reports on a thin-client interface.

The question is, how do we optimize this architecture to fully leverage an agile development approach?

We can’t always do much about data sources; the company may not even own all of them. On the other hand, we can do a lot to improve the data integration layer. Vendors such as Composite Software, IBM, and Informatica offer tools to integrate data without physically moving it. Integration happens in a virtual layer; source data is cached in the virtualization server and refreshed as needed by the business or as agreed with the data source owner.

Virtual integration lets business users visualize data much earlier in the development cycle, which helps them further refine requirements. Such an architecture also sustains near-real-time BI more easily than the standard ETL model.

4. The Cloud and Agile BI

When does the Cloud make sense for BI and how does it improve Agility? Companies without BI programs should look closely at the Cloud as a way to jump-start their initiatives. With Cloud services, BI and ETL software can be provisioned as a service. Companies with problematic in-house systems can use the Cloud to avoid having to upgrade hardware and software.

Another time the Cloud can help is when data sources feeding the data warehouse change. Say a legacy system is replaced by a commercial, off-the-shelf system all the underlying BI mappings and infrastructure must be redone. When a packaged ERP system is adopted to eliminate a multitude of homegrown siloed apps, there may be BI modules associated with the packaged tool that can be implemented in the Cloud. Most companies will take a hybrid approach. Remember, where a system is hosted is less important than how fast and how well users can be served.

5. IT and Agile BI

Agility is driven by the need to serve end users. It’s about always being relevant and responsive. To achieve maximum effectiveness, IT must interact with the business it serves and also connect with the business problems. A BI team with high turnover from project to project will find it much harder to leverage lessons learned.

Finally, beware of red tape, which is usually imposed by ingrained processes and Project Management offices. Business and IT executive sponsors must commit fully to agile development. Only then can the need for speed be reflected in the BI infrastructure.