Agile innovations in data management

A successful agile data warehouse can speed the delivery of information to organizations. Ideally, initial raw data can be delivered to analysts quickly, then integrated into a formal enterprise data warehouse (EDW) for more robust and broad reporting in ensuing development cycles.

Many companies have run into challenges with making agile work. Data warehouse projects tend to be large and complex, with many inter-related parts. The business is not willing to wait years, or even months, to have data available for reporting and analytics. Gartner studies continue to show that 70 to 80 percent of BI projects fail to deliver the BI capabilities users want. Yet, no one will give up on BI: CIOs have pegged BI as the top technical priority through at least 2017, according to Gartner.

As pressures mount to have more immediate availability to data, and to have the flexibility to change priorities based on data insights and changing business conditions, agile development is becoming more of an imperative. It can play a key role in aligning and flexing more with business stakeholder needs.  

I’ve spent the past 2 ½ years developing an EDW using agile methodologies. Prior to this most recent initiative, all of my experience in data warehouse and business intelligence has been using waterfall methodologies, which were effective, but slow.  These processes were fine when the business need for data was much more stable and consistent. However, the need for data has changed dramatically over the course of time, and supporting that need is a much larger challenge.

Building the foundation for agile

Switching to agile was tough, but we discovered that building the right foundation can really improve the speed of delivery and result in successful business adoption. Within a few months, an environment with raw data was available for data exploration for our data analysts. Within the first year, business stakeholders were using the new EDW, and within 18 months it became the single source of truth for the organization. All analytics, standard reporting, and dashboards were transitioned to the new EDW, and the two legacy databases were decommissioned. This new EDW continues to grow, with new sources of data being added on an ongoing basis.

Step one on the road to agile data management is developing a robust business conceptual model. Spend some time interviewing key organizational leads, and document the company’s key activities based on “what” not “how.” Group and consolidate these activities into logical processes. The result will be a top-level enterprise process framework (which is a good document for every company to have).

The enterprise process framework can then be used to develop a conceptual business model. As a result, this model will reflect all of the key capabilities of the company. Even when organizations change, the basic process framework is typically consistent. Thus, the conceptual business model will remain a viable model from which to build.

Starting with the conceptual business model enables agile development by:

– defining a master plan for long term development

– allowing incremental development, mostly by addition

– deferring significant details until development time

By taking this first step to develop a conceptual business model, the team (and the business stakeholders) will be able to better understand where all development fits into the eventual future state. Higher priority areas within the model can be identified as ones to tackle first, with lower priority areas to follow. I was amazed at how easy it was to see the pieces start fitting together over the course of time.

From here, development can take an evolutionary approach, with just-in-time data modeling and development, one “user story” at a time. The data warehouse is developed as parts of a whole, and both the development team and the business stakeholders can better understand how the parts fit together.

Additional steps in building this foundational approach to agile data warehouse development include:

– building a grass roots data governance capability

– developing a high level architecture with repeatable design patterns

– ensuring solid testing and tools

– implementing a robust data quality program

– giving the development team the ability to self manage their agile development approach, incorporating continuous improvement

I will cover these remaining steps in the next few upcoming articles.