Data Warehousing: Making Smart Decisions Via Business Intelligence Systems

In the mid-1980s, when Ralph Kimball started selling what were then called decision support systems, he felt he was working less against normal buyer skepticism than against the culture itself. Decision support systems (DSSs) were and are methods of filtering aggregate numbers (such as total sales per week) out of the data generated by the transactional devices (such as cash registers, telephone switches, ATMs, bar-code scanners) that pervade everyday life. The beneficiaries of those systems were supposed to be managers, and the logic of the case was easy to make, or so Kimball thought.

Unfortunately, the consensus at the time was that real managers did not depend on numbers. Manly managers went down to the floor or out into the field (a.k.a. the front lines), rolled up their sleeves, got their hands dirty and managed by intuition. Those who managed by numbers were called bean counters, among other, less neutral terms, and were universally derided as a pallid, thin-lipped tribe with no human sentiments. No one wanted to be a bean counter, and the DSS industry suffered accordingly. Year after year the same 100 to 200 people would show up at the industry conferences. Then one year?he thinks it was 1993?Kimball stepped up to a podium and saw an ocean of unfamiliar faces in front of him. “What happened here?” he remembers asking himself.

Part of the answer was that a new generation of managers?the children of VisiCalc and Lotus 1-2-3?was taking over. They had cut their teeth on text about lean manufacturing and total quality management, philosophies that emphasized the importance of transparency, and large, dynamic flows of information. Since DSS was in a position to supply that need, doors began to open. CIO reflected and advanced that change, arguing in an article (“Information Preservation,” July 1993) that the technology allowed managers to “make intelligent business decisions faster.”

As often happens, with success came a new name: data warehousing. The moniker was probably a step backward in terms of clarity, since strictly speaking a data warehouse is just one piece of a DSS. (An entire DSS includes a transactional device that produces the raw data, software that filters this data and writes it into storage?the latter being the actual warehouse?and a client on the manager’s desk for probing the warehouse.) On the other hand, the new name provided the impression that an updated technology had arrived, which never hurts.

During the ’90s the industry grew steadily, with revenues reaching into the low billions, according to an IDC report from the period. Toward the end of the decade, the technology underwent another change. Before then, analysts measured growth by the number of installations; in the late ’90s the number of users per installation also began to grow. Partly this reflected changes in utility: As hardware grew faster and cheaper, data warehouses were able to remember more types of data at finer levels of detail, allowing quicker responses to more questions. Vendors built systems specialized on department-level missions, such as ERP or CRM, which simplified interfaces and training. (Those are sometimes called data marts.) New categories of data, such as clickstream, which can capture the behavior of specific customers on a granular level, opened up for filtering and analysis.

As those changes worked their way through the market, data warehousing changed from an analytic tool, whose users were concentrated in senior management, to a platform for distributing quantitative material to every corner of the enterprise. Querying the data warehouse became an integral part of almost everyone’s job routine.

Some observers see this change as the cusp of a new era, in which data warehouses become the language, even the skeleton, of the enterprise itself. From its earliest days DSSs have required managers to impose consistent, rigorous definitions of objects and processes, such as sales or product names or regions or dates. At the time that was more of a nuisance than a feature, but today it has become clear that the possibilities of both large-scale collaborations among enterprises (and departments) and software-based intelligence depend on establishing the largest possible library of unambiguous, enterprise-specific references. “The warehouse has become a data hub, gathering information from both operational systems and users,” says Richard Tanler, CEO of Minneapolis-based e.Intelligence, a vendor that utilizes the data warehouse as the foundation for planning and predictive intelligence.

Certainly Kimball no longer worries about getting his calls returned. Along with his wife Julie, he now operates Ralph Kimball Associates in Boulder Creek, Calif., which in turn runs Kimball University, “the authoritative source for data warehouse education.” He does offer private consulting, although according to his website, Ralphkimball.com, he has little time for public speaking. Kimball stuck with his technology when it was languishing on the periphery, and now he is collecting a return on that loyalty. There may be a lesson in that; if not, we should all be so lucky.