I recently host edited an issue of the Cutter IT Journal (Vol. 21, No. 4, April 2008), a magazine that publishes debates on controversial issues. As I was thinking of a topic, I thought what could be more controversial than asking the question whether business intelligence (BI) should always be done based on a data warehouse (DW) or if BI can be done directly against operational source data without a DW. My lead into this topic included the following arguments for both camps:

BI without DW arguments

Nobody disputes that bringing order into chaos is desired, but building a DW for the entire organization takes a lot of time. The more data there is to be standardized, the more departmental views there are to be resolved. And the smaller the DW team is, the longer it takes to build, maintain, and expand the DW. End-users, especially at the operational and middle-management layer of the organization, cannot wait for the DW team to catch up with their BI requirements. In addition, DW teams themselves have been promoting the idea of user self-sufficiency for a long time, acknowledging that the DW teams cannot possibly get to all the user requirements in a timely fashion. Therefore, there should be no reason why end-users should not take advantage of the modern BI tools and build their own BI solutions. For example, today BI applications can easily be created by power users with Microsoft Excel, a tool many end-users are very familiar with and love. Excel and other specialized BI software provide easy point and click capabilities to extract data directly from the operational source systems without the need for a DW, load the data into personalized databases, and provide analytical capabilities. It’s a dream come true for many end-users.

BI with DW arguments

On the other hand, one could argue that BI without DW is simply an excuse to create silo point solutions using BI technology with no coordinated data standardization and integration effort among the end-users. In addition, while end-users frequently claim that they know their data best, they know where to get it, and they know how to manipulate and analyze it, these same users do not agree on what the data means, what business rules and policies the data is subject to, and how to use the data properly. Each business unit or department has its own view of the data, which collides with many other views from other departments. The purpose of a DW is to standardize the data in order to provide a consistent and trustworthy Single Version of the Truth for all end-users throughout the organization. Moreover, why should data be treated any differently than other organizational assets? After all, organizations apply one consistent set of standards, policies and rules for managing human resources, equipment, buildings, parts inventory, financial assets, and so on. No organization in the world would allow each department to develop its own financial chart of accounts or to use a different salary structure and benefits policy from the corporate one. So, why allow each department to create their own set of financial reports from their own private databases?

In my call for papers, I asked the following questions: Is an organization better off building BI solutions when they need them and as quickly as they need them, without the overhead of a costly and complex DW? Are the risks an organization takes acceptable or unacceptable when end-users add more redundant and probably more inconsistent or plain wrong data by building silo BI solutions without a standardized and integrated DW? Who in the organization or what types of situations should determine which BI/DW strategy to pursue? After all, there are good arguments on both sides.

The resulting articles I got back were very interesting. While all authors to some extent acknowledged the benefits of data warehousing, opinions were split about BI’s necessity for, or reliance on, a DW.

It’s an evolutionary process

One author summarized his observations about BI as an evolutionary process that many organizations go through. The Lone Ranger depicts the first phase of this evolution. This is a “go to person” either in IT or on the business side to whom end-users turn for their reporting needs. The lone ranger is the defacto BI analyst who pulls operational source data from here, there, and everywhere, and gives the end-users what they want and when they want it. Sooner or later, the lone ranger becomes overwhelmed with too many and too complicated end-user requests and the end-users begin to recognize the limited capacity of a lone ranger strategy. At this point, the organization evolves into the second phase. A data-driven department in IT is chartered to consolidate operational source data into reporting databases and provide the end-users with standardized data and reports, which are properly audited and secured. This works for a period of time until IT becomes the bottleneck as end-user requests pour in faster than IT is able fulfill in a timely manner. This frequently leads to the third phase, the informed organization, which is often driven by a senior executive. In this phase, data integrity and business process management are critical to the perceived success of BI. As a result, data is cleansed, a mature methodology is used, databases are carefully architected to control redundancy and ensure consistency, a governance structure is put in place, and so on. The author concluded his article by pointing out that all phases in the evolutionary process should be considered valid BI strategies.

It depends on the type of end-user requirements

Another article strongly supported the position that BI with DW and BI without DW support two completely different types of end-user requirements, which the author called end-usercontracts. While some end-user contracts can best be supported by a Single Version of the Truth, other contracts don’t have that type of requirement. Instead, other contracts may demand the capability to respond to change quickly, which might best be supported by an independent (silo) BI solution. In the author’s words: “Development of a BI solution is always connected with a specific business case.” End-user contracts for these business cases must take into account the variables of cost, time, quality, and availability. The author gave two examples of how to use cost, time, quality, and availability to choose the best BI solution architecture (one example for BI without DW and another example for BI with DW). While the author did not write about an evolutionary process per se, he did describe various situations that require original end-user contracts to be reevaluated. This reevaluation can lead to a change in strategy from BI without DW to BI with DW. The author concluded his article with tips and best practices for designing BI solution architectures that are positioned to migrate easily from one strategy to the other.

Using agile processes is the answer

The third article conceded that many organizations have succumbed to the challenges and disappointments of BI with DW, but the author warned not to throw the “EDW baby” out with the “bureaucratic bathwater.” The bureaucratic bathwater refers to such things like long-running and rigid waterfall methodologies, lack of end-user involvement throughout the project, assumption that requirements were articulated properly only to find out too late that they were not, and so on. Rather than letting end-users collect their own data using spreadsheets or even BI tools to produce their own reports, which would be throwing out the “EDW baby,” the author suggested changing the method of development and end-user interaction. He elaborated on several agile techniques, such as Rapid Unified Process (RUP), agile data techniques, the myth and limitation of Single Version of the Truth, SOA and Web 2.0 strategies, and lean data governance. For example, how to apply RUP to deliver quickly by breaking requirements into use cases, how to involve end-users in prototyping activities, and how to allow end-users to refine their requirements during those activities. The author also described the concepts of agile data modeling, database refactoring, and continuous database integration, just to name a few. The article concluded with a list of best practices for adopting a lean data governance strategy to support all of the previously discussed agile techniques.

Business drivers demand operational real-time BI

Another author presented his arguments for how the future of BI will depend on business drivers as well as the direction the technology is taking. He suggested that “in a world turning to SOA, the question is whether massive data consolidation and data integration exercises are worth the big effort.” We already see forms of BI without DW, such as Corporate Performance Management (CPM) and Business Activity Monitoring (BAM), which are taking advantage of integration-enabling technologies, and the trend appears to continue in that direction. The author’s observations were not just based on technology trends, but on the changing demands from the business community. He listed the following business drivers, which are inviting a BI without DW strategy: lower infrastructure costs, faster and cheaper development, real-time monitoring, and improved data access and security. Furthermore, he observed that end-users today demand operational real-time BI with browser-based BI applications, portability to hand-helds, instant alerts, and continuous feedback on operational processes. He doubted that the traditional enterprise DW will be relevant in addressing these new end-user needs. After citing a few more future trends that support his position, such as adapters for legacy systems, composite services, BI on Internet platforms leveraging Rich Internet Application (RIA) technologies, SOA, portals, and mashups, the author concluded with a suggestion that a concept like Salesforce.com for BI is totally within reach and would probably make the debate about BI with or without DW mute.

All you need is integration-enabling technologies

Following the thread of the technology trends, another author compared the traditional architectures (BI with DW) to the new architectures (BI without DW). While conceding that there are instances of large-scale enterprise-wide initiatives that still require traditional architectures based on ETL, data marts, DW, and operational data stores (ODS), the author asserted that in many cases the new architectures based on SOA, EII, EAI, ESB, Portal and Web 2.0 technologies, can accomplish the same goals of those initiatives, and more. He provided a detailed accounting of the capabilities of these integration-enabling technology components and offered ideas on how to use them. For example, he illustrated how EII can be used to federate across popular relational databases and file dumps, and how EII opens up possibilities to “mashup” data in non-conventional ways. His in-depth review of how to use the capabilities of these integration-enabling technologies on BI projects continued to cover portals, RIA, composite applications, dedicated clients, Web clients, desktop widgets, and application widgets.

Who cares? It’s not an IT decision to make

The last article I received challenged the entire premise for a technical discussion of this topic. In fact, the author stated emphatically that the question “BI with or without DW?” is irrelevant from a business perspective. Business people consider that topic a technical implementation matter. Having said that, the author quickly pointed out that in his mind “the long-term implications for the business about data chaos and spiderweb operational implementations trump the (also) important short-term individual business unit considerations.” However, he insisted that as long as IT folks couch their case in technical terms, the benefits of one BI strategy over another will remain obscure to the business management team. Since making the BI/DW decision is based on business issues, the author offered five key questions to ask before choosing a BI strategy. The questions were basically governance and planning questions that identified two types of organizations: one with a sound strategic foundation and strong governance, and the other with a weak strategic foundation and weak or no governance. The author suggested that each type of organization will choose a different BI strategy. He concluded his article with the statement that in his view, “it doesn’t matter how strong the arguments are for either position [BI with or without DW], or how well they are made. They’re subordinate to the business issues. In effect – the answer depends on the business.”

Conclusion

At the risk of shooting myself in the foot as a DW professional, I personally am of the opinion that data warehousing is the biggest band aid IT has ever invented – but for now, it is a necessary band aid. The real issue is the data chaos in our operational systems. If we could get that under control, then BI without DW would not only be totally plausible, it would even be desirable. We have the technology to deliver BI without DW, we just don’t have a non-redundant, clean, and historical pool of operational source data. Master data management, if done correctly with the appropriate data governance, could be a step in the right direction. Therefore, the question for me is timing. Has the time come for BI without DW or is it premature? I would like to hear your opinions on this topic. Please e-mail them to me at methodfocus@earthlink.net.

About the Author

Larissa Moss is president of Method Focus Inc., and a senior consultant for the BI Practice at the Cutter Consortium. She has 27 years of IT experience, focused on information management. She frequently speaks at conferences worldwide on the topics of data warehousing, business intelligence, master data management, project management, development methodologies, enterprise architecture, data integration, and information quality. She is widely published and has co-authored the books Data Warehouse Project Management, Impossible Data Warehouse Situations, Business Intelligence Roadmap, and Data Strategy. Her present and past associations include Friends of NCR-Teradata, the IBM Gold Group, the Cutter Consortium, DAMA Los Angeles Chapter, the Relational Institute, and Codd & Date Consulting Group. She was a part-time faculty member at the Extended University of California Polytechnic University Pomona, and has been lecturing for TDWI, the Cutter Consortium, MIS Training Institute, Digital Consulting Inc. and Professional Education Strategies Group, Inc. She can be reached at methodfocus@earthlink.net