This article is excerpted from Data Warehouse Project Management (Addison-Wesley, 2000). All material is copyright Addison Wesley Longman, Sid Adelman and Larissa Moss. This is the firs article in a two-part series on Data Warehouse Indicators of Success.

There has been much heated discussion over the failure rate of data warehouses. Luminaries disagree on the percentage of those that have succeeded. The problem may be with the definition of success and failure.  In fact, very few organizations have identified, up front, what for them will be success or failure.  That being the case, any industry-wide numbers on failure are meaningless.

We have all seen projects that are grossly over budget, years behind schedule, and delivering a fraction of the promised function.  For various reasons some of these projects have been declared a success with promotions, industry and publication awards and bonuses for all involved.  There is a clear problem in that we have no accepted definition for success.

So that everyone is working off the same set of goals, and so that you are legitimately able to declare victory, we suggest that you list your measures of success and have that list approved by both IT and by the business sponsor.

Measures of Success

There are a number of examples of success indicators. Let’s take a look at some measures of success.

1. Return on Investment (ROI)

ROI can be achieved a number of ways:

• Lower cost – Costs could be lowered through better inventory management, fewer dollars spent on unproductive measures, product promotions, etc.
• Improved productivity – Greater productivity could be expected from both IT and the user. Today user analysts may spend 80% of their time gathering data and only 20% analyzing the data. The data warehouse should reverse those numbers. IT will still be responsible for developing complex reports as well as writing reports for production systems. The data warehouse can provide reporting tools with a well documented, clean, and easily accessible database. This capability should significantly improve IT productivity.
• Increased revenue – This could be as a result of greater market share and increased sales as marketing is able to more effectively target customers and provide the right products at the right time to the right market.

The effects of costs and revenues may be difficult to assign to the impact of the data warehouse. As the data warehouse is being implemented, the organization is not standing still. There are both internal and external factors that impact both costs and revenues, and so the actual benefits of the data warehouse may be difficult to determine.

2. The Data Warehouse is used

One of the easiest categories to understand can be measured by the number of users and the total number of queries and reports generated. If queries and reports are run regularly, it’s a good indication that the users are achieving some benefit.

3. The Data Warehouse is useful

The data warehouse may be used, but the users may find the benefits to be marginal and illusive. It’s important to ask the users what they see as the benefits of the data warehouse, how it has changed the way they do business, how it may have improved their productivity, and how it may have improved the quality of their decisions.

4. Project delivered on time 

This measure is problematic, as schedules are often set without an understanding of what is involved and how long each project task will take. “On time” is only relevant if a realistic schedule is the base for comparison.

5. Project delivered within budget 

This criterion may be difficult to achieve since the total costs of a data warehouse are difficult to determine. Initially, you may not have known how many users to expect, how many queries and reports they would be generating, and the complexity and resources used by the queries and reports.  You did not know how large the data warehouse would be, or how many indexes and summary tables would be required and desired.  You may not have anticipated needing a larger CPU.  You may not have known that the software was more difficult than the vendors represented, and that teams of software consultants would be required.  You may not have anticipated needing to upgrade our network to support the increased line traffic.  You may not have anticipated needing to raise the salaries of the data warehouse team nor the increased cost of recruiting the talent required to make the project a success. All these factors will contribute to severely underestimating the budget. “Within budget” is only relevant if a realistic budget is the basis for comparison.

6. Improved user satisfaction

Users may be internal, external or both. In all cases, the goal is to have users who are happy with the features and capabilities, the performance, the quality of data and the level of support.

7. Additional requests for data warehouse functions and data

You’ll know you were successful if other user departments are beating down your door with requests for access to the data warehouse, and current users are requesting new data and functions to be added to the existing data warehouse.

8. Business performance-based benchmarks

This is the most subjective of all the measures and will become the most controversial. Most industries have sets of industry averages as well as benchmark (the best) companies against which they make comparisons. For example, the industry average for quality, represented by the number of defects for a new car, may be three while the best is one. With better information, a car manufacturer in the middle of the pack, may have a goal to manufacture a mid-size sedan using eight worker days. The data warehouse may be able to provide improved, more complete and timelier information and, with this information, the auto manufacturer may be able to achieve their productivity goals.

9. Goals and objectives met 

On the assumption that you have developed goals and objectives, success will be defined by how well these goals and objectives were met. No doubt, not all were met or were only partially met. A scorecard will give you an initial and then an ongoing measure of your project’s success. Use the scorecard at the end of the article to score your results.

10. Business problems solved 

The data warehouse was developed for some specific reason. Perhaps marketing was unable to identify customer demographics for target marketing. If the data warehouse now provides this capability, it should be considered a success.

11. Business opportunity realized

The identified opportunity might have been the ability to provide information to suppliers through the web so they would be able to respond more quickly to your demands for components you need for your manufacturing process. If the supplier now has fewer stock-outs, the project is successful.

12. The data warehouse has become an agent of change

The world is changing, and the rate of change is accelerating dramatically. Successful organizations must be able to respond and respond quickly. Decisions must be made more quickly, but this can only happen with better and more timely information. There can be some fundamental changes to the business in the manner and speed in which decisions are made and, the data warehouse can be the vehicle for that change.

Measures of Failure

We may not be sure if the data warehouse is a success but we will always know when we have failed. Some of the indications of failure are:

1. Funding has dried up

This could be because the sponsor has moved on or is no longer interested in the project. There could be other factors that have nothing to do with the success of the project such as the company being bought by Philistines who have no appreciation of your efforts or the data warehouse possibilities. Real failure results if the project is perceived as having no real benefit to the organization.

2. Users are unhappy with the quality of the data

If you took a shortcut and decided not to clean the data, users will reject the data warehouse. The users may even have told you not to spend the time to understand and clean the data, that they want the data warehouse as soon as possible, and they don’t care if the data is dirty.  Don’t believe them. Even if the initial users don’t care if the data is dirty, other users in the organization who don’t know how to interpret the dirty data do care.  These new users would have to learn how to decipher the dirty data or they would get the wrong results if they don’t realize it’s dirty. If the quality of the data is poor, the data warehouse will be a failure.

3. Users are unhappy with the query tools

The notion of “one size fits all” is inappropriate for selecting tools. Power users need and want different tools than those fitted to the inexperienced and technologically intimidated users. The needs of each user segment must be met with the appropriate tool and the appropriate training and environment. Users will tell when they don’t like the query tool you have foisted on them.

4. Only a small percentage of users take advantage of the data warehouse

You may have expected everyone in marketing to be actively writing queries. They all went through training and graduated with a query tool certificate.  However, it turns out that whenever anyone needs some information, they all go to only one person in the department who is a frequent and knowledgeable user.

5. Your sponsor avoids you in the hallway 

When the only available seat in the lunchroom was at your table and the sponsor chose to eat standing up you may have a problem.

6. Poor performance 

Response time is so bad that users only launch queries right before leaving on extended vacations. It’s not only that response time is bad, but also there does not seem to be any means of improving performance that anyone is aware of.

7. Inability to expand the data warehouse (it’s not scalable)

You expect the data warehouse to grow, but if the technical architecture does not allow the expansion of the number of users and the size of the data base, the infrastructure must then be recast at enormous cost and lost opportunity.

8. Data that is not integrated –

One of the goals for the data warehouse is the ability to integrate data coming from many and heterogeneous source files and databases.  You wanted departments to be able to share data and to have data of record for the organization; data that everyone in the organization would accept as correct. A proliferation of data marts that do not have common keys or common data definitions is an indication of islands of data marts that cannot be shared.

9. Extract, Transform and Load (ETL) steps do not fit in the batch window 

ETL can gobble up machine resources and time. A number of data warehouses have failed because of the very lengthy time required for ETL. This problem is especially critical to global companies who have to be up and running in several time zones and are therefore starting with a reduced batch window.

Conclusion

To determine if your data warehouse is a success or a failure, you first need to identify your own measures for success and failure. These measures should be mapped to your goals and objectives. Only then can you objectively assess how well you are doing.

About The Authors

Sid Adelman is a charter member of Business Information Alliance.  He co-authored a methodology and project planning product tailored specifically for data warehouse.  Sid is an international speaker at data warehouse and industry conferences.

He can be reached at sidadelman@aol.com

Larissa Moss is president of Method Focus Inc., a consulting firm specializing in data warehousing.  Larissa has published numerous articles on various data  management topics. She frequently teaches seminars in the United States, Canada and Europe.

Larissa can be reached at methodfocus@earthlink.net.