Curing Bad Performing Warehouses

With the number of unsuccessful data warehouse projects in the last several years reaching significant numbers, more and more organizations are becoming frustrated with a return on value of analytics solutions.  Recently there has been a wave of unpleasant reminders of how significant amounts of money and time can be spent, only for an organization to receive little value and have to start over or abandon their efforts.  Oftentimes entire teams are laid-off or let go for lack of success.  In the last couple of months I heard several key leaders of technology companies speak about how to avoid all of the failures inherent in data warehousing by turning it into a hardware problem.  That has inspired what will hopefully be a thought provoking article on the root of unsuccessful data warehouses.

Parallel Patterns

The rampant dissatisfaction with data warehousing solutions seems in many ways eerily similar to the collapse of the financial industry.  Data warehouse failures and response time challenges are becoming common place, almost a normal occurrence.  When digging into the root causes of these failures, you can see the parallels of everyone wanting to jump in and have a data warehouse, but only pay pennies for it.  Saving a few bucks by hiring unqualified resources that may have data warehouse related terms and acronyms on their resume, but have never successfully designed a production solution that truly had a radical impact on the business’ success.  Data warehousing has turned into a programming build instead of a proven architect-led effort with heavy business analyst interaction directly with business subject matter experts. There is no predicate in this sentence.  It is a fragment. Thus, came the building of many solutions that were not set up to succeed.  Common failure points like:

• Not properly scoped or designed
• Not properly funded
• Lacked a roadmap, architecture, or vision
• Untrained resources treating it like a plain application
• Not business driven

The Wrong Goals

It seems to a large degree that the pressures and the challenges executive leaders face are at the root of the reckless approach taken towards many efforts.  Pay and various incentives are now yearly or even quarterly-based in all industries.  This has most that are charged with building analytics solutions focusing on the immediate future and not caring or planning for long term success.  Much like the financial industry they are focused on immediate press worthy highlights (call it a success even when it may not be).  Who or what project is based not on meeting an artificial timeline, but on delivering value to the organization/corporation in a big way?  Hit the date and the budget and you are a success.  If it is a poor long term solution or doesn’t add business value, bonuses are still received and everyone but the corporation wins.

Throw Hardware At It?

Unfortunately, amongst other irrational reactions, this has caused hardware vendors to rush to the “quick fix bailout plan”.  Throw hardware at it.  I was appalled to hear a hardware vendor speak about how data warehouses “never” have good response time and that the only way to have a usable data warehouse required the right hardware – of course, they have the perfect solution.  Most of them will describe how they have gone into an organization and applied their solution such that the customer received a 10 or even 100 fold decrease in response time.  Buyers beware, most experienced data warehousing professionals I know can say that they have done this many times in their career with simple design or tuning techniques at a fraction of both the near term and long term cost.  In fact, most have done it themselves more than these hardware approaches have solved.

Those that have been doing successful data warehousing for years can come up with many examples of very fast, powerful, effective data warehouses that have been in use for years.  Perhaps at its root is more the willingness of many hardware vendors, if not most, when times get tough, to say whatever is needed to capture the elusive revenue.  Yet, isn’t that what has us in this problem in the first place?

It is also a nice escape to offer organizations whose projects have not been successful.  If we can blame it on the hardware, we can say we did well, but need better hardware in order to realize all of the value possible.

Summary

In challenging times like these, it is even more critical that any data warehousing effort follows critical success factors in the design, development, and performance tuning of any analytics based efforts.  Successful teams ingrain their value in what they can provide for the organization, making them invaluable.  Unsuccessful teams are one of the first areas targeted for reorganization or layoffs.  Not only are unsuccessful efforts a risk to the individuals involved and the needs of the organization sponsoring the effort, but it is also a widening of the pipeline of negative feelings about data warehouse cost vs value.  The leadership resources receive their financial rewards and kudos, while in the long run the responsible technical resources receive their pink-slips.  If your organization struggles with performance or receiving value out of your analytics, 9 times out of 10, at least, it is not a hardware issue, but a design or tuning issue.  Many of these issues can be resolved quickly with experienced resources that are trustworthy.  All of these issues can be avoided with a sound architecture, approach, and design at the beginning.  The single biggest reason for my devotion to the EIM Institute is the quality and integrity of all who established it as well as the pure mission and goals of bringing respectability to information management.  May your next solution be a smashing success!

About the Author

Bruce has over 20 years of IT experience focused on data / application architecture, and IT management, mostly relating to Data Warehousing. His work spans the industries of healthcare, finance, travel, transportation, retailing, and other areas working formally as an IT architect, manager/director, and consultant. Bruce has successfully engaged business leadership in understanding the value of enterprise data management and establishing the backing and funding to build enterprise data architecture programs for large companies. He has taught classes to business and IT resources ranging from data modeling and ETL architecture to specific BI/ETL tools and subjects like “getting business value from BI tools”. He enjoys speaking at conferences and seminars on data delivery and data architectures. Bruce D. Johnson is the Managing director of Data Architecture, Strategy, and Governance for Recombinant Data (a healthcare solutions provider) and can be reached at bjohnson@recomdata.com