In working with many large corporations across a variety of industries, it is clear that we all face the reality of rearchitecting data warehouses being common place, whether we want to admit it or not. Since Data warehouses are business driven (rethink your strategy if you have IT driving your warehouse), it is very important that the business leaders championing your effort understand the perils of skirting a well thought out architecture and design.
You have probably heard the now famous words from Benjamin Franklin: “The definition of insanity is doing the same thing over and over and expecting different results”. Yet, we continue to see corporations of many sizes having to throw out their existing solutions and spend significant amounts of time and resources to build new ones. Look at the number of data warehouse vendors and BI tools vendors that have emerged on the seen in the last several years. Plenty of work for everyone. Too often organizations look to technology vendors to solve their problems, thinking it is the technology itself that is the issue. Rarely, is that the case.
Many individuals that attend data warehousing seminars, conferences, and classes have told me that they can’t understand why there is so much focus on data warehousing theory. My thought is that there is still so much unnecessary failure in data warehousing and it all starts with understanding the right leadership (business and IT), approach, and learning how to design a successful architecture.
Business Sponsors / Champions
In Oct of 2006 I was speaking at a large conference on an enterprise data warehouse effort I am currently managing for a large healthcare provider. It was amazing how many attendees stayed to chat or contacted me via email. Most of them had one consistent theme: they insisted how lucky our project was to have executive sponsorship. Indeed, we would not have the funding and direction required to deliver to the business needs without that sponsorship. However, sponsors don’t find you and tell you the right way to build a sound data warehouse solution. While, oftentimes we need to find them, we always need to work with them to repeat their needs back and explain the solution that is required in order to meet their real needs. So next time a business sponsor/champion tells you exactly what they need, or insists on cutting corners, use that as an opportunity to work with them to come up with approaches/solutions that hit their pain points, but enable a timely/valuable business solution. Generally, leaders/executives are in that role because of their ability to reason, rationalize, and strategize. If you are knowledgeable about the topic, they will likely listen. If this is not one of your strengths, consider bringing in an outside expert (again, make sure they are not a technology vendor, but a solution vendor).
What do these business executives need to know? Start here: Regardless of the business need, the challenge with building a successful data warehouse is directly related to the fact that most corporations have transactional systems with little to no intelligent integration. That makes it very difficult to bring data together into a data warehouse. In most cases, this is not a technology problem, although many software vendors insist they have tools that can alleviate the problem. Perhaps one day a silver bullet technology solution will arrive, but I am not holding my breath. Let us consider, if your data was standardized in terminology, definition, and quality across systems, building a data warehouse would be downright simplistic. Transformations would only occur for repositioning data into the most efficient database structure. Users would already be comfortable with the terminology and quality of the data. All of these are inherent in well designed information management. Yet, for most large corporations, this isn’t reality. Your sponsors need to understand this.
Consider that in simplistic terms, data warehouses seem to be thought of two ways:
First, as simple, quick solutions (90 – 120 days), that answer a specific question. Many consultants and teachers preach this today, but don’t tell you the requirements and ramifications of this type of approach. This definition of a data warehouse has significant value for a specific need, but is more often than not an easy out for someone that has many needs and justs wants at least one problem solved at some point in their life (can you blame them).
Secondly, as a larger enterprise structure for large corporations that can afford to spend a lot of money, and wait many years to get any results. This misconception of an enterprise approach stems from those that have likely read various horror stories about implementations that had no business value or were never delivered. In all likelihood, this was not due to the design or architecture, but was due to demands and shortcuts that led to getting something quickly rather than well. More often than not an enterprise approach is what is needed, but those that don’t know how to implement it iteratively fall back to the excuse that it is the monolithic, boiling of the ocean, and we all know that never works.
Often times, when we design something as complex and yet valuable as a data warehouse, we are pressed by needs or priorities that cause us to repeat the exact same problem that made this so difficult to begin with. We don’t have time to properly define the data of our business. Yet the companies that we all hear the great data warehouse stories from have all taken the time and resources to plan, architect, and design the data, approach, and delivery.
You can be assured that your business leaders hear about successes and failures and want to make sure their business in on the success side of that equation. Make sure you share some of this background with them so they can navigate all of the vendors and information they see with what is really needed for your organization.
Where do you start?
The only way to avoid repeating chaos in your data warehousing efforts is to work to educate the business leaders you work with such that they understand the cost, decreased value, and limited lifespan of efforts that meet a single need, but not the needs of the business. This doesn’t mean that they can’t get what they want in short order. Whether you are building an Enterprise Data Warehouse or a specific data warehouse/business intelligence solution, it can be built in stages, incrementally, but it has to be designed with the end in mind. It sounds pretty simple, yet most businesses continually look at redesigning / architecting / building their data warehouses and when they do, they repeat the exact same mistakes yet again.
IT resources understanding the business needs (based on business strategy), not the specific solution. Instead of taking what the business tells you as a direct order, ask questions of leadership, not line management. Then listen to what they are really in need of and work with an expert architect to design a solution that will meet their needs. I read an article several years ago about gathering business requirements. It referenced the difference between taking someone’s order (much like being in the drive thru of a fast food restaurant) and understanding someone’s needs. Over 20 years ago in my college years, it was repeatedly drilled into our minds that in order to be successful in IT we needed to determine requirements by working with the business, not taking their order. Today, it is amazing to think that one of our greatest challenges in IT is that we expect the business to tell us exactly what they want. This field of IT is not so simplistic that any executive can tell us exactly what to build and we will do precisely what we are told. Isn’t that why we are IT professionals – to help the business understand how to build a technology solution that will solve their needs? When the business pushes for you to cut corners, you need to have all of the pitfalls to explain to them what happens if they do. I can tell you that all of the leaders and executives in business and IT that I have worked with have much more respect for our team because of our ability to define an appropriate solution that meets their usage needs, delivers in a timely manner, and very importantly, follows through on what we said we would do. Believe me, we all get the same requests for corner cutting.
The second major key is to not jump to defining metrics, facts, dimensions, etc… Many organizations move straight to metrics analysis, spending months, sometimes years, trying to define all of the metrics and they don’t know what approach/architecture they need yet. If you are new to warehousing, strive to understand the various options for architectures first before you learn the appropriate steps required to deliver it. Focus on two things initially – Data and Usage.
Your business users will be much more successful and thus so will you, if you deliver a solution they use than if you deliver exactly what they ask for.
Bruce D. Johnson
Unit Head of the Enterprise Data Trust for Mayo Clinic
About the Author
Bruce has over 20 years of IT experience focused on data / application architecture, and IT management, mostly relating to Data Warehousing. His work spans the industries of healthcare, finance, travel, transportation, retailing, and other areas working formally as an IT architect, manager/director, and consultant. Bruce has successfully engaged business leadership in understanding the value of enterprise data management and establishing the backing and funding to build enterprise data architecture programs for large companies. He has taught classes to business and IT resources ranging from data modeling and ETL architecture to specific BI/ETL tools and subjects like “getting business value from BI tools”. He enjoys speaking at conferences and seminars on data delivery and data architectures. Bruce D. Johnson is the Managing director of Data Architecture, Strategy, and Governance for Recombinant Data (a healthcare solutions provider) and can be reached at firstname.lastname@example.org