Successful re-architecture of data warehousing / business intelligence / analytics environments pose business challenges in addition to information technology issues
Many large corporations across a variety of industries face the reality of re-architecting data warehouse environments to accommodate business intelligence and analytics services. Since data warehouses are business driven and should not originate in the Information Technology department, it is important that the business leaders who are the program’s champions understand the need for a properly planned DW/BI/analytics architecture and design.
The following quote has been attributed to Benjamin Franklin and others: “The definition of insanity is doing the same thing over and over and expecting different results.” Yet, corporations continue to discard their existing solutions and spend significant amounts of time and resources to build or buy new systems. Too often, organizations look to technology vendors to solve their problems, thinking it is the technology itself that is the issue, when it is rare that technology is the primary cause for the failure or lack of success.
Informal survey of attendees at data warehousing seminars and conferences indicate they do not understand the continuing focus on data warehousing theory, as opposed to best-practice solutions advice. Based on much observation, it seems there is still so much unnecessary failure in DW/BI/analytics for many reasons. Those responsible for the DW/BI/analytics programs must a.) recruit the right business and IT leadership, b.) develop the correct approach that meets the organization’s goals, and c.) learn how to design a successful architecture.
DW/BI/Analytics Business Sponsors and Champions
Every organization needs executive sponsors and champions for every major program and project. Indeed, no successful DW/BI/analytics initiative has the funding and direction required to deliver to the business needs without strong, continuing executive sponsorship. However, sponsors do not find the project team and tell them the right way to build a sound data warehouse solution. Sponsors do not deliver architecture knowledge; they deliver funding and championship to leadership to promote and sustain the program and its goals.
Although it is usual for the project team to find the sponsors (and it is important to find multiple sponsors for a DW/BI/analytics program), it is essential to work with them to review their needs and explain the solution that is required to meet their actual requirements. Therefore, next time a business sponsor/champion changes the requirements, or insists on cutting corners, use that as an opportunity to work with them to develop approaches/solutions that hit their pain points, but enable a timely/valuable business solution.
Generally, leaders and executives are in that role because of their ability to reason, rationalize, and strategize. If the project team’s members are knowledgeable about the topic, the sponsors usually will listen. If the project team does not have a strong speaker or advocate, consider bringing in an outside expert who is not a technology vendor, but who is a true subject matter expert.
Educating Business Sponsors and Champions
What do DW/BI/analytics business executives need to know? Start here: Regardless of the business need, the challenge with building a successful DW/BI/analytics program is related directly to the fact that most corporations have transactional systems with little to no intelligent integration. That makes it very difficult to bring data together into an analytical system.
In most cases, this is not a technology problem, although many software vendors insist they have tools that can alleviate the problem. Consider that if a data warehouse uses standard terminology, definition, and has managed data quality across systems, building a high-performing business intelligence environment would be straightforward. Transformations would only occur for repositioning data into the most efficient database structure. Users would already be comfortable with the terminology and quality of the data. All of these characteristics are inherent in well-designed information management environments. Yet, for most large corporations, this isn’t reality, and the sponsors must understand the implications of the lack of enterprise information management.
Two Approaches to DW/BI/Analytics Development
First, many DW/BI/analytics efforts are sold as simple, quick solutions that answer a specific question and require 90 – 120 days to complete. Many consultants and vendors support this approach, but they do not tell you the requirements and ramifications. This definition of a data warehouse has significant value for a specific need, but it is often an easy out for someone that has many needs and simply wants at least one problem solved without exploring the many other issues lurking around that glaring one challenge.
Second, other organizations use DW/BI/analytics as a larger enterprise structure for addressing large corporate reporting / decision/ analytics challenges. These large corporations can afford to spend a lot of money, and wait many years to get any results. They fail to understand the real meaning of the word “enterprise,” a misconception that may come from horror stories about programs that were started with no business value defined at the start or were never delivered.
In all likelihood, these failures were not due to the design or architecture, but were caused by the demands and shortcuts that forced quick results over proper results. Usually, a properly applied enterprise data management approach and architecture can solve the problem. Unfortunately, in many cases, project teams have not been taught how to design an enterprise DW/BI environment, or how to implement it iteratively. So, they fall back to the excuse that it is monolithic, that “enterprise” means “boiling of the ocean”, and that leads always to failure.
Often, when designing something as complex and yet valuable as a data warehouse, the project team is pressed by needs or priorities that cause them to repeat the same wrong decisions that form the basis for the initial system. Perhaps it is a perception of a lack of time to define the data properly. Yet the successful organizations, with effective DW/BI/analytics environments that scale and meet business needs have taken the time and resources to plan, architect, and design the data, approach, and delivery.
All business leaders hear about successes and failures and want to make sure their business in on the success side of that equation. Share this background with them so they can understand the theory and examine how it can be implemented against the organization’s requirements.
Starting Point
The only way to avoid repeating chaos in a DW/BI/analytics effort is to work to educate the business leaders so they understand the cost, decreased value, and limited lifespan of programs that meet a single need, but not the enterprise needs of the business. This does not mean that they have to wait a long time for results. An Enterprise Data Warehouse or a specific data warehouse/business intelligence solution can be built in stages, incrementally, but it has to be designed and architected with the end in mind (“What are the ultimate business goals? Why”). Although it sounds simple, most businesses continually look at redesigning / architecting / building their data warehouses instead of going back to their ultimate business goals and starting from there. If they only focus on re-building what exists, they repeat the exact same mistakes yet again.
The Keys to Success
IT resources must understand the business needs, not the specific solution. IT leaders should ensure that the IT team develop an understanding of the business strategy, since it informs the business needs for the DW/BI/analytics solutions.
IT team leaders should ask questions of business leadership, not line management. Listen to what they need and work with an expert architect to design a solution that will meet their needs. Good IT teams do not function as “order takers.” The difference between taking someone’s order (much like being in the drive line thru of a fast food restaurant) and understanding someone’s needs is demonstrated by working with the business to gather and understand requirements, not “taking their order”. A major challenge in IT is the expectation that the business can explain exactly what they want and that IT can build it. The role of IT is to help the business understand how to build a technology solution that will address their business goals and meet their needs. When the business pushes to cut corners, IT teams must explain to them what happens if those shortcut measures are taken. Many leaders and executives in business and IT respect IT teams that can define an appropriate solution that meets their usage needs, delivers in a timely manner, and very importantly, follows through on promises and speaks truth to power with knowledge.
The second major key is not to jump too soon defining metrics, facts, dimensions, etc.… Many organizations move straight to metrics analysis, spending months, sometimes years, trying to define all the metrics and they don’t know what approach/architecture they need yet. New DW/BI/analytics teams should strive to understand the various options for architectures first before learning the appropriate steps required to deliver it. Focus on these things initially – Data, Metadata, and Usage.
Conclusion
There are many business challenges in DW/BI/analytics re-architecture. Focus on business sponsors, their requirements and needs, and the appropriate architecture for the final goal. Focus on data, metadata, usage. Business users will be much more successful and so will the IT team if the solution meets the needs and requirements.
