Meta Data ROI: The Evolution of Technology (part 2 of 3)
By David Marco
In order to fully understand meta data’s value it’s imperative to identify the changes impacting businesses today. Last month we examined the three key areas impacting today’s corporate landscape. These three areas are:
-
Emergence Of One-To-One Marketing
-
Evolving Distribution Channels (internet)
-
Global Marketplace
In this our second installment on meta data return on investment (ROI) we will examine the technical challenges that arise as a result of these evolutionary corporate changes. In part 3 of this series we will present the remedies that meta data provides to these technical and business challenges.
Technical Challenges
Poor Systems Integration And Adaptability
The days of building an operational system and using it for 10 – 20 years, without major enhancements (typically complete rewrites), are over. In order for companies to adapt to market’s rapidly changing landscape the enterprise’s data processing systems must be more flexible and robust than ever before.
Many to our current systems have been built as “stovepipe” applications. Meaning that they do not communicate easily with other corporate systems. Moreover, these “stovepipe” applications form their own system “islands” with their own hardware platforms, databases, and development languages. Corporations are demanding new systems changes at an astounding rate and unfortunately these old “legacy” systems do not adapt well to change. This is most clearly evident with the Y2K problem that has haunted most every company in the world. If we stop and think about it Y2K is merely a date field that has been designed to hold a two-digit year, instead of a four-digit year. When the problem is considered in these terms is doesn’t appear to be very significant. Of course we know that the Y2K problem is very significant as the code that manipulates the date field is very convoluted, difficult to identify, and hard to locate.
For those that believe that the need to make global systems changes end with Y2K keep in mind that immediately, following the Y2K will be the new standardized European currency (EURO) challenge. While the EURO will significantly impact all corporations, financial institutions will be most effected.
Along with the rapid change in business needs there is an equally rapid change in the technology that fuels systems development. As a result, corporations are working feverishly to keep their company’s systems up-to-speed with the extreme rate of change in technology. It seems that weekly there is a new software application or piece of hardware that would aid a company in meeting their corporate goals. This situation is certainly not going to slow down in the upcoming years. Corporations are demanding faster hardware, and more software functionality and integration to gain a corporate advantage.
Systems Must Deliver Better Business Value
Another challenge facing companies today is getting their systems to meet the needs of their business users in a manner and language in which the business users understand. As an industry we as IT professionals are still not sufficiently meeting the needs of our business users. One recent survey asked CEO if their IT systems were meeting the needs of their business. Over 84% of the CEO felt that their systems do not. As an IT professional this is a statistic that cannot go unnoticed. Instead of designing systems that speak to them in business terms, the systems we are building still communicate to their users in IT terms. For example, I experienced one such system that would show the user a report called XC001AB, which is a report that shows product sales, by region, by marketing campaign, over time. Clearly a marketing analyst would much rather see this report entitled “product sales, by region, by marketing campaign, over time” as oppose to XC001AB. This small example illustrates why so very few senior executives believe that their corporation’s Information Technology strategy is well-integrated with their business strategy. The good news is that the vast majority of these senior executives do believe that their systems are critical to the success of their business.
Decision Support Moves to the Forefront
Most companies have come to the realization that a decision support system is critical to their enterprise goals. Operational systems are typically designed to manage products. These systems have evolved over the years to produce, deliver, and invoice products/services. . Unfortunately, when remedial questions are posed:
-
“Who are our most profitable customers?”
-
“Which segment of our market offers the greatest future potential profit?”
-
“Which of our products are complimentary (market basket analysis)?”
-
“Which competitors pose the greatest threat to our existence?”
-
“Which product/service is providing the greatest value to our customers?”
These operational systems cannot easily or quickly provide answers to such fundamental questions that are redefining the world of business. The emergence of decision support looks to transition these legacy (operational) systems to decision support systems (DSS) that help companies answer these questions, which support a one-to-one marketing future. DSS system is designed to manage customers as oppose to products. DSS systems are build to handle the “strategic” questions that a company’s key decision-makers need to answer. Moreover, companies now understand that without a DSS system that they will not be able to compete effectively enough to survive in tomorrow’s marketplace. In fact I believe the in the near future, 5 – 10 years that decision support will be a key component to any major corporate IT initiative. It will no longer be a decision of “if” we should build a DSS system, but “how” quickly we can build them.
DSS Challenges
The construction of decision support systems to aid in the strategic planning and decision making process. The task of replacing old operational systems is being impeded by out dated systems designed to manage products, and not a company’s customer base. These “operational” systems have evolved over the years to produce, deliver, and invoice products/services. Unfortunately, when a remedial question is posed “who are our most profitable customers?” These systems cannot easily or quickly provide answers to such fundamental questions that are redefining the world of business.
It’s easy to see the need for DSS systems, unfortunately the task of implementing these systems is anything but easy. The challenges for implementing a DSS system (also know as a data warehouse) come from both the business and technical arenas.
Business Challenges
The most common cause for DSS project failure is that once the systems is built it didn’t meet the business objectives of the organization. Data warehouses that don’t satisfy the business user’s decision support needs are not accessed and eventually die.
Clear business objectives can be clearly defined and are measurable. This activity is critical since once the DSS project is completed the management team will have to justify the expenditures. Moreover, it’s important to understand that a data warehouse is NOT a project, it is a process. Data warehouses are organic in nature. They grow very fast and in directions that are very difficult to anticipate. Most warehouses double in size and in the number of users in their first year of production. Once a cost justification can be quantified for the initial release the process for gaining funding for the follow up releases is greatly simplified.
From a political perspective an enterprise data warehouse requires consent and commitment from all of the key departments within a corporation. DSS systems pull data from all of the key operational systems from across the enterprise and looks to create an integrated view of the corporation. This challenge is particularly difficult since rarely do separate departments agree on what that view should look like. As a result, the political aspects of implementing these systems are particularly challenging.
Technical Challenges
DSS projects technically stretch an organization in ways unlike that of traditional system projects. Typically a data warehouse will source data from most, if not all of the key operational systems within a company. The task for integrating all of this data from across the enterprise is considerable and requires the largest effort of any project activity. For example, it is probable that a company would want to store customer data in the data
warehouse. More than likely there is customer data in several of the firm’s operational systems. All of this dispersed customer data has to be integrated and cleansed before it can be loaded into the data warehouse. The process for integrating the data is complicated since it takes a good deal of knowledge on the data in order to integrate it.
DSS systems tend to store large quantities of data. It is even becoming somewhat common to see one or more terabytes of data being stored and accessed. By adding massive amounts of data into the equation the points of failure increase significantly. Moreover, large volumes of data push the envelope of the database management system (DBMS), middleware, hardware, and could force developers into using parallel development techniques, if massively parallel processing (MPP) architecture is needed. Keep in mind the answer to many of these challenges comes in the form of a hefty price tag. As a result, adding the dimension of size can be painful and costly.
Many legacy systems contain redundant or inaccurate data. This lack of data quality in the operational systems has caused more than one decision support effort to fail. As a result, this operational data must be cleaned before it is loaded into the data warehouse to ensure its usability. Metadata is critical for monitoring and improving the quality of the data coming from the operational systems. Metadata tracks the number of errors that occurred during each data warehouse load run and can report when certain error thresholds are reached. In addition, the DSS data quality metrics should be stored over the history of the DSS system. This allows corporations to monitor their data quality over time.
Next month we will wrap up this series on meta data ROI by examining the solutions meta data provides to these business and technical challenges.