Managed Meta Data Environment: A Complete Walkthrough (part 4 of 8)
By David Marco
This article is adapted from the book “Universal Meta Data Models” by David Marco & Michael Jennings, John Wiley & Sons
In part 3 of this series, the Meta Data Sourcing Layer of a Managed Meta Data Environment (MME) was presented, along with a walkthrough of two of the most common sources of meta data: End Users, and Documents and Spreadsheets. In part 4, I will walk through the remaining common meta data sources:
Messaging and transactions
Web sites and E-commerce
Messaging and Transactions
Many companies and government agencies are using some form of messaging and transactions, either Enterprise Application Integration (EAI) or XML (sometimes EAI applications use XML), to transfer data from one system to another. The use of EAI and XML is a popular trend as enterprises struggle with the high cost of maintaining current point-to-point approaches to data integration. The problem with point-topoint integration is that the information technology (IT) environment becomes so complex that it is impossible to manage it effectively or efficiently, especially if you do not have an enterprise level MME. An EAI messaging paradigm should help companies unravel their current point-to-point integration approaches. Figure 1 shows an EAI messaging bus which provides the technical engine for the EAI messages.
Figure 1: EAI Messaging Bus
While the vast majority of companies are not very advanced in their use and application of EAI and XML, these types of processes can be used to capture highly valuable meta data: business rules, data quality statistics, data lineage, data rationalization processes, etc. Since the EAI tools are designed to manage the messaging bus, not the meta data around it, it is important to bring this meta data from the EAI tools into the MME to allow for global access, historical management, publishing and distribution. Without a good MME it becomes very difficult to maintain these types of applications. Large government organizations and major corporations are using their MMEs to address this challenge.
Within the wide array of applications a corporation uses, some will be custom-built by the enterprise’s IT department (e.g. data warehouses, general ledge systems, payroll, supply chain management), others will be based on packages (e.g. PeopleSoft, SAP, Siebel), and some may be outsourced or based on an Application Service Provider (ASP) model. This proliferation of applications can be quite voluminous. For example, we know of several corporations and government agencies whose applications number in the thousands.
Each of these applications contains valuable meta data that may need to be extracted and brought into the MME application. Assuming the applications are built on one of the popular relational database platforms (i.e. IBM, Oracle, Microsoft, Sybase, Teradata) the Meta Data Sourcing Layer can read the system tables or logs of these databases. There is also considerable meta data stored within these varied applications. Business rules and lookup values are buried within the application code or control tables. In these situations, a process needs to be built to bring in the meta data.
Web Sites and E-Commerce
One of the least used sources of meta data is corporate Web sites. Many companies forget the amount of valuable meta data that is contained (or should we say locked away) in hypertext markup language (HTML) on Web sites. For example, healthcare insurance industry analysts need to know the latest information about the testing of a new drug for patient treatments. Research is typically conducted by a doctor working with a hospital. The doctor usually posts his findings to the hospital’s Web site or portal, so it’s important to capture meta data around these Web sites, such as when the site was updated, what was updated, and so on.
For many companies it is a standard business process to interact heavily with third parties. Certainly companies in the banking, national defense-related agencies, healthcare, finance, and certain types of manufacturing need to interact with business partners, suppliers, vendors, customers and government or regulatory agencies (such as the Food & Drug Administration and Census bureau) on a daily basis. For every systematic interaction, these external data sources generate meta data1
that should be extracted and brought into the MME. 1See Chapter 2 of “Building and Managing the Meta Data Repository” (David Marco, Wiley 2000) for a more detailed discussion of external meta data sources