Metadata, and its management, has had a long and winding history. Frequently overlooked, the value of metadata should never be discounted.
Computer aided software engineering (CASE) tools, introduced in the 1970s, were among the first commercial tools to offer metadata services. CASE tools greatly aid the process of designing databases and software applications; they also store data about the data they manage (“metadata”).
It did not take long before users started asking their CASE tool vendors to build interfaces to link the metadata from various CASE tools together. These vendors were reluctant to build such interfaces because they believed that their own tool’s repository could provide all of the necessary functionality and, understandably, they did not want companies to be able to easily migrate from their tool to a competitor’s tool. Nevertheless, some interfaces were built, either using vendor tools or dedicated interface tools.
In 1987, the need for CASE tool integration triggered the Electronic Industries Alliance (EIA) to begin working on a CASE data interchange format (CDIF), which attempted to tackle the problem by defining meta models for specific CASE tool subject areas by means of an object-oriented entity relationship modeling technique. In many ways, the CDIF standards came too late for the CASE tool industry.
During the 1980s, several companies, including IBM, announced mainframe-based metadata repository tools. These efforts were the first metadata initiatives, but their scope was limited to technical metadata and almost completely ignored business metadata. Most of these early metadata repositories were just glamorized data dictionaries, intended, like the earlier data dictionaries, for use by DBAs and data modelers. In addition, the companies that created these repositories did little to educate their users about the benefits of these tools. As a result, few companies saw much value in these early repository applications.
It was not until the 1990s, as newer tools expanded the scope of metadata addressed to include business metadata, when some business managers finally began to recognize the value of metadata repositories. Some of the potential benefits of business metadata identified in the industry during this period included:
- Provide the semantic layer between a company’s systems (operational and business intelligence) and their business users
- Reduce training costs
- Make strategic information (e.g. data warehousing, CRM, SCM, etc.) much more valuable as it aids analysts in making more profitable decisions
- Create actionable information
- Limit incorrect decisions
The metadata repositories of the 1990s operated in a client-server environment rather than on the traditional mainframe platform that had previously been the norm. The introduction of decision support tools requiring access to metadata reawakened the slumbering repository market. Vendors such as Rochade, RELTECH Group, and BrownStone Solutions were quick to jump into the fray with new exciting repository products. Many older, established computing companies recognized the market potential and attempted, sometimes successfully, to buy their way in by acquiring these pioneer repository vendors. For example, Platinum Technologies purchased RELTECH, BrownStone, and LogicWorks, and was then swallowed by Computer Associates in 1999.
With the growing focus around the World Wide Web, data warehousing and the pending year 2000 (Y2K) deadline looming, the mid to late 1990’s saw metadata becoming more relevant to corporations who were struggling to understand their information resources. Efforts began to try to standardize metadata definition and exchange between applications in the enterprise. Examples include the CASE Definition Interchange Facility (CDIF) developed by the Electronics Industries Alliance (EIA) in 1995 and the Dublin Core Metadata Elements developed by the Dublin Core Metadata Initiative (DCMI) in 1995 in Dublin, Ohio. The first parts of ISO 11179 standard for Specification and Standardization of Data Elements were published in 1994 through 1999. Microsoft and Oracle battled over development of a metadata standard throughout this period. The Object Management Group (OMG), supported by Oracle, developed the Common Warehouse Metadata Model (CWM) in 1998. While rival Microsoft supported the Metadata Coalition’s (MDC) Open Information Model in 1995, by 2000, the two standards were merged into CWM. Many of the metadata repositories adopted the CWM standard.
In the early years of the 21stcentury, existing metadata repositories were updated for deployment on the web, plus some level of support for CWM was introduced in the products. During this period, many data integration vendors began focusing on metadata as an additional product offering. Many of the products that existed in the early 2000’s became part of larger organizations’ suites (e.g., Superglue is part of Informatica, Ascential is part of IBM, etc.).
However, relatively few companies actually purchased or developed metadata repositories, let alone achieved the ideal of implementing an effective enterprise wide Managed Metadata Environment as defined in “Universal Metadata Models”. There are a number of reasons for this, including:
- The scarcity of people with real world metadata skills
- The difficulty of the metadata development and integration effort
- The less-than-stellar success of some of the initial efforts at some companies that did not plan with enterprise data management best practices (no enterprise data governance, no enterprise data architecture, etc.)
- Relative stagnation of the tool market after the initial burst of interest in the late 90’s
- The still less-than-universal understanding of the business benefits of managing metadata
- The too-heavy emphasis many in the industry placed on legacy applications and technical metadata at the expense of business metadata and business usage
Due to the attention given analytics and the need for understanding the data used in decision-making, companies are beginning to focus more on the need for and importance of metadata. Emphasis is also expanding on how to incorporate metadata beyond the traditional structured sources and include unstructured sources.
Some of the factors driving this renewed interest in metadata management are:
- Entry into the market of a variety of new and established software vendors, some concentrating on metadata management, others adding metadata to their existing suites
- The challenges that many companies are facing in trying to address regulatory and privacy requirements with unsophisticated tools
- The emergence of enterprise wide initiatives like information governance, compliance, enterprise architecture and automated software reuse
- Improvements to the exiting metadata standards
- A recognition at the highest levels of the organization by some of the most sophisticated companies and organizations that information is an asset (for some companies the most critical asset), and must be actively and effectively managed
Conclusion
The history of metadata management, and managed metadata environments continues to expand with each adoption of metadata standards by an organization, by the recognition of the value of metadata to managing the information assets of the organization, and the need to optimize that value systematically and at an enterprise level.
