The Importance of Data Governance In A “Super Numbers Cruncher” Era
By Anne Smith
A recent book by a Yale University econometrician, Ian Ayers, looks at a new trend in organizations, changing the decision-making process from one based on expertise and intuition to a data-based effort. This change is possible due to an almost inexhaustible supply of data on every topic, gathered from many sources and made available by the development of huge databases and the tools to manipulate the data in a variety of ways. The book, Super Crunchers: Why Thinking-by-Numbers Is the New Way to Be Smart, calls the data set analysts “super crunchers” and discusses the changes they are making to industries as varied as medical diagnostics, air travel pricing, screenwriting and online dating services. Although the author presents both sides of this debate (intuition versus use of data), clearly, he is convinced that the use of large amounts of data for “objective” decision-making is the better approach. The difference in the two approaches is not just a matter of managerial preference according to the author: “We are in an historic moment of horse vs. locomotive competition where intuitive and experiential expertise is losing out time and time again to number crunching.” Ayers shows that some older industries, such as wine-making, still rely more on feeling and experience than on the quantitative method. He believes that the data-based approach is needed to improve performance in every operation, using the incredible volumes of data accumulated in every organization, regardless of field.
This trend, which started with the development of data warehouses and other large databases for decision support in the late 1980’s and early 1990’s, is increasing due to the availability of enormous amounts of raw data, the relatively inexpensive data storage mechanisms and the creation of many sophisticated data mining and artificial intelligence software systems. As these factors continue their inexorable progression, the use of very large data sets to make “objective” decisions will increase.
This trend shows the need for improved and consistently applied data governance, so that the decisions are made with accurate, timely and valid data. Humans can overcome data anomalies with experience and intuition (“that data just doesn’t look right”, “I don’t think those values are accurate”, etc.) but software is programmed to accept the data as it is presented and is expected to use it according to rules and routines instantiated in the code. Without well-governed data for super-crunching applications to use, the decisions made by the “machines” will be flawed, and could result in loss of revenue, loss of market share, loss of lives. Without well-governed processes that represent accurately the business activities and rules, the analysis software will not perform as Dr. Ayers expects, and will provide inaccurate or false or misleading results. The governance of data and process becomes increasingly important as the trend toward data-based decision-making permeates organizations from every field.
The author still wants both human and machine to be in a mutually supportive relationship, with more weight given to machine predictions as time proceeds. Dr, Ayers answers the fundamental question of what place humans are to have in this “new world order” by identifying the need for humans to lay the foundations that enable super-crunching to occur. Humans must still “hypothesize”, he states; they must make the decisions about the variables to be used, while the computers actually perform the statistical analysis.
Humans govern data and process, humans act as data stewards, humans make the decisions about the data to be used in a data set or with an analytical application; all of these actions can fall under the role that Dr. Ayers describes for people: “laying the foundations” that enable super-crunching. This foundation must be solid, using accepted best practices for developing data governance and executing the role of stewardship, using data quality approaches and relevant software to ensure the accuracy and validity of the data. These foundations are also important for processes and analysis methods, since it is essential to use good data with good analysis methods to ensure good results.
Any organization that uses data-based decision-making or is contemplating it, should institute a governance program for both data and business processes to provide the right data for the “super-crunchers” to load into their very large databases for their statistical packages to operate against. Since Dr. Ayers’ research shows that most types of organizations are using or planning to develop the data-based decision-making capabilities, the book Super Crunchers can be viewed as a testament for the development of a data governance program in all organizations.
About the Author
Anne Marie Smith is a leading consultant in Information Management and is a frequent contributor to various IS publications. Anne Marie has over 20 years experience in information management for several corporate entities and has successfully led the development of data resource management departments within corporations and consulting organizations. Anne Marie is active in the local chapter of DAMA and serves on the board of directors of DAMA International, and is an advisor to the DM Forum. She has been an instructor of Management Information Systems (MIS) with Philadelphia, PA area colleges and universities. Anne Marie has taught topics such as: data stewardship and governance, data warehousing, business requirements gathering and analysis, metadata management and metadata strategy, information systems and data warehouse project management. Anne Marie’s areas of consulting expertise include metadata management including data stewardship and governance, information systems planning, systems analysis and design, project management, data warehouse systems assessment and development, information systems process improvement and information resource management/data resource management. Anne Marie holds the degrees Bachelor of Arts and a Master's of Business Administration in Management Information Systems from La Salle University; she has earned a PhD in MIS at Northcentral University. She is a certified logical data and process modeler and holds project management certification. Anne Marie can be reached at firstname.lastname@example.org