Staffing A Data Warehouse – Part I
Throughout the years, I have seen as many different ways of building IT and business organizations to support data warehousing efforts as I have seen data warehousing architectures. Inasmuch as the establishment of an effective architecture is critical to success, so is the establishment of an organization that works specifically to satisfy your data warehousing needs and is successful within your organization’s culture. I frequently receive questions related to this topic from resources that have read articles or attended classes and yet still struggle to define how it should work in their organization. Oftentimes, consultants that haven’t worked as an employee in a corporation have a hard time customizing a standard organization to fit the uniquenesses of that business.
Due to the many layers of complexity involved in defining the right organizational needs for any business, the data warehousing industry has tried to put organizational focus onto “Centers of Excellence”. Just by attending classes or searching the internet, one can find many options for how your business should be organized. Yet, just like defining the right warehouse for your business, defining the right organization just isn’t simple.
As I continue to hear from and see organizations that struggle getting organized effectively, I will devote some time to this critical success factor. I feel very strongly that an entire book could be devoted to this topic and I am challenged to condense it enough to still contain significant value. Thus, this article will kick off a 3 part series on establishing an effective data warehousing organization.
- In part I, we will look at the general components of a data warehousing team.
- In part II, we will explore building and retaining a strong team.
- In part III, we will go through best practices, critical success factors, and simple things you can do that will greatly enhance your odds of success.
Breaking Down The Roles
The place to begin is outlining the relationship of business and IT resources that are required to work together for success:
Leadership: A core staple of data warehousing and analytics success is having the right leadership. A successful data warehouse is Business Driven and IT Lead. This means that IT should not run with an idea and have the “build it and they will come” plan for adoption/usage. Thus, there is a valid business need for a project to build an analytics solution. It doesn’t mean that the business should tell IT what to build. The success of almost all analytics solutions is heavily dependent on defining the right architecture to support your business needs. With so many options, most of which would lead to severe limitations or failure, expert resources can quickly layout the optimal solution for your business now and for the future. Key leadership roles include:
Sponsor: This is the “business” champion that leads the support and endorsement for all phases and components of your solution. On larger efforts, this person is ideally at an executive level.
Owner: This is the “business” leader/manager that is responsible for realizing business value from the solution that is being designed and delivered.
Business Manager: This is the “business” leader that is responsible for building and organizing the appropriate business resources required to collaborate with IT on the building and delivery of the solution as well as the adoption of the solution by business resources.
Business Users: These are the resources that will test and actually use the system once it is complete.
There are many other business roles, many of which are specific to the type of solution that you are building. It is imperative that the specific roles are tied to the specific business application, processes, and future usage needs.
IT Leader: This is the IT champion responsible for supporting this effort and working with business executives to fund and prioritize. On large efforts, this resource is usually the CIO or at the minimum a direct report to the CIO.
Manager: This is the IT manager that is responsible for staffing, budgeting, planning, sharing status reporting with leadership, and ensuring the overall progress of the efforts.
Project Manager: This resource is responsible for performing project management tasks on individual projects. Ideally this resource will help develop project processes, templates, and reporting/status tools that can be used by all projects associated with your efforts.
Data Integration: These resources are responsible for the design, delivery, and maintenance of the data that is moving into and through your data warehousing environment. This is typically referred to as ETL (Extract, Transformation, and Load). I feel that ETL is too simplistic and makes it sound too easy. Data Integration is the most time consuming aspect to most data warehouses and is also one of the most frustrating for management resources – both business and IT.
Solution Delivery: These resources are responsible for the design, delivery, and maintenance of the end user information access solutions often referred to as BI (Business Intelligence). They work with the various BI front ends and also work directly with data marts. They work very closely with the business to build and enhance their methods of using information.
Database Analyst: These resources are responsible for the design, delivery, and maintenance of the physical database structure itself. They work closely with Data Integration and Solution Delivery resources to optimize the solutions to user’s needs.
Data Modeler: These resources are responsible for the design, delivery, and maintenance of the data models that are the basis for the data warehouse. This includes working with the business on terminologies and definitions, data examples, relationships, and governance rules.
Quality Assurance: These resources are responsible for two key components prior to turning over to production:
- Validation of the data – accuracy of terminology, quality of data, comprehensiveness of data, etc…
- Front-end solution validation – does it meet requirements, application testing, reports/analytics testing, and security.
One final note on these roles: You should fully expect that your more effective resources will be able to fulfill several of these roles on a given project, but newer resources should focus on one or at the most two.
Sound Organizational Area Options For IT
With many different ways of building data warehouses, there are also practically unlimited ways of organizing those resources. Since every business need and solution is potentially different, I will not explore that advice here. This will focus on the IT side of the equation only. Each organization has a culture or method of working that they employ. In very successful organizations, these structures often translate over to data warehousing well, but in organizations that struggle with many projects and politics overrules success, I caution you that you may need to change in order to realize value from your warehouse. I will touch on what I feel are the most effective from two perspectives: large efforts and medium to small efforts.
For “Large Efforts”, since there are many areas of expertise and typically significant numbers of FTEs, the best practice is to focus on areas of expertise. Many organizations desire for all data warehousing IT resources to know every aspect of their solution. While this is an ideal scenario, it is impossible to achieve in large solutions. You will have a couple of key resources that know the solution end to end at a high level. This is merely practicality. Over the years of growth of your solution, it is good to have resources work on various different parts to expand their overall knowledge and value. A best practice for large efforts would have IT resources organized by area of expertise:
- Data Integration
- Solution Delivery (BI)
- Data Modeling and Administration
- Quality Assurance
- Project Management
These organizations are often referred to by terms like: Center of Excellence, Competency Center, Shared Service, Best Practice, etc… This type of structure will greatly reduce cost and increase value as it will leverage focused toolsets, single infrastructures, development processes, consistent security, etc… In this case, having strong, seasoned managers and a very strong architect is critical.
For “Small to Medium Efforts”, it is important to recognize the desired solution and future phases of it. This typically results in individuals having more involvement in multiple components of a solution. In some small efforts, an individual or two can design, develop, and effectively deliver all components of the desired solution. Ideally in these scenarios, you have a team of resources that each learn most of the layers, but you have some focused specialties that are too challenging and not worthy for all to learn – like a DBA and a Data Modeler. In these cases, having a seasoned technical manager guide this group is crucial.
Handling Support And Development
This is an oft-debated topic. There are two main approaches, with several variations possible. I have seen each of these types of organizations work successfully and have seen the challenges that come with them as well.
- Do you have some resources only do development while others only support what has been put in place?
- Pros: This can be a valuable learning process for new employees. When you put them into the maintenance side, they will gain more experience on your solution faster than would otherwise be possible. With experienced resources, it will provide them with the opportunity to stay focused on delivery.
- Cons: This can alienate resources that are relegated to only maintenance. They do not get the glory or learning that is inherent in development. There is also the potential negative impact of development resources just moving quick to throw solutions out, even of low quality, since they know they aren’t the ones that will have to support them.
- Pros: This is generally a very strong team building solution. It keeps all employees learning together. It also keeps development teams focused on delivering high-quality solutions that require less maintenance.
- Cons: Maintenance work can put a sudden stop to project work as key resources are taken off of projects to provide support. Really strong resources are not provided as much of a forum for excelling on projects – these are the resources you want to insure you keep on your team.
There are many ways of organizing your resources that could lead to success. It would take a small book to share every effective option, all of the variables, and provide a formula that you could use to make sure you arrive at the right one for your need. My advice if you are unsure or struggling is to get some expert advice and guidance. In part I of this series, I outlined key business and IT resources as well as some organizational considerations. I would like to hear from those of you that have successful examples of organization structures that you feel are unique and would like permission to share your examples as a follow-up to this article series.
About the Author
Bruce has over 20 years of IT experience focused on data / application architecture, and IT management, mostly relating to Data Warehousing. His work spans the industries of healthcare, finance, travel, transportation, retailing, and other areas working formally as an IT architect, manager/director, and consultant. Bruce has successfully engaged business leadership in understanding the value of enterprise data management and establishing the backing and funding to build enterprise data architecture programs for large companies. He has taught classes to business and IT resources ranging from data modeling and ETL architecture to specific BI/ETL tools and subjects like “getting business value from BI tools”. He enjoys speaking at conferences and seminars on data delivery and data architectures. Bruce D. Johnson is the Managing director of Data Architecture, Strategy, and Governance for Recombinant Data (a healthcare solutions provider) and can be reached at firstname.lastname@example.org