Defining the optimal organizational structure for a data warehouse, business intelligence, analytics program is a critical success factor that starts with the proper foundation.
There are many ways to build IT and business organizations that support data warehousing / business intelligence and analytics efforts, In fact, they are as varied as data warehousing architectures. It is essential that the organization establish a unit / department that works specifically to satisfy the data warehousing / BI and analytics needs and is successful within the organization’s culture. Many organizations struggle to define such a unit and with what resources to staff it.
Some organizations create “Centers of Excellence” for analytics and business intelligence, believing this will solve the problem. However, like defining the right architecture for the data warehouse, defining the right organization to build and support it is not simple.
The optimal DW/BI/analytics organizational structure is a critical success factor. A three-part series examines establishing an effective data warehousing organization.
- Part I, the general components of a data warehousing team.
- Part II, building and retaining a strong team.
- Part III, best practices, critical success factors, and simple things that enhance success.
General Components of DW/BI/Analytics Teams
The foundation begins with outlining the relationship of business and IT resources that are required to work together for success:
BUSINESS
Leadership: A core staple of data warehousing and analytics success is having the right leadership. A successful data warehouse is business-driven and IT supported. This means that IT should not run with an idea and have the “build it and they will come” plan for adoption/usage. There must be a valid business need for a project to build an analytics solution. This does not mean that the business should tell IT what to build.
Sponsor: This is the business champion who leads the support and endorsement for all phases and components of the solution. Ideally, this person is at an executive level.
Owner: This is the business leader/manager who is responsible for realizing business value from the solution that is being designed and delivered.
Business Manager: This is the business leader who is responsible for building and organizing the appropriate business resources required to collaborate with IT on the building and delivery of the solution as well as the adoption of the solution by business resources.
Business Users: These are the resources who will test and actually use the system once it is complete.
There are many other business roles, many of which are specific to the type of solution. It is imperative that the specific roles are connected to the specific business application, processes, and future usage needs.
Information Technology (IT) ROLES
IT Leader: This is the IT champion responsible for supporting this effort and working with business executives to fund and prioritize. This person is usually the CIO / CDO or at the minimum a direct report to the CIO / CDO.
Manager: This is the IT manager who is responsible for staffing, budgeting, planning, sharing status reporting with leadership, and ensuring the overall progress of the efforts.
Project Manager: This resource is responsible for performing project management tasks on individual projects. Ideally, this resource will help develop project processes, templates, and reporting/status tools that can be used by all projects associated with your efforts.
Data Integration: These resources are responsible for the design, delivery, and maintenance of the data that is moving into and through your data warehousing environment. This is typically referred to as ETL (Extract, Transformation, and Load). Data Integration is the most time consuming aspect to most data warehouses and is one of the most frustrating for management resources – both business and IT.
Solution Delivery: These resources are responsible for the design, delivery, and maintenance of the end user information access solutions often referred to as BI (Business Intelligence). They work with the various BI front ends and work directly with data marts. They work very closely with the business to build and enhance their methods of using information.
Database Analyst: These resources are responsible for the design, delivery, and maintenance of the physical database structure. They work closely with Data Integration and Solution Delivery resources to optimize the solutions to users’ needs.
Data Modeler: These resources are responsible for the design, delivery, and maintenance of the data models that are the basis for the data warehouse. This includes working with the business on terminologies and definitions, data examples, relationships, and data governance rules.
Data Quality Assurance: These resources are responsible for two key components prior to turning over to production:
- Data Validation – accuracy of terminology, quality of data, comprehensiveness of data, etc.…
- Front-end solution validation – does it meet requirements, application testing, reports/analytics testing, and security.
One final note on these roles: More effective resources will be able to fulfill several of these roles on a given project, but newer resources should focus on one or at the most two.
Sound Organizational Area Options for IT
With many different ways of building analytical solutions, there is a wide variety of ways to organize those resources. Since every business need and solution can be different, this advice will focus on the IT side of the equation only. Each organization has a culture or method of working that they employ. In very successful organizations, these structures often translate over to data warehousing / BI / analytics well. However, in many organizations struggle with many projects and politics overrules success. Therefore, changes to the organizational structure for DW/BI analytics are necessary to realize value from DW/BI/analytics.
For large efforts, since there are many areas of expertise and typically significant resources, the best practice is to focus on areas of expertise. Many organizations desire that all data warehousing IT resources know every aspect of their solution. While this is an ideal scenario, it is impossible to achieve in large solutions. A few key resources should know the solution end to end at a high level, for practical reasons. As the solution grows, resources should work on various different parts to expand their overall knowledge and value. A best practice for large efforts would have IT resources organized by area of expertise:
- Data Integration
- Solution Delivery (BI)
- Data Modeling and Data Governance
- Data Quality Assurance
- Project Management
Often, these groups are referred to as: Center of Excellence, Competency Center, Shared Service, Best Practice, etc.… This type of structure can reduce cost and increase value since it can leverage focused toolsets, single infrastructures, development processes, consistent security, etc.… In this case, having strong, seasoned managers and a very strong architect is critical.
For small to medium efforts, it is important to recognize the desired solution and future phases of it. Typically, in this sized effort, individuals have more involvement in multiple components of a solution. In some small efforts, an individual or two can design, develop, and effectively deliver all components of the desired solution. Ideally in these scenarios, a team of resources learn most of the layers, with some focused specialties that are too challenging and not worthy for all to learn – like a DBA and a Data Modeler. In these cases, having a seasoned technical manager guide this group is crucial.
Handling Support and Development
This is an oft-debated topic. There are two main approaches, with several variations possible. Each of these types of organizations can work successfully, and each type has challenges.
- Some resources only do development while others only support what has established
- Pros: This can be a valuable learning process for new employees. When you put them into the maintenance side, they will gain more experience on your solution faster than would otherwise be possible. With experienced resources, it will provide them with the opportunity to stay focused on delivery.
- Cons: This can alienate resources who are relegated only to maintenance. They do not get the glory or learning that is inherent in development. There is also the potential negative impact of development resources moving very quickly to throw solutions out, even of low quality, since they know they are not the ones that will have to support them.
- All resources perform both development and support
- Pros: This is generally a very strong team building solution. It keeps all employees learning together. It also keeps development teams focused on delivering high-quality solutions that require less maintenance.
- Cons: Maintenance work can put a sudden stop to project work as key resources are taken from projects to provide support. Strong resources are not provided a forum for excelling on projects – these are the resources to keep engaged and productive on a team.
Conclusion
There are many ways of organizing DW/BI/analytics resources that could lead to success. Each could be the optimal solution for any organization, with the key business and IT resources noted. The organizational considerations should not be overlooked when building the foundation of any DW/BI/analytics team.
