Every business intelligence and analytics initiative needs access to the source data. This can be a complex task, but there are ways to make that activity smoother
One commonly overlooked challenge to developing any business intelligence or analytics solution is getting access to the source systems and data. Whether it is just getting permission to get the data, or working with source system resources to acquire the data, this can be a significant challenge.
The Root of the Problem
Every organization has many system and data variables, some industry specific, others appropriate to the organization. The vast number of systems, the unique nature of data, the prevalence of unstructured data, broad technologies, and number of proprietary data platforms are all commonplace in most organizations. Combine this with the strong need for security and protection of data privacy, and any request for access to data can be met with intense scrutiny.
Many companies demand for access to data from many different sources for many different reasons. In addition to financial and operational analysis needs that most businesses face, analysis and decision support are activities that frequently require systems to provide data extracts. Since the need for access to this information is extremely high, most of these needs are never met, with only the most urgent provided in a timely manner. Due to this demand, information technology (IT) departments prioritize those requests and satisfy them one at a time. Thus, for those organizations without an integrated data warehouse, requests for data from many systems may not be satisfied.
Therefore, it is easy to understand how any requests for getting data are met with sincere skepticism and concerns for what will be done with the data. The resources that support these systems feel responsible for the data in their system and often act as a gatekeeper for it usage. Operations staff may not understand the importance of analytics or business intelligence, not giving the requests for data appropriate priority.
Develop the Proper Approach
Getting access to source system data will be littered with roadblocks, but it could also prove to be a major issue that shuts down the analytics projects. To mitigate this condition, have a clear recognition of the challenges ahead and have a sound approach for appropriate access that is communicated/shared across the organization. The most important factors for a proper approach to getting access include:
- Change the perception that application developers own the data in their systems and are the gatekeepers for anyone who wants to use it. While it is important that these resources are accountable for the quality of data in their system, it is the institution as a whole that owns the data. When the business decides it can provide access to many resources faster, easier, and more cost effectively, an integrated. This starts with leadership and must be driven down to all levels consistently.
- Identify and document all relevant metadata, for source data and the data to be sent to the business intelligence / analytics solution.
- Minimize impact on source system resources. Use source system technical support only as needed to understand source structures/schemas. Since the data comes from many disparate sources will mean the employees of each source system must learn and know the tools and techniques to build ETL, and they must approach the ETL process consistently. If this common approach is not implemented, the organization will suffer from cleansing and transformation rules that are not applied consistently.
- Minimize impact on source systems and their SLAs. Do not do all the work when extracting the data. Just get a copy of the data and do all cleansing, staging, transformation, and integration work on the appropriate servers. Doing so can reduce response time issues in source systems, demands on source system servers, contention with outages/downtime, etc. In general, ETL work is much more effective when data is staged on one platform, before applying consistent rules and performing integration tasks. Doing this creates the basis for writing code that is more effective, consistent, and maintainable.
- Have data and access security clearly defined. When talking with IT managers from the relevant areas, provide them with sufficient information to understand the business projects, the business intelligence effort’s purpose and methods, and knowledge of the appropriate security/protection. Show them how their data is protected as it moves to the BI / analytics environment and how security is controlled and audited throughout the process, including the front-end. They will be much more receptive to requests.
Obtain Active Support and Commitment
Even with the right approach, great planning, and strong resources, there will be obstacles to getting access to data sources. One of the best ways to mitigate this eventual roadblock is to have leadership aware of this dilemma and engaged in helping to mitigate it. Any executive leader who is sponsoring an effort will want to know about roadblocks and how they can offer help and support. That is the purpose of executive sponsors. How to help them help:
- Provide background – Help them understand how systems developers are responsible and diligent in securing their systems. Then help them understand how most systems deal with requests for data – show them the number of source systems needed and the reason for a consistent, enterprise approach. Educate them before they approve the effort and alter them to the essential value of the data being requested.
- Suggest a communication method or mechanism that sponsors can use. The goal of the communication is for senior management to understand the business needs, value, and challenges to the effort. Doing so will establish that help is needed to ensure that this roadblock does not arise.
- Help executive leadership to see that data is an asset and that it belongs to the institution. When senior leadership level truly understands this concept, the planning, monitoring, and addressing related issues provides a forum to make these challenges more obvious to executives who can offer responses to reduce them.
- Develop an approach to enterprise data management so data assets are supported by data governance policies, data quality metrics, and other aspects of effective data management.
- Prepare a presentation for the team to use – include some of these key slides:
- Some of the same slides that executive leaders gave for background information
- Team composition and purpose
- Project goals, timelines, and key resources – basic project management information
- Security in the data integration process and into the BI / analytics environment and any tools that access data.
- How can they help?
- Lastly, make sure the team understands that when they run into issues with getting access to systems they should ask for assistance / support from leadership.
Obtain support and interaction from audit and compliance areas. Make sure these resources are added to steering committees and prioritization groups so that they can help address any questions or concerns. They can help the organization by being a key active member of the leadership of this effort. If compliance and security officers support the BI / analytics effort, it will be much harder for anyone to scrutinize security and privacy aspects. If necessary, consider asking for help in engaging and convincing those key resources.
Conclusion
Information Technology is challenged with large numbers of systems, many data complexities, privacy/security demands, and technology platforms of every possible combination. Invariably, getting access to the source system data will be challenging. In planning for a BI / analytics effort, do not forget to include the extra time it will take to gain access to the critical systems that house the data. Also, educate sponsors and leadership of this challenge from the start, so they can help educate those groups and individuals who can supply access to the right data.
