Are They Using It?
Building an analytical solution can be an arduous task. There are many factors that are required in order to deliver a successful solution. It takes good planning, a solid architecture and design, good setup and usage of your front end delivery tools, and validation that the data you have retrieved is accurate. While the building of the solution is challenging, ultimately, value is only realized when users leverage the system.
Too often the project to provide the solution is deemed a success, but the usage isn’t there. Rest assured that the success of a project in the short term may be evaluated by whether it was on schedule and budget. However, in the long run, every effort is judged much more by the level of usage and value received from the solution. I once had someone tell me that they delivered their system, taught their users how to write SQL themselves, and it’s great, we never hear from them. I strongly suggested that they take a look at audit logs and see if anyone is using the system (other than to extract all of the data and download it to Excel). So how do you determine if your system is delivering value?
Let’s start by looking at the real usage that is occurring. It is one thing to estimate what activity users will engage against your database during the requirements phase of your effort. It is quite another to see the level of activity that is actually occurring 3, 6, 9, 12 months or more down the road. Here are some of the many reasons that having an audit trail of all usage of your data solution is important:
- Security Auditing– Protection of the data in your solution is almost always a concern. Oftentimes we define the security we want to protect access to the data, but don’t have someone assigned to monitor the actual access. In healthcare in particular, the protection of access to patient data in a proven, secure manner is crucial.
Performance Monitoring– There are many minor enhancements you can make to your solution that will greatly improve the overall experience of the users. Simply adding an index or an aggregation table for commonly submitted requests can often be the difference between a frustrating user experience and having business resources singing the praises of the tool. Knowing what queries are running really long, which ones are running frequently, and what data is most often requested will allow you to target performance enhancements.
Usage Auditing– This is probably the most challenging, but then also the most rewarding. One key is to leverage usage auditing for the same reason our users rely on business intelligence – to find out how we can improve the service that we provide. What you are looking for is who is using the system (what type of user), for what type of activity, what data are they most commonly going after, etc… This doesn’t just require numbers, it requires analysis and watching the user experience unfold side by side with them. You may discover a tool being used for purposes beyond it’s strength – like reports being used as dashboards, summarized data trying to be used for data mining, etc.
Setting up the tools and tracking of results to enable the ability to audit is only the first step. If it is to be of any value, you must actively engage the appropriate resources to monitor the results. There are many areas and techniques that can be explored to help you insure the success of your solution.
Query Numbers Low – Why?
Now, let’s focus on what to do when you aren’t seeing high levels of user access or queries as per the defined requirements. How you react to this situation is much more important than the fact that you are not getting the level of interaction from users that you expect. First, we need to look at the possible causes for the lack of usage. To investigate this, you may want to use several techniques. Very importantly, make sure you’re getting feedback from your users.
- A survey is a great way to find out who is using the system and who really sees potential in the system.
- If you send the survey out and almost no one returns it, they likely could care less about the solution – this would be a requirements gap or a communications gap. Perhaps leadership has seen great potential benefits, but they haven’t shared the strategy and ideas with the resources actually using the solution. We will assume that you didn’t use the “Build it and they will come” approach to the solution.
- If you get many responses and they are all negative, that shows people really care and see potential value in the solution. There are probably specific nuances that are extremely frustrating to them that are causing their displeasure with the solution. As long as the data is accurate, there are always going to be ways to improve the delivery mechanism.
- If you get a smattering of responses of varying levels of acceptance, you are probably in a great position. You can leverage those that like it to share with others the techniques and results that have caused them to be early adopters. You can also leverage the negative comments to find performance improvement or usage opportunities.
- Watching over their shoulders. Sit with some of your business users and see how they are using the system. This is my favorite mechanism as you get to see first hand what they are looking at. You will quickly find out what they really like – this is critical as you can make sure you don’t change anything that could have a negative impact on something very important. You also will be able to see areas they are struggling that you thought should have been simple or intuitive. Look at the data they go after, look for how they are formatting their output and look for who they share results with and why.
- Talk to business management responsible for the activity. Try to find out what they are hearing, what they are doing to encourage usage, etc… Perhaps there are other initiatives going on with their users that do not provide them with the time or resources to leverage the solution.
- Look at your database access activity. It is important to have some of this to analyze prior to meeting with the users as above. It can help you target who to sit with on the business – I would try watching one of the heaviest volume users, a medium volume user, and a light or no volume user. It can help you identify requests that everyone is running that you might be able to schedule for them so it is there instantly. There might be key things you observe that you can then tie to requirements and watch to see if the users are able to get what was defined in requirements.
Here are a few of the findings you will probably encounter in your research: “The system is SLOW”, “the tool is not user-friendly”, “I don’t trust the data”, “I can’t do the functions I want, can’t I just export it to Excel?”, “I just don’t have time to learn the tool”.
One Key – Usability
Just to focus on one area, I am going to pick on usability. Listen to the complaints you are hearing and witness the actual usage patterns taking place. This will help you define why usage is low and what you can do about it.
One of the first aspects of usability is to determine if you are using the right front end delivery mechanism for the true need. By watching and listening to what the users want to be able to do, you should be able to define if the front end you have provided is really the right tool for the job. I have heard many users ask for a dashboard or scorecard, when what they really were looking for was reports, and vice versa. Ideally, you catch these kinds of requests up front, but too often since the users haven’t seen the solution or used these types of tools before, it only becomes apparent after they have had access to it for a few months.
Next, look for how they navigate the front end access tools and data, and the frustrations that are obvious. Any number of things could become apparent.The tool itself is hard to navigate– this could come from a poor front-end or using the wrong tool for the job.Too many clicks– having lots of options can be wonderful, having to click many times for a simple request can be frustrating.Too technical of a tool– takes a rocket scientist to use it.No meta data– users don’t know what the data means and don’t know where to go to find out more about the data definition, heritage, and lineage.
We build analytical solutions to help us better understand our business and operations. This enables us to apply tactics, change strategies or approaches, and monitor/measure the impact. Likewise, as the designers and builders of these applications, our technical resources need to provide the same support and same desire for improvement that our end users expect. By searching through the activities taking place against our systems and staying closely engaged with our users, we can best serve them and the institutions that compensate us for our service. When the project is over, the real work starts. How you handle the rollout and improvements to your system will often have an equal or greater impact to your success and it will bring a tremendous partnership with the business. How you adjust to the needs identified and focus on the right tools for the job will go a long ways towards providing value that fully engages the business.
About the Author
Bruce has over 20 years of IT experience focused on data / application architecture, and IT management, mostly relating to Data Warehousing. His work spans the industries of healthcare, finance, travel, transportation, retailing, and other areas working formally as an IT architect, manager/director, and consultant. Bruce has successfully engaged business leadership in understanding the value of enterprise data management and establishing the backing and funding to build enterprise data architecture programs for large companies. He has taught classes to business and IT resources ranging from data modeling and ETL architecture to specific BI/ETL tools and subjects like “getting business value from BI tools”. He enjoys speaking at conferences and seminars on data delivery and data architectures. Bruce D. Johnson is the Managing director of Data Architecture, Strategy, and Governance for Recombinant Data (a healthcare solutions provider) and can be reached at firstname.lastname@example.org