Managing Data Warehouse Schedule Expectations
By Sid Adelman
There have been a number of data warehouse projects that any outside expert observer would have labeled as successful. Within the organization, however, the projects were considered failures. They were considered failures because management and users had much higher expectations than what was actually achieved. There is nothing more critical to the success of a data warehouse than managing the expectations of users and management.
There are five major areas of expectations: schedule, cost, function/usability, service level agreements (for performance and availability), and expandability. This piece will focus on the expectations for schedule.
Management has expectations as to when the project will be completed. Statements such as “putting a stake in the ground” with a specific date attached is a management technique designed, ostensibly, to give the data warehouse team motivation and incentives for working hard. These dates are usually developed in a vacuum without any understanding of the tasks involved and the duration of those tasks. The data warehouse project manager will often accept those dates because there is no basis to refute the schedule. If the project manager had a project plan, there would be a basis for discussion and negotiation.
Once the project manager has a good idea of how long the project should take, she is in a position to face management with a set of alternatives. In no case should the unrealistic schedule be accepted with a “let’s give it our best shot” attitude. The results are predictable. The project will slip, functions will eventually be deleted or the quality of deliverables will be below standard or there may a horrible combination of all three. In any case, the project will be considered a failure.
There are alternatives, some more acceptable than others.
Alternative #1 Cut the Scope
The first alternative is to stay with management’s schedule and deliver the first phase on time but deliver only a subset of the initial functions. This assumes that the deferred functions will be delivered in subsequent phases and that a phased approach is acceptable to management.
The project manager must be careful to assure that, even with the reduced function, the first phase can be completed on time and that the functions delivered have real and measurable value. Two areas in particular must be examined very carefully: training needs and dirty data. These areas are chronically underestimated.
Alternative #2 Renegotiate
The second alternative is to negotiate for a more realistic schedule. Some schedules are hard and fast (Year 2000, governmental reporting requirements,…) but others are without any underlying real requirement (a fast-track manager may want to demonstrate quick results). Determine the underlying reason for the imposed schedule and let it determine your negotiating strategy.
Alternative #3 Add Resources
The third alternative, throwing more resources at the problem has perils and inherent costs. Adding more people, be they employees, contractors, or consultants, will often delay a project, as the new people have to be brought up to speed, trained and managed. This can mean that valuable resources are pulled away from productive activities. In certain cases, however, the right resources, if they are trained, experienced and familiar with the project, can be of significant help.
Another type of resources to add could be software that would improve the productivity of key activities. An example would be data quality software that would help automate the process of cleansing and transforming the source data and software to facilitate prototyping. The project manager should be careful not to accept the vendors’ claim of improved productivity but should check these claims with reliable references. Be aware that new software requires time to install, it requires time to train the employees who will be devoted to the software, and requires time before the product is fully functional. The learning curve for a functionally rich ETL tool can be as high as three to four months.
A word of caution, the project manager should be careful about accepting new resources and should not accept a shortened schedule as a trade-off unless the resources are the right ones and can truly impact the timeline.
Alternative #4 Taking Shortcuts
The next alternative is not really an alternative, but is often an inevitable sequence of actions that result from the pressures of arbitrary and unrealistic deadlines. We have seen cases when, in the name of expediency, quality was allowed to suffer. Shortcuts were taken in the following ways:
- There was not enough time allocated to recruit a competent team and there was not enough time allowed for training the data warehouse team. Therefore, tasks were performed poorly or had to be redone.
- There was not enough time allowed to develop a project plan or to manage the development effort.
- The time to understand user requirements were severely shortened or the requirements gathering activity just given a cursory effort. The users did not get what they wanted and walked away from the warehouse.
- The effort to determine what source data to migrate to the data warehouse was eliminated. All the production data was dropped into the data warehouse. The result was a bloated data warehouse with poor performance. It was accompanied by poor documentation and with users who didn’t know the meaning of the data they were accessing.
- The time-consuming and labor-intensive task of cleansing the data was bypassed. This resulted in user dissatisfaction with their queries and reports and ultimately in the failure of the project.
- A data model was not created. Subsequent projects were not integrated and the result was disparate data marts with redundant and inconsistent data. It became almost impossible to create a query that accessed more than one data mart.
- Metadata was neither created nor maintained. It was not clear what the reports meant, their timeliness, or the source of the data. Since, in this process, transformation and cleansing were performed without the benefit of metadata, it meant the users were not even able to find or recognize their legacy data which was now stored in the data warehouse.
- There was not enough time allocated to develop and carry out user training. Users had more problems and become frustrated.
- Not enough time was dedicated to testing the system.
- There was no effort exerted to proactively tune the system. Performance was poor.
It should be obvious that the effect of an unrealistic schedule on quality will have a devastating impact and should not be considered as an acceptable alternative.
Alternative #5 Major Overtime
The final alternative should also not be seriously considered. There are managers who plan schedules to include employees working weekends and very long hours during the week. These managers have also been known to ask the team to defer vacation, holidays and needed dental work. While the team may put in the hours, their productivity and the quality of their work will almost always suffer, not to mention their morale. In other words, the managers get the time but not the results. The now disgruntled employees often transfer during or shortly after the project is complete, or leave the company entirely.
If management is aware of the downsides to insisting on unrealistic schedules, they are likely to accept either alternative one or two and may be willing to spend the extra money for alternative three but should be careful to avoid alternatives four and five.
About the Author
Sid Adelman is a principal consultant with Sid Adelman & Associates, an organization specializing in planning and implementing data warehouses, performing data warehouse and BI assessments, and in establishing effective data strategies. He is a regular speaker at “The Data Warehouse Institute” and IBM’s “DB2 and Data Warehouse Conference”. Sid chairs the “Ask the Experts” column on www.dmreview.com, and has had a bi-monthly column in DMReview. He is a frequent contributor to journals that focus on data warehousing. He co-authored one of the initial works in data warehousing, Data Warehousing, Practical Advice from the Experts, and is co-author of Data Warehouse Project Management with Larissa Moss. He is the principal author of Impossible Data Warehouse Situations with Solutions from the Experts and his newest book, Data Strategy, was co-authored by Larissa Moss and Majid Abai. He can be reached at 818 783 9634and email@example.com. His web site is www.sidadelman.com.