Data warehousing / business intelligence / analytics programs require a different approach to project management. Quick deliverables, simplified scope, and attention to rapid results are combined in a method called “Extreme Scoping”
Every organization needs a methodology to manage projects and develop / integrate applications. Traditional waterfall methodologies don’t work for data warehouse (DW) and business intelligence (BI) projects because they are much too rigid. It is common for seasoned project managers who use a traditional methodology on a DW or BI project to feel completely out of control. The requirements appear to be a “moving target,” communication between staff members takes too long, assigning tasks in a traditional way seems to result in too much rework, etc.
Ultimately, the business users pressure the project teams for quick deliverables (90 days or less) while they refine their requirements. As the project team scrambles to meet those user expectations, data standardization is skipped, testing is cut short, documentation is not done, and quality is compromised. Sound familiar? So, how can an organization “have the cake and eat it too?” Try an agile project management approach called “Extreme Scoping.”
Waterfall vs. Extreme Scoping Foundations
Extreme scoping and traditional waterfall methodologies have one thing in common. They both support the notion that the earlier mistakes are caught in the development process, the cheaper and less complicated it is to fix them. Waterfall methodologies accomplish this with a rigorously followed sequence of deliverables: project charter (planning document), requirements definition (requirements document), external specifications (analysis document), internal specifications (design document), code reviews and testing, and finally user acceptance test (UAT). Each of these deliverables must be approved formally by the end users to control scope creep. However, there are always changes to the requirements, even on traditional operational projects.
Traditional waterfall methodologies discourage scope creep, especially in the later phases when the application is coded and tested. After all, making changes during application design is bad enough, but once the team has to alter the code – well, forget the project deadline!
For DW/BI/analytics projects, the foundation is constant change. Using the traditional approach will deliver anything in 90 days or less. A different approach is necessary, since “it is not logical to expect the results to be different if we do not change our ways to produce them.” That means changing the perception of the value of a methodology for DW/BI/analytics, and the way that DW/BI/analytics projects are organized and completed.
Breaking Traditional Software Release Concept
The first thing to do is to break the habit of delivering any application with one project. While most organizations realize that they cannot build an entire DW/BI/analytics environment at once, they do not realize that they cannot and should not build any application all at once for many reasons. These reasons include but are not limited to: requirements may be unstable; the scope may be too large or the deadline is too tight; some technology components might be unproven; data integration complexity may not be understood; and the quality of the source data can be poor, and its business rules may not documented or even known; the project team may be too small or too large or untrained; and business users may be too busy to be full-time members on the core team.
With a project like this, following a traditional methodology and staying on paper for weeks or months is a project death sentence. It is essential to produce tangible results quickly as possible. That means prototyping, prototyping, and more prototyping. How does a team prototype an entire application without falling back into the waterfall sequence of requirements, analysis, design, coding, testing, without returning to the traditional challenges?
Multiple Software Release Concept: “Extreme Scoping”
There is another way: break the application into multiple software releases. Each software release becomes a project. The scope of each software release will contain only a small portion of the application requirements, hence the name “extreme scoping.” Each project is developed using an iterative prototyping approach. Each prototype is time-boxed (anywhere from 10-60-90 days). Most software releases should produce a production-worthy deliverable (although there are occasions where it is necessary to refine [“refactor”] a deliverable in the next software release before putting it into production). After several software releases, the team will have a completed and fully-functioning application.
The quality of the application may be higher with this approach than with a traditional waterfall approach because mistakes can be found and fixed earlier in the development process. With this approach the development process crosses the application scope vertically, instead of horizontally across the development phases. This change allows manage improvements: the requirements to be refined with each software release until they become stable; the scope of each software release can be small enough to fit into a tight deadline; technology components can be evaluated earlier; data integration complexity can be handled in small increments; the quality of the source data can be vigorously profiled, finding and documenting the business rules; the project team can be small during the early software releases, with minimal impact on the entire application; and business users often are more involved in prototyping activities than they are on traditional projects where they hand over their service requests and wait for User Acceptance Testing.
In the traditional approach, many mistakes cannot be found early, and would be found once the entire application is in production. With the “extreme scoping” approach, these mistakes can be found and corrected during the development process, before the entire application is completed. Smaller projects, with focused teams, working on extremely scoped requirements, often have more pleasant interactions between IT and business users than traditional methods.
Extreme Scoping Software Release Planning Process
Extreme scoping project planning is slightly different from traditional project planning. Traditionally, the project manager reviews the methodology and extracts the activities and tasks that are needed to build the entire application into a work breakdown structure. The project manager uses the work breakdown structure as a guide to create a detailed project plan, which typically is arranged in the sequence of the development phases of the methodology: requirements definition, analysis, design, coding, testing, and implementation. The detailed project plan is a multi-page Gantt chart, which is used to guide the day-to-day work activities, manage the change control process, and report the project status to management.
With the “extreme scoping” approach, the project management function is performed by the entire core team, not by a single project manager. The core team members review the methodology and extract the activities and tasks needed to complete the entire application into a work breakdown structure. Using the work breakdown structure as a guide, the core team members create a high-level project plan (not 30-40 pages) to give them broad estimates and an understanding of the overall effort, resources, cost, schedule, risks, and assumptions for the entire application.
This detail is necessary to develop the right number of software releases, the right sequence of those releases, the dependencies among the requirements, and thus, the deliverables and scope for each release. Without this crucial step, the process of breaking an application into software releases (and separate projects) would be completely arbitrary, with no foresight and no control over the whole development process for the entire application. It would be like throwing darts at a calendar and picking the dates hit by the darts as the software release dates, with the requirements and deliverables for each software release following.
Once the core team members are comfortable with the scope and sequence of the proposed software releases and are confident that each software release can be achieved within the allotted time-box (deadline), they create a detailed project plan with weekly milestones for the first software release. Starting with the deadline and working backwards, the core team members determine where they must be the week before the deadline to make the deadline, or put another way, they determine in what state the project or deliverable must be the week before the deadline. They name and describe that milestone, and then repeat the process by backing up another week and another week, etc. If they pass the project start date, the core team members must determine if the scope is too large for the deadline or if the time periods between the milestones are overestimated. All team members must agree that the milestones are achievable, and that quality will not be compromised, given the scope and resources.
After the project activities for the first software release are organized into milestones, the core team members self-organize into the appropriate number of work teams. Knowing the makeup of the work teams and understanding the weekly milestones, the core team members decide on the detailed tasks and task deliverables for each milestone by referring to the work breakdown structure. They also decide how to assign the tasks and deliverables for this release. The detailed daily task assignments and task deliverables are documented on a white board, a flip chart, a spreadsheet, or other informal media, which can be modified quickly and easily, since detailed 30-40 page project plans created with professional project management tools are too cumbersome to maintain. The core team members use this informal detailed project plan daily to guide the work activities, manage the change control process during prototyping, and monitor the progress of the project. They do not use this detailed plan to report the project status to management. Instead, they create a short 1-2 page Gantt chart showing only the milestones. If a milestone is delayed and the core team members are confident that they can catch up the following week, the delay is not reported to management. However, if the delay has a domino effect and the next milestone is missed, then the delay is reported to management along with a proposed course correction.
Extreme Scoping Release Adjustments
If the first software release was completed on time and without unanticipated problems, the core team members can plan the second software release in the same manner. However, if there were problems with the first software release, such as underestimated tasks, incomplete deliverables, friction on the core team, constant adjustments to the scope, etc., the core team members must review and adjust the high-level project plan for the entire application. They must revisit their broad estimates and their understanding of the overall effort, resources, cost, schedule, risks, and assumptions for the entire application. Then they must make the necessary adjustments to the remaining software releases. These changes can include altering the scope for the second software release, changing the number of software releases, re-prioritizing and varying the sequence of the software releases, modifying the deliverables for one or more software releases, changing the deadlines, and so on. Only then can the core team proceed with the detailed planning of the second, and future, software releases.
Conclusion
In summary, the popular concept of iterative development applies not only to developing a data warehouse / business intelligence / analytics effort, but it should be practiced at the application level as well. The traditional definition of a project is to deliver a completed, fully-functioning application. However, for DW/BI/analytics, and for applications in general, a project should equate to a software release with smaller scope so it requires multiplesoftware releases, i.e. multiple projects, to deliver a completed, fully-functioning application.
