Extreme Scoping - An Agile Project Management Approach
By Larissa Moss
In my last article “We Don’t Need No Stinkin’ Methodology,” I illustrated the need for using a methodology to manage projects and develop applications. My story also pointed out that traditional waterfall methodologies don’t work for data warehouse (DW) and business intelligence (BI) projects because they are much too rigid. It is not uncommon for seasoned project managers who use a traditional methodology on a DW or BI project to feel completely out of control. The requirements appear to be a “moving target,” communication between staff members takes too long; assigning tasks in a traditional way seems to result in too much rework; and so on. To top it all off, the business users are pressuring the project teams for quick deliverables (90 days or less) as they are still “fine-tuning” their requirements. As the project team scrambles to meet those user expectations, data standardization is skipped, testing is cut short, documentation is not done, and quality is compromised. Sound familiar? So, how can you “have the cake and eat it too?” Try an agile project management approach that I call “extreme scoping.”
Extreme scoping and traditional waterfall methodologies have one thing in common. They both support the notion that the earlier you catch mistakes in the development process, the cheaper and less complicated it is to fix them. Waterfall methodologies accomplish this with a rigorously followed sequence of deliverables: project charter (planning document), requirements definition (requirements document), external specifications (analysis document), internal specifications (design document), code reviews and testing, and finally user acceptance test (UAT). Each of these deliverables must be signed off in blood by the end users in order to control scope creep. But as we all know, there are always changes to the requirements, even on traditional operational projects. Traditional waterfall methodologies discourage scope creep, especially in the later phases when the application is being coded and tested. After all, making changes while we’re still designing the application on paper is bad enough, but once we have to touch the code – well, you might as well forget your project deadline now!
In the DW and BI world, it’s all about change. It’s about constant change. Thus, using the traditional approach, we would never get off paper much less deliver anything in 90 days or less. We need a different approach, or as Mr. Spock from Startrek might say:”it is not logical to expect the results to be different if we do not change our ways to produce them.” That means changing how we organize our projects and how we use a methodology.
Software Release Concept
The first thing to do is to break the habit of delivering an application with one project. While most organizations realize by now that they cannot build an entire DW all at once, they don’t realize yet that they cannot and should not build an application all at once either. Why? Because the requirements are most likely unstable; the scope is probably too large or the deadline is too tight; some technology components might be unproven; data integration complexity is most likely not fully understood; not to mention the quality of the source data, which probably has not been vigorously profiled because many (if not most) business rules are not documented or even known; the project team may be too small or too large or untrained; and business users are probably too busy to be full-time members on the core team. With a project like this, following a traditional methodology and staying on paper for weeks or months is the “kiss of death.” You have to “get physical” as quickly as possible. That means prototyping, prototyping, and more prototyping. But how do you prototype an entire application without falling back into the waterfall sequence of requirements, analysis, design, coding, testing? You’d be right back where you started.
Here’s how you do it. You break the application into multiple software releases. The scope of each software release will contain only a small portion of the application requirements, hence the name “extreme scoping.” Each software release becomes a project. Each project is developed using an iterative prototyping approach. Each prototype is time-boxed (anywhere from 10-60 days). Most software releases should produce a production-worthy deliverable (although there are occasions where you want to further refine [“refactor”] a deliverable in the next software release before putting it into production). After several software releases, you will have a completed and fully-functioning application.
The quality of the application will probably be higher with this approach than with a traditional waterfall approach because mistakes can be found and fixed early in the development process! Except with this approach we have sliced the development process vertically across the application scope instead of horizontally across the development phases. This allows the requirements to be refined with each software release until they become stable; the scope of each software release can be small enough to fit into a tight deadline; technology components can be evaluated early on; data integration complexity can be handled in small increments; the quality of the source data can be vigorously profiled because you will incrementally find and document the business rules; the project team can be small or even in training during the early software releases with minimal impact on the entire application; and business users are usually more involved in prototyping activities than they are on traditional projects where they hand over their service requests and wait for UAT. In addition, many mistakes cannot be found on paper and would only be found once the entire application is in production and being used by the business users. With the “extreme scoping” approach, these mistakes can now be found and corrected during the development process before the entire application is completed. And the “icing on the cake” is that these types of projects are much more fun than traditional projects because of the core team dynamics, which I will describe in my next article.
Software Release Planning Process
Creating a project plan using “extreme scoping” is slightly different from the traditional way. Traditionally, the project manager reviews the methodology and extracts the activities and tasks that are needed to build the entire application into a work breakdown structure. The project manager then uses the work breakdown structure as a guide to create a detailed project plan, which is typically arranged in the sequence of the development phases of the methodology: requirements definition, analysis, design, coding, testing, and implementation. The detailed project plan is usually a 30-40 page Gantt chart, which is used to guide the day-to-day work activities, manage the change control process, and report the project status to management.
With the “extreme scoping” approach, the project management function is performed by the entire core team (the makeup of which I will describe in my next article), not by a single project manager. The core team members start out by reviewing the methodology and extracting the activities and tasks that are needed to complete the entire application into a work breakdown structure. Using the work breakdown structure as a guide, the core team members create a high-level project plan (not 30-40 pages) to give them broad estimates and an understanding of the overall effort, resources, cost, schedule, risks, and assumptions for the entire application. This is necessary in order to come up with the right number of software releases, the right sequence of those releases, the dependencies among the requirements, and thus, the deliverables and scope for each release. Without this crucial step, the process of breaking an application into software releases would be completely arbitrary, with no foresight and no control over the whole development process for the entire application. It would be like throwing darts at a calendar and picking the dates hit by the darts as your software release dates. And Lord only knows how you would pick the requirements and deliverables for each software release.
Once the core team members are comfortable with the scope and sequence of the proposed software releases and are confident that each software release is doable within the allotted time-box (deadline), they create a detailed project plan with weekly milestones for the first software release. Starting with the deadline and working backwards, the core team members determine where they must be the week before the deadline in order to make the deadline, or put another way, they determine in what state the project or deliverable must be the week before the deadline. They name and describe that milestone, and then repeat the process by backing up another week and another week and so on. If they pass the project start date, the core team members must determine if the scope is too large for the deadline or if the time periods between the milestones are overestimated. All team members must agree that the milestones are achievable, and that quality will not be compromised, given the scope and resources.
After the project activities for the first software release are organized into milestones, the core team members self-organize themselves into the appropriate number of work teams (more on self-organizing project teams in my next article). Knowing the makeup of the work teams and knowing the weekly milestones, the core team members decide on the detailed tasks and task deliverables for each milestone (referring to the work breakdown structure they created earlier). They also decide which tasks and deliverables are assigned to what person on what work team. The detailed daily task assignments and task deliverables are documented on a white board, a flip chart, a spreadsheet, or other informal media, which can be modified quickly and easily. (Detailed 30-40 page project plans created with professional project management tools are too cumbersome to maintain.) The core team members use this informal detailed project plan on a daily basis to guide the day-to-day work activities, manage the change control process during prototyping, and monitor the progress of the project. They do not use this detailed plan to report the project status to management. Instead, they create a short 1-2 page Gantt chart showing only the milestones. If a milestone is delayed and the core team members are confident that they can catch up the following week, the delay is not reported to management. However, if the delay has a domino effect and the next milestone is also missed, then the delay gets reported to management along with a proposed course correction.
If the first software release was completed on time and without unanticipated problems, the core team members can plan the second software release in the same manner. However, if there were problems with the first software release, such as underestimated tasks, incomplete deliverable, friction on the core team, constant adjustments to the scope, and so on, the core team members must review and adjust the high-level project plan for the entire application. They must revisit their broad estimates and their understanding of the overall effort, resources, cost, schedule, risks, and assumptions of the entire application. Then they must make the necessary adjustments to the remaining software releases. That can include changing the scope for the second software release, changing the number of software releases, reprioritizing and changing the sequence of the software releases, changing the deliverables for one or more software releases, changing the deadlines, and so on. Only then can the core team proceed with the detailed planning of the second software release.
In summary, the popular concept of “iterative development” applies not only to developing the DW in small increments, but it should be practiced at the application level as well. While traditionally the definition of a project was to deliver a completed, fully-functioning application, in the DW/BI space a project equates to a software release, and it takes multiple software releases, i.e. multiple projects, to deliver a completed, fully-functioning BI application. If you want to learn more about “extreme scoping,” I’m afraid you’ll have to wait for my next book. I am hoping to publish it in 2008, provided I finally get started on compiling and organizing my materials on this subject. In the meantime, come and attend the project management and methodology seminars that I teach at various conferences.
About The Author
Larissa Moss is the president of Method Focus Inc., a company specializing in improving the quality of business information systems. She has 26 years of IT experience, with a focus on data warehousing for the past 18 years. She frequently speaks at conferences worldwide on the topics of data warehousing, business intelligence, information quality, project management, development methodologies, and other information asset management topics, such as enterprise architecture, data integration, and information quality. She is co-author of Data Warehouse Project Management, Impossible Data Warehouse Situations, Business Intelligence Roadmap: The Complete Project Lifecycle for Decision Support Applications, and Data Strategy. She can be reached at firstname.lastname@example.org.