Examining the Value of Project Reviews and Post-Mortems

All projects can benefit from an impartial final review, also called a “post-mortem,” to learn about successful activities, to improve upon mistakes, and to develop skills for future projects

Many project managers dread project reviews because it meant that the system that was just implemented was DOA (dead on arrival).  Most project teams do not have to wait until the system went into production to know that it was in trouble.  However, many teams do not have time to address the telltale signs of a failing project: missed deadlines, managers making not-so-subtle threats about career and employment uncertainties, etc.  So, in many cases, the project was completed, jobs were maintained, and management was content until something happened.  The system had crashed.  It was time for a post-mortem!  Management wanted to know what went wrong, when it apparent, who was responsible, and how the situation could have been avoided.  Unfortunately, all of the answers came too late to prevent the system from failing in the first place.  In addition, the lessons learned were rarely taken into consideration on the next project and were almost never shared with other project teams.  What are the benefits of post-mortems?  What is done is done.  Perhaps it is best to fix the current situation and move on to the next system.

An organization could take the opposite approach and apply lessons learned during project development to improve all forms of quality.  This approach is especially appropriate in data warehousing / business intelligence / analytics, where solutions are created incrementally in small releases after the initial phase.  Since this is a process of continuous refinement and expansion, there are benefits to periodic project reviews that increase the quality of deliverables and improve development process.

Software Release Concept

The software release concept breaks a system development project into multiple software releases.  Each software release becomes a separate project.  Each project is developed using an iterative prototyping approach.  Each prototype is time-boxed (anywhere from 10 to 120 days).  Each software release should produce a production-worthy deliverable.  After several software releases, there is completed and fully-functioning application.  In other words, the idea of “get it right the first time” has never worked, regardless how many times project teams performed post-mortems. This is especially true for data warehousing / business intelligence / analytics efforts.

Project Retrospectives

The term retrospective is very popular in agile methodologies like Scrum or XP (eXtreme Programming).  It refers to a project review that is conducted after every software release of an application under development.  Unlike a post-mortem, a retrospective is performed regardless whether the deliverable from a software release runs perfectly or has problems.  It is imperative that lessons are learned during the development of an application to improve its quality and to increase the speed of the development process.

Project retrospectives are also excellent forums for IT managers and business executives to become comfortable with the dynamic development process and the software release concept of BI / analytics applications.  In addition, retrospectives are excellent venues for sharing the lessons learned with other project teams as well as with other business managers in the organization.

Topics for Project Retrospectives

To be effective, no topic should be off-limits for project retrospectives.

Schedule

• Were we able to deliver all of the requirements we had planned for the last release?  If not, why not?
• Was the scope realistic?
• Was the schedule realistic?
• What slowed the project down?
• How can delays be prevented in future releases

Budget

• Is the application staying within budget?  If not, why not?
• Is the budget realistic?
• How can cost overruns be prevented in future releases?

Satisfaction

• Are the users seeing the benefits they expected?
• Are the BI tools satisfying the analytical business needs?
• Are users satisfied with the data so far?  If not, why not
• Are the users satisfied with the functionality so far?  If not, why not?

Scope

• Were scope changes requested during the last release?  Why?
• Were scope changes made to the overall application because of the last release?
• Was the impact analyzed and measured?
• What was the impact?
• Could the changes have been anticipated or avoided?
• What did we learn about scope changes and the existing change management procedure?

Negotiating Skills

• Did anything in the last release have to be renegotiated with the users (scope, schedule, resources, budget, and quality)?
• Was the renegotiating process with the users painless or did it create friction between the business people and IT?
• What needs to be done to improve the renegotiating process in future releases?

Staffing

• Did we lose any key people during the last release?  Why did they leave?
• What was the impact of their departure?
• How can we avoid that type of loss in future releases?
• Was the core team staffed properly?
• Were there too few or too many team members?
• Were the roles and responsibilities assigned appropriately?
• Did the team work well together?
• Was there friction?
• What was the reason for the friction?
• How can team spirit and team morale be increased and / or maintained in future releases?

Skills and Training

• Were the skills of the team members sufficient?
• Was “emergency training” required during the last release?
• Was the training effective?
• What should be done differently next time?

Project Planning and Reporting

• Did the team track their “actual time” truthfully?  If not, why not?
• Were the activities estimated correctly?
• If not, do we know why they were overestimated or underestimated?
• Does our procedure for tracking time and reporting include project status work?
• How can we improve it?
• What other lessons were learned about project planning, tracking, and reporting?

Development Approach

• Were the appropriate steps, activities, and tasks selected from the methodology?  If not, why not?
• Were important tasks left out?
• Were unnecessary tasks included?
• Did we use prototyping or other agile development techniques like XP or Scrum?  If not, why not?
• Did the agile development techniques work for us?  If not, why not?

Contractors, Consultants, Vendors

• Were outside consultants or contractors used?
• Were they used effectively?  How was “effective” measured?
• Did they take time to transfer their knowledge to our staff?
• What lessons did we learn from negotiating with outside vendors?
• Are the vendors following the rules or trying to go around them?
• How can that situation be controlled or improved in future releases?

General

• Is communication effective through the team?  How was “effective” measured?
• Were business representatives available when needed?  Were the “right” business representatives allocated sufficiently to the project?  If not, why not?
• What other lessons did we learn?
• What should be done differently in future releases?

Organizing Project Retrospectives

Project retrospectives can be formal or informal.  During the development of an application, most retrospectives can be informal and limited to the core team and the participating users, including the business data stewards.  However, the final retrospective should be a formal meeting to which additional stakeholders are invited.  In either case, consider the following items when organizing a project retrospective.

Preparation

• The issues log must be examined to see which issues were effectively resolved and which ones were not.
• The change control procedure should be assessed for its effectiveness.
• The project plan should be reviewed to determine if all the appropriate tasks were included.
• The estimated and actual task completion times on the project plan should be studied to determine which tasks were underestimated and which were overestimated.
• Any problems with the technology platform should be noted, such as problems with tools, vendors, hardware, network, etc.
• The budget should be reviewed to see if the actual expenditures came close to the estimated ones.
• The effectiveness of the training sessions should also be assessed.
• All these items are potential topics for discussion at the project retrospective.

Frequency and Timing

• While the application is under development, schedule the retrospective the day after the software release is declared done, regardless whether done means that it went into production or not.
• Once the application is completed (after the final software release), it may be advisable to wait for two or three weeks before holding the final formal review.  This will give the project team time to iron out the final glitches in production.  It will also give the project manager and the business sponsor time to:
• Review the project charter, project plan, project reports, project activities, and the budget
• Collect information and metrics about the usage of the BI application, the DW databases, and the metadata repository
• Organize the meeting

Meeting Venue

• Most project retrospectives can be held onsite in a large conference room, or virtually using teleconference and video conference facilities.  Virtual attendees should be asked to focus attention on meeting
• Cell phones should be used for emergencies only and should be set to “vibrate.”
• The room should be set up as a conference room, and should be supplied with sufficient materials to facilitate a successful meeting (white boards, computers and presentation software, sufficient refreshments if onsite, etc.) 

Duration

• A well-organized and frequently performed retrospective can be accomplished in a few hours.
• The final formal retrospective (and occasionally an interim retrospective) may last a full day, with the option of a follow-up session within a week, if necessary.

Attendees

All team members from the core team, including the development track teams, and the extended team should be invited to participate in the project retrospectives.

Team members must be prepared to contribute.  That means that they must review the agenda and be prepared to discuss the topics listed on it.  They must also review any documents and be prepared to discuss them.  In short, every project team member should be an active participant!

Content

An agenda must be prepared in advance for each project retrospective.  Include the following items:

• A list of all topics to be discussed, including introduction and wrap-up, with estimated time allocations for each topic.  The time estimates must take into account the complexity of the topic and the number of people participating.
• A list of attendees.  Everyone who is invited should be given the opportunity to add to the agenda and to ask for any pertinent documents to be reviewed.

Session Flow for Project Retrospectives

Project retrospectives are structured and follow a prescribed procedure, which the attendees should follow for success.  Certain people have the responsibility for conducting certain parts of the meeting.

Business Sponsor

It would be best if the business sponsor could open the meeting with a brief acknowledgement and appreciation for the last software release deliverable before turning the meeting over to the project manager.

Project Manager

The project manager should discuss the flow, the rules, and the expectations of the project retrospective, and turn the meeting over to a skilled facilitator.

Facilitator

The facilitator should lead the group through the topics on the agenda.  The responsibilities of a facilitator include:

• Ask the person owning a topic on the agenda to introduce the topic.
• Solicit comments and feedback from the other participants.
• Assure that the meeting does not focus solely on one topic.

 Monitor the allocated time for each topic and interrupt the discussion when the time limit has been reached.  At that point, the facilitator must ask the project manager for a decision on whether to schedule a follow-on meeting or to continue the discussion at the expense of dropping another topic on the agenda.

Scribe

A scribe is a person who was not involved with the project.  The main purpose for having a third-party scribe is to have a knowledgeable but neutral note taker who can:

• Document the highlights of all conversations and comments.
• Document identified action items and to whom they were assigned.

At the end of each project retrospective, all action items are reviewed and the person to whom an action item was assigned will give an estimate for a completion date or a reply date (a date on which an estimate will be provided for the effort to complete the action item).  The group must decide who will follow up on the action items, and whether an additional working session is necessary and schedule it.

Conclusion

As George Santayana once said: “Those who cannot remember the past are condemned to repeat it.”  .  This statement is as applicable to projects as it is to life and politics.  To know how to improve the next project, and the next software release, one has to learn from the mistakes made on the last software release.  The project retrospective is the vehicle to discover the mistakes and to take corrective action before completing the application.  Excluding this step would result in repeating the same mistakes in an environment that is growing rapidly and is affecting more people with each release.  Correcting mistakes on small systems is much easier than correcting mistakes on large systems.