By Larissa Moss

In my last article, I explained how I chose my column name: Back to Basics. As an example, I mentioned how amazed I was to discover that many companies don’t have a data strategy; something I consider to be very basic. I am equally astounded that many companies don’t require a methodology for project planning or application development.

Yes, I have heard the complaints: methodologies are a thing of the past; methodologies belong with the IT-dinosaurs back in the 1970’s; good riddance to the rigidity of methodologies and all that paper work; dynamic teams don’t need no stinkin’ methodology; and so on. I was around during those dinosaur days, and I’ll be the first to agree that those were not “the good old days.” But I’ve also been around long enough to know not to throw the baby out with the bathwater. To illustrate an extreme example of a team that operated without any guidance of a methodology, and to share what I learned from that experience, I will recall a true story that happened about six years ago.

Agility versus rigor

I was asked by a barely surviving dot-com (now dot-gone) company to conduct a methodology/project management seminar for their very young and very enthusiastic staff. When I arrived one hour prior to the seminar, the workbooks were not ready yet, but “on their way” from Kinko’s. Once the workbooks arrived 15 minutes prior to class, I noticed that they were collated and bound incorrectly. Rather than causing panic or even mild concern, this situation raised the level of excitement among the staff. They jumped on their skateboards and rolled in “to the rescue.” They manually re-collated and re-bound all 50 workbooks in a record time of 25 minutes. When they were done, there was cheering in the hallways, and “high fives” were exchanged among the “heroes” who “pulled off the impossible” one more time. It was immediately clear to me that “constant fire fighting” was how they worked on their projects, and that this was the very reason why I was brought in. But I must also admit that I was delighted to see their teamwork, their enthusiasm to solve the problem, their ability to reorganize themselves intuitively, and to “pull off” what seemed impossible. There was a lesson here for me to learn.

I share this experience because it demonstrates the potential productivity that could come from the “thrills of chaos” (agility) if channeled effectively, and the devastating effects of killing enthusiasm and morale that could result from the “dregs of structure” (rigor). In fact, to continue my story, I abandoned my lesson plan for this group and started with an exercise of two questions: “What are the thrills of chaos?” and “What are the dregs of structure?”

The first question, “What are the thrills of chaos?” produced these answers:

  • It’s like a “high”
  • Unlimited creativity
  • You are free, unrestrained
  • Spontaneity
  • Camaraderie
  • Be a hero
  • Stand out and be noticed
  • Boost to self-esteem
  • Excitement, fun
  • Defiance
  • Sense of accomplishment
  • You can hide your incompetence

There is an indisputable payback for operating in chaos. Whether the payback is a boost to one’s self-esteem or to hide one’s incompetence, there are distinct personal benefits, which spur people on to accomplish “missions impossible.” It would be counter-productive to try and squelch these paybacks. On the contrary, these paybacks must be considered and carefully architected into any structure or discipline, such as a methodology.

The second question, “What are the dregs of structure?” produced an even longer list of answers:

  • Oppressive
  • Big brother
  • Being under somebody’s thumb
  • Kills creativity
  • Boring
  • Never works anyway
  • Wastes time
  • Suffocating
  • Military
  • The establishment
  • Being old and stale
  • Dictatorship
  • No individualism
  • Choking, controlling
  • Un-American

These fears and anxieties are even greater motivators to defend the “thrills of chaos.” It is no wonder that any type of structure or discipline (rigor) is rejected when it results in such strong personal feelings as choking or working under a dictatorship.

While it appeared that my students had found bliss in their chaos, I suspected that many had not slept too much during the previous week because they had to finish a project. I also heard from their manager that they wouldn’t get too much sleep in the future because portions of their work had to be redone due to errors. I also picked up that some had family problems because they were never home. It was time to ask two more questions: “What is bad about chaos?” and “What benefits can structure provide?”

I was surprised to get such honest answers to the first question, “What is bad about chaos?”:

  • You don’t have a life
  • Not enough sleep
  • Lost concentration
  • It’s frustrating to keep redoing work
  • High error rate
  • You run 200 mph but cover less than 100 [miles]
  • Management is displeased
  • Spouses threaten divorce
  • Sets precedents for unreasonable expectations

Equally surprising were the answers to the second question, “What benefits can structure provide?”:

  • Calmness
  • Being organized feels good
  • Sense of being in control
  • Being able to plan activities outside of work
  • Time to think
  • Be creative
  • Balance
  • Rest and vacations
  • Sense of accomplishment, job well done
  • Recognition
  • Learning
  • Raised self-esteem

It was striking to me to see that some of my students’ responses were the same as to the previous questions, such as “be creative” and “raised self-esteem.” They did not reject structure and methodologies per se, only the “rigor” that accompanied it and only when imposed by their manager. The lesson was clear: agility and rigor not only can complement each other, they should. The other lesson, of course, was that agility without rigor is chaos, and chaos has extremely high risk and high cost. The purpose of a methodology is to reduce risks and costs. The more complex a project, the higher its risks and costs, and the more rigor is required to manage it.

BI/DW methodologies

Business intelligence (BI) and data warehouse (DW) projects are particularly complex for several reasons. Their scope is often much too large and the project deadline is much too short. The requirements are constantly changing or expanding without adjustments to the deadline. Data is integrated from multiple legacy source systems that were never designed with integration in mind. Dirty data, duplicates, and inconsistencies are everywhere and add to the complexity. Most project teams are understaffed and insufficiently trained. And the list goes on.

When I wrote the Business Intelligence Roadmap methodology, I compiled 900 tasks that are potentially applicable to BI and DW projects. Since then, I have received numerous emails from readers pointing out additional tasks that I omitted. In other words, there are close to 1,000 tasks that apply to BI and DW projects. Of course, not all tasks are applicable to all projects, but they must be considered and either selected or rejected, depending on the nature of the project. Nobody can remember 1,000 potential tasks without the help of a methodology. Nobody should ever commit to a deadline without having reviewed the 1,000 potential tasks to understand the effort that may be required for the project.

On the other hand, traditional methodologies are much too rigid for the dynamic nature of BI and DW projects. Traditional methodologies prescribe a sequence to their tasks and deliverables, which presumes that requirements can be finalized before analysis starts, that analysis can be completed before commencing the design, that the design can be approved before coding starts, and so on. None of these assumptions are true for BI and DW projects. Therefore, following a rigid traditional methodology is a prescription for failure, and anyone rejecting traditional methodologies for their BI and DW projects is justified in doing so.

A BI/DW spiral methodology is a prime candidate where agility and rigor can and should meet. The rigor should be documented in the methodology as entry and exit criteria for all development steps, things to consider, activities and tasks, deliverables, dependencies, roles and responsibilities, do’s and don’t, tips and rules of thumb. The agility should be reflected in the methodology in two ways: extreme scoping, an agile approach to scope management and project management, and self-organizing project teams, which are a modified version of my “kids on skateboards.” These will be the subjects of my next two articles.

If you would like to learn more about the Business Intelligence Roadmap methodology, where rigor is coupled with agility, here are two references:

“Business Intelligence Roadmap: The Complete Project Lifecycle for Decision-Support Applications” by Larissa Moss and Shaku Atre, Addison-Wesley, 2003. ISBN 0-201-78420-3

“Business Intelligence Methodologies: Agile with Rigor?” by Larissa T. Moss, Cutter IT Journal, Vol. 14, No. 12, December 2001

About the Author

Larissa Moss is president of Method Focus Inc., and a senior consultant for the BI Practice at the Cutter Consortium. She has 27 years of IT experience, focused on information management. She frequently speaks at conferences worldwide on the topics of data warehousing, business intelligence, master data management, project management, development methodologies, enterprise architecture, data integration, and information quality. She is widely published and has co-authored the books Data Warehouse Project Management, Impossible Data Warehouse Situations, Business Intelligence Roadmap, and Data Strategy. Her present and past associations include Friends of NCR-Teradata, the IBM Gold Group, the Cutter Consortium, DAMA Los Angeles Chapter, the Relational Institute, and Codd & Date Consulting Group. She was a part-time faculty member at the Extended University of California Polytechnic University Pomona, and has been lecturing for TDWI, the Cutter Consortium, MIS Training Institute, Digital Consulting Inc. and Professional Education Strategies Group, Inc. She can be reached at methodfocus@earthlink.net

Free Expert Consultation