Bob Lambert

Need uber-guru types who are willing to challenge the existing groupthink on design and architecture, especially on TDD and emergent design and pair programming anti-pattern” – job post at Monster.com 2/9/2010

I stumbled upon that quote following links on the role of the architect on an agile project. Maybe one important role of the architect is to help the team avoid groupthink.

Groupthink is a situation where a team’s decision process breaks down and the team reaches decisions that aren’t fully vetted and evaluated.  Both Watergate and the Bay of Pigs fiasco are cited as examples (here).  I’ve seen groupthink operate on IT projects, and to me the agile method’s effectiveness in enabling groups to work together means agile projects are particularly susceptible.

This post reviews groupthink then discusses how the architect on an agile project might help prevent it.

Groupthink

From the Wikipedia article on groupthink, “groupthink is a type of thought exhibited by group members who try to minimize conflict and reach consensus without critically testing, analyzing, and evaluating ideas. Individual creativity, uniqueness, and independent thinking are lost in the pursuit of group cohesiveness. During groupthink, members of the group avoid promoting viewpoints outside the comfort zone of consensus thinking…Highly cohesive groups are much more likely to engage in groupthink, because their cohesiveness often correlates with unspoken understanding and the ability to work together with minimal explanations.”

In my experience risk of groupthink can manifest in several ways on IT projects:

• Not Invented Here: Successful teams that work through conflict can settle into a shared culture that resists new ideas from outside the team.
• The Know It All: Less successfully integrated teams can be dominated by a single strong-willed individual, and can habitually avoid conflict by accepting without question the ideas of that one dominant team member.
• The Abilene Paradox: Team members sometimes collectively decide on a course of action that no one on the team likes, when each member actually disagrees with the decision but mistakenly believes that their own preferences are counter to the group’s.

The Agile Architect

According to the Psychologists for Social Responsibility, the standard remedies for groupthink include this: “At least one articulate and knowledgeable member should be given the role of devil’s advocate (to question assumptions and plans)”. Of course the architect is an integral part of the overall project, but the skilled practitioner participates with the Agile team while maintaining separateness in order to remain a source of ideas from outside the team, and therefore provide a counterweight to groupthink by recognizing it and taking measures to prevent it.  Andrew Johnston’s site agilearchitect.org describes some of the ways the architect is in but not totally of the team (here). Among the architect’s responsibilities, he or she:

• Ensures “the delivered system is consistent with the agreed architecture, and will meet the requirements”
• “Is frequently an evangelist for new or different technologies, processes or solutions”
• “Acts as a bridge between developers, managers and other communities, and spends much of his time translating and mediating between them”
• “Recognizes the wide range of stakeholders, and their needs and concerns.”

While core agile team members are immersed in the scope and design that defines the current sprint, the architect retains a larger perspective that encompasses alternative designs, emerging technologies, business fit, stakeholder concerns, and more. The architect is therefore uniquely positioned to recognize groupthink effects on the team’s technical activities. Here are two examples of how that works on agile projects:

• Estimations and retrospectives: Mark Needham, in this post, cites risk of groupthink in agile estimation sessions and retrospectives.  The architect can address both of these risks. In estimation, the architect brings the diverse perspective that Mr. Needham says is important when team members estimate incorrectly due to incorrect team-shared assumptions. In retrospectives, the architect can be the one to increase the “safety” of different perspectives by raising or encouraging others to raise “things that have gone well, not gone well, and things that are confusing”.
• Work product reviews: I’m an advocate of code walkthroughs and design reviews, and making them explicit sprint tasks. The team can set aside an hour or two a week to review one or two representative work products in order to share ideas, ensure quality, and uncover overlooked errors or opportunities. In this forum the architect has the opportunity to reinforce quality work that is aligned with the requriements and architecture, or supportively correct deficiencies.

Of course there are risks:

• The architect shouldn’t be the know it all: In some cases the architect can be the strong-willed individual who stifles creativity and causes the team to avoid conflict.  Strong teamwork and interpersonal skills are core to the agile method, and those who staff the project must include those skills in selection and evaluation of the architect.
• Keep the architect different: If the architect is a core member of the team, he or she can become integrated into the group and therefore part of a groupthink dynamic.  For this reason, I advocate architects being assigned part-time to agile efforts. Otherwise, the architect risks becoming the extra developer, as near term sprint tasks expand to fill the available team bandwidth.  Consider sharing the architect among two or three projects, or assigning him or her responsibility for technical strategy and planning.

In a recent article at Information Management, Maria Villar and Theresa Kushner offer 4 Steps to Create an Effective IT and Business Partnership, a very useful list of ways to ensure “strong partnership between IT and business”.  To the authors this partnership “is the most important, and often overlooked, component to successfully managing critical business data. Undertaking business intelligence, data quality or an enterprise data management [program] without full cooperation and collaboration between IT and the business is a formula for frustration.”  The authors suggest these four steps: “know your partner, develop a relationship, define roles and responsibilities, and establish open, regular communication channels.”  I recommend reading this article because IT folks (like me) seem tempted to neglect the habits that enable building a solid relationship with business people.

That said, it seems to me that there’s something missing.  Consider one BI manager I know who has fractious relations with his business customers.  I won’t go into detail, but trust me, relations have been rocky, and reviews from key business players poor.  What this person does extremely well is to build rock-solid, reliable systems, deliver on time, meet business needs, and ensure that the solution meets regulatory and audit concerns.  This BI group is essentially unchanged after many years, enduring even the recent recession in a devastated industry segment, and outlasting many of its critics.

To me, building a good relationships is important, but execution is the sine qua non of IT/business alignment.  Think about it.  Say you hire a really nice contractor to fix a leaky roof.  However personable he is, you won’t hire him to replace your windows if the roof still leaks after you’ve paid the bill.

My view: if you want to do it right adopt Villar’s and Kushner’s excellent suggestions but the fundamentals remain the same:

1. Either (a) present a robust business case that all accept, or (b) pay attention to what you’ve been asked to do
2. Deliver what was requested/promised in 1
3. When things change go to 1

“When you are in the middle of a story it isn’t a story at all, but only a confusion; a dark roaring, a blindness, a wreckage of shattered glass and splintered wood; like a house in a whirlwind, or else a boat crushed by the icebergs or swept over the rapids, and all aboard powerless to stop it.  It is only afterwards that it becomes anything like a story at all.  When you are telling it, to yourself or to someone else.” – from Alias Grace by Margaret Atwood

Whatever project management approach a team uses, sometimes everything falls apart, commonly due to work piling up at the end, but sometimes due to a key individual leaving, or a pivotal assumption no longer holding true, or many other reasons.  When that happens, the project can become like a whack-a-mole game, with leads working from issue to issue as they pop up faster and faster.

I served as one of many workstream PMs on one very large project where this didn’t happen.  Out of the seeming chaos the multi-million dollar IT project came in on time.  Here’s my view of what we did to succeed:

• Had very clear interim milestones that were generally known and served as reference points for discussion.  I’d characterize them as a milestone per workstream per month.
• Held seemingly interminable weekly risk/issue discussions.  These were open, no holds barred reviews of anything at all that could endanger achieving milestones.  Often risk/issue discussions are polite exercises in avoiding the fact that the emperor has no clothes, with team members carefully avoiding forbidden topics.  On this project everything was open for discussion.
• The program manager excelled at visibly not sweating the small stuff, directing workstream leadership to handle their localized risks and issues themselves, and focusing program energy only on those he judged to have overall impact.
• Each of the many workstreams had its own project schedule; the program insisted on detail where it mattered but not where the detail was irrelevant.  For example, there was a minutely detailed cutover plan for production migration.

Many of us were surprised when the chaos all came together on time with surprisingly few glitches. I attribute this program’s success to the program manager’s unflappable focus on milestones, encouragement of unfettered group risk/issue analysis, and ability to parse program from project concerns.

In a recent very thoughtful post on data quality, Paul Erb plays out an analogy comparing data users with Don Quixote and data quality professionals with Sancho Panza, then reverses the analogy to cleverly coin the “Sancho Panza” test of data quality professionals.  He encourages data quality professionals promoting the critical role of data quality to apply a what would Sancho say test to ensure that they are aligned with the needs and interests of data consumers.

Here’s Paul’s description of the Sancho Panza test:

Think of Don Quixote [DQ] as the data-quality specialist or even the data management specialist or software vendor, bringing to the world his specialist’s perspective and vocabulary and enthusiasm, influenced by the books he’s read, visioning everyday business practices, with his value added, as goldmines for the organization.  Meanwhile Sancho Panza represents the person who does a practical job every day, who knows what works around here and what doesn’t.

I advocate to Data Quality (let’s call it DQ) consultants that they listen to this Sancho Panza, and consider themselves as Don Quixote.  Sancho doesn’t know much about data, but he knows what he likes… He’s open to listening, but slow to change, and he’ll tell you what he thinks.

Paul’s article reminded me that as a child I thought the problem with Don Quixote was that he tilted at windmills and attempted to ambush acting troupes because of his bad eyesight.  Of course this is not the case, but to me it provides a relevant perspective on data quality in many organizations.

Here’s the problem I’ve seen play out on a number of IT application projects:

1. A high level business study recommends replacement or improvement of a current application.
2. The organization approves the project described in a business case citing benefits named in the business study and costs detailed for infrastructure, package software, and application development, but data-related costs are glossed over or left out entirely.
3. The project begins with a requirements phase that collects hundreds of imperative statements (”The system shall…”)  from business people who will use the system.
4. Late in the requirements phase, the team finds that data integration work in system interfaces will be more complex than expected.  A common example: the project requires changes to a feeder application with no documentation and no in-house support expertise.
5. Project leadership goes back to the sponsor seeking more money.

In these situations the business case was incorrect because it did not account for all of the costs of data integration.  I’ve seen projects weather steps four and five well, but often discovery of previously unseen data complexity starts a disruptive chain of events.  (Sadly for the project manager, such situations are often seen as a failure of project management and corrected accordingly, but that’s a topic for another post.)

In my view the root cause of unforeseen data complexity on projects is the lack of a data constituency in current IT. It is only recently that success of companies like Google and Amazon have motivated emergence of data as a key business resource in the collective consciousness. Famous success stories notwithstanding (see this link), there are relatively few senior IT managers with data quality backgrounds.  Conversely, many rose through the ranks of the infrastructure, application development, or business (process) analysis groups.

It will be a while before, for example, a Mobil CIO’s predecessor jobs include definition of a metadata repository or elimination of multipurpose data, but in the meantime here’s what we can do:  add a business case to the application lifecycle as the last step in requirements.  Stop the project when the real costs are known, recalculate the cost/benefit, and ask the sponsors if the project should continue.  Give Sancho (in this case the project team) a chance to speak to the reality of the situation, and hand to Don Quixote (project sponsors) the eyeglasses of in-depth visibility into real costs. If the decision is to move ahead with the project, then all share the same vision and the sponsors have endorsed the actual project, not the fuzzy image from earlier on that might have been a windmill.

In a new post at Insurance Networking News Ara Trembly provides a balanced perspective on IT/business misalignment (Business/IT Misalignment: Whose Responsibility?).  He describes the problem as cultural, more amenable to relational than management solutions.    His conclusion sums it up: “Take a geek/suit to lunch today!”

To me (speaking as an IT professional) IT should take the initiative to solve the problem.  Quoting Trembly, “business executives … make decisions, but they are for the most part mystified at the magical incantations and actions that produce IT results” and “IT people, on the other hand, are jealous of the sheer power wielded over them by business people who just don’t get IT.”  In other words, business people contend with an emotional and a substantive problem, “fear and lack of knowledge,” while IT people have only the emotional problem of jealousy.

If we take the emotions out of the picture (its just a job, right?) then that leaves IT folks with knowledge that business people need in order to maximize the value of IT and efficiency of business processes.  Ever since mainframes roamed the prehistoric rain forests of the ’60s application developers have often been the most knowledgeable about how business processes really work, understanding both the intricacies of the application logic and how business people use the system to get things done.  These individuals can add value to the business discussion by bringing their knowledge to the table in a way that business people can understand.

In many organizations IT manages the forum in which these conversations can occur: the requirements process.  In my experience a good requirements process is long enough for the business and IT teams to get to know each other, offers generous opportunity for both structured and unstructured conversations about business needs, and brings together knowledgeable business and IT participants.  IT is typically able to bring the insights of seasoned application developers to the fore in a well planned requirements effort.

Yes, everyone has responsibility to “cultivate personal relationships based on mutual need and respect,” but IT can and should bring substance to the relationship in requirements definition.

Here’s a little-recognized fact about data integration: if you run a business or any sizable chunk of one, someone is integrating your data.

In my professional life I have on occasion suggested data integration efforts.  Sometimes my suggestions have been accepted and sometimes not.  As an IT professional I understand that different managers have different priorities, and in a given business situation sometimes other things are more important for example than having a single, consistent source for all customer records, or making sure production data matches financial data.

But as a customer?  That’s different.

A couple of years ago I bought a laptop from a company renowned for quality and customer service.  For the first weeks the computer was all it was cracked up to be, but then it cracked up.  The screen developed a mysterious flicker.  After a few diagnostic conversations they replaced the main logic board.  The problem recurred a few months later, and this time the company traded the lemon for a new computer.

All was well, but here we encounter the first data integration problem: they said I needed to call a different number to have my three-year service agreement transferred, which I did.

Months later I called service for a minor problem, and they had no record of the service agreement for the computer.  My warranty was still connected to the lemon.  After about an hour on the phone this company’s outstanding support staff came up with a more than satisfactory solution.

Even so, this company’s service records weren’t integrated with its warranty records.  In this case data integration happened because of my insistence and the service staff’s creativity.  The cost?  Only considering my last encounter, three service professionals were tied up for about an hour, and I’ll think twice before I buy again from this company.

It seems the choice is either pay now to integrate so all applications work from consistent data or pay later by having staff, customers, and suppliers do it on a case-by-case basis.

History is littered with IT application projects that end late, go way over budget, or abandoned altogether.  I was fortunate enough to see one work out really well (almost – please read on).  It was no mistake.  It came down to a simple method advocated by a gentleman named named John Carpenter.

The project was an HR management software conversion from one commercial off-the-shelf software (COTS) package to another.  The company concerned was conservative about spending money.  A previous business case had proposed a similar project.  The problem with that business case was that the benefits were really tough to conceptualize, so the cost/benefit analysis relied on soft benefits like “improved access to information” and “more consistent reporting data”.  The folklore was that the CFO had physically thrown that business case out of his office.

Mr. Carpenter’s method was to divide requirements definition and implementation into two distinct projects, with a different business case for each.  Under his direction, we wrote a ~1m business case for requirements definition only.  We proposed that this first project would result in another business case precisely specifying the schedule, method, cost, and benefits of the implementation project.

According to John, “the approach we used would not be considered a textbook approach for an ERP (enterprise resource planning) implementation.  What we did was more of a strategy to address the the CFO’s concerns.  The company was very risk-averse so we needed a way to take out as much risk as we could.  This was a large project because it involved four major modules affecting the three main areas of HR, and the company wanted to know costs and benefits at each step.  Complicating matters, HR business processes and therefore requirements were not clearly understood – the HR department seemed to rely on on the job training rather than documented procedures.  So we presented the first phase as an investment into understanding HR processes, as well a precise roadmap for implementation.”

This first business case was accepted by that same CFO and we got started on the 7-month effort. We brought in a consulting team experienced in the proposed COTS package, and followed their lead in requirements definition and prototyping.  During the prototyping step they walked HR staff through each relevant function in the software package, detailing how to configure the package for their specific needs and where we’d need to customize it.   The result was a definitive, detailed document that showed how the package fit HR process and how it would need to be customized.  Then, we used those results to build a business case that included specific configuration, customization, hardware, and software costs, as well as the process and organizational changes that would be required, not to mention the benefits that would accrue.  The business case showed substantial improvement, predicting real financial benefits within 4 years.  Even better, on a depreciated basis the project literally was almost free, costing only $1,800 in the first year and returning benefits thereafter.

The business case was accepted by the company’s executive committee and the project started.  It ran exactly as outlined by the results of the requirements effort, with very few of the nasty surprises often typical of large projects, and it tracked to forecast schedule and budget.

Proving that no good deed goes unpunished, the company, whose core business was real estate, in effect folded in the financial crash of last autumn ‘09 , one month from implementation.

At any rate, the lesson I took away from the effort was that dividing requirements and development into separate projects gives business visibility into a project, helps manage financial risk, and enables the project to ground predictions rather than guessing at costs and benefits before they can be known.

Recently I was asked for a high level requirements plan for a large IT conversion.  I googled around a little for something standard.  I found some good references (see links at the bottom of this post), but not exactly what I was looking for: a simple, method-agnostic layout of the high level steps and checkpoints in requirements for a big project, emphasizing interactions with business people.   I then rifled my files to find the example below.

This summary plan frames up interactions with business subject matter experts and their review of results.  The table lays out the steps, “granularity” (meaning how often each step is carried out), what comes out of each step, who does the work, and offers a few notes.

Before the table, here are two important definitions:

• Sponsor: Very early on in your project you should identify the one person who you need to make happy in order to succeed.   The bigger the project, the higher up the sponsor.  Keep in touch to make sure they know what’s going on, keep them happy, and if something happens that will make them not happy don’t keep it secret.  Maintain a vibrant risk/issue process so that you can give them early warning of bad possibilities and they can help early.
• Stakeholder: A stakeholder is anyone who will benefit from or be harmed by your project.  Requirements come from stakeholders.  Be sure to build support even with the latter group if at all possible, and at least make the adjustments that will keep them from working to prevent you from succeeding.

Careful, this is just an empty vessel.  Within this high level framework a team can apply whatever requirements techniques they want.  (In fact, I highly recommend structured analysis techniques like use cases or process models, but that’s for the requirements team and this framework is for the PM).  Of course a smaller effort suitable for agile techniques wouldn’t need something like this.  This is for big transitions like conversion to a new COTS package, for example, where it is easy to get lost in the detail.

Hopefully if you’re a PM on a big project you’ll find this framework as useful as I did.

 Here are some of the other references I found along the way, caveat emptor:

• http://www.outsource2india.com/software/RequirementAnalysis.asp
• http://en.wikipedia.org/wiki/Requirements_analysis
• http://www.jiludwig.com/HCI_Journal_article.html

To me, calling someone a “user” contributes to the gulf between IT and other business people.

Most of us don’t think about it and say “user” or “end-user”. Those who do think about it say something like client, customer, or business customer, or something similar.  Sometimes these terms work and sometimes they seem contrived, but there’s a better way.

Lior Arussy of the Strativity Group argues that what we call our customers is critical to effectively serving them (http://www.crmxchange.com/focus_customer/Oct06_2.asp). According to Arussy the problem is that many of us use terms for customers “based on their role in our process, encouraging a “one size fits all” approach that treats all customers the same. The diversity and complexity of customers is subsequently ignored and replaced with a customer that does not exist but who fits perfectly within the organization’s business model. The customer is thus subjected to the company’s processes and efficiency needs rather than the other way around.”

When we in IT call them “users” we do the same thing. Rather than thinking about a business person’s role in a business process, we think about them in terms of how they use “our” system. This encourages us to think narrowly about how, for example, a recruiter uses a personnel workflow system. The word “user” encourages us to disconnect from the business process rather than understanding how the system fits into the hiring process as a whole.

Maybe I’m exaggerating and this one small point doesn’t matter much. However, in most organizations there is a a communications gap between IT and business, and as the service provider it is IT’s responsibility to bridge the gap. Communications gaps are made up of lots of little habits of thinking and practice, and this is one of them.

So what should you call the user of a business application? When I’m talking generally about business people who use IT applications, I use the term business people, which after all is what they are. When I’m talking about the users of a specific system I try to use the term that everyone else in the organization uses for what they do, like “financial analyst” or “recruiter” or “production line supervisor.”

I hope that being specific and correct about business people’s role encourages me to understand the system I’m building in business context, making it easier for me to build the application exactly as they need it.

I’ve heard that “A good project manager can manage any project, as long as they’ve got good people on the project”. In the perfect world, maybe. In the real world, no way.

Those promoting project management focus on characteristics of PMs that are common across all disciplines: interpersonal skills, organizational skills, communication skills, and problem-solving skills.  These are necessary but not sufficient qualities.  Good project managers also have interest and depth in the subject matter of the project.

It seems that the world has forgotten about history’s great project managers. They were overwhelmingly individuals with abiding interests in their fields.  Werner von Braun led the massively successful development of the Saturn moon rocket. General Bernard Schriever, the man cited as “The Father of Project Management” (here), who led the Air Force’s space and missile programs in the 1950s, was a long time pilot with an engineering degree.  Frederick Brooks, with a PhD in Mathematics from Harvard, led development of IBM’s OS/360 operating system, fundamental to IBM’s dominance 0f commercial computing during the 70s and 80s. “As of 2008^[update] he is still engaged in active research there, primarily in virtual worlds and molecular graphics.” (Wikipedia)

I recently witnessed two IT projects, both multimillion dollar efforts and both business-critical.  Project A, a migration to a commercial-off-the-shelf (COTS) human resources system, tracked to budget and schedule and met business objectives with almost no customization.  Project B, another COTS effort for core business operations, suffered huge budget and schedule overruns associated with frequent changes in direction and scope expansions.

Project A was outsourced to an outside consulting firm, who brought in a PM with deep technical experience configuring and customizing the COTS package.  Project B was run by an internal Project Management Office, staffed by project managers who did not have technical or business analysis backgrounds.

Project A’s leadership had the background to understand the road ahead, and spent six months of a two year project in definition, completing detailed business requirements, business process design, and system architecture deliverables.  On Project B, a senior IT manager decreed that the team skip requirements definition, since future business processes would be dictated by the COTS application: a critical early mistake.  Project B’s management did not have the IT background needed to recognize the mistake, which would have been obvious to the Project A team.

For a PM to be effective he or she needs sufficient knowledge to envision the end product in a scope and objectives document, create a work breakdown structure that makes sense to business and technical team members, and know which risks are pivotal and which don’t matter.

In spite of stereotypes about IT folks, many are experienced, articulate, well organized, have great interpersonal and problem-solving skills, technically curious, and would welcome an opportunity to lead an ambitious project.  Make one of them your project manager, provide strong organizational support, and you’ll be more likely to achieve objectives on budget and on schedule.