Bob Lambert

Need uber-guru types who are willing to challenge the existing groupthink on design and architecture, especially on TDD and emergent design and pair programming anti-pattern” – job post at Monster.com 2/9/2010

I stumbled upon that quote following links on the role of the architect on an agile project. Maybe one important role of the architect is to help the team avoid groupthink.

Groupthink is a situation where a team’s decision process breaks down and the team reaches decisions that aren’t fully vetted and evaluated.  Both Watergate and the Bay of Pigs fiasco are cited as examples (here).  I’ve seen groupthink operate on IT projects, and to me the agile method’s effectiveness in enabling groups to work together means agile projects are particularly susceptible.

This post reviews groupthink then discusses how the architect on an agile project might help prevent it.

Groupthink

From the Wikipedia article on groupthink, “groupthink is a type of thought exhibited by group members who try to minimize conflict and reach consensus without critically testing, analyzing, and evaluating ideas. Individual creativity, uniqueness, and independent thinking are lost in the pursuit of group cohesiveness. During groupthink, members of the group avoid promoting viewpoints outside the comfort zone of consensus thinking…Highly cohesive groups are much more likely to engage in groupthink, because their cohesiveness often correlates with unspoken understanding and the ability to work together with minimal explanations.”

In my experience risk of groupthink can manifest in several ways on IT projects:

• Not Invented Here: Successful teams that work through conflict can settle into a shared culture that resists new ideas from outside the team.
• The Know It All: Less successfully integrated teams can be dominated by a single strong-willed individual, and can habitually avoid conflict by accepting without question the ideas of that one dominant team member.
• The Abilene Paradox: Team members sometimes collectively decide on a course of action that no one on the team likes, when each member actually disagrees with the decision but mistakenly believes that their own preferences are counter to the group’s.

The Agile Architect

According to the Psychologists for Social Responsibility, the standard remedies for groupthink include this: “At least one articulate and knowledgeable member should be given the role of devil’s advocate (to question assumptions and plans)”. Of course the architect is an integral part of the overall project, but the skilled practitioner participates with the Agile team while maintaining separateness in order to remain a source of ideas from outside the team, and therefore provide a counterweight to groupthink by recognizing it and taking measures to prevent it.  Andrew Johnston’s site agilearchitect.org describes some of the ways the architect is in but not totally of the team (here). Among the architect’s responsibilities, he or she:

• Ensures “the delivered system is consistent with the agreed architecture, and will meet the requirements”
• “Is frequently an evangelist for new or different technologies, processes or solutions”
• “Acts as a bridge between developers, managers and other communities, and spends much of his time translating and mediating between them”
• “Recognizes the wide range of stakeholders, and their needs and concerns.”

While core agile team members are immersed in the scope and design that defines the current sprint, the architect retains a larger perspective that encompasses alternative designs, emerging technologies, business fit, stakeholder concerns, and more. The architect is therefore uniquely positioned to recognize groupthink effects on the team’s technical activities. Here are two examples of how that works on agile projects:

• Estimations and retrospectives: Mark Needham, in this post, cites risk of groupthink in agile estimation sessions and retrospectives.  The architect can address both of these risks. In estimation, the architect brings the diverse perspective that Mr. Needham says is important when team members estimate incorrectly due to incorrect team-shared assumptions. In retrospectives, the architect can be the one to increase the “safety” of different perspectives by raising or encouraging others to raise “things that have gone well, not gone well, and things that are confusing”.
• Work product reviews: I’m an advocate of code walkthroughs and design reviews, and making them explicit sprint tasks. The team can set aside an hour or two a week to review one or two representative work products in order to share ideas, ensure quality, and uncover overlooked errors or opportunities. In this forum the architect has the opportunity to reinforce quality work that is aligned with the requriements and architecture, or supportively correct deficiencies.

Of course there are risks:

• The architect shouldn’t be the know it all: In some cases the architect can be the strong-willed individual who stifles creativity and causes the team to avoid conflict.  Strong teamwork and interpersonal skills are core to the agile method, and those who staff the project must include those skills in selection and evaluation of the architect.
• Keep the architect different: If the architect is a core member of the team, he or she can become integrated into the group and therefore part of a groupthink dynamic.  For this reason, I advocate architects being assigned part-time to agile efforts. Otherwise, the architect risks becoming the extra developer, as near term sprint tasks expand to fill the available team bandwidth.  Consider sharing the architect among two or three projects, or assigning him or her responsibility for technical strategy and planning.

Information Management recently sent around their pick of best IM blog articles of 2009.  Among them was Forrester’s James Kobelius’s reaction to Bill Inmon’s “incineration of a straw man concept that he refers to as ‘virtual data warehousing (DW).’” 

According to Mr. Inmon, virtual data warehousing reminds him of the carnival game called whac-a-mole.  He says “just when you think this incredibly inane idea has died and just when someone has delivered what should have been a deathly blow, out it pops again from another hole.” There’s just a very informal definition of virtual DW in Mr. Inmon’s post (remember, he says he’s whacked this mole before), but, as I interpret, he’s talking about a system built after a decision to avoid all the expense of building a data warehouse by just having a query engine that pulls the data from wherever it lives. Mr. Inmon argues that a query accessing diverse databases would leave data integration to the user, and there’s no guarantee that two users would integrate data the same way.  He cites virtual database query inefficiency risks and, on the assumption that the query is trolling operations focused databases, says that source data would be “tuned” to operational rather than informational specifications for history retention and completeness.

Mr. Inmon’s ideas drew quick reaction from Mr. Kobelius and Neil Raden.  Each in his own measured way stresses that integration can be compatible with distributed architectures, and that there is a DW solution architected for efficiency that includes effective data integration from diverse sources: the Federated Data Warehouse.

Experience and emerging tools reinforce their point.  According to a colleague at CapTech, for smaller organizations “you can deal with this issue using a BI tool with a metadata layer that has joins predefined: the data integration is done by the BI metadata modeler.”  Another CapTech’er cites mashup as a potential quick and dirty approach.  Check out “7 Mashups Every Company Needs” here.

A well-architected federated warehouse certainly can integrate and deliver data, maintain history, and enable a “single version of the truth”, perhaps in a more timely manner than a “traditional” DW architecture.  On this question the devil is in the specifics of the situation.  It is difficult to argue one way or another out of the context of a real project in a real organization.

However, even though it certainly has a technical side, data integration is first a business activity.  Sometimes when we apply terms like “semantic rationalization” to software components, we in IT start believing you can actually build a machine that does the things you need to do to rationalize data semantics, like figure out the corporate definition of a customer.  Of course all we can do in IT is to build the empty shell.  The real work happens when business people from departments whose data is being integrated sit down and decide how they are going to define “staff member”, “customer”, and so on.  Only business professionals can say, for example, whether they want to include contractors in staffing reports or whether the term “customer” includes homebuyers under contract but not yet closed.

Integration tools that support data warehouses, whether centralized or federated, are only as good as the business consensus behind them. The consensus behind integrated data is arguably more rewarding to the business that the tools because with consensus on critical objects and events come non-IT-specific improvements like reduction of repetitive and conflicting business processes, reduced communication breakdown due to terminology disconnects, and more.

To me the beauty of the Inmon DW model is that it provides a mechanism that can assist an organization in evolving toward improved information maturity.  Organizations achieve some benefit by simply integrating data into a single data warehouse.  However, the data warehouse also makes source data quality problems obvious and blatantly reveals differences in data meaning from one operational source to another.  So the warehouse delivers some benefit early and also shows how much better it would be if data were integrated.  It therefore becomes a tool for identifying, assessing, prioritizing, and motivating correction of data deficiencies.

For organizations not so far along on the maturity curve, the additional complexity of the federated warehouse tends to obscure this data quality feedback loop. Federation based on drawing from operational sources integrates data from a set of different databases built toward different architectural goals.  On the other hand, the logical data model for the enterprise warehouse is the enterprise data model, and its architectural objective is to integrate enterprise data to provide a single source of truth.  Therefore, the enterprise data warehouse provides an architectural focal point for integration.  It isolates responsibility for improving data integration crisply at either the source or the warehouse, and — within the framework of solid information management strategy, management, and facilitation — motivates diverse business players to work toward consensus definition of enterprise data.

Federation, or virtual data warehousing if you will, can be the best strategy for the mature organization that has already integrated business data to a consistent enterprise view.  For the rest of us, the single centralized warehouse with its unambiguous architectural goals and borders seems the shortest distance to achieving the business benefits of data integration.

Many see IT as application of technology to solve business problems. 

Of course, this is true but it leaves out the third element, which is to apply the right architectural pattern to solve the problem.  For example, when the business problem is that reporting is slow and reports from different departments don’t match, the astute IT professional immediately thinks in terms of a data warehousing pattern employing technologies like databases, extract-transform-load (ETL) tools, and multi-dimensional reporting suites.

A strategy based on the tools alone may solve the immediate problem, but understanding the solution-pattern enables the IT professional to bring to the business the additional benefits that come with the pattern, organizational and IT support impacts, and any risks that might emerge by applying the pattern.  In the data warehousing case the informed architect might cite improved executive dashboards and ability to drill down to root causes from summary reports, the need for data stewardship, and potential long term increase in data storage capacity needs.

Beyond that, when the architect lacks the pattern approach he or she seems to the business person like Bob Dylan’s debutante: “Your debutante just knows what you need, but I know what you want.” On the projects I’ve been on where designers lacked a pattern-based perspective the technical team did exactly what the business folks said they wanted, but for the most part didn’t contribute to the business value of the solution.  On projects like this developers slavishly ensure the solution matches each and every requirement, rarely bring to the table new business requirements that are logical consequences of the design, and tend to avoid questioning defined requirements even if they are contradictory or counterproductive.

Sure, patterns aren’t strictly necessary.  An outstanding architect can design from whole cloth an original solution that precisely matches business need.  However, that’s not how outstanding architects do business, at least the ones I’ve known.  In my experience the best IT architects know patterns, like data warehousing, SOA, and others, well enough to match the business problem to the right pattern and then evolve the architecture from the generalized pattern into a problem-specific architecture based on the particulars of the business problem at hand.

Some examples of common solution patterns in the world of business IT are Data Warehousing, Master Data Management, and Service Oriented Architecture (the Wikipedia article on this one is preliminary at this writing but still a good intro for the uninitiated).   Those interested in a more technical introduction to patterns might start with Avel Avram’s quick intro at InfoQ (Membership at InfoQ is required but free and worthwhile).

And of course, apologies to Mr. Dylan for the title…

Today, the foundation of most of our custom-built systems is a relational dbms.  While development frameworks vary, they overwhelmingly access and maintain data in relational tables and columns.  As I write I routinely save this post in a MySQL database, and at work I tend SQL Server applications.  Millions of others develop, use, and extract analytical data from thousands of SQL Server, DB2, and Oracle applications, on servers and networks maintained in-house by in-house administrators.

Some claim that the relational dbms may be out of style very soon.  Cool new “cloud computing” and “SaaS” apps and services  delivered over the internet seem to be popping up everywhere – just look at Salesforce.com, the well-established Customer Relations Manager vendor, and the many cloud-based PC backup sites.  As part of that trend, Amazon, Google, Microsoft and others offer database services over the internet that don’t look much like relational dbms’s.  Some supporters of the cloud-db options seek alternatives to the standard relational DBMS (note this widely read article).  Of these, many are OO developers.  There’s a fundamental dissonance between OO and relational approaches, requiring an intermediate object/relational mapping (ORM) layer for OO systems to operate effectively with relational DBMSs.  Many of the new cloud-db options are open source, lightly structured data services provided via the internet, capable of storing and delivering large data stores for high availability, fast response applications.

The convenient thing about relational databases is that they pretty much match the business view of data, and therefore give business people and developers common ground.  A well thought out relational data model is one way to express the inherent structure of business rules (see this previous post).  A relational model at the back end of a custom-built system means that both developers and business people can talk about the real guts of a system in ways that make sense to both, like this:

• Developer to business person: “Should we allow a part_order to include items from only one division”
• Business person to developer: “After a call from our shipping department, I ran a query on the part_order table and found a that there is a part_order with null shipper_phone_number. I thought it was a required column, what’s up?

How’s it going to be when those comments don’t reflect the underlying structure of the database?  Today’s cloud db offerings vary in structure, but tend to favor highly efficient and flexible models like name-value pairs, and avoid the overhead required by semantic layers like the relational model.  According to the MongoDB site, “by reducing transactional semantics the db provides, one can solve an interesting set of problems where performance is very important.”

In such databases the structure of the data will be hidden from business people; there will be no shared business/IT view.  Rather than talking with business people about the actual database structure we’ll talk about its custom abstraction, and when things go poorly with performance and functionality the developer will in effect say “trust me on this one” to the business person rather than explaining what’s up.

For a long time cloud databases will be another option alongside the relational model, but the more prominent cloud databases become the more difficult it will be for developers and business people to communicate about business data in IT applications, and it could be a serious challenge for developers to learn to cross that communications gap without the bridge provided by the relational model.

While many organizations understand the value of managing the information resource, for many others information management remains abstract and difficult to define.  In an effort to make it concrete here’s a hypothetical proposal to provide an Enterprise Information Architect for a hypothetical organization that really needs one.

Today: inconsistent data of uncertain quality blurs enterprise view and restricts planning

Today managers, planners, and analysts lack the information required to run the organization as a single enterprise rather than a collection of diverse units.

• Data quality in IT applications varies to the point that, outside financials, it is impossible to gather consistent data supporting an enterprise view of operations.
  • Application development efforts have focused narrowly on departmental interests without accounting for enterprise concerns, making application data incomplete in describing business processes and inconsistent with data in other applications.
  • Focus on departmental concerns and tight development timelines has resulted in incomplete validation of data critical to the enterprise but not critical to the application’s focus.  For example, customer demographics are not critical to the sales process and therefore zip codes and telephone numbers are not consistently collected at point of sale, substantially reducing value of market analysis based on sales data.
• Enterprise planners work with only the highest level summaries of operational data, those summaries suffer large margins of error, and planners cannot definitively answer questions required to make critical business decisions.
• Regulators have questioned the validity and repeatability of reporting because of the organization’s heavy reliance on spreadsheets and manual processes in gathering and compiling data for reports.

Solution: enable sound planning and management by identifying data assets and setting processes to manage them

Empower an Enterprise Information Architect to lead an effort that (1) identifies data that describes the organization, (2) defines how to integrate and improve quality of that data, and (3) improves the ability of information technology to maintain data quality.

(1) Lead definition of an Enterprise Information Architecture identifying information required to manage the organization as a single integrated enterprise, and data quality standards that ensure that data supports enterprise goals.

• Identify and define events and objects critical to the enterprise

• Identify and define relationships among those events and objects and attributes that describe them

• Classify data managed by the organization by type (operational, statistical, financial, decision support, etc.) and define standards for managing and integrating each type.
• Compile the above into a plan that explicitly supports the enterprise strategic plan

(2) Working with senior business managers, put in place a program of data quality improvement that plans and executes specific measures and sustained commitment to improving data quality in business processes and IT applications

• Identify the business group responsible for maintaining quality and integrity of each business object, event, relationship, and attribute

• Identify for each data item of interest to the enterprise its “system of origination” and “system of record”.

• System of origination is the application that provides the entry point of a given data object to the organization.
• System of record is the application that is the source of record for the data object.

• Define and deploy standards and practices for for business process and IT application definition that support data quality and integrity standards

(3) Working with senior IT managers define and put in place standards for application requirements definition, data management, and metadata management to

• Define and deploy application development and interface standards that support data quality objectives.
• Ensure that application development efforts support enterprise data quality
• Continually monitor new developments in data management best practices and make that information available to the enterprise.

Here’s a little-recognized fact about data integration: if you run a business or any sizable chunk of one, someone is integrating your data.

In my professional life I have on occasion suggested data integration efforts.  Sometimes my suggestions have been accepted and sometimes not.  As an IT professional I understand that different managers have different priorities, and in a given business situation sometimes other things are more important for example than having a single, consistent source for all customer records, or making sure production data matches financial data.

But as a customer?  That’s different.

A couple of years ago I bought a laptop from a company renowned for quality and customer service.  For the first weeks the computer was all it was cracked up to be, but then it cracked up.  The screen developed a mysterious flicker.  After a few diagnostic conversations they replaced the main logic board.  The problem recurred a few months later, and this time the company traded the lemon for a new computer.

All was well, but here we encounter the first data integration problem: they said I needed to call a different number to have my three-year service agreement transferred, which I did.

Months later I called service for a minor problem, and they had no record of the service agreement for the computer.  My warranty was still connected to the lemon.  After about an hour on the phone this company’s outstanding support staff came up with a more than satisfactory solution.

Even so, this company’s service records weren’t integrated with its warranty records.  In this case data integration happened because of my insistence and the service staff’s creativity.  The cost?  Only considering my last encounter, three service professionals were tied up for about an hour, and I’ll think twice before I buy again from this company.

It seems the choice is either pay now to integrate so all applications work from consistent data or pay later by having staff, customers, and suppliers do it on a case-by-case basis.

Free form diagramming doesn’t only mean “no rules”, it also means “just right”.

This post, last in a three part series on free form diagramming, gives some simple guidelines for getting the technique right.  Part one talked about the tension between rigor and expression in diagramming for analysis and design, and how more precise diagrams can hinder rather than help communications with business people.  Part two reviewed free form diagramming in practice.

While it is impossible to specify format and structure of a free form diagram in advance, here are some useful guidelines to follow when building your free form diagram:

•    Rule number one: draw it as you see it. Typically, an analyst uses a free form diagram because he/she already pictures a business process.  Trust your mental picture and get it down on the page.  Then, go through the following checklist to make sure it says what you want it to say.

•    Model real world processes, things, and events. Free form diagrams have one great advantage over Use Case Diagrams, Data Flow Diagrams, and the rest: they are concrete rather than abstract.  For example, in a free form diagram you can symbolize a shopper with a clipart picture of someone choosing a soup can from a grocery store shelf.  The free form diagram should clearly represent things from the real world: organizations, locations, business processes, interfaces, etc.

•    Use symbols consistently. Look at each repeated rectangle, line, circle, icon, etc, and verify that everything with the same shape represents the same type of thing or event.

•    Speak the language of the audience. A free form diagram should depict things business people care about in recognizable terms.  For example, accountants might readily understand boxes labeled GL, AP and AR for general ledger, accounts payable, and accounts receivable.  A shipping clerk might quickly interpret a process illustration showing labeled icons shaped like trucks and warehouses.

•    Arrangement on the page conveys meaning. Frequently, free form diagrams group objects that belong together on the page.  In other cases, the diagram shows a definite process flow by the arrangement of objects.  For example, Business Intelligence Architecture diagrams typically show information flow from source systems on the left to business reporting on the right.  Could this kind of flow or grouping improve your diagram?

•    Limit the number and complexity of objects on the diagram. Most often, a meaningful diagram shows relatively few objects, organizes them in a sensible way, and does not cross lines.  If you need many objects to tell the story, reduce complexity by arranging them logically.

•    Work at one level of detail, or clearly indicate differences between levels of detail. If your diagram includes a box labeled “AP System” then it would not likely make sense for it also to contain another labeled “Journal Voucher Key”.  Diagrams that communicate well are all at the same level of detail or clearly indicate differences in level of detail.

The free form diagram can be an essential part of a successful IT application project by enabling all to understand the target system in the same way and helping business people understand critical functionality.

This is part two of a three part series on free form diagramming for IT projects.  This entry reviews free form diagramming in practice. Part one talked about the tension between rigor and expression in diagramming for analysis and design, and how more precise diagrams can hinder rather than help communications with business people.  Part three will provide a few simple guidelines for getting it right.

Some current development tools and methods do include the free form diagram in their toolbox.  For example, Scott Ambler’s Agile Modeling site includes a page on free form diagrams (http://www.agilemodeling.com/artifacts/freeForm.htm), and the IBM tool XDE has provided an integrated free form diagram (http://www-128.ibm.com/developerworks/rational/library/915.html).

A Free Form Diagram Showing the Broad Outlines of a System

Free form diagrams can be especially useful in illustrating the overall scope of an IT development effort.  For example, XDE product literature cites the free form diagram’s “capabilities to communicate broad ideas about the direction of [a] solution”.  IT software product marketers frequently bear out this advantage over structured modeling with attractive diagrams describing their products and services, as in the diagram above.  Such diagrams enable companies to “level set” the terms of a conversation, providing a reference point for providing more detailed information about the product.

A similar diagram can provide a reference for IT project participants.  One, produced jointly early on in custom development of a financial system by the primary business contact and project architect, helped build consensus with designers, database administrators, programmers, and business people.  Some on the project considered this shared vision of the planned system one key to the project’s success.

Both of these examples illustrate the most common use of the free form diagram: to depict “the fundamental organization of a system [or business function], embodied in its components, their relationships to each other and the environment, and the principles governing its design and evolution” (http://en.wikipedia.org/wiki/System_architecture).

Because the free form diagram can seem more concrete than other, more abstract, models, a well-constructed free form diagram is a great way to communicate complexity to an audience of both technical and non-technical participants.

The free form diagram is also useful in other situations.  Sometimes a requirements team works with business users who have a diagram describing their current process.  In that case the team might use the same format to describe the new process.  In other cases it might be useful to improve readability by drawing a process flow on a map of the shop floor, or using icons representing real objects and events rather than class symbols on a class model.  The point is to create a medium that precisely matches the audience with the message.

Part 3 of the Free Form Diagrams series provides a few simple guidelines for success with this technique.

One effective way of communicating complexity, especially in the overall architecture of a system, is the free form diagram.  A free form diagram can directly address unique characteristics of a system in a way that business people can understand.

Out on a walk some years ago I met an acquaintance who happened to be a professor of Computer Science.  We talked shop (I worked in the IT department of the local electric utility) and he asked me what methods we used for mathematically proving program correctness.  I confess I laughed.  My team was struggling to figure out just what the business really wanted – forget mathematical proofs!

Our conversation highlights the tension between rigor and expression in diagramming to support IT projects.  Developers need precise diagrams rigorous enough to accurately reflect complex processes.  However, that precision and detail can make the diagrams at best boring and at worst intimidating to business people.  Often they don’t have the time or patience required to sit through the explanations needed to understand such diagrams. Requirements meetings frequently end early with business participants walking off grumbling about IT non-alignment.

UML is today’s de facto analysis and design diagramming standard. To the OO professional UML provides a rich, expressive, and consistent set of conceptual tools that continue to evolve.  Work is underway to improve the accuracy and precision of UML with respect to the target code. (http://www.omg.org/docs/ad/00-01-07.pdf).
The problem is that more precision will make the UML diagrams more complex and less understandable to non-coders, making the diagrams even less useful in communication with business people.

For project success business people need to be able to communicate their needs freely and completely.  Requirements analysts need to capture, record, and replay business needs for confirmation, and let the business expert return to other work as quickly as possible.  These communications often work best with pictures rather than words, and of course when everyone understands the picture the same way.  Asking a busy floor supervisor to review a formal class model is literally having him or her review specifications in a foreign language.

One solution is for developers to communicate with business people using free form diagrams, expressive yet rigorous diagrams with drawing conventions tailored to the audience.

More to come in parts two and three of this series.  Part 2 talks about using free form diagrams in practices, and Part 3 provides some simple guidelines for successfully using free form diagrams.