As good as it is, to me the article conjured an unlikely vision of the Dead as business men in hippie drag self-consciously making strategy decisions that altered the marketing landscape. I agree that the Dead took the actions cited on purpose, but I believe core product, not marketing strategy, consumed the band’s energies during its formative and peak years.   Could it be that their innovative market strategies grew organically from a quality product, where quality included the entire fan experience?

I hope those teaching Grateful Dead management include this:

• Develop and maintain a strong product
• Risk everything to make the product top quality
• Perfect every detail of the customers’ experience

Develop and maintain a strong product

Given their image, it is easy for non-fans to lose sight of the fact that the Dead were good at what they did.  While one could argue aimlessly about how good they were, they certainly didn’t suffer the unevenness of musicality of some rock ‘n’ roll bands, and they proved it live most days of any given year.  Phil Lesh’s memoir recalls the words of Dizzy Gillespie, passing by an outdoor concert in the late ’60s: “Those cats can swing!”  Lesh himself was a student of jazz and then avant garde composers like Stockhausen before joining the Warlocks, as they were first named.  Mickey Hart was and remains a student of primitive percussion. According to Lesh, Jerry Garcia learned to play the pedal steel guitar in a matter of weeks, quite an accomplishment for an instrument that requires both hands, both feet, and knees to control.

They applied these skills in a joyous, educated, and well-crafted way that reflected musical practice, discipline, and breadth, gluing songs of separate genres together with rocking transitions that frequently dissolved into organic free jams, only to come back together somewhere entirely unexpected.  Even beyond their widely varying originals, their diverse covers were an encyclopedia of mid-twentieth century folk/pop, including Harry Belafonte, Merle Haggard, Buddy Holly, Marty Robbins, and many others.

Risk everything for top quality

Early on the members of the Grateful Dead were steadily less satisfied with the quality of the then-prevailing live sound technology.  Their dissatisfaction peaked in the early seventies, when, according to soundman Dan Healy, they sunk “90 percent of their total earnings” toward building a sound system so that their faithful could “go to the show and hear the heavenly choir, so to speak, through the heavenly sound system.”  Their sacrifice to quality was such that “there were times when we spent the money on speakers and nobody got paychecks, from Jerry on down.”

The dream emerged as the Wall of Sound, a behemoth sound system that “took four semi-trailer trucks and more than 20 crew members to haul and set up.” While the wall of sound itself became too costly to lug around with the fuel crisis of the mid seventies, the Dead retained top notch sound quality after its demise, and its technical innovation remains with us today.  For example, the wall of sound introduced the practice of mic-ing each instrument separately, enabling the sound engineer to deliver a live show with the balance of a recording – standard practice today but revolutionary at the time.

Perfect every detail of the customer’s experience

The Grateful Dead were central to the Haight-Ashbury hippie scene in the sixties, a culture that embraced “Eastern philosophy, championed sexual liberation, promoted the use of psychedelic drugs which they felt expanded one’s consciousness, [and] used alternative arts, street theatre, folk music, and psychedelic rock as a part of their lifestyle and as a way of expressing their feelings.” (here)  Hippie culture came together in “happenings”, free form gatherings “during which music, psychedelic experimentation, a unique sense of personal style and … light shows combined to create a new sense of community”.   The focus of the happening was on the totality of the experience, bringing all elements together to join the participants and spectators (in as much as there was such a distinction) into a single mind.

As freaky as all this sounds, the happening’s focus on the shared gestalt surely fostered the Grateful Dead’s attention to audience experience.  I saw the Dead only once, but it was an outstanding show. Every detail seemed to have been choreographed for the experience of the listener.  The sound was big but not loud and every nuance was clearly audible. Stage and house lighting were perfect. Security was ubiquitous, proactive, and polite.  No detail interfered with band/audience community. Needless to say the Dead rocked the house.

—

Sadly, it couldn’t last forever.  A reviewer of Lesh’s book recounts the decline that started during the late eighties as drug-related health problems, constant touring, the changing nature of their fan base, and the sheer weight of their growing organization bore down on the band. It seems every long, strange, trip has its long strange decline – and perhaps there are management secrets there as well.

I stumbled upon that quote following links on the role of the architect on an agile project. Maybe one important role of the architect is to help the team avoid groupthink.

Groupthink is a situation where a team’s decision process breaks down and the team reaches decisions that aren’t fully vetted and evaluated.  Both Watergate and the Bay of Pigs fiasco are cited as examples (here).  I’ve seen groupthink operate on IT projects, and to me the agile method’s effectiveness in enabling groups to work together means agile projects are particularly susceptible.

This post reviews groupthink then discusses how the architect on an agile project might help prevent it.

Groupthink

From the Wikipedia article on groupthink, “groupthink is a type of thought exhibited by group members who try to minimize conflict and reach consensus without critically testing, analyzing, and evaluating ideas. Individual creativity, uniqueness, and independent thinking are lost in the pursuit of group cohesiveness. During groupthink, members of the group avoid promoting viewpoints outside the comfort zone of consensus thinking…Highly cohesive groups are much more likely to engage in groupthink, because their cohesiveness often correlates with unspoken understanding and the ability to work together with minimal explanations.”

In my experience risk of groupthink can manifest in several ways on IT projects:

• Not Invented Here: Successful teams that work through conflict can settle into a shared culture that resists new ideas from outside the team.
• The Know It All: Less successfully integrated teams can be dominated by a single strong-willed individual, and can habitually avoid conflict by accepting without question the ideas of that one dominant team member.
• The Abilene Paradox: Team members sometimes collectively decide on a course of action that no one on the team likes, when each member actually disagrees with the decision but mistakenly believes that their own preferences are counter to the group’s.

The Agile Architect

According to the Psychologists for Social Responsibility, the standard remedies for groupthink include this: “At least one articulate and knowledgeable member should be given the role of devil’s advocate (to question assumptions and plans)”. Of course the architect is an integral part of the overall project, but the skilled practitioner participates with the Agile team while maintaining separateness in order to remain a source of ideas from outside the team, and therefore provide a counterweight to groupthink by recognizing it and taking measures to prevent it.  Andrew Johnston’s site agilearchitect.org describes some of the ways the architect is in but not totally of the team (here). Among the architect’s responsibilities, he or she:

• Ensures “the delivered system is consistent with the agreed architecture, and will meet the requirements”
• “Is frequently an evangelist for new or different technologies, processes or solutions”
• “Acts as a bridge between developers, managers and other communities, and spends much of his time translating and mediating between them”
• “Recognizes the wide range of stakeholders, and their needs and concerns.”

While core agile team members are immersed in the scope and design that defines the current sprint, the architect retains a larger perspective that encompasses alternative designs, emerging technologies, business fit, stakeholder concerns, and more. The architect is therefore uniquely positioned to recognize groupthink effects on the team’s technical activities. Here are two examples of how that works on agile projects:

• Estimations and retrospectives: Mark Needham, in this post, cites risk of groupthink in agile estimation sessions and retrospectives.  The architect can address both of these risks. In estimation, the architect brings the diverse perspective that Mr. Needham says is important when team members estimate incorrectly due to incorrect team-shared assumptions. In retrospectives, the architect can be the one to increase the “safety” of different perspectives by raising or encouraging others to raise “things that have gone well, not gone well, and things that are confusing”.
• Work product reviews: I’m an advocate of code walkthroughs and design reviews, and making them explicit sprint tasks. The team can set aside an hour or two a week to review one or two representative work products in order to share ideas, ensure quality, and uncover overlooked errors or opportunities. In this forum the architect has the opportunity to reinforce quality work that is aligned with the requriements and architecture, or supportively correct deficiencies.

Of course there are risks:

• The architect shouldn’t be the know it all: In some cases the architect can be the strong-willed individual who stifles creativity and causes the team to avoid conflict.  Strong teamwork and interpersonal skills are core to the agile method, and those who staff the project must include those skills in selection and evaluation of the architect.
• Keep the architect different: If the architect is a core member of the team, he or she can become integrated into the group and therefore part of a groupthink dynamic.  For this reason, I advocate architects being assigned part-time to agile efforts. Otherwise, the architect risks becoming the extra developer, as near term sprint tasks expand to fill the available team bandwidth.  Consider sharing the architect among two or three projects, or assigning him or her responsibility for technical strategy and planning.

I just received this from the SQL Saturday crew: “We regret to inform you we’re postponing SQL Saturday #30 until 10 April 2010 due to the weather forecast for Friday and Saturday in Richmond VA.  We contacted the Hilton Garden Inn at 804-521-2900 to discuss reservations, but you will need to contact them about your reservation individually. (Apologies – we tried!)

“If you know you will not be able to make the new date, please cancel your registration on the SQLSaturday website.”

—— 

On Saturday 1/30 the local SQL Server community will host an event surely of interest to anyone in the Central Virginia who works with SQL Server databases.  Review the program and sign up here: http://www.sqlsaturday.com/eventhome.aspx?eventid=34.  I’ll be hosting two sessions, “The Business End of Data Modeling” and “Normalize Metadata for Data Integration Analysis.”  After those two morning presentations I’ll look forward to seeing many of the other presentations.  I’ll see you there, and watch this space for future posts on both topics.

According to Mr. Inmon, virtual data warehousing reminds him of the carnival game called whac-a-mole.  He says “just when you think this incredibly inane idea has died and just when someone has delivered what should have been a deathly blow, out it pops again from another hole.” There’s just a very informal definition of virtual DW in Mr. Inmon’s post (remember, he says he’s whacked this mole before), but, as I interpret, he’s talking about a system built after a decision to avoid all the expense of building a data warehouse by just having a query engine that pulls the data from wherever it lives. Mr. Inmon argues that a query accessing diverse databases would leave data integration to the user, and there’s no guarantee that two users would integrate data the same way.  He cites virtual database query inefficiency risks and, on the assumption that the query is trolling operations focused databases, says that source data would be “tuned” to operational rather than informational specifications for history retention and completeness.

Mr. Inmon’s ideas drew quick reaction from Mr. Kobelius and Neil Raden.  Each in his own measured way stresses that integration can be compatible with distributed architectures, and that there is a DW solution architected for efficiency that includes effective data integration from diverse sources: the Federated Data Warehouse.

Experience and emerging tools reinforce their point.  According to a colleague at CapTech, for smaller organizations “you can deal with this issue using a BI tool with a metadata layer that has joins predefined: the data integration is done by the BI metadata modeler.”  Another CapTech’er cites mashup as a potential quick and dirty approach.  Check out “7 Mashups Every Company Needs” here.

A well-architected federated warehouse certainly can integrate and deliver data, maintain history, and enable a “single version of the truth”, perhaps in a more timely manner than a “traditional” DW architecture.  On this question the devil is in the specifics of the situation.  It is difficult to argue one way or another out of the context of a real project in a real organization.

However, even though it certainly has a technical side, data integration is first a business activity.  Sometimes when we apply terms like “semantic rationalization” to software components, we in IT start believing you can actually build a machine that does the things you need to do to rationalize data semantics, like figure out the corporate definition of a customer.  Of course all we can do in IT is to build the empty shell.  The real work happens when business people from departments whose data is being integrated sit down and decide how they are going to define “staff member”, “customer”, and so on.  Only business professionals can say, for example, whether they want to include contractors in staffing reports or whether the term “customer” includes homebuyers under contract but not yet closed.

Integration tools that support data warehouses, whether centralized or federated, are only as good as the business consensus behind them. The consensus behind integrated data is arguably more rewarding to the business that the tools because with consensus on critical objects and events come non-IT-specific improvements like reduction of repetitive and conflicting business processes, reduced communication breakdown due to terminology disconnects, and more.

To me the beauty of the Inmon DW model is that it provides a mechanism that can assist an organization in evolving toward improved information maturity.  Organizations achieve some benefit by simply integrating data into a single data warehouse.  However, the data warehouse also makes source data quality problems obvious and blatantly reveals differences in data meaning from one operational source to another.  So the warehouse delivers some benefit early and also shows how much better it would be if data were integrated.  It therefore becomes a tool for identifying, assessing, prioritizing, and motivating correction of data deficiencies.

For organizations not so far along on the maturity curve, the additional complexity of the federated warehouse tends to obscure this data quality feedback loop. Federation based on drawing from operational sources integrates data from a set of different databases built toward different architectural goals.  On the other hand, the logical data model for the enterprise warehouse is the enterprise data model, and its architectural objective is to integrate enterprise data to provide a single source of truth.  Therefore, the enterprise data warehouse provides an architectural focal point for integration.  It isolates responsibility for improving data integration crisply at either the source or the warehouse, and — within the framework of solid information management strategy, management, and facilitation — motivates diverse business players to work toward consensus definition of enterprise data.

Federation, or virtual data warehousing if you will, can be the best strategy for the mature organization that has already integrated business data to a consistent enterprise view.  For the rest of us, the single centralized warehouse with its unambiguous architectural goals and borders seems the shortest distance to achieving the business benefits of data integration.

I recently read Testimony, Solomon Volkov’s controversial publication of the memoirs of Dimitri Shostakovich, the great 20th century Russian composer.  There were three different individuals in the book who demonstrated three different ways of “leading”, or behaving with character within a group.  (See the note at the end of this post on the question of authenticity)

The Cheerful Individualist

Volkov presents Modest Mussorgsky, known to most of us today as the composer of Pictures at an Exhibition, as an eccentric but cheerful individualist who would listen attentively to criticism as “everyone who felt like it harangued and criticized him…When he was criticized, he kept quiet, nodded, almost agreed.  But the agreement lasted only as far as the door; once he was outside, he took up his work again, like one of those dolls you can’t knock down.”  Mussorgsky was a unique individual with a unique musical voice, and in Volkov’s account had the confidence to overcome negative reaction to harsh feedback.

The Enabler

Alexander Glazunov, known as the “Russian Brahms,” was head of the Leningrad Conservatory from 1906 to 1928.  Glazunov was Shostakovich’s mentor during his time at the conservatory.  Glazunov enjoyed the company of young musicians, “performers came to his house every day”.  From Shostakovich’s account of Glazunov I’m reminded of a teacher I had who didn’t seem to teach at all, but by being welcoming, cheerful, and obviously talented and passionate about the subject matter, created an atmosphere in which we couldn’t help but learn.  More importantly, Glazunov saved lives during years of starvation and repression in Russia by “giving away his salary to needy students” and saving Jewish musicians from repression in their home towns by signing petitions for them to live in Petersburg without having them play for him.  As Volkov atrributes to Shostakovich after relating this story: “All things in life can be separated into the important and the unimportant.  You must be principled when it comes to the important things and not when it comes to the unimportant.”

The Supportive Subversive

Volkov’s Shostakovich lived bemused in a world of individual and institutional stupidity.  For example, one official order demanded “a quartet of 10 musicians.”  In this world  Shostakovich trod a fine line but never crossed far enough over to draw retribution.  Finally Shostakovich (among others) was denounced as a “decadent formalist” at the first Composer’s Congress in 1948, perhaps in part due to his 8th Symphony.  It was a solemn response to the end of World War II rather than the expected victory celebration.

Life during the Stalin years and World War II was characterized by deprivation and repression.  Many of Shostakovich’s friends and associates were denounced, exiled to Siberia, or killed in mysterious circumstances.  As a leading Soviet composer Shostakovich provided the soundtrack for the regime.  However, to the careful listener it seems not to celebrate the Stalinists but rather to channel the anguish of the people.  A telling example is his Fifth Symphony, ostensibly a joyous tribute to Party but arguably a veiled protest (see this analysis by Michael Tilson Thomas).

Volkov’s Shostakovich seems to have blundered on in spite of the worst possible conditions, sublimating his genius into musical irony and thereby doing what little he could in small ways to help his fellow Russians.

–

Many question Testimony’s authenticity, but this from the Wikipedia entry on the book might sum up a reasonable assessment: “the book gives a true picture of the political situation in the USSR and correctly represents his father’s political views, but [Maxim Shostakovich] continues to speak of the book as being ” ‘about my father, not by him.’ “  I’m neutral on whether or not Testimony is authentic, and whether Shostakovich was a toady of Stalin’s regime or an undercover dissident.  This post reflects subjective impressions from the book, nothing more.

Many see IT as application of technology to solve business problems. 

Of course, this is true but it leaves out the third element, which is to apply the right architectural pattern to solve the problem.  For example, when the business problem is that reporting is slow and reports from different departments don’t match, the astute IT professional immediately thinks in terms of a data warehousing pattern employing technologies like databases, extract-transform-load (ETL) tools, and multi-dimensional reporting suites.

A strategy based on the tools alone may solve the immediate problem, but understanding the solution-pattern enables the IT professional to bring to the business the additional benefits that come with the pattern, organizational and IT support impacts, and any risks that might emerge by applying the pattern.  In the data warehousing case the informed architect might cite improved executive dashboards and ability to drill down to root causes from summary reports, the need for data stewardship, and potential long term increase in data storage capacity needs.

Beyond that, when the architect lacks the pattern approach he or she seems to the business person like Bob Dylan’s debutante: “Your debutante just knows what you need, but I know what you want.” On the projects I’ve been on where designers lacked a pattern-based perspective the technical team did exactly what the business folks said they wanted, but for the most part didn’t contribute to the business value of the solution.  On projects like this developers slavishly ensure the solution matches each and every requirement, rarely bring to the table new business requirements that are logical consequences of the design, and tend to avoid questioning defined requirements even if they are contradictory or counterproductive.

Sure, patterns aren’t strictly necessary.  An outstanding architect can design from whole cloth an original solution that precisely matches business need.  However, that’s not how outstanding architects do business, at least the ones I’ve known.  In my experience the best IT architects know patterns, like data warehousing, SOA, and others, well enough to match the business problem to the right pattern and then evolve the architecture from the generalized pattern into a problem-specific architecture based on the particulars of the business problem at hand.

Some examples of common solution patterns in the world of business IT are Data Warehousing, Master Data Management, and Service Oriented Architecture (the Wikipedia article on this one is preliminary at this writing but still a good intro for the uninitiated).   Those interested in a more technical introduction to patterns might start with Avel Avram’s quick intro at InfoQ (Membership at InfoQ is required but free and worthwhile).

And of course, apologies to Mr. Dylan for the title…

Some claim that the relational dbms may be out of style very soon.  Cool new “cloud computing” and “SaaS” apps and services  delivered over the internet seem to be popping up everywhere – just look at Salesforce.com, the well-established Customer Relations Manager vendor, and the many cloud-based PC backup sites.  As part of that trend, Amazon, Google, Microsoft and others offer database services over the internet that don’t look much like relational dbms’s.  Some supporters of the cloud-db options seek alternatives to the standard relational DBMS (note this widely read article).  Of these, many are OO developers.  There’s a fundamental dissonance between OO and relational approaches, requiring an intermediate object/relational mapping (ORM) layer for OO systems to operate effectively with relational DBMSs.  Many of the new cloud-db options are open source, lightly structured data services provided via the internet, capable of storing and delivering large data stores for high availability, fast response applications.

The convenient thing about relational databases is that they pretty much match the business view of data, and therefore give business people and developers common ground.  A well thought out relational data model is one way to express the inherent structure of business rules (see this previous post).  A relational model at the back end of a custom-built system means that both developers and business people can talk about the real guts of a system in ways that make sense to both, like this:

• Developer to business person: “Should we allow a part_order to include items from only one division”
• Business person to developer: “After a call from our shipping department, I ran a query on the part_order table and found a that there is a part_order with null shipper_phone_number. I thought it was a required column, what’s up?

How’s it going to be when those comments don’t reflect the underlying structure of the database?  Today’s cloud db offerings vary in structure, but tend to favor highly efficient and flexible models like name-value pairs, and avoid the overhead required by semantic layers like the relational model.  According to the MongoDB site, “by reducing transactional semantics the db provides, one can solve an interesting set of problems where performance is very important.”

In such databases the structure of the data will be hidden from business people; there will be no shared business/IT view.  Rather than talking with business people about the actual database structure we’ll talk about its custom abstraction, and when things go poorly with performance and functionality the developer will in effect say “trust me on this one” to the business person rather than explaining what’s up.

For a long time cloud databases will be another option alongside the relational model, but the more prominent cloud databases become the more difficult it will be for developers and business people to communicate about business data in IT applications, and it could be a serious challenge for developers to learn to cross that communications gap without the bridge provided by the relational model.

Excerpt from "Show Me the Money: A DM/BI Business Value Primer", Bob Lambert and Tri Truong, Information Management Special Reports, March 24, 2009

1. Intangible benefits don’t count.
2. BI has no inherent value.
3. Senior managers often make decisions about future outcomes with insufficient data.

Intangible Benefits Don’t Count: An effective business case communicates tangible future value in a convincing way.  An argument has a chance of convincing a skeptical reader if the reader agrees that the argument’s assumptions are reasonable and that the conclusion follows logically from the assumptions.  Quantifying financial metrics like Return on Investment (ROI) or Net Present Value (NPV) help build the case, but such measures are credible only if readers agree with the underlying assumptions and the logic built upon them.

BI has no inherent value: We in the BI field believe that any organization’s fortunes would improve if it rationalized its data stewardship, integrated its data, and applied analytics creatively in management and operations.  However true, that view must ring hollow to senior business managers.  Without a compelling and motivating story about how a new system contributes to revenue or reduces costs, that system’s business case stops dead in its tracks.  Of course sometimes “someone at a high level” just wants BI, but organizations don’t often embark on BI efforts without first evaluating tangible costs and benefits.

Senior Managers Make Decisions about Future Outcomes with Insufficient Data: Although BI practitioners must make a convincing case for future business value, there’s room for uncertainty.  Executives and senior managers aren’t highly compensated for playing it safe, but rather for understanding current conditions and setting direction based on educated but sometimes courageous predictions of future conditions.  A successful BI business case matches or extends the executive’s knowledge of current conditions and expands his or her view of potential future outcomes of near term actions.

That said, it seems to me that there’s something missing.  Consider one BI manager I know who has fractious relations with his business customers.  I won’t go into detail, but trust me, relations have been rocky, and reviews from key business players poor.  What this person does extremely well is to build rock-solid, reliable systems, deliver on time, meet business needs, and ensure that the solution meets regulatory and audit concerns.  This BI group is essentially unchanged after many years, enduring even the recent recession in a devastated industry segment, and outlasting many of its critics.

To me, building a good relationships is important, but execution is the sine qua non of IT/business alignment.  Think about it.  Say you hire a really nice contractor to fix a leaky roof.  However personable he is, you won’t hire him to replace your windows if the roof still leaks after you’ve paid the bill.

My view: if you want to do it right adopt Villar’s and Kushner’s excellent suggestions but the fundamentals remain the same:

1. Either (a) present a robust business case that all accept, or (b) pay attention to what you’ve been asked to do
2. Deliver what was requested/promised in 1
3. When things change go to 1