It is an old story: about 30% of IT application projects succeed, 45% are “challenged,” and the other quarter fail altogether.   That’s the consistent result over the years of the Standish Group Study of Project Outcomes.  Jorge Dominguez, here, displays a chart of the remarkably similar results since 1994.  Not a pretty picture, right?  Some question the validity of the Standish studies, but Scott Ambler parallels the Standish story in a recent Dr Dobbs column called “Lies, Great Lies, and Software Development Project Plans,” which itemizes the strategies commonly used by IT project managers to “stay out of trouble” when schedule/budget results don’t match initial estimates.  For example, “18% change the original schedule to reflect the actual results”.

The frequent reaction to stats like these is to scapegoat the IT folks by finding fault with their tools, processes, or skills.  If we just had a more efficient methodology, or a slick development suite, or a more highly skilled team, or a better project process, were more agile, or whatever, then application projects would be on time, resulting systems would be faultless, and we could drive down outrageous IT costs.

Mr. Ambler plays out the IT perspective in this column about Agile project estimation.  Here’s how I paraphrase the gist of his logic: you can’t accurately estimate early on due to normal uncertainty of high level requirements; you will get new requirements that will expand scope.  If you offer a range of predicted outcomes, management will hold you to the most optimistic, which will turn out to be a gross underestimate. Your best strategy is to use an accurate estimation method (net or average velocity, described in the article), and be straight with management even if they “don’t like what they are hearing”.

Based on my experience that sequence of events can be distressingly true to life.  But I’ve been on the other kind of IT project as well, the kind that ends on time, stays on budget, and satisfies the business. The scope of Mr. Ambler’s article doesn’t include asking why the requirements are changing so drastically, but if it had, maybe he would have asked questions like these:

• Does the business unit involved have defined objectives and processes?
• Are strategic and tactical business objectives of the project defined, and do they support the business unit’s objectives?
• Are the critical business success factors of the project defined?
• Does the new project change business processes and if so have the new ones been defined yet?
• Are there documented business requirements and a process for evaluating/approving changes?

Based on what I’ve seen requirements shifts that nearly double the cost of an IT project, like the ones Mr. Ambler describes, result from inadequate business definition and business requirements analysis.  Of course it is true that the “initial high-level requirements and architecture envisioning” early in a typical project only enables an estimate “in the +/- 30% range”.  But effective requirements and architecture definition, when based on effective business definition, can enable a much closer estimate.  Further, it is possible to accurately estimate how long it will take to provide the more accurate estimate.

In some cases the business definition prerequisites clearly don’t yet exist at the outset of an application development effort.  One thing I’ve seen work in such cases is to divide the project into two separate projects: one to closely define the effort, and if needed make up for any lacking business definition, and another to build, test, and install the application. In between the two phases management will have the opportunity to review the costs/benefits of the project and evaluate whether to continue or not.  This post describes one project that successfully used that strategy.

Please don’t read this as repudiation of Agile methods and advocation of “big requirements up front”.  Agile methods work well whether or not business prerequisites are defined, but they seem not to work well when (1) the project’s goals shift with evolving business definition and (2) the plan and budget aren’t adjusted accordingly.

Business definition, completed as part of a discrete requirements phase, that leads to a management decision to continue or not, gives the team the opportunity to build on a solid business foundation.  It also gives management a reasonable estimate and a chance to bail if the project isn’t worth it.

As good as it is, to me the article conjured an unlikely vision of the Dead as business men in hippie drag self-consciously making strategy decisions that altered the marketing landscape. I agree that the Dead took the actions cited on purpose, but I believe core product, not marketing strategy, consumed the band’s energies during its formative and peak years.   Could it be that their innovative market strategies grew organically from a quality product, where quality included the entire fan experience?

I hope those teaching Grateful Dead management include this:

• Develop and maintain a strong product
• Risk everything to make the product top quality
• Perfect every detail of the customers’ experience

Develop and maintain a strong product

Given their image, it is easy for non-fans to lose sight of the fact that the Dead were good at what they did.  While one could argue aimlessly about how good they were, they certainly didn’t suffer the unevenness of musicality of some rock ‘n’ roll bands, and they proved it live most days of any given year.  Phil Lesh’s memoir recalls the words of Dizzy Gillespie, passing by an outdoor concert in the late ’60s: “Those cats can swing!”  Lesh himself was a student of jazz and then avant garde composers like Stockhausen before joining the Warlocks, as they were first named.  Mickey Hart was and remains a student of primitive percussion. According to Lesh, Jerry Garcia learned to play the pedal steel guitar in a matter of weeks, quite an accomplishment for an instrument that requires both hands, both feet, and knees to control.

They applied these skills in a joyous, educated, and well-crafted way that reflected musical practice, discipline, and breadth, gluing songs of separate genres together with rocking transitions that frequently dissolved into organic free jams, only to come back together somewhere entirely unexpected.  Even beyond their widely varying originals, their diverse covers were an encyclopedia of mid-twentieth century folk/pop, including Harry Belafonte, Merle Haggard, Buddy Holly, Marty Robbins, and many others.

Risk everything for top quality

Early on the members of the Grateful Dead were steadily less satisfied with the quality of the then-prevailing live sound technology.  Their dissatisfaction peaked in the early seventies, when, according to soundman Dan Healy, they sunk “90 percent of their total earnings” toward building a sound system so that their faithful could “go to the show and hear the heavenly choir, so to speak, through the heavenly sound system.”  Their sacrifice to quality was such that “there were times when we spent the money on speakers and nobody got paychecks, from Jerry on down.”

The dream emerged as the Wall of Sound, a behemoth sound system that “took four semi-trailer trucks and more than 20 crew members to haul and set up.” While the wall of sound itself became too costly to lug around with the fuel crisis of the mid seventies, the Dead retained top notch sound quality after its demise, and its technical innovation remains with us today.  For example, the wall of sound introduced the practice of mic-ing each instrument separately, enabling the sound engineer to deliver a live show with the balance of a recording – standard practice today but revolutionary at the time.

Perfect every detail of the customer’s experience

The Grateful Dead were central to the Haight-Ashbury hippie scene in the sixties, a culture that embraced “Eastern philosophy, championed sexual liberation, promoted the use of psychedelic drugs which they felt expanded one’s consciousness, [and] used alternative arts, street theatre, folk music, and psychedelic rock as a part of their lifestyle and as a way of expressing their feelings.” (here)  Hippie culture came together in “happenings”, free form gatherings “during which music, psychedelic experimentation, a unique sense of personal style and … light shows combined to create a new sense of community”.   The focus of the happening was on the totality of the experience, bringing all elements together to join the participants and spectators (in as much as there was such a distinction) into a single mind.

As freaky as all this sounds, the happening’s focus on the shared gestalt surely fostered the Grateful Dead’s attention to audience experience.  I saw the Dead only once, but it was an outstanding show. Every detail seemed to have been choreographed for the experience of the listener.  The sound was big but not loud and every nuance was clearly audible. Stage and house lighting were perfect. Security was ubiquitous, proactive, and polite.  No detail interfered with band/audience community. Needless to say the Dead rocked the house.

—

Sadly, it couldn’t last forever.  A reviewer of Lesh’s book recounts the decline that started during the late eighties as drug-related health problems, constant touring, the changing nature of their fan base, and the sheer weight of their growing organization bore down on the band. It seems every long, strange, trip has its long strange decline – and perhaps there are management secrets there as well.

Excerpt from "Show Me the Money: A DM/BI Business Value Primer", Bob Lambert and Tri Truong, Information Management Special Reports, March 24, 2009

1. Intangible benefits don’t count.
2. BI has no inherent value.
3. Senior managers often make decisions about future outcomes with insufficient data.

Intangible Benefits Don’t Count: An effective business case communicates tangible future value in a convincing way.  An argument has a chance of convincing a skeptical reader if the reader agrees that the argument’s assumptions are reasonable and that the conclusion follows logically from the assumptions.  Quantifying financial metrics like Return on Investment (ROI) or Net Present Value (NPV) help build the case, but such measures are credible only if readers agree with the underlying assumptions and the logic built upon them.

BI has no inherent value: We in the BI field believe that any organization’s fortunes would improve if it rationalized its data stewardship, integrated its data, and applied analytics creatively in management and operations.  However true, that view must ring hollow to senior business managers.  Without a compelling and motivating story about how a new system contributes to revenue or reduces costs, that system’s business case stops dead in its tracks.  Of course sometimes “someone at a high level” just wants BI, but organizations don’t often embark on BI efforts without first evaluating tangible costs and benefits.

Senior Managers Make Decisions about Future Outcomes with Insufficient Data: Although BI practitioners must make a convincing case for future business value, there’s room for uncertainty.  Executives and senior managers aren’t highly compensated for playing it safe, but rather for understanding current conditions and setting direction based on educated but sometimes courageous predictions of future conditions.  A successful BI business case matches or extends the executive’s knowledge of current conditions and expands his or her view of potential future outcomes of near term actions.

In his book Money Mischief Milton Friedman made the point that money has no intrinsic value: “The value of money is the value people attribute to what they want to exchange, no more, no less.” Likewise, data has no value in itself.  Its value is derived from people’s desire to know about the things the data describes, and how reliably and accurately it describes those things.  So an organization’s data, like its money, is not a resource in itself.  It is an asset that represents the resources that an organization manages and controls.  It follows then that data management should look a lot like money management.

A cornerstone of our economic stability is consensus that organizations must manage money well and make their internal money management visible to investors, regulators, and independent standards groups.  We’ve evolved a standard for money management where a department represented by a C-level executive administers formal accounting, budgeting, planning, and financial reporting.  The organization evaluates every manager’s compliance to money management policies, and independent auditors evaluate the organization’s soundness in terms of its money management.  Accounting professionals meet rigorous, generally respected certification standards.

Overall, our volume of online purchases and use of FDA-approved drugs, for example, attest to our general confidence in current data management practices.  But still, data  professionals know that it could be a lot better.  Scarcely a week goes by without another scandal involving lost customer data, and consider these snafus:

• This article cites multiple non-compliant databases as a significant contributor to the chaos in reuniting families in the wake of the Katrina disaster
• “The Mars Climate Orbiter, a key part of NASA’s program to explore the planet Mars, vanished in September 1999 after rockets were fired to bring it into orbit of the planet. An investigative board later discovered that NASA engineers failed to convert English measures of rocket thrusts to newtons, a metric system measuring rocket force, and that was the root cause of the loss of the spacecraft. The orbiter smashed into the planet instead of reaching a safe orbit.” (cited here)
• One Fortune 1000 services company carried separate customer records in each of its operating units resulting in a number of anomalies visible to the customers.  For example, the same customer would receive separate invoices with different terms for each of the services purchased from the company.

In parallel with emergence of these types of issues, regulators and industry associations have set data management standards for many industries and practice areas.   Food and consumer product safety rests on a regulatory foundation of correctly recording and managing results of inspections.  The International Air Transport Association sets standards for safety data collection and management.  Likewise, the US Food and Drug Administration and other governing bodies set clinical safety data management and reporting standards.

It is just a matter of time before the many separate externally imposed data management guidelines congeal into a a set of general best practices that apply across the organization.  Then investors, regulators, and standards groups will hold organizations responsible for effective data management in the same way they are held to account for effectively managing money. An internal department represented by a C-level executive will administer formal data management standards and procedures.  The organization will evaluate every manager’s compliance with data management policies, independent auditors will evaluate the organization’s soundness in terms of the quality of its data management, and data management professionals will be held to rigorous, generally respected certification standards.

Farfetched? Maybe.  But it isn’t farfetched to think that as a society we’ll begin to recognize what data professionals have known for a long time: that the quality of an organization’s products, its care of and protection of its customers, workforce, resources, stewardship of the environment, and even its financial health depend to a significant degree on sound data management practices.

—

Here are some resources on data management:

DAMA, the organization for data management.

The Wikipedia page quotes this definition: “Data management is the development, execution and supervision of plans, policies, programs and practices that control, protect, deliver and enhance the value of data and information assets.”

Data Stewardship Strategy: 6 Keys to Success by Jill Dyché: “As executives increasingly agree that data is a corporate asset, they are also funding data governance and data quality efforts more willingly. But … entrenched organizational behaviors are much more difficult to shift. Many companies have introduced the role of data steward before fully defining the role. In these cases, the beleaguered data stewards are doomed before they even begin. ”

Leverage Data Quality to Build an Effective Enterprise Architecture by Mark Amspoker.  “It might be time to rethink the notion that effective information architecture development will solve the data quality problem.”

Guidelines for Responsible Data Management in Scientific Research from the Office of Research Integrity, US Department of Health and Human Services.   “Data management is one of the essential areas of responsible conduct of research, as outlined by the Office of Research Integrity. This educational course will educate new investigators about conducting responsible data management in scientific research.”

Here’s Paul’s description of the Sancho Panza test:

Think of Don Quixote [DQ] as the data-quality specialist or even the data management specialist or software vendor, bringing to the world his specialist’s perspective and vocabulary and enthusiasm, influenced by the books he’s read, visioning everyday business practices, with his value added, as goldmines for the organization.  Meanwhile Sancho Panza represents the person who does a practical job every day, who knows what works around here and what doesn’t.

I advocate to Data Quality (let’s call it DQ) consultants that they listen to this Sancho Panza, and consider themselves as Don Quixote.  Sancho doesn’t know much about data, but he knows what he likes… He’s open to listening, but slow to change, and he’ll tell you what he thinks.

Paul’s article reminded me that as a child I thought the problem with Don Quixote was that he tilted at windmills and attempted to ambush acting troupes because of his bad eyesight.  Of course this is not the case, but to me it provides a relevant perspective on data quality in many organizations.

Here’s the problem I’ve seen play out on a number of IT application projects:

1. A high level business study recommends replacement or improvement of a current application.
2. The organization approves the project described in a business case citing benefits named in the business study and costs detailed for infrastructure, package software, and application development, but data-related costs are glossed over or left out entirely.
3. The project begins with a requirements phase that collects hundreds of imperative statements (“The system shall…”)  from business people who will use the system.
4. Late in the requirements phase, the team finds that data integration work in system interfaces will be more complex than expected.  A common example: the project requires changes to a feeder application with no documentation and no in-house support expertise.
5. Project leadership goes back to the sponsor seeking more money.

In these situations the business case was incorrect because it did not account for all of the costs of data integration.  I’ve seen projects weather steps four and five well, but often discovery of previously unseen data complexity starts a disruptive chain of events.  (Sadly for the project manager, such situations are often seen as a failure of project management and corrected accordingly, but that’s a topic for another post.)

In my view the root cause of unforeseen data complexity on projects is the lack of a data constituency in current IT. It is only recently that success of companies like Google and Amazon have motivated emergence of data as a key business resource in the collective consciousness. Famous success stories notwithstanding (see this link), there are relatively few senior IT managers with data quality backgrounds.  Conversely, many rose through the ranks of the infrastructure, application development, or business (process) analysis groups.

It will be a while before, for example, a Mobil CIO’s predecessor jobs include definition of a metadata repository or elimination of multipurpose data, but in the meantime here’s what we can do:  add a business case to the application lifecycle as the last step in requirements.  Stop the project when the real costs are known, recalculate the cost/benefit, and ask the sponsors if the project should continue.  Give Sancho (in this case the project team) a chance to speak to the reality of the situation, and hand to Don Quixote (project sponsors) the eyeglasses of in-depth visibility into real costs. If the decision is to move ahead with the project, then all share the same vision and the sponsors have endorsed the actual project, not the fuzzy image from earlier on that might have been a windmill.

Today: inconsistent data of uncertain quality blurs enterprise view and restricts planning

Today managers, planners, and analysts lack the information required to run the organization as a single enterprise rather than a collection of diverse units.

• Data quality in IT applications varies to the point that, outside financials, it is impossible to gather consistent data supporting an enterprise view of operations.
  • Application development efforts have focused narrowly on departmental interests without accounting for enterprise concerns, making application data incomplete in describing business processes and inconsistent with data in other applications.
  • Focus on departmental concerns and tight development timelines has resulted in incomplete validation of data critical to the enterprise but not critical to the application’s focus.  For example, customer demographics are not critical to the sales process and therefore zip codes and telephone numbers are not consistently collected at point of sale, substantially reducing value of market analysis based on sales data.
• Enterprise planners work with only the highest level summaries of operational data, those summaries suffer large margins of error, and planners cannot definitively answer questions required to make critical business decisions.
• Regulators have questioned the validity and repeatability of reporting because of the organization’s heavy reliance on spreadsheets and manual processes in gathering and compiling data for reports.

Solution: enable sound planning and management by identifying data assets and setting processes to manage them

Empower an Enterprise Information Architect to lead an effort that (1) identifies data that describes the organization, (2) defines how to integrate and improve quality of that data, and (3) improves the ability of information technology to maintain data quality.

(1) Lead definition of an Enterprise Information Architecture identifying information required to manage the organization as a single integrated enterprise, and data quality standards that ensure that data supports enterprise goals.

• Identify and define events and objects critical to the enterprise

• Identify and define relationships among those events and objects and attributes that describe them

• Classify data managed by the organization by type (operational, statistical, financial, decision support, etc.) and define standards for managing and integrating each type.
• Compile the above into a plan that explicitly supports the enterprise strategic plan

(2) Working with senior business managers, put in place a program of data quality improvement that plans and executes specific measures and sustained commitment to improving data quality in business processes and IT applications

• Identify the business group responsible for maintaining quality and integrity of each business object, event, relationship, and attribute

• Identify for each data item of interest to the enterprise its “system of origination” and “system of record”.

• System of origination is the application that provides the entry point of a given data object to the organization.
• System of record is the application that is the source of record for the data object.

• Define and deploy standards and practices for for business process and IT application definition that support data quality and integrity standards

(3) Working with senior IT managers define and put in place standards for application requirements definition, data management, and metadata management to

• Define and deploy application development and interface standards that support data quality objectives.
• Ensure that application development efforts support enterprise data quality
• Continually monitor new developments in data management best practices and make that information available to the enterprise.

The project was an HR management software conversion from one commercial off-the-shelf software (COTS) package to another.  The company concerned was conservative about spending money.  A previous business case had proposed a similar project.  The problem with that business case was that the benefits were really tough to conceptualize, so the cost/benefit analysis relied on soft benefits like “improved access to information” and “more consistent reporting data”.  The folklore was that the CFO had physically thrown that business case out of his office.

Mr. Carpenter’s method was to divide requirements definition and implementation into two distinct projects, with a different business case for each.  Under his direction, we wrote a ~1m business case for requirements definition only.  We proposed that this first project would result in another business case precisely specifying the schedule, method, cost, and benefits of the implementation project.

According to John, “the approach we used would not be considered a textbook approach for an ERP (enterprise resource planning) implementation.  What we did was more of a strategy to address the the CFO’s concerns.  The company was very risk-averse so we needed a way to take out as much risk as we could.  This was a large project because it involved four major modules affecting the three main areas of HR, and the company wanted to know costs and benefits at each step.  Complicating matters, HR business processes and therefore requirements were not clearly understood – the HR department seemed to rely on on the job training rather than documented procedures.  So we presented the first phase as an investment into understanding HR processes, as well a precise roadmap for implementation.”

This first business case was accepted by that same CFO and we got started on the 7-month effort. We brought in a consulting team experienced in the proposed COTS package, and followed their lead in requirements definition and prototyping.  During the prototyping step they walked HR staff through each relevant function in the software package, detailing how to configure the package for their specific needs and where we’d need to customize it.   The result was a definitive, detailed document that showed how the package fit HR process and how it would need to be customized.  Then, we used those results to build a business case that included specific configuration, customization, hardware, and software costs, as well as the process and organizational changes that would be required, not to mention the benefits that would accrue.  The business case showed substantial improvement, predicting real financial benefits within 4 years.  Even better, on a depreciated basis the project literally was almost free, costing only $1,800 in the first year and returning benefits thereafter.

The business case was accepted by the company’s executive committee and the project started.  It ran exactly as outlined by the results of the requirements effort, with very few of the nasty surprises often typical of large projects, and it tracked to forecast schedule and budget.

Proving that no good deed goes unpunished, the company, whose core business was real estate, in effect folded in the financial crash of last autumn ’09 , one month from implementation.

At any rate, the lesson I took away from the effort was that dividing requirements and development into separate projects gives business visibility into a project, helps manage financial risk, and enables the project to ground predictions rather than guessing at costs and benefits before they can be known.