- The Business End of Data Modeling (2.5m powerpoint presentation)

- Normalize Metadata For Data Integration Analysis (5.5m full version, zip including presentation and code samples)

- Normalize Metadata For Data Integration Analysis (small) (2m reduced size version, graphics removed from ppt file)

Here are some quick notes for those looking to run the Metadata prototype:

The prototype metadata database includes SQL Server 2008 data definition language and data manipulation language (DDL and DML) needed to create the database and populate it with tables and columns from Microsoft’s AdventureWorksDW sample database. It also includes a sample requirements spreadsheet and source-to-target map, and SSIS jobs to load the spreadsheets to corresponding metadata tables. These define fictional requirements and mappings to populate the AdventureWorksDW FACTInternetSales table from tables in the AdventureWorks sample database.

AdventureWorks and AdventureWorksDW are available here: http://msftdbprodsamples.codeplex.com/Wikipage (accessed 4/14/2010)

According to Mr. Inmon, virtual data warehousing reminds him of the carnival game called whac-a-mole.  He says “just when you think this incredibly inane idea has died and just when someone has delivered what should have been a deathly blow, out it pops again from another hole.” There’s just a very informal definition of virtual DW in Mr. Inmon’s post (remember, he says he’s whacked this mole before), but, as I interpret, he’s talking about a system built after a decision to avoid all the expense of building a data warehouse by just having a query engine that pulls the data from wherever it lives. Mr. Inmon argues that a query accessing diverse databases would leave data integration to the user, and there’s no guarantee that two users would integrate data the same way.  He cites virtual database query inefficiency risks and, on the assumption that the query is trolling operations focused databases, says that source data would be “tuned” to operational rather than informational specifications for history retention and completeness.

Mr. Inmon’s ideas drew quick reaction from Mr. Kobelius and Neil Raden.  Each in his own measured way stresses that integration can be compatible with distributed architectures, and that there is a DW solution architected for efficiency that includes effective data integration from diverse sources: the Federated Data Warehouse.

Experience and emerging tools reinforce their point.  According to a colleague at CapTech, for smaller organizations “you can deal with this issue using a BI tool with a metadata layer that has joins predefined: the data integration is done by the BI metadata modeler.”  Another CapTech’er cites mashup as a potential quick and dirty approach.  Check out “7 Mashups Every Company Needs” here.

A well-architected federated warehouse certainly can integrate and deliver data, maintain history, and enable a “single version of the truth”, perhaps in a more timely manner than a “traditional” DW architecture.  On this question the devil is in the specifics of the situation.  It is difficult to argue one way or another out of the context of a real project in a real organization.

However, even though it certainly has a technical side, data integration is first a business activity.  Sometimes when we apply terms like “semantic rationalization” to software components, we in IT start believing you can actually build a machine that does the things you need to do to rationalize data semantics, like figure out the corporate definition of a customer.  Of course all we can do in IT is to build the empty shell.  The real work happens when business people from departments whose data is being integrated sit down and decide how they are going to define “staff member”, “customer”, and so on.  Only business professionals can say, for example, whether they want to include contractors in staffing reports or whether the term “customer” includes homebuyers under contract but not yet closed.

Integration tools that support data warehouses, whether centralized or federated, are only as good as the business consensus behind them. The consensus behind integrated data is arguably more rewarding to the business that the tools because with consensus on critical objects and events come non-IT-specific improvements like reduction of repetitive and conflicting business processes, reduced communication breakdown due to terminology disconnects, and more.

To me the beauty of the Inmon DW model is that it provides a mechanism that can assist an organization in evolving toward improved information maturity.  Organizations achieve some benefit by simply integrating data into a single data warehouse.  However, the data warehouse also makes source data quality problems obvious and blatantly reveals differences in data meaning from one operational source to another.  So the warehouse delivers some benefit early and also shows how much better it would be if data were integrated.  It therefore becomes a tool for identifying, assessing, prioritizing, and motivating correction of data deficiencies.

For organizations not so far along on the maturity curve, the additional complexity of the federated warehouse tends to obscure this data quality feedback loop. Federation based on drawing from operational sources integrates data from a set of different databases built toward different architectural goals.  On the other hand, the logical data model for the enterprise warehouse is the enterprise data model, and its architectural objective is to integrate enterprise data to provide a single source of truth.  Therefore, the enterprise data warehouse provides an architectural focal point for integration.  It isolates responsibility for improving data integration crisply at either the source or the warehouse, and — within the framework of solid information management strategy, management, and facilitation — motivates diverse business players to work toward consensus definition of enterprise data.

Federation, or virtual data warehousing if you will, can be the best strategy for the mature organization that has already integrated business data to a consistent enterprise view.  For the rest of us, the single centralized warehouse with its unambiguous architectural goals and borders seems the shortest distance to achieving the business benefits of data integration.

In his book Money Mischief Milton Friedman made the point that money has no intrinsic value: “The value of money is the value people attribute to what they want to exchange, no more, no less.” Likewise, data has no value in itself.  Its value is derived from people’s desire to know about the things the data describes, and how reliably and accurately it describes those things.  So an organization’s data, like its money, is not a resource in itself.  It is an asset that represents the resources that an organization manages and controls.  It follows then that data management should look a lot like money management.

A cornerstone of our economic stability is consensus that organizations must manage money well and make their internal money management visible to investors, regulators, and independent standards groups.  We’ve evolved a standard for money management where a department represented by a C-level executive administers formal accounting, budgeting, planning, and financial reporting.  The organization evaluates every manager’s compliance to money management policies, and independent auditors evaluate the organization’s soundness in terms of its money management.  Accounting professionals meet rigorous, generally respected certification standards.

Overall, our volume of online purchases and use of FDA-approved drugs, for example, attest to our general confidence in current data management practices.  But still, data  professionals know that it could be a lot better.  Scarcely a week goes by without another scandal involving lost customer data, and consider these snafus:

• This article cites multiple non-compliant databases as a significant contributor to the chaos in reuniting families in the wake of the Katrina disaster
• “The Mars Climate Orbiter, a key part of NASA’s program to explore the planet Mars, vanished in September 1999 after rockets were fired to bring it into orbit of the planet. An investigative board later discovered that NASA engineers failed to convert English measures of rocket thrusts to newtons, a metric system measuring rocket force, and that was the root cause of the loss of the spacecraft. The orbiter smashed into the planet instead of reaching a safe orbit.” (cited here)
• One Fortune 1000 services company carried separate customer records in each of its operating units resulting in a number of anomalies visible to the customers.  For example, the same customer would receive separate invoices with different terms for each of the services purchased from the company.

In parallel with emergence of these types of issues, regulators and industry associations have set data management standards for many industries and practice areas.   Food and consumer product safety rests on a regulatory foundation of correctly recording and managing results of inspections.  The International Air Transport Association sets standards for safety data collection and management.  Likewise, the US Food and Drug Administration and other governing bodies set clinical safety data management and reporting standards.

It is just a matter of time before the many separate externally imposed data management guidelines congeal into a a set of general best practices that apply across the organization.  Then investors, regulators, and standards groups will hold organizations responsible for effective data management in the same way they are held to account for effectively managing money. An internal department represented by a C-level executive will administer formal data management standards and procedures.  The organization will evaluate every manager’s compliance with data management policies, independent auditors will evaluate the organization’s soundness in terms of the quality of its data management, and data management professionals will be held to rigorous, generally respected certification standards.

Farfetched? Maybe.  But it isn’t farfetched to think that as a society we’ll begin to recognize what data professionals have known for a long time: that the quality of an organization’s products, its care of and protection of its customers, workforce, resources, stewardship of the environment, and even its financial health depend to a significant degree on sound data management practices.

—

Here are some resources on data management:

DAMA, the organization for data management.

The Wikipedia page quotes this definition: “Data management is the development, execution and supervision of plans, policies, programs and practices that control, protect, deliver and enhance the value of data and information assets.”

Data Stewardship Strategy: 6 Keys to Success by Jill Dyché: “As executives increasingly agree that data is a corporate asset, they are also funding data governance and data quality efforts more willingly. But … entrenched organizational behaviors are much more difficult to shift. Many companies have introduced the role of data steward before fully defining the role. In these cases, the beleaguered data stewards are doomed before they even begin. ”

Leverage Data Quality to Build an Effective Enterprise Architecture by Mark Amspoker.  “It might be time to rethink the notion that effective information architecture development will solve the data quality problem.”

Guidelines for Responsible Data Management in Scientific Research from the Office of Research Integrity, US Department of Health and Human Services.   “Data management is one of the essential areas of responsible conduct of research, as outlined by the Office of Research Integrity. This educational course will educate new investigators about conducting responsible data management in scientific research.”

I’ve cited here the need to start data interface analysis early to avoid budget and schedule blow-ups when, as a result of not thinking early about interface complexity, data integration work turns out to be bigger and nastier than anticipated.

Early data study also helps business analysts elicit more detailed and accurate business requirements.  Say a mid-level football (soccer) team in the UK is looking to recruit a couple of strikers who can reliably punch home goals for the club.  The obvious data they seek is (1) the number of goals scored per game by each prospect, and (2) over their careers how much time have they spent on the bench due to injury.  At the same time, this club is building a strategic recruiting system to support growth into the higher echelons of English football.  A process-oriented requirements strategy (like the one described here) asks the team’s recruiters what they need to in order to get good people into the club, and often emerges with a list of statements about what the system will do (”The system shall provide an interface enabling entry of the following player statistics” or “The system shall provide a report ranking players by the following criteria:…”).

It isn’t necessarily wrong to start with process analysis, especially when backed up with formal techniques like use cases, data flow diagramming, or others, but addition of data analysis early provides ability to be far more perceptive into the real business needs.  Without interviewing anyone a data analyst can know that there are many goals in a game of soccer (OK, to some not nearly enough, but that’s another story), that the attributes of a game include location, weather conditions, date and time, whether it’s regular season or playoff, and more.  Attributes of a goal: time during the game; left foot, right foot, or head; did it come from a set play or in the run of play; from the left or right side of the field, and much more.

The analyst who knows the data and understands its structure can probe with questions like whether a player tends to score at the end of games, or would it be useful to find one striker who tends to score from the left side of the field and another who scores from the right?  By understanding the data an analyst can understand the business problem more deeply, build better rapport with business people  by asking more informed questions, and cross the business/IT communications gap to define the right requirements so that the right system gets built.

It may be just the organizations I’ve been exposed to, but in my experience data analysis isn’t typically part of the requirements effort.  Supporting this point, the author of the wikipedia page on business analysis entirely omits data analysis, apparently favoring a process-only approach.  On the other hand, object-based techniques offer a balanced approach, studying both data and process by representing things like goals, games, and players as objects with their own attributes and behaviors.  In addition, the International Institute of Business Analysts (IIBA) includes data-oriented along with process-oriented techniques in its Business Analysis Body of Knowledge (BABOK).

As process/data balance early on in the application lifecycle becomes more widespread analysts should generate more insightful requirements and, other things being equal, the success rate of IT application projects should improve.

Here’s Paul’s description of the Sancho Panza test:

Think of Don Quixote [DQ] as the data-quality specialist or even the data management specialist or software vendor, bringing to the world his specialist’s perspective and vocabulary and enthusiasm, influenced by the books he’s read, visioning everyday business practices, with his value added, as goldmines for the organization.  Meanwhile Sancho Panza represents the person who does a practical job every day, who knows what works around here and what doesn’t.

I advocate to Data Quality (let’s call it DQ) consultants that they listen to this Sancho Panza, and consider themselves as Don Quixote.  Sancho doesn’t know much about data, but he knows what he likes… He’s open to listening, but slow to change, and he’ll tell you what he thinks.

Paul’s article reminded me that as a child I thought the problem with Don Quixote was that he tilted at windmills and attempted to ambush acting troupes because of his bad eyesight.  Of course this is not the case, but to me it provides a relevant perspective on data quality in many organizations.

Here’s the problem I’ve seen play out on a number of IT application projects:

1. A high level business study recommends replacement or improvement of a current application.
2. The organization approves the project described in a business case citing benefits named in the business study and costs detailed for infrastructure, package software, and application development, but data-related costs are glossed over or left out entirely.
3. The project begins with a requirements phase that collects hundreds of imperative statements (“The system shall…”)  from business people who will use the system.
4. Late in the requirements phase, the team finds that data integration work in system interfaces will be more complex than expected.  A common example: the project requires changes to a feeder application with no documentation and no in-house support expertise.
5. Project leadership goes back to the sponsor seeking more money.

In these situations the business case was incorrect because it did not account for all of the costs of data integration.  I’ve seen projects weather steps four and five well, but often discovery of previously unseen data complexity starts a disruptive chain of events.  (Sadly for the project manager, such situations are often seen as a failure of project management and corrected accordingly, but that’s a topic for another post.)

In my view the root cause of unforeseen data complexity on projects is the lack of a data constituency in current IT. It is only recently that success of companies like Google and Amazon have motivated emergence of data as a key business resource in the collective consciousness. Famous success stories notwithstanding (see this link), there are relatively few senior IT managers with data quality backgrounds.  Conversely, many rose through the ranks of the infrastructure, application development, or business (process) analysis groups.

It will be a while before, for example, a Mobil CIO’s predecessor jobs include definition of a metadata repository or elimination of multipurpose data, but in the meantime here’s what we can do:  add a business case to the application lifecycle as the last step in requirements.  Stop the project when the real costs are known, recalculate the cost/benefit, and ask the sponsors if the project should continue.  Give Sancho (in this case the project team) a chance to speak to the reality of the situation, and hand to Don Quixote (project sponsors) the eyeglasses of in-depth visibility into real costs. If the decision is to move ahead with the project, then all share the same vision and the sponsors have endorsed the actual project, not the fuzzy image from earlier on that might have been a windmill.

In a database null means that there is literally no value, or the value is indeterminate.  Null is not the same as zero or blank.  When a database operation involves nulls the result can be difficult to predict for someone not practiced in SQL.  In many cases the answer is null.  For example, 1+0=0 but 1+null=null.  In plain English, what you’re asking the DBMS to do in the latter case is to add 1 to [I don't know what], and of course 1+[I don't know what] equals [I don't know what].

So, if you use null to represent a business value then you might not get the results you’re looking for when you try to get business answers out of your database.  For example, say “C” represents credit customers and null represents cash customers, and you have 2 cash and 1 credit customers.   In SQL Server if you use a Count function to tally all of your cash customers the answer isn’t 2, it is null.

That’s one example of why it’s not a good idea to try to represent a business fact with a null value.  It doesn’t make business sense and in this case the DBMS, correctly, won’t make sense of it for you.

To be clear, whether or not a given database column permits null values is an entirely different question, best left to database designers.  For example, a database table might record which patient occupies which hospital bed.  It may be reasonable and correct to assign a null patient ID if the bed is currently available.  However, there are alternative methods of representing this situation, and the database designer should be free to choose the right alternative taking into account the specifics of the application under development.

In my professional life I have on occasion suggested data integration efforts.  Sometimes my suggestions have been accepted and sometimes not.  As an IT professional I understand that different managers have different priorities, and in a given business situation sometimes other things are more important for example than having a single, consistent source for all customer records, or making sure production data matches financial data.

But as a customer?  That’s different.

A couple of years ago I bought a laptop from a company renowned for quality and customer service.  For the first weeks the computer was all it was cracked up to be, but then it cracked up.  The screen developed a mysterious flicker.  After a few diagnostic conversations they replaced the main logic board.  The problem recurred a few months later, and this time the company traded the lemon for a new computer.

All was well, but here we encounter the first data integration problem: they said I needed to call a different number to have my three-year service agreement transferred, which I did.

Months later I called service for a minor problem, and they had no record of the service agreement for the computer.  My warranty was still connected to the lemon.  After about an hour on the phone this company’s outstanding support staff came up with a more than satisfactory solution.

Even so, this company’s service records weren’t integrated with its warranty records.  In this case data integration happened because of my insistence and the service staff’s creativity.  The cost?  Only considering my last encounter, three service professionals were tied up for about an hour, and I’ll think twice before I buy again from this company.

It seems the choice is either pay now to integrate so all applications work from consistent data or pay later by having staff, customers, and suppliers do it on a case-by-case basis.

Excerpt from "Illusions, Allusions – Let’s Get Real about Database Design", October 4, 2002

Much of today’s IT application development – custom or off-the-shelf – involves integrating data from legacy systems, third- party software products and external data sources such as demographics or mail lists.  More often than not, data integration is unexpectedly complex, either due to data quality issues or the nature of the data integration itself.

Here are some typical examples:

• One ERP package uses the same table for both Sales Quotes and Sales Orders. Columns that mean one thing for Quotes mean quite something else Orders. One team extracting data from this ERP package continually mixed up, for example, Date Received on the Quote with Date Prepared for the Order. The designer who blindly copies data from input systems can propagate these issues. In this case, the correct solution is to extract the two documents into separate tables in the destination system, making each column describe either a quote or an order, not both.
• Marketing databases often store data purchased from several third parties on the same set of customers. These sources usually include overlapping columns with different values. For the same customer, different sources might store different values for the person’s address, credit scores or even name. It is sometimes important to preserve all of the columns from all of the sources and to maintain the information on where the data came from as well as what its value was. This can result in a messy database design, where columns again carry dual meaning: their value and their source.
• Codes from legacy databases tend to evolve into complex forms, embedding more and more information into a single field. This is perhaps a natural reaction to the slow evolution of the system relative to changes in business, as users shoehorn information into the system that it was not designed to store. For instance, in a legacy system a one- character code might classify customers by “customer category,” with values 1 for small business, 2 for mid-size, and 3 for Fortune 5000. Users might add codes 4, 5 and 6 for corresponding values for aerospace customers, then 7 for federal government, and so on. The database designer must know the data well to extract each embedded concept into a different destination column.

When data integration is part of a project, expect complexity and leave room in interface development estimates for devils in the details of source system analysis and integration design.