Bob Lambert

I’ve often thought that conceptual data modeling was an underused tool in the arsenal available to requirements analysts, and in a recent conversation I found that many were surprised that it would be used in the requirements phase at all.  Checking the Business Analysis Body of Knowledge (BABOK) I found data modeling listed among the tools available to requirements analysts to “to describe the concepts relevant to a domain, the relationships between those concepts, and information associated with them.”  There’s also Steve Hoberman’s excellent book on the topic, Data Modeling for the Business, an introduction to data modeling aimed at a business audience.

Data modeling has long been one of my requirements analysis tools of choice for custom operational applications.  To me, using data modeling techniques in requirements analysis reduces errors by improving requirements completeness, consistency, and communication, and provides unique continuity between analysis and design.   As David Elliott told me in conversation, “development of a data model uncovers many opportunities for clarification of existing requirements, or uncovering of additional detail.  At the very least, it confirms to one’s business customer that the BSA understands and can graphically demonstrate many business rules and relationships.”

I’ll hasten to add a these caveats.  (1) Perhaps strangely, conceptual data modeling is not useful in the same way in requirements for informational systems like data warehouses and marts (I’ll save that discussion for another post).  (2) Requirements definition for commercial off-the-shelf (COTS) applications follows a different methodology in which data modeling might be less applicable.  (3) This post is not about database design, but rather about use of conceptual data modeling as a tool for organizing and validating requirements.

Conceptual vs. logical data modeling

It is easy to see why in practice there are varying definitions of the different types of data models.  The Wikipedia entry on data modeling reflects the standard terminology based on the ANSI four-level database architecture, but features a confusing diagram that to me blurs the distinction between conceptual and logical models. The entries on Logical Data Model and Conceptual Data Model make them sound like the same thing: implementation-independent representations of business data.  Then, the entry on Database Design contradicts them by stating that the logical model “contains all the needed logical and physical design choices and physical storage parameters needed to … create a database.”

For this post I’ll follow the definitions offered in Simison and Witt’s Data Modeling Essentials:

• Conceptual data modeling identifies a set of data structures that will meet requirements, focusing on business and not on technical or DBMS-specific concerns
• Subsequent logical data modeling maps the conceptual model to structures supported by the particular DBMS, finalizing the design in DBMS-appropriate constructs but not yet optimizing for performance, which comes next in physical modeling

Use of data modeling in requirements definition

Conceptual data modeling is hardly an outlier technique in requirements definition:

• Perhaps in reaction to problems experienced by adopters of Structured techniques in the 80s, data modeling was the cornerstone analytical technique in Clive Finkelstein’s and James Martin’s widely-adopted Information Engineering methodology.
• The BABOK includes class modeling, data modeling’s object-oriented cousin, in its chapter on data modeling.  Class modeling is a core technique of object-oriented analysis.
• Scott Ambler’s Agile Modeling site offers conceptual (or “slim”) data modeling as an option in the initial envisioning stage.
• Informally searching requirements definition templates available on the web, I found that about a third recommend including conceptual data models.

Benefits of data modeling in requirements analysis

The BABOK separates requirements gathering from requirements analysis, defining requirements analysis as an essential step to organize, prioritize, and validate elicited requirements. Elicited requirements are the business objectives of the system. The analysis step organizes those objectives in a way that both makes sense to the business and guides subsequent application design.  Conceptual data modeling in this stage helps ensure requirements completeness, consistency, and communications:

Completeness: In my experience most requirements analysis is process-based, and the most common tool the “swim lane” activity diagram.  While such techniques are essential for understanding complex processes, they can miss requirements that aren’t directly involved in the process itself.  For example, a complex process might reference federal, state, and local tax rates by zip code.  Analysts who are heads-down in defining the process might neglect the need for at least annual refresh of the tax rate tables.  Data modelers thinking in terms of business objects and events and their life cycles would be less likely to miss that one.  This kind of review is formalized in the “CRUD Matrix” a table identifying which business activities create, read, update, or delete which business entities.

Consistency: Another challenge with process-oriented techniques is, for large systems, the risk of inconsistency in definition of business objects and events.  For example, I worked on requirements definition of a specialized order processing system.  Separate sub-teams defined field and headquarters processes, and as a result there were incompatible definitions of critical concepts like “customer” and “order”.  Time pressures made it difficult for the two sub-teams to work together to make their work consistent.  A separate data modeling sub-team can provide a reference point for object and event definitions and promote consistency between separate process analysis teams (on COTS installations the product database itself serves as the reference data model for separate process definition teams).

Communications: Data management professional Peter Carr recounted to me his experience as a consultant on a large project: “the Conceptual Diagram helped us think about all the current state situations, and broadly about the relationships between entities in the organization.  It helped us to ask questions of the business when they were looking to enhance or build new systems to solve business requirements.  Paraphrasing executive colleague Rich Hartt, “the enterprise data model is like a piece of art, it provides a picture into the business that offers new insight through its drawing and interpretation’.  He went on to tell the key business leaders that the enterprise data model will continually be changing, but that it helped them gain understanding of their business in a different way than written business rules”.

For relational database applications, data modeling applies the same conceptual tool throughout the development cycle.  A conceptual data model used to define the problem domain uses the same structure and symbols as a physical database design, although of course it uses fewer.  On the other hand, activity diagrams and data flow diagrams are fundamentally different in nature from the software that they describe.  In effect, process designers need to translate analysis artifacts into a different language.  Logical and physical data modelers use exactly the same language as the business analysts who complete the conceptual data model.

My colleague Grayson Gorman cites “Poorly Defined / Missed Requirements” as a key contributor to IT project failure.  In my view making data modeling a more prevalent part of requirements definition could help by improving requirements completeness, consistency, and communication with business participants, and promote a seamless transition from requirements to design.

In a new post at Insurance Networking News Ara Trembly provides a balanced perspective on IT/business misalignment (Business/IT Misalignment: Whose Responsibility?).  He describes the problem as cultural, more amenable to relational than management solutions.    His conclusion sums it up: “Take a geek/suit to lunch today!”

To me (speaking as an IT professional) IT should take the initiative to solve the problem.  Quoting Trembly, “business executives … make decisions, but they are for the most part mystified at the magical incantations and actions that produce IT results” and “IT people, on the other hand, are jealous of the sheer power wielded over them by business people who just don’t get IT.”  In other words, business people contend with an emotional and a substantive problem, “fear and lack of knowledge,” while IT people have only the emotional problem of jealousy.

If we take the emotions out of the picture (its just a job, right?) then that leaves IT folks with knowledge that business people need in order to maximize the value of IT and efficiency of business processes.  Ever since mainframes roamed the prehistoric rain forests of the ’60s application developers have often been the most knowledgeable about how business processes really work, understanding both the intricacies of the application logic and how business people use the system to get things done.  These individuals can add value to the business discussion by bringing their knowledge to the table in a way that business people can understand.

In many organizations IT manages the forum in which these conversations can occur: the requirements process.  In my experience a good requirements process is long enough for the business and IT teams to get to know each other, offers generous opportunity for both structured and unstructured conversations about business needs, and brings together knowledgeable business and IT participants.  IT is typically able to bring the insights of seasoned application developers to the fore in a well planned requirements effort.

Yes, everyone has responsibility to “cultivate personal relationships based on mutual need and respect,” but IT can and should bring substance to the relationship in requirements definition.

It seems I’m frequently in conversations about using null to represent a business value.  To paraphrase, say there are credit and cash customers, and there’s a suggestion to set “Customer_Type” to “C” for credit and null for cash.  To data and database professionals this is obviously a bad idea, but it’s not obvious from a business point of view.

In a database null means that there is literally no value, or the value is indeterminate.  Null is not the same as zero or blank.  When a database operation involves nulls the result can be difficult to predict for someone not practiced in SQL.  In many cases the answer is null.  For example, 1+0=0 but 1+null=null.  In plain English, what you’re asking the DBMS to do in the latter case is to add 1 to [I don't know what], and of course 1+[I don't know what] equals [I don't know what].

So, if you use null to represent a business value then you might not get the results you’re looking for when you try to get business answers out of your database.  For example, say “C” represents credit customers and null represents cash customers, and you have 2 cash and 1 credit customers.   In SQL Server if you use a Count function to tally all of your cash customers the answer isn’t 2, it is null.

That’s one example of why it’s not a good idea to try to represent a business fact with a null value.  It doesn’t make business sense and in this case the DBMS, correctly, won’t make sense of it for you.

To be clear, whether or not a given database column permits null values is an entirely different question, best left to database designers.  For example, a database table might record which patient occupies which hospital bed.  It may be reasonable and correct to assign a null patient ID if the bed is currently available.  However, there are alternative methods of representing this situation, and the database designer should be free to choose the right alternative taking into account the specifics of the application under development.

Recently I was asked for a high level requirements plan for a large IT conversion.  I googled around a little for something standard.  I found some good references (see links at the bottom of this post), but not exactly what I was looking for: a simple, method-agnostic layout of the high level steps and checkpoints in requirements for a big project, emphasizing interactions with business people.   I then rifled my files to find the example below.

This summary plan frames up interactions with business subject matter experts and their review of results.  The table lays out the steps, “granularity” (meaning how often each step is carried out), what comes out of each step, who does the work, and offers a few notes.

Before the table, here are two important definitions:

• Sponsor: Very early on in your project you should identify the one person who you need to make happy in order to succeed.   The bigger the project, the higher up the sponsor.  Keep in touch to make sure they know what’s going on, keep them happy, and if something happens that will make them not happy don’t keep it secret.  Maintain a vibrant risk/issue process so that you can give them early warning of bad possibilities and they can help early.
• Stakeholder: A stakeholder is anyone who will benefit from or be harmed by your project.  Requirements come from stakeholders.  Be sure to build support even with the latter group if at all possible, and at least make the adjustments that will keep them from working to prevent you from succeeding.

Careful, this is just an empty vessel.  Within this high level framework a team can apply whatever requirements techniques they want.  (In fact, I highly recommend structured analysis techniques like use cases or process models, but that’s for the requirements team and this framework is for the PM).  Of course a smaller effort suitable for agile techniques wouldn’t need something like this.  This is for big transitions like conversion to a new COTS package, for example, where it is easy to get lost in the detail.

Hopefully if you’re a PM on a big project you’ll find this framework as useful as I did.

 Here are some of the other references I found along the way, caveat emptor:

• http://www.outsource2india.com/software/RequirementAnalysis.asp
• http://en.wikipedia.org/wiki/Requirements_analysis
• http://www.jiludwig.com/HCI_Journal_article.html

Free form diagramming doesn’t only mean “no rules”, it also means “just right”.

This post, last in a three part series on free form diagramming, gives some simple guidelines for getting the technique right.  Part one talked about the tension between rigor and expression in diagramming for analysis and design, and how more precise diagrams can hinder rather than help communications with business people.  Part two reviewed free form diagramming in practice.

While it is impossible to specify format and structure of a free form diagram in advance, here are some useful guidelines to follow when building your free form diagram:

•    Rule number one: draw it as you see it. Typically, an analyst uses a free form diagram because he/she already pictures a business process.  Trust your mental picture and get it down on the page.  Then, go through the following checklist to make sure it says what you want it to say.

•    Model real world processes, things, and events. Free form diagrams have one great advantage over Use Case Diagrams, Data Flow Diagrams, and the rest: they are concrete rather than abstract.  For example, in a free form diagram you can symbolize a shopper with a clipart picture of someone choosing a soup can from a grocery store shelf.  The free form diagram should clearly represent things from the real world: organizations, locations, business processes, interfaces, etc.

•    Use symbols consistently. Look at each repeated rectangle, line, circle, icon, etc, and verify that everything with the same shape represents the same type of thing or event.

•    Speak the language of the audience. A free form diagram should depict things business people care about in recognizable terms.  For example, accountants might readily understand boxes labeled GL, AP and AR for general ledger, accounts payable, and accounts receivable.  A shipping clerk might quickly interpret a process illustration showing labeled icons shaped like trucks and warehouses.

•    Arrangement on the page conveys meaning. Frequently, free form diagrams group objects that belong together on the page.  In other cases, the diagram shows a definite process flow by the arrangement of objects.  For example, Business Intelligence Architecture diagrams typically show information flow from source systems on the left to business reporting on the right.  Could this kind of flow or grouping improve your diagram?

•    Limit the number and complexity of objects on the diagram. Most often, a meaningful diagram shows relatively few objects, organizes them in a sensible way, and does not cross lines.  If you need many objects to tell the story, reduce complexity by arranging them logically.

•    Work at one level of detail, or clearly indicate differences between levels of detail. If your diagram includes a box labeled “AP System” then it would not likely make sense for it also to contain another labeled “Journal Voucher Key”.  Diagrams that communicate well are all at the same level of detail or clearly indicate differences in level of detail.

The free form diagram can be an essential part of a successful IT application project by enabling all to understand the target system in the same way and helping business people understand critical functionality.

This is part two of a three part series on free form diagramming for IT projects.  This entry reviews free form diagramming in practice. Part one talked about the tension between rigor and expression in diagramming for analysis and design, and how more precise diagrams can hinder rather than help communications with business people.  Part three will provide a few simple guidelines for getting it right.

Some current development tools and methods do include the free form diagram in their toolbox.  For example, Scott Ambler’s Agile Modeling site includes a page on free form diagrams (http://www.agilemodeling.com/artifacts/freeForm.htm), and the IBM tool XDE has provided an integrated free form diagram (http://www-128.ibm.com/developerworks/rational/library/915.html).

A Free Form Diagram Showing the Broad Outlines of a System

Free form diagrams can be especially useful in illustrating the overall scope of an IT development effort.  For example, XDE product literature cites the free form diagram’s “capabilities to communicate broad ideas about the direction of [a] solution”.  IT software product marketers frequently bear out this advantage over structured modeling with attractive diagrams describing their products and services, as in the diagram above.  Such diagrams enable companies to “level set” the terms of a conversation, providing a reference point for providing more detailed information about the product.

A similar diagram can provide a reference for IT project participants.  One, produced jointly early on in custom development of a financial system by the primary business contact and project architect, helped build consensus with designers, database administrators, programmers, and business people.  Some on the project considered this shared vision of the planned system one key to the project’s success.

Both of these examples illustrate the most common use of the free form diagram: to depict “the fundamental organization of a system [or business function], embodied in its components, their relationships to each other and the environment, and the principles governing its design and evolution” (http://en.wikipedia.org/wiki/System_architecture).

Because the free form diagram can seem more concrete than other, more abstract, models, a well-constructed free form diagram is a great way to communicate complexity to an audience of both technical and non-technical participants.

The free form diagram is also useful in other situations.  Sometimes a requirements team works with business users who have a diagram describing their current process.  In that case the team might use the same format to describe the new process.  In other cases it might be useful to improve readability by drawing a process flow on a map of the shop floor, or using icons representing real objects and events rather than class symbols on a class model.  The point is to create a medium that precisely matches the audience with the message.

Part 3 of the Free Form Diagrams series provides a few simple guidelines for success with this technique.

One effective way of communicating complexity, especially in the overall architecture of a system, is the free form diagram.  A free form diagram can directly address unique characteristics of a system in a way that business people can understand.

Out on a walk some years ago I met an acquaintance who happened to be a professor of Computer Science.  We talked shop (I worked in the IT department of the local electric utility) and he asked me what methods we used for mathematically proving program correctness.  I confess I laughed.  My team was struggling to figure out just what the business really wanted – forget mathematical proofs!

Our conversation highlights the tension between rigor and expression in diagramming to support IT projects.  Developers need precise diagrams rigorous enough to accurately reflect complex processes.  However, that precision and detail can make the diagrams at best boring and at worst intimidating to business people.  Often they don’t have the time or patience required to sit through the explanations needed to understand such diagrams. Requirements meetings frequently end early with business participants walking off grumbling about IT non-alignment.

UML is today’s de facto analysis and design diagramming standard. To the OO professional UML provides a rich, expressive, and consistent set of conceptual tools that continue to evolve.  Work is underway to improve the accuracy and precision of UML with respect to the target code. (http://www.omg.org/docs/ad/00-01-07.pdf).
The problem is that more precision will make the UML diagrams more complex and less understandable to non-coders, making the diagrams even less useful in communication with business people.

For project success business people need to be able to communicate their needs freely and completely.  Requirements analysts need to capture, record, and replay business needs for confirmation, and let the business expert return to other work as quickly as possible.  These communications often work best with pictures rather than words, and of course when everyone understands the picture the same way.  Asking a busy floor supervisor to review a formal class model is literally having him or her review specifications in a foreign language.

One solution is for developers to communicate with business people using free form diagrams, expressive yet rigorous diagrams with drawing conventions tailored to the audience.

More to come in parts two and three of this series.  Part 2 talks about using free form diagrams in practices, and Part 3 provides some simple guidelines for successfully using free form diagrams.