Bob Lambert

I’ve often thought that conceptual data modeling was an underused tool in the arsenal available to requirements analysts, and in a recent conversation I found that many were surprised that it would be used in the requirements phase at all.  Checking the Business Analysis Body of Knowledge (BABOK) I found data modeling listed among the tools available to requirements analysts to “to describe the concepts relevant to a domain, the relationships between those concepts, and information associated with them.”  There’s also Steve Hoberman’s excellent book on the topic, Data Modeling for the Business, an introduction to data modeling aimed at a business audience.

Data modeling has long been one of my requirements analysis tools of choice for custom operational applications.  To me, using data modeling techniques in requirements analysis reduces errors by improving requirements completeness, consistency, and communication, and provides unique continuity between analysis and design.   As David Elliott told me in conversation, “development of a data model uncovers many opportunities for clarification of existing requirements, or uncovering of additional detail.  At the very least, it confirms to one’s business customer that the BSA understands and can graphically demonstrate many business rules and relationships.”

I’ll hasten to add a these caveats.  (1) Perhaps strangely, conceptual data modeling is not useful in the same way in requirements for informational systems like data warehouses and marts (I’ll save that discussion for another post).  (2) Requirements definition for commercial off-the-shelf (COTS) applications follows a different methodology in which data modeling might be less applicable.  (3) This post is not about database design, but rather about use of conceptual data modeling as a tool for organizing and validating requirements.

Conceptual vs. logical data modeling

It is easy to see why in practice there are varying definitions of the different types of data models.  The Wikipedia entry on data modeling reflects the standard terminology based on the ANSI four-level database architecture, but features a confusing diagram that to me blurs the distinction between conceptual and logical models. The entries on Logical Data Model and Conceptual Data Model make them sound like the same thing: implementation-independent representations of business data.  Then, the entry on Database Design contradicts them by stating that the logical model “contains all the needed logical and physical design choices and physical storage parameters needed to … create a database.”

For this post I’ll follow the definitions offered in Simison and Witt’s Data Modeling Essentials:

• Conceptual data modeling identifies a set of data structures that will meet requirements, focusing on business and not on technical or DBMS-specific concerns
• Subsequent logical data modeling maps the conceptual model to structures supported by the particular DBMS, finalizing the design in DBMS-appropriate constructs but not yet optimizing for performance, which comes next in physical modeling

Use of data modeling in requirements definition

Conceptual data modeling is hardly an outlier technique in requirements definition:

• Perhaps in reaction to problems experienced by adopters of Structured techniques in the 80s, data modeling was the cornerstone analytical technique in Clive Finkelstein’s and James Martin’s widely-adopted Information Engineering methodology.
• The BABOK includes class modeling, data modeling’s object-oriented cousin, in its chapter on data modeling.  Class modeling is a core technique of object-oriented analysis.
• Scott Ambler’s Agile Modeling site offers conceptual (or “slim”) data modeling as an option in the initial envisioning stage.
• Informally searching requirements definition templates available on the web, I found that about a third recommend including conceptual data models.

Benefits of data modeling in requirements analysis

The BABOK separates requirements gathering from requirements analysis, defining requirements analysis as an essential step to organize, prioritize, and validate elicited requirements. Elicited requirements are the business objectives of the system. The analysis step organizes those objectives in a way that both makes sense to the business and guides subsequent application design.  Conceptual data modeling in this stage helps ensure requirements completeness, consistency, and communications:

Completeness: In my experience most requirements analysis is process-based, and the most common tool the “swim lane” activity diagram.  While such techniques are essential for understanding complex processes, they can miss requirements that aren’t directly involved in the process itself.  For example, a complex process might reference federal, state, and local tax rates by zip code.  Analysts who are heads-down in defining the process might neglect the need for at least annual refresh of the tax rate tables.  Data modelers thinking in terms of business objects and events and their life cycles would be less likely to miss that one.  This kind of review is formalized in the “CRUD Matrix” a table identifying which business activities create, read, update, or delete which business entities.

Consistency: Another challenge with process-oriented techniques is, for large systems, the risk of inconsistency in definition of business objects and events.  For example, I worked on requirements definition of a specialized order processing system.  Separate sub-teams defined field and headquarters processes, and as a result there were incompatible definitions of critical concepts like “customer” and “order”.  Time pressures made it difficult for the two sub-teams to work together to make their work consistent.  A separate data modeling sub-team can provide a reference point for object and event definitions and promote consistency between separate process analysis teams (on COTS installations the product database itself serves as the reference data model for separate process definition teams).

Communications: Data management professional Peter Carr recounted to me his experience as a consultant on a large project: “the Conceptual Diagram helped us think about all the current state situations, and broadly about the relationships between entities in the organization.  It helped us to ask questions of the business when they were looking to enhance or build new systems to solve business requirements.  Paraphrasing executive colleague Rich Hartt, “the enterprise data model is like a piece of art, it provides a picture into the business that offers new insight through its drawing and interpretation’.  He went on to tell the key business leaders that the enterprise data model will continually be changing, but that it helped them gain understanding of their business in a different way than written business rules”.

For relational database applications, data modeling applies the same conceptual tool throughout the development cycle.  A conceptual data model used to define the problem domain uses the same structure and symbols as a physical database design, although of course it uses fewer.  On the other hand, activity diagrams and data flow diagrams are fundamentally different in nature from the software that they describe.  In effect, process designers need to translate analysis artifacts into a different language.  Logical and physical data modelers use exactly the same language as the business analysts who complete the conceptual data model.

My colleague Grayson Gorman cites “Poorly Defined / Missed Requirements” as a key contributor to IT project failure.  In my view making data modeling a more prevalent part of requirements definition could help by improving requirements completeness, consistency, and communication with business participants, and promote a seamless transition from requirements to design.

I always thought the analogy would be cheesy, but Adrian Cho’s “Jazz Process” is a carefully researched and well presented “framework for improving collaboration, innovation and agility inspired by the way in which jazz musicians deliver strong, innovative performances.”  Mr. Cho, with deep roots in both jazz and application development, presents a method for app dev teams to work together the way jazz musicians do to “deliver on-time, high-quality performances that will attract and retain customers and do it all in real-time under continuous scrutiny.”

To put cards on the table, while Mr. Cho is a dedicated dual-career professional who plays jazz at the highest level, I’m a dedicated IT professional who spends some evenings as a serious amateur at local watering holes and private parties.  To me, Mr Cho really nails it.  He groups 14 principles into four categories on how teams can effectively work, collaborate, execute, and innovate together, bringing honed skills, “big ears”, trust, and commitment to deliver successful outcomes.

Does the Jazz Process work?  Only time will tell if it catches on in the business world, but from my experience the Jazz Process does describe how musicians work together.  The two concepts I particularly like are the importance of developing a technical base and the need for humility.  Mr. Cho quotes Vijay Govindarajan, who wrote: “The more humble you are, the more you know what you don’t know; you seek to learn” - something John Coltrane might have said.

From the point of view of a jazz-playing IT guy, however, two important topics seem to be de-emphasized in the Jazz Process:

Those who don’t know history don’t know how to repeat it

Every jazz player I know is also a jazz historian, and knows that Coleman Hawkins’ version of “Body and Soul” is a landmark of harmonic improvisation, and if they heard Pentangle’s song “I’ve Got a Feeling” they would know it as Mile’s Davis’s “All Blues.” An accomplished jazz improviser might weave into a solo phrasing or melodic fragments borrowed from jazz greats of the past, which jazz aficionados would recognize and applaud.

How many application developers know that during the early ’70s Larry Constantine first conceived the concepts basic to Object Oriented design, and how many know the (perhaps apocryphal) story that he borrowed them from the social sciences?  How many know about Edsger Dijkstra’s landmark article called “Go To Statement Considered Harmful”?

If I’ve stumped you, don’t worry, you could probably stump me just as easily.  The point is that, unlike jazz musicians, we in app development have a culture that tends not to respect and build upon the past.  As we gain new tools and methods we seem to discard the old ones (does anyone remember Warnier-Orr diagrams?), and we often forget the hard-won lessons of the past (how many process analysis and redesign projects skip logical modeling, causing the new process to retain unnecessary steps related to old systems?)

Play it for the people

In a very real and direct way, the audience is a huge part of any musical performance.  The jazz player is not only aware of the style needed for the gig (background, dance, concert, etc) but also of the “feeling in the room”.  There’s a gestalt in the performance space that is difficult to explain but very apparent to a musician.  Jazz musicians not only constantly listen and  interact with each other, but also constantly feel and feed the vibe in the room.

Similarly, excellent development teams are constantly aware of and constantly cultivate good feeling in their organizational community.  I was fortunate enough to be involved in a development effort where key business people we were building the application for worked overtime so they could be involved as testers, in part because the system was critical to their process but also because it was a really dedicated and energized group.

–

Mr. Cho’s book is not yet available at the time of this writing, but there’s a wealth of information at www.jazzprocess.com.  I’ll look forward to learning more and applying this method at work.