Bob Lambert

I’ve often thought that conceptual data modeling was an underused tool in the arsenal available to requirements analysts, and in a recent conversation I found that many were surprised that it would be used in the requirements phase at all.  Checking the Business Analysis Body of Knowledge (BABOK) I found data modeling listed among the tools available to requirements analysts to “to describe the concepts relevant to a domain, the relationships between those concepts, and information associated with them.”  There’s also Steve Hoberman’s excellent book on the topic, Data Modeling for the Business, an introduction to data modeling aimed at a business audience.

Data modeling has long been one of my requirements analysis tools of choice for custom operational applications.  To me, using data modeling techniques in requirements analysis reduces errors by improving requirements completeness, consistency, and communication, and provides unique continuity between analysis and design.   As David Elliott told me in conversation, “development of a data model uncovers many opportunities for clarification of existing requirements, or uncovering of additional detail.  At the very least, it confirms to one’s business customer that the BSA understands and can graphically demonstrate many business rules and relationships.”

I’ll hasten to add a these caveats.  (1) Perhaps strangely, conceptual data modeling is not useful in the same way in requirements for informational systems like data warehouses and marts (I’ll save that discussion for another post).  (2) Requirements definition for commercial off-the-shelf (COTS) applications follows a different methodology in which data modeling might be less applicable.  (3) This post is not about database design, but rather about use of conceptual data modeling as a tool for organizing and validating requirements.

Conceptual vs. logical data modeling

It is easy to see why in practice there are varying definitions of the different types of data models.  The Wikipedia entry on data modeling reflects the standard terminology based on the ANSI four-level database architecture, but features a confusing diagram that to me blurs the distinction between conceptual and logical models. The entries on Logical Data Model and Conceptual Data Model make them sound like the same thing: implementation-independent representations of business data.  Then, the entry on Database Design contradicts them by stating that the logical model “contains all the needed logical and physical design choices and physical storage parameters needed to … create a database.”

For this post I’ll follow the definitions offered in Simison and Witt’s Data Modeling Essentials:

• Conceptual data modeling identifies a set of data structures that will meet requirements, focusing on business and not on technical or DBMS-specific concerns
• Subsequent logical data modeling maps the conceptual model to structures supported by the particular DBMS, finalizing the design in DBMS-appropriate constructs but not yet optimizing for performance, which comes next in physical modeling

Use of data modeling in requirements definition

Conceptual data modeling is hardly an outlier technique in requirements definition:

• Perhaps in reaction to problems experienced by adopters of Structured techniques in the 80s, data modeling was the cornerstone analytical technique in Clive Finkelstein’s and James Martin’s widely-adopted Information Engineering methodology.
• The BABOK includes class modeling, data modeling’s object-oriented cousin, in its chapter on data modeling.  Class modeling is a core technique of object-oriented analysis.
• Scott Ambler’s Agile Modeling site offers conceptual (or “slim”) data modeling as an option in the initial envisioning stage.
• Informally searching requirements definition templates available on the web, I found that about a third recommend including conceptual data models.

Benefits of data modeling in requirements analysis

The BABOK separates requirements gathering from requirements analysis, defining requirements analysis as an essential step to organize, prioritize, and validate elicited requirements. Elicited requirements are the business objectives of the system. The analysis step organizes those objectives in a way that both makes sense to the business and guides subsequent application design.  Conceptual data modeling in this stage helps ensure requirements completeness, consistency, and communications:

Completeness: In my experience most requirements analysis is process-based, and the most common tool the “swim lane” activity diagram.  While such techniques are essential for understanding complex processes, they can miss requirements that aren’t directly involved in the process itself.  For example, a complex process might reference federal, state, and local tax rates by zip code.  Analysts who are heads-down in defining the process might neglect the need for at least annual refresh of the tax rate tables.  Data modelers thinking in terms of business objects and events and their life cycles would be less likely to miss that one.  This kind of review is formalized in the “CRUD Matrix” a table identifying which business activities create, read, update, or delete which business entities.

Consistency: Another challenge with process-oriented techniques is, for large systems, the risk of inconsistency in definition of business objects and events.  For example, I worked on requirements definition of a specialized order processing system.  Separate sub-teams defined field and headquarters processes, and as a result there were incompatible definitions of critical concepts like “customer” and “order”.  Time pressures made it difficult for the two sub-teams to work together to make their work consistent.  A separate data modeling sub-team can provide a reference point for object and event definitions and promote consistency between separate process analysis teams (on COTS installations the product database itself serves as the reference data model for separate process definition teams).

Communications: Data management professional Peter Carr recounted to me his experience as a consultant on a large project: “the Conceptual Diagram helped us think about all the current state situations, and broadly about the relationships between entities in the organization.  It helped us to ask questions of the business when they were looking to enhance or build new systems to solve business requirements.  Paraphrasing executive colleague Rich Hartt, “the enterprise data model is like a piece of art, it provides a picture into the business that offers new insight through its drawing and interpretation’.  He went on to tell the key business leaders that the enterprise data model will continually be changing, but that it helped them gain understanding of their business in a different way than written business rules”.

For relational database applications, data modeling applies the same conceptual tool throughout the development cycle.  A conceptual data model used to define the problem domain uses the same structure and symbols as a physical database design, although of course it uses fewer.  On the other hand, activity diagrams and data flow diagrams are fundamentally different in nature from the software that they describe.  In effect, process designers need to translate analysis artifacts into a different language.  Logical and physical data modelers use exactly the same language as the business analysts who complete the conceptual data model.

My colleague Grayson Gorman cites “Poorly Defined / Missed Requirements” as a key contributor to IT project failure.  In my view making data modeling a more prevalent part of requirements definition could help by improving requirements completeness, consistency, and communication with business participants, and promote a seamless transition from requirements to design.

Thanks to all who attended my presentation at Richmond Code Camp 2010.1 on May 22.  Here’s a link to the PowerPoint presentation:  The Business End of Data Modeling (2.5m)

The presentation was about what to expect before you start database design and what to do if you don’t get it. Software development  professionals apply technology solutions to meet business needs. The problem is that sometimes we’re expected to meet business needs that aren’t well defined, or not defined at all. The presentation reviewed the requirements side of data modeling and how it prepares a developer to design and build the right solution. Then, it covered typical scenarios where critical business definition elements are missing and what the app dev professional can do make up for the missing requirements pieces and produce a successful result.

I always thought the analogy would be cheesy, but Adrian Cho’s “Jazz Process” is a carefully researched and well presented “framework for improving collaboration, innovation and agility inspired by the way in which jazz musicians deliver strong, innovative performances.”  Mr. Cho, with deep roots in both jazz and application development, presents a method for app dev teams to work together the way jazz musicians do to “deliver on-time, high-quality performances that will attract and retain customers and do it all in real-time under continuous scrutiny.”

To put cards on the table, while Mr. Cho is a dedicated dual-career professional who plays jazz at the highest level, I’m a dedicated IT professional who spends some evenings as a serious amateur at local watering holes and private parties.  To me, Mr Cho really nails it.  He groups 14 principles into four categories on how teams can effectively work, collaborate, execute, and innovate together, bringing honed skills, “big ears”, trust, and commitment to deliver successful outcomes.

Does the Jazz Process work?  Only time will tell if it catches on in the business world, but from my experience the Jazz Process does describe how musicians work together.  The two concepts I particularly like are the importance of developing a technical base and the need for humility.  Mr. Cho quotes Vijay Govindarajan, who wrote: “The more humble you are, the more you know what you don’t know; you seek to learn” - something John Coltrane might have said.

From the point of view of a jazz-playing IT guy, however, two important topics seem to be de-emphasized in the Jazz Process:

Those who don’t know history don’t know how to repeat it

Every jazz player I know is also a jazz historian, and knows that Coleman Hawkins’ version of “Body and Soul” is a landmark of harmonic improvisation, and if they heard Pentangle’s song “I’ve Got a Feeling” they would know it as Mile’s Davis’s “All Blues.” An accomplished jazz improviser might weave into a solo phrasing or melodic fragments borrowed from jazz greats of the past, which jazz aficionados would recognize and applaud.

How many application developers know that during the early ’70s Larry Constantine first conceived the concepts basic to Object Oriented design, and how many know the (perhaps apocryphal) story that he borrowed them from the social sciences?  How many know about Edsger Dijkstra’s landmark article called “Go To Statement Considered Harmful”?

If I’ve stumped you, don’t worry, you could probably stump me just as easily.  The point is that, unlike jazz musicians, we in app development have a culture that tends not to respect and build upon the past.  As we gain new tools and methods we seem to discard the old ones (does anyone remember Warnier-Orr diagrams?), and we often forget the hard-won lessons of the past (how many process analysis and redesign projects skip logical modeling, causing the new process to retain unnecessary steps related to old systems?)

Play it for the people

In a very real and direct way, the audience is a huge part of any musical performance.  The jazz player is not only aware of the style needed for the gig (background, dance, concert, etc) but also of the “feeling in the room”.  There’s a gestalt in the performance space that is difficult to explain but very apparent to a musician.  Jazz musicians not only constantly listen and  interact with each other, but also constantly feel and feed the vibe in the room.

Similarly, excellent development teams are constantly aware of and constantly cultivate good feeling in their organizational community.  I was fortunate enough to be involved in a development effort where key business people we were building the application for worked overtime so they could be involved as testers, in part because the system was critical to their process but also because it was a really dedicated and energized group.

–

Mr. Cho’s book is not yet available at the time of this writing, but there’s a wealth of information at www.jazzprocess.com.  I’ll look forward to learning more and applying this method at work.

Thanks to all who attended my presentations at SQL Saturday on April 10.  Here are the materials from my two presentations:

- The Business End of Data Modeling (2.5m powerpoint presentation)

- Normalize Metadata For Data Integration Analysis (5.5m full version, zip including presentation and code samples)

- Normalize Metadata For Data Integration Analysis (small) (2m reduced size version, graphics removed from ppt file)

Here are some quick notes for those looking to run the Metadata prototype:

The prototype metadata database includes SQL Server 2008 data definition language and data manipulation language (DDL and DML) needed to create the database and populate it with tables and columns from Microsoft’s AdventureWorksDW sample database. It also includes a sample requirements spreadsheet and source-to-target map, and SSIS jobs to load the spreadsheets to corresponding metadata tables. These define fictional requirements and mappings to populate the AdventureWorksDW FACTInternetSales table from tables in the AdventureWorks sample database.

AdventureWorks and AdventureWorksDW are available here: http://msftdbprodsamples.codeplex.com/Wikipage (accessed 4/14/2010)

“Our goals can only be reached through a vehicle of a plan, in which we must fervently believe, and upon which we must vigorously act. There is no other route to success.” – Pablo Picasso

It is an old story: about 30% of IT application projects succeed, 45% are “challenged,” and the other quarter fail altogether.   That’s the consistent result over the years of the Standish Group Study of Project Outcomes.  Jorge Dominguez, here, displays a chart of the remarkably similar results since 1994.  Not a pretty picture, right?  Some question the validity of the Standish studies, but Scott Ambler parallels the Standish story in a recent Dr Dobbs column called “Lies, Great Lies, and Software Development Project Plans,” which itemizes the strategies commonly used by IT project managers to “stay out of trouble” when schedule/budget results don’t match initial estimates.  For example, “18% change the original schedule to reflect the actual results”.

The frequent reaction to stats like these is to scapegoat the IT folks by finding fault with their tools, processes, or skills.  If we just had a more efficient methodology, or a slick development suite, or a more highly skilled team, or a better project process, were more agile, or whatever, then application projects would be on time, resulting systems would be faultless, and we could drive down outrageous IT costs.

Mr. Ambler plays out the IT perspective in this column about Agile project estimation.  Here’s how I paraphrase the gist of his logic: you can’t accurately estimate early on due to normal uncertainty of high level requirements; you will get new requirements that will expand scope.  If you offer a range of predicted outcomes, management will hold you to the most optimistic, which will turn out to be a gross underestimate. Your best strategy is to use an accurate estimation method (net or average velocity, described in the article), and be straight with management even if they “don’t like what they are hearing”.

Based on my experience that sequence of events can be distressingly true to life.  But I’ve been on the other kind of IT project as well, the kind that ends on time, stays on budget, and satisfies the business. The scope of Mr. Ambler’s article doesn’t include asking why the requirements are changing so drastically, but if it had, maybe he would have asked questions like these:

• Does the business unit involved have defined objectives and processes?
• Are strategic and tactical business objectives of the project defined, and do they support the business unit’s objectives?
• Are the critical business success factors of the project defined?
• Does the new project change business processes and if so have the new ones been defined yet?
• Are there documented business requirements and a process for evaluating/approving changes?

Based on what I’ve seen requirements shifts that nearly double the cost of an IT project, like the ones Mr. Ambler describes, result from inadequate business definition and business requirements analysis.  Of course it is true that the “initial high-level requirements and architecture envisioning” early in a typical project only enables an estimate “in the +/- 30% range”.  But effective requirements and architecture definition, when based on effective business definition, can enable a much closer estimate.  Further, it is possible to accurately estimate how long it will take to provide the more accurate estimate.

In some cases the business definition prerequisites clearly don’t yet exist at the outset of an application development effort.  One thing I’ve seen work in such cases is to divide the project into two separate projects: one to closely define the effort, and if needed make up for any lacking business definition, and another to build, test, and install the application. In between the two phases management will have the opportunity to review the costs/benefits of the project and evaluate whether to continue or not.  This post describes one project that successfully used that strategy.

Please don’t read this as repudiation of Agile methods and advocation of “big requirements up front”.  Agile methods work well whether or not business prerequisites are defined, but they seem not to work well when (1) the project’s goals shift with evolving business definition and (2) the plan and budget aren’t adjusted accordingly.

Business definition, completed as part of a discrete requirements phase, that leads to a management decision to continue or not, gives the team the opportunity to build on a solid business foundation.  It also gives management a reasonable estimate and a chance to bail if the project isn’t worth it.

I was preparing a presentation about databases and thought of a Dilbert strip I’d seen years ago.  Easily distracted, I searched for it.  I didn’t find the strip itself, but here’s the transcript I found:

Boss: I just got our consultant report and he has identified our biggest problem

Wally: I recommend that we build a tracking database

Dilbert: We can put it on the network

Boss: Would you like to hear what the problem is first?

Wally: I hate to dwell on the negative

Dilbert: We like databases

Boss: You haven’t heard what the problem is yet, how could you recommend building a database to resolve it?

Wally: We always build a database

Dilbert: And we will need coffee mugs for the project team.

Boss: The problem is that we have poor process

Wally: That could be the slogan on our mugs.

Need uber-guru types who are willing to challenge the existing groupthink on design and architecture, especially on TDD and emergent design and pair programming anti-pattern” – job post at Monster.com 2/9/2010

I stumbled upon that quote following links on the role of the architect on an agile project. Maybe one important role of the architect is to help the team avoid groupthink.

Groupthink is a situation where a team’s decision process breaks down and the team reaches decisions that aren’t fully vetted and evaluated.  Both Watergate and the Bay of Pigs fiasco are cited as examples (here).  I’ve seen groupthink operate on IT projects, and to me the agile method’s effectiveness in enabling groups to work together means agile projects are particularly susceptible.

This post reviews groupthink then discusses how the architect on an agile project might help prevent it.

Groupthink

From the Wikipedia article on groupthink, “groupthink is a type of thought exhibited by group members who try to minimize conflict and reach consensus without critically testing, analyzing, and evaluating ideas. Individual creativity, uniqueness, and independent thinking are lost in the pursuit of group cohesiveness. During groupthink, members of the group avoid promoting viewpoints outside the comfort zone of consensus thinking…Highly cohesive groups are much more likely to engage in groupthink, because their cohesiveness often correlates with unspoken understanding and the ability to work together with minimal explanations.”

In my experience risk of groupthink can manifest in several ways on IT projects:

• Not Invented Here: Successful teams that work through conflict can settle into a shared culture that resists new ideas from outside the team.
• The Know It All: Less successfully integrated teams can be dominated by a single strong-willed individual, and can habitually avoid conflict by accepting without question the ideas of that one dominant team member.
• The Abilene Paradox: Team members sometimes collectively decide on a course of action that no one on the team likes, when each member actually disagrees with the decision but mistakenly believes that their own preferences are counter to the group’s.

The Agile Architect

According to the Psychologists for Social Responsibility, the standard remedies for groupthink include this: “At least one articulate and knowledgeable member should be given the role of devil’s advocate (to question assumptions and plans)”. Of course the architect is an integral part of the overall project, but the skilled practitioner participates with the Agile team while maintaining separateness in order to remain a source of ideas from outside the team, and therefore provide a counterweight to groupthink by recognizing it and taking measures to prevent it.  Andrew Johnston’s site agilearchitect.org describes some of the ways the architect is in but not totally of the team (here). Among the architect’s responsibilities, he or she:

• Ensures “the delivered system is consistent with the agreed architecture, and will meet the requirements”
• “Is frequently an evangelist for new or different technologies, processes or solutions”
• “Acts as a bridge between developers, managers and other communities, and spends much of his time translating and mediating between them”
• “Recognizes the wide range of stakeholders, and their needs and concerns.”

While core agile team members are immersed in the scope and design that defines the current sprint, the architect retains a larger perspective that encompasses alternative designs, emerging technologies, business fit, stakeholder concerns, and more. The architect is therefore uniquely positioned to recognize groupthink effects on the team’s technical activities. Here are two examples of how that works on agile projects:

• Estimations and retrospectives: Mark Needham, in this post, cites risk of groupthink in agile estimation sessions and retrospectives.  The architect can address both of these risks. In estimation, the architect brings the diverse perspective that Mr. Needham says is important when team members estimate incorrectly due to incorrect team-shared assumptions. In retrospectives, the architect can be the one to increase the “safety” of different perspectives by raising or encouraging others to raise “things that have gone well, not gone well, and things that are confusing”.
• Work product reviews: I’m an advocate of code walkthroughs and design reviews, and making them explicit sprint tasks. The team can set aside an hour or two a week to review one or two representative work products in order to share ideas, ensure quality, and uncover overlooked errors or opportunities. In this forum the architect has the opportunity to reinforce quality work that is aligned with the requriements and architecture, or supportively correct deficiencies.

Of course there are risks:

• The architect shouldn’t be the know it all: In some cases the architect can be the strong-willed individual who stifles creativity and causes the team to avoid conflict.  Strong teamwork and interpersonal skills are core to the agile method, and those who staff the project must include those skills in selection and evaluation of the architect.
• Keep the architect different: If the architect is a core member of the team, he or she can become integrated into the group and therefore part of a groupthink dynamic.  For this reason, I advocate architects being assigned part-time to agile efforts. Otherwise, the architect risks becoming the extra developer, as near term sprint tasks expand to fill the available team bandwidth.  Consider sharing the architect among two or three projects, or assigning him or her responsibility for technical strategy and planning.

Update 1/28/2010: Saturday’s event postponed to April 10 due to threat of inclement weather

I just received this from the SQL Saturday crew: “We regret to inform you we’re postponing SQL Saturday #30 until 10 April 2010 due to the weather forecast for Friday and Saturday in Richmond VA.  We contacted the Hilton Garden Inn at 804-521-2900 to discuss reservations, but you will need to contact them about your reservation individually. (Apologies – we tried!)

“If you know you will not be able to make the new date, please cancel your registration on the SQLSaturday website.”

—— 

On Saturday 1/30 the local SQL Server community will host an event surely of interest to anyone in the Central Virginia who works with SQL Server databases.  Review the program and sign up here: http://www.sqlsaturday.com/eventhome.aspx?eventid=34.  I’ll be hosting two sessions, “The Business End of Data Modeling” and “Normalize Metadata for Data Integration Analysis.”  After those two morning presentations I’ll look forward to seeing many of the other presentations.  I’ll see you there, and watch this space for future posts on both topics.

Information Management recently sent around their pick of best IM blog articles of 2009.  Among them was Forrester’s James Kobelius’s reaction to Bill Inmon’s “incineration of a straw man concept that he refers to as ‘virtual data warehousing (DW).’” 

According to Mr. Inmon, virtual data warehousing reminds him of the carnival game called whac-a-mole.  He says “just when you think this incredibly inane idea has died and just when someone has delivered what should have been a deathly blow, out it pops again from another hole.” There’s just a very informal definition of virtual DW in Mr. Inmon’s post (remember, he says he’s whacked this mole before), but, as I interpret, he’s talking about a system built after a decision to avoid all the expense of building a data warehouse by just having a query engine that pulls the data from wherever it lives. Mr. Inmon argues that a query accessing diverse databases would leave data integration to the user, and there’s no guarantee that two users would integrate data the same way.  He cites virtual database query inefficiency risks and, on the assumption that the query is trolling operations focused databases, says that source data would be “tuned” to operational rather than informational specifications for history retention and completeness.

Mr. Inmon’s ideas drew quick reaction from Mr. Kobelius and Neil Raden.  Each in his own measured way stresses that integration can be compatible with distributed architectures, and that there is a DW solution architected for efficiency that includes effective data integration from diverse sources: the Federated Data Warehouse.

Experience and emerging tools reinforce their point.  According to a colleague at CapTech, for smaller organizations “you can deal with this issue using a BI tool with a metadata layer that has joins predefined: the data integration is done by the BI metadata modeler.”  Another CapTech’er cites mashup as a potential quick and dirty approach.  Check out “7 Mashups Every Company Needs” here.

A well-architected federated warehouse certainly can integrate and deliver data, maintain history, and enable a “single version of the truth”, perhaps in a more timely manner than a “traditional” DW architecture.  On this question the devil is in the specifics of the situation.  It is difficult to argue one way or another out of the context of a real project in a real organization.

However, even though it certainly has a technical side, data integration is first a business activity.  Sometimes when we apply terms like “semantic rationalization” to software components, we in IT start believing you can actually build a machine that does the things you need to do to rationalize data semantics, like figure out the corporate definition of a customer.  Of course all we can do in IT is to build the empty shell.  The real work happens when business people from departments whose data is being integrated sit down and decide how they are going to define “staff member”, “customer”, and so on.  Only business professionals can say, for example, whether they want to include contractors in staffing reports or whether the term “customer” includes homebuyers under contract but not yet closed.

Integration tools that support data warehouses, whether centralized or federated, are only as good as the business consensus behind them. The consensus behind integrated data is arguably more rewarding to the business that the tools because with consensus on critical objects and events come non-IT-specific improvements like reduction of repetitive and conflicting business processes, reduced communication breakdown due to terminology disconnects, and more.

To me the beauty of the Inmon DW model is that it provides a mechanism that can assist an organization in evolving toward improved information maturity.  Organizations achieve some benefit by simply integrating data into a single data warehouse.  However, the data warehouse also makes source data quality problems obvious and blatantly reveals differences in data meaning from one operational source to another.  So the warehouse delivers some benefit early and also shows how much better it would be if data were integrated.  It therefore becomes a tool for identifying, assessing, prioritizing, and motivating correction of data deficiencies.

For organizations not so far along on the maturity curve, the additional complexity of the federated warehouse tends to obscure this data quality feedback loop. Federation based on drawing from operational sources integrates data from a set of different databases built toward different architectural goals.  On the other hand, the logical data model for the enterprise warehouse is the enterprise data model, and its architectural objective is to integrate enterprise data to provide a single source of truth.  Therefore, the enterprise data warehouse provides an architectural focal point for integration.  It isolates responsibility for improving data integration crisply at either the source or the warehouse, and — within the framework of solid information management strategy, management, and facilitation — motivates diverse business players to work toward consensus definition of enterprise data.

Federation, or virtual data warehousing if you will, can be the best strategy for the mature organization that has already integrated business data to a consistent enterprise view.  For the rest of us, the single centralized warehouse with its unambiguous architectural goals and borders seems the shortest distance to achieving the business benefits of data integration.

Many see IT as application of technology to solve business problems. 

Of course, this is true but it leaves out the third element, which is to apply the right architectural pattern to solve the problem.  For example, when the business problem is that reporting is slow and reports from different departments don’t match, the astute IT professional immediately thinks in terms of a data warehousing pattern employing technologies like databases, extract-transform-load (ETL) tools, and multi-dimensional reporting suites.

A strategy based on the tools alone may solve the immediate problem, but understanding the solution-pattern enables the IT professional to bring to the business the additional benefits that come with the pattern, organizational and IT support impacts, and any risks that might emerge by applying the pattern.  In the data warehousing case the informed architect might cite improved executive dashboards and ability to drill down to root causes from summary reports, the need for data stewardship, and potential long term increase in data storage capacity needs.

Beyond that, when the architect lacks the pattern approach he or she seems to the business person like Bob Dylan’s debutante: “Your debutante just knows what you need, but I know what you want.” On the projects I’ve been on where designers lacked a pattern-based perspective the technical team did exactly what the business folks said they wanted, but for the most part didn’t contribute to the business value of the solution.  On projects like this developers slavishly ensure the solution matches each and every requirement, rarely bring to the table new business requirements that are logical consequences of the design, and tend to avoid questioning defined requirements even if they are contradictory or counterproductive.

Sure, patterns aren’t strictly necessary.  An outstanding architect can design from whole cloth an original solution that precisely matches business need.  However, that’s not how outstanding architects do business, at least the ones I’ve known.  In my experience the best IT architects know patterns, like data warehousing, SOA, and others, well enough to match the business problem to the right pattern and then evolve the architecture from the generalized pattern into a problem-specific architecture based on the particulars of the business problem at hand.

Some examples of common solution patterns in the world of business IT are Data Warehousing, Master Data Management, and Service Oriented Architecture (the Wikipedia article on this one is preliminary at this writing but still a good intro for the uninitiated).   Those interested in a more technical introduction to patterns might start with Avel Avram’s quick intro at InfoQ (Membership at InfoQ is required but free and worthwhile).

And of course, apologies to Mr. Dylan for the title…