Bob Lambert

—–

Update 10 January 2010: Thanks to Gints Plivna for observing that we had not posted the slides to this presentation, here they are: Pretty Good Row Level Security Slides.  – Bob

—–

Thanks to those who attended Saturday’s Microsoft Code Camp (see http://richmondcodecamp.org/).  Here are materials for the presentation “Pretty Good Row Level Security” which I did with Nic Morel, my fellow CapTech Ventures Lead Consultant.

Summary:

The presentation reviewed a security solution that limits access at the data level, leverages simple database protocols, requires minimal administration resources, applies to both users and application developers, and overcomes some of the vulnerabilities of default approaches like having the application use a “system” user id.

The solution presents a three layer security architecture:

1. Base tables are accessible only to DBAs.

2. A security layer including cross-reference tables relating user ids to key data like department or sales territory, and table-valued functions that accept user id as a parameter and deliver data from base tables with a join to the security cross reference tables.

3. A data access layer exposed to users consisting of views that access the table-valued functions, providing the current user id as a parameter.  End users and developers only have access to this data access layer.

Email me if you’d like the powerpoint (a little big to attach here, even without the accent graphics), and I’ve pasted in the SQL scripts as a comment on this post.  The second two examples run against the SQL Server AdventureWorks database that ships with 2005 Developer Edition.

In a new post at Insurance Networking News Ara Trembly provides a balanced perspective on IT/business misalignment (Business/IT Misalignment: Whose Responsibility?).  He describes the problem as cultural, more amenable to relational than management solutions.    His conclusion sums it up: “Take a geek/suit to lunch today!”

To me (speaking as an IT professional) IT should take the initiative to solve the problem.  Quoting Trembly, “business executives … make decisions, but they are for the most part mystified at the magical incantations and actions that produce IT results” and “IT people, on the other hand, are jealous of the sheer power wielded over them by business people who just don’t get IT.”  In other words, business people contend with an emotional and a substantive problem, “fear and lack of knowledge,” while IT people have only the emotional problem of jealousy.

If we take the emotions out of the picture (its just a job, right?) then that leaves IT folks with knowledge that business people need in order to maximize the value of IT and efficiency of business processes.  Ever since mainframes roamed the prehistoric rain forests of the ’60s application developers have often been the most knowledgeable about how business processes really work, understanding both the intricacies of the application logic and how business people use the system to get things done.  These individuals can add value to the business discussion by bringing their knowledge to the table in a way that business people can understand.

In many organizations IT manages the forum in which these conversations can occur: the requirements process.  In my experience a good requirements process is long enough for the business and IT teams to get to know each other, offers generous opportunity for both structured and unstructured conversations about business needs, and brings together knowledgeable business and IT participants.  IT is typically able to bring the insights of seasoned application developers to the fore in a well planned requirements effort.

Yes, everyone has responsibility to “cultivate personal relationships based on mutual need and respect,” but IT can and should bring substance to the relationship in requirements definition.

It seems I’m frequently in conversations about using null to represent a business value.  To paraphrase, say there are credit and cash customers, and there’s a suggestion to set “Customer_Type” to “C” for credit and null for cash.  To data and database professionals this is obviously a bad idea, but it’s not obvious from a business point of view.

In a database null means that there is literally no value, or the value is indeterminate.  Null is not the same as zero or blank.  When a database operation involves nulls the result can be difficult to predict for someone not practiced in SQL.  In many cases the answer is null.  For example, 1+0=0 but 1+null=null.  In plain English, what you’re asking the DBMS to do in the latter case is to add 1 to [I don't know what], and of course 1+[I don't know what] equals [I don't know what].

So, if you use null to represent a business value then you might not get the results you’re looking for when you try to get business answers out of your database.  For example, say “C” represents credit customers and null represents cash customers, and you have 2 cash and 1 credit customers.   In SQL Server if you use a Count function to tally all of your cash customers the answer isn’t 2, it is null.

That’s one example of why it’s not a good idea to try to represent a business fact with a null value.  It doesn’t make business sense and in this case the DBMS, correctly, won’t make sense of it for you.

To be clear, whether or not a given database column permits null values is an entirely different question, best left to database designers.  For example, a database table might record which patient occupies which hospital bed.  It may be reasonable and correct to assign a null patient ID if the bed is currently available.  However, there are alternative methods of representing this situation, and the database designer should be free to choose the right alternative taking into account the specifics of the application under development.

While many organizations understand the value of managing the information resource, for many others information management remains abstract and difficult to define.  In an effort to make it concrete here’s a hypothetical proposal to provide an Enterprise Information Architect for a hypothetical organization that really needs one.

Today: inconsistent data of uncertain quality blurs enterprise view and restricts planning

Today managers, planners, and analysts lack the information required to run the organization as a single enterprise rather than a collection of diverse units.

• Data quality in IT applications varies to the point that, outside financials, it is impossible to gather consistent data supporting an enterprise view of operations.
  • Application development efforts have focused narrowly on departmental interests without accounting for enterprise concerns, making application data incomplete in describing business processes and inconsistent with data in other applications.
  • Focus on departmental concerns and tight development timelines has resulted in incomplete validation of data critical to the enterprise but not critical to the application’s focus.  For example, customer demographics are not critical to the sales process and therefore zip codes and telephone numbers are not consistently collected at point of sale, substantially reducing value of market analysis based on sales data.
• Enterprise planners work with only the highest level summaries of operational data, those summaries suffer large margins of error, and planners cannot definitively answer questions required to make critical business decisions.
• Regulators have questioned the validity and repeatability of reporting because of the organization’s heavy reliance on spreadsheets and manual processes in gathering and compiling data for reports.

Solution: enable sound planning and management by identifying data assets and setting processes to manage them

Empower an Enterprise Information Architect to lead an effort that (1) identifies data that describes the organization, (2) defines how to integrate and improve quality of that data, and (3) improves the ability of information technology to maintain data quality.

(1) Lead definition of an Enterprise Information Architecture identifying information required to manage the organization as a single integrated enterprise, and data quality standards that ensure that data supports enterprise goals.

• Identify and define events and objects critical to the enterprise

• Identify and define relationships among those events and objects and attributes that describe them

• Classify data managed by the organization by type (operational, statistical, financial, decision support, etc.) and define standards for managing and integrating each type.
• Compile the above into a plan that explicitly supports the enterprise strategic plan

(2) Working with senior business managers, put in place a program of data quality improvement that plans and executes specific measures and sustained commitment to improving data quality in business processes and IT applications

• Identify the business group responsible for maintaining quality and integrity of each business object, event, relationship, and attribute

• Identify for each data item of interest to the enterprise its “system of origination” and “system of record”.

• System of origination is the application that provides the entry point of a given data object to the organization.
• System of record is the application that is the source of record for the data object.

• Define and deploy standards and practices for for business process and IT application definition that support data quality and integrity standards

(3) Working with senior IT managers define and put in place standards for application requirements definition, data management, and metadata management to

• Define and deploy application development and interface standards that support data quality objectives.
• Ensure that application development efforts support enterprise data quality
• Continually monitor new developments in data management best practices and make that information available to the enterprise.

Excerpt from "Show Me the Money: A DM/BI Business Value Primer", Bob Lambert and Tri Truong, Information Management Special Reports, March 24, 2009

Before starting the Business Intelligence business case, the BI advocate should do the homework required to ensure its success, including these essential steps:

1. Know the organization’s goals and objectives.
2. Identify a BI champion.
3. Identify and work with BI stakeholders.
4. Identify an application with tangible business value.
5. Define and quantify a quick win prototype project.

Know the organization’s goals and objectives. It is human nature for any of us, including executives, to be receptive to help with our own goals and objectives but less receptive to new ideas that aren’t related to our own goals. Furthermore, senior executives facilitate intensive strategic planning processes to set the right corporate goals and objectives. A proposed BI initiative should clearly and tangibly help achieve strategic objectives already in place.

Identify a BI champion. BI is in a unique position within the application stack. Most organizations can operate without a BI strategy. However, most companies would greatly improve their market position with a comprehensive BI solution. The impetus for deploying such a solution needs to come from a leader within the corporation who champions the value that BI brings to the organization as a whole. Often, this champion is someone at the top level of the business chain of command with a solid grasp of the BI’s potential.

Identify and work with BI stakeholders. BI projects should be driven by BI stakeholders, those who will see direct effects (good or bad) from the BI project. Some stakeholders look to benefit from BI-based solutions to concrete problems. Other stakeholders will have to be convinced about the potential value of BI. Both types of stakeholder must be involved in defining and supporting the goals of a BI project.

Identify an application with tangible business value. Again, in order for the BI application to return value, it must focus on achieving business goals. These goals should be measurable so that the value of the BI application can be determined, and the application should contribute to overall organizational strategy.  Scroll down to “Business Value Examples” here for more.

Define and quantify a quick win prototype project. Businesses must quickly see the value that BI brings in order for it to catch fire in the organization. A prototype project is often the best way to showcase BI’s value proposition. These projects should typically produce tangible results in a matter of weeks and target a well-defined business area. The prototype should have a well-defined goal and ROI metric, and produce data or case studies that show progress toward, if not achievement of, that goal.

- Thanks to co-author Tri Truong for assistance with this post.

Excerpt from "Protecting Your Data with Row Level Security for SQL Server Databases," March 17, 2009

Data security is not optional in today’s business environment. High-visibility hacking and fraud, Sarbanes-Oxley, HIPAA regulations, and the Patriot Act all reinforce the need to present the right data to the right users and prevent the wrong ones from gaining access. Typically, “row level security” (RLS) is one requirement: to allow or permit access to particular users based on data in a particular database row. SQL Server does not provide built-in row level security.

In an article at Dr. Dobbs I present a way to build your own SQL Server row level security that restricts user access to data based on data in the row, without changing content of business tables, without affecting application or presentation developers, and regardless of how users access the data. Here’s my example application: How to can I add security to my existing orders database that will limit managers to departments they manage — and departments that report to those they manage — regardless of how users get to the tables and what queries and reports are developed against the database?

There are a number of ideas and code samples for hand-built RLS solutions ideas available on the web.  My favorite is A Fairly Capable Authorization Sub-System with Row-Level Security Capabilities (AFCAS) by Kemal Erdogan.  His solution is based on a lookup table, similar to the one I present. His solution is better than many others because it doesn’t involve business table changes, like adding a security code to each row, but still leaves tables unsecured if users access tables directly rather than through a given application.

The essence of the approach is to (1) use cross reference tables linking userid (or Active Directory group id) to business data values, and (2) protecting data tables behind table-valued functions that accept userid as a parameter and join it to the cross reference tables, returning only those rows that the user is permitted to see.

Using this solution can reduce security admin overhead and enable users to get to what they need without putting secure data into the wrong hands.

Here’s a little-recognized fact about data integration: if you run a business or any sizable chunk of one, someone is integrating your data.

In my professional life I have on occasion suggested data integration efforts.  Sometimes my suggestions have been accepted and sometimes not.  As an IT professional I understand that different managers have different priorities, and in a given business situation sometimes other things are more important for example than having a single, consistent source for all customer records, or making sure production data matches financial data.

But as a customer?  That’s different.

A couple of years ago I bought a laptop from a company renowned for quality and customer service.  For the first weeks the computer was all it was cracked up to be, but then it cracked up.  The screen developed a mysterious flicker.  After a few diagnostic conversations they replaced the main logic board.  The problem recurred a few months later, and this time the company traded the lemon for a new computer.

All was well, but here we encounter the first data integration problem: they said I needed to call a different number to have my three-year service agreement transferred, which I did.

Months later I called service for a minor problem, and they had no record of the service agreement for the computer.  My warranty was still connected to the lemon.  After about an hour on the phone this company’s outstanding support staff came up with a more than satisfactory solution.

Even so, this company’s service records weren’t integrated with its warranty records.  In this case data integration happened because of my insistence and the service staff’s creativity.  The cost?  Only considering my last encounter, three service professionals were tied up for about an hour, and I’ll think twice before I buy again from this company.

It seems the choice is either pay now to integrate so all applications work from consistent data or pay later by having staff, customers, and suppliers do it on a case-by-case basis.

History is littered with IT application projects that end late, go way over budget, or abandoned altogether.  I was fortunate enough to see one work out really well (almost – please read on).  It was no mistake.  It came down to a simple method advocated by a gentleman named named John Carpenter.

The project was an HR management software conversion from one commercial off-the-shelf software (COTS) package to another.  The company concerned was conservative about spending money.  A previous business case had proposed a similar project.  The problem with that business case was that the benefits were really tough to conceptualize, so the cost/benefit analysis relied on soft benefits like “improved access to information” and “more consistent reporting data”.  The folklore was that the CFO had physically thrown that business case out of his office.

Mr. Carpenter’s method was to divide requirements definition and implementation into two distinct projects, with a different business case for each.  Under his direction, we wrote a ~1m business case for requirements definition only.  We proposed that this first project would result in another business case precisely specifying the schedule, method, cost, and benefits of the implementation project.

According to John, “the approach we used would not be considered a textbook approach for an ERP (enterprise resource planning) implementation.  What we did was more of a strategy to address the the CFO’s concerns.  The company was very risk-averse so we needed a way to take out as much risk as we could.  This was a large project because it involved four major modules affecting the three main areas of HR, and the company wanted to know costs and benefits at each step.  Complicating matters, HR business processes and therefore requirements were not clearly understood – the HR department seemed to rely on on the job training rather than documented procedures.  So we presented the first phase as an investment into understanding HR processes, as well a precise roadmap for implementation.”

This first business case was accepted by that same CFO and we got started on the 7-month effort. We brought in a consulting team experienced in the proposed COTS package, and followed their lead in requirements definition and prototyping.  During the prototyping step they walked HR staff through each relevant function in the software package, detailing how to configure the package for their specific needs and where we’d need to customize it.   The result was a definitive, detailed document that showed how the package fit HR process and how it would need to be customized.  Then, we used those results to build a business case that included specific configuration, customization, hardware, and software costs, as well as the process and organizational changes that would be required, not to mention the benefits that would accrue.  The business case showed substantial improvement, predicting real financial benefits within 4 years.  Even better, on a depreciated basis the project literally was almost free, costing only $1,800 in the first year and returning benefits thereafter.

The business case was accepted by the company’s executive committee and the project started.  It ran exactly as outlined by the results of the requirements effort, with very few of the nasty surprises often typical of large projects, and it tracked to forecast schedule and budget.

Proving that no good deed goes unpunished, the company, whose core business was real estate, in effect folded in the financial crash of last autumn ’09 , one month from implementation.

At any rate, the lesson I took away from the effort was that dividing requirements and development into separate projects gives business visibility into a project, helps manage financial risk, and enables the project to ground predictions rather than guessing at costs and benefits before they can be known.

Excerpt from "Illusions, Allusions – Let’s Get Real about Database Design", October 4, 2002

Much of today’s IT application development – custom or off-the-shelf – involves integrating data from legacy systems, third- party software products and external data sources such as demographics or mail lists.  More often than not, data integration is unexpectedly complex, either due to data quality issues or the nature of the data integration itself.

Here are some typical examples:

• One ERP package uses the same table for both Sales Quotes and Sales Orders. Columns that mean one thing for Quotes mean quite something else Orders. One team extracting data from this ERP package continually mixed up, for example, Date Received on the Quote with Date Prepared for the Order. The designer who blindly copies data from input systems can propagate these issues. In this case, the correct solution is to extract the two documents into separate tables in the destination system, making each column describe either a quote or an order, not both.
• Marketing databases often store data purchased from several third parties on the same set of customers. These sources usually include overlapping columns with different values. For the same customer, different sources might store different values for the person’s address, credit scores or even name. It is sometimes important to preserve all of the columns from all of the sources and to maintain the information on where the data came from as well as what its value was. This can result in a messy database design, where columns again carry dual meaning: their value and their source.
• Codes from legacy databases tend to evolve into complex forms, embedding more and more information into a single field. This is perhaps a natural reaction to the slow evolution of the system relative to changes in business, as users shoehorn information into the system that it was not designed to store. For instance, in a legacy system a one- character code might classify customers by “customer category,” with values 1 for small business, 2 for mid-size, and 3 for Fortune 5000. Users might add codes 4, 5 and 6 for corresponding values for aerospace customers, then 7 for federal government, and so on. The database designer must know the data well to extract each embedded concept into a different destination column.

When data integration is part of a project, expect complexity and leave room in interface development estimates for devils in the details of source system analysis and integration design.

Not everyone gets to be a leader, and most leaders are also followers in their own right.  The project manager follows instructions from the project sponsor, the CEO from the board, the politicians from the polls, and so on.

Followership is the yang to leadership’s yin, and according to many interesting sources following can be as fulfilling and important as leadership. For example, check out this site: http://www.exe-coach.com/followerPartnership.html.  Quoting: “When both the leader and follower are focused on the common purpose a new relationship between them arises. This new relationship is candid, respectful, supportive and challenging. It is a relationship that honors open communication, honesty and trust from both parties.“  The article argues that effective followership is the key to making today’s flat organizational models successful and mitigating risk of corporate malfeasance and scandal.

Think about it: more people are followers than leaders, so isn’t it more important to cultivate effective followership than effective leadership?