There’s consensus among data quality experts that, generally speaking data quality is pretty much bad (here, here, and here). Data quality approaches generally focus on profiling, managing, and correcting data after it is already in the system. This makes sense in a data science or warehousing context, which is often where quality problems surface. To quote William McKnight at the first of those sources:
“Data quality is no longer the domain of just the data warehouse. It is accepted as an enterprise responsibility. If we have the tools, experiences, and best practices, why, then, do we continue to struggle with the problem of data quality?”
So if the data quality problem is Garbage In Garbage Out (GIGO), then I would think that it would be easy to find data quality guidelines for app dev, and that those guidelines would be lightweight and helpful to those projects. Based on my research there are few to none such sources (please add them to the comments if you find otherwise).
So, all that said here’s my cut at app dev data quality guidelines by project activity: Continue reading →
I believe that early, effective big picture diagrams are key to application development project success. According to the old saw, no project succeeds without a catchy acronym. Maybe so, but I’d say no project succeeds without a good big picture diagram. The question: what constitutes a good one? To me good high-level diagrams have four key characteristics: they are simple, precise, expressive, and correct.
Out of curiosity I recently reviewed articles critical of Agile Methodologies. I had expected agile-versus-waterfall arguments and attacks from vendors selling new alternatives, but even given the reputation that advocates have for flaming well-intentioned critics, I wasn’t prepared for the level of emotion I found.
My opening position was that Agile techniques are great, but like any other tool there are limits and prerequisites. The critical articles I read strengthened that view. Let’s review three examples that stood out, in reverse order: Continue reading →
Recently the BBC posted this video. On first view it is just funny, but watching those dogs learn to drive really reminded me of personal experiences with IT teams making big learning transitions. To represent those real situations let’s consider a fictional team of SQL developers facing the daunting task of deploying a functional Hadoop-based analytics prototype in two months. The video parallels their critical learning success factors: (1) set audacious goals, (2) learn bit by bit, and (3) know your limits.
When Tom Petty sang, “Hey baby, there ain’t no easy way out” he wasn’t referring to business intelligence (BI) reporting but he might have been. Current generation reporting engines, AKA data visualization or data discovery tools, market their products with statements like these, emphasizing quick development and ease of use:
“The democratization of data is here. In minutes, create an interactive viz (sic) and embed it in your website. Anyone can do it— and it’s free.” (Tableau Products Page)
“Easy yet sophisticated report design empowers your employees to design professional and telling reports in minutes not days” (Windward)
I like these tools, and I do believe that they can provide a leaner, more productive, and more informative approach to BI reporting than some more mature products. However, none is a silver bullet for all data integration and reporting woes. Continue reading →
In my experience, some BI projects ultimately finish as a success, but exceed budget and schedule targets and fall short of functional goals along the way. On projects like this, somewhere in the midst of report development, things get sticky and tasks fall behind schedule as the team runs into unexpected complexities. Continue reading →
I’ve written a number of posts about agile techniques of project management on this site, all in a spirit of advocacy. A comment on the most recent reminded me that agile/scrum isn’t necessarily the right solution in all situations, and in some it may work but needs to be applied carefully. After that comment I thought it would be interesting to write about situations in which agile methods should be applied with care, if at all: Continue reading →
OK, I’ve lost a five-metre scrum, my pack has been overrun, and the ref has raised his arm between the sticks for a penalty try. My colleague Margy Thomas, with support of fellow rugger Billy Tilson, has convincingly argued that agile development in fact is very like rugby union. Margy cleverly fended my meager one-point case with a point-by-point list of the ways that agile projects and rugby are similar. I’ll hold on to my view that sports analogies are generally weak in describing application development, but I’ve come to observe a fundamental similarity between rugby and agile/scrum. Continue reading →
What if you could double the efficiency of your software testing process, and substantially reduce errors found during the test, deployment, and maintenance phases, without purchasing any tool or method? The November 28 InformationWeek offers just that in a reprint of a recent Dr. Dobbs article on formal inspections by Capers Jones and Olivier Bonsignour. They call formal inspections the “defect removal tool of choice” and back up their claim with lots of hard evidence, but I think they are still selling short. Continue reading →