Bob Lambert

Jazz on the harmonica

What is Big Data Creativity and How Do You Get It?


Thomas EdisonIn a recent Smart Data Collective post, Bernard Marr cites creativity as a top big data skill, but what is creativity?

His point is, since big data applications are often off the beaten IT path, big data professionals must solve “problems that companies don’t even know they have – as their insights highlight bottlenecks or inefficiencies in the production, marketing or delivery processes,” often with “data which does not fit comfortably into tables and charts, such as human speech and writing.”

Clearly, Mr. Marr is correct in citing creativity as a critical big data skill. For example, this article undermines its title by citing two data-derived insights that enabled successful marketing campaigns: “Only 4% of women consider themselves beautiful” triggered Dove’s campaign for real beauty, and the “‘Keep Walking’ campaign for Johnnie Walker grew out of data finding that its drinkers were successful but still striving.” More seriously, big data specialist Jonathan Wasson makes the point that the tragic peculiarities of the Ebola crisis “practically scream please find a creative data solution”.

Of the six skills Mr. Marr cites, creativity seems by far the least tangible and raises the most questions: What exactly is creativity? Is it something people are born with, or can it be learned? How can we recognize it in a job candidate? When I looked into the answers to those questions, informed sources echoed Thomas Edison’s advice that it’s “one percent inspiration, ninety-nine percent perspiration.

The great choreographer Twyla Tharp’s book The Creative Habit explores creativity by recounting the elements of her approach to work. The book’s practical advice to fellow artists is summed up here, but I took away a different message: creativity is directly proportional to hard work and good habits. When Ginger Rogers asked Fred Astaire why he worked so hard, he replied “So it looks easy.” Mozart, portrayed as a super gifted party animal in the movie Amadeus (the dances of which Tharp choreographed), was a dedicated professional who practiced piano so much in his youth that his hands were deformed by young adulthood. Tharp’s book fairly crackles with uncompromising energy as she recounts her routine, teaching methods, and personal sacrifices for her art.

Tharp avoids discussion of natural talent, preferring to focus on individual differences in “creative DNA”. For example, one musician’s interests and abilities might lead him or her to be a great blues guitarist, another a great classical violinist. Other things being equal, their creative achievement depends directly upon hours in the woodshed and effectiveness of work habits.

Turning to the academics, there is a generally accepted “componential theory of creativity”, described here by Teresa M. Amabile, its originator. Defining creativity as “the production of a novel and appropriate response, product, or solution to an open-ended task,” Ms. Amabile describes these components of the componential theory:

  • Domain-Relevant Skills: Basically, this means how much a person knows or can do within the relevant domain. For example, big data creativity depends on skills in Unix, Java, software design, distributed data management, and so on.
  • Creativity-Relevant Processes: Based on reading from Ms. Amabile’s paper, I boil this one down to two seemingly contradictory traits. First, the creative individual is an independent, expansive thinker, willing to take risks and able to draw connections between seemingly unrelated things. On the other hand, this component also includes self discipline and high tolerance for hard work in spite of ambiguity.
  • Intrinsic Motivation: Does the person really like the field, or are they involved for “extrinsic” reasons like doing it for money or being directed to by their supervisor.
  • Creativity-Promoting Social Environment: In the big data world “social environment” translates to workplace. Does the organization encourage and reward creativity or not? Amabile lists these potential blockers: “political problems in the organization, criticism of new ideas, emphasis on the status quo,” and a prevailing low risk attitude generally. On the other hand, organizations featuring high creativity tend to reward risk (and even failure!), are collaborative, diversely skilled, and generally encourage sharing of ideas.

After reading from these two sources, creativity seems simple to describe but hard to achieve. Do you want to become more creative? Get motivated, get disciplined, and get to work – reading Tharp’s excellent book is a great way to start on the motivation and discipline.  There is no shortage of great low or no cost ways to learn about various big data tools. To get started, check out training available from Cloudera, Hortonworks, and MongoDB.

If you are a manager looking to build big data capability in your organization, then your challenge is to find and hire these creative folks, and provide the “Creativity-Promoting Social Environment”. This site provides an advice page on recruiting and hiring. As to providing the right work environment, Ms. Amabile’s theory has been extended to organizational innovation. A good initial resource is her 1998 Harvard Business Review article “How to Kill Creativity“.

So creativity isn’t magic, and if you believe the experts isn’t a gift. Instead, it can be learned and developed the old fashioned way with diligence and discipline assuming big data specialists and their employers want it badly enough.





Leave a Reply

Your email address will not be published. Required fields are marked *