Big data is a prime example of the snowball effect, gaining buzzword status within a couple of years. The term and the technology behind it share celebrity status within the cloud, and for good reason, but now there’s a gap to bridge what will likely evolve into a lengthy, multi-sector shift: the task of supplementing organizations’ existing talent pools with the right people for the job.
The job in this case is asking and answering the right questions at the sweet spot adjoining statistics, engineering and business wits; The role in question is the Data Scientist and the demand from employers is just starting to rise.
Here’s a quote from Wikibon’s Big Data Manifesto that addresses this topic:
“As with any young, raw technology, implementing and managing Hadoop clusters and performing advanced analytics on large volumes of unstructured data requires significant expertise, skill and training. Unfortunately, there is currently a dearth of Hadoop developers and Data Scientists available, making it impracticale for many enterprises to maintain and take advantage of complex Hadoop clusters.”
According to a study cited in a WSJ article, the hole that enterprises need to fill in their analytics plans will soon grow to 1.5 million data savvy managers and analysts in the U.S alone. That’s quite far from the realistic number range in 2012, but the IT industry is working on changing that.
Companies like EMC are busy educating engineers and scientists on how to fit in this big data outlook, a direction two of its executive executives discussed in this interview. The storage giant has special course dedicated to training would-be data scientists alongside its cloud workshops, and other firms like Cloudera are working on similar initiatives.