In “Big Data’s Big Problem: Little Talent,” an article published yesterday, the Wall Street Journal’s Ben Rooney used the success of the Splunk IPO to discuss how quickly the big data market has exploded – and how the talent pool hasn’t caught up yet.
Most of what Rooney talks about should be familiar to our readers. The act of gathering so much data is great, but a new type of professional – the big data scientist (or engineer or architect) – is required to help make sense of the chaos and turn it into more actionable business intelligence. Nothing new there.
But when Rooney describes his conversations with top-level industry executives on the problems facing the field of big data for some context, things get interesting.
“IBM started a generation of Cobol programmers. Thirty years ago we didn’t have computer-science departments; now every quality school on the planet has a CS department. Now nobody has a data-science department; in 30 years every school on the planet will have one,” EMC Corp. President and COO Pat Gelsinger told Rooney.
There’s something to that parallel: Just this past March, Frank Coleman, another EMC executive, wrote about how many data scientists may be “hiding in the shadows,” outside the traditional IT structure. That’s simply because it takes a lot of skills that, historically, haven’t been mission-critical to the IT hiring process. A data scientist needs to combine database skills and the hacker mindset with an understanding of the business and experience with complex analytical models.
Finding one or the other out there can be tricky, but as an integrated field of study, well, we’re only seeing the tip of the iceberg. President Obama’s administration has announced a $200 million Big Data Research and Development Fund that’s priming itself to help close the education gap (among other things), but in the meanwhile, it’s touch-and-go.
The Journal article quotes a few IT recruiting managers and headhunters dealing with the big data hiring problem, who say that they’re only figuring out now how to suss out the best candidates. The only consensus on what makes a good data scientist is that there is no consensus, save perhaps that familiarity with the data is as important, if not more so, than mathematical ability.
Of course, there’s a dissenting opinion: HP apparently believes that tools like its recently-acquired Autonomy suite can help companies take advantage of big data and analytics with smarter algorithms, not the support of a dedicated professional.
But, as a recent Wikibon report shows, for those who can find work as a data scientist, the rewards are out there, with salaries ranging from $60,000 to $115,000.