Question: What is the difference between a business intelligence professional and a Data Scientist?
Answer: $40,000 a year, or potentially hundreds of thousands of dollars or more over the course of a career.
That’s not the only difference, of course, but it sure is significant incentive for business intelligence and data warehouse pros to up their skills to meet the growing demand for Data Scientists, that rare breed of man or woman with a blend of statistical and business, analytic and communication, computer science and social science skills needed to turn Big Data into Big Dollars for the enterprise.
The real question is are the skills most business intelligence pros have applicable to data science? For Bill Schmarzo, the answer is a resounding yes.
“There are lots of similarities between business intelligence skills and Big Data skills,” Schmarzo told me in a recent interview. “The need for data preparation, in particular, is very similar.” BI pros also understand key performance indicators; they understand how decisions get made in their business; and they understand how the related systems and tools have evolved over time – all important areas for aspiring Data Scientists, he said.
Schmarzo is in a position to know. A self-described “old-school BI guy,” Schmarzo has spent most of his career working with business intelligence and data warehousing systems, including three years overseeing Business Objects’ analytic applications division in the mid-2000s. Today Schmarzo serves in EMC’s Global Services unit overseeing information management and analytics-related services and consulting. Increasingly, Schmarzo advises EMC customers making the transition to Big Data.
He said the biggest obstacle for BI pros to transition to the role of Data Scientists is not a lack of skills around writing MapReduce jobs or confusion around Hadoop and its funny-sounding subprojects. Those obstacles can be overcome by smart, committed BI pros with some training, he said.
No, “The challenge is not so much [learning] the programming model [of Hadoop and MapReduce], but the need to get comfortable and to change mindset,” he said.
I think Schmarzo is right. But what exactly is the difference in mindset between BI pros and Data Scientists? In my opinion, it boils down to this: BI pros apply their skills to data in order to provide answers to pre-existing questions. Data Scientists look at data as a blank slate, searching for answers to questions nobody has even bothered to ask before.
Put another way, BI pros are used to massaging data to fit into a particular paradigm that the business is comfortable with. The job of the Data Scientist is to question the paradigm itself and the assumptions – sometimes faulty – that led to it.
That’s a pretty significant shift in thinking. And unfortunately, changing ones way of thinking, often engrained after years of experience, is not a trivial undertaking. But I agree with Schmarzo that it is not an impossible chasm to cross. To do it, BI pros must commit themselves to undertaking the necessary training – both formal and informal – to make the journey. That means staying current on the latest trends in analytic practices and research methods, taking educational courses and seminars dedicated to developing a curious and authority-questioning mindset, and learning from peers that have already made the journey.
But BI pros that want to make the transition to Data Scientist must challenge themselves in other ways too. Namely, they must develop a genuine interest in how the rest of the business operates, how data can help find new business opportunities, and how analytics can identify areas to improve operational efficiency. How should BI pros go about this? Ask to shadow sales or marketing or finance pros for a day or a week to better understand their concerns and challenges; take a course in philosophy or economics or in some other area outside your comfort zone at a local college; and, most importantly, never stop asking questions.
Successful Data Scientists must be willing to take chances and fail on occasion. The trick is not giving up, remaining persistent, questioning conventional wisdom, and being open to new experiences. Schmarzo thinks BI pros with the inclination and the motivation to make the move to data science can do it if only they’re willing to change their mindsets. I agree, and company’s out there looking to expand into Big Data would be wise to invest in such BI pros when they can.
As Schmarzo put it to me, “to write off all BI practitioners is foolish.”
