The meteoric rise of interest in big data and technology to exploit the reams of information flowing into companies has not been matched by the needed growth of those with the skills to gain actionable insight from data.
Various research firms have forecast a looming shortage of data scientists and others with the skills needed to corral big data.
For example, 34% of CIOs cite a skills shortage in big data, according to a survey by recruitment firm Harvey Nash. That category wasn’t even on the radar in 2011 and only emerged in 2012.
But does this mean that people who have not been formally trained in statistics are excluded from this growing field?
Not according to Nate Silver, a former New York Times blogger and a statistician. Silver used data analysis to successfully predict that President Barack Obama would be re-elected.
Because of his success, the election was not only notable as a win for Obama but also a win for big data.
Silver, author of the book “The Signal and the Noise,” which details the power of predictive analytics on our daily lives, recently spoke to the Harvard Business Review about how people can get involved with big data without going back to school.
“I think the best training is almost always going to be hands on training,” Silver says. “But my experience is all working with baseball data, or learning game theory because you want to be better at poker, right? Or [you] want to build better election models because you’re curious and you think the current products out there aren’t as strong as they could be. So, getting your hands dirty with the data set is, I think, far and away better than spending too much time doing reading and so forth.”
He goes on to advise that companies integrate employees with statistical skills throughout the organization instead of creating a silo for them.
Silver recognizes that there are times when the data is good and the analysis is accurate but people still push back and challenge results. He advises that those relying on data analysis present their results to at least a modestly large audience.
“Einstein supposedly said that, ‘I don’t trust any physics theory that can’t be explained to a 10-year-old.’ Now, if you feel like you’re expressing yourself and getting the gist of something and you’re still not being listened to, then maybe it’s time to change careers,” Silver notes. “It is the case [that] people who have analytic talent are very much in demand right now across a lot of fields so people can afford to be picky to an extent.”
One of the biggest challenges for those trying to weave data analysis into the fabric of a company are the people who are resistant to relying on big data, but prefer to make decisions based on intuition.
“The question should be, ‘How good is a model relative to our spitball, gut-feel approach?’ And also, ‘How much do we know about this problem?’ There are some issues where you just don’t have a good answer and you have to hedge your risks as a business and not pretend that you’re more certain than you really are,” he notes.
In addition, he adds that businesses often are reluctant to deal with uncertainty in their outlooks.
“Just because a model is not going to be very precise or accurate doesn’t mean that therefore you should trust your gut instinct after a couple of whiskeys and assume it’s going to be very much better,” Silver adds.
Next Steps:
- Subscribe to our blog to stay up to date on the latest insights and trends in big data and data analysis.
