Has big data been the victim of too much hype? Has it failed to live up to its promise? Or have businesses asked too much of big data and not given enough?
Gartner Inc. says thousands of businesses have tried – but failed – to figure out how big data can provide them with the insights they need to make critical business decisions.
Right about now these same businesses that initially had such high expectations for big data are feeling quite disillusioned. And they’re about ready to give up on something that they thought was so brilliant not so long ago.
But maybe they should reconsider says Gregory Piatetsky-Shapiro (@kdnuggets).
Yesterday, we shared a recap of KDNuggets.com editor Piatetsky-Shapiro’s view on the big data and data scientist landscape for 2013. Today, we’re bringing you a more in-depth prediction – the death of big data and big data startup companies.
Read on to find out why he feels that the term big data is set to decline in 2013 while the phenomenon will continue to grow.
Piatetsky-Shapiro is deeply rooted in the big data field and brings a lot to the table with his background in artificial intelligence research and knowledge discovery.
Nearly everything he predicted last year in our interview on predictive analytics is what we’re seeing today.
There’s a huge amount of press and buzz around big data uses and data scientist jobs, but as he pointed out last year and this year – the data is a tool, but it’s not a miracle. Also, the demand for these jobs is centered on data engineers and analysts – not necessarily a true scientist.
With that info in mind, he predicts that we’ll see a number of mergers and acquisitions as well as a number of startup failures in the big data space this year.
“I mentioned in a recent webcast that there were 315 or so startups that work on big data and data analytics. There’s just not enough room for all of them.”
And that matches up with one of the trends we know is growing – self-service access to big data to find answers to questions.
The Human Element Matters
Big data projects typically start with a question and Piatetsky-Shapiro says these projects come from all over the organization – the CEO, other top management and marketing. He says he’s also seen projects that come from the bottom up.
Regardless of where the project comes from, he says you need to have a good sense of what problems you want to solve. However, he calls it an “iterative process.”
“You start with a good question,” he says. “You do some analysis. You find something and maybe find a second question.”
And as he said in the first part of this interview, the human intuition doesn’t always match up with the outcome. But the human element is essential because it’s humans asking the questions. The data just changes things or brings up new questions.
Visualization and Automation
Next, he says we need to look at tools and the purposes they serve. One area that is key is data visualization tools.
And this is a key area for the “human element.”
“But we cannot rely everything on visualization tools because that requires a human to be in the loop and watch everything,” he says. “I think there is a need for more automation for many of the processes.”
Will Big Data Bring on Big Brother?
Another big headline Piatetsky-Shapiro mentions is the focus on big data and privacy. But as much as we’re seeing it in the headlines, he’s not sure there’s much that’s going to change in addressing it – at least in the US this year.
“We leave so many digital crumbs that it’s difficult to remain private,” he says. “For example, a competition on Netflix that produced data that was completely anonymous. Still, some researchers managed to find out who were some of the anonymous users by looking at the movie ratings.
“If we take for example, healthcare, there is a question in there. Once the data is collected, it’s very hard for it to stay private.”
Piatetsky-Shapiro predicts that we will see more privacy breaches, but just like the big data “hype,” nothing much will come in the way of change.
“I think we will see [more anonymous data revealed], but I’m not sure of the result,” he says. “In the US, privacy has been very lax.”
He predicts that we may see increased attention to it, but he doesn’t think anything will happen in terms of laws and regulations.
The Big Data Death Prediction
But what he does think will happen is a change in our terminology in light of the dying hype around the term big data.
I think we’ll see the decline of the ‘big data’ buzzword,” he says. “Maybe it will be replaced by ‘smart data.’”
