Big data is here to stay.
And that’s a good thing for companies that understand how to capture and analyze the massive amounts of data generated every day to make the right business decisions, improve operations, become more competitive, and increase profits.
But putting big data to good use is about more than Hadoop and a “bunch of fancy technology,” according to IT consultant Graham Oakes. For big data to do its job, companies have to overcome the “very real” organizational silos that have grown around that data.
In a recent survey, 56% of respondents cite organizational silos as the biggest obstacle to effective decision making that’s driven by big data.
Oakes shares six barriers that will get in the way of big data in a typical organization:
- Infrastructure: Big data uses a lot of technical infrastructure, storage, bandwidth, CPU, etc., all of which generate highly variable workloads. That means the amount of infrastructure you need varies as well – sometimes you need a lot, sometimes you need a little. The answer to this challenge – the cloud. But not so much as it pertains to the technical side of things. Rather it’s about selecting the right cloud vendor for your company’s needs and ensuring you don’t break the bank doing it.
- Applications: Behind big data is a complex application stack, some of it not very mature. As an example, Oakes points to the Cloudera Hadoop distribution, which contains a dozen applications, some of which are pretty new. To overcome this barrier, you have to “get up several learning curves at once,” integrate a number of tools with your existing application stack, and build a stable operating environment out of these different pieces.
- Skills: Not only do you need business analysts to ask the right questions, and technologists to “tame the infrastructure and applications,” you also need data scientists. You know, the people who can do things like understand the statistical algorithms, and drive the visualization tools. They’re out there, but they’re hard to find. But when you do find them, be sure to make them part of your team, Oakes says.
- Attitude: When it comes to big data projects, you have to “experiment, learn and evolve,” rather than “plan then do,” according to Oakes. But it’s important to balance that research with the objectives of the business.
- Fragmentation: Most organizational data is highly fragmented. That’s because every business unit seems to own a different piece of the data, which creates data quality issues. No one department is responsible for all the data and no one ensures that it’s “correct, consistent and up to date,” Oakes says.
- Valuation: Not very many companies can accurately value their current data, “let alone on the fuzzy web that Big Data exposes,” Oakes says.
Oakes says the most important thing for organizations to do to over come these challenges is to build multi-skilled teams with the right attitudes.
“Right now, many big data projects are merely playing with the data, exploring the tools and shifting data around within its silos,” he says. “If we could build some stable, cross-functional teams and focus them on business-led experimentation, then we’d probably begin to find real value in the data we have stashed away.”
Next Steps:
- Subscribe to our blog to stay up to date on the latest insights and trends in big data and data analytics.
- Join us on August 23 at 1 p.m. EDT for our complimentary webcast, “In-Memory Computing: Lifting the Burden of Big Data,” presented by Nathaniel Rowe, Research Analyst, Aberdeen Group and Michael O’Connell, PhD, Sr. Director, Analytics, TIBCO Spotfire. In this webcast, Rowe will discuss recent findings from Aberdeen Group’s December 2011 study on the current state of big data, which shows that organizations that have adopted in-memory computing are not only able to analyze larger amounts of data in less time than their competitors – they do it much, much faster. TIBCO Spotfire’s Michael O’Connell will follow with a discussion of Spotfire’s big data analytics capabilities.
- Download a copy of the Aberdeen In-Memory Big Data whitepaper here.
Linda Rosencrance
Spotfire Blogging Team
