Gartner’s Ian Finley spoke this afternoon at the 2013 Gartner AADI Conference in London. In what was a primer on Big Data, Finley predicted that there will be 4.4 million jobs in Big Data by 2015, but only a third of those positions will be filled due to a lack of resources in the labor market. He used that prediction to emphasize the value of first understanding the business need for a Big Data solution and formulation of solid use cases.
Finley provided examples of common Big Data opportunities that Gartner is seeing:
- Public sector: real-time monitoring of traffic flows, vehicle locations, resource use
- Public safety: Forecasting the extend and impact of natural and man-made disasters as well as surveillance
- Utilities: Understanding individual use patterns and improving demand management
- Retail: Web and e-commerce by analysis of clickstream and customer feedback. Providing faster and more accurate customer responses.
Finley referred to retail as the most popular use case and said that it attracts the greatest number of resources and publicity.
Iterative Experimentation
Finley brought up the need to experiment with technologies and analytics to find the right combination that works for the organization. As a relatively new technology for most companies, working with Big Data will likely be a very iterative project. Part of being successful is being able to recognize when an answer is valuable or not, and for that he recommends being skeptical of outcomes. Finley also said that being successful is very much a team effort that draws in multiple skills sets.
This iterative approach allows “tinkering” to happen at the data level that can be moved into production as it proves useful to the business. By keeping production and Big Data development separate, the flexibility required to find solutions can happen unhindered by the structure and security of business operations. In the same way, Big Data projects can happen in parallel by operating multiple data environments that take advantage of inexpensive storage and computing. Each project may have different data types and sources, meaning that flexibility is a key part of the development environment.
The Data Scientist
Finley touched on the skill sets necessary for Big Data projects. He said that a good data scientist needs to be able to understand the business, data and analytics in equal measure. While the business may create requirements, Finley said that a data scientist works as a consultant to the business while also being comfortable with the IT department and its language.
In wrapping up, Finley talked about NoSQL and unstructured data. he listed the biggest challenge to NoSQL as the need to code features of that we’ve come to expect from relational databases, like access control.
His advice for the near term was to inventory existing and potential sources of data and to review analytics and development capabilities. In the medium term, he recommended identification of ideas from outside one’s industry to drive an agile program that takes a new approach to data. Lastly, Finley recommended refocusing investment and resources away from BI report development and into diagnostic, predictive and prescriptive analytics. And of course, his final point was to update organizational skills to be ready for the Big Data future.
Minding the Talent Gap
Catching up with Ian afterward, he said that organizations will need to cobble together the resources necessary to make up the shortfall in resources. He pointed out that there are over 50 degree programs around the world that are related to the data scientist role. Finley felt that smaller organizations will likely need to contract experts on a temporary basis as the training investment could be outsized.
Of course, underlying the need to have the right talent for Big Data is the need to be able to find, understand and act on the things that matter to the business. A common topic in Big Data discussions is the need to have strong data integration capabilities that feed these solutions.
Come see us at the Gartner AADI TIBCO booth for a continuing discussion of Big Data and integration.
