Big Bigger Biggest Data in Oil & Gas

For decades the oil & gas industries have used advanced analytics to attempt to understand the earth’s subsurface. And now with the advent of ‘Big Data’ and the ‘Internet of Things’ the Oil & Gas Industries are primed to make revolutionary advances in how they use data to manage and grow their business.

The first wave of ‘Big Data’ was more or less about people communicating with other people—email, chat, social media, video, etc. This next wave is really driven by much of the work being done in oil & gas will be a tsunami in comparison, as it’s driven largely by machines talking to other machines. These machines (or “ Things” in IoT parlance) are instrumented with sensors that collect data about the physical ‘thing’ and communicate that data over a network to a central point for analysis.

Sensors in oil & gas create some of the biggest data sets we will see over the next decade. We work with a number of large energy companies to help them stand up Apache Hadoop-based enterprise data hubs (EDH) that are perfectly suited to receive this sensor data, process the large volumes efficiently, and serve key insights from the data back to plant and field applications. This helps our customers improve efficiencies, ensure safety, and deliver better products.

One customer in particular is looking to provide analytic insights and correlations around well performance based on subsurface datasets. They’re seeking not just to capture and process this data, but also to unify it in a single system (it had previously been spread across data silos) and make it accessible to more TIBCO users for analysis. This organization is using this data to test well production based on certain environmental factors (location, rock properties, subsurface characteristics) to determine why one well or cluster of wells may perform better than another. This information will enable them to predict future success (or failure) and plan accordingly.

Anything physical can be instrumented with sensors. Sensors that measure the earth’s reflected seismic waves along with pressure, rotation, and flow tracking can be used to understand the properties of the earth’s subsurface to help interpret the likely presence of petroleum reservoirs.

Sensors that measure physical properties such as temperature, movement, density, and rotation can be used to understand equipment health and predict maintenance cycles. Data from sensors that measure gas, depth, and stability can be used to improve field safety.

Using an enterprise data hub to store, analyze, and serve insights gleaned from this sensor data, oil & gas companies can directly integrate data from the physical world into their computing and analysis systems. Detailed data about physical attributes of the earth and functional attributes of the large equipment used to explore the earth data can be captured in near real time, stored at very low costs, and processed quickly to help those businesses improve plant and field performance.

To learn more about Cloudera in Energy, visit

Credits: Cloudera is a technology partner of TIBCO Spotfire and a Gold sponsor at the TIBCO Energy Forum being held in Houston, TX from September 1- 2, at the Norris Conference Center.