Increasingly, federal agencies are tapping the power of big data analytics to better perform a myriad of tasks, including predicting killer tornadoes, thwarting terrorist operations and disrupting the networks in Iraq and Afghanistan that build improvised explosive devices (IEDs).
As most of the East Coast hunkers down and braces for Hurricane Sandy, what better time to tell you how federal data scientists are using data analysis to track weather events, among other things?
For example, the National Weather Service is combining big data streaming in from its satellites with automated weather services to create models that are aimed at identifying potential tornadoes quickly enough so that the public can be warned.
“It used to be that weather reports came from weather stations,” says Greg Carbin, the warning coordination meteorologist at the NWS Storm Prediction Center in Norman, Oklahoma. “Now, there are all sorts of automated services. Computers can digest the data, and they can make the models look like the real atmosphere.”
The agency’s weather forecasting approach could be a model for other federal agencies that are developing their own prediction models, Carbin notes.
For example, he suggests that the Federal Reserve Board could use the method for analyzing long-term economic data. The NWS model is sensitive to external data sources like fluctuating economic news, he adds.
“What’s interesting about economic data is it shares the dynamic of being difficult to predict behavior,” Carbin says. “What you need to look at is what is the common characteristic.”
One needs to look no further than the CIA to see how monumental the influx of big data has been to the national intelligence arm of the government.
In the past, the CIA has been criticized for not sharing data with other agencies in the wake of events like the 2009 attempt by a Nigerian man to detonate plastic explosives aboard a Northwest Airlines flight. In fact, the overwhelming volume of data flowing into the agency has been blamed for the security mishap. That’s according to Air Force Gen. Michael Hayden, former director of the CIA and the National Security Agency.
“The issue [with the Northwest Airlines incident] was we couldn’t deal with the volume of information we were sharing,” Hayden says. “There is so much data coming at us, we are now challenged by the volume of what’s coming at us.”
Ira “Gus” Hunt, the CIA’s chief technology officer, has said publicly that the CIA is using big data tools and data analysis to slash the time it takes to process information. Two years ago it took the CIA 63 days to process collected data; the agency now can do this in 27 minutes, according to Hunt.
For its part, the U.S. Department of Defense’s Joint Improvised Explosive Device Defeat Organization (JIEDDO) is using big data tools like data analytics and data mining to connect data from multiple sources to create counter measures for dealing with IEDs, disrupt the networks that build them and better train troops on the ground to deal with the deadly devices, says James Craft, the organization’s chief information officer.
Before the creation of the organization, information related to IEDs was scattered throughout multiple Defense Department systems. Now, the organization is a central hub for gathering and analyzing data that’s related to IEDs so that personnel can be provided real-time data and trained using the latest and most accurate data, Craft adds.
JIEDDO uses data mining and social networking aimed at supporting new, innovative ways to carry out its mission, Craft notes. The agency also uses various intelligence systems to gather feedback from warfighters on the ground.
“If there’s a collaborative environment that adds to the mission and we can leverage it, then we’ll look into it,” he says.
