For those who haven’t noticed, big data has jumped the shark. When it comes to data processing and analytics, the name of the game is Fast Data. And based on early use cases, Fast Data best practices are starting to emerge, according to an article in Enterprise Apps Today.
At the In-Memory Computing Summit in June, vendors and users indicated that because flash memory is relatively inexpensive, it’s driving databases and applications toward exploiting Fast Data, i.e., mobile and sensor cloud data, “using systems whose storage is predominantly or even entirely composed of main and flash memory,” says Wayne Kernochan, author of the article.
One presenter at the event points to a use case that employed one terabyte of main memory and one petabyte of flash, Kernochan notes.
Enterprises are beginning to understand that handling the massive amounts of data flowing into their businesses from cars, cell phones, GPS, etc., in almost real time is the future of analytics as well as of the operational systems that handle the Internet of Things, he says.
“It gives each customer product ‘thing’ in the Internet of Things adaptability as well as intelligence,” Kernochan says. “On the analytics side, its real-time monitoring allows more rapid feedback and adjustment in the sales process.”
So as the cost of flash memory comes down, enterprises can handle Fast Data without going broke and databases aimed at taking advantage of “flash-only” are appearing, he says.
Kernochan offers five best practices of flash-memory database architectures:
- Treat Flash as Extension of Main Memory—This is the “flat-memory” approach, Kernochan says. Vendors have to ensure that processors treat flash as a slower version of main memory. And flash modules should provide new interfaces best suited for random instead of disk accesses. “The user should look for vendors who do this best, and design fast-data-using apps to view all storage as flat and equally available,” he says.
- Implement Variable Tiering for Flash—Kernochan says this means that at times flash should be used for storage, and other times it should be used for processing, depending on the needs of the apps.
- Understand Tradeoff between Data Correctness and Speed to Process—The vendors’ abilities to “write to storage” without losing data, as well as optimize the speed of processing at the risk of some “data incorrectness” will vary.
- Mesh New Database Architecture with Existing Big Data Architectures—Attendees at the In-Memory Computing Summit agree that the new database architectures are unable to handle the scope of today’s big data analytics—or, to put it another way, “big data can’t do fast data’s job, and vice versa,” Kernochan says. Two approaches to this problem are frameworks and brokers that parcel out application data between operational and analytical databases.
- Accept and Plan for Multiplication of Vendor Databases—Enterprises should understand columnar database technology. “Real-world use cases are already adding NoSQL/Hadoop-based databases and columnar databases to handle the operational and analytical sides of fast data,” he says.Kernochan adds that providers are moving rapidly to offer the basic tech for the new flash-memory Fast Data architectures.
Next Steps:
- Try Spotfire and start discovering meaningful insights in your own data.
- Subscribe to our blog to stay up to date on the latest insights and trends in Fast Data and Big Data analytics.
