The foundation of Digital Business is Fast Data—data that’s in motion, emitted by connected products and services like streams of events coming from your car, streams of data about package movements through FedEx or streams of GPS location data from Uber. But there’s one problem: with so much fast-moving information, it’s hard to find critical business moments within the noise, so most firms rely on traditional backwards-looking databases, warehouses, and data lakes stored in Hadoop. So they can’t see—much less control—data while it’s still changing. The ability to act on critical business moments is lost once you start relying on digested and summarized data.
For example, in 2012, one firm—Knight Capital—lost $460M in 40 minutes because so much decision-making on Wall Street is automated. Wall Street got a much-needed wake-up call. That’s when TIBCO created a strategic alliance with a world-leading Wall Street firm who asked us to build a product that turns traditional data warehouses upside down. The mission was simple: build a platform that allows us to query the now and query the future.
The result of that alliance is now called TIBCO Live Datamart. A live data mart is like a data warehouse, but connects directly to streaming data while it’s still moving: sensors, GPS, and mobile data can all be queried as it flows by, and queries can be posed about future conditions that business users want to become aware of immediately.
The magic of Live Datamart is its innovative continuous query processor, or CQP. Let’s decompose how the CQP processes data as it’s moving, and allows business users to query for key conditions in the future.
01: Query streams. Let’s say you’re running an oil-well operation. You have 1,000 wells with sensors attached that emit thousands of readings a second—engine temperature, vibration, current, intake pressure, oil flow, and so on. This sensor data is brought directly into the Live Datamart, which maintains in-memory “tables” of streams that can be queried.
To end users, Live Datamart looks like any other database. They can pose questions like: “Show me all wells that are showing signs of failure for more than 30 seconds.” The difference is that the result set they get back is based on sensor readings that are just milliseconds old.
Moreover, this query is a bit oversimplified—an example of a query for a “sign of failure” on a pump can actually be a complex streaming analytic query like, when the current increases by more than 25%, voltage decreases by 25%, and flow also decreases by more than 10%, this might be a sign that the pump has a short, and is about to fail. Live Datamart maintains streaming analytics on the stream by using complex event processing techniques under the covers.
These real-time computations are done in the TIBCO Live Datamart; staff members in drilling operations centers simply compose these questions, like they would use any analytics tool, and submit them to Live Datamart.
In our example, at the start of the day, this query might return a result that’s empty—all’s quiet. So of our 1,000 oil wells, our query returns six wells (A-F) with mild error readings, with an error rate more than five, but less than 30.
02: Query the future. As if querying streams isn’t cool enough, here’s where the real magic begins: Live Datamart remembers each end-user query—like an elephant. It never forgets. Drilling operators can submit every interesting question they can think of during the day and Live Datamart remembers them for the future.
The update phase of a continuous query keeps updating individual values as they change. So as the day’s drilling activity ramps up, we see some pumps running hot. The continuous query model pushes the new matches to the query to the user’s workstation, mobile phone or tablet. In our example, any users watching this query will see that oil well C’s error rate just hit 100. This is a warning sign.
03: Future matching rows are pushed to the query result set as new wells become stressed in the future, according to our real-time analytic—the CQP adds new rows to the result set in real time without the user having to re-ask the question. In our example, well G is now showing signs of increased error rates and the user’s screen automatically receives this new row in its graphical view.
04: As time proceeds, and live readings make the row fail to match the query, results are removed automatically. Over time, in our example, some wells return to their normal state, while C and G are still struggling. The CQP continuously updates the result set, which allows our operator to filter out the noise and really focus on these problem areas. Well C’s error rate is easing up a bit, so perhaps our action from before is starting to work.
05: Like a database, we can sort, filter, aggregate, graph, and do math against streaming data. Just like a database, users can drill down and analyze information. But with the Live Datamart, that data is live. For example, Live Datamart can allows users to query the CQP for a streaming weather forecast feed, to examine the current and forecasted weather conditions and explore how they might be impacting our error rates from the wells in that region.
06: Live (or historical) drill down. Users can click on each well and look at their real-time sensor readings to investigate further based on real-time readings over the last three hours, take action by sending an alert to the local team for predictive maintenance or schedule a work flow to investigate the well. Users can also choose to display historical data as a way to compare current conditions to historically normal readings.
So, TIBCO Live Datamart allows users to ask every interesting question about now and the future, and analyze and provide a command and control center for operational users to act on those questions in real time.
Here’s what Live Datamart looks like to an end user. The desktop is the end-user command and control center for business users where you can compose continuous queries, submit them to the Live Datamart, analyze the continuous results, and interactively act to explore real-time conditions. For example, here we see a desktop with five windows, which map to five continuous queries. Query results can be displayed on a map, in a table, as a series of alerts, in a heat map or conditionally formatted with colors allowing big problems to grab the attention of the user.
TIBCO Live Datamart was born on Wall Street, in 2015, it’s now available on Main Street.
Live Datamart is being used to query the future in logistics operations to answer predictive queries in real time, like “Tell me whenever a package is stuck or delayed against its scheduled deliver time by more than five minutes.”
Live Datamart can be used to monitor networks, alert network operations of errors, and identify which parts of the network is having problems right now.
Of course, Live Datamart is used all over Wall Street, allowing firms to be alerted in real time about orders that are stuck, not working well, too risky or out of compliance.
In transportation, Live Datamart can monitor the entire transportation system in real time and continuously correlate streaming weather, traffic, and GPS data to keep buses and trains on schedule, and make adjustments.
This is the new era of Digital Business and the new era of Live Datamart: as IT continues to be literally a part of your product or service, you simply can’t afford to fly blind in real time. For the Digital Business of the future, you must monitor real-time operations effectively, seize opportunities, avoid risk, and increase revenues now, instead of waiting for your end-of-day report that only tells you the opportunities you missed yesterday.
TIBCO Live Datamart: continuous analytics for the Digital Business of tomorrow. Check out this video to learn more.
