I frequently discuss big data with executives from some of the largest Fortune 500 companies in the US. I continually act as a sounding board for their frustration as they try to extract value from historical big data. There is a prevalent misbelief among them that there is a great deal of value in years and countless rows of data, but they struggle to monetize this hidden value, and for good reason.
Their concern is valid. Investing millions on clusters of servers with little return on investment is simply not acceptable in these demanding financial times, nor should it ever be. The adage that a data warehouse can just as easily be called a data cemetery is valid in many cases, even for the largest corporations in the world.
Put Data in Gear
A relatable analogy is to explain data analysis in terms of your driving habits. Your brain is an excellent processor of patterns of historical data. As you drive to work each morning, you process the drive time depending on if you have left your home at 7:00am or 7:10am. You know that if you leave your house just 10 minutes later than usual, the drive time will increase 25 minutes. Your brain has also processed that when there is light rain, you need to add five minutes to your drive time, and heavy rain requires 10 minutes. If it is snowing, you may have to add 15 minutes to your commute.
In addition to weather, you have found out that when there is an accident in different parts of the city, it can affect the amount of delay on your trip. In fact, depending on the location of the accident, you may take a different route to work. You receive this real-time stimuli by watching the morning news during your breakfast.
This is similar to using big data to influence real-time actions. Your mind is the big data processor in this analogy. It has pre-processed different results based on many different situations that you have encountered over the years. In this situation, it has calculated the drive time based on different weather and traffic conditions. You have then selected the appropriate pattern based on your real-time observation of those conditions and made an appropriate decision. Combining historical, big data at-rest with various real-time stimuli allows patterns of events to be discovered and to respond intelligently to current issues.
Take Your Data Shopping
If we extend this situation to a sporting goods retailer, we can use big-data processing combined with a data analytics tool on years of buying history to discover that the top five items customers that bought a tent also purchased:
- A lantern – 88%
- A cook stove – 82%
- Some ropes – 78%
- Sleeping bags – 72%
- A tent repair kit – 64%
However, in only 42% of the purchases did they buy water jugs.
Using complex event processing, the retailer can suggest appropriate items to a purchaser of a tent. If the cart has four of the top five, the retailer can issue a reminder for the fifth item. The retailer doesn’t need to sacrifice margin in the form of a coupon in this case, the customer is pre-disposed to want the item and a simple reminder may be sufficient.
However, in the case of water jugs not being in the cart, the retailer knows there is a likelihood of a need by the customer but a reminder may not be sufficient. In this case, a coupon may be necessary to incentivize the purchase.
There is even more added benefit to analyze the transaction logs of customers receiving those offers. The retailer can now personalize this historical big-data behavior with the specific actions of a unique customer. By correlating the specific buying action of an individual customer with the buying actions of other customers, the retailer can find specific patterns of behavior of similar people across the entire chain of sporting goods stores. By making these correlations and storing them in a real-time memory grid, the retailer can leverage the historical big-data patterns along with the current real-time patterns to create a highly focused Event-Enabled Marketing program.