Many of the conversations about Big Data focus on information flowing into the organization from somewhere else. It isn’t as widely discussed that there is an enormous amount of information coming out of every enterprise’s IT infrastructure that is just as critical, offers enormous insights and is just as time sensitive. Big Data isn’t complete without log data.
What is Log Data?
Log data, effectively, is like non-stop tweets coming from IT assets and is generated by almost every element within an enterprise’s infrastructure. By managing this data proactively instead of just when something goes wrong, organizations mitigate risk, ensure service availability and promote operational efficiency.
This data provides an immutable fingerprint of user and system activity that can be, at the lowest level, a failed logon and at the higher levels, a significant diversion from baselines, runaway application or an actual security breach. Logs leave behind a track that can be followed to answer questions like: “Who did what and when?”; “Are we following regulations?”; “Is our network performing optimally?” and “Is our data safe and secure?” These are all critical to business operations and can bring down an organization that isn’t paying attention.
Getting specific, log data gives us a view into:
- Threat management – Logs contain the evidence of an security event but also provide information before and after an attack begins that can be used to head off the problem as it happens.
- Regulatory compliance controls – Log data contains the evidence that supports PCI DSS, HIPAA, SOX, ISO and other audits by demonstrating internal and external policy adherence. Dashboards contain data that shows where and when compliance is being met, and allows the organization to put effort where it is needed to shore up requirements.
- Cloud auditing – Cloud computing is getting more complex and finding more uses. This kind of growth needs to be monitored and managed to ensure everything works as advertised.
- Technology utilization and performance – Operational performance monitoring is key to getting the most out of enterprise assets.
Truly Big Data
Log data is managed physically, virtually or in the cloud, and is ENORMOUS. According to Gartner, a medium-sized enterprise creates 20,000 messages per second of operational data in activity logs. In a single, eight-hour day this comes to 500 million messages, adding up to more than 150 GB of operational data. Without automation technology, collecting, moving and analyzing that data is impossible. There has to be a big filter for this big data that can sort through and pass key events to other systems to be used to manage opportunities, threats and efficiency in the best ways possible.
The equation becomes 1 + 1 = 3 when log data can be blended in real time with loyalty, supply chain, marketing, ERP, social and click stream information. If you’re not managing log data, what’s hiding in your logs?
