Splunk looks to help companies dig through data by breaking up the data into smaller chunks and simultaneously running search functions on each. (Zachary Fryer-Biggs)
SAN FRANCISCO ó Accumulating data on attackers isnít difficult for most companies. Within every network log lies the information about any attack that might have occurred. Companies keep the logs, as terabyte after terabyte piles up.
But while gathering the data isnít difficult, finding something meaningful in the data is. Part of the problem lies in the very nature of the process: Thereís a lot of data. Traditional methods for searching through the data, where a search function digs through one large database, can take days for many network administrators saddled with a wealth of logs.
Splunk looks to help companies dig through that data using a method gleaned from Google: breaking up the data into smaller chunks and simultaneously running search functions on each. Think of it in terms of criminals facing jail time. If a person is convicted of 20 counts of a crime and faces five years for each count, that can total 100 years in prison if the jail time is served consecutively. But if the counts are served concurrently, the criminal would be out in five years.
In much the same way, Splunk digs through data concurrently, whittling searches that would otherwise take eons down to minutes or even seconds.
The company provides an interface to ease the access to the information. But the underlying approach could be used for a wide range of applications, from mining customer data to digging through tax records.
Much of Splunkís federal business is focused on the security application of the software, digging through network logs looking for anomalous activity that might indicate a threat.
ďOne thing that they canít hide is that they do not act as a normal employee,Ē said Joe Goldberg, a senior manager for the company.
On top of the underlying search functionality, the company has a site that includes 400 applications that leverage the capability to display different subsets of the data.
Cyber red teams, the groups that penetrate friendly networks to detect vulnerabilities and then report them to improve security, are using Splunkís software as well. Red teams that are working for the intelligence community have been using the software to review information inside of networks that they penetrate, a company executive said.
Splunk isnít the only company to offer solutions to the increasingly prevalent problem of big data, but it is a company that is widely partnered with those in the cybersecurity industry. Several large security firms use the system to help them review data.
But Splunk is not alone. An expert who has used the software pointed to open source upstarts that might disrupt its market share in time.