Having only been with iDelta for a few months, I wasn’t sure what to expect when I found out I was to attend this year’s Splunk .conf event. Yes, I was aware it was a showcase of the bleeding-edge capabilities of Splunk and a chance to hear about the current vision for Splunk but I wasn’t prepared for the sheer immensity of the three days that transpired.
The event took place at the Venetian in Las Vegas – a venue that I think would be capable of housing more than the 11,000 Splunkers who made the trip to the city. Proverbially, what happens in Vegas, stays in Vegas but heeding this would make for a very short article and there is so much to tell.
As a relative Splunk newbie I had to be judicious in choosing which sessions to attend – I had to make sure that I went to sessions that were accessible but yet would give me some new knowledge/insight into how Splunk works and how I might apply that in my day-to-day work. One session that ticked all of the boxes was Eurus Kim’s and Amir Malekpour’s “The Two Most Common Machine Learning Solutions Everyone Needs to Know”.
This was an excellent session which struck a good balance between describing the technical detail sitting behind Splunk’s Machine Learning Tool Kit (MLTK) (which appealed to my inner maths nerd) and emphasising what the use-cases are (which ensured it stayed relevant and didn’t end up being simply an academic lecture).
The Two Most Common Machine Learning Solutions Everyone Needs to Know turn out to be:
- Using DensityFunction
- Forecasting using StateSpaceForecast
These are useful for when looking to perform anomaly detection. The density function maps different outcomes to their likelihood of occurrence. Splunk’s Machine Learning toolkit is able to take data and fit this to the best-fitting of its collection of distributions. The density function can then be used to look at any new data to determine its likelihood – anything lying outwith a certain tolerance when compared to the density function would be flagged as being an outlier.
The key thing that the presenters were keen to stress was that Forecast and Prediction aren’t coterminous. They explained the difference as follows:
- A forecast is the determination of the value of a particular metric at some future time point given what the values of that metric were for timepoints in the past.
- A prediction is the determination of the value of a particular field, given the values of other related fields – many other inputs are required to help explain the relationship between those inputs.
The speakers perfectly encapsulated forecasting with the following quote:
Getting a picture of what the future might look like.
Hitherto, my day-to-day work has been concerned with reporting stats relating to one of our customer’s implementation of Open Banking. In the course of this work I have become increasingly aware of just how rich their data is and how this could be used to turn them from being “reactive” to “proactive”. Financial institutions face myriad challenges from different directions (cyber attacks, fraudsters, competitor propositions) and are increasingly needing to engage with their data in order to find ways of anticipating threats and/or finding an edge in a competitive market.
Splunk’s MLTK can give our customers these things and whilst we will never be able to predict anything with 100% accuracy (after all, we don’t live in a Laplacian, deterministic universe), machine learning can give firms a glimpse into the financial future to allow them to make the best decisions for their customers.
The conference was such an eye opener for me in terms of the scale of operations and the number of opportunities for Splunk et al to add real value in an increasingly data-driven world.
Session Link: https://conf.splunk.com/files/2019/recordings/FN1213.mp4
Session Slides: https://conf.splunk.com/files/2019/slides/FN1213.pdf
Speakers: Eurus Kim, Amir Malekpour