Today LogRhythm officially released our Advanced Intelligence (AI) Engine – a fully integrated extension of our core solution that performs advanced correlation and pattern recognition.  And while the concept isn’t new, we’re pretty sure that once you take a look, you’ll see that the execution is pretty groundbreaking. 

So what is advanced correlation and pattern recognition?  In practical terms, it’s being able to automatically identify a sequence of events and recognize a relevant pattern of behavior that will have some sort of impact on another event that will happen as a result.  If this were to take place in your brain, it might happen something like this…  You walk in the house from the garage and shut the door behind you.  Once that’s done, you might expect to see moonlight coming through the next doorway as you make your way up the stairs.  So logically if it’s too dark something is wrong.  You should automatically realize that the absence of light once you shut the door behind you means that the next door is shut.  Typically you would register the correlation between the two and you would know to put your hand out to open the next door.  Unless you, like me, are on autopilot by the time you get home from work.  Because there are better ways to make that connection than with your forehead.

Unlike your brain, an IT environment is not typically capable of recognizing an important sequence on its own.  That’s why solutions exist to analyze event data and let you know when something has or is about to happen that may cause you pain.  From a security standpoint that might be five failed login attempt from a single unknown IP followed by a successful login indicating that you may have suffered an external breach.

Tools for doing this have been around for a while, but with limitations.  One of them is that they are typically restricted to analyzing a subset of data that has already filtered out significant information before it ever gets to the correlation engine.  This narrows the scope of coverage to specific security-oriented use cases and takes away the flexibility to cast a wider net for event sequences that may not be as well defined.  AI Engine has removed this limitation by allowing advanced correlation rules to be run against all log data.  By doing so we have expanded the scope of what you can do with advanced correlation and pattern recognition to extend well beyond the standard security use cases.  An operations example would be when you have a critical process that may start and stop on a regular basis.  Operationally this is standard behavior.  You don’t need to know every time the process stops or you’ll quickly start ignoring the alerts.  But you do need to know when it doesn’t start back up within a certain time frame.

Perhaps the biggest drawback of most advanced correlation tools is their complexity.  They may work well within the confines of their preconfigured rules, but adaptability and usability are limited.  They’re kind of like a remote control that handles a few functions and operates your television.  Your options for adding new features to the remote are limited.  If you have the tools, the time and the knowledge, you can take the remote apart and rebuild it to handle your new blue-ray player.  Practically speaking you will probably be adding a second remote to your collection or you’ll be waiting for a remote that someone else builds that is capable of working with your tv and blue-ray.  Add a receiver for a new sound system and you have a third remote.  Now this may not be that bad at home, but imagine you have thousands of heterogeneous devices that you need to control, and limited time to do so.  What you really need is one universal remote that handles all of your devices and can be easily programmed to handle what you have now and what you will add in the future.  And as new features are added, the remote has to adapt to those as well.

It’s the same thing with pattern recognition.  Sure there are specific behavior patterns that you know you have to identify because they are similar for all environments, but over time, that behavior may change.  And there are things that you may want to watch for that are less defined but no less important to detect.  For those things you need the ability to cast a wider net and once you are able to detect and understand those behavior patterns that are more general, you also need to be able to quickly put a rule in place that will narrow in on specific activities.  Without a usable interface, advanced correlation tools lack the flexibility to adapt to what you need in your environment or to keep up with new behavior patterns that are critical to security and operations.

AI Engine is accessed through LogRhythm’s console, with the same consistent look and feel inherent to all LogRhythm tools.  A wizard-based interface with a drag-and-drop GUI for defining advanced correlation makes creating and customizing even complex rules simple to learn and quick to execute.  It also correlates against all log data – not just a pre-filtered subset of security events.  AI Engine analyzes over 50 different metadata fields and many more sub-fields that provide highly relevant data for analysis and correlation.  The metadata fields map to system, network and application information extracted from the logs themselves, but they also include context that is derived from the log information such as direction, impacted entities, the city from which activity originated and more.  The extensive metadata from which advanced correlation rules and patterns can be defined, combined with the entirety of all log data against which these rules can be applied, offers unprecedented visibility and context to threats and operational issues that have been blind spots for many organizations until now.    At the same time, AI Engine can easily be used to cast a wide net with more general correlation rules, ensuring that significant incidents are captured despite changes in event behavior.  Sure AI Engine comes with over 100 rules ready to go out-of-the-box covering a wide range of common use cases, both general and tightly focused.  But we’ve also designed it to work for you.

If you want to know more about our AI Engine, we’ll be happy to show you how it works.  Just let us know.  Or check out Chris Petersen’s video in which he demonstrates AI Engine.  You can hear the entire LogRhythm story or simply jump to the chapter on AI Engine.   Watch the Demo.