Recently, I learned a valuable lesson from what appeared as though it would be a regular Monday. My day started off routinely, but along the way some surprising events unfurled.
I was scheduled to go on-site with a federal customer for a “knowledge transfer” (aka OJT) as a new NOC/SOC team was coming online. When I got there, it started out as a rather typical meeting—get an understanding of the team’s knowledge of LogRhythm, assess their goals, and introduce them to any new features/products available.
Prior research led me to believe that this was an older deployment that had been neglected for some time. I was pleasantly surprised to find out that this was not the case. Rather, it was a rather new XM-6350 running LogRhythm 6.3.3 and the latest KB. Not having to perform a system upgrade was a relief.
However, as I listened to the customers’ requirements and dug into the deployment deeper, I quickly realized that the system was only being used for log collection, and rarely were people logging in or monitoring the data being collected by the XM. I’d say they were using about 30% of the LogRhythm features and functionality. Moreover, the WebUI wasn’t installed and AIE was disabled.
After a quick install of the WebUI, initialization of AIE, addition of the basic LOW/LOW-LOW rules found in the Post-Install Guide for POCs and setting up the third-party threat feeds (as well as some other customer requested tweaks)—the XM was back in fighting shape!
After a brief conversation, my sales rep and I started to walk the team through a quick demo using the customers XM, Data and WebUI. Within 10 minutes of talking, we happened to click on the Alarms tab, and to our surprise, we found some interesting data.
A few red alarm cards with ratings of 90 appeared denoting “Malware Found.” The team asked “What’s that all about?” We began to pivot drilldown and discovered the systems affected by the malware. Apparently this information was being reported by their Symantec and McAfee systems for quite some time and on the regular.
The NOC/SOC team quickly sprang into action to remedy the issue (albeit they were a bit annoyed). Moments later, we returned to the demo and another alarm fired with a score of 97. The team (almost in unison) said “What now!?”
After a quick drilldown, we discovered that their Fortinet deployment was reporting a breach coming from an “external IP” (outside the U.S.). Being that this was a government customer, we’d just tripped DEFCON-1 (and were called into the manager’s offices to explain the situation). The customer thanked us for helping to spot the external breach and asked us to leave for the day, as they were about to get very busy.
What was the lesson learned? Make sure that you always look under the hood of an existing LogRhythm deployment to verify that the basics have been covered. It’s important to understand what LogRhythm is capable of and how to best leverage the system to your advantage.
As one of the NOC/SOC employees put it, “If we’d been paying attention and using LogRhythm more regularly, we could have caught this sooner. Who knows how long we’ve been under attack.”
A typical Monday, indeed…