Testing & Evaluating SIEM Systems: A Review of Rapid7 InsightIDR

As companies continue to get breached by the hour, IT and security teams are constantly scrambling their defenses in hopes of eradicating attackers from their networks. The (sort of) good news is that security software and hardware vendors are overflowing with product and service offerings designed to help you. Many of them even promise to keep the bad guys and gals out of your systems 24 hours a day, 7 days a week!

So all you have to do is get out your credit card, install one of these products and get a good night’s sleep, right? Unfortunately, during the penetration tests we do for clients around the country, we find time and time again that these expensive security solutions have considerable blind spots. Many fail to fully detect even the most basic attacks. And security products also need to be fine-tuned for your environment; default settings will not cover all concerns specific to your environment.

In this article, we will show you how to run some of these “hacker low-hanging fruit” attacks to test the effectiveness of your organization’s security services. For the sake of having a real commercial tool to conduct these attacks against, we chose a security solution called InsightIDR from Rapid7 to install in our lab environment.

InsightIDR has SIEM at its foundation and scales out to essentially be an XDR solution covering endpoints, network traffic analysis, UEBA, incident response and more. In our testing we found a number of issues that are common in SIEM systems. The good news is vendors are responsive to these issues – they’d rather hear them from “white hat” folks and customers than the bad guys – and we’ve been in touch with Rapid7 throughout our work. We found Rapid7 to be responsive and straightforward to deal with – a good sign for user support, and critically important in cybersecurity. Rapid7 is working on a number of fixes to our findings – some of which were known issues the company was already aware of – so we’ll update this article in a few months to reflect completion of that work. We’ve also included Rapid7’s comments in a number of places.

This is, to our knowledge, one of the few times that test results have been published on a cybersecurity product of this magnitude, so our goal is largely educational. We encourage you to use this article as a guide to conduct your own in-house testing of security solutions. We’ve focused on a number of detectable “warning shots” that can happen during an attack (or a pentest) that will let you know that bad things are brewing behind the scenes. If one or more of these attacks are not detected, challenge your vendors to write a signature for them. The result will be a stronger product for their entire user base.

We also created a separate Guide to Getting Started with Rapid7 InsightIDR to reflect our impressions on the product’s ease of use and functionality. We found the product easy to install and configure in just a few hours, and immediately started receiving notifications about key security events. Our focus in this article is on common attacks we conduct during penetration tests, and how Rapid7’s InsightIDR system responded.

Testing InsightIDR’s ability to detect attacks

• Active Directory enumeration
• Password spraying
• Kerberoasting
• AS-REP Roasting
• Network traffic poisoning
• Dumping hashes from domain controllers
• Pass the hash (PTH) attacks
• Conclusion

Once the agent was installed on all systems in our test environment, we were ready to simulate actions an attacker or ransomware operator would perform to see what type of security threats are detected. (Note: many of the attacks below are taken from Light Pentest LITE: eBook Edition, which is designed to be a practical, step-by-step playbook for internal network penetration testing.)

Active Directory enumeration

One of the first things we do in a penetration test is try to learn more about the client’s Active Directory environment. Active Directory configurations, especially those that have been in production for a number of years, are often rife with attack paths that are difficult to see unless you take a deeper look under the hood.

One such tool we use for Active Directory enumeration is SharpHound, which vacuums up information about all the objects in the directory and how they relate to each other from a security standpoint. SharpHound has many different configuration settings you can tweak to make the data collection stealthier, but we usually just run it “wide open” like so:

sharphound.exe -d pwn.town -c all

The -d pwn.town specifies our domain, and -c all tells SharpHound to collect all data about the Active Directory environment.

InsightIDR did not detect SharpHound being run. This is not surprising, however, as we have yet to see a security solution detect it.

The good news is InsightIDR will soon be able to detect this attack. Rapid7 told us, “We are working on a detection for Active Directory enumeration using SharpHound that we think is possible using Windows log events. This will likely be available by the end of the year.”

Password spraying

Early in a penetration test, we want to see if we can get access to other Active Directory user accounts. A popular and semi-stealthy way to do this is via password spraying, where we attempt to log into each account one time with a password we know people are bound to use. From our experience, we know that people absolutely love to use season + year combinations. So we downloaded Rubeus and attempted to log into each user account using the following syntax:

rubeus.exe spray /password:Spring2022! /outfile:pwned.txt

The spray flag tells Rubeus we are doing a password spray, the /password:Spring2022! specifies the password we want to spray with, and /outfile:pwned.txt saves and valid credentials to a text file called pwned.txt.

InsightIDR did not detect our password spray attempts. What we find is that security solutions are very good about notifying you when an account gets locked out, but in this case we are only attempting to log into each account one time, so the accounts stay active.

Rapid7 told us a honeypot we didn’t configure would have helped. The company’s response: “For password spraying we have 2 detections, the first is the honey user that you didn’t configure during the test and the second is a brute force detection that looks for a burst of failed authentications from a single host. Our threshold is at least 100 different users (our smallest customers have ~500 users). We are reevaluating that threshold.”

Kerberoasting

Kerberoasting is one of our favorite attacks. The technical explanation of it can get a bit dense, so we recommend checking out this article from Black Hills Information Security for a deeper dive. However, when we talk about Kerberoasting, it is usually in a room full of managers and executives who just want a plain English explanation of what the attack is, and why they should care. So we will say something like this:

“Essentially, any Active Directory account can say ‘Hey Active Directory, could you please give me the hashes for any service accounts associated with things like IIS and SQL, for example?’ And the Active Directory environment happily answers, ‘No problem, Bri-guy, HERE YOU GO!'”

This attack makes more sense when you see it in action. To check if an environment is vulnerable to the Kerberoasting attack, we can use Rubeus again with the following syntax:

rubeus.exe kerberoast

As we can see here, the Ray user account in our lab is susceptible to Kerberoasting. So what we can do is bring over the account hash into a high-powered password-cracking rig and potentially figure out the plain text password. In production environments, the Kerberoastable accounts we see are frequently part of the Domain Admins group, and the passwords are not rotated frequently. That means if we crack those accounts, we might have complete control of the domain before morning snack time!

InsightIDR did not detect the Kerberoasting attempt. That is not entirely surprising, though, as we have only seen a few security solutions generate an alert for this specific attack. However, solutions such as Blumira have built a detection for it, and even make their code for a Kerberoast honey credential available to the public.

Rapid7 response: “Several of our customers are concerned about kerbroasting and we are actively working on a detection for this sort of activity that we expect to have live by the end of the summer. Our major concern right now is doing it in a way that limits the false positives.”

AS-REP Roasting

Similar to Kerberoasting, AS-REP Roasting is something any Active Directory account can do to gain access to account password hashes. There is a very good article from Stealthbits that covers the technical nuts and bolts of this attack. But in a room mixed with tech and non-techie folks, we explain the attack as follows:

With the AS-REP Roasting attack, any user in Active Directory can basically say, “Hey Active Directory, if any user accounts are set to not require Kerberos preauthentication, let me have a bit of encrypted data about that user that I can bring offline and crack!”

To find vulnerable users in Active Directory, you can use the following PowerShell command:

Get-ADUser -Filter {DoesNotRequirePreAuth -eq $True} -Properties DoesNotRequirePreAuth

For any users returned from this command, you can open them up in the Active Directory Users and Computers tool and click the Account tab to see the vulnerable configuration:

Since we want to see the vulnerable users and their hashes, we will use Rubeus once again:

rubeus.exe asreproast

InsightIDR did not detect the AS-REP Roast attempt. Much like Kerberoasting, we have seen very few security solutions alert on this attack. Again, users can likely expect a detection from Rapid7 in the near future.

Network traffic poisoning

If we are unsuccessful in making any progress with password spraying, Kerberoasting and AS-REP Roasting, we will use a variety of tools to poison certain vulnerable network traffic protocols – such as NBT-NS (NetBIOS Name Service), LLMNR (Link-Local Multicast Name Resolution) and mDNS (multicast DNS).

To help visualize this attack, imagine a scenario where user Sally opens up Windows Explorer and tries to browse to the PT-APP01 server, but she fat-fingers the server name by putting in an extra zero and types \\pt-app001. In a few moments, she will get a pop-up error saying “Windows cannot access \\pt-app001.” No big deal, right? Well actually, in the background, something much more sinister could be going on!

If we as attackers are running network poisoning tools such as Inveigh, we can actually listen for Sally to make typos like this, and then intercept them. Check out this visual to get a feel for what this looks like:

Essentially, Sally’s machine asks the DNS server where to find \\pt-app001, and when the DNS server cannot find it, Sally’s machine does some shout-outs to the rest of the network using insecure protocols (NBT-NS, LLMNR and mDNS). When that happens – BOOM! – we trick her machine into sending us Sally’s username and password hash.

To start this attack on network traffic, we can use Inveigh:

inveigh.exe -nbns y -llmnr y -mdns y

This syntax tells Inveigh to fire up and poison the insecure NBT-NS, LLMNR and mDNS traffic.

After a while, hopefully we will see entries in the Inveigh log that look like this:

We can now take the hash for user Beverly and try to crack it.

InsightIDR did not detect the network poisoning, but because InsightIDR is an endpoint-based tool, we did not expect it to. However, if your security solutions are looking at all network traffic in addition to endpoint activity, we recommend running Inveigh to see if it generates alerts. Alternatively, there is an awesome free tool you can run to detect network poisoning attacks called CanaryPi.

The good news is here too a solution is in the works, per Rapid7: “We do have a detection for this which we have tested on a tool called Responder in the past and it has fired. We did try it with the tool you used (inveigh) and could not get it to fire. We are investigating why but will get it working soon.”

Dumping hashes from domain controllers

If we are able to capture and crack credentials from a member of the Domain Admins group, the next thing we will do is extract all the usernames and password hashes from Active Directory. The mimikatz tool works perfect for this:

lsadump::dcsync /domain:pwn.town /all /csv

This syntax tells Mimikatz to spit out a list of all pwn.town domain users and their hashes in a clean CSV format:

In our opinion, this specific attack is the worst thing that can happen to your Active Directory environment. Why? Well, if the attacker has this information and is still inside of your network, they can conduct pass-the-hash (PTH) attacks to take action in the Active Directory while impersonating any user they want! And even if you break the foothold that attackers have in your environment, they may be able to crack some of those hashes and then get right back into your network via email, VPN, VDI, etc.

InsightIDR did detect the presence of the mimikatz.exe on endpoints that had the InsightIDR agent on them. However, when we dumped the Active Directory hashes from a non-monitored endpoint using impacket, InsightIDR did not throw an alert. Traditionally, we will see security solutions alert on the hash dumping action. And while that is good news, it is also kind of frustrating because basically the solution detected the attacker’s Active Directory “finishing move.”

Pass the hash (PTH) attacks

Armed with the hashes we dumped out of Active Directory, we can use a tool like CrackMapExec to “pass” the hash around the network to other systems:

cme smb 10.0.7.0/24 -u brian -H PASSWORD-HASH-FOR-BRIAN

In this syntax, cme calls CrackMapExec, smb specifies the protocol to use, 10.0.7.0/24 specifies the subnet of systems we are going to “spray” the hash to, -u brian specifies the Brian user, and the -H PASSWORD-HASH-FOR-BRIAN is Brian’s password hash:

As you can see, we have “sprayed” this hash to the entire 10.0.7.0/24 subnet and found that not only is the user/hash combination valid, but it is a high privilege account on many systems (indicated by Pwn3d!).

InsightIDR did not detect the pass-the-hash attacks. In general, we are seeing corporate endpoint detection and response (EDR) endpoint protection tools throw alerts when they see pass-the-hash behavior.

Pentesting Your Environment for ‘Warning Shot’ Detection

It is important to note that these kinds of blind spots are common in many SIEMs. In a perfect world, we could wave a magic wand and find a combination of security solutions that would make your network hack-proof. In the meantime, though, we feel there are a lot of detectable “warning shots” that can happen during an attack (or a pentest) that will let you know that bad things are brewing behind the scenes. We encourage you to use this article as a guide to conduct your own in-house testing of security solutions. If one or more of these attacks are not detected, challenge your vendors to write a signature for them. And if you are in the market for a new monitoring and logging solution, check out our SIEMple SIEM questionnaire. It contains a list of pre-sales questions you can ask to better understand what the solution does and does not do, as well as additional technical tests you can run to determine if the solution is effective at detecting and/or stopping threats.

Explore Other Top SIEM Solutions

Read next:

• Getting Started with Rapid7 InsightIDR: A SIEM Tutorial
• Best SIEM Tools & Software