Part 2 of a 2-part story detailing the real-life forensics effort that got to the bottom of a break-in at a major IT infrastructure player. Part 1 was posted earlier. Registered users were able to access the complete version earlier than unregistered users. Click here to register and see exclusive content, and to sign up for the SecurityGram newsletter.

Day 2: Developing an Action Plan

Upon completion of the initial incident forensics, BlueLeaf management expressed concern that the attacks may be a sign of corporate espionage. Management feared that the timing of the extranet compromise, which occurred at the same time as the acquisition talks, was more than a mere coincidence. Riptech believed that espionage was highly unlikely-typically compromised systems that are used by an attacker as a launching point for additional intrusions are indicative of a "random" attack. Despite Riptech's recommendation, the fear of corporate espionage led BlueLeaf to make the decision that further investigation was warranted in this case.

In order to determine the motive driving the attack, Riptech first employed a "fishbowl" tactic, which allows Riptech to monitor the attacker's activities without being observed. In order to limit further damage to BlueLeaf's internal and external systems, the compromised system was left on-line, but moved to an isolated network segment, thus minimizing the negative effects of the intruder's packet sniffer. Prior to this move, the server resided in a prime location, thus allowing the intruder to sniff all inbound and outbound connections into the corporate network.

While the intruder's network scanner was left running, the corporate firewall was modified to block outgoing traffic in order to limit BlueLeaf's potential downstream liability. While case law has not yet clarified the legal issues surrounding downstream liability, BlueLeaf's legal counsel was not willing to risk the legal liability involved in knowingly allowing the intruder to harm non-BlueLeaf systems.

Finally, Riptech provided a monitoring system to view future intruder activity remotely via an encrypted tunnel. Riptech also formulated an emergency action plan, which was to be engaged if the hacker became more destructive or attempted to gain additional access to BlueLeaf systems. If necessary, Riptech could immediately terminate the intruder's access remotely.

Days 3-5: Monitoring and Tracking the Intruder

Over the next 72 hours, Riptech observed three external sites connect to the compromised BlueLeaf server using the intruder's backdoors. None of these sites had any official business with BlueLeaf.

With BlueLeaf's permission, Riptech contacted the sites to inform them that they were involved in a hacking incident and that they were likely compromised as well. Riptech initiated these calls to the involved ISPs without identifying BlueLeaf as a victim, thus protecting BlueLeaf's identity while working to resolve the incident. Using a non-invasive approach from a remote location, Riptech was able to determine that all three systems had a backdoor similar to the one on BlueLeaf's server. Riptech observed that the compromised system was running the Linux operating system, a consistent trend up to this point in time.

Using the source code for the encrypted backdoor that was previously found on the BlueLeaf system, Riptech decrypted the intruder's connections, which allowed a keystroke-by-keystroke replay of the hacker's activity. This was a key piece of evidence for several reasons. First, it heightened suspicions that this intruder was extremely skilled. Second, it demonstrated clearly that damage was inflicted on the client infrastructure.

Analysis of the intruder's sessions also revealed that immediately after connecting to the system through the backdoor, the intruder checked recent system activity for possible detection. The intruder then immediately transferred the collection of passwords from the system, thoroughly cleaned the activity from the system logs, and exited. The entire session lasted little more than two minutes. The methods and speed of the intruder's search would have impressed even the most skilled Unix systems administrator. Because Riptech had taken the precaution of clearing the system and logs of their own activity, the intruder was completely unaware he had been discovered despite an exhaustive search for signs of detection.

Figure 2:
Anatomy of a forensic effort:
Riptech discoveries in the BlueLeaf case.

Riptech Figure 2

After contacting the three ISPs, only one site, GreenRoot, was willing to cooperate with the Riptech investigation. With permission, Riptech isolated and monitored GreenRoot's compromised system, employing the same methodologies used at BlueLeaf. Riptech investigated GreenRoot's servers cooperatively with GreenRoot system administrators and identified intruder fingerprints and attack tools on one of the servers. After closer analysis, Riptech determined that the tools were very similar to those found in the BlueLeaf system.

In this case, the intruder also left behind an Internet Relay Chat (IRC) client and a hidden web page with the message "KSE falcons rule!!!." Intruders often communicate with peers via IRC both socially and as a source of mentoring. Riptech suspected that the hidden web page referred to an obscure hacker group and was left by the intruder as a trophy.

On both the BlueLeaf and GreenRoot systems, the intruder was meticulous about cleaning system logs to thwart any efforts to track him. However, the intruder made one log cleaning mistake and missed two connections in GreenRoot's web access logs. According to the web acess logs, an IP address from an ISP in Kalamazoo, North Dakota visited the hacker's hidden web pages twice. Further investigation using this passive monitoring system revealed that IP addresses at an ISP in Chicago, IL and a high school in Kalamazoo, North Dakota were using the backdoors into GreenRoot's system.

Riptech contacted the ISP in Chicago, but, unfortunately, it did not log any of its users and claimed no responsibility for the actions of its users, hostile or benign. Open source searching revealed that quite a bit of known hacker activity originated from the Chicago ISP due to its liberal usage policies. This site had classic characteristics of an intruder "jump-point," not the source of the attack.

The connection from the North Dakota high school was more promising. Riptech fingerprinted the high school system, revealing that it was a Windows 98 system. Windows 98 systems are less often used as jump points as are other, more sophisticated operating systems. This find, combined with the two web access logs from the Kalamazoo ISP, indicated the high school system was a possible source rather than a "jump-point".

The IRC server to which the GreenRoot IRC client connected revealed that a hacker with the handle of "mojo" had last used it. Riptech was able to confirm that the hacker handle "mojo" had used the North Dakota ISP in the past. In addition, the "mojo" handle was used to publish several articles in 2600: The Hacker Quarterly. Riptech did more open source searching, this time targeting Kalamazoo, North Dakota. The search revealed a high school in whose mascot was a falcon and whose initials were KSE (Kalamazoo South East).

It's a wrap

These findings were sufficient to convince both Riptech and BlueLeaf that the intruder was most likely a student at Kalamazoo South East High School. With regard to a motive, Riptech's monitoring did not suggest that BlueLeaf's intellectual property was a target for the attack, nor did it reveal a pattern of attacks against BlueLeaf's partners.

With the possibility of corporate espionage ruled out, and no desire to publicly acknowledge that a teenager had compromised their corporate security, BlueLeaf and GreenRoot agreed to cease pursuing the intruder. At the direction of the client, Riptech served as an independent third party to notify all affected organizations of the incident and to provide them with several system recovery recommendations.

Riptech then helped BlueLeaf and GreenRoot recover from the intrusion by assisting with the transitioning of data on the compromised server to a secured system. Riptech also performed a limited security assessment of the BlueLeaf network to confirm proper system and firewall configuration as well as verify that BlueLeaf successfully changed all passwords compromised by the intruder. Riptech continued to monitor the affected systems for one month following the compromise. No signs of further activity from mojo were detected.

As this case study illustrates, tracking an intruder back to point of origin is a complex and time intensive process. The cost of pursuing attackers as well as the potential damage to corporate image deters most companies from pursuing attackers or involving legal authorities. For these reasons, rather than attempting to pursue the intruder, Riptech typically assists clients with recovery from the attack and take steps to minimize the future risk of system compromises.

Security Incident Checklist

Riptech encourages companies to take the following steps if they believe they have been compromised. This checklist was extracted from Steps for Recovering from a UNIX or NT System Compromise co-authored by the Computer Emergency Response Center/Coordination Center (CERT/CC) and the Australian CERT (AUSCERT).

  1. Consult your security policy. A site's security policy should address the proper internal procedures for handling a computer security incident, including whom to contact, whether to involve law enforcement, public relations preparation, documentation requirements, etc. It is is ineffective to create policy during an incident, so do this now.
  2. Regain control. If possible, disconnect the suspected system(s) from the network and perform a low level back up using as little of the compromised system in the process. For example, backup utilities on the compromised system should not be used to perform the backup, since the intruder may have modified them. In addition, deploy an incident response monitoring system to immediately begin monitoring the victim's network for additional intruder activity.
  3. Analyze the intrusion. Review the compromised system(s) for any modifications, sniffer files, intruder tools, etc. Logs from network firewalls, intrusion detection systems and the like should also be reviewed to gain insight into the intruder's activity. Intruder's techniques are constantly evolving and fully analyzing such an incident is often better left for an experienced security response team. A majority of network incidents involve more than one compromised system; sites risk a failure to identify other compromised systems if security professionals do not handle this step.
  4. Contact other sites involved. If possible, contact owners of other compromised systems so they can go through this same checklist. Many sites only discover they have been compromised when they are notified by another company that they have, in turn, compromised. A trusted third party incident response team can perform this step discretely to minimize potential adverse public response to the incident.
  5. Recover from the intrusion. Install a clean version of the operating system and follow industry guidelines for securing the system, including the installation of vendor patches and disabling of all unnecessary services. Be cautious of old backups, since they may be compromised as well. Finally, ensure that all potentially compromised passwords are immediately changed.
  6. Solidify the security of your network and system. Take steps to keep the intruder from re-compromising the network immediately and take final steps to identify other potentially compromised systems. Use the lessons learned from past security breaches and apply these lessons to remaining corporate assets. Review and improve firewall configuration and upgrade monitoring/logging capability based on deficiencies discovered in the incident. Consider subscribing to a service that provides real-time security monitoring and around-the-clock analysis.
  7. Reconnect to the Internet. Restore connectivity, with diligent monitoring in place. Be alert for additional attempts to compromise the system.
  8. Update your security policy. Take the lessons learned from the incident and apply it to your site security document to ensure that costly lessons are not wasted. Consider using the total cost of the incident to justify changes to the security posture, as well as additional security resources.

Tim Belcher is chief technology officer of Riptech, Inc., a security service provider based in Alexandria, Va. that offers real-time security monitoring as well as professional security services including forensics, assessment, auditing and policy development.

Copyright Riptech, Inc. 2000. All Rights Reserved.