Those of you who run your corporation's networks are probably aware of the term honeypot, a computer that is designed from the start to be compromised and attacked. The idea is that an ordinary off-the-shelf server, typically without any particular security patches or other modifications, is placed out on the Internet, and several packet monitoring tools are setup to watch what happens. The idea is to use the honeypot to learn the tools and tactics used by the bad guys (which for the purposes of this essay we'll call black hats).

But honeypots only tell you what can happen to a single computer. These days, black hats can compromise entire networks, entering through one or more weak links. These attacks are more trouble: it can be harder to track down the security breaches and harder to fix all of them.

Enter the Honeynet Project. A group of 30 computer security professionals banded together and created several different challenges, in the best spirits of the Internet community. Each member of the group, who go by such colorful monikers as Rain Forest Puppy and K2, has a real full-time job, working in various end-user and vendor companies. They do this in their spare time, in the hope of learning from watching black hat attacks, and also partly for the intellectual challenge of coming up with new tools and techniques to track down network attacks. Think of parking a brand new car on some deserted downtown street and watching it from a closed circuit TV until someone tries to steal it. Now picture doing this before TV was invented.

What I find curious is that the project doesn't advertise or draw out any of the attacks on its systems. Rather, participants just place a bunch of computers on the Internet and expect the black hats will find their way inside their networks and try to compromise the systems.

"We are being scanned constantly," said one project participant. Often, the attacks happen within a few days of putting up the networks. That is a somewhat depressing thought. Those of you who monitor your own networks can probably attest to frequent port scans and the electronic equivalent of rattling doorknobs.

The group posted a "forensic challenge" earlier this year, providing evidence of a real attack and asking people to analyze it and provide a blow-by-blow explanation. The attack on which the challenge was based actually happened to a university Linux computer last fall, when it took all of about a minute for someone to break into the system and install some automated attack tools. (Various packet capture utilities happened to be monitoring the compromised system, and extensively collected various bits.)

The notion was to provide everyone with the same data, compare the responses and see what various people learned and how they went about documenting the attack. The challenge asked contestants to identify the intrusion method and as much as possible about the intruders, and to list all the files that were added/modified by the intruders. The contest also asked for other documentation, including an estimate of the cost to respond to this intrusion.

Thirteen teams took on the challenge and submitted their entries. Most of the explanations are over my head and beyond my level of expertise, to be sure. But what I did learn was that the hard part isn't understanding how your network has been compromised, it is figuring out how to fix it and return it to a state where it won't be threatened again. The average time spent to investigate the attack turned out to be about 34 hours per person. That is a staggering amount of person-hours, and these are the supposed experts with years of security training.

As the project leaders state:

"One thing is for certain. It is much harder and takes more skill to figure out what was damaged than to do the damage. When a system is compromised, and the data on it and its network are compromised, it is not simple to determine the extent of the damage without a lot of work. We do not know if the black hat stole people's passwords, hacked other systems, has implemented sniffers, etc. This argues for strong prevention, defense in depth (including monitoring in depth), and trained responders. If all the administrator does is re-install the OS, they are doing a wholly inadequate job of responding to a security incident, as the extent of damage may be far greater then a single system."

You can read more about the project, the challenge, and view the various results at http://project.honeynet.org/challenge. It is a fabulous resource, and a tremendous learning experience. The project team should be commended for making this possible, and to help others study black hat attacks "in the wild."

There is a ton of material on the site. I particularly liked (in the white papers section) the conversation recorded between D1ck and J4n3 (Dick and Jane), two black hats talking about various exploits, over an IRC channel, giving plenty of insight into some of their motives. All in all, a very sobering experience.

David Strom is a leading expert on network and Internet technologies and has written extensively on the topic for over 13 years for a variety of publications. This story originally appeared in Web Informant, the weekly essay he has published since 1995 that goes to several thousand subscribers. He can be reached at david@strom.com.