When the Worst Happens

by Fred Avolio, Avolio Consulting, Inc.

My last column discussed how to prepare for the worst when a computer security "incident" occurs. We know that preparation is not the same as prevention, but the two go hand-in-hand if you are to successfully minimize the damage from an incident. This column will focus on providing an overview of the Computer Security Incident Response Team (CSIRT) in action.

Signs you've been attacked

Signs of an attack are varied. Sometimes it is obvious: the attacker calls - either to be a "hero," to brag about it, or for extortion; Internet access all but disappears under a denial of service attack; or perhaps your web site now sports pictures of naked people. Often they are not so obvious: your intrusion detection system indicates anomalous behavior, for example. Or they may just be indicators, such as system crashes, new user accounts, new files and folders or missing ones, or strange entries in log files.

Often, users are among the first to notice abnormalities. Therefore, all users should know what to do should they receive an attacker's phone call or notice that something is not right. You should train all employees to write things down on paper and sign and date the page, as most people do not remember that they tend to forget things.

Responding to an Incident

After some quick assessment, the CSIRT leader declares an incident and calls the team. All CSIRT meetings start the same: with a reminder of the need for confidentiality and record keeping. The CSIRT is not going to cover it up, but strives for a combination of right communication and right time.

The CSIRT keeps records in a notebook with numbered pages and the recording secretary will date and sign the pages. By now you are getting the idea: this is serious stuff and if you want to take someone to court, you had better do things in a way that will help rather than bog down any case you may try to make against the attacker.

At some point, the incident - if big enough - will be public. The media relations person on the team will be the only one to talk to the media, because she knows what to say and how to say it.

Additionally, and this is really important, everyone else in the company should be aware that only this person discusses incidents with the media. That way everyone else will know to say, "Let me transfer you to Judy Jones. She can help you with that." This is so much better than, "We got hacked or something. There are FBI guys everywhere, people are panicking."

Securing the Crime Scene

The CSIRT - or someone under its direction - will secure the crime scene, which may be the data center, individual servers, or users' PCs. There are usually many crime scenes, so apply these examples to every one of them.

Remember that we are dealing with latent evidence - evidence that is "present or potential, but not evident." [www.dictionary.com] Like fingerprints, it may be there but it must be discovered, and may be easily altered, damaged, or destroyed.

Two people should gather the evidence. One will take notes (or pictures) and verify what the other does, before he does it. We want to disturb as little as possible. For a computer, for example, it means knowing first what not to do. Do not run shutdown, do not make backups (you should have done that yesterday) and do not open or alter any log files. Do act calmly. Do think clearly. Do follow your written Incident Response Procedures.

Observe. Is the screen lit? If not, touch the "Shift" key or carefully and slightly move the mouse. Photograph and write down what you see on the screen. Write down what you hear. Almost always at this point you will, and this is going to sound crazy, shut the computer by pulling the power cord from the back of the computer (not from the wall, as you want to make sure you've got the right one). It should be noticeably quieter.

Now, record everything you can about the computer including model, make, and serial number. Disconnect all the cables, labeling which ones went where, and place tape over the sockets. Again, take photos and write down everything you do.

If the evidence is in logs on various servers, you need not shut them all down (your ISP will be pleased to know this), but whomever makes copies of the logs must be just as meticulously careful. Keep it simple, document everything, and do not alter anything, as far as you are able. Finally, store all evidence in a secured container or location. Log if and when the evidence changes hands as it is important to establish a clear chain of custody. Always think, "I may have to testify in court."

Data Recovery and Investigation

Some readers were waiting for this sexy part of the discussion, and I am sorry to disappoint. Someone with experience, someone who does this every week for example, should recover the data and investigate the incident. All FBI field offices have expertise in computer forensics. Some local law enforcement does as well. Or your managed security service provider can help.

Last steps

After a real incident, have a team meeting when it is over to discuss what worked and what has to be changed. The lessons you learn from each incident will make you more effective in dealing with future incidents. This is all very involved, which is why it is vital to drill and test the procedures under a mock attack.

Resources

CERT CSIRT

RFC 2196, Site Security Handbook

U.S. Department of Justice, Office of Justice Programs, National Institute of Justice Electronic Crime Scene Investigation: A Guide for First Responders

RFC 2350, "Expectations for Computer Security Incident Response"