NOT JUST FOR SCRIPTS

Short circuiting is useful in many places

Short circuiting isn’t just for scripts. It can be useful when you have a series of commands that might take a while to run when each command depends on the success of the command before it. For example, the command sudo apt-get update && sudo apt-get -y upgrade will first update the local software repository cache and then upgrade any packages that have newer versions. The -y option automatically says yes to any prompts. If you are unable to connect to your repositories for some reason the upgrade command is never executed.

Another common use of this technique is building software from source when you do not want to sit around and wait to see if each stage completes successfully. Many packages require a configure script to be run that checks dependencies and optionally sets some non-default options (such as library and tool locations), followed by a make and sudo make install. It can take some time for all three stages to complete. The command ./configure && make && sudo make install can be used to do this all on one line.

The only real work done in this script is in the echo line near the bottom. We have already seen the echo command, but there are a few new things on this line. First, echo has a -e option. The option enables interpretation of backslash characters. This allows us to put newlines (\n) in our string in order to produce multiple lines of output with a single echo command.

There are a couple reasons why we want to use a single echo command here. First, we will be passing (piping actually) the results to the netcat talker which will send this data to our forensics workstation. We want this done as one atomic transaction. Second, this allows a more compact and easily understood script.

There is also something new in the echo string, the $@ variable. $@ is equal to the entire set of command line parameters passed to the script. We first use $@ to create a header that reads “++++Sending log for {command with parameters} at {date} ++++”. We then use our $() trick yet again to actually run the command and insert its output into our string. Finally, a “–-end–-” footer is added after the command output.

The last client script is used to send files to the forensics workstation for analysis. It will make a log entry, then send the filename to the appropriate port, then delay a few seconds to give the server time to create a listener to receive the file, and finally send the file. The script for doing this follows.

send-file.sh

#

Simple script to send a new file # to listener on forensics workstation. Intended to be # used as part of initial live response. # by Dr. Phil Polstra (@ppolstra) as developed for # PentesterAcademy.com.

defaults primarily for testing

[ -z “$RHOST” ] && { export RHOST=localhost; }

[ -z “$RPORT” ] && { export RPORT=4444; }

[ -z “$RFPORT” ] && { export RFPORT=5555; } [ -z “$RFTPORT” ] && { export RFTPORT=5556; } usage () { echo “usage: $0 <filename>”

echo “Simple script to send a file to listener” exit 1

} # did you specify a file? if [ $# -lt 1 ] ; then usage fi #log it

echo “Attempting to send file $1 at $(date)” | nc $RHOST $RPORT #send name

echo $(basename $1) | nc $RHOST $RFPORT #give it time sleep 5 nc $RHOST $RFTPORT < $1

As with the other client scripts, there is no she-bang at the beginning of the script so it must be run manually by typing bash ./send-file.sh {filename}. The short

circuiting technique is again used to set environment variables to defaults if they have not been set. The script is very straightforward. First, the number of parameters passed in is checked, and if no filename was passed in, the usage statement is displayed. Second, the filename is echoed to the filename listener which causes the server to start a listener to receive the file. Note that the basename command is used to strip any leading path from the filename (the full path does appear in the log, however). Third, the script sleeps for five seconds to allow the server time to start the listener. This is probably not needed, but it is well worth waiting a few seconds to have a reliable script. Finally, the file is sent to the file listener and then the script exits.

INTRODUCING OUR FIRST SUBJECT SYSTEM

Throughout this book we will work through a few example subject systems. If you wish to follow along, you may download the example images from http://philpolstra.com. This website is also the place to get updates and other materials from this book (and also past and future books). To keep things simple I will install this example system in a virtual machine using VirtualBox running on my Ubuntu 14.04 computer. Recall that I said earlier in this book that using a USB 2.0 response drive is less problematic when trying to mount the drive in a virtual machine.

Our first example is also an Ubuntu 14.04 64-bit system. You have received a call from your new client, a development shop known as Phil’s Futuristic Edutainment (PFE) LLC. Your initial interviews revealed that one of the lead developer’s computers has been acting strangely and PFE suspects the machine has been hacked. They have no in-house Linux forensics people so you were called in. One of the things that seems to be happening on the subject system is that warning such as those from Figure 2.13 keep popping up.

FIGURE 2.13

Suspicious system warnings on subject system.

One of the first things you need to do with the subject system is mount your knowngood binaries. The steps required are shown in Figure 2.14. The ifconfig utility is also run as a verification that everything is working correctly.

FIGURE 2.14

Mounting a response drive and loading a known-good shell and binaries.

A sequence of commands to run the know-good binaries and then use the above client scripts is shown in Figure 2.15. Some of the results that appear on the forensics workstation are shown in Figure 2.16 and Figure 2.17.

FIGURE 2.15

Mounting know-good binaries and then running some client scripts on the subject system.

FIGURE 2.16

Partial log entry for for the commands shown in Figure 2.15.

FIGURE 2.17

Files created by the commands in Figure 2.15.

COLLECTING VOLATILE DATA

There is plenty of volatile data that can be collected from the subject system. Collecting this data will help you make a preliminary determination as to whether or not there was an incident. Some of the more common pieces of data you should collect are discussed below.

Date and time information

One of the first things you want to collect is the date and time information. Why? The subject system might be in a different timezone from your usual location. Also, computer clocks are known to be bad at keeping good time. If the system has not been synchronized with a time server recently the clock could be off, and you will want to note this skew to adjust times in your reports. Despite its name, the date command outputs not only the date, but the time and timezone as well.

Operating system version

You will need to know the exact operating system and kernel version you are running should you later decide to do memory analysis. The uname -a command will provide you with this information and also the machine name and kernel build timestamp. The results of running this command on the PFE subject system are shown in Figure 2.18.

FIGURE 2.18

Results of running uname on a subject system.

Network interfaces

What network interfaces are on the machine? Is there anything new that shouldn’t be there? This might sound like a strange question, but an attacker with physical access could add a wireless interface or USB interface pretty easily. Other strange but less common interfaces are a possibility.

What addresses have been assigned to various interfaces? What about the netmask? Has someone taken a network interface down in order to have traffic routed through something they control or monitor? All of these questions are easily answered using the ifconfig -a command. The results of running ifconfig -a on the subject system are shown in Figure 2.19.

FIGURE 2.19

Results from the ifconfig -a command.

Network connections

What other machines are talking with the subject machine? Are there any suspicious local network connections? Is the system connecting to something on the Internet when it is not supposed to have such access? These questions and more can be answered by running the netstat -anp command. The options a, n, and p are used to specify all sockets, use only numeric IPs (do not resolve host names), and display process identifiers and programs, respectively.

Open ports

Are there any suspicious open ports? Is the system connecting to ports on another machine that is known to be used by malware? These questions are also easily answered by running the netstat -anp command. The results of running this command on the subject system are shown in Figure 2.20.

FIGURE 2.20

Results of running netstat -anp on the subject system.

Malware can affect any of the commands you are running during your initial scan of the subject system. This is true even with know-good binaries as underlying memory structures may be altered by rootkits and the like. The results of running netstat -anp on the subject system after a rootkit is installed are shown in Figure 2.21. Note that the netstat process is killed and a system warning is also displayed. Every command that fails like this increases the probability that the machine has been compromised.

FIGURE 2.21

The results of running netstat -anp after a rootkit has been installed.

Programs associated with various ports

Some ports are known to be home to malicious services. Even safe ports can be used by other processes. The output of netstat -anp can be used to detect programs using ports they should not be using. For example, malware could use port 80 as it will look like web traffic to a casual observer.

Open Files

In addition to asking which programs are using what ports, it can be insightful to see which programs are opening certain files. The lsof -V (list open files with Verbose search) command provides this information. The results of running this command on the subject system are shown in Figure 2.22. As with the netstat command, this will fail if certain rootkits are installed.

FIGURE 2.22

Results of running lsof -V on subject system. Note that this command failed when it was rerun after installation of a rootkit (Xing Yi Quan).

Running Processes

Are there any suspicious processes running? Are there things being run by the root user that should not be? Are system accounts that are not allowed to login running shells? These questions and more can be answered by running the ps -ef command. The -e option lists processes for everyone and -f gives a full (long) listing. This is another command that might fail if a rootkit has been installed. Partial results of running this command on the subject system are shown in Figure 2.23.

FIGURE 2.23

Results of running ps -ef on the subject system.

Routing Tables

Is your traffic being rerouted through an interface controlled and/or monitored by an attacker? Have any gateways been changed? These and other questions can be answered by examining the routing table. There is more than one way to obtain this information. Two of these ways are to use the netstat -rn and route commands. I recommend running both commands as a rootkit might alert you to its presence by altering the results of one or both of these commands. If you get conflicting results it is strong evidence of a compromise. The results of running both of these commands on the subject system are shown in Figure 2.24.

FIGURE 2.24

Results of running netstat -rn and route on the subject system.

Mounted filesystems

Are any suspicious volumes mounted on the system? Is one of the filesystems suddenly filling up? What are the permissions and options used to mount each partition? Are there unusual temporary filesystems that will vanish when the system is rebooted? The df (disk free) and mount commands can answer these types of questions.

As with many other commands, a rootkit might alter the results of one or both of these commands. Whenever two utilities disagree it is strong evidence of a compromised system. The results of running df and mount on the subject system are shown in Figure

2.25.

FIGURE 2.25

Results of running df and mount on the subject system.

Loaded kernel modules

Are there any trojaned kernel modules? Are there any device drivers installed that the client does not know anything about? The lsmod command provides a list of installed kernel modules. Partial results from running lsmod are shown in Figure 2.26.

FIGURE 2.26

Partial results of running lsmod on the subject system.

Users past and present

Who is currently logged in? What command did each user last run? These questions can be answered by the w command. For those who are not familiar, w is similar to the who command, but it provides additional information. Results for the w command are shown in Figure 2.27.

FIGURE 2.27

Results of running the w command on the subject system.

Who has been logging in recently? This question is answered by the last command. A list of failed login attempts can be obtained using the lastb command. The last command lists when users were logged in, if the system crashed or was shut down while a user was logged in, and when the system was booted. Partial results from running last are shown in Figure 2.28. Note that there are multiple suspicious logins on March 9^th. A new user johnn who should not exist has logged on as has the ligthdm system account.

FIGURE 2.28

Partial results of running last on the subject system. The logins by johnn and lightdm are indicators of a compromise.

The results from running lastb on the subject system are shown in Figure 2.29. From the figure it can be seen that John struggled to remember his password on May 20^th. The much more interesting thing that can be seen is that the lightdm account had a failed login on March 9^th. When you combine this information with the results from last, it would appear that an attacker was testing this new account and did not correctly set things up the first time. Furthermore, it seems likely the john account was used by the attacker.

FIGURE 2.29

Are there any new accounts created by an attacker? Has someone modified accounts to allow system accounts to login? Was the system compromised because a user had an insecure password? Examination of the /etc/passwd and /etc/shadow files help you answer these questions.

A partial listing of the /etc/passwd file can be found in Figure 2.30. Notice the highlighted portion is for the johnn account. It appears as though an attacker created this count and tried to make it look a lot like the john account for John Smith. Also of note is the hidden home directory for johnn located at /home/.johnn.

FIGURE 2.30

Partial listing of /etc/passwd file from subject system. The highlighted line is for a new johnn account which appears at first glance to be the same as john. Note the hidden home directory.

Looking at the line for the lightdm account in Figure 2.30 we observe that the login shell has been set to /bin/false. This is a common technique used to disable login of some system accounts. From the last command results it is clear that this user was able to login. This is cause for investigation of the /bin/false binary.

Putting it together with scripting

There is no good reason to type all of the commands mentioned above by hand. Since you already are mounting a drive with your know-good binaries, it makes sense to have a script to do all the work for you on your response drive. A simple script for your initial scan follows. The script is straightforward and primarily consists of calling the send-log.sh script presented earlier in this chapter.

initial-scan.sh

# Simple script to collect basic information as part of # initial live incident response. # by Dr. Phil Polstra (@ppolstra) as developed for # PentesterAcademy.com. usage () { echo “usage: $0 [listening host]”

echo “Simple script to send a log entry to listener” exit 1

} # did you specify a listener IP?

if [ $# -gt 1 ] || [ “$1” == “—help” ] ; then usage fi # did you specify a listener IP?

if [ “$1” != “” ] ; then source setup-client.sh $1

fi # now collect some info!

send-log.sh date send-log.sh uname -a send-log.sh ifconfig -a send-log.sh netstat -anp send-log.sh lsof -V send-log.sh ps -ef send-log.sh netstat -rn send-log.sh route send-log.sh lsmod send-log.sh df send-log.sh mount send-log.sh w send-log.sh last send-log.sh lastb send-log.sh cat /etc/passwd send-log.sh cat /etc/shadow

SUMMARY

We have covered quite a bit in this chapter. Ways of minimizing disturbance to a subject system while determining if there was an incident were discussed. Several scripts to make this easy were presented. We ended this chapter with a set of scripts that can allow you to determine if there was a compromise in mere minutes. In the next chapter we will discuss performing a full live analysis once you have determined that an incident occurred.

CHAPTER