FINDING THINGS THAT ARE OUT OF PLACE
Thus far in this chapter we have seen a lot of theory on how the extended filesystem works. We can leverage that knowledge to find things that are out of place or not quite right. After all, this is normally what the forensic investigator is after. Inconsistencies and uncommon situations often point to an attacker’s attempt to hide his or her actions.
Some of the system directories such as /sbin and /bin are highly targeted by attackers.
Even the lowly ls command can often be enough to detect alterations in these directories. How can we detect tampering in a system directory? When the system is installed, files in the system directories are copied one after the other. As a result, the files are usually stored in sequential inodes. Anything added later by an attacker will likely be in a higher inode number.
The command ls -ali will list all files (-a) with a long listing (-l) which will begin with an inode number (-i). If we take the results of this command and pipe them to sort -n, they will be sorted numerically (-n) by inode number. The results of running ls ali bin | sort -n from within the mount directory (subject’s root directory) of the PFE subject system are shown in Figure 7.16. Files associated with the Xing Yi Quan rootkit are highlighted. Notice that the inodes are mostly sequential and suddenly jump from 655,549 to 657,076 when the malware was installed.
FIGURE 7.16
Results of running ls -ali bin | sort -n on PFE subject image. The highlighted files are from a rootkit installation.
Some readers may have expected the inode numbers to be considerably higher. The primary reasons that they are only a few thousand inodes away is that Linux will do its best to store files in a given directory in the same block group. Only when there is no space in the containing directories block group will the file be stored elsewhere. This is a performance optimization that attempts to minimize movement of the read/write heads on a hard drive.
In this case the attacker did not attempt to alter the timestamps on the rootkit files. There are many tools that will allow you to easily modify these timestamps. The important thing to realize is that no matter what you do to the timestamps, the inodes will reveal recently added files every time.
The ls command supports a few basic sorts. The default sort is alphabetical, but it will also sort by extension, size, time, and version. The command ls -aliR <directory> –sort=size will perform a recursive (-R) listing of a directory with everything sorted by size (largest to smallest). Partial results of running ls -aliR bin –sort=size are shown in Figure 7.17.
FIGURE 7.17
Partial results of running ls -aliR bin –sort=size against the PFE subject image.
Have a look at the highlighted files bash and false from Figure 7.17. Notice anything unusual? The only thing /bin/false does is return the value false when called. Yet this is one of the three largest files in the /bin directory. It is also suspiciously the exact same size as /bin/bash. What appears to have happened here is that the attacker copied /bin/bash on top of /bin/false in an attempt to login with system accounts. Notice that this attacker was not terribly sophisticated as evidenced by the fact that they didn’t change the timestamp on /bin/false when they were done.
Because the attacker failed to modify the timestamps, a listing sorted by time will also show what has happened pretty clearly. A partial listing of results from the ls -aliR bin –sort=time command is shown in Figure 7.18. The highlighted portion shows both the rootkit files and the recently altered /bin/false.
FIGURE 7.18
Partial results from running ls -aliR bin –sort=time against the PFE subject image.
The advantage of running the ls command as shown in this section is that it is simple. The downside is that you have to carefully scrutinize the output to detect any gaps in the inode numbers. Some readers might have seen monthly bank statements that list checks that have been cashed against the account listed in order with a notation if a check number is missing.
Wouldn’t it be great if we could have a shell script that did something similar to the bank statement? Well, you are in luck, such a script follows. You are extra fortunate as there are some new shell scripting techniques used in this script.
!/bin/bash
# out-of-seq-inodes.sh
# Simple script to list out of # sequence inodes.
Intended to be used as part of # a forensics investigation.
As developed for PentesterAcademy.com # by Dr. Phil Polstra (@ppolstra) usage () { echo “Usage: $0 <path>” echo “Simple script to list a directory and print a warning if” echo “the inodes are out of sequence as will happen if system” echo “binaries are overwritten.” exit 1
} if [ $# -lt 1 ] ; then
usage fi # output a listing sorted by inode number to a temp file templs=’/tmp/temp-ls.txt’ ls -ali $1 | sort -n > $templs # this is for discarding first couple lines in output foundfirstinode=false declare -i startinode while read line do
there is usually a line or two of non-file stuff at start of output # the if statement helps us get past that if [ “$foundfirstinode” = false ] \ && [ “\echo $line | wc -w\” -gt 6 ] ; then startinode=expr $(echo $line | awk ‘{print $1}’) echo “Start inode = $startinode” foundfirstinode=true fi
q=$(echo $line | awk ‘{print $1}’) if [[ $q =~ ^[0-9]+$ ]] && \
[ “$startinode” -lt $q ]; then
if [ $((startinode + 2)) -lt $q ] ; then echo -e “\e[31mOut of Sequence inode detected \
expect $startinode got $q\e[0m” else
echo -e “\e[33mOut of Sequence inode detected \
expect $startinode got $q\e[0m” fi
reset the startinode to point to this value for next entry startinode=expr $(echo $line | awk ‘{print $1}’) fi echo “$line”
startinode=$((startinode+1))
done < $templs rm $templs
The script starts with the usual she-bang, followed by a usage function which is called when no command line arguments were passed in. To improve performance and also make the script simpler, the output from ls -ali <directory> | sort -n is sent to a temporary file. Two variables foundfirstinode and startinode are created. The line declare -i startinode is new. This line creates a new variable that is an integer value. It also sets the variable to zero. We will see later in the script why this is needed.
The line while read line begins a do loop. You might wonder where this line is coming from. If you look down a few lines, you will see a line that reads done < $templs. This construct redirects the temporary file to be input for the loop. Essentially, this do loop reads the temporary file line by line and performs the actions listed within the loop code.
The if block at the start of the do loop is used to consume any headers output by ls. The if has two tests anded together with the && operator. The first test checks to see if foundfirstinode is false which indicates we have not made it past any header lines to the actual file data yet. The second test executes the command echo $line | wc -w and tests if the result is greater than six. Recall that enclosing a command in back ticks causes it to be executed and the results converted to a string. This command echoes the line which is piped to the word count utility, wc. Normally this utility will print out characters, words, and lines, but the -w option causes only words to be printed. Any output lines pertaining to files will have at least seven words. If this is the case the startinode variable is set, a message is echoed to the screen, and foundfirstinode is set to true;
The line startinode=expr $(echo $line | awk ‘{print $1}’) is easier to understand if you work from the parentheses outward. The command echo $line | awk ‘{print $1}’ echoes the line and pipes it to awk which then prints the first word (start of the line up to the first whitespace character). Recall that the inode number is listed first by the ls command. This inode number then gets passed to expr
After the if block (which is only run until we find our first file), the line q=$(echo $line | awk ‘{print $1}’) sets the value of q equal to the inode number for the current line. If startinode is less than the current inode number, a warning message is printed. The nested if/else statement is used to print a message in red if the inode number jumped up by more than two. Otherwise the message is printed in yellow. The funny characters in the echo statements are called escape sequences. The -e option to echo is needed to have echo evaluate the escape sequences (which begin with \e) rather than just print out the raw characters. If the message was displayed, the startinode variable is reset to the current inode number.
Regardless of whether or not a message was printed, the line is echoed. The startinode variable is incremented on the line startinode=$((startinode+1)). Recall that $(()) is used to cause bash to evaluate what is contained in the parentheses in math mode. This is why we had to declare startinode as an integer earlier in the script. Only integer values can be incremented like this. When the loop exits, the temporary file is deleted. Partial results of running this script against the PFE subject system’s /bin directory are shown in Figure 7.19. Notice the red warnings before the rootkit files.
FIGURE 7.19
Partial results of running out-of-seq-inode.sh against the PFE subject system’s /bin directory. Note the warning for the rootkit files.