Episode #122: More Whacking of Moles

Tim prepares for a fight:

In my home town we have a college with a team who intends to compete in the CCDC Competition. The students are in control of a number of systems that are under attack by professional penetration testers (hackers) and the students need to defend the systems from the attackers.

The mentor of the group asked if I had any nasty little tricks to help defend the systems. I first pointed him to our Advanced Process Whack-a-Mole. I was then asked if there was a good way to baseline the system for running processes, and then kill any that aren't in that group. I said sure, but with two caveats: 1) most exploits aren't going to kick off a separate process and 2) this may have unexpected consequences. But we went a head an did it anyway to experiment. After all, college is a time to experiment isn't it?

Let's first use cmd.exe to create our baseline file.

C:\> for /f "skip=3" %i in ('tasklist') do @echo %i >> knowngood.txt

The Tasklist command lists all the running processes. The For loop is used to strip off the column headers and to give us just the name of the executable. The knowngood.txt file now contains a list of all the executables that we trust and looks like this:

winlogon.exe
services.exe
lsass.exe
ibmpmsvc.exe
nvsvc32.exe
svchost.exe
svchost.exe
svchost.exe
S24EvMon.exe
svchost.exe
...

Now, a little while later, we come back and check the running processes. We compare the running processes against our file to find ones we don't approve of.

C:\> for /f "skip=3" %i in ('tasklist') do @type knowngood.txt |
  find "%i" > null || echo "bad process found: %i"

bad process found: calc.exe

Uh Oh, it looks like someone is doing some unauthorized math. We'll stop that, but first let's see how this command works?

The first part of our For loop parses the output of tasklist. The variable %i contains the name of the executable of a currently running process. We then need to search the file to see if %i is a good process, or a bad one.

We write out the contents of knowngood.txt and use the Find command to see if the file contains the process %i. And we don't care to see the output, so the output is redirected to NUL. The next part is a little trick using the Logical Or (||) operator.

As you probably know, if either input to our Logical Or is a true, then the result is true.

Input 1    Input 2    Result
true       true       true
true       false      true
false      true       true
false      false      false

If the first part of the command is successful, meaning we found a string in the file, then it returns true, otherwise the result is false. Also notice, if the first input is true, then we don't need to check the second input since the result is already true. Operators that operate in such a manner are known as Short-Circuit Operators, and we can use the functionality to our advantage.

If our Find command finds our process in the file, then the result is true, so there is no need to do the second portion. Only if our Find does not find a match do we execute the second portion of our command, in this case Echo.

We can upgrade the command to kill our process too.

C:\> for /f "skip=3" %i in ('tasklist') do @type knowngood.txt | find "%i" > NUL || taskkill /F /IM %i
SUCCESS: The process "calc.exe" with PID 1932 has been terminated.

Cool, we have an automated killing machine, now to do the same thing in PowerShell. We'll start off creating our known good file.

PS C:\> Get-Process | select name | Export-Csv knowngood.csv

The Export-Csv cmdlet is the best way to import and export object via PowerShell. Once we export our list we can come back later and look for any "rogue" processes.

PS C:\> Compare-Object (Import-Csv .\knowngood.csv) (Get-Process) -Property Name |
  ? { $_.SideIndicator -eq "=>" }

Name    SideIndicator
----    -------------
calc    =>

We use the Compare-Object cmdlet to compare the Known Good processes from our csv file and the results from the Get-Process command. The comparison takes place on the Name property of each. We then filter the results for objects that aren't in our csv but are returned by Get-Process. And as we can see that pesky calc is back. Silly math, let's make it stop by killing it automatically. To do that, all we need to do is pipe it into the Stop-Process cmdlet.

PS C:\> Compare-Object (Import-Csv .\knowngood.csv) (Get-Process) -Property Name |
  ? { $_.SideIndicator -eq "=>" } | Stop-Process

Those silly mathmeticians hackers, we have foiled them. No taking over our Windows machines.

Hal joins the throwdown

This challenge turned out to be a lot of fun because what I thought was going to be a straightforward bit of code turned out to have some unexpected wrinkles. I was sure at the outset how I wanted to solve the problem. I would set up an array variable indexed by process ID and containing the command lines of the current processes. Then I would just re-run the ps command and compare the output against the values stored in my array.

So let's first set up our array variable:

$ ps -e -o pid,cmd | tail -n +2 | while read pid cmd; do proc[$pid]=$cmd; done

Here I'm telling ps just to dump out the PID and command line columns. Then I use tail to filter off the header line from the output. Finally, my while loop reads the remaining input and makes the appropriate array variable assignments.

Seems like it should work, right? Well imagine my surprise when I tried to do a little testing by printing out the information about my current shell process:

$ echo ${proc[$$]}

$

Huh? I should be getting some output there. I must admit that I chased my tail on this for quite a while before I realized what was going on. The array assignments in the while loop are happening in a sub-shell! Consequently, the results of the array variable assignments are not available to my parent command shell.

But where there's a will, there's a way:

$ eval $(ps -e -o pid,cmd | tail -n +2 | 
                 while read pid cmd; do echo "proc[$pid]='$cmd';"; done)
$ echo ${proc[$$]}
bash

Did you figure out the trick here? Rather than doing the array variable assignments in the while loop, the loop is actually outputting the correct shell code to do the assignment statements. Here's the actual loop output:

$ ps -e -o pid,cmd | tail -n +2 | 
        while read pid cmd; do echo "proc[$pid]='$cmd';"; done
proc[1]='/sbin/init';
proc[2]='[kthreadd]';
proc[3]='[migration/0]';
[...]

So I then take that output and process it with "eval $(...)" to force the array assignments to happen in the environment of the my parent shell. And you can see in the example output above that this sleight of hand actually works because I get meaningful output from my "echo ${proc[$$]}" command. By the way, for those of you who've never seen it before "$$" is a special variable that expands to be the PID of the current process-- my command shell in this case.

OK, we've got our array variable all loaded up. Now we need to create another loop to check the system for new processes. Actually, since we want these checks to happen over and over again, we end up with two loops:

$ while :; do 
    ps -e -o pid,cmd | tail -n +2 | while read pid cmd; do 
        [[ "${proc[$pid]}" == "$cmd" ]] || echo $pid $cmd; 
    done; 
    echo check done; 
    sleep 5; 
done
4607 ps -e -o pid,cmd
4608 tail -n +2
4609 bash
check done
4611 ps -e -o pid,cmd
4612 tail -n +2
4613 bash
check done
4615 ps -e -o pid,cmd
4616 tail -n +2
4617 bash
check done
...

The outermost loop is simply an infinite loop to force the checks to happen over and over again. Inside that loop we have another loop that looks a lot like the one we used to set up our array variable in the first place. In this case however, we're comparing the current ps output against the values stored in our array. Similar to Tim's solution, I'm using short-circuit logical operators to output the information about any processes that don't match up with the stored values in our array. After the loop I throw out a little bit of output an sleep for five seconds before repeating the process all over again.

But take a look at the output. Our comparison loop is catching the ps and tail commands we're running and also the sub-shell we're spawning to process the output of those commands. These processes aren't "suspicious", so we don't want to kill them and we don't want them cluttering our output. But how to filter them out?

Well all of these processes are being spawned by our current shell. So they should have the PID of our current process as their parent process ID. We can filter on that:

$ while :; do 
    ps -e -o pid,ppid,cmd | tail -n +2 | while read pid ppid cmd; do 
        [[ "${proc[$pid]}" == "$cmd" || "$ppid" == "$$" ]] || echo $pid $cmd; 
    done; 
    echo check done; 
    sleep 5; 
done
check done
check done
check done
4636 gcalctool
check done
4636 gcalctool
check done
4636 gcalctool
check done
...

So the changes here are that I'm telling the ps command to now output the PPID value in addition to the PID and command line. This means I'm now reading three variables at the top of my while loop. And the comparison operator inside the while loop gets a bit more complicated, since I want to ignore the process if it's already a known process in the proc array variable or if it's PPID is that of my command shell.

From the output you can see that all is well for the first three checks, and then the evil mathematicians fire up their calculator of doom. If you want the calculator of doom to be stopped automatically, then all you have to do is change the echo statement after the "||" in the innermost loop to be "kill -9 $pid" instead. Of course, you'd have to be running as root to be able to kill any process on the system.

Shell trickery! Death to evil mathematicians! What's not to like?

Friend of the blog Jeff Haemer wrote in with an alternate solution that uses intermediate files and some clever join trickery. Check out his blog post for more details.