Tuesday, January 26, 2010

Episode #79: A Sort of List

Hal starts off:

Way back in Episode #11 I showed you a little trick for sorting directory listings by inode number. But it struck me recently that we hadn't talked about all of the other interesting ways you can sort directory listings.

For example, you can use "ls -S" to sort by file size:

$ ls -lS
total 6752
-rw-r----- 1 syslog adm 1271672 2010-01-18 05:36 kern.log.1
-rw-r----- 1 syslog adm 1016716 2010-01-18 05:39 messages.1
-rw-r----- 1 syslog adm 499580 2010-01-18 05:38 daemon.log.1
[...]

Add "-h" if you prefer to see those file sizes with human-readable units:

$ ls -lSh
total 6.6M
-rw-r----- 1 syslog adm 1.3M 2010-01-18 05:36 kern.log.1
-rw-r----- 1 syslog adm 993K 2010-01-18 05:39 messages.1
-rw-r----- 1 syslog adm 488K 2010-01-18 05:38 daemon.log.1
[...]

Also, adding "-r" (reverse sort) can be useful so that the largest files end up at the bottom of the directory listing, closer to your next command prompt:

$ ls -lShr
total 6.6M
[...]
-rw-r----- 1 syslog adm 488K 2010-01-18 05:38 daemon.log.1
-rw-r----- 1 syslog adm 993K 2010-01-18 05:39 messages.1
-rw-r----- 1 syslog adm 1.3M 2010-01-18 05:36 kern.log.1
$

You have to do much less scrolling around this way.

In addition to sorting by size, you can also sort by the so-called "MAC time" values: last modified (mtime), last access (atime), and last inode or meta-data update (ctime). By default, "ls -t" will sort by last modified time. This is another good one to use "-r" on so you can quickly find the most recently modified files in a directory:

$ ls -lrt
total 6752
[...]
-rw-r----- 1 syslog adm 86080 2010-01-18 08:10 kern.log
-rw-r----- 1 syslog adm 120492 2010-01-18 08:17 syslog
-rw-r----- 1 syslog adm 3310 2010-01-18 08:17 auth.log
$

If you want to sort by ctime you use "-c" in addition to "-t". However, to sort by atime you need to use "-u" ("-a" was reserved for something else, obviously):

$ ls -lrtu
total 6752
[...]
-rw-r--r-- 1 root root 219990 2010-01-18 08:00 udev
-rw-r--r-- 1 root root 120910 2010-01-18 08:00 Xorg.0.log
-rw-r----- 1 root adm 56275 2010-01-18 08:00 dmesg
$

Now let's see what my Windows brethren have up their sleeves, shall we?

Ed Responds:
Although not as full featured as the Linux ls command, the humble dir command offers us a bunch of options, allowing us to mimic pretty much everything Hal has done above. The main options we'll use here are:
  • /o followed by a one-character option that lets us specify a sort order (we'll use /os to sort by size and /od by date... with a - sign in front of the one character to reverse order)
  • /t, also followed by one character which lets us specify a time field we're interested in (the field options we have and their definitions, according to the dir command's help, are /tc for Creation time, /ta for Last Access time, and /tw for Last Written time).

So, to get a directory listing sorted by size (smallest to largest), we'd run:

C:\> dir /os

Want them reversed? We would use:

C:\> dir /o-s

Want those sizes in human readable form? Install Cygwin and use the ls command, for goodness sakes. This is the dir command we're talking about here. We don't need no stinkin' human readable format. Actually, the default output for dir does show commas in its size numbers, making things a little more readable than the stock Linux output.

To see directory contents listed by Last Written (which is what dir calls them... roughly the same as last modified times in Linux parlance), in reverse order (with the most recently modified near the top), you could execute:

C:\> dir /o-d /tw

But, like we see with the ls command, Last Written is the default, so you can leave off the /tw to get the same results.

Wanna sort by creation time, again in reverse? Use:

C:\> dir /o-d /tc

And, how about last access? You could go with:

C:\> dir /o-d /ta

It's a good thing that the /od and /o-d sort options pick up the proper timestamp specified by the /t option, or else we'd be forced to do some waaaaay ugly sort command nonsense. Whew!

Tim responds too:

To get a directory listing we use Get-ChildItem. The name is a bit odd, but it is a generic command and can be used to get the child items from any container such as the registry, file system, or the certificate store. Today we are just looking at the file system.

First, let's take a look at the aliases for this useful cmdlet.

PS C:\> Get-Alias -Definition Get-ChildItem

CommandType Name Definition
----------- ---- ----------
Alias dir Get-ChildItem
Alias gci Get-ChildItem
Alias ls Get-ChildItem


I typically use ls since it is 33% more efficient to type than dir. But I digress...

Let's sort by file size:

PS C:\> gci | sort length

Directory: C:\

Mode LastWriteTime Length Name
---- ------------- ------ ----
-a--- 6/10/2009 4:42 PM 10 file1.txt
-a--- 6/10/2009 4:42 PM 24 file2.txt
-a--- 11/24/2009 3:56 PM 1442522 file3.zip


The Get-ChildItem cmdlet does not have sorting capability built in, none of the cmdlets do. But that is what the pipeline and the Sort-Object cmdlet are for.

Want to sort by file size in reverse order? Use the Descending parameter.

PS C:\> gci | sort length -descending

Directory: C:\

Mode LastWriteTime Length Name
---- ------------- ------ ----
-a--- 11/24/2009 3:56 PM 1442522 file3.zip
-a--- 6/10/2009 4:42 PM 24 file2.txt
-a--- 6/10/2009 4:42 PM 10 file1.txt


We can sort by any property, including LastAccessTime, LastWriteTime, or CreationTime.

PS C:\> gci | sort LastWriteTime


We can even sort on two properties.

PS C:\> gci | sort LastWriteTime, Length

Directory: C:\

Mode LastWriteTime Length Name
---- ------------- ------ ----
-a--- 6/10/2009 4:42 PM 24 file2.txt
-a--- 6/10/2009 4:42 PM 10 file1.txt
-a--- 11/24/2009 3:56 PM 1442522 file3.zip


The files will first be sorted by write time. If two files have the same write time, they will then be sorted by length.

Finally, we come to displaying the size in a human readable format, and it isn't pretty. We have to write a custom expression to display the size in KB or MB.

PS C:\> gci | format-table -auto Mode, LastWriteTime, Length,
@{Name="KB"; Expression={"{0:N2}" -f ($_.Length/1KB) + "KB" }},
@{Name="MB"; Expression={"{0:N2}" -f ($_.Length/1MB) + "MB" }},
Name


Mode LastWriteTime Length KB MB Name
---- ------------- ------ -- -- ----
-a--- 6/10/2009 4:42 PM 10 0.01KB 0.00MB file1.txt
-a--- 6/10/2009 4:42 PM 24 0.02KB 0.00MB file2.txt
-a--- 11/24/2009 3:56 PM 1442522 1,408.71KB 1.38MB file3.zip


We can specify custom properties to display. This format works with any of the format cmdlets (Get-Command -Verb Format) or select-object. The custom columns are created by using a hashtable. A hashtable is specified by using @{ key1=value1, key2=value2 }. In our case we specify a name and an expression. Here is a simple example.

..., @{Name="Foo"; Expression={ $_.Length + 1 }}, ...


In this case we would add a column with the heading Foo and with a value of the Length plus 1. The expression can include all sorts of math or other crazy PowerShell fu.

Ironically, getting a human readable output comes from a non-human readable command.

Tuesday, January 19, 2010

Episode #78: Advanced Process Whack-a-Mole

Ed Prepares to Open Up a Can of Process Whoop-Ass:

I've never considered myself a particularly violent man. But, I have to admit it: Sometimes it just feels good to kill processes. I've even been heard to mutter a deadpan "Dodge This" in my lab late at night as I obliterate errant or evil processes just begging to meet their maker. Then, to make sure such a process doesn't pop back up to start bothering me again, I sow the ground with command-line kung fu salt to strangle any other similar process that might pop up in its place.

This technique, which we've taken to calling "Process Whack-a-Mole", can be helpful to people in all walks of life. I'm sure it's happened to pretty much everyone at some point. You find yourself playing defense in a Capture the Flag tournament against an elite team of ninjas from a three-letter government agency who want to completely control your boxen. They repeatedly gain access, and you have to shew them out before they score and you lose points. To deal with such situations, we can run a command to continuously look for processes with certain characteristics of our adversaries, and then kill them when they appear. We touched upon the idea of killing a process that starts to listen on a given TCP port in Episode #76. But, let's go further this time, discussing how you can make much more flexible whack-a-mole commands to deal with various process characteristics.

These techniques are useful even outside of Capture the Flag games. I often use them in malware analysis and even system administration when I want to suppress some activity temporarily while I'm analyzing or configuring something else.

The basic structure I use for process whack-a-mole consists of the following three parts:

<Continuous Loopinator> <Process Selector> <Process Terminator>

Quite often the Process Selector and Process Terminator are combined together in a single command, because we can select or filter for the process we want in the same command we use to whack it. However, to filter for certain specific process characteristics, we'll have to split out these two entities. I'll show you what I mean in a bit.

We start out with our Continuous Loopinator:

C:\> for /L %i in (1,0,2) do @ping -n 2 127.0.0.1 >nul

This is a simple FOR /L loop that starts counting at 1, goes up to 2, in steps of 0. In other words, it's the cmd.exe equivalent of while (true), used to keep something running continuously. At the start of the loop, we introduce a 1-second delay by pinging ourselves twice (-n 2) and throwing the standard output away so as not to clutter our output (>nul). That way, we'll run our Process Selector and Process Terminator approximately every 1 second, helping to minimize our impact on performance. If you want a faster Process Selector, simply omit that ping, and your system will run our whack-a-mole command as fast as it can, but performance may drag.

We then follow with our Process Selector. If you keep in the ping delay, put in an & followed by the Process Selector, which lets us make one command run after another. Otherwise, just put the Process Selector after the @ (which turns off command echo, by the way... no sense having our output clogged up with commands).

The two most common Process Selectors I use are wmic and taskkill, which have the nice property of also including the ability to act as Process Terminators in the same command. Let's look at wmic first.

The wmic command can be used to select given processes based on our constructing a where clause, using the following syntax:

C:\> wmic process <where clause> <verb clause>

In the where clause, we can specify any attribute or group of attributes of processes that can be listed via wmic process. To get a list of these attributes, you could run:

C:\> wmic process get /?

So, for example, if you want to select a processID of 4242, you could write your wmic command as:

C:\> wmic process where processid=4242

Or, we could look for processes that have a given Parent Process ID:

C:\> wmic process where parentprocessid=3788

Or, we could look for processes with a given name:

C:\> wmic process where name="cmd.exe"

These where clauses also support AND and OR, but you've got to make sure you put the guts of your where clause inside of parentheses. The where clauses also support not equals (!=). Check this out:

C:\> wmic process where (name="cmd.exe" and processid != 676)

Now, we haven't supplied a verb clause here, so all our wmic commands are simply displaying raw, unformatted process information on Standard Output.

Let's start doing our whack-a-mole by specifying a Process Terminator by using the verb clause of wmic with the simple verb "delete". That'll kill a process.

Putting these pieces together, suppose you want to kill all cmd.exe processes other than the current cmd.exe you, the administrator, are running. Let's assume that your own cmd.exe has a process ID of 676. You could run:

C:\> for /L %i in (1,0,2) do @ping -n 2 127.0.0.1 >nul & wmic process where (name=
"cmd.exe" and processid!=676) delete

Now, let's see 'em try to run a cmd.exe. Every time someone tries to launch one, your loop will kill it.

Next, suppose you want to prevent a given process with processid 4008 from spawning child processes. Maybe process ID 4008 is a cmd.exe shell, and you want to prevent the person who is using it from being able to run any commands that aren't built-into the shell itself. Or, better yet, maybe process ID 4008 is Tim Medin's PowerShell process, and you wanted to pee in his Corn Flakes, depriving him of the ability to run any separate EXE commands, forcing him to rely solely on built-in capabilities of Powershell itself. We can do this one without our ping-induced delay to really confound him:

C:\> for /L %i in (1,0,2) do @wmic process where parentprocessid=4008 delete

These wmic where clauses also support substring matches, with the use of "like" and %. For example, suppose you want to continuously kill every process that is running from an EXE with a path in a given user's directory. You could run:

c:\> for /L %i in (1,0,2) do @ping -n 2 127.0.0.1 >nul & wmic process where
(executablepath like "c:\\users\\tim\\%") delete

Note that in a "where" clause with the "like" syntax, you have to surround the elements with parens. Also, note that if you have a \ in your where clause, you have to specify it as \\, the first \ indicating an escape, and the second indicating your backslash.

You can combine these where clause elements (=, !=, AND, OR, LIKE, and %) in all kinds of ways to mix and match against various process attributes for whack-a-mole. I'm sure our readers can dream up all kinds of interesting and useful combinations.

But, there is a missing attribute from "wmic process get /?" -- it's the user name that process is running under. To play whack-a-mole based on user names, we can turn to another Process Selector and Terminator: taskkill. I wrote about taskkill filters back in Episode 22, showing how we can use it to kill a process based on its owner username. Here, we'll wrap that in our whack-a-mole construct:

C:\> for /L %i in (1,0,2) do @ping -n 2 127.0.0.1 >nul & taskkill /F /FI
"username eq tim"

Sorry, Tim. That's what you get for using a shell on the same system I'm on. Hal's over on some lonely Linux that no one ever uses, so I leave him alone. :)

Anyway, where was I? Ah, yes, we were discussing attributes of processes that wmic doesn't include, but which may be handy in whack-a-mole games. How about DLLs? Suppose a bad guy is hacking your system and keeps trying to inject a DLL into some process, and you want to kill that process. Maybe Evil Badguy (yes, that's his full name) has injected metsrv.dll, the Metasploit Meterpreter, into a running process, and uses process migration to jump from process to process. Sorry, but getting the system to unload that DLL using only built-in tools at the command line is very difficult, but killing that process is totally doable:

C:\> for /L %i in (1,0,2) do @ping -n 2 127.0.0.1 >nul & taskkill /F /FI
"modules eq metsrv.dll"

Now, I mentioned above that the Process Selector and Process Terminator components are typically combined in a single command, as we've seen so far with wmic and tasklist. When would you have two different commands for these pieces? Well, one example is with netstat, which can show TCP and UDP port usage and the processes associated with each port. That's exactly what we used in Episode #76, where our Process Selector was netstat (whose output I parsed with a FOR /F loop to pull out the ProcessID number), and the Process Terminator was taskkill:

C:\> for /L %i in (1,0,2) do @(ping -n 2 127.0.0.1>nul & for /f "tokens=5"
%j in ('netstat -nao ^| find ^":47145^"') do @taskkill /f /PID %j)

So, keeping in mind those three components of process whack-a-mole, you can use almost any command that lists processes in pretty much any arbitrary way to build a whack-a-mole command for fun and profit.

And now, for a routine disclaimer: Be careful with any of these commands. If you kill a vital system process, such as lsass.exe, you could bring your whole box down. You have been warned. So, now that you are armed and dangerous, go have fun!

Tim prepares for war:

It seems that Ed has a bit of shell envy. So let's kick that inferior shell off "our" machine and keep it (and him) off our machine.

As Ed described, the structure for whack-an-Ed whack-a-mole has three parts, and that basic structure will be very similar in PowerShell.

<Continuous Loopinator> { <Process Selector> | <Process Terminator> }

The Continuous Loopinator repeatedly calls the Process Selector whose results are piped into the Process Terminator. Let's see how each piece works.

Continuous Loopinator:

There are many ways to do a continuous loop, but the easiest and quickest method is to use the While loop.

 PS C:\> while (1) { <code to run ad infinitum>; Start-Sleep 1 }


This loop is pretty self explanatory. It is a simple While loop that runs while 1 is true, which it always will be. The Start-Sleep cmdlet (alias sleep) will suspend activity for the specified amount of time. If we wanted a shorter nap we could use the -milliseconds parameter. Since Ed's command runs every second, we should run ours a bit faster just because we can. How about 5 times a second?

 PS C:\> while (1) { <code to run ad infinitum>; Start-Sleep -milliseconds 200 }


Process Terminator:

I'm covering this a bit out of order because the Terminator is so simple, so indulge me for a bit. The cmdlet used for killing is Stop-Process (alias spps or kill). It can even be used for some rudimentary process selection before the assassination. We can kill based on the Process Id:

PS C:\> Stop-Process 1337


...or the process name.

PS C:\> Stop-Process -Name cmd


In the second example every process with the name "cmd" would be stopped, but what if we wanted to be a little more high tech in making Ed's processes "sleep with the fishes?"

As described earlier, the results of the Process Selector can be piped into our Process Terminator. We can pick any method to retrieve the process(es) to be killed, but more on that later. Here is what it would look like:

<Get Processes> | Stop-Process


By default, Stop-Process will ask for confirmation prior to terminating any process not owned by the current user. To get around that safety mechanism we can use the Force parameter.

<Get Processes> | Stop-Process -Force


Short version:

<Get Processes> | kill -f


We could just kill the processes with Stop-Process by giving it a Process Id or process name, but we want more options. Now let's see how we can find more processes to kill.

Process Selector:

To get a process or a number of processes we use Get-Process (aliases ps and gps). This is our Process Selector. We have covered this before, but we have a number of ways to get a process or a list of processes. To get help on the command you can run:

PS C:\> Get-Help Get-Process


...or for those who have seen the light and come from the linux side but have a bad memory, this works too:

PS C:\> man ps


To see the examples use the Examples parameter, or use the Full parameter to see everything. From the help we can see how to get a process with a given Process ID. We will be looking for PID 4242:

PS C:\> Get-Process -Id 4242
PS C:\> ps -Id 4242


To get all the cmd.exe processes:

PS C:\> Get-Process cmd
PS C:\> ps cmd


Note that the process name does NOT include .exe.

We can also use filters in order to get more granular. We already had our loop to kill all cmd processes, but what if Ed wants to use PowerShell? We need to make sure that we are King of the PowerShell Hill, and any other PowerShell usurper is destroyed. This will find any PowerShell processes that aren't ours.

PS C:\> Get-Process powershell | ? { $_.ID -ne 21876 }


The weaponized version of the command could look like this:

PS C:\> While (1) { ps powershell | ? { $_.ID -ne 21876 } | kill -f; sleep 1 }


The next thing Ed did, after tee-tee'ing in my Kelloggs, was to prevent me from kicking off any processes from a cmd or PowerShell process. So let's do the same to him. Unfortunately, the objects returned by Get-Process do not have a Parent Process Id property, so we will have to use WMI to find processes with a given parent.

PS C:\> Get-WmiObject win32_process -Filter "ParentProcessId=5552" |
% { Get-Process -Id $_.ProcessID }


Get-WmiObject (alias gwmi) is used to access WMI in order to get all processes with a Parent Process Id of 5552. The results are piped into a ForEach-Object (alias %) loop. In the loop we use Get-Process and the Process Id retrieved from WMI in order to get the process object. We can then pipe that into our kill(er). Similar to what Ed did, we want to run this continuously so he doesn't have a chance. Here is what our command looks like:

PS C:\> While (1) { gwmi win32_process -Filter "ParentProcessId=5552" |
% { ps -Id $_.ProcessID } | kill -f }


We also want to make sure that Ed isn't able to run anything from his user directory (which includes his desktop).

PS C:\> While (1) { ps | ? { $_.Path -like "c:\users\ed\*" } | kill -f }


We use the Where-Object (alias ?) to filter our list of processes based on the path. The Like operator is used with our wildcard search string in order to find any of Ed's processes. Again, we pipe the results into Stop-Process in order to kill it.

Just to make sure that Ed doesn't run anything, we will kill any process where he is the owner. Again, we will have to use WMI in order to find the owner of a process.

PS C:\> While (1) { Get-WmiObject Win32_Process |
? { $_.GetOwner().User -eq "ed" } | % { Get-Process -Id $_.ProcessId } |
Stop-Process -Force }


This command is a little complicated, so let's break it down piece by piece. The While loop portion should be obvious so we'll skip that bit of explanation. The first chunk...

Get-WmiObject Win32_Process | ? { $_.GetOwner().User -eq "ed" }


We start off by querying WMI and retrieving WMI objects representing each process running on the current machine. The results are then piped into our filter, Where-Object (alias ?). The "current pipeline object", represented by the variable $_, allows us to access the properties of each object passed down the pipeline. For all intents and purposes, the $_ variable is used to iterate through each object passed down the pipeline. It takes the first object, in our case the first process, and accesses the GetOwner method's User property. We then check to see if the value is equal (-eq) to "ed". If it is equal, then our WMI object passes our filter and is sent further down the pipeline. Remember, the objects are WMI Process Objects, not PowerShell Process Objects, and they will need to be converted to the PowerShell version so we can natively deal with them in PowerShell. On to the conversion.

... | % { Get-Process -Id $_.ProcessId } ...


The objects that passed through the filter are now sent into our ForEach-Object (alias %) loop. This loop is used to iterate through each object and execute some fu on each of the WMI objects. Again, $_ represents the current object. To retrieve the PowerShell version of the process object we use Get-Process. We need to use Id parameter with the Process Id property of the current object ($_.ProcessId). Now we have PowerShell Process Objects. YAY!

... | Stop-Process -Force


Finally, the processes are piped into Stop-Process to be destroyed. The Force option is used since we don't want a confirmation to kill each process.

Next, let's look for the processes with the injected Meterpreter dll. How do we find this dll? We need to look at the modules a process has loaded. Here is what the Modules property looks like for the PowerShell process.

PS C:\> Get-Process powershell | select modules

Modules : {System.Diagnostics.ProcessModule (powershell.exe), System.Diagnostic
s.ProcessModule (ntdll.dll), System.Diagnostics.ProcessModule (kernel
32.dll), System.Diagnostics.ProcessModule (KERNELBASE.dll)...}


As you can see the dll name is wrapped in parenthesis. So here is how we find it and kill it.

PS C:\> Get-Process | ? { $_.Modules -like "*(metsrv.dll)*" } | Stop-Process


EDIT: In MetaSploit v2 and v3.0-3.2 this technique worked to find meterpreter. In v3.3 (and presumably future versions) this does not work since MetaSploit uses Reflective DLL injection to load the dll. I wrote a separate blog post on my personal blog on how to find the footprints of meterpreter: Finding Meterpreter.

Actually, the modules property is a collection of module objects. So we can use a nested Where-Object to filter.

PS C:\> ps | ? { $_.Modules | ? {$_.ModuleName -eq "metsrv.dll" }} | kill


In this command we retrieve all the processes. We then filter the Modules, where the ModuleName is metsrv.dll. The results are piped into Stop-Process.

We can also parse netstat in order to kill a process similar to Episode #76. Let's take that command and wrap it in our infinite loop.

PS C:\> While (1) { netstat -ano | ? {$_ -like "*:47145 *"} |
% { $_ -match "\d+$"; stop-process $matches[0] } }


And as Ed said, be careful not to kill the wrong process or the whole box could go down. Of course, when it is down it is pretty dang hard to attack. Of course, it is also pretty dang hard to use too.

Now that Ed and I have spent all of our energy going after each other, Hal is going to show up and mop the floor with our tired carcases.

Disclaimer: No Eds where harmed in the making of this episode.

Hal's Analysis:

Why are Ed and Tim so angry all the time? It couldn't have anything to do with the platform they've chosen to work on, could it? Hey guys, don't worry, be happy! You can always install Linux for free, or even just use Cygwin.

When Ed first proposed this topic, I was pretty stoked because I thought it was going to be a cake-walk for me with my little friend lsof. But not all of Ed's challenges that could be answered purely with lsof. Some required a bit more shell fu.

Let's start with the simple stuff first. The "infinite loop with 1 second delay" idiom for bash is something we've seen before in previous Episodes:

# while :; do [...your commands here...]; sleep 1; done

In this case, the commands we put into the while loop are going to be a kill command and usually some variant of "`lsof -t ...`" we'll be using to select the PIDs we want to kill. Remember from previous Episodes that "lsof -t" causes lsof to print out just the PIDs of the matching processes, specifically so we can use the output as arguments to the kill command.

For example, let's suppose we want to kill all of Ed's processes. We can use lsof's "-u" option to select processes for a particular user:

# while :; do kill -9 `lsof -t -u skodo`; sleep 1; done


Or we could nuke all the bash shells on the machine, using "-c" to select commands by name:

# while :; do kill -9 `lsof -t -c bash`; sleep 1; done

Of course, this would hammer our own shell, so it pays to be more selective:

# while :; do kill -9 `lsof -t -a -c bash -u^root -u^hal`; sleep 1; done

Here I've added the "-a" flag which means do a logical "and" on my selection criteria. Those criteria are "all commands named bash" ("-c bash") and "not user root" ("-u^root") and "not user hal" ("-u^hal"). Note that lsof's negation operator ("^") only works when selecting user names, PIDs (with "-p"), process group IDs (with "-g"), command names (with "-c"), and protocol state info ("-s", as in "-s^TCP:LISTEN").

Another one of Ed's challenges was killing processes where the binary is in a particular directory. Again we can do this with lsof:

# while :; do kill -9 `lsof -t -a -d txt +d /home/skodo`; sleep 1; done

Here we're looking for process binaries using "-d txt". In the lingo, the binary is what's used to create the "text segment" of a process (where the executable code lives), hence "-d txt" for lsof. The "+d" tells lsof to look for open files under a particular directory. Yes, lsof has so many command line options that the author had to start doubling up on letters using "+" instead of "-" (there's a reason the lsof manual page is nearly 50 pages long when printed out).

Note that "+d" only searches "one level deep". So if Ed were running "/home/skodo/evil", then our loop above would whack that process. But if Ed were running "/home/skodo/bin/evil", then we wouldn't catch it. If you want to do full directory recursion, use "+D" instead of "+d". lsof distinguishes these with separate options because full directory searches are so time-consuming.

However, as I mentioned earlier, Ed had challenges that I wasn't able to come up with a "pure lsof" solution for. For example, while lsof has the "-R" option for displaying parent process ID (PPID) values, there aren't any switches in lsof to select particular processes by PPID. So we'll need to resort to some awk:

 # while :; do kill -9 `lsof -R -d cwd | awk '($3 == 8552) { print $2 }'`; sleep 1; done

Here the lsof command is outputting PPID values ("-R") in addition to the normal lsof output, and we're only outputting the lines showing the current working directory of each process ("-d cwd"). The "-d cwd" hack is a good way of ensuring that you only get one line of lsof output per process-- so we don't end up outputting the same PID multiple times and generating spurious error messages from kill. The awk code simply matches against a particular PPID value in column #3 and outputs the PID value in column #2.

Even though I had to resort to a bit of awk in the last example, you have to admit that having lsof makes this challenge unfairly easy for us Unix/Linux folks. Ahhh, lsof! How I love thee! Let me count the ways...

Tuesday, January 12, 2010

Episode #77: USB History

Ed Embarks:

Believe it or not, one of the things that we strive for in this blog is to be, not to put too fine a point on it, actually useful. We keep our musings here away from the theoretical and focused on the practical, in the hopes of helping people do their jobs better with a little bit of command-line fun. Almost* nothing gives Hal that special warm glow like e-mail from readers telling us that a recent article saved them tons of work, or allowed them to accomplish something they thought impossible. And, once Tim taught his mom to spoof e-mail addresses, the amount of such e-mail we receive has gone up a full 33% since we brought Tim on board.

Regular readers know that most of the techniques we cover here are focused on system administration and security work (such as audit, penetration testing, or, occasionally, forensics). For this episode, I'd like to address a technique that I find quite useful in my forensics work. As is our way, I'll talk about it in good ol' cmd.exe. But, I'm especially interested in seeing what magickal incantations Tim can add with PowerShell, and then I really want to see how Hal can tease this kind of information from Linux.

Sometimes, when working on a forensics case, we need to determine whether a given USB device was plugged into a given computer. Perhaps we are wondering if a user brought in an unauthorized USB token and attached it, infecting their system with malware which then spread. Or, maybe we want to get an idea if an intruder with physical access to a machine plugged in a USB hard drive so he or she could have copied large numbers of files. Sometimes, for a huge number of reasons, we need historical information about USB activities on the box for some extra proof of a perpetrators actions.

Windows stores information in the registry about every USB device plugged into the box. We can view this information with the following command:

c:\> reg query hklm\system\currentcontrolset\enum\usbstor /s

HKEY_LOCAL_MACHINE\system\currentcontrolset\enum\usbstor\Disk&Ven_SanDisk&Prod_Enterprise&Rev_6.52

HKEY_LOCAL_MACHINE\system\currentcontrolset\enum\usbstor\Disk&Ven_SanDisk&Prod_Enterprise&Rev_6.52\0B919380B2629895&0
DeviceDesc REG_SZ @disk.inf,%disk_devdesc%;Disk drive
Capabilities REG_DWORD 0x10
HardwareID REG_MULTI_SZ USBSTOR\DiskSanDisk_Enterprise______6.52\0USBS
TOR\DiskSanDisk_Enterprise______\0USBSTOR\DiskSanDisk_\0USBSTOR\SanDisk_Enterprise______6\0SanDisk_Enterprise______6\0USBSTOR\GenDisk\0GenDisk
CompatibleIDs REG_MULTI_SZ USBSTOR\Disk\0USBSTOR\RAW
ClassGUID REG_SZ {4d36e967-e325-11ce-bfc1-08002be10318}
Driver REG_SZ {4d36e967-e325-11ce-bfc1-08002be10318}\0001
Class REG_SZ DiskDrive
Mfg REG_SZ @disk.inf,%genmanufacturer%;(Standard disk drives)
Service REG_SZ disk
ConfigFlags REG_DWORD 0x0
FriendlyName REG_SZ SanDisk Enterprise USB Device

HKEY_LOCAL_MACHINE\system\currentcontrolset\enum\usbstor\Disk&Ven_SanDisk&Prod_Enterprise&Rev_6.52\0B919380B2629895&0\Device Parameters

HKEY_LOCAL_MACHINE\system\currentcontrolset\enum\usbstor\Disk&Ven_SanDisk&Prod_Enterprise&Rev_6.52\0B919380B2629895&0\Device Parameters\MediaChangeNotification

HKEY_LOCAL_MACHINE\system\currentcontrolset\enum\usbstor\Disk&Ven_SanDisk&Prod_Enterprise&Rev_6.52\0B919380B2629895&0\Device Parameters\Partmgr
Attributes REG_DWORD 0x0

HKEY_LOCAL_MACHINE\system\currentcontrolset\enum\usbstor\Disk&Ven_SanDisk&Prod_Enterprise&Rev_6.52\0B919380B2629895&0\LogConf

HKEY_LOCAL_MACHINE\system\currentcontrolset\enum\usbstor\Disk&Ven_SanDisk&Prod_Enterprise&Rev_6.52\0B919380B2629895&0\Properties
HKEY_LOCAL_MACHINE\system\currentcontrolset\enum\usbstor\Disk&Ven_SanDisk&Prod_Enterprise&Rev_6.52\0B919380B2629895&0\Control

The /s indicates that I want the reg command to recurse the Registry, showing all settings under this area. In my output, I first see an indication of the vendor and product information, which is prefaced with "Disk&Ven". I can just pull that information by piping the output through the find command in a case-insensitive (/i) fashion:

c:\> reg query hklm\system\currentcontrolset\enum\usbstor /s | find /i "Disk&Ven"

Here, you'll see stuff like SanDisk (for a lot of thumb drive memory tokens), Lexar, WD, SAMSUNG, and much more. You'll also typically see a "Prod" indication on this same line, showing the product name or number the vendor associates with the device. A quick Google search on the vendor and product data will often show you more details about the product.

After that, we get a registry entry with a unique id number associated with the device (0B919380B2629895&0 in the example above). This value is derived from a serial number from the device. With this number, I can track the usage of a single USB device across multiple machines.

Also, keep in mind that the reg key can be used remotely, so I can pull this data from target systems where I have admin credentials and SMB access, by simply prefixing the hklm above with \\MachineName\. Oh, and one more nifty bonus: The reg key is case insensitive, so I don't have to memorize the annoying CamelCase nonsense of registry keys when using reg. No, unfortunately, the cmd.exe reg command doesn't have tab autocomplete with registry paths, something that I'm sure Tim is going to extol in his PowerShell write-up.

*Don't ask. You really don't wanna know.

Tim finds his way in:

So we are digging to the registry huh? Well, that's easy. Especially with tab completion! (Insert angel AAAAAAAAHHHHH sound here) Tab completion in the registry is such a hand feature, expecially when you can't remember if the CurrentVersion key has a space in the middle (it doesn't).

PS C:\> gci -r HKLM:\SYSTEM\curr<Press tab>

PS C:\> gci -r HKLM:\SYSTEM\CurrentControSet


See how easy that was? That saved a dozen key strokes, keystrokes that can be shipped to the keystroke challenged and help them with their crops and...uh...let's get back to what we are supposed to be working on.

So we are using Get-ChildItem, the same command we use to browse the file system. However, when we use Get-ChildItem with the registry it doesn't display the values. Why? Well, you don't see the contents of files with you do a directory listing do you? Ok, that is a bit weak, but it does make sense if you think about it just right.

Another thing that might take you by surprise is that you can "change directory" into the registry. Technically "cd" is an alias for Set-Location, but it still works the same as the "cd" you know and love from the other shells.

PS C:\> cd HKLM:
PS HKLM:\> cd software
PS HKLM:\SOFTWARE> ls

Hive: HKEY_LOCAL_MACHINE\SOFTWARE

SKC VC Name Property
--- -- ---- --------
1 0 ActiveTouch {}
4 0 Adobe {}
1 0 AGNS {}
2 0 Alps {}
...


Now that you have seen some of the mind-bending features of PowerShell let try to recreate what Ed did.

PS C:\> gci -r HKLM:\SYSTEM\CurrentControlSet\Enum\USBSTOR | select name
Name
----
HKEY_LOCAL_MACHINE\system\currentcontrolset\enum\usbstor\Disk&Ven_SanDisk&Prod_Enterprise&Rev_6.52
HKEY_LOCAL_MACHINE\system\currentcontrolset\enum\usbstor\Disk&Ven_SanDisk&Prod_Enterprise&Rev_6.52\0B919380B2629895&0
HKEY_LOCAL_MACHINE\system\currentcontrolset\enum\usbstor\Disk&Ven_SanDisk&Prod_Enterprise&Rev_6.52\0B919380B2629895&0\Device Parameters
HKEY_LOCAL_MACHINE\system\currentcontrolset\enum\usbstor\Disk&Ven_SanDisk&Prod_Enterprise&Rev_6.52\0B919380B2629895&0\Device Parameters\MediaChangeNotification
HKEY_LOCAL_MACHINE\system\currentcontrolset\enum\usbstor\Disk&Ven_SanDisk&Prod_Enterprise&Rev_6.52\0B919380B2629895&0\Device Parameters\Partmgr
HKEY_LOCAL_MACHINE\system\currentcontrolset\enum\usbstor\Disk&Ven_SanDisk&Prod_Enterprise&Rev_6.52\0B919380B2629895&0\LogConf
HKEY_LOCAL_MACHINE\system\currentcontrolset\enum\usbstor\Disk&Ven_SanDisk&Prod_Enterprise&Rev_6.52\0B919380B2629895&0\Properties
HKEY_LOCAL_MACHINE\system\currentcontrolset\enum\usbstor\Disk&Ven_SanDisk&Prod_Enterprise&Rev_6.52\0B919380B2629895&0\Control
...



We use -r, which is short for Recurse, in order to show all the keys below our current location. Now let's find all of the devices that have been plugged in.

PS C:\> gci -recurse HKLM:\SYSTEM\CurrentControlSet\Enum\USBSTOR |
? { $_.PSPath -match ".*Disk&Ven[^\\]*$" } | select PSChildName

Disk&Ven_SanDisk&Prod_Enterprise&Rev_6.52
Disk&Ven_&Prod_USB_Flash_Memory&Rev_PMAP
Disk&Ven_Generic&Prod_&Rev_6000
Disk&Ven_IronKey&Prod_Secure_Drive&Rev_1.00
...


First, we get the results of Get-ChildItem. Next, we filter the results based on a regular expression. The regular expression looks for Disk&Ven followed by any characters that are NOT a backslash. Finally, we select the name of the node using the PSChildName property. The PSChildName is the name of the node in question. The property name leaves a little to be desired, but it works.

We got the Vender &: Product info, and we can use a similar command to get all the unique ids.

PS C:\> gci -recurse HKLM:\SYSTEM\CurrentControlSet\Enum\USBSTOR |
? { $_.PSParentPath -match ".*Disk&Ven[^\\]*$" } | select PSChildName

0B919380B2629895
...


The difference between the commands is subtle. In the second command we use our regular expression on the parent item, not the current item. This takes us one step deeper and gets us the unique id.

PowerShell allows us to use the .NET framework, which gives us access to a lot of cool tools. While I wasn't able to finish it for this episode, we can use the Windows API in order to get the last write time of the registry keys. It isn't pretty, but it works. Much like some friends of mine.

Hal is happy to come along

I was really happy when Ed proposed this idea because the Linux side was something I'd been meaning to research for quite some time now. But now Ed just let me in on where all of those fan emails are coming from and I'm all depressed. Ah well, nothing to do but get really geeky in order to take my mind of things.

So your first question might be, how exactly does one research something like this? I went for the "brute force and massive ignorance" approach, using a technique similar to the find trick I discussed back in Episode #29. I simply used touch to update the timestamp on an empty file, plugged in the USB device, and then did a "find ... -newer ..." to locate any files that had changed as a result of the device being inserted. So the command line looked like:

# touch /root/timestamp
[... plug in USB device ...]
# find / -newer /root/timestamp

The results were actually quite interesting and fairly consistent across the two Linux distros I tested against: CentOS 5.4 (a Red Hat derivative running kernel v2.6.18) and Ubuntu 9.10 aka "Karmic Koala" (a Debian derivative running kernel v2.6.31). Here's what I found:


  1. When the device is plugged in, it inherits the next available "sd" device name. So on my laptop, for example, the internal drive is "sda", so the first USB device gets "sdb", the second "sdc", and so on. Also a number of symlinks are created under /dev pointing to the sd device for the USB drive that was just inserted. Here's some of the pathnames to show you what I'm taking about:

    /dev/disk/by-uuid
    /dev/disk/by-uuid/80a5317f-8e4c-4c83-88ea-b5192b442f97
    /dev/disk/by-path
    /dev/disk/by-path/pci-0000:02:02.0-usb-0:1:1.0-scsi-0:0:0:0-part1
    /dev/disk/by-path/pci-0000:02:02.0-usb-0:1:1.0-scsi-0:0:0:0
    /dev/disk/by-id
    /dev/disk/by-id/usb-Kingston_DataTraveler_2.0_8990000000000000000000B0-part1
    /dev/disk/by-id/usb-Kingston_DataTraveler_2.0_8990000000000000000000B0
    /dev/.udev/links
    /dev/.udev/links/disk\x2fby-uuid\x2f80a5317f-8e4c-4c83-88ea-b5192b442f97
    /dev/.udev/links/disk\x2fby-uuid\x2f80a5317f-8e4c-4c83-88ea-b5192b442f97/b8:17
    /dev/.udev/links/disk\x2fby-path\x2fpci-0000:02:03.0-usb-0:1:1.0-scsi-0:0:0:0-part1
    /dev/.udev/links/disk\x2fby-path\x2fpci-0000:02:03.0-usb-0:1:1.0-scsi-0:0:0:0-part1/b8:17
    /dev/.udev/links/disk\x2fby-id\x2fusb-Kingston_DataTraveler_2.0_8990000000000000000000B0-0:0-part1
    /dev/.udev/links/disk\x2fby-id\x2fusb-Kingston_DataTraveler_2.0_8990000000000000000000B0-0:0-part1/b8:17
    /dev/.udev/links/disk\x2fby-path\x2fpci-0000:02:03.0-usb-0:1:1.0-scsi-0:0:0:0
    /dev/.udev/links/disk\x2fby-path\x2fpci-0000:02:03.0-usb-0:1:1.0-scsi-0:0:0:0/b8:16
    /dev/.udev/links/disk\x2fby-id\x2fusb-Kingston_DataTraveler_2.0_8990000000000000000000B0-0:0
    /dev/.udev/links/disk\x2fby-id\x2fusb-Kingston_DataTraveler_2.0_8990000000000000000000B0-0:0/b8:16

    You can see the manufacturer name for the device ("Kingston DataTraveler 2.0"), the serial number of the device ("8990000000000000000000B0"), and the UUID associated with the file system on the device ("80a5317f-8e4c-4c83-88ea-b5192b442f97") in the path names above. And if you look closely, you'll see that there's a single partition on the device (in addition to the links that point to the device as a whole). If you wanted to, you could also run the mount command (see Episode #59) to get more information about this file system.

    The only problem with all of this information is that it's completely ephemeral. As soon as the drive is removed, all of this information goes away. So from an "after the fact" sort of forensic perspective, it's not really all that useful.


  2. In addition to the device files and links under /dev, the OS also creates some information files under /dev/.udev/db. CentOS calls these files names like "/dev/.udev/db/block@sdb" and "/dev/.udev/db/block@sdb@sdb1", while the newer kernel on my Ubuntu box uses names like "/dev/.udev/db/block:sdb" and "/dev/.udev/db/block:sdb1", but the basic information in the files is pretty much the same. Here's the data in the "/dev/.udev/db/block:sdb1" file from my Ubuntu system:

    N:sdb1
    S:block/8:17
    S:disk/by-id/usb-Kingston_DataTraveler_2.0_8990000000000000000000B0-0:0-part1
    S:disk/by-path/pci-0000:02:03.0-usb-0:1:1.0-scsi-0:0:0:0-part1
    S:disk/by-uuid/80a5317f-8e4c-4c83-88ea-b5192b442f97
    W:51
    E:ID_VENDOR=Kingston
    E:ID_VENDOR_ENC=Kingston
    E:ID_VENDOR_ID=0951
    E:ID_MODEL=DataTraveler_2.0
    E:ID_MODEL_ENC=DataTraveler\x202.0
    E:ID_MODEL_ID=1603
    E:ID_REVISION=1.00
    E:ID_SERIAL=Kingston_DataTraveler_2.0_8990000000000000000000B0-0:0
    E:ID_SERIAL_SHORT=8990000000000000000000B0
    E:ID_TYPE=disk
    E:ID_INSTANCE=0:0
    E:ID_BUS=usb
    E:ID_USB_INTERFACES=:080650:
    E:ID_USB_INTERFACE_NUM=00
    E:ID_USB_DRIVER=usb-storage
    E:ID_PATH=pci-0000:02:03.0-usb-0:1:1.0-scsi-0:0:0:0
    E:ID_FS_UUID=80a5317f-8e4c-4c83-88ea-b5192b442f97
    E:ID_FS_UUID_ENC=80a5317f-8e4c-4c83-88ea-b5192b442f97
    E:ID_FS_SEC_TYPE=ext2
    E:ID_FS_VERSION=1.0
    E:ID_FS_TYPE=ext3
    E:ID_FS_USAGE=filesystem
    E:DKD_PARTITION=1
    E:DKD_PARTITION_SCHEME=mbr
    E:DKD_PARTITION_NUMBER=1
    E:DKD_PARTITION_TYPE=0x83
    E:DKD_PARTITION_SIZE=4018885632
    E:DKD_PRESENTATION_NOPOLICY=0

    There isn't a whole lot here we couldn't have pulled out of the link names shown above, but this format is definitely a lot more readable. Unfortunately, this file is also ephemeral and gets cleaned up as soon as the drive is disconnected from the system.

    At this point you're probably asking yourself if there's any forensic technique we could use to recover the deleted. The bad news is that both the links and the db files created by the device insertion are created in a memory-based file system:

    $ df /dev/.udev
    Filesystem 1K-blocks Used Available Use% Mounted on
    udev 1992768 276 1992492 1% /dev
    $ df /dev/disk
    Filesystem 1K-blocks Used Available Use% Mounted on
    udev 1992768 276 1992492 1% /dev

    Assuming the system hasn't been rebooted since the device was inserted, I suppose it's possible that you might find some strings still floating around in the memory image of the system (or possibly in the disk-based swap area). But frankly, I don't hold out much hope for you. At least the db file has a nice regular structure to assist in searching.


  3. There are, however, some more permanent records of the device having been connected to the system. Any of you who've been using Linux for a while know that when you hook up a removable device to the system, the Nautilus agent in the standard Gnome desktop will position an icon on the desktop that represents the device. You may have noticed that if you disconnect and later reconnect the same device, the icon will always re-appear in the same spot on the desktop. Nautilus keeps track of this in a file in your home directory.

    On CentOS, the file is called "~/.nautilus/metafiles/x-nautilus-desktop\:%2F%2F%2F.xml" and looks like this:

    <?xml version="1.0"?>
    <directory><file name="trash" timestamp="1200596020" icon_position="64,182"/>
    <file name="home" timestamp="1200596020" icon_position="64,102"/>
    <file name="computer" timestamp="1200596020" icon_position="64,22"/>
    <file name="CD-ROM%20Drive.volume" timestamp="1206558600" icon_position="64,282"/>
    <file name="CD-RW%2FDVD%C2%B1RW%20Drive.volume" timestamp="1212709603" icon_position="64,282"/>
    <file name="465.8%20GB%20Volume.volume" timestamp="1263069239" icon_position="64,282"/>
    <file name="USB%20DISK%202.0.volume" timestamp="1263070714" icon_position="64,282"/>
    <file name="Kingston%20DataTraveler%202.0.volume" timestamp="1263073745" icon_position="64,582"/></directory>

    What's interesting to me about this file is the time stamp value. The time stamp is in the Unix "seconds since Jan 1, 1970" format, but this can easily be converted to something human readable, if my co-authors will permit me to use a little bit of Perl:

    $ perl -e 'print scalar(localtime(1263073745)), "\n";'
    Sat Jan 9 13:49:05 2010

    Loyal reader Adrian Shaw wrote in to point out that the GNU date command will actually allow us to do this conversion without resorting to Perl:

    $ date -d @1263073745
    Sat Jan 9 13:49:05 PST 2010

    So this is an awesome trick for Linux users, but don't expect it to work on other Unix-like operating systems unless you install the standard GNU tools.


    The only problem is that this time stamp value gets updated if the user moves the icon for the device around on their desktop. But since users don't tend to do this very often, it usually reflects the time the device was first connected to the system.

    On my Ubuntu box running a newer version of Gnome, the Nautilus agent actually creates a separate file for each new device. For example, the information for the "4.0 GB Filesystem" on my Kingston device ends up in a file called "~/.gconf/apps/nautilus/desktop-metadata/4@46@0@32@GB@32@Filesystem@46@volume/%gconf.xml", which looks like this:

    <?xml version="1.0"?>
    <gconf>
    <entry name="nautilus-icon-position-timestamp" mtime="1263074552" type="string">
    <stringvalue>1263074552</stringvalue>
    </entry>
    <entry name="icon-scale" mtime="1263074552" type="string">
    <stringvalue>1</stringvalue>
    </entry>
    <entry name="nautilus-icon-position" mtime="1263074552" type="string">
    <stringvalue>64,10</stringvalue>
    </entry>
    </gconf>

    What's nice about the newer file format is that there are multiple time stamps in the file. The "icon-scale" time stamp does not get updated if the user repositions the icon on their desktop, and therefore may give you a truer reading on when the device was first connected to the system.

    The only problem with data in a user's home directory, is that an astute user may clean up after themselves if they're doing something nefarious. And they may use a secure deletion tool (see Episode #32) to prevent recovery of the data.


  4. The last place to find information about the device is in the system logs. Both the kernel and the file system drivers log information about the device being inserted.

    Here's the log from my CentOS system when the device is inserted:

    Jan  9 13:51:22 localhost kernel: usb 1-1: new high speed USB device using ehci_hcd and address 8
    Jan 9 13:51:22 localhost kernel: usb 1-1: configuration #1 chosen from 1 choice
    Jan 9 13:51:22 localhost kernel: scsi7 : SCSI emulation for USB Mass Storage devices
    Jan 9 13:51:28 localhost kernel: Vendor: Kingston Model: DataTraveler 2.0 Rev: 1.00
    Jan 9 13:51:28 localhost kernel: Type: Direct-Access ANSI SCSI revision: 02
    Jan 9 13:51:28 localhost kernel: SCSI device sdb: 7856128 512-byte hdwr sectors (4022 MB)
    Jan 9 13:51:28 localhost kernel: sdb: Write Protect is off
    Jan 9 13:51:28 localhost kernel: sdb: assuming drive cache: write through
    Jan 9 13:51:28 localhost kernel: SCSI device sdb: 7856128 512-byte hdwr sectors (4022 MB)
    Jan 9 13:51:28 localhost kernel: sdb: Write Protect is off
    Jan 9 13:51:28 localhost kernel: sdb: assuming drive cache: write through
    Jan 9 13:51:28 localhost kernel: sdb: sdb1
    Jan 9 13:51:28 localhost kernel: sd 7:0:0:0: Attached scsi removable disk sdb
    Jan 9 13:51:28 localhost kernel: sd 7:0:0:0: Attached scsi generic sg1 type 0
    Jan 9 13:51:29 localhost kernel: kjournald starting. Commit interval 5 seconds
    Jan 9 13:51:29 localhost kernel: EXT3 FS on sdb1, internal journal mode.
    Jan 9 13:51:29 localhost hald: mounted /dev/sdb1 on behalf of uid 500

    And here's what happens when the device is removed:

    Jan  9 14:00:43 localhost hald: unmounted /dev/sdb1 from '/media/disk' on behalf of uid 500
    Jan 9 14:00:47 localhost kernel: usb 1-1: USB disconnect, address 8

    You'll notice that the system logs the UID of the user on the console of the system, as well as giving you information about the device name (no serial number though), and the size and type of file system found on the device.

    The information logged on my Ubuntu box with a newer kernel version is slightly different:

    Jan  9 14:02:26 ubuntu kernel: [ 2350.984617] usb 1-1: new high speed USB device using ehci_hcd and address 5
    Jan 9 14:02:26 ubuntu kernel: [ 2351.365130] usb 1-1: configuration #1 chosen from 1 choice
    Jan 9 14:02:26 ubuntu kernel: [ 2351.377732] scsi6 : SCSI emulation for USB Mass Storage devices
    Jan 9 14:02:26 ubuntu kernel: [ 2351.378321] usb-storage: device found at 5
    Jan 9 14:02:26 ubuntu kernel: [ 2351.378330] usb-storage: waiting for device to settle before scanning
    Jan 9 14:02:31 ubuntu kernel: [ 2356.399268] usb-storage: device scan complete
    Jan 9 14:02:31 ubuntu kernel: [ 2356.404119] scsi 6:0:0:0: Direct-Access Kingston DataTraveler 2.0 1.00 PQ: 0 ANSI: 2
    Jan 9 14:02:31 ubuntu kernel: [ 2356.406027] sd 6:0:0:0: Attached scsi generic sg2 type 0
    Jan 9 14:02:31 ubuntu kernel: [ 2356.438414] sd 6:0:0:0: [sdb] 7856128 512-byte logical blocks: (4.02 GB/3.74 GiB)
    Jan 9 14:02:31 ubuntu kernel: [ 2356.441587] sd 6:0:0:0: [sdb] Write Protect is off
    Jan 9 14:02:31 ubuntu kernel: [ 2356.441594] sd 6:0:0:0: [sdb] Mode Sense: 23 00 00 00
    Jan 9 14:02:31 ubuntu kernel: [ 2356.441599] sd 6:0:0:0: [sdb] Assuming drive cache: write through
    Jan 9 14:02:31 ubuntu kernel: [ 2356.463161] sd 6:0:0:0: [sdb] Assuming drive cache: write through
    Jan 9 14:02:31 ubuntu kernel: [ 2356.463231] sdb: sdb1
    Jan 9 14:02:31 ubuntu kernel: [ 2356.495857] sd 6:0:0:0: [sdb] Assuming drive cache: write through
    Jan 9 14:02:31 ubuntu kernel: [ 2356.495900] sd 6:0:0:0: [sdb] Attached SCSI removable disk
    Jan 9 14:02:32 ubuntu kernel: [ 2356.945441] kjournald starting. Commit interval 5 seconds
    Jan 9 14:02:32 ubuntu kernel: [ 2356.957462] EXT3 FS on sdb1, internal journal
    Jan 9 14:02:32 ubuntu kernel: [ 2356.957535] EXT3-fs: mounted filesystem with writeback data mode.
    [... device is removed ...]
    Jan 9 14:12:17 ubuntu kernel: [ 2942.693376] usb 1-1: USB disconnect, address 5

    You'll notice that the Ubuntu system isn't logging the UID of the user on the console at the time the drive is connected. But you could always get this information by running "last" to get the last login history of the system. The other problem, however, is that the newer kernel logs use a slightly different format for reporting the device manufacturer info, so there isn't a simple grep expression that will work when parsing logs from multiple different kernel versions.


Phew! That's a whole lot of information! The short summary, though, is that the only data that may exist long after the drive has been removed is the limited data in the system logs plus possibly some data in user home directories. I have to say that Windows leaves a much more detailed forensic trail. Whether you consider this a "feature" or not is up to you. I suppose it depends on whether you're representing the prosecution or the defense.

Tuesday, January 5, 2010

Episode #76: Say Hello to My Little Friend Netstat

Ed greets 2010:

Many times on our little blog here, I've spoken appreciatively of some of my favorite Windows command-line friends, including wmic, netsh, and sc. Yes, I've fought the desire to personify them, but I can't help but think of them as buddies... Buddies with annoying quirks, a bad attitude, and flaws that sometimes let you down, but compadres nonetheless.

While spending some time over the holidays hanging out with these trustworthy sidekicks, something occurred to me: We haven't spent nearly enough time on this blog with the good ol' netstat command. Sure, we touched upon him a little bit in analyzing protocol statistics, checking out whether we've got a SYN flood, and even as an example of ways to get help from Windows commands. But, so far, I've truly let one of my most faithful and useful commands down by not giving him the spotlight he deserves. So, without further adieu, let's look at some really useful invocations of netstat:

Detecting when a scan reaches a given target box: So, Mike Poor and I were at a client's facilities conducting a large-scale vulnerability scan of thousands of hosts. Mike was running the scanning tool from the corporate headquarters, while I was located at another building sitting with the client in front of one of their Windows servers that was included in Mike's scan. The client and I wanted to know when Mike's scan would reach us. I asked the client if I could install a sniffer on their server, or connect my laptop to a span port on the switch so I could use my own sniffer to see when Mike's scan reached us. "No dice," said the client, who wasn't authorized to let us install any software or get access to the switch. I asked the client if we could logon to the Windows server and run a simple single netstat command that would show us when Mike's scan reached us, with approximately 1 second accuracy. He said, "Sure thing," so I ran:

C:\> netstat -na | find /i "Listening"

Here, I'm just checking to make sure there is at least one listening TCP port, piping netstat's output through the find command in a case-insensitive fashion (/i) because I didn't want to hold down the shift-lock key to type LISTENING. Sure enough, on this Windows server, I saw several listening ports, including TCP 445. When Mike's scanner reached us, it would open a connection to that port.

Then, I ran:
C:\> netstat -na 1 | find "[MikePoorIPaddr]"

That little space followed by a 1 means that Netstat will re-run approximately every 1 second. I scrape through its output looking for the string of Mike's address. If I see no output, there is no connection from Mike. When I start seeing output, I'll know that Mike's scan has reached me. Sure, this isn't perfect. It's quite possible that the scanning tool will make and drop connections so quickly that they will fall in between the 1-second interval we're working with here. However, that's quite unlikely. And, sure enough, we were able to determine when Mike's scan reached us, and finished with us. We then headed for lunch.

Detecting when a piece of malware starts interacting with the network: I was in my lab, fully clothed, analyzing some malware. After it finished some initialization actions, this little gem would start listening on TCP port 47145. Again, I wanted to know when it would start to listen, with 1-second accuracy... I ran:

C:\> netstat -na 1 | find ":47145"

I then wanted to go a little further, and find when the bad guy actually made a connection to this listener. For that, I executed:

C:\> netstat -na 1 | find ":47145" | find /i "Established"

Finding the mysterious cause of an ICMP Host Unreachable Message: A student who was taking my SANS 504 class told me that he was getting a mysterious ICMP Host Unreachable message coming from their border firewall back into their network a couple times per day, and he wanted to know what was triggering it. I asked him what the destination address of the packet was. It was one of their Windows 2003 servers. So, the Windows 2003 server was trying to send a message out through the firewall, but the firewall was responding that it didn't know how to reach the destination machine. The student wanted to know which program on the Win2K3 box was sending the original message. I had him run:

C:\> netstat -nao 1 | find "[DestIPaddr]"

As long as that program was trying to make a TCP connection, we'd see output from netstat for the half open connection, and could see the process ID number (the -o in netstat gives me the PID of the process using the port). We were then able to run the wmic command to get all kinds of info about that process:

C:\> wmic process where processid="[pid]" list full

Playing process whack-a-mole: I was playing a king-of-the-hill style capture the flag game, and my adversaries were trying to listen on a given TCP port on the target box, which I had already compromised. I wanted to simply kill any process that started on that port as quickly as possible. The netstat invocations listed above are quite nice, but they don't really help much for this need. You see, putting the integer after the netstat invocation makes netstat run continuously, showing us its output every 1 second. We can scrape through that output with the find command, but we cannot then run any other command after that, such as a taskkill. That's because netstat never stops running, so we cannot kickoff another command. For this, we'll have to wrap things in a FOR /L loop to get our continuity of operation, and in a FOR /F loop, to parse out the PID from our netstat output, as follows:

C:\> for /L %i in (1,0,2) do @for /f "tokens=5" %j in ('netstat -nao ^| find ^":47145^"')
do @taskkill /f /PID %j
SUCCESS: The process with PID 3152 has been terminated.
SUCCESS: The process with PID 2432 has been terminated.
SUCCESS: The process with PID 4068 has been terminated.

Here, I've simply set up a FOR /L loop to count from 1 to 2 in steps of zero to implement a continuous loop. Then, at each iteration through the loop, I run a FOR /F loop set to parse the fifth column (the PID) from the output of a netstat -nao command, whose output itself is filtered by the find command looking for our port. Once I've got my PID, I then use taskkill to forcefully (/f) kill that PID. The evil listening process cannot listen for very long now.

Note that I've eliminated the 1 second delay -- this thing will run as fast as it can, useful in a CtF game as long as you aren't worried about being a performance hog. If you want a delay, we can simply add "& ping -n 2 127.0.0.1>nul" inside our FOR /F loop to ping ourselves twice (which takes about 1 second), as follows:

C:\> for /L %i in (1,0,2) do @(ping -n 2 127.0.0.1>nul & for /f "tokens=5" 
%j in ('netstat -nao ^| find ^":47145^"') do @taskkill /f /PID %j)

Create a real-time network sentinel by adding a date and timestamp and dumping process info: Now, let's really get jiggy with it. Instead of killing processes or using the above commands in an interactive mode, wouldn't it be nice if we could run a command that would just watch a given TCP port, and when it finds something using that port, it displays the date/time and various vital aspects of that process? Of course it would!

But, we'd only want it to print information when a new processID starts to listen on the given port. That way, if one process starts and listens for a while, and then stops, and another starts up, we would capture on our output the time and details of each process that interacted with the port, a diligent sentinel gathering data for us. That would allow us to run a command that can watch for given network activity over night, and then come back to see what happened and when it happened later. I use this concept and the resulting command a lot during investigations. Check this out:

C:\> cmd.exe /v:on /c "set status=0 & for /l %i in (1,0,2) do @for /f "tokens=5"
%j in ('"netstat -nao ^| find ":47145""') do @if NOT !status!==%j echo !date!
!ti
me! & wmic process where processid="%j" list full & set status=%j"

Sun 01/03/2010 5:36:47.79

CommandLine=nc -l -p 47145
CSName=FRED2
Description=nc.exe
ExecutablePath=C:\tools\netcat\nc.exe
ExecutionState=
Handle=3116
HandleCount=33
InstallDate=
KernelModeTime=0
MaximumWorkingSetSize=1413120
MinimumWorkingSetSize=204800
Name=nc.exe
OSName=Microsoft Windows XP Professional|C:\WINDOWS|\Device\Harddisk0\Partition1
OtherOperationCount=208
OtherTransferCount=124718
PageFaults=475
PageFileUsage=540672
ParentProcessId=3584
PeakPageFileUsage=540672
PeakVirtualSize=14987264
PeakWorkingSetSize=1953792
Priority=8
PrivatePageCount=540672
ProcessId=3116
QuotaNonPagedPoolUsage=2376
QuotaPagedPoolUsage=29780
QuotaPeakNonPagedPoolUsage=2376
QuotaPeakPagedPoolUsage=29924
ReadOperationCount=5
ReadTransferCount=15977
SessionId=0
Status=
TerminationDate=
ThreadCount=1
UserModeTime=300432
VirtualSize=14987264
WindowsVersion=5.1.2600
WorkingSetSize=1953792
WriteOperationCount=1
WriteTransferCount=72

In this command, I start by running a cmd.exe with /v:on to enable delayed variable expansion. I'll need that, because I need to store status to track if the PID using the port changes. I then set my initial status to zero, since no process should have that PID. I then start a FOR /L loop to run everything continuously. Then, I run a FOR /F loop to invoke my netstat command, piping its output through the find command to look for that port (47145, although you could also look for an IP address, as shown earlier in the article). I take the fifth column of output, the PID, and store it in iterator variable %j. Then, in the do clause of my FOR /F loop, I use an IF command to see if my status has changed (if the status variable is NOT the same as my PID, it has changed since the last iteration). When I detect such a change, I print out the date and time (note that !status!, !date!, and !time! have the exclamation points because I want their values to float over time with delayed variable expansion). I also run wmic to get full details of the given process. Finally, I set my status to the current PID. That way, I can check to see if it changes in the next go-round. This may look complicated, but it is darn useful. I'm simply dumping output to the screen here, but you could append it to a results file if you'd like by using >>results.txt at the end.

As you can see, there are a lot of fun things my little friend netstat can do using these building blocks.

Hal jumps in:

It's always been interesting to me how much Windows completely ripped off the Windows netstat command is like the Unix version. With some minor changes, most of Ed's examples work perfectly well on most Unix systems. Obviously, you end up using "grep" instead of "find" to filter the output, but the netstat options are almost 100% compatible. Here's the Unix equivalent of Ed's first netstat command as an example of what I'm talking about:

$ netstat -na | grep -i listen

Note that the output that Ed wants to match in this example is "LISTEN" in the Unix universe, not "LISTENING".

One important difference between Unix and Windows netstat is the way you get netstat to run continuously rather than producing one set of output and then exiting. With the Unix version, you need to use "-c" to specify continuous mode:

$ netstat -nac | grep <Mike's IP addr>

The interval for "-c" is fixed at one second-- you can't specify your own interval like you can with Windows. If you want to use a less frequent interval, you'll have to write your own loop and use "sleep", like so:

$ while :; do netstat -na | grep <Mike's IP addr>; sleep 5; done

Frankly, though, I think I'd just use our old friend the "watch" command instead of either "netstat -c" or the loop option:

$ watch -n 1 'netstat -na | grep <Mike's IP addr>'

The watch command is not only quicker to type than the loop, it allows us to easily choose our preferred monitoring interval, which we can't do with "netstat -c".

But really I wouldn't use netstat for this at all as long as I was on a machine that had the lsof command installed. With lsof, not only can I do this in less keystrokes, but I'll also get more information than I would from the netstat. lsof is *my* little friend and the friend of Unix admins everywhere.

The downside is that you usually have to be root to get complete output from lsof:

# watch -n 1 lsof -nPi @<Mike's IP addr>

"lsof -i @<ipaddr>" show you all connections related to the specified IP address. As with netstat, "-n" suppresses IP address to hostname mappings. However, while "netstat -n" also suppresses port number to port name mappings, lsof wants you to use the "-P" option to show numeric port numbers.

We can also use lsof to do something similar to Ed's port monitoring examples:

# watch -n 1 lsof -nPi :47145

Notice that you prefix port numbers with a colon when using lsof, while IP addresses are prefixed with "@" as we saw above. This allows us to stack IP address and port combinations to be even more specific. For example, suppose I was interested in SSH connections to a particular box, I could write:

# watch -n 1 lsof -nPi tcp@<ipaddr>:22

The above example also shows how to specify a particular protocol type (usually "tcp" or "udp") along with the IP address and port information. If I just wanted SSH connections to all IP addresses, I could do:

# watch -n 1 lsof -nPi tcp:22

Another advantage to using lsof instead of netstat is that lsof's "-t" option makes "process whack-a-mole" dead easy:

# while :; do kill -9 `lsof -t -i :47145`; done

As we discussed back in Episode #69, "lsof -t" just outputs the PIDs of the matching processes. This means it's easy to kill those processes by using backticks and the kill command as we're doing here.

Ed's final monitoring loop is interesting. Here's how I'd do something similar on a Linux machine:

# while :; do
pid=`lsof -ti :47145`
[[ "X$pid" == "X" || "X$pid" == "X$lastpid" ]] && continue
lastpid=$pid
echo -n '*** '
date
ps e --cols 1600 -fl -p $pid
lsof -p $pid
done

I'm first getting the PID of the process we're interested in using "lsof -t...". If there's currently no PID associated with this port, or if the current PID is the same as the previous time we ran the loop then we just start the loop all over again.

Otherwise I update the value of $lastpid and output some info about the new process. You could choose to use whatever commands you want, but here I start by throwing out some asterisks with "echo -n" so that it's easier to find the start of each section of output. Then I follow that up with the date command to get a timestamp. Since I used "echo -n", the date output will appear on the same line as the asterisks.

Then I use the ps command to dump some information about the process. "ps e" will dump any environment variables that are set in the process context. Since this output tends to be long, I also add the "--cols 1600" option to specify the screen width as a larger value so that the output doesn't get truncated (on narrower displays, the lines will simply wrap). I also use "-fl" to get the most detailed output possible. The "-p" option is used to specify the PID we're interested in. While the ps command will give us plenty of output, I'm also using lsof to dump information about all of the open files associated with the process.

It's worth noting that both Ed's loop and mine assume that only one process at a time is ever going to be associated with the port we're monitoring. But if the port is a back-door login port, you might actually have multiple connections to the port at the same time. I'd finesse this in bash by using an array to track multiple PIDs at the same time, but I think it would be next to impossible for Ed to deal with this in cmd.exe.

Tim leaps in:

Of course we can use netstat in PowerShell, but it would be mostly redundant (with the exception of the all of Ed's For Loops). Also, Hal seems upset about Windows "stealing" netstat so let's try the same thing in PowerShell without using netstat. Without netstat we will have to use the .NET framework available to us through PowerShell.

Here is the PowerShell and .Net equivalent of the first netstat command.

PS C:\> ([Net.NetworkInformation.IPGlobalProperties]::
GetIPGlobalProperties()).GetActiveTcpListeners()

AddressFamily Address Port
------------- ------- ----
InterNetwork 0.0.0.0 80
InterNetwork 0.0.0.0 135
InterNetwork 0.0.0.0 445
InterNetwork 192.168.1.5 139
...


Most of the information we are going to need is accessed via the IPGlobalProperties object's methods and properties. Since we are going to be using it a lot let's dump it into a variable so we can save keystrokes in future commands.

PS C:\> $a = [Net.NetworkInformation.IPGlobalProperties]::GetIPGlobalProperties()


Now we can run the first command again like this:

PS C:\> $a.GetActiveTcpListeners()


The name of this method is pretty self explanatory, it returns the active TCP listeners. Not much to talk about, so we'll move on.

The next task is to find out when Mike connects.

PS C:\> while (1) {$a.GetActiveTcpConnections() | ?
{$_.RemoteEndPoint.Address -eq "[MikePoorIPaddr]"} }


We use a while loop that will run forever since "1" is always true (except for very small values of 1). Then we get the active TCP connections and look for Mike's IP address.

This will really hit the processor, so we probably should put in a delay of a at least a half a second.

PS C:\> while (1) {$a.GetActiveTcpConnections() | ?
{$_.RemoteEndPoint -like "[MikePoorIPaddr]:*";
sleep -milliseconds 500}}


...or maybe two seconds

PS C:\> while (1) {$a.GetActiveTcpConnections() | ?
{$_.RemoteEndPoint -like "[MikePoorIPaddr]:*";
sleep 2}}


So no we move on to the malware that Ed was analyzing. In the example, the port to be monitored is TCP 47145.

PS C:\> $a.GetActiveTcpListeners | ? {$_.Port -eq 47145}


We get the active tcp listeners and then filter the results for connections where the local port is 47145.

Now we have the problem of killing the process. Without using netstat we don't have a way to determine the PID. It looks like we will have to cave in an use our old friend netstat (even though Hal is trying to poison this friendship).

PS C:\> netstat -ano | ? {$_ -like "*:47145 *"} |
% { $_ -match "\d+$"; stop-process $matches[0] }


The output of netstat is filtered for connections on port 47145. Using a regular expression inside the ForEach-Object loop we look for a number at the end of the line. In regular-expression-land \d+ means at least one numeric digit and $ means it has to be at the end of the line. The number that we found is represented by the $matches object so we use the first and only match, $matches[0], with stop-process to kill it.

Ed's grand finale, the monitoring loop done in PowerShell.

PS C:\> while(1) {netstat -ano | ? {$_ -like "*:47145 *"} |
% {
$_ -match "\d+$";
Get-Process -id $matches[0] | Format-List *;
(Get-Process -id $matches[0]).WaitForExit()
}
}


While it ain't pretty, I think we can safely say that the PowerShell version is the easiest to read of the bunch (finally), but I did add a few extra line breaks and spaces to make it even easier to read. How does it work?

First, we have our friendly infinite while loop. Nested inside the loop is our netstat command and our filter to search for port 47145. As before, we use the match operator to find the PID. Then we get all of the properties for the process. Piping the Get-Process command into Format-List * will display all the properties, not just the most commonly used properties. Finally we wait for the process to end and then our loop continues from the beginning. By using the WaitForExit method we know when that process dies and we can look for a new process. We don't have to worry about finding, keeping track of, and comparing the PID. It makes it much easier and cleaner. It also has the added benefit of being very light on system resources while it is waiting.