Episode #132: Enigma

Tim goes stealth-mode:

We are always looking for new topic ideas from you readers, but this week we only received one email and trying to use it was difficult.

Hey there,

I've been searching for kung fu teachers, and I was excited to find you.

It's too hard to find trustworthy, quality service providers, and <dumb site> is changing that. We're growing really fast, we need more kung fu teachers, and I think you're a perfect fit!

Posting on <dumb site> is a great way to advertise yourself and it's completely free for service providers like you.

All you need to do to post your information is visit: <dumb site's URL>

Thanks!

~Heather

Apparently Heather didn't know that we don't do physical kung fu. We do try to kick bit booty, but we leave any physical effort to people in another line of work, like accountants. Oh, and the site name was redacted because I don't want to give them any business.

Since that didn't give us any great ideas, I came up with one on my own using some really secure encryption.

One method of encryption involves the use of On-time Pads. When used correctly it is impossible to crack. "If the key is truly random, as large as or greater than the plaintext, never reused in whole or part, and kept secret, the ciphertext will be impossible to decrypt or break without knowing the key." That is different from taking billions of years to crack AES, this is uncrackable!

Let's use this technique to send a message to Hal:
I secretly teach Kung Fu

Hal and I meet up, and exchange one-time pads. I generated the pad by flipping a coin, where 1 is head and 0 is tails. The 1s and 0s were converted to bytes to create our encryption key.

PS C:\> $key = [byte[]](0x70,0xB3,0xDE,0xC0,0xDE,0xDF,0xAC,0xE1,0x7B,
  0xA5,0x41,0x51,0x33,0x7E,0xD5,0xC1,0x11,0xDF,0x00,0x15,0xDE,0xAD,0xD0,0x0D)

With this clear text...

PS C:\> $cleartext = "I secretly teach Kung Fu"

...we do a bit of encrypting

PS C:\> 0..($cleartext.length -1) | % {
  $ciphertext += ($key[$_] -bxor [int]$cleartext[$_]).ToString("x2") }
PS C:\> $ciphertext
3993ada5bdadc99517dc6125561fb6a93194757bb98d9678

This command counts from 0 to 24, the length of the clear text minus 1 (remember, base 0). The current pipeline object ($_) represents the counter and is used as the index in the array to grab the respective bytes from the key and clear text. The -bxor operator does a binary XOR. The result is then converted to a two character Hex string (x2), then is appended it to our cipher text. I would then send Hal this message:

Dear Hal,

Don't tell Heather, but 3993ada5bdadc99517dc6125561fb6a93194757bb98d9678

The cool thing is that the same process can be used to decrypt.

PS C:\> $cipherbytes = [byte[]](0x39,0x93,0xad,0xa5,0xbd,0xad,0xc9,0x95,0x17,
  0xdc,0x61,0x25,0x56,0x1f,0xb6,0xa9,0x31,0x94,0x75,0x7b,0xb9,0x8d,0x96,0x78)

PS C:\> 0..($cipherbytes.length -1) | % {
  $cipherbytes += ($key[$_] -bxor $cipherbytes[$_]).ToString("x2") }
PS C:\> $cipherbytes
I secretly teach Kung Fu

We can even use PowerShell to convert a hex string to bytes:

PS C:\> "3993ada5bdadc99517dc6125561fb6a93194757bb98d9678" | 
Select-String ".." -AllMatches | % { $_.Matches } | % { [byte]("0x" + $_.Value) }

This command will match on each two characters (..), take each match object, take each byte of the match and convert it to a byte.

That isn't really pretty, but we'll see how Hal will send the message, and if Bash is leet.

Hal goes into suck mode

I've got to be honest. This is a challenge where bash just isn't very conducive to solving the problem. But since a "real" scripting language like Perl or Python is disallowed by the rules of our blog, I'll just have to muddle through somehow.

Before we get to the suckage, let's start by assigning the bytes of our one-time pad to an array, just like Tim did:

$ otp=(0x70 0xB3 0xDE 0xC0 0xDE 0xDF 0xAC 0xE1 0x7B 0xA5 0x41 0x51 
       0x33 0x7E 0xD5 0xC1 0x11 0xDF 0x00 0x15 0xDE 0xAD 0xD0 0x0D)

The syntax we're using here "var=(val1 val2 ...)" is a convenient way for assigning a list of values to the elements of an array. We'll use this again in just a moment.

The hard part is when we want to start XOR-ing these bytes with the bytes in our string. The bash XOR operator ("^") wants both operators to be a numeric type. If I tried to do something like "I ^ 0x70", bash treats the "I" as an invalid number and therefore zero-- that's not what we want at all! So what I need to do is somehow convert the ASCII bytes in our input string into their numeric equivalents.

Well guess what? There's no bash built-in function for doing this! So you're left with doing a nasty hack like this:

$ echo -n I secretly teach Kung Fu | xxd -p
49207365637265746c79207465616368204b756e67204675

As you can see, I'm using the hexdumping program xxd as my ASCII-to-hex converter. The "-p" option means "plain mode": just output the hex bytes and nothing else. Of course, the problem here is that the bytes are all run together. But a little sed will fix that:

$ echo -n I secretly teach Kung Fu | xxd -p | sed 's/\(..\)/0x\1 /g'
0x49 0x20 0x73 0x65 0x63 0x72 0x65 0x74 0x6c 0x79 0x20 0x74 0x65 0x61 0x63...

In my sed expression, I'm snapping up two characters at a time and outputting a "0x", then the two characters, then a space.

With a little command substitution, I can now assign these hex bytes to another array:

$ bytes=( $(echo -n I secretly teach Kung Fu | xxd -p | sed 's/\(..\)/0x\1 /g') )

I'm really just doing an array assigment here-- "bytes=( ... )"-- similar to the one we used to set up our one-time pad. But in this case, the values I'm assigning are the output of our shell pipeline.

Now that I've got two arrays of bytes, I can XOR them together with a for loop:

$ for ((i=0; $i < ${#bytes[*]}; i++)); do 
      printf "%x" $((${bytes[$i]} ^ ${otp[$i]})); 
done
3993ada5bdadc99517dc6125561fb6a93194757bb98d9678$

The expression "${#bytes[*]}" evaluates to the number of elements in the array. Inside the loop, I'm using printf to output the results of my XOR operation as hex digits ("%x"). The only downside, as you can see, is that there's no final trailing newline at the end of the output-- the "$" you see there is the shell prompt for the next line! This is maybe not such a big deal, because most likely I'd want to do something like this:

$ secret=$( for ... )

That would assign the output of the loop to a variable, $secret.

But now what about going the other way and converting the contents of $secret back into the original ASCII? This is actually pretty similar to the steps above with a couple of small mods:

$ bytes=( $(echo $secret | sed 's/\(..\)/0x\1 /g') )
$ for ((i=0; $i < ${#bytes[*]}; i++)); do 
      printf "%x" $((${bytes[$i]} ^ ${otp[$i]})); 
done | xxd -r -p
I secretly teach Kung Fu$

Here I'm resetting the value of our bytes array to be the individual bytes from $secret, converted by sed to "0xNN" format. Then I run through the same for loop to XOR these bytes with our one-time pad. However this time the output goes into "xxd -r -p", which "reverts" the resulting string of bytes into their corresponding ASCII values.

All I can say is, thank the shell gods for xxd!

Episode #131: Subject of Attachment

Because Hal sed So:

We got an interesting challenge in the mailbag this week from Ryan Tuthill:

Given the example logs:

562C938C3.A9069|attachment = Interview Results.xls
B1CE33BED.A58D0|subj="Who?"  | subj="Who?"
67BD53BED.AF311|subj="January 14.docx.doc"  | attachment = January 14.docx.doc
19B4D27D2.A8CE7|subj="FW: Two New Bud Light Commercial's"  | attachment = Bud_Lite_Pick_Up_Line.mpg

What I am curious to discover is a way to print only lines that contain subj and attachment, not just one or the other. From there, I would like to print the lines with the same subj and attachment titles.

Looking at the problem, I thought to myself, "Hmmm, irregular pattern matching. Seems like a job for sed." Then I started wondering if I could solve the problem entirely with sed. We haven't spent much time on the blog talking about sed, but you must understand that it's a fully capable programming language in its own right. For proof of that assertion, I refer you to sedtris-- yep, that's a Tetris game written entirely in sed.

Anyway, it turns out there is a sed-only solution to Ryan's problem. Assuming we've got our sample input in a file called "input", simply do this:

$ sed 'h; s/.*subj="\(.*\)".*attachment = \1$/\1/; t hit; d; :hit g' input
67BD53BED.AF311|subj="January 14.docx.doc"  | attachment = January 14.docx.doc

I know the syntax is terse and mysterious here, so let's take this slowly. The first thing you need to understand is that sed has two different data spaces you can work with. The one you normally use is the "pattern space", which is the line you just read in. When you do an operation like "s/.../.../", you're operating on the pattern space. However, sed also has a "hold space" where you can store stuff for later. And there are operators that let you copy or append data into the hold space from the pattern space and vice-versa.

With that in mind, let's walk through our sed "program". The first command we give sed is "h", which means "copy the pattern space into the hold space, overwriting the previous value in the hold space". We're using this to store the original version of the line we just read in, so that we can print it out later if it matches our criteria.

Once we've saved a copy of the original line, we can start mangling the copy in the pattern space. The next sed operation we perform is a substitution. The regular expression on the lefthand side matches 'subj="<string>"' anywhere on the line, followed by 'attachment = <string>' at the end of the line. sed is one of the few Unix regular expression syntaxes that supports matching a string early in the regular expression and then testing for the matched string later in the same regex. If our pattern matches, then we end up replacing the entire original line with just the value of the <string> we matched.

This is where it starts to get tricky. It turns out that the replacement we do on the RHS of the "s/.../.../" expression doesn't matter. I'm really only using the substitution to verify that we have a line that matches our criteria. This, in turn, allows me to use sed's branching operator to control whether or not we output the original line. You see sed has a goto-like operator "t <label>" which will jump to the specified <label> if and only if there has been a successful "s/.../.../" operation since the current line was read in (or since the last "t" operation if there's been more than one).

So if our substitution was successful, then this is a line with matching "subj" and "attachment" titles and we want to print out the line. In this case our "t" operation tells sed to jump to the label "hit" and start executing statements from there. If the substitution didn't work, then we just fall through to the next statement after the "t".

Thus when we don't match what happens is the sed operator "d", which simply means "discard the current pattern space and move on to the next line". Think of it like "continue" or "next" in many popular programming languages. The next thing in our sed program is the label ":hit", which is where we'll jump to if the "s/.../.../" operator worked. The sed command we invoke here is "g", which overwrites the current pattern space with the contents of the hold space. Remember how we saved the original line into the hold space back at the beginning of our program? Well now we bring that back. And finally, there's an implicit "print the contents of the pattern space" at the end of every sed block which takes care of actually outputting the line for us.

Pretty neat, huh? I was feeling really good about myself until I got a follow-up email from Ryan with a few more sample log entries:

BE1342109.A1C1F|attachment = ATT49396.txt | subj="FW: snow"
21E3430E7.A8583|attachment = Scan001.pdf  | subj="FW: PROJECT/TASK"
8657C30E7.AC7A5|attachment = IMG00005.jpg  | subj="IMG00005.jpg"

Yep, it turns out that the "subj" and "attachment" can appear in either order. Well, back to the drawing board:

$ sed 'h; 
       s/.*subj="\([^"]*\)".*attachment = \1$/\1/; t hit; 
       s/.*attachment = \(.*[^ ]\) *|.*subj="\1"$/\1/; t hit; d; 
       :hit g' input
67BD53BED.AF311|subj="January 14.docx.doc"  | attachment = January 14.docx.doc
8657C30E7.AC7A5|attachment = IMG00005.jpg  | subj="IMG00005.jpg"

The new code isn't actually all that different from the original solution-- I've just split things onto multiple lines for greater readability. We have the "h" operator, a substitution that checks for 'subj="<string>"' followed by 'attachment = <string>', and the "t" operator just as before. However, this time if we fail to get a hit, then we try looking for 'attachment = <string>' followed by 'subj="<string>"' instead. Only if both of those operations fail, do we give up and do "d". If either operation succeeds, then we jump to ":hit" and do the operations to print out the line.

The pattern match for the 'attachment = <string>' is a little tricky in the second case because there are no quotes around the attachment name and the attachment name can contain spaces. So we have to explicitly match "'attachment<space>=<space>", followed by "some stuff that ends in a non-space character", followed by some spaces, and then a pipe character ("|") which is our field separator. Yeesh!

Anyway, this has been your first high-speed introduction to the kinky joy that is sed. Tim, I bet you don't have anything this dirty in your Powershell arsenal!

Tim feeds his expressions prunes, so they stay regular:

PowerShell may not have sed, but it does have built-in regular expressions. These expression are way cooler than their incontinent brethren of cmd.exe.

Similar to Hal's approach, we first read the file and look for strings containing a subject. This is accomplished by piping Get-Content (alias gc) into Select-String where we filter using a regular expression.

PS C:\> gc log.txt | Select-String 'subj="(?<Subject>[^"]+)
B1CE33BED.A58D0|subj="Who?" | subj="Who?"
67BD53BED.AF311|subj="January 14.docx.doc" | attachment = January 14.docx.doc
19B4D27D2.A8CE7|subj="FW: Two New Bud Light Commercial's" | attachment = Bud_Lite_Pick_Up_Line.mpg
...

This regular expression matches on lines contain something like the following:
subj="<text that doesn't contain a double quote>

The content of the text that doesn't contain double quotes is part of a named group and contains the subject we would like to use later. The syntax for using a named group is:

(?<GroupName>regex)

Until I prepped for this episode, I didn't realize that the Select-String cmdlet will populate the $matches variable; I thought that was only done via the -Match operator. This little trick makes our command much shorter and easier to read than using the -Match operator.

As regular readers (badabing!) might remember, the $matches variable contains the groups matched by our regular expression. Since we used a named group, it makes it much easier to access the group we want by simply using $matches.GroupName. This variable can be used further down the pipeline, and we can use this little trick to get our final command.

PS C:\> cat .\log.txt | Select-String 'subj="(?<Subject>[^"]+)' | 
  Select-String "attachment = $($matches.subject)"

67BD53BED.AF311|subj="January 14.docx.doc" | attachment = January 14.docx.doc
8657C30E7.AC7A5|attachment = IMG00005.jpg | subj="IMG00005.jpg"

The results from our first command are piped into Select-String which filters for strings containing "attachment = <subject name> and that gives us the output we are looking for.

Episode #130: Eenie, Meanie, Miney, Mo...

Hal gets selective

Recently I was doing an audit of USB devices that had been connected to numerous Windows machines on a network. The input data I had looked like this:

KeyStore XP2G
080909524d94e5
Sat Jun 30 22:42:07 2009
Apple iPod
000A270010C4E86E
Fri Jan 16 23:24:15 2009
M-Sys Dell Memory Key
086086412140E1C2
Fri Jan 16 23:15:30 2009
OLYMPUS u810/S810
000J55024022
Wed Jan 14 19:03:58 2009
...

The input file was a collection of three line "records" for each device. The first line was the "friendly name" of the device, the second line was the device serial number, and the third line was the date the device was last connected. I had one input file per machine.

What I needed to do was to extract just the serial numbers from each input file. Since the serial numbers of USB devices can have widely varying formats, I couldn't easily write a regular expression to match them. Instead I needed to extract the lines by line number-- the 2nd, 5th, 8th lines and so on. awk is actually really useful for this:

$ awk '!((FNR - 2) % 3)' *
080909524d94e5
000A270010C4E86E
086086412140E1C2
000J55024022
...

Recall from Davide Brini's solution a few weeks ago, that FNR is the "record number" in the current file. When using awk's default record separator which is newline, that means that FNR corresponds to the line number in the current file. So I'm subtracting two from the line number and then doing a "modulo 3" operation-- so the expression will be zero on lines 2, 5, 8, 11, ... Therefore "not" that expression, aka "!(...)", will evaluate to true on those lines only. Since there's no command block after the expression "{print}" is assumed, and I output the lines I want.

Now the reason I was pulling out the serial numbers is that I wanted to see which devices had been connected to multiple systems. The easy way to do this is to just slap on a little sort and uniq action:

$ awk '!((FNR - 2) % 3)' * | sort | uniq -d
080909524d94e5
086086412140E1C2
...

But just to save Davide the trouble of sending me an email, here's the "awk-only" version:

$ awk '!((FNR - 2) % 3) && (++a[$1] == 2)' * 
086086412140E1C2
080909524d94e5
...

By adding another clause after a logical "and" ("&&") I ensure that the second clause only gets executed on the lines containing serial numbers. In the second clause I'm creating values in an array that is indexed by the serial number. The values are a count of the number of times we've seen each serial number-- "++a[$1]" adds one to the value of a[$1] before evaluating the conditional. After we've incremented our accumulator value, we check to see if the value is 2, meaning that this is the second time we've encountered a given serial number. If that's true then our implicit "{print}" happens again and we output the serial number. I don't care if the serial number appears in more than two different files, just that it appears in at least two.

Of course the output of the "sort | uniq -d" version is in a different order than the awk-only version because of the "sort" in the first solution. But since the serial number format varies widely anyway, I'm not sure sorting is that useful in any case. You could always pipe the awk output into sort if sorting is important to you.

I'm pretty sure Tim can knock this one out of the park using Powershell. I wonder what a CMD.EXE solution would look like..?

Tim is selective too

Hal likes to taunt me with digs at CMD.EXE, and this week I'll have to concede. I got about 90% of the way through writing the big ol' command and decided it was getting ridiculous. If you want to know what it would have looked like go check out Ed's final episode. The command would have worked but it would be completely impractical and no one would have used it, in short, a circus act. Instead, let's do something practical. On to PowerShell...

The PowerShell cmdlet used to read a file is Get-Content (aliases cat, gc, and type). But we don't just need to read a file, we need to read a file and get every third line. Here's how to do just that:

PS C:\> Get-Content -Path * -ReadCount 3 | % { $_[1] }
080909524d94e5
000A270010C4E86E
086086412140E1C2
000J55024022
...

Get-Content's ReadCount parameter is used to "specifies how many lines of content are sent through the pipeline at a time." The default value is 1, so normally each line is sent down one at a time. Setting this value to 3 means that 3 lines at a time will be sent down the pipeline.

Inside the ForEach-Object's scriptblock, the current pipeline object ($_) contains the three lines passed down the pipeline. In this group of three lines we are looking for the second item in the array. The second item in the array is represented by the array index of 1. Remember, the first item in an array is 0, so the second is 1, and third is 2.

Now that we have the serial numbers, let's look for duplicates. This is really easy with the Group-Object cmdlet.

PS C:\> Get-Content -Path * -ReadCount 3 | % { $_[1] } | Group-Object -NoElement

Count Name
----- ----
    1 080909524d94e5
    2 000A270010C4E86E
    2 086086412140E1C2
...

The NoElement switch means that the original objects are not stored, so we have just the count serial number.

In addition, we can filter for counts greater than 1.

PS C:\> Get-Content -Path * -ReadCount 3 | % { $_[1] } |
   Group-Object -NoElement | Where-Object { $_.Count -gt 1 }

Count Name
----- ----
    2 000A270010C4E86E
    2 086086412140E1C2
...

But that is a long command, and I like to be brief, so this is the short method:

PS C:\> gc * -r 3 | % { $_[1] } | group -n | ? { $_.Count -gt 1 }

So we did what Hal did, now let's up the ante and turn the contents of these files into objects.

PS C:\> gc * -r 3 | select @{Name="Name"; Expression={$_[0]}},
  @{Name="Serial";Expression={$_[1]}},@{Name="Date";Expression={$_[2]}}

Name           Serial            Date
----           ------            ----
KeyStore XP2G  080909524d94e5    Tue Jun 30 22:42:07 2009
Apple iPod     000A270010C4E86E  Fri Jan 16 23:24:15 2009
...

The Select-Object cmdlet uses hashtables to allow you to manually create properties. Hashtables use key-value pairs. To create a property the Name is [obviously] used to set the name of the property. The Expression is used to set the value of the property. So we have objetified everything, but the date is still a string. To convert it to a Date object we can use .NET to convert the string to a native Date Object.

PS C:\> gc * -r 3 | select @{Name="Name"; Expression={$_[0]}},
   @{Name="Serial";Expression={$_[1]}},
   @{Name="Date";Expression={[datetime]::ParseExact($_[2], "ddd MMM dd HH:mm:ss yyyy", $null)}}

Name                   Serial            Date
----                   ------            ----
KeyStore XP2G          080909524d94e5    6/30/2009 10:42:07 PM
Apple iPod             000A270010C4E86E  1/16/2009 11:24:15 PM
M-Sys Dell Memory Key  086086412140E1C2  1/16/2009 11:15:30 PM
...

Now we can do whatever we want with it, such as exporting the results to a CSV to use in Excel, filter based on the properties, or look for duplicates as we did above.

One thing that might be useful is knowing into which computers the USB devices were plugged. Assuming the file name is descriptive we can use it to add an additional property to our objects.

PS C:\> ls | % { $file = $_.Name; gc $_ -r 3 |
   select @{Name="Name"; Expression={$_[0]}},
   @{Name="Serial";Expression={$_[1]}},
   @{Name="Date";Expression={[datetime]::ParseExact($_[2], "ddd MMM dd HH:mm:ss yyyy", $null)}},
   @{Name="File";Expression={$file}}}

Name                  Serial           Date                  File
----                  ------           ----                  ----
KeyStore XP2G         080909524d94e5   6/30/2009 10:42:07 PM Alpha.txt
Apple iPod            000A270010C4E86E 1/16/2009 12:20:00 PM Alpha.txt
...
Apple iPod            000A270010C4E86E 1/21/2009 11:24:15 PM Bravo.txt
M-Sys Dell Memory Key 086086412140E1C2 1/16/2009 11:15:30 PM Bravo.txt
...

This command starts with Get-ChildItem (alias ls) to get each file in the directory. The files are piped into ForEach-Object where the filename is stored in a variable for use later. The rest of the command is very similar as our original command, the only difference is the input to Get-Content is specified by the current pipeline object, instead of our wildcard (*) as before.

Now we can do all sorts of filtering.

PS C:\> ... | ? { $_.Serial -eq "000A270010C4E86E" }
Name       Serial           Date                  File
----       ------           ----                  ----
Apple iPod 000A270010C4E86E 1/16/2009 12:20:00 PM Alpha.txt
Apple iPod 000A270010C4E86E 1/21/2009 11:24:15 PM Bravo.txt
...

Put that in your pipeline and smoke it, Hal!

Episode #129: Writing on the Wall

Tim wants to be heard:

Hal and I realized that in the previous episode we brought up a new topic but never explained it. I broke out the "msg" command, and he used "wall." In case you couldn't figure it out, these commands are used to send a message to users on the system.

The Windows command Msg is used to send a message to one or more users based on username, sessionname, or sessionid. The username is the most common way of directing a message to a user.

C:\> msg hal You have no chance to survive, make your time!

This command simply sends a message to Hal. As mentioned above, a message can also be directed based on the session name or id. To determine the session id or name refer to episode 62.

We can also send a message to all users on the system by using the asterisk.

C:\> msg * Someone set up us the bomb!

What if we don't want to send the message to all the users, but more than one user? We can do that! It does require that we have a file containing a list of usernames to whom we would like to direct our message.

C:\> msg @mostlyeveryone.txt Someone set up us the bomb!

We can also send the messages to users on other systems by using the /SERVER switch.

C:\> msg * /SERVER:otherbox All your base are belong to us!

However, this command doesn't just send messages, but also can be used to get an acknowledgment. The /V option displays information about which actions have been performed, such as sending a message and acknowledgments. The /W option waits for a response from the users. Say we send a message to Hal, and want to make sure he gets its, this is how we would do it:

C:\> msg hal /V /W Did you make your time?
Sending message to session Console, display time 60
Message to session Console responded to by user

The first message lets us know that a message was sent to Hal. The second means that either Hal responded, or the 60 second timer elapsed. Its a bit weird that the message is the same either way, but welcome to the wonderful world of Windows commands.

If we don't think that 60 seconds is long enough for Hal to respond, we can use the /TIME option to explicitly specify the duration of the message.

C:\> msg hal /V /W /TIME:3600 Did you make your time?

This command will wait one hour for a response; more than enough time for Hal to "make his time!"

By the way, these lost-in-translation quotes are taken from the internet meme All Your Base Are Belong To Us!. As a side note, I tried to use these quote in an earlier episode, but Hal corrected my grammar and fixed it. I can't believe Hal doesn't have endless hours to waste watching silly videos on the internet. Is that what the internet is for?

Hal is a little hard of hearing:

I guess I'm just having trouble keeping up with you kids and all your Internet shenanigans. But all of this writing to people's terminals is making me fondly remember my days using dumb terminals to talk with my friends on time-sharing systems. You can't type faster than 9600 baud, so who the heck needs 10Gbps? And get off my lawn!

If I wanted to tell Tim to get off my lawn, I could do that with the write command:

$ write tim
write: tim is logged in more than once; writing to pts/2
Get off my lawn!
^D

Specify the user you want to send a message to as an argument. As you can see, Tim is logged in on multiple PTYs, so the write command will by default send the message to the lowest numbered device. But you can specify a specific device to write to as an optional argument: "write tim pts/2", for example.

Once you hit return on the command line, anything you type in subsequent lines is sent to the specified user (each line normally gets sent immediately when you hit Enter). When you're done entering text, just hit <Ctrl>-D. Tim ends up seeing a message like this:

Message from hal@caribou on pts/1 at 16:43 ...
Get off my lawn!
EOF

Now random grumpy old men shouting into your terminal windows can be distracting when you're trying to get work done. So Tim has the option of blocking messages on his PTY with "mesg n":

$ mesg
is y
$ ls -l /dev/pts/2
crw--w---- 1 tim tty 136, 2 2011-01-10 16:50 /dev/pts/2
$ mesg n
$ ls -l /dev/pts/2
crw------- 1 tim tty 136, 2 2011-01-10 16:50 /dev/pts/2

With no arguments, the mesg command tells you what the current status of your PTY is-- the default is to be accepting messages, or "y". As you can see in the output above, running "mesg n" simply removes the group-writable flag from your PTY. The write command is set-GID to group "tty" so that it can write messages to users terminals when "mesg y" is set.

But in our last Episode, I used the wall command to send New Year's greetings to everybody on the system. In its simplest form, wall accepts input on the standard in and blasts it to all currently connected users:

$ echo Get off my lawn! | wall

And the users see:

Broadcast Message from hal@caribou                                             
        (/dev/pts/1) at 16:58 ...                                              
                                                                               
Get off my lawn!

If you are the superuser, wall can also be used to write messages stored in a text file.

# wall /etc/shutdown-message

The other advantage to running wall as the superuser is that your message will even go to the terminals of those users who have set "mesg n". After all, "mesg n" works by changing permissions on the PTY devices, but root can write to any file regardless of permissions.

Now that I've educated you whipper-snappers on this outmoded technology, it's time for my nap. Why don't you kids go shoot some marbles or play with your hula-hoops?

Episode #128: Happy New Year!

Command Line Kung Fu is taking a little holiday respite. We'll be back with more Fu on 2011-01-11. In the meantime, here's your weekly fix of command-line madness... just a little bit early.

Hal has been replaced by a small shell script

echo "echo 'Happy New Year from your #1 blog!' | wall" | at 11:11 01/01/11

Tim has been replaced by a slightly smaller shell script

at 11:11 /next:1 cmd /c "msg * Happy New Year from your #1 blog!"

See Episode #50 for further explanation...

Episode #127: Making a Difference

Hal went to school

I recently got the opportunity to sit in on (fellow SANS instructor) Lenny Zeltser's "Reverse Engineering Malware" class. It's a terrific course, and I highly recommend it.

During the material on memory analysis, we were comparing the output of "volatility pslist" and "volatility psscan2". It's relatively straightforward for rootkits to hide themselves from pslist, but psscan2 does a much more thorough job of finding the hidden processes. So the differences in the output are always very interesting to the analyst. Here's an example of what I mean:

$ volatility pslist -f memory.img
Name                 Pid    PPid   Thds   Hnds   Time  
System               4      0      55     260    Thu Jan 01 00:00:00 1970  
smss.exe             540    4      3      21     Thu Jan 28 16:11:40 2010  
csrss.exe            604    540    12     363    Thu Jan 28 16:11:46 2010  
lsass.exe            684    628    18     341    Thu Jan 28 16:11:47 2010  
vmacthlp.exe         836    672    1      24     Thu Jan 28 16:11:47 2010  
svchost.exe          848    672    18     201    Thu Jan 28 16:11:47 2010  
svchost.exe          1024   672    51     1178   Thu Jan 28 16:11:47 2010  
svchost.exe          1072   672    4      75     Thu Jan 28 16:11:47 2010  
svchost.exe          1132   672    15     212    Thu Jan 28 16:11:48 2010  
spoolsv.exe          1476   672    10     115    Thu Jan 28 16:11:49 2010  
explorer.exe         1592   1572   12     4021   Thu Jan 28 16:11:50 2010  
VMwareUser.exe       1656   1592   8      416    Thu Jan 28 16:11:50 2010  
VMwareService.e      1996   672    3      1026   Thu Jan 28 16:11:58 2010  
wscntfy.exe          1396   1024   1      27     Thu Jan 28 16:12:03 2010  
taskmgr.exe          1624   628    3      20201  Tue Feb 02 02:45:05 2010  
mike022.exe          1956   672    2      30     Tue Feb 02 03:25:29 2010  
wordpad.exe          1992   1260   4      102    Tue Feb 02 22:17:03 2010  
calc.exe             828    1592   1      26     Thu Feb 04 00:01:00 2010  
cmd.exe              968    1592   1      32     Thu Feb 04 00:01:13 2010  
wordpad.exe          2008   1256   5      101    Thu Feb 04 00:02:56 2010  
$ volatility psscan2 -f memory.img
PID    PPID   Time created             Time exited              Offset     PDB        Remarks
------ ------ ------------------------ ------------------------ ---------- ---------- ----------------

   932    672 Thu Jan 28 16:11:47 2010                          0x01ea3558 0x082c0100 svchost.exe     
  1744    848 Thu Feb 04 00:02:53 2010 Thu Feb 04 00:04:23 2010 0x01eaea88 0x082c0380 wmiprvse.exe    
  1132    672 Thu Jan 28 16:11:48 2010                          0x01eb4970 0x082c0160 svchost.exe     
  1956    672 Tue Feb 02 03:25:29 2010                          0x020155d8 0x082c02c0 mike022.exe     
  1072    672 Thu Jan 28 16:11:47 2010                          0x02016978 0x082c0140 svchost.exe     
  1172   1592 Tue Feb 02 02:40:48 2010                          0x0204c850 0x082c01c0 cmd.exe         
  1476    672 Thu Jan 28 16:11:49 2010                          0x0209db38 0x082c01a0 spoolsv.exe     
  1996    672 Thu Jan 28 16:11:58 2010                          0x021f0da0 0x082c0180 VMwareService.e 
  1664   1592 Thu Jan 28 16:11:50 2010                          0x021feb88 0x082c0240 msmsgs.exe      
  1024    672 Thu Jan 28 16:11:47 2010                          0x02202880 0x082c0120 svchost.exe     
   604    540 Thu Jan 28 16:11:46 2010                          0x0221f020 0x082c0040 csrss.exe       
  1624    628 Tue Feb 02 02:45:05 2010                          0x02256da0 0x082c02e0 taskmgr.exe     
   272   1820 Thu Feb 04 00:00:55 2010                          0x02293b08 0x082c0300 wordpad.exe     
  1012    672 Thu Jan 28 16:12:02 2010                          0x023a78b0 0x082c0260 alg.exe         
  1656   1592 Thu Jan 28 16:11:50 2010                          0x023a9c28 0x082c0220 VMwareUser.exe  
  1648   1592 Thu Jan 28 16:11:50 2010                          0x023ae980 0x082c0200 VMwareTray.exe  
   848    672 Thu Jan 28 16:11:47 2010                          0x023b3020 0x082c00e0 svchost.exe     
  1748   1592 Thu Feb 04 00:02:10 2010 Thu Feb 04 00:06:19 2010 0x0240b9a0 0x082c03a0 cmd.exe         
   836    672 Thu Jan 28 16:11:47 2010                          0x02412b58 0x082c00c0 vmacthlp.exe    
   672    628 Thu Jan 28 16:11:47 2010                          0x02448cf8 0x082c0080 services.exe    
   968   1592 Thu Feb 04 00:01:13 2010                          0x024707e8 0x082c0340 cmd.exe         
   684    628 Thu Jan 28 16:11:47 2010                          0x02483da0 0x082c00a0 lsass.exe       
  1992   1260 Tue Feb 02 22:17:03 2010                          0x02491130 0x082c0360 wordpad.exe     
  1396   1024 Thu Jan 28 16:12:03 2010                          0x02492d78 0x082c0280 wscntfy.exe     
  2008   1256 Thu Feb 04 00:02:56 2010                          0x02494988 0x082c03e0 wordpad.exe     
   828   1592 Thu Feb 04 00:01:00 2010                          0x024c86b8 0x082c02a0 calc.exe        
  1592   1572 Thu Jan 28 16:11:50 2010                          0x024ddda0 0x082c01e0 explorer.exe    
   540      4 Thu Jan 28 16:11:40 2010                          0x024f8368 0x082c0020 smss.exe        
   628    540 Thu Jan 28 16:11:46 2010                          0x025314e8 0x082c0060 winlogon.exe    
     4      0                                                   0x025c8830 0x00319000 System          

Visually you can see that the psscan2 output lists several more processes than pslist, but just using your eyeballs it can be difficult to figure out exactly what the differences are. Seems like a job for command-line kung fu!

My first thought was to simply extract the list of .EXEs from each command and diff them. In order to do the diff properly, I'll need to sort them into canonical order, but that's no problem. Here's how we manage the output from pslist:

$ volatility pslist -f memory.img | tail -n +2 | awk '{print $1}' | sort
calc.exe
cmd.exe
csrss.exe
...

I use tail to chop off the header line, then awk to extract the name of the .EXE from the first column, and finally pipe the whole thing into sort.

Dealing with the psscan2 output is very similar:

$ volatility psscan2 -f memory.img | tail -n +4 | awk '{print $NF}' | sort
alg.exe
calc.exe
cmd.exe
...

In this case, there are three header lines we need to skip. Also the .EXE name is in the last column of output-- "print $NF" is a useful awk idiom for printing the value in the last column.

So now we need to diff the output of these two commands. We could do this by creating temporary files, but why bother when have the magic bash "<(...)" syntax that lets us substitute command output in a place where a command would normally be looking for a file name:

diff <(volatility psscan2 -f memory.img | tail -n +4 | awk '{print $NF}' | sort) \
     <(volatility pslist -f memory.img | tail -n +2 | awk '{print $1}' | sort)
1d0
< alg.exe
4,5d2
< cmd.exe
< cmd.exe
10,11d6
< msmsgs.exe
< services.exe
18d12
< svchost.exe
23d16
< VMwareTray.exe
25,27d17
< winlogon.exe
< wmiprvse.exe
< wordpad.exe

Wicked! There are 10 processes that appear in the psscan2 output that don't show up in the pslist output. Since we don't see any lines starting with ">" there are no processes in the pslist output that don't show up in psscan2-- this is what we'd expect, but it's always nice to get confirmation.

The only problem here is that as we got further into the in-class exercises, I realized I really wanted all of the extra detail about each of the hidden processes from the psscan2 output. For example, the hex offset values end up being very useful, and I'd like to know exactly which two of the three command.exe processes are the hidden ones. Let me show you the command line I came up with and then explain it to you:

$ join -v 1 -1 1 -2 2 \
    <(volatility psscan2 -f memory.img | tail -n +4 | sort -n -k 1,1) \
    <(volatility pslist -f memory.img | tail -n +2 | sort -n -k2,2)
272 1820 Thu Feb 04 00:00:55 2010 0x02293b08 0x082c0300 wordpad.exe 
628 540 Thu Jan 28 16:11:46 2010 0x025314e8 0x082c0060 winlogon.exe 
672 628 Thu Jan 28 16:11:47 2010 0x02448cf8 0x082c0080 services.exe 
932 672 Thu Jan 28 16:11:47 2010 0x01ea3558 0x082c0100 svchost.exe 
join: file 1 is not in sorted order
join: file 2 is not in sorted order
1012 672 Thu Jan 28 16:12:02 2010 0x023a78b0 0x082c0260 alg.exe 
1172 1592 Tue Feb 02 02:40:48 2010 0x0204c850 0x082c01c0 cmd.exe 
1648 1592 Thu Jan 28 16:11:50 2010 0x023ae980 0x082c0200 VMwareTray.exe 
1664 1592 Thu Jan 28 16:11:50 2010 0x021feb88 0x082c0240 msmsgs.exe 
1744 848 Thu Feb 04 00:02:53 2010 Thu Feb 04 00:04:23 2010 0x01eaea88 0x082c0380 wmiprvse.exe 
1748 1592 Thu Feb 04 00:02:10 2010 Thu Feb 04 00:06:19 2010 0x0240b9a0 0x082c03a0 cmd.exe

In this case I'm using join rather than diff because the output of the two commands is so differently formatted. Essentially I'm doing a join on the PID columns of the psscan2 ("-1 1") and pslist ("-2 2") output and telling join to output the non-matching lines from psscan2 ("-v 1"). The tricky bit is that each command output needs to be sorted by its PID column for join to work. So if you look in the "<(...)" clauses, you'll see that the final element of the pipeline in each case is a numeric sort on the PID column. Easy, right?

The only fly in the ointment is the "not in sorted order" error messages from join. The problem is that join only understands alphabetic sorting. So when we go from 9xx PIDs to 1xxx PIDs, join thinks the file has gone all unsorted. There's no "-n" option to join like there is for sort, but in some versions of join we can use the "--nocheck-order" option to suppress the error messages:

$ join -v 1 -1 1 -2 2 --nocheck-order \
    <(volatility psscan2 -f memory.img | tail -n +4 | sort -n -k 1,1) \
    <(volatility pslist -f memory.img | tail -n +2 | sort -n -k2,2)
272 1820 Thu Feb 04 00:00:55 2010 0x02293b08 0x082c0300 wordpad.exe 
628 540 Thu Jan 28 16:11:46 2010 0x025314e8 0x082c0060 winlogon.exe 
672 628 Thu Jan 28 16:11:47 2010 0x02448cf8 0x082c0080 services.exe 
932 672 Thu Jan 28 16:11:47 2010 0x01ea3558 0x082c0100 svchost.exe 
1012 672 Thu Jan 28 16:12:02 2010 0x023a78b0 0x082c0260 alg.exe 
1172 1592 Tue Feb 02 02:40:48 2010 0x0204c850 0x082c01c0 cmd.exe 
1648 1592 Thu Jan 28 16:11:50 2010 0x023ae980 0x082c0200 VMwareTray.exe 
1664 1592 Thu Jan 28 16:11:50 2010 0x021feb88 0x082c0240 msmsgs.exe 
1744 848 Thu Feb 04 00:02:53 2010 Thu Feb 04 00:04:23 2010 0x01eaea88 0x082c0380 wmiprvse.exe 
1748 1592 Thu Feb 04 00:02:10 2010 Thu Feb 04 00:06:19 2010 0x0240b9a0 0x082c03a0 cmd.exe 

The other alternative is obviously to sort the PID columns alphabetically, but that offends my sensibilities somehow.

Mmmm, hmmm! That was some tasty fu! Hey Tim, volatility runs on Windows-- what can you do with the output? I double-dog-dare you to try it in CMD.EXE first...

Tim skipped school:

Do cmd.exe, dang Hal. Happy Freaking New Year to me, huh?

Here is what I came up with based on the assumption that pslist returns a subset of psscan2.

C:\> python.exe volatility psslist -f memory.img > plist.txt
C:\> cmd /v:on /c "for /F "skip=2 tokens=1,5,10,15" %a in ('python.exe volatility psscan2 -f lab3.img') do
  @(if not "%d"=="" (set name=%d) else (if not "%c"=="" (set name=%c) else (set name=%b))) &
  set pid=%a & (type pslist.txt | findstr /B /R /C:"!name! *!pid! " > NUL || echo !name! !pid!)"

svchost.exe 932
wmiprvse.exe 1744
cmd.exe 1172
msmsgs.exe 1664
wordpad.exe 272
alg.exe 1012
VMwareTray.exe 1648
cmd.exe 1748
services.exe 672
winlogon.exe 628

I split this command into two for the sake of readability; however, it could be easily combined into a one-liner. But I'll leave that simple experiment to you. The first line takes the output of psslist and dumps the contents into a file. This file will be read numerous times so it is significantly faster to just read the file in the second "half" of our command. Now, regarding that second half...

We start off by using invoking our shell with /v:on to enable delayed variable expansion and /c to cause our spawned shell to exit upon completion. Inside the shell we use our trusty For loop. The first three lines are skipped as they are headers. The For loop then splits the line based on white space. We are trying to get the name of the process, and due to spacing, it may be in the 5th, 10th, or 15th token. Yes, it is that confusing. Here is a little diagram of what I mean:

PID    PPID   Time created             Time exited              Offset     PDB        Remarks
------ ------ ------------------------ ------------------------ ---------- ---------- ----------------

Token1      2   3   4  5        6    7                                   8          9     10
   932    672 Thu Jan 28 16:11:47 2010                          0x01ea3558 0x082c0100 svchost.exe

Token1      2   3   4  5        6    7   8   9 10       11   12         13         14     15
  1744    848 Thu Feb 04 00:02:53 2010 Thu Feb 04 00:04:23 2010 0x01eaea88 0x082c0380 wmiprvse.exe

Token1      2                                                            3          4      5
     4      0                                                   0x025c8830 0x00319000 System

Our for loop will give us 4 variables a, b, c, and d which represent the 1st, 5th, 10th, and 15th token. We have to use a little trick to figure out which of the three variables contains the process name by checking each variable from right to left. If %d is not empty, then it contains the process name so we set Name equal to %d. If %d is empty we try %c, and if %c is empty we use %b. For the sake of nice variable names we set !pid! equal to %a. We then have the variable !pid!, which contains the process id, and !name!, which contains the process name.

We then search the pslist.txt file to see if the current process, represented by !name! and !pid!, is in the file. We output the file, using the Type command, and use FindStr to search for the matching name and process id. The /B switch says our search string must be at the beginning of the line, the /R enables regular expression searches. The default FindStr setting is to treat a space in our search string as a logical OR, but the /C switch "uses [the] specified string as a literal search string," meaning it doesn't treat a space as a logical OR. In short, it looks for the process name at the beginning of the line, followed by some number of spaces, then the process id, and then another space.

We then use the logical OR (||) in conjunction with the FindStr command to determine whether FindStr found something or not. This trick has been used repeatedly, but most recently in episode 122. If FindStr doesn't find anything we then output the process name and PID. This effectively gives us a list of processes that are found with psscan2 but not pslist.

Now for a more robust solution using...

PowerShell

I'm going to deviate into script land here, only because this mini-script may be very useful for manipulating the output of these commands. It will take the output and objectify it.

Objectifying psscan2:

PS C:\> $null, $pslist = python volatility pslist -f memory.img
PS C:\> [regex]$regex = '(?<Name>\S+)\s+(?<PID>[0-9]+)\s+(?<PPID>[0-9]+)\s+(?<Threads>[0-9]+)\s+(?<Handles>[0-9]+)\s+(?<Time>.*)'
PS C:\> $pslistobjects = foreach ($p in $pslist) {
...        $psobj = "" | Select-Object Name, PID, PPID, Threads, Handles, Time
...        $p -match $regex | Out-Null
...        $psobj.Name = $matches.Name
...        $psobj.PID = $matches.PID
...        $psobj.PPID = $matches.PPID
...        $psobj.Threads = $matches.Threads
...        $psobj.Handles = $matches.Handles
...        $psobj.Time = [datetime]::ParseExact($matches.Time.Trim(), "ddd MMM dd HH:mm:ss yyyy", $null)
...        $psobj
...     }

PS C:\> $pslistobjects | Format-Table
Name            PID  PPID Threads Handles Time
----            ---  ---- ------- ------- ----
System          4    0    55      260     1/1/1970 12:00:00 AM
smss.exe        540  4    3       21      1/28/2010 4:11:40 PM
csrss.exe       604  540  12      363     1/28/2010 4:11:46 PM
...

This takes the output from pslist and converts it to PowerShell objects. Let's look at each line, one at a time.

PS C:\> $null, $pslist = python volatility pslist -f memory.img

Here we get the output from pslist, send the first line to null, and the remainder is put into the variable pslist. This effectively skips the first line (header).

PS C:\> [regex]$regex = '(?<Name>\S+)\s+(?<PID>[0-9]+)\s+(?<PPID>[0-9]+)\s+(?<Threads>[0-9]+)\s+(?<Handles>[0-9]+)\s+(?<Time>.*)'

The next chunk sets up our Regular Expression with named groupings.

PS C:\> $pslistobjects = foreach ($p in $pslist) {
...        $psobj = "" | Select-Object Name, PID, PPID, Threads, Handles, Time
...        $p -match $regex | Out-Null
...        $psobj.Name = $matches.Name
...        $psobj.PID = $matches.PID
...        $psobj.PPID = $matches.PPID
...        $psobj.Threads = $matches.Threads
...        $psobj.Handles = $matches.Handles
...        $psobj.Time = [datetime]::ParseExact($matches.Time.Trim(), "ddd MMM dd HH:mm:ss yyyy", $null)
...        $psobj
...     }

Inside the ForEach-Object loop is where the heavy lifting is done. First, an empty object is created. Then the Match operator is used to match the string using the regular expression and automatically populate the $matches variable. We then set each property of our object. The Time property is a bit special since the time format used by pslist isn't one of the formats that PowerShell/Windows natively understands. The variable $pslistobjects then contains PowerShell'ed objects from volatility's pslist. We can then sort, filter, or do perform all sorts of tricks once it has been PowerShellized.

A similar mini-script will objectify the output from psscan2:

PS C:\> $null, $null, $null, $psscan2 = \python25\python.exe volatility psscan2 -f memory.img
PS C:\> [regex]$regex = '\s*?(?<PID>[0-9]+)\s+(?<PPID>[0-9]+)\s(?<Created>.{24})\s(?<Exited>.{24})
          \s(?<Offset>[0-9a-fx]{10})\s(?<PDB>[0-9a-fx]{10})\s(?<Name>.+)'
PS C:\> $psscan2objects = foreach ($p in $psscan2) {
...        $psobj = "" | Select-Object Name, PID, PPID, Created, Exited, Offset, PDB
...        $p -match $regex | Out-Null
...        $psobj.Name = $matches.Name
...        $psobj.PID = $matches.PID
...        $psobj.PPID = $matches.PPID
...        $psobj.Offset = $matches.Offset
...        $psobj.PDB = $matches.PDB
...        if ($matches.Created.Trim()) {
...            $psobj.Created = [datetime]::ParseExact($matches.Created, "ddd MMM dd HH:mm:ss yyyy", $null)
...        }
...        if ($matches.Exited.Trim()) {
...            $psobj.Exited = [datetime]::ParseExact($matches.Exited, "ddd MMM dd HH:mm:ss yyyy", $null)
...        }
...        $psobj
...     }

PS C:\> $psscan2objects | ft

Name             PID  PPID Created              Exited               Offset     PDB
----             ---  ---- -------              ------               ------     ---
svchost.exe      932  672  1/28/2010 4:11:47 PM                      0x01ea3558 0x082c0100
wmiprvse.exe     1744 848  2/4/2010 12:02:53 AM 2/4/2010 12:04:23 AM 0x01eaea88 0x082c0380
svchost.exe      1132 672  1/28/2010 4:11:48 PM                      0x01eb4970 0x082c0160
mike022.exe      1956 672  2/2/2010 3:25:29 AM                       0x020155d8 0x082c02c0
...

If you are going to use these commands often I would highly suggest making these into script files. You could even pass the file name to these scripts and have it wrap the volititlity commands.

Ok, so now we have two variables, each contains the output of the respective volatility command.

PS C:\> $pslistobjects | ft

Name            PID  PPID Threads Handles Time
----            ---  ---- ------- ------- ----
System          4    0    55      260     1/1/1970 12:00:00 AM
smss.exe        540  4    3       21      1/28/2010 4:11:40 PM
csrss.exe       604  540  12      363     1/28/2010 4:11:46 PM
lsass.exe       684  628  18      341     1/28/2010 4:11:47 PM
...

PS C:\> $psscan2objects | ft

Name             PID  PPID Created              Exited               Offset     PDB
----             ---  ---- -------              ------               ------     ---
svchost.exe      932  672  1/28/2010 4:11:47 PM                      0x01ea3558 0x082c0100
wmiprvse.exe     1744 848  2/4/2010 12:02:53 AM 2/4/2010 12:04:23 AM 0x01eaea88 0x082c0380
svchost.exe      1132 672  1/28/2010 4:11:48 PM                      0x01eb4970 0x082c0160
mike022.exe      1956 672  2/2/2010 3:25:29 AM                       0x020155d8 0x082c02c0
...

Finally Now, we can then use the Compare-Object cmdlet to compare the two sets of processes.

PS C:\> Compare-Object $pslistobjects $psscan2objects -Property name,pid

name           pid  SideIndicator
----           ---  -------------
svchost.exe    932  =>
wmiprvse.exe   1744 =>
cmd.exe        1172 =>
msmsgs.exe     1664 =>
wordpad.exe    272  =>
alg.exe        1012 =>
VMwareTray.exe 1648 =>
cmd.exe        1748 =>
services.exe   672  =>
winlogon.exe   628  =>

The Property parameter is used to specify the properties to use for comparison. We can either use a single property or a comma separated list of property names.

From this output it is quickly apparent that there are 10 processes found by psscan2 that were not found by pslist.

Whew, that was a lot of work this week. I hope it gets me on Santa's Nice list...next year.

Davide is too cool for school

Davide Brini has once again punk'd me with this full-on awk attack:

awk 'FNR>1 && NR==FNR {a[$1,$2]; next} 
     FNR>3 && !(($NF,$1) in a)' \
        <(volatility pslist -f memory.img) \
        <(volatility psscan2 -f memory.img)

Obviously, Davide has a PhD in awk, so let me explain what's going on here. FNR is an internal awk variable that tracks the current "input record number"-- usually the line number-- of the current file. NR, on the other hand, tracks the total number of records (lines) seen so far across all files.

If you look at the first awk clause, the "FNR>1" is how Davide is skipping the first header line in the pslist output. The "NR=FNR" expression will only be true if we're processing the first input "file", i.e. the output of "volatility pslist ...". Once awk moves on to the second "file" (the psscan output), NR will keep on accumulating, but FNR will be reset to zero.

So the first clause is for handling the psscan output. If you look at what's happening in the curly braces, Davide is creating empty array entries indexed by process name ($1) and PID ($2). The "next" just tells awk to read and process the next line of input, skipping the second clause which applies to the psscan output.

So let's look at that second clause. We can only get here if "NR!=FNR", which means we're dealing with the psscan output from the second input "file". Here Davide is using "FNR>3" to skip the header lines. For all the other lines, "!(($NF,$1) in a)" is true if and only if there is no entry in the array "a" for this combination of process name ($NF) and PID ($1). If we don't find an entry then psscan is telling us about a process that's been hidden from pslist and we want to output the information about this process. Davide is relying on the implicit "{print}" behavior of awk to make this happen.

Davide points out that the output from the above command will not be sorted, but you can always pipe the results into sort if that's important to you:

awk 'FNR>1 && NR==FNR {a[$1,$2]; next} 
     FNR>3 && !(($NF,$1) in a)' \
        <(volatility pslist -f memory.img) \
        <(volatility psscan2 -f memory.img) | sort -n -k2,2

Nice job, Davide!

Michael has to stay late for passing notes

Wow, this Episode sure provoked a lot of interesting commentary. Michael Hale Ligh gave us a shout out from the volatility camp. He even wrote a small plugin for volatility, psdiff.py, that does the same thing as our command line kung fu:

# For http://volatility.googlecode.com/svn/branches/Volatility-1.4_rc1

import volatility.plugins.psscan as psscan 
import volatility.win32.tasks as tasks
import volatility.utils as utils

class PsDiff(psscan.PSScan):
    """Produce a process diff"""

    def calculate(self):
        addr_space = utils.load_as(self._config)

        # Build a dictionary of processes found by scanning. The keys are 
        # physical addresses and the values are the objects
        procs_scan = dict((p.obj_offset, p) for p in psscan.PSScan.calculate(self))

        # Build a dictionary of processes found by walking the linked list. 
        # The virtual addresses are converted to physical with vtop. 
        procs_list = dict((addr_space.vtop(p.obj_offset), p) for p in tasks.pslist(addr_space))
        
        # Create two sets of addresses so we can easily compute the difference 
        scan_addrs = set(procs_scan.keys())
        list_addrs = set(procs_list.keys())

        # Yield any objects that are found by psscan but not pslist 
        for addr in (scan_addrs - list_addrs):
            yield procs_scan[addr]

    def render_text(self, outfd, data):
        for p in data:
            outfd.write("{0:<8} {1:<16} {2}\n".format(p.UniqueProcessId, p.ImageFileName, p.ExitTime))

Michael's plugin uses "psscan" instead of "psscan2", so the output will be slightly different, but it shouldn't be that hard to switch things over to use "psscan2" instead if you prefer. Michael also provided a bit more explanation in his original email:

$ python volatility.py psdiff -f memory.dmp

Volatile Systems Volatility Framework 1.4_rc1
0        Idle             1970-01-01 00:00:00
940      cmd.exe          2008-11-26 07:45:49
660      services.exe     1970-01-01 00:00:00
808      taskmgr.exe      2008-11-26 07:45:40
924      svchost.exe      1970-01-01 00:00:00
592      csrss.exe        1970-01-01 00:00:00
992      alg.exe          1970-01-01 00:00:00
1016     svchost.exe      1970-01-01 00:00:00
828      svchost.exe      1970-01-01 00:00:00

The exit time of "1970-01-01 00:00:00" just means the field is empty (process is still active). I am doing the diff based on the address of EPROCESS objects, however its possible, though not very likely, that an address could get re-used...so for a more robust diff you may check other fields as well.

If you want to see other fields in the output, its rather easy because the Volatility types are auto-generated from Microsoft's PDB symbol files. For example since Windows defines a structure like this:

typedef struct _EPROCESS {
  ...
  char  ImageFileName[16];
  DWORD UniqueProcessId;
  ...
} EPROCESS, *PEPROCESS;

You can print those fields like p.ImageFileName and p.UniqueProcessId in the plugin.

Lastly, the csrpslist plugin discussed in Malware Analyst's Cookbook produces a diff using two alternate sources of process listings (the csrss.exe handle table and an internal linked list found in the memory of csrss.exe). There are many other sources as well...

Episode #126: Cleaning Up The Dump

Hal's directories are bloated

It's not politically correct to say, but sometimes in Unix your directories just get fat. And like most of us, as your directories get fat, they also get slow. This is because in standard Unix file systems, directories are implemented as sequential lists of file names. They aren't even sorted, so you can't binary search them.

For example, suppose you'd just been dumping your logs into a single directory for years. You could end up with a big pile of stuff that looks like this:

# ls -ld logs
drwxr-xr-x 2 root root 266240 Dec 18 15:49 logs
# ls logs | wc -l
7188
# ls logs
authpriv.20070808.gz
authpriv.20070809.gz
authpriv.20070810.gz
...

Almost 7200 files-- and as you can see the directory itself has grown to be about a quarter of a megabyte! In our example, the file names are "<log>.YYYYMMDD" with an optional ".gz" extension on the older log files that have been compressed to save space.

Well I want my directories to be fit and lean again, so I decided to move the files into a tree structure based on year and month. So I'll need to move each file to a new location such as "YYYY/MM/<log>.YYYYMMDD". That should prevent any single sub-directory from getting too bloated.

I think there are a lot of ways you could attack this one, but I decided to make some noise with sed:

# cd logs
# for file in *; do 
    dir=$(echo $file | sed 's/.*\.\([0-9][0-9][0-9][0-9]\)\([0-9][0-9]\).*/\1\/\2/'); 
    mkdir -p $dir; 
    mv $file $dir; 
done

Yep, that sed expression sure is noisy-- as in "line noise". What's going on here? Well I'm taking the file name as input and using sed to pull out the YYYY and the MM and reformatting them into a subdirectory name like "YYYY/MM". First I match "anything followed by a literal dot", aka ".*\.". Then I match four digits-- four instances of the set "[0-9]"-- followed by two digits. However, I enclose both groups of digits in parens-- "\( ... \)"-- so that I can use the matched values on the righthand side of the substitution. On the RHS, "\1" is the four-digit year we matched in the first parenthesized expression and "\2" is the month we matched second. So "\1\/\2" is the year and the month with a literal slash in between-- "YYYY/MM". Obvious, right?

But the sed is the hard part. Once that's over, it's a simple task to make the directory and move the file. And now our directory should be nice and skinny:

# ls
2007  2008  2009  2010
# ls -ld .
drwxr-xr-x 6 root root 266240 Dec 18 15:58 .

Wait a minute! We've only got four top-level directories under our logs directory, but the logs directory itself hasn't shrunk at all. Unfortunately, this is normal behavior for Unix-- once a directory gets big, it never loses the weight.

So how do we stop our directory from looking like Jabba the Hut? In Unix, you make a new directory and wipe out the old one:

# mkdir ../newlogs
# mv * ../newlogs
# cd ..
# rmdir logs
# mv newlogs logs
# ls -ld logs
drwxr-xr-x 6 root root 4096 Dec 18 16:09 logs

It's liposuction via cloning! A miracle of the modern age! OK, really it's a lame mis-feature of the Unix file system. But at least you now know what to do about it.

And now I want to see Tim push his big directories around. Hey Tim, your directory is so fat...
Tim feels bloated from all the Christmas food:

Hal, yo directories is so fat, when they floated around the ocean Spain claimed them as a new world.

Ok, so the joke is terrible, but the problem is real. Directories with a lot of files can really be a pain.

On Windows there isn't one directory that contains all the logs. Each service typically has its own subdirectory under C:\Windows\System32\LogFiles\. For example, the subdirectory W3SVC1 would contain the logs for the first instance of an IIS webserver. Also, with older version of Windows C:\Windows is replaced with C:\WinNT.

This LogFiles directory is used by Microsoft products and some third-party products, but of course the third-party products can put their log files in all sorts of other weird locations. For the sake of this article, we'll assume we are looking at IIS logs.

By default IIS log files are created daily with the naming convention of exyymmdd.log. Microsoft doesn't put the full four digit year, so we'll assume 20XX. Why assume post 2000? Because if you are running an IIS server from the last millennium it probably isn't your server any more (see pwned).

Let's start off by getting the names for our directories, and then we'll build on that. According to Microsoft's IIS Log File Naming Syntax, no matter what file format or regular rotation interval (month, week, day, hour), the format always is always:

<some chars describing format><YY><MM><other numbers as used in date format>.log

We can build a regular expression replace pattern to derive directory names from the file names:

PS C:\Windows\System32\LogFiles\W3SVC1> ls *.log | % { [regex]::Replace($_.name, '[^0-9]*([0-9]{2})([0-9]{2}).*', '20$1\$2') }
2010\01
...
2010\02
...
2010\03
...

We use a ForEach-Object (alias %) loop on the output of our directory listing (Get-ChildItem is aliased as ls). Inside the loop we use .Net to call the static Replace method in the Regex class. The Replace method takes three arguments: the input, the search pattern, and the replacement string. The input is the name of the file. The search pattern is slightly more complicated. Here is how the search pattern maps to the portions of the log created on January 16th of 2009, ex090116.log.

[^0-9]*    = ex (all the non-digits at the beginning of the file name)
([0-9]{2}) = 09 (two digit year)
([0-9]{2}) = 01 (two digit month)
.*         = 16.log (the rest of the name)

We then use the replacement string to build the directory name, where $1 represents the first grouping (year) and $2 represents the second grouping. Each grouping is designated by parenthesis. For more information on .Net and Regular Expression Replacement, see this article.

Notice, in our command above we used single quotes. That is because PowerShell will expand any strings inside double quotes before our Replace method had a chance to do any replacing. This means that PowerShell would try to convert $1 into a variable and not pass the literal string to the Replace method. Here is what I mean:

PS C:\> echo "Here is my string $1"
Here is my string

PS C:\> echo 'Here is my string $1'
Here is my string $1

We could use double quotes, but we would have to add a backtick (`) before the dollar sign. The resulting command would look like this:

PS C:\Windows\System32\LogFiles\W3SVC1> ls *.log | % {
  [regex]::Replace($_.name, '[^0-9]*([0-9]{2})([0-9]{2}).*', "20`$1\`$2") }

So now we have the directory name, let's create the directory structure and move some files! I'm not going to show the full prompt so the command is less cluttered.

> Get-ChildItem *.log | ForEach-Object {
  $dir = [regex]::Replace($_.Name, '[^0-9]*([0-9]{2})([0-9]{2}).*', "20`$1\`$2");
  mkdir $dir -ErrorAction SilentlyContinue;
  Move-Item $_ $dir }

Wow, that is a rather large command, so let's trim it down with aliases and shortened parameter names. We can't have a big ol' fat command with our nice lean directories.

> ls *.log | % {
  $dir = [regex]::Replace($_.name, '[^0-9]*([0-9]{2})([0-9]{2}).*', "20`$1\`$2");
  mkdir $dir -ea;
  move $_ $dir }

Inside our ForEach-Object loop we set $dir equal to the new directory name. We then create the directory. The ErrorAction (ea for short) switch tells the shell not to show us an error message or stop processing if there is a problem. In our case, we want to make sure the command continues to run even if the directory already exists. After the directory is created we move the file, which is represented by $_.

PS C:\Windows\System32\LogFiles\W3SVC1> ls

    Directory: Microsoft.PowerShell.Core\FileSystem::C:\Windows\System32\LogFiles\W3SVC1

Mode           LastWriteTime       Length Name
----           -------------       ------ ----
d----    12/19/2010 12:28 AM        <DIR> 2008
d----    12/19/2010 12:28 AM        <DIR> 2009
d----    12/19/2010 12:28 AM        <DIR> 2010

So now we can enter the new year with leaner and meaner directories. And yes, they are meaner. Directories get pretty ticked off when you trim their children.