Episode #29: Finding the Time

Hal Says:

When analyzing a file system after an incident, it's often useful to search for recently modified files. Yes, an attacker who's broken root could have reset the timestamps on the files they modified, but I've found that in many cases they don't bother. That means you can do something like this:

# find / -mtime -2 -ls

That will give you a detailed listing of all files that have been modified within the last two days.

This example points up a shortcoming of find, however-- the finest granularity you have for time-based searches is in terms of some number of days. But what if your IDS or firewall logs can tell you exactly when the attack occurred down to the second? Wouldn't it be nice if you could have your find command show you only the files that have been modified since that exact point in time?

Turns out there's a clever little idiom for doing precisely this:

# touch -t 200904291446.53 /tmp/timestamp
# find -newer /tmp/timestamp -ls

The touch command allows the root user to explicitly set the timestamp on a file. The format is YYYYMMDDhhmm.ss (the seconds are optional), so in the example above we're setting the timestamp to 2:46:53pm on Apr 29, 2009. Once you've set the timestamp appropriately, it's a simple matter of using "-newer" with find to locate all files that have been modified more recently than that date.

Ed responds:
I really hate it when Hal pulls something out of his... uh... ear that is trivially easy to do in Linux with bash but is a real pain in the... uh... neck in Windows cmd.exe. But, saddled with this shell, we must make do.

One option we have for getting files' last modification dates and times in Windows cmd.exe is to use the dir command as follows:

C:\> dir /a /tw /o-d

That'll display the contents of the current directory, with all files regardless of their attributes (/a), showing the timestamp (/t) of the last written time (w), ordered (/o) in reverse (-) by date (d). OK... it's a start. The most recently written-to files (i.e., "modified") are at the top of the list.

But, Hal's find command is doing recursion through the Linux file system. What happens when we add a happy little /s to dir to make it recurse through all of c:\?

C:\> dir /a /s /tw /o-d c:\

In the words of Steve Jobs, it's a bag of hurt. It just sucks. It sucks horrifically bad. You see, the contents of each directory are displayed, and those files in each directory are sorted by modified date, as we'd like... but it's done on a directory-by-directory basis, without regard to when each directory was altered. Doh! Adding a /b (for bare form of output) doesn't help us, because it that'll leave off the timestamp information that we crave.

Often in cmd.exe, if we can't get the file information we'd like from the dir command because of all of its baked-in limitations, we can get far closer to what we want with a FOR loop.

So, let's try this again, this time with a FOR loop. We'll use a FOR /R loop, because it recurses through the file system for us, finding files. How about this:

C:\> for /r %i in (*) do @echo %~ti, %~fi

This loop will recurse (/r) through all subdirectories (*) of our current directory (cd into c:\ if you want to go from the top), with an iterator variable of %i taking on the name of each file that we find. In our do clause, we turn off display of commands (@), and echo to standard output %~ti. That reference will display the timestamp associated with %i (that is, our file). As it turns out, the timestamp displayed here is the last modified time. We then display a comma (you'll see why shortly), followed by %~fi, which is a way to reference the full path of %i.

The output here will show modified_time, full_path_to_file, for all files in our given directory and its subdirectories.

(BTW, with the FOR /R, our iterator variable will only take on the value of file names... if you want files and directories, you could use FOR /D /R... yup... both /D and /R together in the same FOR loop).

You might think that we could sort them using the sort command. Unfortunately, Windows cmd.exe built-in sort is an alphanumeric sort, and cannot in anyway understand the dates we want to sort by. Ouch.

So, what can we do? Well, we could dump our output into a csv file thusly:

C:\> for /r %i in (*) do @echo %~ti, %~fi >> file_list.csv

Now you can see why I put that comma in between the timestamp and full file path. Now, we could open up that csv file in a spreadsheet, and sort it however we want. I know that's kinda cheating with respect to our rules of using only built-in features, but we simply lack a good sort program in cmd.exe. Thus, we often have to create csv's and use a spreadsheet.

But, wait... Hal threw us a bone with his follow-up command, which is looking for a file with a very particular date and timestamp during a forensics investigation. We can look for that easily in the output of our FOR /R loop, as follows:

C:\> for /r %i in (*) do @echo %~ti, %~fi | find "04/28/2009 02:06 PM"

Mr. Bucket mentioned to me that the forfiles command is worthy of a mention in this article. My first thought was... "That's a great tool, but it's only in the Resource Kits." I've used it before in investigations, and carry it around with me on a USB token along with a ton of other useful tools. But, Mr. Bucket's bringing it up did make me double check... and, lo and behold, forfiles is built-in to Vista and Windows 2008 Server! That's scrumptious! The best news I've had in the last 30 minutes, at the very least. Sorry, but it's not built into earlier versions of Windows, though, but is a separate download from the Resource Kits.

Using forfiles can really help us out here, and, on Vista and 2008, we've got it built in. Here's how we can satisfy Hal's original hankerin':

C:\> forfiles /p c:\ /s /d -2

This one will display files inside of c:\ (the /p means "path"), recursing subdirectories (/s), displaying those with a last modification date earlier than the current date minus two days. Note that the /d option can be used with a date as well, instead of a number of days, with the date expressed in MM/DD/YYYY format. Instead of a -, you can also use a + in front of the number or date to show all files modified after that given date.

So, we have many options... dir for really simple stuff, FOR /R and FOR /D /R for more complicated stuff, and forfiles if we're on Vista or 2008. It's nice to have options in life. :)

Episode #28: Environment-a-list

Ed starts out:

At a cmd.exe prompt, you can list all environment variables using the "set" command by itself:

C:\> set

Pretty simple. By default, there are a typically a lot of variables set, upwards of two or three dozen.

Another property of set involves the listing of groups of environment variables that start with a substring. For example, to see all of your environment variables that start with the letters pr, you could type:

C:\> set pr

It's kinda weird, but environment variables in Windows cmd.exe are case insensitive.

We can refer to any one of these variables in a command by using the %[var]%. Thus, we can see the value of the username variable by running:

C:\> echo %username%

Or, we can see the value using the set command with:

C:\> set username

Prior to Vista, Windows didn't include a built-in whoami command. The closest thing we have is this "set username". However, be aware that, depending on how your shell is instantiated, you may or may not have this variable set. For example, this variable is typically not set when Metasploit launches a shell on a compromised box. Windows Vista does include a whoami command, but that's a story for another article. We must keep focus when performing command line kung fu.

Other environment variables besides username that are worthwhile include our path variable:

C:\> set path
Path=C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\Program
PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH

The output here will show you both your path, plus the pathext variable (remember, set [string] shows all variables that start with string). The pathext variable is particularly interesting, because it tells us the priority of execution on the machine for various file name suffixes. By default, if I type in "command", but don't specify a suffix, command.com will be run first. If that doesn't exist, the system tries command.exe. Then, it moves onto command.bat and so forth. That behavior can be used for some evil stuff by creating a backdoor called cmd.com or ipconfig.com and inserting it somewhere in the user's path.

Windows, by the way, includes an implicit "." at the very beginning of every user's path. That's a huge bummer from a security perspective, because users can be tricked into running a backdoor out of their current directory when they try to invoke some other command. Also, note that Windows doesn't seem to be too proud that "." is in the path, because it doesn't show it... but, it behaves as though it's there allright.

Other useful environment variables include %systemroot%, which tells you which partition and folder your primary Windows software is located in:

C:\> echo %systemroot%

Often when I'm handling an incident, I see attackers who assume the system root is on C:\. But, sometimes, it's not, and attackers get wrapped around the axel trying to figure out why their tweaks to the OS aren't having an impact. Whenever I'm doing a penetration test and I compromise a Windows box, I quickly display my systermroot and username variables, because that helps to orient me to the target machine.

Also, the prompt environment variable is called, simply enough, prompt. It's default value is $P$G, which indicates your current working directory ($P) followed by a greater than sign ($G). You could really freak yourself out by running:

C:\> set prompt=$$
$ ls

You now have a Linux wanna-be for your shell. I tell ya, when I see that $ prompt, and I just want to type "ls". Alas, no dice. :)

Hal responds:

It's interesting to me how even something as simple as environmental variables points out the differences between the Windows and Unix command-line philosophies. For example, instead of packing all kinds of functionality into a single command like Windows does with "set", Unix handles this by providing a toolkit of different programs that all interact with each other. For example, you use the "env" command to dump all of your current environment variable settings. If you want a specific subset, you pipe the output of "env" into "grep":

$ env | grep PATH
MANPATH=/usr/man:/usr/share/man:/usr/openwin/man:/usr/dt/man:/usr/local/man
CDPATH=.:/home/hal
LD_LIBRARY_PATH=/usr/lib:/usr/openwin/lib:/usr/dt/lib:/usr/local/lib
PATH=/bin:/usr/bin:/usr/X11R6/bin:/usr/local/bin:/sbin:/usr/sbin:...

This seems cumbersome-- you have to use two commands and a pipe instead of using just one "set" command with different parameters. But the power of this idiom is that you use the same knowledge of "grep" to search any kind of textual data in Unix, whereas under Windows you have to learn the particular idiosyncracies of a whole host of separate command line interfaces that each only deal with very specific types of data.

Anyway, returning to Ed's examples, you can display the value of a particular variable using "echo", and we typically set new values using "export" so that the variable setting will be passed along to any subshells we spawn:

$ echo $PATH
PATH=/bin:/usr/bin:/usr/X11R6/bin:/usr/local/bin:/sbin:/usr/sbin:...
$ export PATH=.:$PATH
$ echo $PATH
PATH=.:/bin:/usr/bin:/usr/X11R6/bin:/usr/local/bin:/sbin:/usr/sbin:...
$ export PATH=${PATH/\.:/}
$ echo $PATH
PATH=/bin:/usr/bin:/usr/X11R6/bin:/usr/local/bin:/sbin:/usr/sbin:...

Here I'm using the current value of $PATH in the "export" expression so that I can prepend "." (the current working directory) at the front of my search path. Obviously, as Ed points out above, this is a terrible idea from a security perspective, so I then use the same variable substitution operator we used back in Episode 26 to remove the leading dot. Again, I'm leveraging prior knowledge of basic shell building blocks to solve a new problem.

As Ed points out, Unix does have the "whoami" command, plus a number of other mechanisms for figuring out what user you are that provide different types of information:

$ whoami
hal
$ who am i
hal      pts/2        2009-04-26 14:44 (:0.0)
$ id
uid=1000(hal) gid=1000(hal) groups=4(adm),20(dialout),24(cdrom),...

Also, the your login program will generally set the $LOGNAME and $HOME environment variables, and your shell may also set the $USER and/or $USERNAME variables as well.

You can change your prompt by setting the $PS1 environment variable. For example, the default on my Ubunty machine is:

$ export PS1='\u@\h:\w\$ '
hal@elk:~$

There are a whole host of different escape sequences you can use here: "\u" is your user ID, "\h" is the unqualified host name, and "\w" is the current working directory. So if we wanted to emulate the standard Windows command prompt in our Unix shell we could use:

hal@elk:~$ export PS1='\w> '
~> cd /tmp
/tmp>

Of course, it doesn't look exactly like the Windows prompt, because when you're in your home directory, the directory is listed as "~", not $HOME. However, you can use the magic $PWD variable in your prompt instead of "\w" to get a more Windows-like experience:

/tmp> export PS1='$PWD> '
/tmp> cd
/home/hal> 

The emulation isn't perfect because Unix uses forward slashes in directory paths, not backslashes. I present the following for the truly psychotic:

/home/hal> export PS1='C:${PWD//\//\\\\}> '
C:\home\hal> cd /tmp
C:\tmp> 

You think having a "$" prompt will confuse the Windows folks? Try putting the above recipe into your /etc/profile file next April Fool's Day and watch your Unix users freak out.

By the way, you'll notice that the variable is $PS1. There's also $PS2:

C:\home\hal> export PS1='$ '
$ export PS2='more> '
$ for i in *
more> do
more> ...

Believe it or not, there are even $PS3 (the prompt used by the "select" command) and $PS4 (usually set to "+", this is the value printed before each command in an execution trace as with "set -x").

Paul Throws His Hat In:

One of the environment variables that I've encountered in the past that has really helped me is "LD_LIBRARY_PATH". The variable stores the path that will be used by programs to find software libraries. For example, if you've got a bunch of custom libraries in /opt/local/lib you could run the following command:

$ export LD_LIBRARY_PATH="/opt/local/lib:/usr/local/lib:/usr/lib:/lib"

This comes in handy when you are using OS X and something such as MacPorts is installed in /opt and has the libraries you are looking for. Yes, these are in fact the libraries you are looking for...

Episode #27: You'll Be History

Ed starts with:

In Episode #14, I touched briefly on how a cmd.exe user could hit the F7 key to view shell history, and we received a lot of positive feedback on that from folks. But, there's some more aspects of cmd.exe history that I use all the time that I think are worth bringing up here.

In particular, if you want to get access to your shell history at the command line itself so you can dump it to a file or manipulate it (rather than just displaying it on the screen so you can scroll through it with F7), you could run the following command:

C:\> doskey /history

Dude! The 80's just called, and they want their doskey command back. :)

Seriously, though, it remains a very useful command. Running doskey this way will show all of the commands typed into the current cmd.exe session. Unfortunately, you won't see history for other cmd.exe's on the screen or those invoked in the past. Still, when handling an incident, running this command with the output redirected to a file can be very useful evidence. If the bad guy was kind enough to leave a cmd.exe on the console screen of the machine he controlled the GUI on, you can quickly dump the history to an evidence file. Alternatively, if you, the incident handler or forensics analyst, are typing commands at a cmd.exe and want a record of what you've typed, invoking:

C:\> doskey /history > \\evidence_server\share\mycommands.txt

...can be very helpful to shoot your results to an evidence server somewhere.

Oh, and note that cmd.exe "telescopes" commands down, so that if you run the same command multiple times in a row, the history will only show the command once. For example:

C:\> dir > nul

C:\> ipconfig > nul

C:\> ipconfig > nul

C:\> ipconfig > nul

C:\> hostname > nul

C:\> doskey /history
dir
ipconfig
hostname
doskey /history

As we can see, the ipconfig command only appears in history once, even though we ran it three times in a row, due to this telescoping down effect.

By default, the command history stores 50 commands, but is configurable on a user-by-user basis. To change it at the command line for the current user, you could run:

C:\> reg add hkcu\console\^%systemroot^%_system32_cmd.exe /v HistoryBufferSize
        /t reg_dword /d 0x400

That last number there is a hexadecimal value for the number of commands you want it to remember. 0x400 will make it remember the last 1024 commands. This new history depth will only take effect when you launch new cmd.exe's going forward. But, you don't have to logoff or reboot for it to take effect. Just launch another cmd.exe, and you should have your new history depth (which can be viewed in the GUI by right clicking the title bar of a cmd.exe window, selecting "Properties", and going to the Options tab to view Command History Buffer Size).

Oh, and another thing about cmd.exe history... I often see bashketeers get a little freaked out by how cmd.exe handles shell history when you scroll through it with the up-arrow and down-arrow keys. As we all know, you can hit the up-arrow key in cmd.exe to scroll back to a previous command in modern Windows machines. When you find the command you want, you can hit Enter to re-run that command. No problem there. The issue that puzzles some people is what happens after you've run that command from your history, and you then hit the up arrow key again. Let me illustrate it with this screen shot.

Here, I've just hit 1 followed by Enter, to seed my history with some bogus commands so we can track things easily. Because there is no program called 1.exe, my shell tells me so. I do this for 2, 3, 4, and 5 as well.

Then, I hit the up arrow key to scroll back to my 3 command. I hit Enter on that. If I hit the up arrow again, of course I get the 3 again.

Now for the interesting part. What happens when I hit the up arrow again? If you were a bash guy, you might think that the system would display the number 5, because before you did 3, you did 5. And, you'd be wrong in a cmd.exe shell, which would display the number 2. What? Why 2?

Well, cmd.exe maintains a shell history pointer, and when you scroll back into your history and then run a command from it, this pointer jumps back to your earlier command. All up or down arrow movements from that point are relative to that older command, not your most recent commands.

This used to flummox me, with my ancient Unix background. But, you know what? I'll share a dirty little secret with you if you don't tell anyone. I actually have grown to like (and... gulp... prefer) this behavior. You see, there is a certain locality of command execution from history, and, if I execute a command that was 20 commands ago, there is a reasonably good probability that I'll want to run another command from 19 commands ago after it.

So, cmd.exe history scrolling isn't broken... it's just different. And, it's different for a reason. You might not agree with the reason, but at least there is one. So many things in Windows are different for no reason whatsoever, you know... but don't get me started about tracert. ;)

Hal Responds:

I was really happy when Ed brought us back around to command-line history again, because there was a bunch of stuff that I wanted to put into Episode #14, but decided not to in the interests of space. But there's no stopping me now...

While "doskey /history" is an obvious and intuitive way to access your command history, those wacky Unix folks have decided to force you to use the obscure "history" command instead (that was sarcasm for those of you who missed it because you were trying to remember the silly cmd.exe syntax):

# history
[...]
 1001  /etc/init.d/httpd restart
 1002  vi /etc/httpd/conf/httpd.conf
 1003  vi /etc/httpd/conf.d/ssl.conf
 1004  /etc/init.d/httpd restart
 1005  ps -ef | grep http
 1006  ps -ef | grep http
 1007  history

Oh, and notice that the bash history saves all of the commands you've typed, even the repeats. I think that's more useful-- especially from a forensic perspective-- than removing repeated commands, but I'm sure lord high doskey knows better than me.

See the numbers next to the commands in the history output above? You can use "!<n>" to execute the previous command with number <n>. For example, "!1001" would execute "/etc/init.d.http restart" again (as would "!1004"). I often use this when I'm debugging a problem with a server process like Apache-- try to start the server and fail, enter a bunch of commands to fix the problem, "!<whatever>" to try and start the server, lather, rinse, repeat. In other words, "!<n>" is useful when you are constantly re-running the same command with an arbitrary number of other commands in between attempts.

By default, the command shell keeps your last 500 commands, but you can change this by setting the HISTSIZE variable in your .bashrc or .profile:

export HISTSIZE=1024

Wow, you can even set it in decimal! Crazy! What will those wacky Unix folks think of next?

Episode #26: Renaming Files With Regular Expressions

Hal Says:

I admit it, I'm a fan of the CommandLineFu site (hey, it's a community, not a competition), and like trolling through it occasionally for interesting ideas. This post by vgagliardi shows a cool trick for renaming a bunch of files using regular expressions and the substitution operator in bash. For example, suppose I wanted to convert spaces to underscores in all file names in the directory:

$ for f in *; do mv -- "$f" "${f// /_}"; done

I realize that syntax looks a little crazy. The general form is "${variable/pattern/substitution}", but in this case we have an extra "/" at the front of "pattern", which means "replace all instances" rather than only replacing the first instance.

By the way, you can use the standard Unix regular expression syntax for your substitution pattern. For example, here's a loop to remove all characters from file names except for alphanumeric characters, dot, hypen, and underscore:

$ for f in *; do mv -- "$f" "${f//[^-_.A-Za-z0-9]/}"; done

In this case "[^...]" means match any characters not in the specified set, and we're performing a null substitution.

Did you notice that we're also using the "--" argument to the "mv" command, just in case one of our file names happens to start with a "-"? These files are typically a huge pain in Unix. What if we wanted to replace all the "-" characters at the beginning of a file name with underscores?

$ for f in *; do mv -- "$f" "${f/#-/_}"; done

As you can see, starting the pattern with "#" means "match at the front of the string. Or we can match at the end of the string with "%":

$ find docroot -type f -name \*.htm | \
     while read f; do mv -- "$f" "${f/%.htm/.html}"; done

Here we're using "find" to locate all of the *.htm files in our web docroot and then piping the output into a while loop that renames all these files to be *.html files instead.

There are a couple of problems with the method that I'm using here: (1) if multiple files map to the same name, you'll end up clobbering all but the last instance of that file, and (2) you get errors if your substitution doesn't actually modify the file name because the "mv" command refuses to rename a file to itself. We can fix both of these problems with a little extra logic in the loop. Let's return to our first example of converting spaces to underscores:

$ for f in *; do n="${f// /_}"; [ -f "$n" ] || mv -- "$f" "$n"; done

First we assign the new file name to the variable $n. Then we check to see if a file named "$n" exists-- the "mv" command after the "||" is only executed if there is no "$n" file.

I admit that I usually use the Perl rename program for renaming large numbers of files, because (a) the syntax is much more terse, and (b) I love Perl. But this program isn't always available on all the different flavors of Unix that I end up having to work on. So having this functionality built into the shell is a huge win.

Ed Responds:
When I quickly glanced at Hal's challenge initially, I thought... "Yeah, that's pretty easy... findstr supports regex, and I'll use the ren command to rename the files... No prob."

And then, I started to write the command, and it got horribly ugly really quickly. Hal squealed with delight when I told him how ugly it was... and believe me... you ain't seen nothing until you've seen Hal squeal with delight.

Anyway, to keep this article from getting unreasonably long, I'm going to address Hal's original command, which replaced the spaces in file names with underscores. Unfortunately, you see, the parsing, iteration, and recursion capabilities within a single command in cmd.exe are really limiting. For parsing strings, we've got FOR /F and a handful of substring operations I covered in Episode 12. For running a command multiple times, we've got for /L, as I mentioned in Episode 3. For recursion, well, that's just plain bad news in a single command unless we bend our rules to create a bat file that calls itself.

To start to address Hal's original challenge, we can use the following command to determine if there are any files that have at least one space in their names in our current directory:

C:\> dir /b "* *"

That's pretty straightforward to start, with the /b making dir show only the bare form of output, omitting cruft about volume names and sizes. Note that it will only show files that do not have the hidden attribute set. If you want, you can invoke dir with /a to make it show files regardless of their attributes, hidden or otherwise. Now, let's see what we can do with this building block.

Plan A: Every File Should Have Four Spaces in Its Name, Right?
My original plan was to wrap that command inside a FOR /F loop, iterating over each file using FOR /F functionality to parse it into its constituent elements. I was thinking something like this:

C:\> for /F "tokens=1-4" %i in ('dir /b "* *"') do ren "%i %j %k %l" "%i_%j_%k_%l"

Well, that's all very nice, but we've got a problem... let me show an example of what this beast creates when I run it in a directory with a file named "file 1.txt" and "file 2 is here.txt":

C:\> dir /b
file_1.txt__
file_2_is_here.txt

Ooops... the file1.txt name has two underscores after it. This option only works if files have exactly four spaces in their names. That's no good.

Plan B: Let's Just Change the First Space into an Underscore
Well, how about this... We could write a one-liner that assumes a file will have only one space in its name, and convert that one space into an underscore. That's not too bad:

C:\> for /f "tokens=1,*" %i in ('dir /b "* *"') do ren "%i %j" "%i_%j"

I'm parsing the output of the dir /b command using parsing logic of "tokens=1,*", which means use your default delimiters of space and break each line of the output of the dir command into the entity before the first space into %i, and everything afterward into the next iterator variable, %j.

Let's run that with our same file names as before, yielding:

C:\> dir /b
file_1.txt
file_2 is here.txt

Well, we got it right for file_1.txt, because there is only one space. But, we only fixed the first space in file 2 is here.txt. Hmmmm... How could we move closer to our result?

Hit the up arrow a couple times to go back to our Plan B FOR /F loop, and hit enter again. Now, running our dir, we get:

C:\> dir /b
file_1.txt
file_2_is here.txt

Ahh... we nailed our second space. Hit the up arrow again and re-run... and... well, you get the picture. We can take care of one space at a time. Not so bad.

But, who wants to hit the up arrow again and again and again until we get rid of all the spaces? You'd have to re-run my Plan B command N times, where N is the maximum number of spaces inside a file name in the current directory.

Plan C: Make the Shell Re-Run the Command Instead of Doing it Manually
Well, instead of re-running a command a bunch of times, let's make the shell do our work for us. We'll just wrap the whole thing in a FOR /L loop to count through integers 1 through 10 (1,1,10) and invoke the FOR /F loop at each iteration through our FOR /L loop:

C:\> for /L %a in (1,1,10) do @for /f "tokens=1,*" %i in ('dir /b "* *"')
  do ren "%i %j" "%i_%j"

That works, provided that none of the files have more than ten spaces in their name. Ummm... but what if they do? We could raise the number 10 to 20... but that's kind of a cheap hack, no?

Plan D: Violate the Rules -- Make a 3-Line Script
OK... if we had a while construct in cmd.exe, we could simply run my FOR /F loop of Plan B while the dir /b "* *" still returned valid output. But, we don't have a while command in cmd.exe. If we want to check a condition like that, we only have IF statements. And, if we want to jump around based on the results of IF statements, we need to use GOTOs. And, if we want to use IFs and GOTOs, we can't dump everything on a single one-line command, but will instead have to create a little bat file.

So, I'm going to have to bend our ground rules for this blog, which require a single command, and instead use a three-line bat file. Here's a bat file I wrote that converts all of the names of files in the current directory with spaces in them into underscores:

:begin
for /F "tokens=1,*" %%i in ('dir /b "* *"') do ren "%%i %%j" "%%i_%%j"
if exist "* *" goto begin

There you have it.... kind of an ugly little hack, but it works. Note that I had to change my iterator variables in my FOR loop from %i and %j into %%i and %%j. You have to do that to convert command-lines into bat files in Windows. Also, I'm using an IF statement to test for the existence of "* *", which would match any file with a space in its name.

A small script in cmd.exe can satisfy Hal's original challenge. To start addressing his other feats to convert other characters in file names, we could specify options for the FOR /F loop of everything we want to parse out with the syntax "tokens=1,* delims=~!@#$%^&*()+=" and whatever else you wanna take out.

I could drone on and on endlessly here, but I think you get the idea. It ain't pretty, but it is doable.... Now that should be the cmd.exe mantra.

PS: I too am a fan of the CommandLineFu site. It rocks.

Episode #25: My Shell Does Math

Ed says:

On Linux and Unix, I frequently rely on bc to do math at the command line. But, bc isn't built-in to Windows. A lot of people don't realize that the cmd.exe shell has some built-in capabilities for doing math... really lame math. But still, if you are caught in a pinch, don't wanna invoke the calc.exe GUI, and want to focus on integer math, cmd.exe can do your bidding. It's all invoked with the "set /a" command, as in:

C:\> set /a 2+2
4

Sexy, huh? Well, keep in mind that it is integer math only, as illustrated by:

C:\> set /a 3/2
1

That leads to some weirdness, as in:

C:\> set /a 3/2*2
2

Also, we're working only with signed 32-bit numbers here, so make sure your results stay under approximately two billion. Bigger than that will either push you into negative territory, or outright give you an error message:

C:\>set /a 1000000000*2
2000000000

c:\test>set /a 2000000000*2
-294967296

c:\test>set /a 3000000000*2
Invalid number.  Numbers are limited to 32-bits of precision.

We've also got some options for using hex (specified with 0x as a prefix):

C:\> set /a 0xA + 0x2
12

Or, you can do octal, by prefacing your numbers with a zero:

C:\> set /a 010 + 01
9

You can even mix and match:

C:\> set /a 0xA + 01 + 1
12

It gets exciting to think that you can do hex or octal math at the cmd.exe command line, until you realize that all of your outputs are in, well, decimal. What genius thought up adding hex and octal as input, without providing options for hex and octal in the output?

Also, we've got bitwise AND with &, bitwise OR with |, and XOR with ^. But, again, the output is in... ugh... decimal.

Who uses decimal anymore? I mean, when you are driving, those speed limit signs are all in hex, right? Cruising around at 0x65 mph or 0x100 km/hr can be very exciting. When pulled over, tell the officer that you thought the speed limit sign was hexadecimal. Of course, you'll have to say that you assumed your speedometer is in decimal... kinda like set /a. Inputs in hex, outputs in decimal. See if that'll get you out of a ticket. Good luck.

Anyway, for really simple integer math of smallish numbers, set /a can come in handy. I do sometimes use it if I've gotta do a quick integer calculation and I don't wanna take focus off of my cmd.exe on the screen.

Honestly, given the quite limited abilities for cmd.exe to do math this way, I'm kind of expecting my command line kung fu sparring partners working in bash to knock me out with this one. But, in leading with my jaw, I'm hoping in the process that we'll all learn some completely magical fu from them when doing math in bash. So, lay it on us, guys. What fu have you?

Mr. Bucket read the draft of this post, and kicked in some interesting fu from the dawn of time for doing some hex match using built-in Windows command line tools. I remembered this one, but barely. It was buried in the dark recesses of my mind... I haven't used this since I bought a really good hex calculator a long long time ago.

Mr. Bucket suggested running the built-in Windows debugger tool:

C:\> debug

Then, at the goofy little "-" prompt, you could type:

- H 4a44 90

The debugger will then print out the sum and difference of the two numbers you provided:

4AD4  49B4

So there.... next time you are stranded on a desert island with only a Windows box and need to do hex addition or subtraction to hack your way off, you'll be all set. Thanks for the interesting point, Mr. Bucket!

Paul Says:

On UNIX/Linux systems I've always found that the bc command was available to me if I need to perform simple math (or even that "new" math people talk about):

$ bc
bc 1.06
Copyright 1991-1994, 1997, 1998, 2000 Free Software Foundation, Inc.
This is free software with ABSOLUTELY NO WARRANTY.
For details type `warranty'.
3/2*2
2
quit

It can even do fancy things like square root:

$ bc
bc 1.06
Copyright 1991-1994, 1997, 1998, 2000 Free Software Foundation, Inc.
This is free software with ABSOLUTELY NO WARRANTY.
For details type `warranty'.
sqrt(3.14)
1.77
quit

Of course Bash itself can do math as well. This is handy in shell scripts. Below is a command line "one-liner" example:

$ let RESULT=2+2 ; echo $RESULT
4

As long as "2+2" continues to equal 4 the universe stays in balance and life is good :) Of course, if you leave out the "let" directive you get very different results:

$ RESULT=2+2 ; echo $RESULT
2+2

Here "2+2" is being treated as a string, and therefore no math will be performed. So remember, when using Bash always "Let there be math".

Hal Says:

Ummmm, Paul? How about just:

$ echo $((2+2))
4

Using "$((...))" to put arithmetic expressions in-line is a heck of a lot easier than that tedious "let ..." nonsense.

By the way, for those of you that aren't using bash or a more modern shell with built-in arithmetic, there's also the really old-school expr command in addition to bc:

$ expr 2 + 2
4

Episode #24: Copying and Synchronizing (Remote) Directories

Hal Says:

Copying files and directories around is a pretty common task, and there are all sorts of ways to accomplish this in Unix. But I have a strong preference for using the rsync command because (a) it allows me to copy files either within a single machine or between systems over the network, (b) it can keep directories in sync, preserve timestamps, ownerships, etc, and (c) it's smart enough to only copy files that have changed, which is a big savings on time and bandwidth.

Basic rsync usage is straightforward:

$ rsync -aH dir1/ dir2/                              #copy dir1 to dir2 on same machine
$ rsync -aH remotehost:/path/to/dir/ localdir/       #copy remote dir to local machine
$ rsync -aH dir/ remotehost:/path/to/dir/            #copy local dir to remote machine
$ rsync -aH dir/ someuser@remotehost:/path/to/dir/   #copy dir to remote host as someuser

The "-a" option combines the most common rsync options for copying and preserving directories: "-r" for recursive, "-ptog" to preserve permissions/timestamps/owner/group owner, "-l" to copy symlinks as symlinks, and "-D" to preserve device and special files. The only useful option not included in "-a" is the "-H" option to preserve hard links. Even though "-H" adds extra processing time, I tend to always use it, just to be on the safe side.

By the way, the trailing "/" characters on the source directory paths are significant. I could explain it in words, but it's easier to just show you an example:

$ ls source
bar.c  baz.c  foo.c
$ rsync -aH source dir1
$ ls dir1
source
$ rsync -aH source/ dir2
$ ls dir2
bar.c  baz.c  foo.c

Without the trailing "/", rsync copies the entire directory, including the top directory itself. With the trailing "/", the contents of the source directory are put into the top of the destination directory.

If you want to keep two directories in sync, you probably want to include the "--delete" option, which deletes files in the target directory that aren't present in the source:

$ rsync -aH source/ target
$ rm source/foo.c
$ rsync -aH source/ target
$ ls target
bar.c  baz.c  foo.c
$ rsync -aH --delete source/ target
$ ls target
bar.c  baz.c

Sometimes there are certain files that you don't want to copy:

$ ls source
bar.c  bar.o  baz.c  baz.o  foo.c  foo.o
$ rsync -aH --delete --exclude=\*.o source/ target
$ ls target
bar.c  baz.c  foo.c

Here I'm using "--exclude" to not copy the object files, just the source files. You can put multiple "--exclude" options on a single command line, but after a while it gets annoying to keep typing the same set of excludes over and over again. So there's also an "--exclude-from=<file>" option to read in a list of exclude patterns from the file "<file>".

rsync also has a "-n" option like the "make" command, which shows you what would happen without actually copying any files. You usually want to combine "-n" with "-v" (verbose) so you can actually see what rsync would be doing:

$ rsync -aHnv --delete --exclude=\*.o source/ newdir
sending incremental file list
created directory newdir
./
bar.c
baz.c
foo.c

sent 90 bytes  received 24 bytes  228.00 bytes/sec
total size is 0  speedup is 0.00 (DRY RUN)

Ed Joyfully Responds:
You thought you had me with your fancy shmancy rsync, didn't ya, Hal? Well... ha! A couple of years ago, you'd have been right, because there was no good built-in Windows tool for synchronizing directories. The xcopy command is a decent file copy tool, but it doesn't really police file updates to maintain synchronicity (and yes, that oblique reference to "The Police" was intentional). A couple of years ago, you'd have to install a separate tool for synchronizing. A really nice tool for doing that is robocopy, available in various resource kits, such as the Windows 2003 kit here.

But, here's the really good news: robocopy is built-in to Vista and Windows 2008 server! It's a really nice synchronization tool, and its name is far cooler than the rather pedestrian "rsync". "Robocopy" sounds like a cool mechanized buddy, or perhaps even a robotized superhero law enforcement officer.

So, how can we use robocopy to achieve what Hal does with rsync above? Well, to mirror from one local directory to another local directory, you could run:

C:\> robocopy dir1 dir2 /s

The /s makes robocopy recurse subdirectories. You could put the /s up front, but, generally speaking, it's best to put the source and destination directories first with robocopy, especially when you start to define file and directory exclusions. Note that robocopy cannot copy hard links, so we lose them (i.e., there is no rsync -H equivalent). Note also that robocopy works the same way whether you specifiy dir1 or dir1/, unlike rsync. That's ok with me even though it is slightly less flexible, as there is less of a chance that I'll screw something up.

To use robocopy to replicate something to or from a remote directory, just refer to the directory as \\[machine]\[share]\[dir], as you might expect, as in:

C:\> robocopy plans_for_world_domination \\backup\rainbowsANDunicorns /s /z

Another nice feature associated with using robocopy across a network involves what happens when network connectivity is lost. If you invoke it the right way, robocopy maintains status so that it can pick up where it left off when doing a copy. When you invoke robocopy, use the /z option to run it in restartable mode. The /z makes it maintain the status information necessary to restart a copy that is interrupted.

If you want to keep directories in sync (removing files from the destination that have been deleted from the source), you can use the /MIR option (/MIR actually means /E plus /PURGE). As with rsync, robocopy will copy all files by default. To omit certain files, we have a huge number of exclusion options, such as /XF to exclude files and /XD to exclude directories, both of which support wildcards with *.

Thus, to mimic Hal's fu above, we could run:

C:\> robocopy source target /S /MIR /XF *.o 

Oh, and to do a dry run, just printing out information about what would be copied, instead of actually doing the copies, we can invoke robocopy with the /L option, as in:

C:\> robocopy source target /L /S /MIR /XF *.o 

If you'd like to copy all file attributes, ownership information, and file permissions, invoke robocopy with /COPYALL.

The output of robocopy is quite nice as well, showing detailed statistics about what was copied, what was skipped (because it was already there, or was excluded), detailed times, and a timestamp of invocation and completion:

c:\> robocopy source target /L /S /MIR /XF *.o

-------------------------------------------------------------------------------
ROBOCOPY     ::     Robust File Copy for Windows

-------------------------------------------------------------------------------

Started : Sun Apr 12 07:54:51 2009

Source : c:\test\source\
Dest : c:\test\target\

Files : *.*

Exc Files : *.o

Options : *.* /L /S /E /COPY:DAT /PURGE /MIR /R:1000000 /W:30

------------------------------------------------------------------------------

                     4    c:\test\source\
      New File                   4        bar.c
      New File                   4        baz.c
      New File                   4        foo.c
                     1    c:\test\source\hello\

------------------------------------------------------------------------------

          Total    Copied   Skipped  Mismatch    FAILED    Extras
Dirs :         2         0         2         0         0         0
Files :         5         3         2         0         0         0
Bytes :        24        12        12         0         0         0
Times :   0:00:00   0:00:00                       0:00:00   0:00:00

Ended : Sun Apr 12 07:54:51 2009

Yeah, robocopy!

Episode #23: Job Control

Hal Says:

Paul's last post left me hankering to talk a little bit more about job control in the Unix shell. Besides, it'll make Ed feel really bad about the Windows command shell, so what's not to like?

The simplest form of job control is to hit "^Z" to suspend a currently running process. Once the process is suspended, you get your command prompt back and can issue other commands. You can also type "bg" to force the suspended process into the background, where it will continue to run:

# tail -f /var/log/httpd/error_log
[... output not shown ...]
^Z
[1]+  Stopped                 tail -f /var/log/httpd/error_log
# bg
[1]+ tail -f /var/log/httpd/error_log &

Now you can fiddle with your Apache config file, restart the web server, etc. The "tail -f" process is still running in the background and displaying errors as they appear in the log file. I find this very useful if I'm working remotely to resolve problems on my web server. Actually, I could have simply started that "tail" command in the background by putting a "&" at the end of the command line:

# tail -f /var/log/httpd/error_log &
[1] 14050

I often use job control to flip in and out of a superuser shell:

$ /bin/su
Password:
# [... do some stuff as root ...]
# suspend

[1]+  Stopped                 /bin/su
$ [... do some stuff as a normal user ...]
$ fg
/bin/su
# 

Notice that you use "fg" to start a suspended process running again. This sure beats having to keep using "su" and re-enter the root password all the time. Of course, you probably should be using "sudo", but that's a topic for another day.

By the way, you can also use job control to blip in and out of remote SSH sessions too:

[hal@localhost ~]$ ssh remotehost
hal@remotehost's password:
Last login: Sun Apr  5 10:06:26 2009 from ...
[hal@remotehost ~]$
[hal@remotehost ~]$ ~^Z [suspend ssh]

[1]+  Stopped                 ssh remotehost
[hal@localhost ~]$ fg
ssh remotehost

[hal@remotehost ~]$ 

Note that the key sequence to suspend an SSH session is "\n~^Z"-- you have to enter the tilde character at the beginning of a line, not while you're in the middle of a command-line. By the way, if you're logged into multiple machines in series, you can use multiple tildes to pick which of the chained SSH sessions you actually want to suspend.

Sometimes you might have multiple jobs suspended at the same time. You can use the "jobs" command to display them, and use the job number displayed in the output to interact with the different jobs:

# jobs
[1]   Stopped                 vi /etc/httpd/conf/httpd.conf
[2]-  Stopped                 vi /etc/httpd/conf.d/ssl.conf
[3]+  Stopped                 tail -f /var/log/httpd/error_log
# bg %3
[3]+ tail -f /var/log/httpd/error_log &
# fg %1
vi /etc/httpd/conf/httpd.conf
^Z
[1]+  Stopped                 vi /etc/httpd/conf/httpd.conf
# %2
vi /etc/httpd/conf.d/ssl.conf
^Z
[2]+  Stopped                 vi /etc/httpd/conf.d/ssl.conf
# fg
vi /etc/httpd/conf.d/ssl.conf

In general you do "[fg|bg] %n" to foreground or background job number "n". However, you can leave off the "fg" if you want because foregrounding a job is the default. If you don't specify a job number after the "fg" or "bg", then the most recently manipulated job number is assumed-- you can see this job with a "+" next to it in the output of the "jobs" command.

Ed Wimpers:

Job control?!? What are you, Hal... just plain cruel? I long for true job control in my cmd.exe. If I want true job control, I just install Cygwin.

But, there are some job control-like things I can do in a pinch on a box without Cygwin.

For example, if I want to start a process in a separate cmd.exe window, I can use the start command. Here, I'll use it to start a separate shell window to search for wmic.exe on the C:\ partition, using the file search capabilities we described in Episode #21:

C:\> start dir /s /b c:\wmic.exe

That pops up a separate Window that will display my results. The window will linger displaying my results after it's done. If I want to start it up minimized, I can run:

C:\> start /MIN dir /s /b c:\wmic.exe

If I want it to start a process in a separate window and then have it disappear when it is done running, I use:

C:\> start cmd.exe /c dir /s /b c:\wmic.exe

Sadly, there's no easy way to get access to standard out of these separately invoked shells windows while they are running, other than to have them dump their output into a file, with "> filename" at the command line invocation.

But now for the tricky part... how can you mimic the & feature of most Linux shells to run something in the background of the current cmd.exe while still dumping their standard output into the current shell's display? We can use the /b option of start to put something in the background, as in:

C:\> start /b dir /s /b c:\wmic.exe

So, we've got &-like behavior! However, it's not really full job control, because I can't pull that job into the foreground, or select one job from a group of backgrounded tasks. But, I can kill my background task, by hitting CTRL-break in the cmd.exe window that spawned it. Also, each time you run "start /b", it will leave an extra cmd.exe process running until you either exit that process (by typing... you guessed it... "exit") or killing it.

To kill "jobs" (if I can call them that) more thoroughly, I usually rely on wmic or taskkill, as defined in Episode #22.

So, there you have it... not quite job control, but an ability to kick things into the background so you can keep your current shell active and get more work done. Oh, and install Cygwin. You'll be glad you did. :)

Paul Says:

I will never forget when I was first learning Linux/UNIX (I actually got my feet wet with Linux/UNIX by using Linux at home, and AIX at work). I was configuring a proxy server, carefully constructing each command line switch and running the command. I'd hit ^Z to put the command in the background and wonder why my proxy server wasn't listening. I had forgotten to put the & after the command, so it truly did "suspend" the command. I never made that mistake again :)

I would also like to quickly highlight the nohup command. I often use this when I run a command that will take a long time and want to go back and check its output:

$ nohup nmap -sS -T4 -iL edswindowsnetwork -oA myscanresults &

Sometimes processes you start while logged into a remote host will SIGHUP when you logoff, and nohup can prevent that.

Episode #22: Death To Processes

Paul Says:

Ever been in the mood to just kill things? This happens to me when I use Windows. There are more interesting ways to kill things in Linux than you think, for example:

# killall top

Sometimes you want to put processes in the background. You can do this by appending "&" after the command, or hitting CTRL-Z and typing "bg" once the command has been started. But what if you try to quit the process and you are ignored? Reboot? (No, that's just if you're running Windows and this happens). You can hit CTRL-Z and then use the kill command as follows:

$ while :; do cat /dev/null; done
^Z
[1]+  Stopped                 cat /dev/null

The process is stopped, but it's still running:

$ ps waux | grep null
root     77241   0.0  0.0   599780    440 s005  R+   10:28PM   0:00.00 grep null
root     77238   0.0  0.0   599700    372 s005  S    10:28PM   0:00.00 cat /dev/null

To kill the previous command that was put in the background, do the following:

$ kill %1

[1]+  Stopped                 cat /dev/null

Now it's not running at all:

$ ps waux | grep null
root     77243   0.0  0.0   599780    388 s005  R+   10:28PM   0:00.00 grep null
[1]+  Terminated              cat /dev/null

Now I will just wait until Hal hands something to me, usually it's something that rhymes with glass.

Hal Says:

Of course, if Paul wanted to kill the process, I'm wondering why he didn't just hit ^C instead of ^Z. Seriously, though, there's all kinds of interesting ways to kill processes in Unix. In addition to the killall command Paul's using, there's also pkill. I actually prefer pkill to killall, because it lets me do things like:

# pkill -P 1 sshd

This command means "kill all sshd processes whose parent process ID is 1", which kills only the master sshd process leaving all of the users on the system still logged in. Or I could kick Paul off the system:

# pkill -u paul

Did you know that you can use lsof to help you kill processes? Check this out:

# umount /home
umount: /home: device is busy
# kill `lsof -t /home`
# umount /home

The "-t" option to lsof tells it just to output the PIDs of the matching processes-- in this case all processes that have files open under /home. Once all of those processes are dead, you can successfully unmount the volume. Note that you're going to have some very unhappy users after this action, but if you absolutely positively have to unmount a file system for some reason...

Another point to mention is that you can kill commands via the top command. Just hit "k" in the window where you're running top and enter the PID you want to kill. This technique is useful for killing a runaway process that's eating all your CPU time. You can also use it to kill processes that are eating tons of memory: hit "O" to pick a different sorting criteria and select "q" to sort by Resident Size. Now you'll see the processes sorted by the "RES" column and it will be easy to pick out and kill the ones eating lots of memory.

Ed Steps up to the Plate:
Killing processes in Windows? Man, we have a lot of options. One of my favorite ways is to use wmic, God's gift to Windows. While you can manipulate many (most?) aspects of Windows using WMIC, here, we'll use it to kill processes by running:

C:\> wmic process where [where_clause] delete

It's the where_clause where we have some real power in fine tuning what we want to kill. To kill a process based on its name (like Paul's killall above), we can run:

C:\> wmic process where name="calc.exe" delete

Here, I've just killed any process named calc.exe, the little Windows calculator.

Would you rather kill something based on its pid number? You could run:

C:\> wmic process where processid="536" delete

You can construct where clauses around all kinds of process attributes, including executablepath, parentprocessid, or even commandline options. To see the attributes exposed via wmic process, run:

C:\> wmic process get /?

You can even write where clauses to match multiple attributes using an "and", putting in some parentheses where appropriate. To mimic Hal's fu about killing a process named sshd with a parent processid of 1, we could run:

C:\> wmic process where (name="sshd" and parentprocessid="1") delete

Of course, on a real Windows box, you likely won't use that specific name and parentprocessid, but you get the idea and can easily adapt it.

In addition to "and", note that "or" is also supported. Plus, you can do substring matching with where clauses that include "like" and "%", as follows:

C:\> wmic process where (name like "%bot.exe") delete

That'll kill any process with a name that ends with bot.exe.

But, there's one important attribute missing from wmic process: the user name! Doh! Thanks for nothing, Microsoft.

But, sometimes you really want to kill all processes running as a given user... Paul sometimes can be a bother when he logs onto a machine, after all. For this, we can rely on the taskkill command. That command allows us to kill processes based on all kinds of things, including processid (specified with /PID [N]) or process name (specificed with /IM [name]). But, taskkill gets really useful with its filtering rules, specified with /FI "[filter]". Let's kill all processes running as user paul:

C:\> taskkill /F /FI "username eq paul"

The /F means to forcefully terminate a process. The taskkill filtering language is very flexible, supporting wildcard strings of *, and offering options for eq (equals), ne (not equals), gt, lt, ge, le for a variety of process attributes. Various attributes we can filter on include status (running or not running), memusage (if you're a memory hog, I'll kill ya!), username (you'd better behave, Paul), modules (if a process has loaded metsrv.dll, the Metasploit Meterpreter, I can kill it! Oh, and if it's a critical system process, I just hosed my machine), services (kill the process inside which a given service runs), and even windowtitle. That's a lot of precision, letting you target your process kills.

Specific filtering syntax is available by running:

C:\> taskkill /?

Episode #21: Finding & Locating Files

Paul Writes In:

I'm one of those messy desktop people. There I said it, I keep a messy desktop with tons of files all over the place (partly due to the fact that when you do <4> in OS X to take a screen grab it puts the file on the desktop). So, it should some as no suprise that I often need help finding files. I don't know how many of you have actually run the find command in OS X (or even Linux), but it can be slow:

# time find / -name msfconsole
real 14m3.648s
user 0m17.783s
sys 2m29.870s

I actually stopped it at around 15 minutes because I couldn't wait that long. There are many factors in the performance equation of the above command, such as the overall speed of the system, how busy the system is when you execute the command, and some even say that find is slower if its checking across different file system types ("/" would also include mounted USB drives). A quicker way to find files is to use the locate command:

$ locate msfconsole | grep -v .svn
/Users/fanboy/metasploit/framework-2.7/msfconsole
/Users/fanboy/metasploit/framework-3.1/msfconsole
/Users/fanboy/metasploit/framework-3.2-release/msfconsole

This command reads from a database (which is generated on a regular basis) that consists of a listing of files on the system. It's MUCH faster:

$ time locate msfconsole
real 0m1.205s
user 0m0.298s
sys 0m0.050s

I'm wondering what Ed's going to do on Windows, unless he's come up with a way to get an animated ASCII search companion dog. :)

Hal Says:

One thing I will note about the locate command is that it's going to do sub-expression matching, whereas "find ... -name ..." will do an exact match against the file name. To see the difference, check out the following two commands:

# find / -name vmware
/etc/vmware
/etc/init.d/vmware
/usr/lib/vmware
/usr/lib/vmware/bin/vmware
/usr/bin/vmware
/var/lib/vmware
/var/lock/subsys/vmware
/var/log/vmware
# locate vmware
/etc/vmware
/etc/init.d/vmware
/etc/pam.d/vmware-authd
/etc/rc2.d/K08vmware
/etc/rc2.d/S19vmware
[... 5000+ addtl lines of output not shown ...]

Also, as Paul notes above, the database used by the locate command is updated regularly via cron. The program that builds the database is updatedb, and you can run this by hand if you want to index and search your current file system image, not the image from last night.

I was curious whether doing a find from the root was faster than running updatedb followed by locate. Note that before running the timing tests below, I did a "find / -name vmware" to force everything into the file cache on my machine. Then I ran:

# time find / -name vmware >/dev/null
real 0m1.223s
user 0m0.512s
sys 0m0.684s

# time updatedb
real 0m0.263s
user 0m0.128s
sys 0m0.132s

# time locate vmware >/dev/null
real 0m0.314s
user 0m0.292s
sys 0m0.016s

It's interesting to me that updatedb+locate is twice as fast as doing the find. I guess this shouldn't really be that surprising, since find is going to end up calling stat(2) on every file whereas updatedb just has to collect file names.

Ed Kicks in Some Windows Stuff:

In Windows, the dir command is often used to search for files with a given name. There are a variety of ways to do this. One of the most obvious but less efficient ways to do this involves running dir recursively (/s) scraping through its results with the find or findstr command to look for what we want. I'll use the findstr command here, because it gives us more extensibility if we want to match on regex:

C:\> dir /b /s c:\ | findstr /i vmware

There are a couple of things here that may not be intuitive. First off, what's with the /b? This indicates that we want the bare form of output, which will omit the extra stuff dir adds to a directory listing, including the volume name, number of files in a directory, free bytes, etc. But, when used with the /s option to recurse subdirectories, /b takes on an additional meaning. It tells dir to show full paths to files, which is what we really want to see to know the file's location. Try running the command without /b, and you'll see that it doesn't show what we want. The /b makes it show what we want: the full path to the file so we know its location. Oh, and the /i makes findstr case insensitive.

But, you know, dumping all of the directory and file names on standard out and then scraping through them with findstr is incredibly inefficient. There is a better way, more analogous to the "find / -name" feature Paul and Hal use above:

C:\> dir /b /s c:\*vmware*

This command seems to imply that it will simply look inside of the c:\ directory itself for vmware, doesn't it? But, it will actually recurse that directory looking for matching names because of the /s. And, when it finds one, it will then display its full path because of the /b. I put *vmware* here to make this look for any file that has the string vmware in its name so that its functionality matches what we had earlier. If you omit the *'s, you'll only see files and directories whose name exactly matches vmware. This approach is significantly faster than piping things through the findstr command. Also note that it is automatically case insensitive, because, well, that's the way that dir rolls.

How much faster? I'm going to use Cygwin so I can get the time command for comparison. The $ prompt you see below is from Cygwin running on my XP box:

$ time cmd.exe /c "dir /s /b C:\ | findstr /i vmware > nul"

real 0m10.672s
user 0m0.015s
sys 0m0.015s

Now, let's try the other approach:

$ time cmd.exe /c "dir /s /b C:\*vmware* > nul"

real 0m6.484s
user 0m0.015s
sys 0m0.031s

It takes about half the time doing it this more efficient way. Oh, and note how I'm using the Cygwin time command here. I use time to invoke a cmd.exe with the /c option, which will make cmd.exe run a command for me and then go away when the command is done. Cygwin's time command will then show me how long the command took. I use time to invoke a cmd.exe /c rather than directly invoking a dir so that I can rely on the dir command built-into cmd.exe instead of running the dir command included in Cygwin.

OK... so we have a more efficient way of finding files than simply scraping through standard output of dir. But, what about an analogous construct to the locate command that Hal and Paul talk about above? Well, Windows 2000 and later include the the Indexing Service, designed to make searching for files more efficient by creating an index. You can invoke this service at the command line by running:

C:\> sc start cisvc

Windows will then dutifully index your hard drive, making searches faster. What kind of searches? Well, let's see what it does for our searches using dir:

$ time cmd.exe /c "dir /s /b C:\*vmware* > nul"

real 0m6.312s
user 0m0.015s
sys 0m0.046s

Uh-oh... The Windows indexing service doesn't help the dir command, whether used this way or in combination with the find command. Sorry, but dir doesn't consult the index, and instead just looks through the complete file system directory every time. But, the indexing service does improve the performance of the Start-->Search GUI based search. You can control which directories are included in the index via a GUI tool that can be accessed by running:

C:\> ciadv.msc

Also, in that GUI, if you select System-->Query the Catalog, you get a nice GUI form for entering a query that relies on the indexing service. I haven't found a built-in cmd.exe feature for searching directories faster using the indexing service, but there is an API for writing your own tools in VBS or other languages for quering the index. Microsoft describes that API and the indexing service in more detail here.

Episode #20: Ping Beep of Death

Ed says:

Paul threw down a challenge to Hal and me in e-mail just a few minutes ago:

"I want a string of commands that will ping a host and for each time a packet is missing (i.e. timeout) send a beep to the console."

The solution to this one in Windows includes some useful constructs, so let's have at it with this fu:

C:\> for /L %i in (1,0,2) do @(ping -n 1 HostIPaddr || echo ^G)
& ping -n 2 127.0.0.1

This command starts out with a FOR /L loop, set with an iterator variable of %i that I won't use, counting from 1 to 2 in steps of 0. In other words, it'll run forever, or until someone kills it. At each iteration through the loop, I turn off command echo (@), and ping the target host one time (ping -n 1). Pretty pedestrian so far.

Now we get to some interesting stuff. If the ping command fails to get a response, I'll have it run the echo command to make a beep. As we've seen in earlier episodes, we can use cmd1 && cmd2 to make the shell run cmd2 only if cmd1 succeeded. Likewise, we can use cmd1 || cmd2 to make the system run cmd2 only if cmd1 fails. This is more efficient than checking the %errorlevel% environment variable with an IF statement, but I digress. Of course, as always cmd1 & cmd2 means run cmd2 regardless of the success or failure of cmd1.

If the ping fails, we have "echo ^G". Note that it looks like I typed Shift-6 G, but I didn't. That little ^G is created by holding down the CTRL key and hitting G. It's just displayed like ^G. It's how we can make the echo command ring the system beep. After this, I simply introduce a 1-second delay by pinging localhost twice (first ping instantly, second ping one second later). So, there you have it.

BTW, if you want prettier output from this, you can dump various Standard Output to nul as follows:

C:\> for /L %i in (1,0,2) do @(ping -n 1 HostIPaddr > nul || echo ^G)
& ping -n 2 127.0.0.1 > nul

Hal Chimes In:

We can easily do the Unix equivalent of what Ed's doing in the Windows command shell:

$ while :; do ping -c 1 -w 1 HostIPaddr >/dev/null || echo -e \\a; sleep 2; done

Note the "echo -e \\a" syntax for emitting an "alert" (^G or BEL).

However, in its default mode the standard Unix ping command will run continuously. So it seems like you should be able to get some joy just by piping the output of ping into another command, without having to resort to a loop. Mr. Bucket had the clever idea of filtering the output of ping through awk and then sending the resulting misses to the "say" command on his OS X machine (because talking is much cooler than beeping):

$ ping x.x.x.x 2>&1 | awk -F: '/sendto:/ {print $3}' | say

The above command seems like it should work, but it wasn't bringing the noise. A confused Mr. Bucket emailed me for clarification.

What's happening here is that Mr. Bucket's clever idea is getting kneecapped by the standard buffering behavior of shell pipes. If you run that first ping command by itself and have the output go to your terminal, then you see the results of each ICMP packet, one line at a time. However, the minute you pipe the output of the ping command into another command, the shell switches to page buffering the output between the two commands. In other words, the awk command gets the output of the ping command in 4096 byte chunks, and the output of awk has to fill up 4096 bytes before the "say" command gets anything. If Mr. Bucket had let the command run long enough, eventually he would have gotten a burst of talking from the "say" command, but only long after the actual failed ping events.

Unfortunately, there is no magic shell variable you can set to fix this problem globally. However, applications in the Unix environment can force their output to be unbuffered if the application developer chooses to add this functionality to their program. Sometimes the software does this automatically-- the tee program in most Unix environments unbuffers its output by default (using setvbuf(3)). Sometimes you need to add an extra command-line option, like with the awk command on my Ubuntu system (which is really mawk):

$ ping x.x.x.x 2>&1 | awk -W interactive -F: '/sendto:/ {print $3}' | ...

That'll get you line-buffered output that you can pipe into another program. The GNU grep utility has the "--line-buffered" option, which is similar.

Paul Chimes In:

I have to confess, I really thought this would be something pretty easy. The page buffering thing threw me through a loop too (pun intended). I also discovered a much better way to do this in OS X:

$ ping -A 192.168.1.1

This command will beep (ASCII 0x07) "when no packet is received before the next packet is transmitted." so says the man pages. This is exactly what I was looking for to monitor hosts while I scan them with various security tools (like Nessus) and alert me if they crash or become unresponsive.

Episode #19 - Clearing The Contents Of A File

Paul Writes In:

Ever want to delete just the contents of a file, but not the file itself? There are many ways to do this on UNIX/Linux systems, here's one:

$ cat /dev/null > my_file

Probably the most common usage for this is to clean out a log file. I'm certain Hal will have several suggestions, and Ed will have to (again!) be creative without /dev/null in Windows :)

Hal Says:

I generally "set -o noclobber" in my .bashrc because I don't want to accidentally overwrite files with output redirection (and you should too, Paul!). Thus, Paul's command line above is not as useful for me. My idiom is usually just:

$ cp /dev/null my_file

It's also two characters (not counting extraneous spaces) shorter than Paul's version. Go me!

Ed Responds:

While we don't have /dev/null in Windows, we do have the very handy nul, the rough equivalent of /dev/null. I use it rather often to make things go away, usually directing unwanted Standard Out to it. But, to delete the contents of a file, you could use nul like this:

C:\> type nul > my_file

This will keep the file's creation time (viewable with "dir /tc my_file", as we discussed in Episode 11), while making the contents of the file empty.

Alternatively, we can remove the contents of a file in Windows using Hal's approach of copying with:

C:\> copy nul my_file

Oh, and my copy command is four less characters less to type than Hal's fu above, a whopping six characters less than Paul's. Efficiency, thy name is Windows. Go Windows! Now there's something that you don't hear often in a battle of the command shells!

BTW, thanks for the "set -o noclobber" tip, Hal. Very useful! Almost always in my SANS 560 class, attendees accidentally hose their systems by flipping a <> and destroying /etc/passwd or /etc/shadow on their Linux boxen, even though I warn them about it. I'll set noclobber by default for their shells, which should prevent that from happening, but will introduce frustrations when their bad command doesn't work. I guess the frustration is better than hosing the system.

And, the hits just keep on coming...

Episode #18 - Clearing The System DNS Lookup Cache

Paul Says:

In an effort to provide some quick command like tips (we lovingly call them "quickies") I wanted to share the OS X command to clear the local client's lookup cache:

# dscacheutil -flushcache

This is especially useful if you are doing some DNS changes and don't want the local client's cache to obscure your results.

Hal Says:

Repeat after me: "Client side caches are a lousy idea that can only lead to confusion and suffering. If your naming services are so unreliable that you can't live without a client cache, then it's time to get a new naming service."

The Name Services Caching Daemon ("nscd") is the client-side cache for Solaris and Linux systems. Generally I just turn it off on all of my machines, but if you leave it running you can flush the cache with a command like:

# nscd -i hosts

"passwd" and "group" are also valid arguments instead of "hosts", but it's the local DNS cache that generally gets you into the most trouble.

Ed Cleans Up:

Per Hal's request to repeat after him: "Client side caches are a lousy idea that can only lead to confusion and suffering. If your naming services are so unreliable that you can't live without a client cache, then it's time to get a new naming service."

So, if it's got confusion and suffering, you better believe its an integral component of the architecture of Windows. :)

In all seriousness, while Hal might not like it, I find the DNS cache in Windows hugely useful when conducting investigations, so I can see what has been recently resolved on the box. You can dump the DNS cache of a Windows machine with:

C:\> ipconfig /displaydns

The output will include all cached records, showing their remaining TTL in seconds. Run it every one second, and you can see the times decrement. If you happen to know the original TTL, you can get a feel for how long the record has been in the cache (realize, however, that the original TTL may not have been the value set by the authoritative server, because the machine may have gotten it from some intermediate server where it was cached for a while). Anyway, that's some useful information in an investigation, so you can piece together a timeline of when certain actions were taken.

But, the challenge at hand deals with clearing the DNS cache, which we can do with:

C:\> ipconfig /flushdns

My buddy Lara Corcoran refers to some things as "easy peasy". When I first heard her use this phrase, I found it annoying. But, it's addictive, and saying it reminds me of Lara, which always makes me smile.

Hal Puts in the Last Word:

Actually, Ed, instead of using the local DNS cache as a proxy to figure out what hosts the local machine has been communicating with recently, in Linux we can actually dump out the kernel's internal routing cache:

$ netstat -rCn
Kernel IP routing cache
Source          Destination     Gateway         Flags   MSS Window  irtt Iface
68.238.96.36    67.18.149.10    67.18.149.10    l         0 0          0 lo
212.118.133.101 67.18.149.10    67.18.149.10    l         0 0          0 lo
72.95.246.206   67.18.149.10    67.18.149.10    l         0 0          0 lo
69.158.249.37   67.18.149.10    67.18.149.10    l         0 0          0 lo
67.18.149.10    68.105.28.76    67.18.149.9            1500 0          0 eth0
67.18.149.10    200.149.55.136  67.18.149.9            1500 0          0 eth0
67.18.149.10    93.190.136.10   67.18.149.9            1500 0          0 eth0
67.18.149.10    62.253.181.25   67.18.149.9            1500 0          0 eth0
[...]

What you see are a bunch of host routes corresponding to the machines that this host has been communicating with recently. This is incredibly useful information in a forensic investigation.

Paul Chiming In One More Time:

To view the DNS cache table in OS X, you can use the following command:

# dscacheutil -cachedump -entries Host

Which will output something like:

DirectoryService Cache Overview:
  AAAA Queries  - Enabled
  Buckets Used  - 40
  Cache Size    - 10

  Entry count by category:
      Host  - 8
      User  - 2

Cache entries (ordered as stored in the cache):

    Category         Best Before         Last Access      Hits    Refs       TTL    Neg  DS Node
  ----------  ------------------  ------------------  --------  ------  --------  -----  ---------
        Host   03/30/09 16:39:19   03/30/09 16:39:12         3       4         9       
                   Key: h_name:twitter.com ipv4:1 ipv6:1
                   Key: h_name:twitter.com ipv6:1
                   Key: h_name:twitter.com ipv4:1

This allows you to see which hosts are in the cache when the command was run. I could not find a way to do the equivalent of a "netstat -C" in OS X.

We Interrupt This Blog...

Hal Comments:

Somehow this seems like an appropriate forum to make an announcement that I've been in "stealth mode" about for some time now. But apparently details have leaked to the press and we're expecting some wild stories to be posted later today, and so I thought I'd blog about this personally, just to reduce the amount of misinformation that's going to be put out there.

For almost a year now, I've been working for Microsoft, helping to architect the next generation Windows operating system (the one that's coming after Windows 7). No one was more surprised than I that I accepted this position, but when I heard about the plans for the new operating system, I knew it was an offer I just couldn't refuse.

You see, Microsoft is only too keenly aware of the environmental impact caused by users having to buy newer, faster, more power-hungry machines just to handle the increasing computing requirements of each new release of Windows (to say nothing of the impact of all of the old, discarded equipment being introduced into landfills around the planet). So as part of a company-wide initiative to be more "green", the decision was made to remove the Windows GUI and instead go with a command-line only interface in the next release of Windows (codenamed WinDOS, btw). Here's a snapshot of Internet Explorer 8 in command-line mode:


This is a very exciting time for us here at Microsoft! Bill Gates was telling the team just the other day how happy he was that Microsoft was "returning to his original vision of MS-DOS".

Of course, I haven't completely abandoned my Unix roots. In fact, my primary area of responsibility in the WinDOS project is re-implementing some of the more useful features of the Unix command shell, pursuant to Microsoft's well-documented "embrace and extend" policy. Most recently, I've been working on an upgraded version of the popular "rear" command. Never heard of "rear"? Just type "man rear" into any Unix/Linux command shell you happen to have handy.

Working with the brilliant minds at Microsoft, helping to create a convergence between Windows and Unix, and saving the planet at the same time? Who wouldn't love this job? Wish me luck, folks!

Ed Responds:
Wow! Hal... this is huge. I remember your stint at Google a couple years back. You seem to have a thing for the siren song of large companies bent on world domination. It must somehow fulfill an obscure longing you have. Just out of curiosity, were you bottle fed as an infant?

Anyway, given that Microsoft hired both Crispin Cowan and Bill Arbaugh last year, I guess this makes it a trifecta for them. Go Microsoft!

So, my friend, here's to you and Microsoft. I wish you the best in your new digs, and I am very much looking forward to seeing your influence permeate the future evolution of Windows!

Paul Responds:

Congrats to Hal and Microsoft! Finally they have seen the light and hired the best when it comes to 1980's style user interfaces (which are, btw, the interfaces of the future). This is a huge win for the community, I can't wait to install the new version of WinDOS on my old 486 laptop!