Episode #47: Fun with Output Redirection and Errorlevels

Ed begins thusly:

We really love it when readers write in with suggestions and questions for our hearty band of shell aficionados. Diligent reader Curt Shaffer writes in with a suggestion:

I have been following this blog for some time. I learn more and more every day and not just for pen testing but system administration as well. For that I thank everyone involved! It has come to a point where I just want to script and do everything from the command line on Windows. I have always opted for command line on Linux. My question/suggestion is to do an episode detailing how we might get output into a log file on commands we run. I know now that 2>c:\log.txt or something similar will provide standard error but what I would like to see is where it possibly errored out. Here is an example:

I want to script adding a group to local administrators for all servers on the network. I get them all into a text file called Servers.txt. I then do the following:

C:\> for /f %a in (C:\Servers.txt) do psexec \\%a net localgroup
    "administrators" "domain\server-admin-gg" /add

Now that should work just fine I believe. The problem is because I need to be sure this happened, I would like to capture the data of failures. So I know I can add a 2>c:\error.txt. What I would like though is a central log file that would give me something along the lines of %a and the error. (i.e. Server 1 – Access Denied, Server 2 – OK, Server 3—OK, Server 4 –Access Denied etc) or something to that effect so I can know which were not successful so I can go back and make sure they get it.

I have to say, Curt, I like the cut of your jib. This is a good question, and its answer will open up areas we haven't yet talked much about in this blog -- output redirection, file descriptor duplication, and ways to deal with and record errorlevels. Thank you for the great suggestion!

First off, it should be noted that you don't have to wrap your psexec in a FOR /F loop to run it against multiple machines. The psexec command (not built-into Windows, but downloadable from Microsoft Sysinternals) has an @filename option, which causes it to run the given command on each remote system listed in filename. I'm going to work with your provided command with the FOR /F loop as is, though, because I think it'll keep our answer here more generally applicable to uses other than just psexec.

Now, as you point out, we can capture standard error information via the file descriptor 2, and send it to a file with a simple redirection (2>c:\error.txt). To capture both standard error and standard output we could append the following to our command:

[command] > results.txt 2>&1

The first part of this is taking the standard output of our command and dropping it into the results.txt file (> results.txt). This is actually a shortened form of explicitly referring to our standard out using the file descriptor 1. We could write it as 1>results.txt if we wanted to be a little more explicit. Now, we want to dump standard error into the same place that standard output is going. If we simply did "2>1", we'd have a problem, because standard output (1) is already busy dumping stuff into results.txt. The syntax "2>&1" tells the shell that we want standard error to be connected to a duplicate of standard output, which itself is already connected to the results.txt file. So, we get both standard output and standard error in the results.txt file.

We've got to get the order right here. If you transpose these redirects with "[command] 2>&1 > results.txt", you wouldn't capture standard error, because your standard error will be dumped to standard output (the screen by default) before you redirect it to the results.txt file. In other words, you will see standard error on the screen, and your results.txt will only get standard output.

Now, I'm sure my shell sparring buddies will point out that their blessed little bash shell supports a nice shortened construct to make this a little easier to type. In bash, you can replace all of this with a simple &>results.txt. Let me preemptively point out that cmd.exe doesn't support that. Sorry, but cmd.exe, but its very nature, makes us work harder.

OK, so with that in hand, and realizing that we can append with >>, we can tweak Curt's command a bit to achieve his desired results:

C:\> for /f %a in (C:\Servers.txt) do @psexec \\%a net localgroup "administrators"
    "domain\server-admin-gg" /add >> c:\results.txt 2>&1

Curt also asked how he could get a copy of %a in his output. Well, we could do that using an "echo %a", adding a few dash marks to separate our output:

C:\> for /f %a in (C:\Servers.txt) do @echo %a---------- >> c:\results.txt & psexec
    \\%a net localgroup "administrators" "domain\server-admin-gg" /add >>
    c:\results.txt 2>&1

Now, that'll work, Curt, but your output will contain everything that psexec displays. That's a bag chock full of ugly. Can we make it prettier, and explore a little more flexibility of cmd.exe and error messages? You bet we can, using our good friends && and ||.

If a command succeeds, it should set the %errorlevel% environment variable to 0. Otherwise, it sets it to some other value. Please note that not all Windows commands properly set this errorlevel! You should experiment with a command first to make sure the errorlevel is set as you wish by running your command followed by "& echo %errorlevel%". Run your command in a successful fashion and also in a way that makes it fail, and verify that %errorlevel% functions as you desire -- with 0 for success and a non-zero value for failure.

The psexec command does set the errorlevel to 0 if the command it runs on the target machine succeeds, provided that we don't use the -d option. Invoked with -d, psexec runs a command on the target machine in detached mode (running it without access to its standard input, output, and error). Since 2005, psexec with the -d switch will return the processid that it created on the target. Anyway, here, we're not using -d, so we should be cool with the %errorlevel% value. But, consider yourself warned about psexec, -d, %errorlevel%, and processids.

We could use an "IF %errorlevel% equ 0" statement to put in some logic about what to store for a result based on this %errorlevel% value, but that's a lot to type into a shell. I'd do it in a heartbeat for a script, but let's keep this focused on more human-typable commands. Instead of IF errorlevel stuff, we can use the shorthand [command1] && [command2] || [command3]. If command1 succeeds, command2 will run. If command1 has an error, command3 will execute.

The result gives us pretty much complete control of the output of our command:

C:\> for /f %a in (C:\Servers.txt) do @echo %a---------- >> c:\results.txt & psexec
    \\%a net localgroup "administrators" "domain\server-admin-gg" /add >nul &&
    echo Success >> c:\results.txt || echo Failure >> c:\results.txt

C:\> type c:\results.txt
10.1.1.1----------
Failure
10.1.1.4----------
Success
10.1.1.12----------
Success
10.1.1.100----------
Failure

But, you know, somehow it just feels wrong to throw away all of those details in nul. The psexec command goes to all the trouble of creating them... we can at least store them somewhere for later inspection. How about we create two files, one with a summary of Success or Failure in results.txt and the other with all of our standard output and standard error in details.txt? We can do that by simply combining the above commands to create:

C:\> for /f %a in (C:\Servers.txt) do @echo %a---------- >> c:\results.txt & psexec
    \\%a net localgroup "administrators" "domain\server-admin-gg" /add >>
    c:\details.txt 2>&1 && echo Success >> c:\results.txt || echo Failure >>
    c:\results.txt

The results.txt has a nice summary of what happened, and all of the details are in details.txt.

Fun, fun, fun!

Verily Hal doth proclaim:

It's fairly obvious that the Windows command shell has... "borrowed" a certain number of ideas from the Unix command shells. Nowhere is it more obvious than with output redirection. The syntax that Ed is demonstrating above for Windows is 100% the same for Unix. For example, you see this construction all the time in cron jobs and scripts where you don't care about the output of a command:

/path/to/some/command >/dev/null 2>&1

You know, Ed. I wasn't going to rag you at all about Windows lacking the "&>" syntax-- I personally prefer ">... 2>&1" (even though it's longer to type) because I think it documents what you're doing more clearly for people who have to maintain your code. Still, if I had to spend my life working with that sorry excuse for a shell, I guess I'd be a little defensive too.

Anyway, we can also emulate Ed's final example as well. In fact, it's a little disturbing how similar the syntax ends up looking:

for a in $(< servers.txt); do
   echo -n "$a " >>results.txt
   ssh $a usermod -a -G adm,root,wheel hal >>details.txt 2>&1 \
       && echo Success >>results.txt || echo Failure >>results.txt
done

Of course the command I'm running remotely is different, because there's no real Unix equivalent of what Curt is trying to do, but you can see how closely the output redirections match Ed's Windows command fu. To make the output nicer, Unix has "echo -n", so I can make my "Success/Failure" output end up on the same line has the host name or IP address. Neener, neener, Skodo.

There are all kinds of crazy things you can do with output redirection in the Unix shell. Here's a cute little idiom for swapping standard input and standard error:

/path/to/some/command 3>&1 1>&2 2>&3

Here we're creating a new file descriptor numbered 3 to duplicate the standard output-- normally file descriptor 1. Then we duplicate the standard error (descriptor 2) on file descriptor 1. Finally we duplicate the original standard output handle that we "stored" in our new file descriptor 3 and associate that with file descriptor 2 to complete the swap. This idiom can be useful if there's something in the error stream of the command you want to filter on. By the way, just to give credit where it's due, I was reminded of this hack from a posting on the CommandLineFu blog by user "svg" that I browsed recently.

But here's perhaps the coolest output redirection implemented in bash:

ps -ef >/dev/tcp/host.example.com/9000

Yes, that's right: you can redirect command output over the network to another machine at a specific port number. And, yes, you can use the same syntax with /dev/udp as well. It's like "netcat without netcat"... hmmm, where have I heard that before? Oh, that's right! That was the title of Ed's talk where he showed me this little tidbit of awesomeness.

Some notes about the "... >/dev/tcp/..." redirection are in order. This syntax is a property of the bash shell, not the /dev/tcp device. So you can't use this syntax in ksh, zsh, and so on. Furthermore, some OS distributions have elected to disable this functionality in the bash shell-- notably the Debian Linux maintainers (which means it also doesn't work under Ubuntu and other Debian-derived distributions). Still, the syntax is portable across a large number of Unix and Linux variants-- I use it on Solaris from time to time, for example.

Paul Chimes In:

I have to say, I'm a bit dizzy after trying to figure out a way to add a users and groups via the command line in OS X Leopard. In fact, this is where I got stuck trying adapt the Windows and UNIX/Linux concepts above to OS X. User/groups creation in OS X is very different from Windows and UNIX/Linux, and even varies depending on which version of OS X you are running. So far I've go this in order to just create a user in the latest version of Leopard:

dscl . -create /Users/testuser
dscl . -create /Users/testuser UserShell /bin/bash
dscl . -create /Users/testuser RealName "Test User"
dscl . -create /Users/testuser UniqueID 505 
dscl . -create /Users/testuser PrimaryGroupID 80 
dscl . -create /Users/testuser NFSHomeDirectory /Users/testuser 
dscl . -passwd /Users/testuser supersecretpassword
dscl . -append /Groups/admin GroupMembership testuser

The article How to: Add a user from the OS X command line, works with Leopard! was extremely useful and documented several methods. Also, there is a really neat shell script published called "Create & delete user accounts from the command line on Mac OS X". One important item to note, most of the documentation either says to reboot or logout and back in again in order to see the new user (yuk). So, I hope the above examples and resources provide you with enough information to adapt the techniques covered in this post for OS X administration, and I plan to cover more on this in upcoming episodes.