Tuesday, December 29, 2009

Episode #75: Yule Be Wanting an Explanation Then

Hal returns to the scene of the crime

I opened last week's post saying there would be no "explanations or excuses", but apparently that wasn't good enough for some of you. So at the request of our loyal readers, we'd like to revisit last week's episode and explain some of the code. Especially that crazy cmd.exe stuff that Ed was throwing around.

Of course the bash code is completely straightforward:

$ ct=12; while read line; do
[ $ct == 1 ] && echo -n Plus || echo -n $ct;
echo " $line";
((ct--));
done <<EoLines
keyboards drumming
... lines removed ...
command line hist-or-y!
EoLines

First we're initializing the "ct" variable we'll be using to count down the 12 Days of Fu. Then we start a while loop to read lines from the standard input.

Within the body of the loop, we use the quick and dirty "[...] && ... || ..." idiom that I've used in previous Episodes as a shortcut instead of a full-blown "if ... then ... else ..." clause. Basically, if we've counted down to one then we want to output the word "Plus"; otherwise we just output the value of $ct. Notice we use "echo -n ..." so that we don't output a newline. This allows us to output the text we've read from the standard input as the remainder of the line. Finally, we decrement $ct and continue reading lines from stdin.

The interesting stuff happens after the loop is over. Yeah, I could have put the text into a file and read the file. But I was looking for a cool command-line way of entering the text. The "<<EoLines" syntax at the end of the loop starts what's called a "here document". Basically, I'm saying that the text I type in on the following lines should be associated with the standard input (of the while loop in this case). The input ends when the string "EoLines" appears by itself on a line. So all I have to do is type in the 12 lines of text for our 12 Days of Fu and then finish it off with "EoLines". After that we get our output. Neat!

Everybody clear? Cool. I now throw it over to Tim to get the lowdown on his PowerShell madness.

Tim goes back in time

Let's unwrap what we did last week.

PS C:\> $ct=12; "keyboards drumming
admins smiling
... lines removed ...
command line hist-or-y!".split("`n") |
% { if ($ct -eq 1) {"Plus $_"} else {"$ct $_"}; $ct-- }


Here is the break down:

First, we initialize the $ct variable to begin our count down.

ct=12;


That was easy. Next, we take a multi-line string and split it in to an array using the new line character (`n) as the delimiter.

"keyboards drumming
... lines removed ...
command line hist-or-y!".split("`n")


We could have just as easily read in content from a file using Get-Content, but I wanted to demonstrate some new Fu that was similar to Hal's Fu. The nice thing is that reading from a file would be an easy drop-in replacement.

Once we have an array of Fu Text, we pipe it into ForEach-Object so we can work with each line individually.

if ($ct -eq 1) {"Plus $_"} else {"$ct $_"}


Inside the ForEach-Object loop we use an IF statement to format the output. If the count is one, then we output "Plus" and the Fu Text, otherwise we output the value of $ct and the Fu Text.

Finally, we decrement the value of $ct.

$ct--


Simple right? That was pretty straightforward. But sit back, and grab some spiked Egg Nog before you proceed further.

Ed Surveys the Madcap Mayhem:
I really liked this episode because it required the use of a few techniques that we haven't yet highlighted in this blog before. Let's check them out, first reiterating that command:

c:\> copy con TempLines.txt > nul & cmd.exe /v:on /c "set ct=12& for /f
"delims=" %i in (TempLines.txt) do @(if not !ct!==1 (set prefix=!ct!) else (set prefix=
Plus)) & echo !prefix! %i & set /a ct=ct-1>nul" & del TempLines.txt
keyboards drumming
----snip----
command line hist-or-y!
^Z (i.e., hit CTRL-Z and then hit Enter)

OK... we start out with the copy command, which of course copies stuff from one file to another. But, we use a reserved name here for our source file: con. That'll pull information in from the console, line by line, dumping the results into a temporary file, very cleverly named TempLines.txt. Of course, there is the little matter of telling "copy con" when we're done entering input. While there are several ways to do that, the most efficient way to do so that has minimal impact on the contents of the file is to hit CTRL-Z and then Enter. Voila... we've got a file with our input. By the way, I throw away the output of "copy con" with a "> nul" because I didn't want the ugly "1 file(s) copied." message to spoil my output. By the way, it kinda stinks that that message is put on Standard Output, doesn't it? Where I come from, that is much more of a Standard Error thing. But, I'm sure we could get into a big philosophical debate about what should go on Std Out and what should go on Std Err. But, let's just cut the argument short and say that many Windows command line tools are all screwed up on this front, regardless of your philosophy. That's because there are no reasonable and consistent rules for directing stuff to Std Out versus Std Err in Windows command line output.

Anyway, back to the point. I then run "cmd.exe /v:on" so I can do delayed variable expansion. That'll let me use variables with values that can change as my command runs. Otherwise, cmd.exe will expand all variables to their values instantly when I hit Enter. I need to let these puppies float. I use the cmd.exe to execute a command, with the /c option, enclosing the command in double quotes.

So far, so good. But, now is where things get interesting.

To map to Hal and Tim's fu, I then set a variable called ct to a value of 12, just a simple little integer. But, what's with that & right after the 12? Well, if you leave it out, you'll see that the output will contain an extra space in "12 keyboards drumming"... it'll look like "12[space][space]keyboards drumming". Where does the extra space come from? Well, check this out:
c:\> cmd.exe /v:on /c "set var=foo & echo !var!bar"
foo bar

c:\> cmd.exe /v:on /c "set var=foo& echo !var!bar"
foobar

Do you see what's happening here? In the first command, the set command puts everything in the variable called var, starting right after the = and going up to the & (which separates the next command), and that includes the space! We have to omit that space by having [value] with the & right after it. For a more extreme example, consider this:
c:\> cmd.exe /v:on /c "set var=stuff      & echo !var!blah"
stuff blah

I typically like to include a space before and after my & command separators when wielding my fu, for ease of readability. However, sometimes that extra space has meaning, so it has to be taken out, especially when used with the set command to assign a string to a variable, like the ct variable in that big command above.

Wait a second... earlier I referred to ct as an integer, and now I'm calling it a string? What gives? Just hold on a second... I'll come back to that in just a bit.

We have to deal with our FOR loop next. I'm using a routine FOR /F loop to iterate through the contents of the TempLines.txt file. I'm specifying custom delimiters, though, to override the default space and tab delimiters that FOR /F loops like to use. With "delims=", I'm specifying a delimiter of... what exactly? The equals sign? No... that's actually part of the syntax of specifying a delimiter. To make an equals a delimiter, I'd have to use "delims==". So, what is the delimiter I'm setting here? Well, friends, it's nothing. Yeah. I'm turning off parsing, because I want the full line of my input to be set to my iterator variable. In the past, when I was but a young cmd.exe grasshopper, I would turn off such parsing by setting a delim of a crazy character I would never expect in my input file, such as a ^, with the syntax "delims=^". But, I now realize that the most effective way to use FOR /F loops is to simply let them use you. I turn off parsing by making a custom delimiter of the empty set. Do not try and bend the spoon... That's impossible. Instead try to realize the truth.... that there is no spoon.

Anyway, so where was I? Oh yea, with my delimiterless lines now being assigned one by one to my iterator variable of %i, I'm off and running. In the DO clause of my FOR loop, I turn off the display of commands (@) and jump right into an IF statement, to check the value of my ct variable. I expand ct into its value with a !ct!, because I'm using delayed variable expansion. Without delayed expansion, variables are referred to a %var%. My IF statement checks to see if !ct! is NOT equal (==) to 1. If it's not, I set another variable called prefix to the value of ct.

Then, I get to my ELSE clause. Although I use ELSE a lot in my work, I have to say that this is the first time I've had to use it in one of our challenges on this blog. The important thing to remember with ELSE clauses in single commands (not in batch scripts) is that you have to put everything in parentheses. So, if !ct! is equal to 1, my ELSE clause kicks in and sets the prefix variable to the string "Plus". That way, later on, I can simply print out prefix, which will contain the ct number for most of the days, but the word "Plus" for the last day.

And, here we back to that string/integer thing I alluded to above. The cmd.exe shell uses loosely typed variables. No, this is not a reference to how hard you hit the keys on your keyboard when typing. Instead, like many interpreted languages, variable types (such as strings and integers) are not hard and fast. In cmd.exe, they are evaluated in real time based on context, and they can even change in a single command. My ct variable behaves like an integer, for the most part. I can add to it, subtract from it, and store its value in another variable. But, when I defined it originally with the set command, if I had used "set ct=12 &...", it would have been a string with the trailing space until I used it in some math, and then that space would disappear. Also, the prefix variable is given the value of ct most of the time, which is just an integer. But, when ct is one, I give the prefix variable a string of "Plus". I'm an old C-language programmer, so this loose type enforcement kinda weirds me out. Still, it's quite flexible.

Then, I echo the prefix (!prefix!) and the line of text (%i). I then subtract one from my ct with the "set /a ct=ct-1", throwing the output of the set command away (>nul). Note that I want to show the prefix and the text on the same line, so I use a single echo command to display both variables on the same line. Most cmd.exe command-line tools actually put their output on standard out with a Carriage Return Line Feed (CRLF) right afterward. Thus, two echo commands, one for each variable, would have broken the prefix and the file content on separate lines, a no-no when trying to reproduce exactly the output of Hal and Tim. When formulating commands that need to display a lot of different items on a single line, I often chunk them into variables and then echo them exactly as I need them on a single line with a single echo statement.

Now, there is one interesting counter-example to the general rule that cmd.exe command-line tools insert a CRLF at the end of their output: the "set /a" command. It does its math, and then displays the output without any extraneous return, as in:
c:\> set /a 8-500 & echo hey
-492hey

I used that little fact in this fun command to spew Matrix-like gibberish on the screen from Episode #58:

C:\> cmd.exe /v:on /c "for /L %i in (1,0,2) do @set /a !random!"

When I first was working on this 12-days challenge, I was thinking about using set /a to display !ct! and then the line from the file. It would all be on the same line because of that special "no-CRLF" property of "set /a". But, I ran into the little problem of the "Plus" for the last line of input, so I instead introduced the prefix variable and played on the loose typing. There are dozens of other ways to do this, but I tried to focus on one that, believe it or not, I thought made the most sense and was simplest.

Oh, and to close things out, I delete the TempLines.txt file. Can't litter our file system with crap, now can we?

So, as you can see, there were a bunch of ideas we haven't used in this blog so far that popped out of cmd.exe in this innocuous-seeming challenge, including empty-set delims, an ELSE clause, weak type enforcement, and variable building for a single line of output. That's a lot of holiday cheer, and it makes me happy.

With that said, all of us at the Command Line Kung Fu blog would like to wish our readers a Happy and Prosperous New Year!

Tuesday, December 22, 2009

Episode #74: Yule Love It!

Hal has indulged in a bit too much holiday cheer:

Presented for your enjoyment with no explanation or excuses:

$ ct=12; while read line; do
[ $ct == 1 ] && echo -n Plus || echo -n $ct;
echo " $line";
((ct--));
done <<EoLines
keyboards drumming
admins smiling
systems thrashing
networks crashing
hosts a-pinging
Windows versions
(billion) Linux distros
Windows loops!
authors coding
shells hacked
types of hosts
command line hist-or-y!
EoLines

12 keyboards drumming
11 admins smiling
10 systems thrashing
9 networks crashing
8 hosts a-pinging
7 Windows versions
6 (billion) Linux distros
5 Windows loops!
4 authors coding
3 shells hacked
2 types of hosts
Plus command line hist-or-y!


Tim got run over by a reindeer:

PS C:\> $ct=12; "keyboards drumming
admins smiling
systems thrashing
networks crashing
hosts a-pinging
Windows versions
(billion) Linux distros
Windows loops!
authors coding
shells hacked
types of hosts
command line hist-or-y!".split("`n") |
% { if ($ct -eq 1) {"Plus $_"} else {"$ct $_"}; $ct-- }

12 keyboards drumming
11 admins smiling
10 systems thrashing
9 networks crashing
8 hosts a-pinging
7 Windows versions
6 (billion) Linux distros
5 Windows loops!
4 authors coding
3 shells hacked
2 types of hosts
Plus command line hist-or-y!

Ed's Nuts Roasting on an Open Fire:


c:\> copy con TempLines.txt > nul & cmd.exe /v:on /c "set ct=12& for /f
"delims=" %i in (TempLines.txt) do @(if not !ct!==1 (set prefix=!ct!) else (set prefix=
Plus)) & echo !prefix! %i & set /a ct=ct-1>nul" & del TempLines.txt
keyboards drumming
admins smiling
systems thrashing
networks crashing
hosts a-pinging
Windows versions
(billion) Linux distros
Windows loops!
authors coding
shells hacked
types of hosts
command line hist-or-y!
^Z (i.e., hit CTRL-Z and then hit Enter)
12 keyboards drumming
11 admins smiling
10 systems thrashing
9 networks crashing
8 hosts a-pinging
7 Windows versions
6 (billion) Linux distros
5 Windows loops!
4 authors coding
3 shells hacked
2 types of hosts
Plus command line hist-or-y!
Best wishes for a happy holiday season and a joyous and prosperous new year!

Tuesday, December 15, 2009

Episode #73: Getting the perfect Perm(s)

Tim unwraps:

One of the things I find myself doing on a regular basis is creating a new directory structure and setting the permissions. The permissions are different for each folder and are based on who in the organization needs access to it. We could just write a script to create the directories and the permissions, but let's say we want to copy permissions from one directory structure to another. For this example let's assume we have a project folder structure that looks something this.

Prjs
+-Project1 (Managers - Full Access, Consultants - Full Access)
|-Budget (Consultants - Deny, Finance - Full Access)
|-Data
+-Docs
|-ForRelease (AdminStaff - Full Access)
+-InProgress


Included above is the appropriate permissions on each folder. All permissions are inherited, so consultants and managers would have access to the Data directory.

We can verify these permissions by using Get-ChildItem (aliases gci, dir, ls) and piping the results into Get-Acl.

PS C:\> ls Prjs -recurse | Get-Acl | fl Path,AccessToString

Path : Microsoft.PowerShell.Core\FileSystem::C:\Prjs\Project1
AccessToString : WINXP\Consultants Allow FullControl
WINXP\Managers Allow FullControl

Path : Microsoft.PowerShell.Core\FileSystem::C:\Prjs\Project1\Budget
AccessToString : WINXP\Consultants Deny DeleteSubdirectoriesAndFiles, Modify,
ChangePermissions, TakeOwnership
WINXP\Consultants Allow FullControl
WINXP\Finance Allow FullControl
WINXP\Managers Allow FullControl

Path : Microsoft.PowerShell.Core\FileSystem::C:\Prjs\Project1\Data
AccessToString : WINXP\Consultants Allow FullControl
WINXP\Managers Allow FullControl

Path : Microsoft.PowerShell.Core\FileSystem::C:\Prjs\Project1\Docs
AccessToString : WINXP\Consultants Allow FullControl
WINXP\Managers Allow FullControl

Path : Microsoft.PowerShell.Core\FileSystem::C:\Prjs\Project1\Docs\ForRelease
AccessToString : WINXP\AdminStaff Allow FullControl
WINXP\Consultants Allow FullControl
WINXP\Managers Allow FullControl

Path : Microsoft.PowerShell.Core\FileSystem::C:\Prjs\Project1\Docs\Working
AccessToString : WINXP\Consultants Allow FullControl
WINXP\Managers Allow FullControl


So now we want to create a second project, Project2, and we want to make sure we have the same permissions. We could copy just the directories without files, but there may be more subdirectories further down that we don't want. So let's create the folders.

PS C:\> mkdir Prjs\Project2\Budget
PS C:\> mkdir Prjs\Project2\Data
PS C:\> mkdir Prjs\Project2\Docs\ForRelease
PS C:\> mkdir Prjs\Project2\Docs\Working


Note, when the Budget directory is created it also creates the Project2 directory since it doesn't exist.

What are the permissions on the new folder?

PS C:\Prjs> Get-Acl Project2 | fl Path,AccessToString

Path : Microsoft.PowerShell.Core\FileSystem::C:\Prjs\Project2
AccessToString : BUILTIN\Administrators Allow FullControl
NT AUTHORITY\SYSTEM Allow FullControl
WINXP\myuser Allow FullControl
CREATOR OWNER Allow 268435456
BUILTIN\Users Allow ReadAndExecute, Synchronize
BUILTIN\Users Allow AppendData
BUILTIN\Users Allow CreateFiles


Those are not the permissions we want. The permissions need to be copied from Project1 to Project2, but how? The Get-Acl and Set-Acl commands will do it.

PS C:\Prjs> Get-Acl Project1 | Set-Acl Project2


Let's verify.

PS C:\Prjs> Get-Acl Project2 | fl Path,AccessToString

Path : Microsoft.PowerShell.Core\FileSystem::C:\Prjs\Project2
AccessToString : BUILTIN\Administrators Allow FullControl
WINXP\Managers Allow FullControl
WINXP\Consultants Allow FullControl


Looks good. Now the subfolder permissions need to be copied as well.

PS C:\Prjs> ls Project1 -Recurse | Get-Acl |% {
Set-Acl $_ -Path ($_.Path -replace "Project1","Project2") }


First we do a recursive directory listing and get the Acl on each folder. We then take that Acl and apply it to a different folder. In our case all we need to do is replace Project1 for Project2 in the Path. Let's verify that the permissions match.

PS C:\Prjs> Compare-Object (ls Project1 -Recurse | Get-Acl)
(ls Project2 -Recurse | Get-Acl) -Property PSChildName, Access


No output, that's good, it means the permissions are identical. How did that work?

The Compare-Object cmdlet is used to find the differences between the collection of objects returned by these two commands:

ls Project1 -Recurse | Get-Acl
ls Project2 -Recurse | Get-Acl


The Property parameter specified in the original command allows us to select the properties to be checked for differences. PSChildName is the directory name and the Access property contains the permissions on the folder. We can't substitute the Path property for PSChildName since Path is the full path and it would always be different.

Copying permissions is pretty easy, I imagine it will be pretty easy for Hal since it isn't as granular. Finally, a bit of a leg up on Hal.

Hal just copies everything:

Do I detect a trace of jealousy and bitterness in my colleague's last comments? Better fix up that attitude Tim, or there will be nothing but coal in your stocking this year.

It's interesting that Tim brings up this subject, because it's another case where the differences in philosophy between Windows and Unix are apparent. In Windows, you need to fix up your directory permissions with an external tool after you copy the files. In Unix, it's just a natural part of the file copying operation-- particularly if you're doing the copy as the superuser.

This is also an area where we've seen some historical evolution in Unix-like operating systems. When I first got started with Unix in the 1980's, the "cp" command didn't have a "-p" option to preserve permissions, ownerships, and timestamps. The way you would copy directories when you wanted to preserve directory permissions was with the so-called "tar cp" idiom (actually, real old-timers will remember doing this with cpio):

# cd olddir
# tar cf - . | (cd /path/to/newdir; tar xfp -)

Here we're running the first tar command to create ("c") a new archive from the current working directory (".") and write it to the standard output ("f -"). We pipe that output to a subshell that first changes directories to our target dir and then runs another tar command to unpack the incoming archive on the standard input. The "p" option means preserve permissions, timestamps, and ownerships. Actually "p" is normally the default if you're running the tar command as root, so you can leave it off, but I prefer being explicit.

These days, however, there are a couple of simpler options. Obviously, you could just use "cp -p":

# cp -Rp olddir /path/to/newdir


I generally prefer rsync though:

# rsync -aH olddir /path/to/newdir


rsync not only allows you to copy directories within the same system, but also gives you the option of copying directories across the network. Also, if you just want to update the permissions on a directory, the rsync command will do that and not actually copy any file data that has previously been copied. For more information on rsync, see Episode #24.

One issue that Tim brought up was that sometimes you want to copy only part of a directory structure, but exclude certain files and/or subdirectories. This is another place where rsync beats cp. rsync has a couple of different ways of excluding content: the --exclude option for specifying patterns to exclude on the command line, and --exclude-from for reading in a list of patterns to exclude from a file. There's no way of excluding files built into the cp command at all. For those old fogies like me out there who still occasionally use "tar cp", the tar command typically has a switch like -X to exclude files and directories from the archive, and GNU tar has --exclude options very similar to rsync.

One thing you do need to be careful with for all of these copy options, however, is that they may not copy special permissions like extended attributes or file ACLs by default. Both cp and rsync have explicit options you can set to preserve these settings:

# cp -R --preserve=all olddir /path/to/newdir
# rsync -aHAX olddir /path/to/newdir

There's no way to do something similar with the "tar cp" idiom, because the tar archive format doesn't preserve extended attributes and ACLs.

Oh dear. Now it's Ed's turn. I hope Tim and I haven't spoiled his holiday cheer...

Ed Joyously Responds:

Ahhhh…. file permissions. They tend to be an absolute pain in the neck to deal with en masse in cmd.exe. Sure, we can use cacls or icacls to manipulate them on single files or directories just swell. But, synchronizing or applying changes to bunches of files using cacls or icacls is often dangerous and painful. When I first read Tim's challenge, I thought to myself, "This is gonna get ugly on us… as ugly as that feud between Snow Miser and Heat Miser." I immediately began to search my mind for a hook or trick to make this a lot easier, hoping to avoid a trip to visit Mother Nature.

Then, it hit me: we can use our little friend robocopy, the wonderfully named tool in Vista, Windows 7, and Windows 2008! Yeah, it's not built in to XP or Windows 2003, but it'll work for the latest version of Windows. We talked about robocopy in Episode #24.

To address Tim's challenge, I'm going to assume that the directory structure where we want to replicate our file permissions does not already exist, avoiding the mkdir commands Tim uses. Robocopy will make those for us, dutifully placing the proper access controls on them if we run:

C:\> robocopy [directory1] [directory2] /e /xf *

All of the subdirectories in directory1 will be created in directory2 with the same file permissions. The /e will make it recurse through those subdirectories, copying both directories with stuff in them and empty directories. The /xf means I want to exclude a certain set of files, which I've selected as *, meaning to exclude all files -- Only directories will be copied, including all of their luscious permissions.

Well, that's all fine and good, but what about Windows XP and 2003? Well, you can download and install robocopy on either of them, which is a pretty good idea. Alternatively, there is a way to trick Windows into applying the permissions from one directory structure to another, which applies to Windows 2003, Vista, 7, and 2008 Server. For this trick, we'll use the handy /save and /restore feature of icacls. Here, let's follow Tim's lead, and assume that we've got directory1 and directory2 already created, and we want to take the permissions from directory1 and its subdirectories and apply them to the already-existing directory2. Check out the following command:

C:\> icacls [directory1]\* /save aclfile /t /c

This command tells Windows to run icacls against directory1 and all of its contents (*), saving the results (/save) in a file called aclfile, recursing through the directory structure (/t), not stopping when it hits a problem (/c). Now, the resulting aclfile is not regular ASCII, but instead a unicode format that includes all of the permissions for the directories _and_ files inside of directory1.

Now, if there is a directory2 that already exists and has a similar overall structure to directory1, but perhaps without having any files in it, we can use icacls to restore the aclfile on a different directory! Wherever there is commonality in the directory structure, the permissions from directory1 will be used to overwrite the permissions on the given entity in directory2. The command to use is:

C:\> icacls [directory2] /restore aclfile /t /c

Voila! We've restored the ACLs from directory1 onto directory2! Now, that is a delicious holiday treat.

But, that leaves out poor little Windows XP, an operating system without robocopy and icacls built in. Sad, sad, sad little XP. Looks like it gets a lump of coal in its stocking this year, not only from Santa-Ed, but also from Microsoft, which has announced its impending withdrawal of support of this very near and dear friend.

Tuesday, December 8, 2009

Episode #72: That Special Time of Year

Tim plays Santa:

A merry listener in the PaulDotCom IRC channel asked:
[Dear Santa]...is there a way to delete certain characters in a for loop from cmd.exe (such as nul, tab, etc)?

Santa slightly nods and exclaims, "Now, Dasher! Now, Dancer! Now, Prancer, and Vixen! On, Cmd! On, For Loop! On, Donner and PowerShell! To the top of the terminal! to the top of the screen! Now bash away! bash away! bash away all!"

Anyway, enough of that crazy old guy.

The question was how to do it from the standard Windows command line, but indulge me for a minute and let's see what PowerShell can do.

Santa has a text file containing the data below, the bits in brackets represent special characters that we want to remove.
Grandma[space]Got[tab]Runover[*]By[']A["]Reindeer[0x06]Last[nul]Night.


So the goal is to delete these special characters. The question is, "How does PowerShell handle special characters?" The answer is the backtick character (by the Esc and 1 keys), and here is a list of special characters and the associated escape sequence:


`n New line
`r Carriage return
`t Tabulator
`a Alarm
`b Backspace
`' Single quotation mark
`" Double quotation mark
`0 Null
`` Backtick character


These can be used in our regular expression to remove the special characters from our file. Here is how we do it.

PS C:\> Get-Content test.bin | % { $_ -replace " |`t|\*|`'|`"| |\0x06|`0", "" }
GrandmaGotRunoverByAReindeerLastNight.


Inside our ForEach-Object (alias %) we use the replace operator to find all of our special characters and replace them with nothing (a.k.a. deleted). For those of you not familiar with regular expressions, the pipe character (|) is a logical OR, so any/all the characters will be replaced with an underscore. To represent the ASCII ACK character (0x06) in the regular expression we use \xNN, where NN is the hexadecimal ASCII code.

We removed the special characters from the text that was read from the file, but we didn't actually change the file. Here is how we do that:

PS C:\> (Get-Content test.bin) | % { $_ -replace " |`t|\*|`'|`"| |\0x06|`0",""} | Set-Content test.bin


There is one very importantly subtlety that can be very easily overlooked. Notice the parentheses used in the first portion of the command. This is necessary so that all of the content is loaded in to memory before it is passed down the pipeline. If we don't do that the command will attempt to write to the file which it is currently reading and will throw an error.

PS C:\> Get-Content test.bin | % { $_ -replace " |`t|\*|`'|`"| |\0x06|`0",""} | Set-Content test.bin
Set-Content : The process cannot access the file 'C:\test.bin' because it is being used by another process.


It is pretty easy with PowerShell, now lets take a look at the Windows command line.

Windows command line

We will start off by using the same file as above; and we will use the standard Windows command line parser, the For Loop.

To see how the For Loop and variables handle the special characters, we will do a quick test of the For Loop without using any delimiters.

C:\> for /f "delims=" %a in ('type test.bin') do @echo %a
Grandma Got Runover*By'A"Reindeer[0x06]Last


Oh No! Notice that we lost the last word. This happened because in the Windows command line variables are null terminated, meaning that the NUL character is deemed to be the end of the string so nothing beyond it will be processed. So we can't work with the NUL character, first strike on Santa's list.

Now, lets try to remove those other pesky characters.

C:\> for /F "tokens=1-8 delims=*^' " %a in ('type test.bin') do @echo %a%b%c%d%e%f%g%h
GrandmaGot RunoverByA"Reindeer[0x06]Last


So we can't represent the tab character, the double quote, or the special character either! Usually the caret character can be used to escape special characters, like the single quote. But for some reason it won't work to escape the double quote. Second strike on Santa's list.

However, we do have a work around for the tab character. We can tell cmd to disable file and directory name completion characters so we can type the tab character. All we have to do is tell cmd to F off.

cmd.exe /F:off


Unfortunately, this can't be prepended to our other command and has to be a separate instance. But now we can type the tab character. All we have to do is add it as a delimiter and we are good to go.

C:\> for /F "tokens=1-8 delims=    *^' " %a in ('type test.bin') do @echo %a%b%c%d%e%f%g%h
GrandmaGotRunoverByA"Reindeer[0x06]Last


One more problem, we can only remove so many characters from a line. Why? Because only the variables a-z are available to represent the tokens.

Given a file with this content:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28


To remove the space we would have to use this command:

C:\> for /F "tokens=1-26" %a in ('type test.txt') do @echo %a%b%c%d%e%f%g%h%i%j%k%l%m%n%o%p%q%r%s%t%u%v%w%x%y%z
1234567891011121314151617181920212223242526


Uh Oh! We lost 27 and 28 since we don't have a way to represent them. Strike three on Santa's list.

We can preserve the rest of the line, but we can't remove the remaining "space" characters.

C:\> for /F "tokens=1-25*" %a in ('type test.txt') do @echo %a%b%c%d%e%f%g%h%i%j%k%l%m%n%o%p%q%r%s%t%u%v%w%x%y%z
1234567891011121314151617181920212223242526 27 28


In the above command the first 25 tokens are now represented by a-y. The 26th token is the remainder of the line, and is represented by z. Ugh!

Too bad cmd.exe is missing a nice easy way to do this, but there are a lot of things missing from cmd. I wish it had a compass in the stock and a thing which tells time, but if it did I would probably shoot my eye out with it.

Hal says 'Oh Holy... Night':

Um. Wow. After reading Tim's fearsome fu I am ever more thankful this holiday season that I spend the vast majority of my time working with the Unix shell. Oh and by the way, Tim, "Bash away, bash away, bash away all!" is my line.

Let's review some basics. Producing "special" characters in the Unix shell is generally pretty easy using some common conventions. On such convention is the use of backwhacks ("\") to protect or "escape" special characters like quoting characters from being interpolated by the shell. Backwhacks can also be used in special escape sequences such as "\t" and "\n" to represent characters like tab and newline. And failing all of those options you can use "\xnn" or "\nnn" to represent arbitrary ASCII codes in either hex or octal.

All of these conventions can be demonstrated with a simple "echo -e" command to output Tim's example input. The "-e" option is necessary here to force proper interpolation of the various backwhacked sequences:

$ echo -e "Grandma Got\tRunover*By'A\"Reindeer\x06Last\000Night." | cat -A
Grandma Got^IRunover*By'A"Reindeer^FLast^@Night.$

The space is represented by a literal space and the tab by "\t". I've enclosed the expression in double quotes so we don't need to backwhack either the "*" or the single quote "'", but we do need a backwhack in front of the "literal" double quote that we want to output (so that it doesn't terminate the double-quoted expression too early). The remaining special characters are produced using their ASCII values in hex and octal. I've piped the output into "cat -A" so that you can more easily see the special characters in the output.

The general quoting rules and escape sequences can vary slightly with different commands. For example, one way to strip characters from our sample input line is with "tr -d". However, while tr understands octal notation for specifying arbitrary characters, it doesn't handle the "\xnn" hex notation. This is not a huge deal, and we could just write:

$ echo -e "Grandma Got\tRunover*By'A\"Reindeer\x06Last\000Night." | tr -d " \t*'\"\006\000" | cat -A
GrandmaGotRunoverByAReindeerLastNight.$

Again I'm using "cat -A" to confirm that I really did strip out all the characters we wanted to remove.

If you really, really insist on having the "\xnn" hex escape sequence, you could use the special $'...' quoting notation in bash that forces interpolation using the same rules as "echo -e":

$ echo -e "Grandma Got\tRunover*By'A\"Reindeer\x06Last\000Night." | tr -d $' \t*\'"\x06\\000' | cat -A
GrandmaGotRunoverByAReindeerLastNight.$

Notice that I now had to backwhack the single quote in my list of characters, but I was able to drop the backwhack in front of the double quote. Also notice that while you can write "\x06" inside of $'...', you need double backwhacks in front of the octal.

Another way to remove characters from a line is with sed. However, while sed understands the "\xnn" hex notation, it doesn't grok specifying characters with octal:

$ echo -e "Grandma Got\tRunover*By'A\"Reindeer\x06Last\000Night." | sed "s/[ \t*\'\"\x06\\000]//g" | cat -A
GrandmaGotRunoverByAReindeerLast^@Night.$
$ echo -e "Grandma Got\tRunover*By'A\"Reindeer\x06Last\000Night." | sed "s/[ \t*\'\"\x06\x00]//g" | cat -A
GrandmaGotRunoverByAReindeerLastNight.$

Even with these annoying little inconsistencies, life with bash, tr, and sed is infinitely preferable to the lump of coal my co-authors have to deal with.

So Merry \x-Mas to all, and to all a good night!

Tuesday, December 1, 2009

Episode #71: Joining Up

Hal fields a question from IRC

Mr. Bucket passed along the following query from the PaulDotCom IRC channel:

What functionality is available to loop through multiple files, and write the output to a single file with some values on the same line? Ex: If one program gives me the hash of a file, and the other program outputs the name/size/etc of a file, can I output to the same file HASH-FileName-Size


I couldn't resist chortling with glee when this question came up, because it's another one of those "easy for Unix, hard for Windows" kinds of tasks. I never can resist sharing these "learning experiences" with my fellow co-authors.

First let's review our inputs. I'm going to use the openssl utility for generating checksums, since it's fairly generic to lots of different flavors of Unix at this point:

$ openssl sha1 *
SHA1(001.jpg)= a088531884ee5eb520e98b3e9e18283f29e13d25
SHA1(002.jpg)= 77febb1498b2926ee6a988c97f3457e38736456d
SHA1(003.jpg)= 922bcb001d025d747c2ee56328811a4270b62079
...

As you can see, it's pretty easy to generate a set of checksums over my directory of image files, but there's a bunch of cruft around the filename that's not really helpful. So let me get rid of that with some quick sed action:

$ openssl sha1 * | sed -r 's/SHA1\((.*)\)= (.*)/\1 \2/'
001.jpg a088531884ee5eb520e98b3e9e18283f29e13d25
002.jpg 77febb1498b2926ee6a988c97f3457e38736456d
003.jpg 922bcb001d025d747c2ee56328811a4270b62079
...

That's better! In the sed expression I'm using the "(.*)" sub-expressions to match the file name and the checksum in each line, and the substitution operator is replacing the original line with just the values of the sub-expressions. Slick.

Now that we've got the checksums, how do we produce the file sizes? I could just use "ls -l" of course. But since the questioner seems to only want "HASH-FileName-Size", I may as well just use "wc -c" to produce simpler output:

$ wc -c *
4227504 001.jpg
4600982 002.jpg
4271719 003.jpg
...

Now that I know what my inputs are going to be, the question is how to stitch them together? Luckily, Unix includes the join command for putting files together on arbitrary fields (we last saw the join command back in Episode #43). Now I could save the checksum output and the file sizes to separate files and then join the contents of the two files, but bash actually gives us a cooler way to handle this:

$ join -1 1 -2 2 <(openssl sha1 * | sed -r 's/SHA1\((.*)\)= (.*)/\1 \2/') <(wc -c *)
001.jpg a088531884ee5eb520e98b3e9e18283f29e13d25 4227504
002.jpg 77febb1498b2926ee6a988c97f3457e38736456d 4600982
003.jpg 922bcb001d025d747c2ee56328811a4270b62079 4271719
...

See the "<(...)" syntax? That's a little bit of bash file descriptor magic that allows us to substitute the output of a command in a place where a program would normally be looking for a file name. In this case it saves us the hassle of having to create intermediate output files to join together. The join command itself is pretty simple. We're telling the program to join the output of the two commands using the file names in the first field of input #1 and the second field of input #2. The only problem is that the join command isn't producing the "HASH-FileName-Size" output that the original questioner wanted. That's because join always outputs the joined field first, followed by the remaining fields from the first input (the checksum in this case), followed by the remaining fields from the second input (the file size). We'll have to use a little awk fu to re-order the fields:

$ join -1 1 -2 2 <(openssl sha1 * | sed -r 's/SHA1\((.*)\)= (.*)/\1 \2/') <(wc -c *) \
| awk '{print $2 " " $1 " " $3}'

a088531884ee5eb520e98b3e9e18283f29e13d25 001.jpg 4227504
77febb1498b2926ee6a988c97f3457e38736456d 002.jpg 4600982
922bcb001d025d747c2ee56328811a4270b62079 003.jpg 4271719
...

Mmmm, that's a tasty little bit of shell magic, isn't it? Let's see what Ed and Tim are cooking up.

Loyal reader Jeff Haemer points out that you don't need awk if you understand how to work join's "-o" option to select your output fields:

$ join -1 1 -2 2 -o 1.2,0,2.1 <(openssl sha1 * | sed -r 's/SHA1\((.*)\)= (.*)/\1 \2/') <(wc -c *)
a088531884ee5eb520e98b3e9e18283f29e13d25 001.jpg 4227504
77febb1498b2926ee6a988c97f3457e38736456d 002.jpg 4600982
922bcb001d025d747c2ee56328811a4270b62079 003.jpg 4271719
...

Yep, join actually lets you select specific fields from each input file and specify the order you want them output in. Nice. Thanks, Jeff!


Ed retorts snidely:
Choosing a topic just because you think it's hard for us Windows guys, huh, Hal? Well, aren't you just a big ball of sunshine, a command-line Scrooge this holiday season? When I first read this one, I though... "Ugh... this is gonna be hard." Perhaps I was psyched out by your juvenile trash talk. Or, maybe I've just been hanging around in cmd.exe too long, and have gotten used to hard problems.

But, this one turned out to be surprisingly straight-forward and even non-ugly (well, beauty is in the eye of the beholder, I suppose). Here's the skinny:
C:\> FOR /f "tokens=1-2" %a in (name-hash.txt) do @for /f "tokens=1,2" %m
in (length-name.txt) do @if %a==%n echo %b %a %m
a088531884ee5eb520e98b3e9e18283f29e13d25 001.jpg 4227504
77febb1498b2926ee6a988c97f3457e38736456d 002.jpg 4600982
922bcb001d025d747c2ee56328811a4270b62079 003.jpg 4271719

I'm assuming that name-hash.txt contains, well, names and hashes, one pair per line. Likewise, length-name.txt contains lengths and names, again one pair per line.

As we know, FOR /F loops can parse through all kinds of crap, including the contents of files. I use a FOR /F loop with two tokens (giving me two variables) of %a (for the file name) and %b (allocated automagically, holding the hash). For each of the files described in name-hash.txt, I then construct the body of my FOR loop. It contains another FOR /F loop, again with two variables (the original question mentioned "etc" for extra stuff there... if you have more stuff, just up the number of tokens and echo the proper variables at the end). My inner FOR /F loop iterates through the length-name.txt file, placing its values in the variables %m (length) and %n (name).

Now, if I just echoed out %a %b %m %n, I'd be making all of the possible combinations of every pair of two lines in the original files. But, we want to pare that down. We only want to generate some output if the name from name-hash.txt (%a) matches the name from length-name.txt (%n). We do this with a little IF operation comparing the two variables. If they match, we then echo out hash (%b), name (%n), and size (%m).

Admittedly, the performance of this little command isn't great, as I have to run through every line of name-hash.txt, comparing the name by running through the entirety of length-name.txt. I don't stop when I've found a match, because, well, there could be another match somewhere. Also, if there is no match of the name between the two files, my command ignores that name, not issuing any output. But, I think that makes sense given what the questioner asks.

So, Tim... does PowerShell have a nifty little built-in or something to make this easier than running through a couple of FOR loops? Inquiring minds what to know.

Tim tags in for Ed:

For loops! We don't need no stinking For loops!

The first thing to do is import the files. Since there is a space between the columes we can use Import-CSV with a delimiter of the space character. Also, there is no header information so we have to specify it.

PS C:\> Import-Csv length.txt,hash.txt -Delimiter " " -Header File,Data
File Data
---- ----
001.jpg 4227504
002.jpg 4600982
003.jpg 4271719
001.jpg a088531884ee5eb520e98b3e9e18283f29e13d25
002.jpg 77febb1498b2926ee6a988c97f3457e38736456d
003.jpg 922bcb001d025d747c2ee56328811a4270b62079
...


We have all the data, so now it can be grouped by the file name using Group-Object (alias group).

PS C:\> Import-Csv length.txt,hash.txt -Delimiter " " -Header File,Data | group file

Count Name Group
----- ---- -----
2 001.jpg {@{File=001.jpg; Data=4227504}, @{File=001.jpg; Data=a088531884ee5eb520e98b3e9e18283f29e13d25}}
2 002.jpg {@{File=002.jpg; Data=4600982}, @{File=002.jpg; Data=77febb1498b2926ee6a988c97f3457e38736456d}}
2 003.jpg {@{File=003.jpg; Data=4271719}, @{File=003.jpg; Data=922bcb001d025d747c2ee56328811a4270b62079}}
...


We have the data grouped like we want, but we still need to massage it a bit so we can get the formate we want.

PS C:\> Import-Csv length.txt,hash.txt -Delimiter " " -Header File,Data |
group file | Select @{Name="Hash";Expression={$_.Group[1].Data}}, Name,
@{Name="Length";Expression={$_.Group[0].Data}}

Hash Name Length
---- ---- ------
a088531884ee5eb520e98b3e9e18283f29e13d25 001.jpg 4227504
77febb1498b2926ee6a988c97f3457e38736456d 002.jpg 4600982
922bcb001d025d747c2ee56328811a4270b62079 003.jpg 4271719
...


The Select-Object (alias select) cmdlet allows for custom expressions which was used to get the hash and the length. The "Group" object contains multiple items and each can be access by its index value, 0 is the length and 1 is the hash.

Fileless PowerShell

The initial task was to get the file name, length, and hash from separate files and combine them in to one. Let's try this again without using files.

This would be very easy if powershell just had a hashing cmdlet, but it doesn't. However, we can do hashing by using the .NET library and some very ugly PowerShell. Maybe in v3 we will get a Get-Hash cmdlet, but it seems as likely as the addition of Get-Unicorn or Get-MillionDollars.

So we need some hash, but not the kind that is illegal in 49 states, we need the hash of a file. Here is how we get it.

PS C:\> PS C:\> gci 001.jpg | % { (New-Object System.Security.Cryptography
.SHA1CryptoServiceProvider).ComputeHash($_.OpenRead()) }


We use the SHA1CryptoServiceProvider .NET class, but it adds another bump since it doesn't take files as input and will only take a stream. It isn't hard to get the stream though, all we need to use is the OpenRead method of our file object. If that wasn't enough, there is another problem, the output.

PS C:\> PS C:\> gci 001.jpg | % { (New-Object System.Security.Cryptography
.SHA1CryptoServiceProvider).ComputeHash($_.OpenRead()) }

160
136
83
24
...


The result is an array of bytes. So we have to convert that to hex and combine it together.

PS C:\> gci 001.jpg | % {$hash=""; (New-Object System.Security.Cryptography
.SHA1CryptoServiceProvider).ComputeHash($_.OpenRead()) | % { $hash += $_.ToString("X2") }; $hash}

a088531884ee5eb520e98b3e9e18283f29e13d25


We use the ToString method with the format string X2 to convert each byte to hex. The X converts it to hex, and the 2 will make sure the output is two characters wide (0A vs A). We then use the variable $hash to stitch our bytes together to get the full hash.

Now let's see the full command.

PS C:\> gci *.* | select @{Name="Hash";Expression={$hash=""; (New-Object
System.Security.Cryptography.SHA1CryptoServiceProvider).ComputeHash($_.OpenRead()) |
% { $hash += $_.ToString("X2") }; $hash}}, name, length

Hash Name Length
---- ---- ------
a088531884ee5eb520e98b3e9e18283f29e13d25 001.jpg 4227504
77febb1498b2926ee6a988c97f3457e38736456d 002.jpg 4600982
922bcb001d025d747c2ee56328811a4270b62079 003.jpg 4271719
...


The first thing we do is get all the files in the currect directory using Get-ChildItem (aliased as gci or dir). That is piped in to Select-Object (aliased as select) to get the hash, filename, and size. The Select-Object cmdlet allows us to get properties of the pipeline object as well as creating a custom expression. In our case we will use the custom expression to calculate the hash.

Our results are in object form and can be piped to a file with Out-File or Out-Csv.

So the task is complete, but let's pretend for a second we had the fictional Get-Hash cmdlet. If we had our leprachaun our command might look something like this:

PS C:\> gci *.* | select @{Name="Hash";Expression={Get-Hash $_ sha1}, name, length


If only getting hash was easier in Windows.

Tuesday, November 24, 2009

Episode #70: The Tangled Web

Hal gets a soft one this week

Lately we've had some of our loyal readers-- mostly the Windows folk-- asking about command-line tools for accessing web pages. When these questions come up, I just smile serenely, because it's easy to do this in Unix. Ed and Tim on the other hand turn a little green and start sweating.

Among the Unix tribes, there seem to be two popular command-line tools for grabbing web pages. Some folks prefer curl, but I generally stick with wget since it's the tool I learned first. Both curl and wget support HTTP, HTTPS, and FTP as well as proxies of various types along with different forms of authentication. curl also supports other protocols like TFTP, FTPS (FTP over SSL), and SCP and SFTP.

Using wget couldn't be simpler:

$ wget blog.commandlinekungfu.com
--2009-11-18 18:47:51-- http://blog.commandlinekungfu.com/
Resolving blog.commandlinekungfu.com... 74.125.113.121
Connecting to blog.commandlinekungfu.com|74.125.113.121|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: `index.html'

[ <=> ] 173,805 18.3K/s in 9.3s

2009-11-18 18:48:01 (18.3 KB/s) - `index.html' saved [173805]


Or, if you don't like all the noisy chatter, you can use the "-q" option:

$ wget -q blog.commandlinekungfu.com
$ ls
index.html index.html.1

Notice that wget doesn't overwrite the first index.html file we downloaded. Instead it appends a uniqe number to the file name. If we downloaded another index.html file, it would end up as index.html.2 and so on. You can also use "-O" to specify an output file name.

Maybe my favorite feature, however, is the "-r" (recursive) option that allows me to grab an entire tree of content in a single operation. So all I have to do when I want to back up the content here at Command Line Kung Fu is just run "wget -r -q blog.commandlinekungfu.com" and wait a few minutes for everything to download.

There are lots of other neat features to wget, but I think I've already demoralized my Windows brethren enough for one Episode. Let's see what Ed and Tim cook up.

Tim orders take-out:
Unfortunately (again), there isn't a built-in cmdlet to do the equivalent of wget or curl, but we can access the .NET libraries and recreate some of the functionality of the Linux commands. By default, the .NET Framework supports URIs that begin with http:, https:, ftp:, and file: scheme identifiers, so it isn't quite as full featured as Linux, but it is all we have.

PS C:\> (New-Object System.Net.WebClient).DownloadString(
"http://blog.commandlinekungfu.com")


<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
...


This will grab files in text format and it can be used further down the pipeline, such as saving the file by piping it in to Out-File. What if we want to grab non-text files or just download the file? We can use the DownloadFile method and specify where we want to save the file.

PS C:\> (New-Object System.Net.WebClient).DownloadFile(
"http://blog.commandlinekungfu.com","c:\downloadedfile.html")


What happens if the file doesn't exist? It raises an 404 error and the file (obviously) isn't opened.

PS C:\> (New-Object System.Net.WebClient).DownloadString(
"http://blog.commandlinekungfu.com/NonExistant.file")


Exception calling "DownloadString" with "1" argument(s): "The remote
server returned an error: (404) Not Found."
At line:1 char:49
+ (New-Object System.Net.WebClient).DownloadString( <<<<
"http://blog.commandlinekungfu.com/NonExistant.file")


Sadly, there isn't way to perform recursive requests without doing some heavy duty scripting.

Ed Sits Here Trying Not To Think About What Hal Called His Part of this Article:

Ahhh... an age-old question. I remember about a year ago, I was having a discussion with Kevin Johnson, bon vivant and web attacker extraordinaire, when this issue came up. I posed the following question to KJ0 (as we call him): "Suppose you've just hacked a Windows box in a pen test. You have command shell access of said machine. Tell me what you want to do now, using only built in command-line capabilities." I went further, boldly mixing my metaphors: "Consider me as /dev/WindowsCommandLine... where do you want to go today?"

Kevin shrugged, smiled, and said, "Easy... wget."

I felt a piercing pain stab at my heart. "The wget tool has sooo many options... can we scale it back a bit? What do you really want to do?" Kevin said, "Sure... I just wanna fetch a web page and write it to the file system. How can I do that at the command line in Windows?"

I spent a little time fitzing around with various Windows commands, and ultimately settled on using the built-in Windows telnet client to formulate HTTP requests manually. It's crude, but works.

"But, Ed," you argue, "Microsoft removed the telnet client from Windows Vista and some of the 6 million versions of Windows 7." I respond: Yes, it's no longer installed by default, but on the professional versions of Vista and 7, you can run:

C:\> start /w pkgmgr /iu:"TelnetClient"


Even if it doesn't finish, kill that cmd.exe, and you should have the Telnet Client ready to run. Note that the installation package for the telnet client is built-in, but the tool itself just isn't installed. The machine doesn't have to reach out to the Internet to get it. Also, note that /iu stands for install update, you need that :, and you should observe the case of the Cap-T and Cap-C in TelnetClient. Oh, and by the way... if you want the Telnet service, change "TelnetClient" to "TelnetServer". To remove either after you've installed, it, run the same command, except substitute /uu: (uninstall update) for /iu:.

So, now equipped with the telnet client, let's make an HTTP request.

Now, you might think that we should simply echo an HTTP request and pipe it into the telnet client, right? Bzzzzzt. Sorry, my friend, but the Windows telnet client doesn't like anything to come into its Standard Input at the command line, and it certainly doesn't send it to the target machine. Try sniffing it sometime. Another annoying part about the Windows telnet client is that it doesn't like you to touch its Standard Output. The telnet client is somehow clairvoyant, and if you follow it with a > or a |, it knows and refuses to send any packets. Gee, thanks, Microsoft.

For this one, we're going to have to go interactive.

C:\> telnet -f log.txt

I'm logging output to the file log.txt, where our output will reside.

At our new telnet prompt, we can open a connection to port 80 on our destination web server. We'll use the "o" option to open a connection:

Microsoft Telnet> o www.google.com 80
Connecting To www.google.com...

Now, depending on the version of Windows you are using, it may or may not look like a connection is made. Either way, just trust that it has, and start typing your HTTP request. Typically, you'll be getting a page, which you can do by entering:

GET [pagename] HTTP/1.1
Host: [TheHostName]

Hit Enter Enter at the end, and you'll have the page displayed on your screen. Hit Enter again, and you'll get back to your telnet prompt. Type "quit" to exit.

More importantly, you should now have your page in the output file log.txt. Yes, you'll have the HTTP response header in front of it, but that's not so bad. It's ugly, but it'll let you grab a file quickly from a server.

Tuesday, November 17, 2009

Episode #69: Destroy All Connections

Ed looks out on the serene waters of Tokyo Bay:

Mr. Byte Bucket sent in a request from the ever insightful Pauldotcom IRC channel:

Can anyone suggest a Windows cmd to disconnect a specific socket?

Nice question! Unfortunately, Windows doesn't offer much in the way of built-in tools that are fine grained enough to operate at the socket level. But, we can either restart the service handling the connection, or, if you want to be a little more violent, just kill its service.

We'll start by taking the violent rout. First off, we need to figure out what processid is associated with the connection you seek. You can get a list of all connections to a given port using the following command:

C:\> netstat -nao | find ":[port]"

The right-most (i.e., fifth) column is the processid number.

If you have multiple different clients clients connected to that same destination port, you can select out the given process that is associated with a specific client connection using:

C:\> netstat -nao | find ":[port]" | find "[ClientIPaddr]"

You can then kill that process using wmic. However, be very careful! That process may be really important, and killing it could end you up in a world of hurt. In fact, a lot of Windows built-in features (such as file sharing and IIS) are all associated with the "System" service (with a PID of 4 or 8 depending on the version of Windows you are using).

C:\> wmic process where processid="[PID]" delete

You can wrap this all up in one very dangerous command... but I really don't recommend doing this. Again, if you inadvertently kill the wrong process, your system could come crashing down around you. I recommend figuring out what process is at issue first, investigating it, and only then killing it.

C:\> for /f "tokens=5" %i in ('netstat -nao ^| find ":[port]" ^| 
find "[ClientIPaddr]"') do @wmic process where processid="%i" delete
Deleting instance \\WILMA\ROOT\CIMV2:Win32_Process.Handle="[PID]"
Instance deletion successful.

Our second approach involves restarting the service associated with the process. Based on the processID information we retrieved above, we can get a list of the associated services by running:

C:\> tasklist /fi "pid eq [PID]" /svc

That command tells us all of the services that are associated with a given process. We'll then have to do a little research to figure out which specific service it is that is handling our given connection. Unfortunately, Windows doesn't give us the ability to map a connection directly to a service, but we must instead map a connection to a process, which we then map to a set of services running from within that process. Now, if there's only one service in that process, we're golden. But, if there are more, we have to do this research step to find out which service it really is that we need to focus on.

Once you've discovered the appropriate service, you can then restart the service using:

C:\> sc stop [service] & sc start [service]

It's kind of annoying, I think, that there is no "restart" option explicitly here, so we have to stop the service and then start it. Not a huge liability, but still a little frustrating.

Tim Sees Something Starting to Stir in the Otherwise Tranquil Waters of the Bay:

PowerShell doesn't (yet) have any cmdlets similar to netstat. The Grand Poobah of PowerShell (@halr9000 on twitter) has a PowerShell script to "objectize" netstat, but that crosses in to scriptland so we will steer clear.

To find the connection and its associated process you will have to refer to the first portion of Ed's section. But once we have the process id we can use PowerShell cmdlets.

So we have the PID, now we can get the process object by using the aptly named command Get-Process. The default method for Get-Process is the process name, so we need to use the Id parameter.

PS C:\> Get-Process -Id 3004
Handles NPM(K) PM(K) WS(K) VM(M) CPU(s) Id ProcessName
------- ------ ----- ----- ----- ------ -- -----------
210 7 5116 13344 61 6.67 3004 Evil


The process can be killed using PowerShell, but as Ed said, "Be very careful!" The Stop-Process cmdlet's default method is the process Id so we aren't required to use Id parameter (but you can if you wish).

PS C:\> Stop-Process 3004


If the offending process is a service, we can't retrieve the service from the process id using by using just Get-Proccess since it doesn't include the Id property. However, we can use wmi in conjunction with Get-Service to "get" the service.

PS C:\> Get-WmiObject win32_service | ? { $_.ProcessID -eq 3004 } | % { Get-Serv
ice $_.Name }


To paraphrase Ed (again), you will have to do some research to since there can be multiple services running from within that process (multiple services be in one process).

Once we find the service that needs a kick, we can stop it using Stop-Service. We also have the ability to restart the service using Restart-Service.

If you felt gutsy you could pipe the command above into Restart-Service or Stop-Service.

PS C:\> Get-WmiObject win32_service | ? { $_.ProcessID -eq 3004 } | % { Get-Serv
ice $_.Name } | Restart-Service


PS C:\> Get-WmiObject win32_service | ? { $_.ProcessID -eq 3004 } | % { Get-Serv
ice $_.Name } | Stop-Service


..or you could do it manually.

PS C:\> Stop-Service [Service]


PS C:\> Restart-Service [Service]


It would be nice if there was a facility in PowerShell to just kill a connection. Maybe we can get that in version 3, but while we wait for the additional capability I get the uneasy feeling Hal is getting ready to squash us like bugs.

And the waters of Tokyo bay begin to boil:

Oh dear. I really am going to have to open my Godzilla-sized can of whup-ass on my poor Windows-using bretheren. But first, let me try to make them feel not so bad by doing a quick netstat-based solution.

On Linux systems at least, netstat has a "-p" option to display process information. Let's take a quick look at some sample output:

# netstat -anp --tcp -4 | grep :22
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 15805/sshd
tcp 0 0 192.168.1.4:60279 192.168.1.2:22 ESTABLISHED 18054/ssh
tcp 0 0 192.168.1.4:32921 192.168.1.2:22 ESTABLISHED 19409/ssh

Here I'm listing all TCP sockets that are using IPv4 ("--tcp -4"), and using "-n" so that the socket numbers are not converted into human-readable names. Anyway, as you can see, the 7th column is "PID/name" (this is what the "-p" option does).

So with the help of awk and cut, we can pull out just the PID of the master SSH daemon:

# netstat -anp --tcp -4 | awk '/:22/ && /LISTEN/ { print $7 }' | cut -f1 -d/
15805

Killing that process just requires appropriate use of backticks:

# kill `netstat -anp --tcp -4 | awk '/:22/ && /LISTEN/ { print $7 }' | cut -f1 -d/`

We could also use cut to pull out the second field, if we wanted to shut down the process using it's init script:

# /etc/init.d/`netstat -anp --tcp -4 | awk '/:22/ && /LISTEN/ { print $7 }' | cut -f2 -d/` stop
bash: /etc/init.d/sshd: No such file or directory

Rats, it's /etc/init.d/ssh on this system, and not /etc/init.d/sshd, but you get the idea.

But really, the 300 foot giant mutated atomic lizard we want to employ here is lsof. You might be aware that we can get lsof to show us just the port 22 stuff with it's "-i" flag:

# lsof -i :22
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
sshd 15805 root 3u IPv6 37028 TCP *:ssh (LISTEN)
sshd 15805 root 4u IPv4 37030 TCP *:ssh (LISTEN)
ssh 18054 hal 3u IPv4 44514 TCP elk.deer-run.com:60279->deer.deer-run.com:ssh (ESTABLISHED)
ssh 19409 hal 3u IPv4 53249 TCP elk.deer-run.com:32921->deer.deer-run.com:ssh (ESTABLISHED)

If we wanted to kill the SSH daemon, we could easily adapt the awk fu from the netstat example to pull out the appropriate PID value and then use backticks to feed this value into the kill command.

But we don't need awk, because lsof has the "-t" ("terse") option, which just spits out the PIDs:

# lsof -t -i :22
15805
18054
19409

The "-t" option is specifically designed so that you can do things like "kill `lsof -t -i :22`". Of course, the problem with doing that is that I'd also end up killing my SSH sessions to the remote machine deer.deer-run.com-- PIDs 18054 and 19409-- which I don't really want to do.

So how do we pull out just the daemon PID? Well looking at the lsof output above, I can see that I want to kill the sshd processes that are listening on port 22 but leave the regular ssh processes alone. I could use the "-c" option to select the PIDs by command name:

# lsof -t -c sshd
15805

But that means I'd have to already know the command name associated with the port in question-- in this case by having already run lsof once to search by port number. And if I knew the command name already, why wouldn't I just use pkill (see Episode #22) instead of lsof?

What I really want is a single command-line using just kill and lsof that lets me reliably destroy the master server process as effectively as Godzilla destroys Tokyo. Luckily, lsof has some more atom-age madness up its sleeve. You see, lsof's "-c" option isn't just limited to simple substring matching: you can use "/.../" to specify a full-on egrep-style regular expression. Because Unix server processes almost always end in "d" (think sshd, httpd, ftpd, and so on), we can construct a similar regular expression to match the daemon process name associated with an arbitrary port number:

# lsof -a -i :22 -c /d$/
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
sshd 15805 root 3u IPv6 37028 TCP *:ssh (LISTEN)
sshd 15805 root 4u IPv4 37030 TCP *:ssh (LISTEN)

The "-a" option means to logically "and" your search criteria together (the default is logical "or" for some strange reason, although this is almost never what you want). So here we're looking for everything using port 22 and where the process name ends with "d".

Adding the "-t" option and some backtick action, we have our final answer:

# kill `lsof -t -a -i :22 -c /d$/`

And with that, I shall swim home to Monster Island and await my next challenge.

Tuesday, November 10, 2009

Episode #68: Help Fu

Tim hits lead off:

This week we get help with help. Everyone needs help, and some in different ways than others. But we aren't going to discuss Ed's problem with crunchy peanut butter or Hal's issue with furry animals with tails. Anyway, back to the issue at hand.

Instead of jumping right in to getting help on commands, let's use PowerShell to find commands by using the Get-Command cmdlet. This command will retrieve all the cmdlets, aliases, and functions available to us. In order to use this command effectively, let's go over a bit of the very basics of PowerShell. As you all probably know, the nomenclature for PowerShell cmdlets is verb-noun. Microsoft's approved verb list defines the verbs so you know which verb to use and to ensure consistent use by cmdlet developers. That way the verbs are consistent so "get" is always used instead of a random choice of get, retrieve, grab, or obtain. The noun part of the name identifies the entity on which the action is performed.

Ed and Hal discussed the command history in episode #27, so we'll use that as the focus of our research.

Let's see a list of all the available nouns.

PS C:\> Get-Command -Type cmdlet | Sort-Object noun | Group-Object noun

Count Name Group
----- ---- -----
2 Acl {Set-Acl, Get-Acl}
5 Alias {Import-Alias, Get-Alias, Set-Alias, Export-...
...
4 History {Invoke-History, Clear-History, Get-History,...
...


You probably could have already guessed, but the noun we want to use is "History." To get a lits of the commands available using that noun we can use this command:

PS C:\> Get-Command -Noun history

CommandType Name Definition
----------- ---- ----------
Cmdlet Add-History Add-History [[-InputObject] ...
Cmdlet Clear-History Clear-History [[-Id] <Int32[...
Cmdlet Get-History Get-History [[-Id] <Int64[]>...
Cmdlet Invoke-History Invoke-History [[-Id] <Strin...


The parameter is not case sensitive and will accept wildcards so "hIST*" would returned similar results. Similarly, we could see all the available commands using the "get" verb using this command.

PS C:\> Get-Command -Verb Get


Back to the subject at hand...

Now we have a list of cmdlets we are interested in. Let's see how we can use the command by using the Get-Help cmdlet.

PS C:\> Get-Help Get-History

NAME
Get-History

SYNOPSIS
Gets a list of the commands entered during the current session.

SYNTAX
Get-History [[-Id] ] [[-Count] ] []
...


We have a bit of a problem here, the basic help isn't super useful since it doesn't provide a good description of the parameters or any examples of the cmdlet's use. However, it does provide syntax and a list of parameters. To see the full help including examples and parameter descriptions use this command:

PS C:\> Get-Help Get-History -Full


To see just the examples:

PS C:\> Get-Help Get-History -Examples


As you have probably noticed in past episodes there are a lot of aliases available in PowerShell. The Get-Alias cmdlet can be used to "get" the list of aliases. Specifically, we can see the aliases for the commands using the History noun.

PS C:\> Get-Alias | Where-Object {$_.Definition -like "*history"}

CommandType Name Definition
----------- ---- ----------
Alias clhy Clear-History
Alias ghy Get-History
Alias h Get-History
Alias history Get-History
Alias ihy Invoke-History
Alias r Invoke-History


The Get-Command cmdlet can be used to get the same results.

PS C:\> Get-Command -Type Alias | Where-Object {$_.Definition -like "*history"}


That's about it for PowerShell, let's see what Ed has for us with the Windows command line: Get-Help -FuMaster Ed -Subject Windows

Ed's Output:

You guys with your fancy help capabilities in your shells are a real hoot. Yes, your shells include intuitive and easy-to-access help features. Where’s the fun in that? Or, more to the point, where’s the fu in that? I’ve spent many countless hours sitting at a cmd.exe trying to reverse engineer… uh… I mean conducting detailed design recovery of cmd.exe functionality to try to figure out what the heck the little bugger was really doing. Ah… good times.

In all seriousness though, cmd.exe offers an embarrassingly paltry amount of help for using it and its commands. Really, it’s kind of stinky. Let’s explore the few skimpy options we have.

First off, we can run the help command, followed by the command we want to analyze, as in:
C:\> help findstr

The good news? Well, we’ll see a brief description of findstr functionality, as well as its command flags.

The bad news? Well, where to start? How about:

Bad News #1) The help command doesn’t provide help for all of the commands available in cmd.exe. It only covers a little less than 80 of them.

To get a list of the commands that help has information about, simply run:

C:\> help

Some of the most interesting and useful commands of all, like netstat, netsh, reg, sc, and wmic, result in this unhappy little message:
C:\> help reg
This command is not supported by the help utility. Try "x /?".

Bad News #2) The output of the help command doesn’t include any examples or much detail at all. It’s just really basic usage, without any interesting fu included.

Bad News #3) Note the little suggestion in the output of my command in Bad News #1: Windows tells us to try running the command with a /? argument for help. As an old Unix guy (shhhh… don’t tell anyone), I’m very hesitant to run a command that I don’t understand just so I can get help. It scares me to do so without knowing what just running the command might do. And, here’s a dirty little secret of the help command itself. It actually runs the command you are trying to get help about! Yes, check this out…

First, start Task Manager, move to the Processes tab, and sort it alphabetically by clicking on the Image Name column.

Now, in a separate cmd.exe, run the following command, which will get help for the findstr command, repeatedly:

C:\> for /L %i in (1,0,2) do @help findstr

Look in your Task Manager window… see it? Every once in a while, the findstr command pops up! So “help findstr” is actually invoking the findstr command itself to scrape out its help options. Yowza. That’s just painful for me.

Bad News #4: Well, with only around 80 commands covered by the help command, we often have to turn to that old stand by, “[command] /?” for more help. But the help included here is spotty at best as well, with major inconsistencies between commands and some commands supporting context-specific help for particular sub-areas of the command. For example, “wmic /?” does offer some help, but there is a wmic-specific option for getting more detailed help:

C:\> wmic /?:full

Besides wmic, none of the other commands I know about support this /?:full option.

Also, some commands have specific help for certain contexts within the command. Consider the following examples:

C:\> wmic process get /?

This shows you a list of all the process attributes wmic supports.

Or, try this:

C:\> wmic process call /?

This one shows you a list of all the process methods you can call from within wmic.

Or, how about this one:

C:\> net /?

And then:

C:\> net use /?

Likewise, we can run “reg /?” to get an overview of the reg command, followed by a “reg query /?” to get more details about the specific syntax for querying the registry.

Want another bewildering inconsistency? Try running this:

C:\> netstat /?

Note that the output actually shows the command flags with dashes instead of slashes (i.e., “netstat –na” instead of “netstat /na”). Thankfully, the actual netstat command on modern Windows boxes lets you use dashes or slashes in its flags.

So, how can we wrap our heads around all of this inconsistency? I’ve found that mastering Linux involves learning certain general principles and then seeing how they are applied throughout the operating system. Whereas, in Windows, mastering the command line involves memorizing a bunch of complex, counter intuitive, and often inconsistent options scattered throughout the operating system without rhyme or reason. So, why bother? Because, once mastered, the Windows command-line is incredibly useful in analyzing systems at a fine-grained level. Also, cmd.exe is pretty much everywhere, pre-installed, so there’s a lot less hit-and-miss than you get with installs of PowerShell. At least, there is for now… this is changing as PowerShell gets more widely deployed.

The bottom line here? When I’m actually looking for help with a command, I apply the following process. I start by trying “help [command]”. If that provides no satisfaction, I proceed to “[command] /?”. If that doesn’t give me what I want, I try to look for context-specific help with “[command] [option] /?”. And, if that doesn’t get me where I need to go, I turn to the web to research commands. One of my absolutely favorite sites for researching Windows (as well as Linux and Unix) commands is the awesome ss64 site. It includes a detailed list of most Windows commands, including their various flags, example usage, and mapping to rough equivalents in Linux, OS X (sometimes), and PowerShell (occasionally). That’s awesome. And, finally, there’s Microsoft’s own command line reference, worth a check from time to time.

Hal cleans up:

There are actually a number of different help systems available on a typical Unix system. First there's the standard on-line manual pages accessed via the "man" command. The Unix manual pages include not only information about programs that you can run from the command line, but also documentation on programming APIs, device interfaces, and even the format of important configuration files on the system.

The trick sometimes is getting to the right manual page. For example, suppose I wanted some information on how to use the chmod() system call in my C program. If I just typed "man chmod", I'd get the manual page for the chmod program and not the documentation on the system call. To distinguish these two different manual pages, the Unix manual has traditionally been organized into sections. Sections 1 (user commands) and 8 (administrator commands) are devoted to command-line tools, while sections 2 (system calls) and 3 (library routines) are devoted to programming APIs. FYI, section 4 is device interfaces, section 5 is configuration file formats, section 6 is games, and section 7 is "miscellaneous" but includes useful tidbits like the table of ASCII values ("man ascii") and additional details on various IPC mechanisms. When a Unix document refers to "chmod(2)" it means "the documentation on the chmod system call in section 2 of the manual".

But how does on pull up the chmod(2) manual page instead of the default chmod(1) page? The man command takes a "-s" option to specify the section: "man -s 2 chmod". But on many Unix variants you can simply drop the "-s" and just type "man 2 chmod".

Suppose, however, that I didn't know which section the chmod() system call was documented in. One of the most useful features of the man command is the "-k" (keyword search) option:

$ man -k chmod
chmod (1) - change file mode bits
chmod (2) - change permissions of a file
fchmod (2) - change permissions of a file
fchmodat (2) - change permissions of a file relative to a directory f...

You don't have to search on command names, of course. Any related keyword will work. For example, if you're new to Unix and don't know how to rename files:

$ man -k rename
dpkg-name (1) - rename Debian packages to full package names
lvrename (8) - rename a logical volume
mmove (1) - move or rename an MSDOS file or subdirectory
mren (1) - rename an existing MSDOS file
mv (1) - move (rename) files
prename (1) - renames multiple files
rename (1) - renames multiple files
rename (2) - change the name or location of a file
rename.ul (1) - Rename files
renameat (2) - rename a file relative to directory file descriptors
vgrename (8) - rename a volume group

By the way, the "apropos" and "whatis" commands are equivalent to "man -k". However, all of these commands operate on special "databases" of information that has been extracted from the manual pages themselves via the "makewhatis" command (aka "mandb" on Linux). Very often your system will have a cron job that builds the whatis databases automatically every week or so, but on some Unix systems you have to build the databases manually (or create your own cron job).

Another tricky issue with the Unix on-line manual is that sometimes you have multiple repositories of manual pages. For example, you might have the standard OS manual pages under /usr/share/man and manual pages for third-party software in /usr/local/man. The man command lets you use the "-M" option to specify a path to use for finding manual pages, but far it's far easier to set MANPATH in your .bashrc file:

export MANPATH=/usr/share/man:/usr/local/man

MANPATH works just like PATH or LD_LIBRARY_PATH. In the above example, the man command will check the system manual page directory first and then look in /usr/local/man if it doesn't find the manual page you're requesting.

The man command is also the source of one of the most famous jokes in Unix:

$ man 'furry animals with tails'
No manual entry for furry animals with tails

OK, so the traditional joke is "man rear", but you get the idea.

Sadly, man pages seem to be dieing off. A lot of Open Source projects have either very scanty or completely non-existent manual pages. It's actually pretty tough to write a good manual page, both in terms of content and also because you need to know nroff/troff formatting commands to format the text. But writing (and reading) manual pages properly is still an important Unix skill IMHO.

Another source of on-line documentation on Unix is the Free Software Foundation's "info" system. The most complete documentation on packages like emacs, gcc, GNU tar, et al is found in the info system-- the manual pages are mostly just stubs extracted from the docs in the info pages. Run the command "info info" to get more information on how to navigate the curses-based info interface. Frankly, I think the user interface for the info system is terrible, but it was mostly designed to be run from within emacs. I suppose that it makes sense if you're one of those people who essentially lives inside of emacs and never touches an actual terminal window.

But the info system never really caught on outside of the FSF. With traditional manual pages being marginalized as well, the main documentation interface these days seems to be the "-h" or "--help" options supported by many commands. Typically, one or both of these options will generate at least a terse summary of command options:

$ info --help
Usage: info [OPTION]... [MENU-ITEM...]

Read documentation in Info format.

Options:
--apropos=STRING look up STRING in all indices of all manuals.
-d, --directory=DIR add DIR to INFOPATH.
--dribble=FILENAME remember user keystrokes in FILENAME.
-f, --file=FILENAME specify Info file to visit.
-h, --help display this help and exit.
--index-search=STRING go to node pointed by index entry STRING.
-n, --node=NODENAME specify nodes in first visited Info file.
-o, --output=FILENAME output selected nodes to FILENAME.
-R, --raw-escapes output "raw" ANSI escapes (default).
--no-raw-escapes output escapes as literal text.
--restore=FILENAME read initial keystrokes from FILENAME.
-O, --show-options, --usage go to command-line options node.
--subnodes recursively output menu items.
-w, --where, --location print physical location of Info file.
--vi-keys use vi-like and less-like key bindings.
--version display version information and exit.

The first non-option argument, if present, is the menu entry to start from;
it is searched for in all `dir' files along INFOPATH.
If it is not present, info merges all `dir' files and shows the result.
Any remaining arguments are treated as the names of menu
items relative to the initial node visited.

Examples:
info show top-level dir menu
info emacs start at emacs node from top-level dir
info emacs buffers start at buffers node within emacs manual
info --show-options emacs start at node with emacs' command line options
info --subnodes -o out.txt emacs dump entire manual to out.txt
info -f ./foo.info show file ./foo.info, not searching dir

Email bug reports to bug-texinfo@gnu.org,
general questions and discussion to help-texinfo@gnu.org.
Texinfo home page: http://www.gnu.org/software/texinfo/

Not as useful as a traditional manual page, IMHO, but at least it's something. The biggest problem is that with the documentation being tied up in each individual command's "--help" output, there's no "man -k" equivalent for doing global keyword searches if you're not sure what command you're looking for.

Our friend Jeff Haemer is at it again, with some more "helpful" pointers on his blog. Man, I can't believe I forgot the "help" command, but Jeff will tell you all about it, plus some other neat shell trickery.

Tuesday, November 3, 2009

Episode #67: Time Lords

Hal is still adjusting:

Mr. Bucket suggested a time-related Episode this week in honor of the shift back to Standard Time here in the U.S. Generally, we try to humor Mr. Bucket as long as his requests aren't too deviant (or if Ed is the only one who has to dress up), so here goes.

First I've got a couple of tricks up my sleeve related to the NTP time synchronization service. My favorite command-line tool for checking up on my NTP servers is "ntpdc -p":

$ ntpdc -pn server.example.com
remote local st poll reach delay offset disp
=======================================================================
=216.14.97.75 192.168.1.2 16 1024 0 0.00000 0.000000 0.00000
=173.45.227.99 192.168.1.2 2 1024 377 0.06984 -0.003889 0.13681
*64.73.32.135 192.168.1.2 2 1024 377 0.09087 0.002323 0.12178
=127.127.1.1 127.0.0.1 5 64 377 0.00000 0.000000 0.03093

The "st" column that shows the remote server's stratum level. This is useful for detecting downed time servers, because they show up as stratum 16 like on the first line above. The "poll" (polling interval in seconds) and "reach" (displayed in octal notation and capped at 255, the number of recent successful polls) columns give you and idea of how well your server is synchronizing with other time servers. "delay" tells you how far away the remote server is from you, and "offset" is the difference between your local clock and the clock on the remote server (if things are working right, the offset should be less than a hundredth of a second). Note that in the above example, I checked on the remote server.example.com machine, but you can leave off the host name portion to check the time server running on your local box.

In addition to checking status with ntpdc, you can also set the time on your box from the command line using ntpdate. Normally you would just run ntpdate and specify the server you wanted to synchronize against, but here's a cute little bit of awk fu for pulling the server names out of your current ntp.conf file:

# ntpdate `awk '/^(server|peer) / && !/127.127./ {print $2}' /etc/ntp.conf`
29 Oct 19:42:43 ntpdate[13376]: adjust time server 66.187.233.4 offset -0.004818 sec

Here I'm pulling off the host name or IP address specified on the "server" or "peer" lines in your ntp.conf file. I'm skipping any "addresses" that start with "127.127." since these are merely place holders for clock driver definitions and not real IP addresses.

Note that you can only run ntpdate if your NTP daemon is not currently active. If ntpd is running at the same time you execute ntpdate, you get a message like this:

# ntpdate `awk '/^(server|peer) / && !/127.127/ {print $2}' /etc/ntp.conf`
29 Oct 19:42:18 ntpdate[13362]: the NTP socket is in use, exiting

Typically you only need to call NTP date by hand if the NTP daemon is not running and your clock has gotten out of synch. After you use ntpdate to jump your clock to something approximating the right time, you should start your NTP daemon to keep it synced (usually "/etc/init.d/ntpd start").

Getting away from NTP now, let me show you a little environment variable hack that's particularly useful for Forensic Analysts. Suppose you've captured a disk image from a system that was being used in a time zone other than the one your analysis machine is set to. You'd like to mount a copy of the image and be able to observe the timestamps on the files relative to the time zone the image was taken from. You could change the global time zone setting on your analysis workstation, but that could cause you lots of hassle.

It turns out that bash allows you to set the TZ environment variable in your shell to alter the local time zone setting for only that shell. For example, my local time zone is US/Pacific, but suppose I had an image from a machine from the East Coast:

$ date
Thu Oct 29 17:59:13 PDT 2009
$ ls -ld /mnt/image/tmp
drwxrwxrwt 23 root root 4096 Oct 29 17:28 /mnt/image/tmp
$ export TZ=US/Eastern
$ date
Thu Oct 29 21:00:07 EDT 2009
$ ls -ld /mnt/image/tmp
drwxrwxrwt 23 root root 4096 Oct 29 20:28 /mnt/image/tmp

Cool! Now I can look at the dates on files in my image and not constantly have to be doing mental arithmetic to convert the time stamps.

So there are three of my favorite time-related hacks. If we can get Ed out of those leather chaps that Mr. Bucket had him wear, I'm sure he'll have some cool Windows Fu to show us...

Ed chimes in:

Ahhh... time. A great idea for an article, Mr. Bucket. Much better than your last idea... you know... the one about the chaps. Anyway, I digress.

Let's walk before we run. To check the time on your local computer, you can simply run the following command:
C:\> time /t
06:05 AM
Want more precision? We can get that, as discussed in Episode #49, by displaying the %time% environment variable.
C:\> echo %time%
6:05:22.75
Alternatively, the "net time" command can be used to pull the time from a remote (or even local) Windows box:
C:\> net time \\[computername]
Current time at \\FRED2 is 10/31/2009 6:04 AM
The command completed successfully.
Note that, depending on whether you are a member of a domain and the privileges of your current account, you may need to first initiate an SMB connection with that machine, by running:
C:\> net use \\[computername] [password] /u:[username]
The command completed successfully.
If you happen to trust that server, you can set your local machine's time by its clock through running:
C:\> net time \\[computername] /set /yes
Well, that's all well and good, but Hal was working with NTP, which offers many more time sources than, well, our local host or domain controller. How can we pull time from an NTP server in Windows? First, please note that officially, with built-in capabilities, Windows relies on the Simple Network Time Protocol (SNTP), a stripped down NTP implementation that is often used in applications where time accuracy isn't as vital. That's why you'll see sntp throughout our commands below.

Let's first look to see which NTP server our Windows machine is configured to use:
C:\> net time /querysntp
The current SNTP value is: time.windows.com,0x1
The command completed successfully.
The time.windows.com server is the default. Do you trust Microsoft to accurately track the time on their server? After missing the release date of Windows Vista by... uh... something like 3 years, maybe we shouldn't (darnit... I couldn't resist pouring salt in those old wounds.) We can change the NTP server we're configured to use as follows:
C:\> net time /setsntp:pool.ntp.org
The command completed successfully.
Note that you can put a whole list of NTP servers there by following the colon with a space-separated list of NTP servers enclosed in double quotes, resulting in something like /setsntp:"pool.ntp.org ntp.colby.edu tick.gatech.edu"

OK... now, once we've set ourselves up to use an NTP server, let's try to check the time:
C:\> net time
Could not locate a time-server.

More help is available by typing NET HELPMSG 3912.
What? Why doesn't "net time" work here? Note that the HELPMSG is not at all helpful, as it explains that we could not locate a time-server, which we already saw in the error message itself. Gee, Microsoft... thanks for nothing.

It turns out that we can't pull time via "net time" using NTP (really SNTP) unless we're a domain controller with the Flexible Single Master Operation (FSMO) role. Lovely. But, what if we're just a lowly client, maybe not even in a domain? Can we sync with an NTP server using only built in tools? As Bob the Builder and Barack Obama might say: Yes, we can!

We can rely on that happy old stand-by for configuring time on Windows boxen, the intuitively named w32tm command. And in homage to Heinz Doofenshmirtz, by "intuitively named", I mean, of course, "completely non-intuitively named."

To sync with an NTP server, we start by configuring our Windows box (whether it's a client or server version of Windows) to allow us to pull time manually from an NTP server we'd like to use (note that you have to specify this with w32tm even if you've already run "net time" with the /setsntp option):
C:\> w32tm /config /syncfromflags:manual /manualpeerlist:pool.ntp.org
The command completed successfully.
Now that we're configured, just about ready to sync times, let's do a quick time check before the sync:
C:\> time /t
06:47 AM
Oh... that looks dreadfully wrong. I think my clock is fast. Let's cause Windows to read our new configuration now:
C:\> w32tm /config /update
The command completed successfully.
And finally, let's resync:
c:\> w32tm /resync
Sending resync command to local computer
The command completed successfully.
Boom! The clock in my tool tray is now synchronized! Let's double check that we have the new and proper time:
C:\> time /t
06:13 AM
We've now synced with a remote NTP server at the command line. Yummy.

Also, there are a lot of other fascinating time-related options available with w32tm. For instance, to display your timezone, you could run:
c:\> w32tm /tz
Time zone: Current:TIME_ZONE_ID_STANDARD Bias: 300min (UTC=LocalTime+Bias)
[Standard Name:"Eastern Standard Time" Bias:0min Date:(M:11 D:1 DoW:0)]
[Daylight Name:"Eastern Daylight Time" Bias:-60min Date:(M:3 D:2 DoW:0)]
We can even convert an NT system time (the number of 100 nanosecond intervals that have elapsed since Jan 1st, 1601, the date DOS was first written by monks) to a human readable date and time:
c:\> w32tm /ntte 128904420852883740
149194 22:21:25.2883740 - 6/25/2009 5:21:25 PM
This is especially useful if you have a Vista box (my condolences) and are researching the installation of patches using "wmic qfe". In XP Pro and Windows 7, "wmic qfe list full" returns a list of patches with an "InstalledOn" field in a human readable form. But, in Vista, this date is now a hexadecimal number of an NT system time. Convert that hex number into decimal using you favorite calculator, and then fire up w32tm /ntte to convert it into a date you can read. Also, the "w32tm /ntpte" command converts NTP times (seconds from Jan 1st, 1900, the date that Nicola Tesla implemented the first Unix kernel) into a human readable format.

But, wait, there's more! Hal showed how to add a timestamp into the output of your commands, useful in forensics. I sometimes will put a date and time into my command prompt itself, so that I can then copy and paste my session to use as evidence:
C:\> prompt $D$T$S$P$G
Sat 10/31/2009 6:15:29.97 C:\>
So, there's a bunch of time-related stuff. Have you got any other fun time kickers, Tim?

Tim clocks in:

Let's take a look at the cmdlets available that use the Time or Date nouns.

PS C:\> Get-Command -Noun date
CommandType Name Definition
----------- ---- ----------
Cmdlet Get-Date Get-Date [[-Date] ...
Cmdlet Set-Date Set-Date [-Date] ...

PS C:\> Get-Command -Noun time
[nothing]


Unfortunately, there aren't any built-in cmdlets for NTP. So to answer Ed's question, "No, no interesting kickers this week." But we can still take a look at the Get-Date cmdlet and its properties.

PS C:\> Get-Date | fl

DisplayHint : DateTime
DateTime : Sunday, November 01, 2009 10:17:23 PM
Date : 11/1/2009 12:00:00 AM
Day : 1
DayOfWeek : Sunday
DayOfYear : 305
Hour : 22
Kind : Local
Millisecond : 206
Minute : 17
Month : 11
Second : 23
Ticks : 633927106432066338
TimeOfDay : 22:17:23.2066338
Year : 2009


The object has a number of methods available, and they can be seen by piping the object into Get-Member. The results aren't shown here since it is a pretty long list. Since the reason for this episode is Daylight Savings Time we'll take a quick peek at the relevant method.

PS C:\> (Get-Date).IsDaylightSavingTime()
False


Ok, that wasn't spectacular, but let's carry on and recreate Ed's command prompt with the date.

PS C:\> function prompt {"PS $(Get-Date) $(Get-Location)>"}
PS 11/01/2009 22:43:53 C:\>


Since the prompt is defined by overloading the prompt function there are endless possibilities for setting a custom prompt.

That's all folks, hope to see you next time.