Tuesday, November 24, 2009

Episode #70: The Tangled Web

Hal gets a soft one this week

Lately we've had some of our loyal readers-- mostly the Windows folk-- asking about command-line tools for accessing web pages. When these questions come up, I just smile serenely, because it's easy to do this in Unix. Ed and Tim on the other hand turn a little green and start sweating.

Among the Unix tribes, there seem to be two popular command-line tools for grabbing web pages. Some folks prefer curl, but I generally stick with wget since it's the tool I learned first. Both curl and wget support HTTP, HTTPS, and FTP as well as proxies of various types along with different forms of authentication. curl also supports other protocols like TFTP, FTPS (FTP over SSL), and SCP and SFTP.

Using wget couldn't be simpler:

$ wget blog.commandlinekungfu.com
--2009-11-18 18:47:51-- http://blog.commandlinekungfu.com/
Resolving blog.commandlinekungfu.com... 74.125.113.121
Connecting to blog.commandlinekungfu.com|74.125.113.121|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: `index.html'

[ <=> ] 173,805 18.3K/s in 9.3s

2009-11-18 18:48:01 (18.3 KB/s) - `index.html' saved [173805]


Or, if you don't like all the noisy chatter, you can use the "-q" option:

$ wget -q blog.commandlinekungfu.com
$ ls
index.html index.html.1

Notice that wget doesn't overwrite the first index.html file we downloaded. Instead it appends a uniqe number to the file name. If we downloaded another index.html file, it would end up as index.html.2 and so on. You can also use "-O" to specify an output file name.

Maybe my favorite feature, however, is the "-r" (recursive) option that allows me to grab an entire tree of content in a single operation. So all I have to do when I want to back up the content here at Command Line Kung Fu is just run "wget -r -q blog.commandlinekungfu.com" and wait a few minutes for everything to download.

There are lots of other neat features to wget, but I think I've already demoralized my Windows brethren enough for one Episode. Let's see what Ed and Tim cook up.

Tim orders take-out:
Unfortunately (again), there isn't a built-in cmdlet to do the equivalent of wget or curl, but we can access the .NET libraries and recreate some of the functionality of the Linux commands. By default, the .NET Framework supports URIs that begin with http:, https:, ftp:, and file: scheme identifiers, so it isn't quite as full featured as Linux, but it is all we have.

PS C:\> (New-Object System.Net.WebClient).DownloadString(
"http://blog.commandlinekungfu.com")


<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
...


This will grab files in text format and it can be used further down the pipeline, such as saving the file by piping it in to Out-File. What if we want to grab non-text files or just download the file? We can use the DownloadFile method and specify where we want to save the file.

PS C:\> (New-Object System.Net.WebClient).DownloadFile(
"http://blog.commandlinekungfu.com","c:\downloadedfile.html")


What happens if the file doesn't exist? It raises an 404 error and the file (obviously) isn't opened.

PS C:\> (New-Object System.Net.WebClient).DownloadString(
"http://blog.commandlinekungfu.com/NonExistant.file")


Exception calling "DownloadString" with "1" argument(s): "The remote
server returned an error: (404) Not Found."
At line:1 char:49
+ (New-Object System.Net.WebClient).DownloadString( <<<<
"http://blog.commandlinekungfu.com/NonExistant.file")


Sadly, there isn't way to perform recursive requests without doing some heavy duty scripting.

Ed Sits Here Trying Not To Think About What Hal Called His Part of this Article:

Ahhh... an age-old question. I remember about a year ago, I was having a discussion with Kevin Johnson, bon vivant and web attacker extraordinaire, when this issue came up. I posed the following question to KJ0 (as we call him): "Suppose you've just hacked a Windows box in a pen test. You have command shell access of said machine. Tell me what you want to do now, using only built in command-line capabilities." I went further, boldly mixing my metaphors: "Consider me as /dev/WindowsCommandLine... where do you want to go today?"

Kevin shrugged, smiled, and said, "Easy... wget."

I felt a piercing pain stab at my heart. "The wget tool has sooo many options... can we scale it back a bit? What do you really want to do?" Kevin said, "Sure... I just wanna fetch a web page and write it to the file system. How can I do that at the command line in Windows?"

I spent a little time fitzing around with various Windows commands, and ultimately settled on using the built-in Windows telnet client to formulate HTTP requests manually. It's crude, but works.

"But, Ed," you argue, "Microsoft removed the telnet client from Windows Vista and some of the 6 million versions of Windows 7." I respond: Yes, it's no longer installed by default, but on the professional versions of Vista and 7, you can run:

C:\> start /w pkgmgr /iu:"TelnetClient"


Even if it doesn't finish, kill that cmd.exe, and you should have the Telnet Client ready to run. Note that the installation package for the telnet client is built-in, but the tool itself just isn't installed. The machine doesn't have to reach out to the Internet to get it. Also, note that /iu stands for install update, you need that :, and you should observe the case of the Cap-T and Cap-C in TelnetClient. Oh, and by the way... if you want the Telnet service, change "TelnetClient" to "TelnetServer". To remove either after you've installed, it, run the same command, except substitute /uu: (uninstall update) for /iu:.

So, now equipped with the telnet client, let's make an HTTP request.

Now, you might think that we should simply echo an HTTP request and pipe it into the telnet client, right? Bzzzzzt. Sorry, my friend, but the Windows telnet client doesn't like anything to come into its Standard Input at the command line, and it certainly doesn't send it to the target machine. Try sniffing it sometime. Another annoying part about the Windows telnet client is that it doesn't like you to touch its Standard Output. The telnet client is somehow clairvoyant, and if you follow it with a > or a |, it knows and refuses to send any packets. Gee, thanks, Microsoft.

For this one, we're going to have to go interactive.

C:\> telnet -f log.txt

I'm logging output to the file log.txt, where our output will reside.

At our new telnet prompt, we can open a connection to port 80 on our destination web server. We'll use the "o" option to open a connection:

Microsoft Telnet> o www.google.com 80
Connecting To www.google.com...

Now, depending on the version of Windows you are using, it may or may not look like a connection is made. Either way, just trust that it has, and start typing your HTTP request. Typically, you'll be getting a page, which you can do by entering:

GET [pagename] HTTP/1.1
Host: [TheHostName]

Hit Enter Enter at the end, and you'll have the page displayed on your screen. Hit Enter again, and you'll get back to your telnet prompt. Type "quit" to exit.

More importantly, you should now have your page in the output file log.txt. Yes, you'll have the HTTP response header in front of it, but that's not so bad. It's ugly, but it'll let you grab a file quickly from a server.

Tuesday, November 17, 2009

Episode #69: Destroy All Connections

Ed looks out on the serene waters of Tokyo Bay:

Mr. Byte Bucket sent in a request from the ever insightful Pauldotcom IRC channel:

Can anyone suggest a Windows cmd to disconnect a specific socket?

Nice question! Unfortunately, Windows doesn't offer much in the way of built-in tools that are fine grained enough to operate at the socket level. But, we can either restart the service handling the connection, or, if you want to be a little more violent, just kill its service.

We'll start by taking the violent rout. First off, we need to figure out what processid is associated with the connection you seek. You can get a list of all connections to a given port using the following command:

C:\> netstat -nao | find ":[port]"

The right-most (i.e., fifth) column is the processid number.

If you have multiple different clients clients connected to that same destination port, you can select out the given process that is associated with a specific client connection using:

C:\> netstat -nao | find ":[port]" | find "[ClientIPaddr]"

You can then kill that process using wmic. However, be very careful! That process may be really important, and killing it could end you up in a world of hurt. In fact, a lot of Windows built-in features (such as file sharing and IIS) are all associated with the "System" service (with a PID of 4 or 8 depending on the version of Windows you are using).

C:\> wmic process where processid="[PID]" delete

You can wrap this all up in one very dangerous command... but I really don't recommend doing this. Again, if you inadvertently kill the wrong process, your system could come crashing down around you. I recommend figuring out what process is at issue first, investigating it, and only then killing it.

C:\> for /f "tokens=5" %i in ('netstat -nao ^| find ":[port]" ^| 
find "[ClientIPaddr]"') do @wmic process where processid="%i" delete
Deleting instance \\WILMA\ROOT\CIMV2:Win32_Process.Handle="[PID]"
Instance deletion successful.

Our second approach involves restarting the service associated with the process. Based on the processID information we retrieved above, we can get a list of the associated services by running:

C:\> tasklist /fi "pid eq [PID]" /svc

That command tells us all of the services that are associated with a given process. We'll then have to do a little research to figure out which specific service it is that is handling our given connection. Unfortunately, Windows doesn't give us the ability to map a connection directly to a service, but we must instead map a connection to a process, which we then map to a set of services running from within that process. Now, if there's only one service in that process, we're golden. But, if there are more, we have to do this research step to find out which service it really is that we need to focus on.

Once you've discovered the appropriate service, you can then restart the service using:

C:\> sc stop [service] & sc start [service]

It's kind of annoying, I think, that there is no "restart" option explicitly here, so we have to stop the service and then start it. Not a huge liability, but still a little frustrating.

Tim Sees Something Starting to Stir in the Otherwise Tranquil Waters of the Bay:

PowerShell doesn't (yet) have any cmdlets similar to netstat. The Grand Poobah of PowerShell (@halr9000 on twitter) has a PowerShell script to "objectize" netstat, but that crosses in to scriptland so we will steer clear.

To find the connection and its associated process you will have to refer to the first portion of Ed's section. But once we have the process id we can use PowerShell cmdlets.

So we have the PID, now we can get the process object by using the aptly named command Get-Process. The default method for Get-Process is the process name, so we need to use the Id parameter.

PS C:\> Get-Process -Id 3004
Handles NPM(K) PM(K) WS(K) VM(M) CPU(s) Id ProcessName
------- ------ ----- ----- ----- ------ -- -----------
210 7 5116 13344 61 6.67 3004 Evil


The process can be killed using PowerShell, but as Ed said, "Be very careful!" The Stop-Process cmdlet's default method is the process Id so we aren't required to use Id parameter (but you can if you wish).

PS C:\> Stop-Process 3004


If the offending process is a service, we can't retrieve the service from the process id using by using just Get-Proccess since it doesn't include the Id property. However, we can use wmi in conjunction with Get-Service to "get" the service.

PS C:\> Get-WmiObject win32_service | ? { $_.ProcessID -eq 3004 } | % { Get-Serv
ice $_.Name }


To paraphrase Ed (again), you will have to do some research to since there can be multiple services running from within that process (multiple services be in one process).

Once we find the service that needs a kick, we can stop it using Stop-Service. We also have the ability to restart the service using Restart-Service.

If you felt gutsy you could pipe the command above into Restart-Service or Stop-Service.

PS C:\> Get-WmiObject win32_service | ? { $_.ProcessID -eq 3004 } | % { Get-Serv
ice $_.Name } | Restart-Service


PS C:\> Get-WmiObject win32_service | ? { $_.ProcessID -eq 3004 } | % { Get-Serv
ice $_.Name } | Stop-Service


..or you could do it manually.

PS C:\> Stop-Service [Service]


PS C:\> Restart-Service [Service]


It would be nice if there was a facility in PowerShell to just kill a connection. Maybe we can get that in version 3, but while we wait for the additional capability I get the uneasy feeling Hal is getting ready to squash us like bugs.

And the waters of Tokyo bay begin to boil:

Oh dear. I really am going to have to open my Godzilla-sized can of whup-ass on my poor Windows-using bretheren. But first, let me try to make them feel not so bad by doing a quick netstat-based solution.

On Linux systems at least, netstat has a "-p" option to display process information. Let's take a quick look at some sample output:

# netstat -anp --tcp -4 | grep :22
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 15805/sshd
tcp 0 0 192.168.1.4:60279 192.168.1.2:22 ESTABLISHED 18054/ssh
tcp 0 0 192.168.1.4:32921 192.168.1.2:22 ESTABLISHED 19409/ssh

Here I'm listing all TCP sockets that are using IPv4 ("--tcp -4"), and using "-n" so that the socket numbers are not converted into human-readable names. Anyway, as you can see, the 7th column is "PID/name" (this is what the "-p" option does).

So with the help of awk and cut, we can pull out just the PID of the master SSH daemon:

# netstat -anp --tcp -4 | awk '/:22/ && /LISTEN/ { print $7 }' | cut -f1 -d/
15805

Killing that process just requires appropriate use of backticks:

# kill `netstat -anp --tcp -4 | awk '/:22/ && /LISTEN/ { print $7 }' | cut -f1 -d/`

We could also use cut to pull out the second field, if we wanted to shut down the process using it's init script:

# /etc/init.d/`netstat -anp --tcp -4 | awk '/:22/ && /LISTEN/ { print $7 }' | cut -f2 -d/` stop
bash: /etc/init.d/sshd: No such file or directory

Rats, it's /etc/init.d/ssh on this system, and not /etc/init.d/sshd, but you get the idea.

But really, the 300 foot giant mutated atomic lizard we want to employ here is lsof. You might be aware that we can get lsof to show us just the port 22 stuff with it's "-i" flag:

# lsof -i :22
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
sshd 15805 root 3u IPv6 37028 TCP *:ssh (LISTEN)
sshd 15805 root 4u IPv4 37030 TCP *:ssh (LISTEN)
ssh 18054 hal 3u IPv4 44514 TCP elk.deer-run.com:60279->deer.deer-run.com:ssh (ESTABLISHED)
ssh 19409 hal 3u IPv4 53249 TCP elk.deer-run.com:32921->deer.deer-run.com:ssh (ESTABLISHED)

If we wanted to kill the SSH daemon, we could easily adapt the awk fu from the netstat example to pull out the appropriate PID value and then use backticks to feed this value into the kill command.

But we don't need awk, because lsof has the "-t" ("terse") option, which just spits out the PIDs:

# lsof -t -i :22
15805
18054
19409

The "-t" option is specifically designed so that you can do things like "kill `lsof -t -i :22`". Of course, the problem with doing that is that I'd also end up killing my SSH sessions to the remote machine deer.deer-run.com-- PIDs 18054 and 19409-- which I don't really want to do.

So how do we pull out just the daemon PID? Well looking at the lsof output above, I can see that I want to kill the sshd processes that are listening on port 22 but leave the regular ssh processes alone. I could use the "-c" option to select the PIDs by command name:

# lsof -t -c sshd
15805

But that means I'd have to already know the command name associated with the port in question-- in this case by having already run lsof once to search by port number. And if I knew the command name already, why wouldn't I just use pkill (see Episode #22) instead of lsof?

What I really want is a single command-line using just kill and lsof that lets me reliably destroy the master server process as effectively as Godzilla destroys Tokyo. Luckily, lsof has some more atom-age madness up its sleeve. You see, lsof's "-c" option isn't just limited to simple substring matching: you can use "/.../" to specify a full-on egrep-style regular expression. Because Unix server processes almost always end in "d" (think sshd, httpd, ftpd, and so on), we can construct a similar regular expression to match the daemon process name associated with an arbitrary port number:

# lsof -a -i :22 -c /d$/
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
sshd 15805 root 3u IPv6 37028 TCP *:ssh (LISTEN)
sshd 15805 root 4u IPv4 37030 TCP *:ssh (LISTEN)

The "-a" option means to logically "and" your search criteria together (the default is logical "or" for some strange reason, although this is almost never what you want). So here we're looking for everything using port 22 and where the process name ends with "d".

Adding the "-t" option and some backtick action, we have our final answer:

# kill `lsof -t -a -i :22 -c /d$/`

And with that, I shall swim home to Monster Island and await my next challenge.

Tuesday, November 10, 2009

Episode #68: Help Fu

Tim hits lead off:

This week we get help with help. Everyone needs help, and some in different ways than others. But we aren't going to discuss Ed's problem with crunchy peanut butter or Hal's issue with furry animals with tails. Anyway, back to the issue at hand.

Instead of jumping right in to getting help on commands, let's use PowerShell to find commands by using the Get-Command cmdlet. This command will retrieve all the cmdlets, aliases, and functions available to us. In order to use this command effectively, let's go over a bit of the very basics of PowerShell. As you all probably know, the nomenclature for PowerShell cmdlets is verb-noun. Microsoft's approved verb list defines the verbs so you know which verb to use and to ensure consistent use by cmdlet developers. That way the verbs are consistent so "get" is always used instead of a random choice of get, retrieve, grab, or obtain. The noun part of the name identifies the entity on which the action is performed.

Ed and Hal discussed the command history in episode #27, so we'll use that as the focus of our research.

Let's see a list of all the available nouns.

PS C:\> Get-Command -Type cmdlet | Sort-Object noun | Group-Object noun

Count Name Group
----- ---- -----
2 Acl {Set-Acl, Get-Acl}
5 Alias {Import-Alias, Get-Alias, Set-Alias, Export-...
...
4 History {Invoke-History, Clear-History, Get-History,...
...


You probably could have already guessed, but the noun we want to use is "History." To get a lits of the commands available using that noun we can use this command:

PS C:\> Get-Command -Noun history

CommandType Name Definition
----------- ---- ----------
Cmdlet Add-History Add-History [[-InputObject] ...
Cmdlet Clear-History Clear-History [[-Id] <Int32[...
Cmdlet Get-History Get-History [[-Id] <Int64[]>...
Cmdlet Invoke-History Invoke-History [[-Id] <Strin...


The parameter is not case sensitive and will accept wildcards so "hIST*" would returned similar results. Similarly, we could see all the available commands using the "get" verb using this command.

PS C:\> Get-Command -Verb Get


Back to the subject at hand...

Now we have a list of cmdlets we are interested in. Let's see how we can use the command by using the Get-Help cmdlet.

PS C:\> Get-Help Get-History

NAME
Get-History

SYNOPSIS
Gets a list of the commands entered during the current session.

SYNTAX
Get-History [[-Id] ] [[-Count] ] []
...


We have a bit of a problem here, the basic help isn't super useful since it doesn't provide a good description of the parameters or any examples of the cmdlet's use. However, it does provide syntax and a list of parameters. To see the full help including examples and parameter descriptions use this command:

PS C:\> Get-Help Get-History -Full


To see just the examples:

PS C:\> Get-Help Get-History -Examples


As you have probably noticed in past episodes there are a lot of aliases available in PowerShell. The Get-Alias cmdlet can be used to "get" the list of aliases. Specifically, we can see the aliases for the commands using the History noun.

PS C:\> Get-Alias | Where-Object {$_.Definition -like "*history"}

CommandType Name Definition
----------- ---- ----------
Alias clhy Clear-History
Alias ghy Get-History
Alias h Get-History
Alias history Get-History
Alias ihy Invoke-History
Alias r Invoke-History


The Get-Command cmdlet can be used to get the same results.

PS C:\> Get-Command -Type Alias | Where-Object {$_.Definition -like "*history"}


That's about it for PowerShell, let's see what Ed has for us with the Windows command line: Get-Help -FuMaster Ed -Subject Windows

Ed's Output:

You guys with your fancy help capabilities in your shells are a real hoot. Yes, your shells include intuitive and easy-to-access help features. Where’s the fun in that? Or, more to the point, where’s the fu in that? I’ve spent many countless hours sitting at a cmd.exe trying to reverse engineer… uh… I mean conducting detailed design recovery of cmd.exe functionality to try to figure out what the heck the little bugger was really doing. Ah… good times.

In all seriousness though, cmd.exe offers an embarrassingly paltry amount of help for using it and its commands. Really, it’s kind of stinky. Let’s explore the few skimpy options we have.

First off, we can run the help command, followed by the command we want to analyze, as in:
C:\> help findstr

The good news? Well, we’ll see a brief description of findstr functionality, as well as its command flags.

The bad news? Well, where to start? How about:

Bad News #1) The help command doesn’t provide help for all of the commands available in cmd.exe. It only covers a little less than 80 of them.

To get a list of the commands that help has information about, simply run:

C:\> help

Some of the most interesting and useful commands of all, like netstat, netsh, reg, sc, and wmic, result in this unhappy little message:
C:\> help reg
This command is not supported by the help utility. Try "x /?".

Bad News #2) The output of the help command doesn’t include any examples or much detail at all. It’s just really basic usage, without any interesting fu included.

Bad News #3) Note the little suggestion in the output of my command in Bad News #1: Windows tells us to try running the command with a /? argument for help. As an old Unix guy (shhhh… don’t tell anyone), I’m very hesitant to run a command that I don’t understand just so I can get help. It scares me to do so without knowing what just running the command might do. And, here’s a dirty little secret of the help command itself. It actually runs the command you are trying to get help about! Yes, check this out…

First, start Task Manager, move to the Processes tab, and sort it alphabetically by clicking on the Image Name column.

Now, in a separate cmd.exe, run the following command, which will get help for the findstr command, repeatedly:

C:\> for /L %i in (1,0,2) do @help findstr

Look in your Task Manager window… see it? Every once in a while, the findstr command pops up! So “help findstr” is actually invoking the findstr command itself to scrape out its help options. Yowza. That’s just painful for me.

Bad News #4: Well, with only around 80 commands covered by the help command, we often have to turn to that old stand by, “[command] /?” for more help. But the help included here is spotty at best as well, with major inconsistencies between commands and some commands supporting context-specific help for particular sub-areas of the command. For example, “wmic /?” does offer some help, but there is a wmic-specific option for getting more detailed help:

C:\> wmic /?:full

Besides wmic, none of the other commands I know about support this /?:full option.

Also, some commands have specific help for certain contexts within the command. Consider the following examples:

C:\> wmic process get /?

This shows you a list of all the process attributes wmic supports.

Or, try this:

C:\> wmic process call /?

This one shows you a list of all the process methods you can call from within wmic.

Or, how about this one:

C:\> net /?

And then:

C:\> net use /?

Likewise, we can run “reg /?” to get an overview of the reg command, followed by a “reg query /?” to get more details about the specific syntax for querying the registry.

Want another bewildering inconsistency? Try running this:

C:\> netstat /?

Note that the output actually shows the command flags with dashes instead of slashes (i.e., “netstat –na” instead of “netstat /na”). Thankfully, the actual netstat command on modern Windows boxes lets you use dashes or slashes in its flags.

So, how can we wrap our heads around all of this inconsistency? I’ve found that mastering Linux involves learning certain general principles and then seeing how they are applied throughout the operating system. Whereas, in Windows, mastering the command line involves memorizing a bunch of complex, counter intuitive, and often inconsistent options scattered throughout the operating system without rhyme or reason. So, why bother? Because, once mastered, the Windows command-line is incredibly useful in analyzing systems at a fine-grained level. Also, cmd.exe is pretty much everywhere, pre-installed, so there’s a lot less hit-and-miss than you get with installs of PowerShell. At least, there is for now… this is changing as PowerShell gets more widely deployed.

The bottom line here? When I’m actually looking for help with a command, I apply the following process. I start by trying “help [command]”. If that provides no satisfaction, I proceed to “[command] /?”. If that doesn’t give me what I want, I try to look for context-specific help with “[command] [option] /?”. And, if that doesn’t get me where I need to go, I turn to the web to research commands. One of my absolutely favorite sites for researching Windows (as well as Linux and Unix) commands is the awesome ss64 site. It includes a detailed list of most Windows commands, including their various flags, example usage, and mapping to rough equivalents in Linux, OS X (sometimes), and PowerShell (occasionally). That’s awesome. And, finally, there’s Microsoft’s own command line reference, worth a check from time to time.

Hal cleans up:

There are actually a number of different help systems available on a typical Unix system. First there's the standard on-line manual pages accessed via the "man" command. The Unix manual pages include not only information about programs that you can run from the command line, but also documentation on programming APIs, device interfaces, and even the format of important configuration files on the system.

The trick sometimes is getting to the right manual page. For example, suppose I wanted some information on how to use the chmod() system call in my C program. If I just typed "man chmod", I'd get the manual page for the chmod program and not the documentation on the system call. To distinguish these two different manual pages, the Unix manual has traditionally been organized into sections. Sections 1 (user commands) and 8 (administrator commands) are devoted to command-line tools, while sections 2 (system calls) and 3 (library routines) are devoted to programming APIs. FYI, section 4 is device interfaces, section 5 is configuration file formats, section 6 is games, and section 7 is "miscellaneous" but includes useful tidbits like the table of ASCII values ("man ascii") and additional details on various IPC mechanisms. When a Unix document refers to "chmod(2)" it means "the documentation on the chmod system call in section 2 of the manual".

But how does on pull up the chmod(2) manual page instead of the default chmod(1) page? The man command takes a "-s" option to specify the section: "man -s 2 chmod". But on many Unix variants you can simply drop the "-s" and just type "man 2 chmod".

Suppose, however, that I didn't know which section the chmod() system call was documented in. One of the most useful features of the man command is the "-k" (keyword search) option:

$ man -k chmod
chmod (1) - change file mode bits
chmod (2) - change permissions of a file
fchmod (2) - change permissions of a file
fchmodat (2) - change permissions of a file relative to a directory f...

You don't have to search on command names, of course. Any related keyword will work. For example, if you're new to Unix and don't know how to rename files:

$ man -k rename
dpkg-name (1) - rename Debian packages to full package names
lvrename (8) - rename a logical volume
mmove (1) - move or rename an MSDOS file or subdirectory
mren (1) - rename an existing MSDOS file
mv (1) - move (rename) files
prename (1) - renames multiple files
rename (1) - renames multiple files
rename (2) - change the name or location of a file
rename.ul (1) - Rename files
renameat (2) - rename a file relative to directory file descriptors
vgrename (8) - rename a volume group

By the way, the "apropos" and "whatis" commands are equivalent to "man -k". However, all of these commands operate on special "databases" of information that has been extracted from the manual pages themselves via the "makewhatis" command (aka "mandb" on Linux). Very often your system will have a cron job that builds the whatis databases automatically every week or so, but on some Unix systems you have to build the databases manually (or create your own cron job).

Another tricky issue with the Unix on-line manual is that sometimes you have multiple repositories of manual pages. For example, you might have the standard OS manual pages under /usr/share/man and manual pages for third-party software in /usr/local/man. The man command lets you use the "-M" option to specify a path to use for finding manual pages, but far it's far easier to set MANPATH in your .bashrc file:

export MANPATH=/usr/share/man:/usr/local/man

MANPATH works just like PATH or LD_LIBRARY_PATH. In the above example, the man command will check the system manual page directory first and then look in /usr/local/man if it doesn't find the manual page you're requesting.

The man command is also the source of one of the most famous jokes in Unix:

$ man 'furry animals with tails'
No manual entry for furry animals with tails

OK, so the traditional joke is "man rear", but you get the idea.

Sadly, man pages seem to be dieing off. A lot of Open Source projects have either very scanty or completely non-existent manual pages. It's actually pretty tough to write a good manual page, both in terms of content and also because you need to know nroff/troff formatting commands to format the text. But writing (and reading) manual pages properly is still an important Unix skill IMHO.

Another source of on-line documentation on Unix is the Free Software Foundation's "info" system. The most complete documentation on packages like emacs, gcc, GNU tar, et al is found in the info system-- the manual pages are mostly just stubs extracted from the docs in the info pages. Run the command "info info" to get more information on how to navigate the curses-based info interface. Frankly, I think the user interface for the info system is terrible, but it was mostly designed to be run from within emacs. I suppose that it makes sense if you're one of those people who essentially lives inside of emacs and never touches an actual terminal window.

But the info system never really caught on outside of the FSF. With traditional manual pages being marginalized as well, the main documentation interface these days seems to be the "-h" or "--help" options supported by many commands. Typically, one or both of these options will generate at least a terse summary of command options:

$ info --help
Usage: info [OPTION]... [MENU-ITEM...]

Read documentation in Info format.

Options:
--apropos=STRING look up STRING in all indices of all manuals.
-d, --directory=DIR add DIR to INFOPATH.
--dribble=FILENAME remember user keystrokes in FILENAME.
-f, --file=FILENAME specify Info file to visit.
-h, --help display this help and exit.
--index-search=STRING go to node pointed by index entry STRING.
-n, --node=NODENAME specify nodes in first visited Info file.
-o, --output=FILENAME output selected nodes to FILENAME.
-R, --raw-escapes output "raw" ANSI escapes (default).
--no-raw-escapes output escapes as literal text.
--restore=FILENAME read initial keystrokes from FILENAME.
-O, --show-options, --usage go to command-line options node.
--subnodes recursively output menu items.
-w, --where, --location print physical location of Info file.
--vi-keys use vi-like and less-like key bindings.
--version display version information and exit.

The first non-option argument, if present, is the menu entry to start from;
it is searched for in all `dir' files along INFOPATH.
If it is not present, info merges all `dir' files and shows the result.
Any remaining arguments are treated as the names of menu
items relative to the initial node visited.

Examples:
info show top-level dir menu
info emacs start at emacs node from top-level dir
info emacs buffers start at buffers node within emacs manual
info --show-options emacs start at node with emacs' command line options
info --subnodes -o out.txt emacs dump entire manual to out.txt
info -f ./foo.info show file ./foo.info, not searching dir

Email bug reports to bug-texinfo@gnu.org,
general questions and discussion to help-texinfo@gnu.org.
Texinfo home page: http://www.gnu.org/software/texinfo/

Not as useful as a traditional manual page, IMHO, but at least it's something. The biggest problem is that with the documentation being tied up in each individual command's "--help" output, there's no "man -k" equivalent for doing global keyword searches if you're not sure what command you're looking for.

Our friend Jeff Haemer is at it again, with some more "helpful" pointers on his blog. Man, I can't believe I forgot the "help" command, but Jeff will tell you all about it, plus some other neat shell trickery.

Tuesday, November 3, 2009

Episode #67: Time Lords

Hal is still adjusting:

Mr. Bucket suggested a time-related Episode this week in honor of the shift back to Standard Time here in the U.S. Generally, we try to humor Mr. Bucket as long as his requests aren't too deviant (or if Ed is the only one who has to dress up), so here goes.

First I've got a couple of tricks up my sleeve related to the NTP time synchronization service. My favorite command-line tool for checking up on my NTP servers is "ntpdc -p":

$ ntpdc -pn server.example.com
remote local st poll reach delay offset disp
=======================================================================
=216.14.97.75 192.168.1.2 16 1024 0 0.00000 0.000000 0.00000
=173.45.227.99 192.168.1.2 2 1024 377 0.06984 -0.003889 0.13681
*64.73.32.135 192.168.1.2 2 1024 377 0.09087 0.002323 0.12178
=127.127.1.1 127.0.0.1 5 64 377 0.00000 0.000000 0.03093

The "st" column that shows the remote server's stratum level. This is useful for detecting downed time servers, because they show up as stratum 16 like on the first line above. The "poll" (polling interval in seconds) and "reach" (displayed in octal notation and capped at 255, the number of recent successful polls) columns give you and idea of how well your server is synchronizing with other time servers. "delay" tells you how far away the remote server is from you, and "offset" is the difference between your local clock and the clock on the remote server (if things are working right, the offset should be less than a hundredth of a second). Note that in the above example, I checked on the remote server.example.com machine, but you can leave off the host name portion to check the time server running on your local box.

In addition to checking status with ntpdc, you can also set the time on your box from the command line using ntpdate. Normally you would just run ntpdate and specify the server you wanted to synchronize against, but here's a cute little bit of awk fu for pulling the server names out of your current ntp.conf file:

# ntpdate `awk '/^(server|peer) / && !/127.127./ {print $2}' /etc/ntp.conf`
29 Oct 19:42:43 ntpdate[13376]: adjust time server 66.187.233.4 offset -0.004818 sec

Here I'm pulling off the host name or IP address specified on the "server" or "peer" lines in your ntp.conf file. I'm skipping any "addresses" that start with "127.127." since these are merely place holders for clock driver definitions and not real IP addresses.

Note that you can only run ntpdate if your NTP daemon is not currently active. If ntpd is running at the same time you execute ntpdate, you get a message like this:

# ntpdate `awk '/^(server|peer) / && !/127.127/ {print $2}' /etc/ntp.conf`
29 Oct 19:42:18 ntpdate[13362]: the NTP socket is in use, exiting

Typically you only need to call NTP date by hand if the NTP daemon is not running and your clock has gotten out of synch. After you use ntpdate to jump your clock to something approximating the right time, you should start your NTP daemon to keep it synced (usually "/etc/init.d/ntpd start").

Getting away from NTP now, let me show you a little environment variable hack that's particularly useful for Forensic Analysts. Suppose you've captured a disk image from a system that was being used in a time zone other than the one your analysis machine is set to. You'd like to mount a copy of the image and be able to observe the timestamps on the files relative to the time zone the image was taken from. You could change the global time zone setting on your analysis workstation, but that could cause you lots of hassle.

It turns out that bash allows you to set the TZ environment variable in your shell to alter the local time zone setting for only that shell. For example, my local time zone is US/Pacific, but suppose I had an image from a machine from the East Coast:

$ date
Thu Oct 29 17:59:13 PDT 2009
$ ls -ld /mnt/image/tmp
drwxrwxrwt 23 root root 4096 Oct 29 17:28 /mnt/image/tmp
$ export TZ=US/Eastern
$ date
Thu Oct 29 21:00:07 EDT 2009
$ ls -ld /mnt/image/tmp
drwxrwxrwt 23 root root 4096 Oct 29 20:28 /mnt/image/tmp

Cool! Now I can look at the dates on files in my image and not constantly have to be doing mental arithmetic to convert the time stamps.

So there are three of my favorite time-related hacks. If we can get Ed out of those leather chaps that Mr. Bucket had him wear, I'm sure he'll have some cool Windows Fu to show us...

Ed chimes in:

Ahhh... time. A great idea for an article, Mr. Bucket. Much better than your last idea... you know... the one about the chaps. Anyway, I digress.

Let's walk before we run. To check the time on your local computer, you can simply run the following command:
C:\> time /t
06:05 AM
Want more precision? We can get that, as discussed in Episode #49, by displaying the %time% environment variable.
C:\> echo %time%
6:05:22.75
Alternatively, the "net time" command can be used to pull the time from a remote (or even local) Windows box:
C:\> net time \\[computername]
Current time at \\FRED2 is 10/31/2009 6:04 AM
The command completed successfully.
Note that, depending on whether you are a member of a domain and the privileges of your current account, you may need to first initiate an SMB connection with that machine, by running:
C:\> net use \\[computername] [password] /u:[username]
The command completed successfully.
If you happen to trust that server, you can set your local machine's time by its clock through running:
C:\> net time \\[computername] /set /yes
Well, that's all well and good, but Hal was working with NTP, which offers many more time sources than, well, our local host or domain controller. How can we pull time from an NTP server in Windows? First, please note that officially, with built-in capabilities, Windows relies on the Simple Network Time Protocol (SNTP), a stripped down NTP implementation that is often used in applications where time accuracy isn't as vital. That's why you'll see sntp throughout our commands below.

Let's first look to see which NTP server our Windows machine is configured to use:
C:\> net time /querysntp
The current SNTP value is: time.windows.com,0x1
The command completed successfully.
The time.windows.com server is the default. Do you trust Microsoft to accurately track the time on their server? After missing the release date of Windows Vista by... uh... something like 3 years, maybe we shouldn't (darnit... I couldn't resist pouring salt in those old wounds.) We can change the NTP server we're configured to use as follows:
C:\> net time /setsntp:pool.ntp.org
The command completed successfully.
Note that you can put a whole list of NTP servers there by following the colon with a space-separated list of NTP servers enclosed in double quotes, resulting in something like /setsntp:"pool.ntp.org ntp.colby.edu tick.gatech.edu"

OK... now, once we've set ourselves up to use an NTP server, let's try to check the time:
C:\> net time
Could not locate a time-server.

More help is available by typing NET HELPMSG 3912.
What? Why doesn't "net time" work here? Note that the HELPMSG is not at all helpful, as it explains that we could not locate a time-server, which we already saw in the error message itself. Gee, Microsoft... thanks for nothing.

It turns out that we can't pull time via "net time" using NTP (really SNTP) unless we're a domain controller with the Flexible Single Master Operation (FSMO) role. Lovely. But, what if we're just a lowly client, maybe not even in a domain? Can we sync with an NTP server using only built in tools? As Bob the Builder and Barack Obama might say: Yes, we can!

We can rely on that happy old stand-by for configuring time on Windows boxen, the intuitively named w32tm command. And in homage to Heinz Doofenshmirtz, by "intuitively named", I mean, of course, "completely non-intuitively named."

To sync with an NTP server, we start by configuring our Windows box (whether it's a client or server version of Windows) to allow us to pull time manually from an NTP server we'd like to use (note that you have to specify this with w32tm even if you've already run "net time" with the /setsntp option):
C:\> w32tm /config /syncfromflags:manual /manualpeerlist:pool.ntp.org
The command completed successfully.
Now that we're configured, just about ready to sync times, let's do a quick time check before the sync:
C:\> time /t
06:47 AM
Oh... that looks dreadfully wrong. I think my clock is fast. Let's cause Windows to read our new configuration now:
C:\> w32tm /config /update
The command completed successfully.
And finally, let's resync:
c:\> w32tm /resync
Sending resync command to local computer
The command completed successfully.
Boom! The clock in my tool tray is now synchronized! Let's double check that we have the new and proper time:
C:\> time /t
06:13 AM
We've now synced with a remote NTP server at the command line. Yummy.

Also, there are a lot of other fascinating time-related options available with w32tm. For instance, to display your timezone, you could run:
c:\> w32tm /tz
Time zone: Current:TIME_ZONE_ID_STANDARD Bias: 300min (UTC=LocalTime+Bias)
[Standard Name:"Eastern Standard Time" Bias:0min Date:(M:11 D:1 DoW:0)]
[Daylight Name:"Eastern Daylight Time" Bias:-60min Date:(M:3 D:2 DoW:0)]
We can even convert an NT system time (the number of 100 nanosecond intervals that have elapsed since Jan 1st, 1601, the date DOS was first written by monks) to a human readable date and time:
c:\> w32tm /ntte 128904420852883740
149194 22:21:25.2883740 - 6/25/2009 5:21:25 PM
This is especially useful if you have a Vista box (my condolences) and are researching the installation of patches using "wmic qfe". In XP Pro and Windows 7, "wmic qfe list full" returns a list of patches with an "InstalledOn" field in a human readable form. But, in Vista, this date is now a hexadecimal number of an NT system time. Convert that hex number into decimal using you favorite calculator, and then fire up w32tm /ntte to convert it into a date you can read. Also, the "w32tm /ntpte" command converts NTP times (seconds from Jan 1st, 1900, the date that Nicola Tesla implemented the first Unix kernel) into a human readable format.

But, wait, there's more! Hal showed how to add a timestamp into the output of your commands, useful in forensics. I sometimes will put a date and time into my command prompt itself, so that I can then copy and paste my session to use as evidence:
C:\> prompt $D$T$S$P$G
Sat 10/31/2009 6:15:29.97 C:\>
So, there's a bunch of time-related stuff. Have you got any other fun time kickers, Tim?

Tim clocks in:

Let's take a look at the cmdlets available that use the Time or Date nouns.

PS C:\> Get-Command -Noun date
CommandType Name Definition
----------- ---- ----------
Cmdlet Get-Date Get-Date [[-Date] ...
Cmdlet Set-Date Set-Date [-Date] ...

PS C:\> Get-Command -Noun time
[nothing]


Unfortunately, there aren't any built-in cmdlets for NTP. So to answer Ed's question, "No, no interesting kickers this week." But we can still take a look at the Get-Date cmdlet and its properties.

PS C:\> Get-Date | fl

DisplayHint : DateTime
DateTime : Sunday, November 01, 2009 10:17:23 PM
Date : 11/1/2009 12:00:00 AM
Day : 1
DayOfWeek : Sunday
DayOfYear : 305
Hour : 22
Kind : Local
Millisecond : 206
Minute : 17
Month : 11
Second : 23
Ticks : 633927106432066338
TimeOfDay : 22:17:23.2066338
Year : 2009


The object has a number of methods available, and they can be seen by piping the object into Get-Member. The results aren't shown here since it is a pretty long list. Since the reason for this episode is Daylight Savings Time we'll take a quick peek at the relevant method.

PS C:\> (Get-Date).IsDaylightSavingTime()
False


Ok, that wasn't spectacular, but let's carry on and recreate Ed's command prompt with the date.

PS C:\> function prompt {"PS $(Get-Date) $(Get-Location)>"}
PS 11/01/2009 22:43:53 C:\>


Since the prompt is defined by overloading the prompt function there are endless possibilities for setting a custom prompt.

That's all folks, hope to see you next time.