Episode #57: At Your Services

Ed embarks:

Sometimes, administrators and users need to alter the state of running services on a Windows box, especially those services that automatically start during system boot. Many users are familiar with the services control panel GUI, with its familiar "Startup Type" column and values of Manual, Disabled, and Automatic. The Automatic services, in particular, are those that startup when the system is booted, even if no one logs onto the machine.

As described in Episode #40, Ed's Heresy, you can launch the services control GUI at the command line using:

C:\> services.msc

Look through all those services and their startup types, and you'll see an awful lot of automatic ones. Unlike Linux, Windows doesn't have the concept of different run-levels which start different sets of services. Instead, all automatic services are... well... automatically started at bootup. Vista and later did introduce a new option for automatic services startup type, called "Automatic (Delayed start)", which causes Windows to activate a service after all of the initial boot activities are completed, making bootup faster. So, I guess we do have a bit of fine-grained differentiation in bootup services: those that are done during the boot itself [Startup Type= Automatic] and those that happen right afterward [Startup Type= Automatic (Delayed Start)]. Of course, "Manual" services are initiated by user action, such as clicking the "Start" button in the services.msc GUI.

But, speaking honestly and bluntly, why use the crappy services.msc GUI when we can interact with our services at the command line? One of the real gems of the Windows command line is the Service Control command, known as "sc" for short. In the olden days, we could use the "net start" command to get a list of running services. But, good old "net start" pales in comparison to the mighty sc. I only use "net start" when I'm on a Windows 2000 box that lacks the sc command from the Resource Kit. I'm happy to report that sc is included in XP Pro and later.

We can use sc to get a list of all services on the box, regardless of their state, by running:

C:\> sc query state= all

Please note that sc is finicky about spaces in its command options. You have to enter it this way: "state equals space all". If you omit the space, it doesn't work. If you put a space before the equals, it doesn't work. Annoying, to be sure, but once you get the hang of it, it's almost tolerable. In fact, that could be the new marketing line for Windows: "Once you get the hang of it, it's almost tolerable." Microsoft marketing reps can feel free to contact me if they'd like to license that line.

Anyway, we can get more detail about each service's state using:

C:\> sc queryex state= all

The queryex there gives us additional information, including the PID each service is running inside of. If you'd like to focus on a single service, looking at all of its details, you could run:

C:\> sc qc [SERVICE_NAME]

The qc stands for "query configuration", and shows all kinds of neat stuff, including the binary path and command line flags used to launch the executable.

That [SERVICE_NAME] option used in the sc command must be the "SERVICE_NAME" and not the "DISPLAY_NAME", both of which are shown in the "sc query" output. The former is the internal name of the command used within Windows, while the latter is a more human-friendly form of the name used in the GUI. Most Windows admins think in terms of DISPLAY_NAME, but the sc command thinks otherwise. To map the DISPLAY_NAME in your head to SERVICE_NAME, you could use WMIC to map one to the other:

C:\> wmic service where displayname="[DISPLAY_NAME]" get name

You can even use substring wildcards on DISPLAY_NAME here with:

C:\> wmic service where (displayname like "%something%") get name

OK, now, fully armed with the sc-friendly SERVICE_NAME, we can proceed. If you want to stop or start a service immediately, you can use this simple command:

C:\> sc [start|stop] [SERVICE_NAME]

Please note, however, if you stop a service whose type is "Automatic" or "Automatic (Delayed Start)", that service will come back the next time you reboot. To change that behavior, you'll have to alter the startup type of the service, making it either manual or disabled. To do so, you could use the sc command as follows:

C:\> sc config [SERVICE_NAME] start= demand

Note that the GUI refers to such services as "manual", but the sc command calls them "demand". Consistency, thy name is Windows.... NOT! (Hey! There's another potential marketing campaign!) Note that if the service is currently running, this change in its configuration won't stop it... we are merely changing its configuration, not its current state.

To configure a service to be disabled at next boot, you could run:

C:\> sc config [SERVICE_NAME] start= disabled

For Automatic, use "start= auto" and for Automatic (Delayed Start), use "start= delayed-auto".

Note that there are two other service start types: boot and system. These are for device drivers, and I wouldn't mess with them unless you first create a snapshot of your virtual machine. What? You are running Windows on _real_ hardware and not a VM? Yikes. Be very, very careful with your config!

Another nifty feature of the sc command is its ability to show us service dependencies, as follows:

C:\> sc enumdepend [SERVICE_NAME] [buffer_size]

Try this for the RPC Service as follows:

C:\> sc enumdepend rpcss

It'll show you the start of the list of services that depend on rpcss, but then complain that its default buffer for gathering this information is too small. You can specify a bigger buffer by putting an integer at the end for the output display buffer, such as 8092. How convenient it is that it allows you to specify your display buffer size. Yeah, right.

Another fine aspect of the sc command is that it can be used remotely, provided that you have admin-level SMB access of a remote system. Any of the sc commands listed above can be directed to a remote system by simply adding \\[IP_addr] right after the sc, as in:

C:\> sc \\[IP_addr] [other_options]

Oh, and one more thing... All those automatic services associated with the underlying operating system do start in a specific order that Microsoft has carefully planned. That order is stored in the registry as a list, and can be displayed using:

C:\> reg query hklm\system\currentcontrolset\control\servicegrouporder

The list looks hideous in the output of the reg command. If you'd like it to look a little prettier, you can open that registry key in regedit.

Hal wants to know where Ed gets off:

Oh rats. Welcome to another episode of "this would be simple if every flavor of Unix didn't do this a little bit differently". The first simplifying assumption I'm going to make is to throw the *BSD folks off the bus. Sorry guys, if you're reading this then you already know how boot services work on your OS and you don't need me to tell you.

For everybody else in the world, the boot service controls on your system have typically evolved from some knockoff version of the System V boot sequencing scheme. This means there's probably a directory on your system called /etc/init.d (or perhaps /etc/rc*/init.d) that contains a whole pile of scripts. Generally, each one of these scripts is responsible for starting and stopping a particular service.

One of the nice features of this system is that you can run the boot scripts manually to start and stop services. The scripts accept "start", "stop", and often the "restart" option as well:

# /etc/init.d/ssh stop
 * Stopping OpenBSD Secure Shell server sshd                             [ OK ] 
# /etc/init.d/ssh start
 * Starting OpenBSD Secure Shell server sshd                             [ OK ] 
# /etc/init.d/ssh restart
 * Restarting OpenBSD Secure Shell server sshd                           [ OK ] 

In general, using the /etc/init.d scripts is the preferred method for starting and stopping services-- as opposed to just killing them-- because the script may perform additional service-specific cleanup actions.

Now when the system is booting we need to be careful that services get started in the correct dependency order. For example, there's no point in starting network services like SSH and Apache until the network interfaces have been initialized. Boot sequencing is the job of the /etc/rc*.d (sometimes /etc/rc*/rc*.d) directories.

If you look into these directories, you'll find a bunch of files named Snn* and Knn*-- for example "S16ssh". These "files" are actually links back to the scripts in /etc/init.d. The numbers are used to make sure the scripts are run by the init process in the correct sequence, and there are usually gaps left in the numbering so that you can add your own scripts at appropriate points in the sequence. The leading "S" tells init to run the script with the "start" option to start the given sequence at boot time. "K" means kill or "stop".

So why are there lots of different rc*.d directories? The basic idea was that Unix systems were supposed to be able to boot to different "run levels", numbered 1-5, that enabled different levels of functionality. I'm old enough to remember a time when booting to run level 2 meant "multi-user mode" and run level 3 meant "multi-user mode plus network file sharing" (yes, I know, get me a walker and you kids stay off my lawn!). These days, the whole "run level" concept has gotten wildly confused. Many Linux systems use run level 3 for multi-user and run level 5 for "multi-user with GUI", but Ubuntu boots into run level 2 by default. Solaris boots using both the run level 2 and run level 3 scripts, which is just wacky.

The reason the whole run level question is relevant is that in order to enable or disable certain services from being started at boot time, you need to change the links in the appropriate rc*.d directory. To do that, you need to know what run level your system is booting into by default. Some systems have a "runlevel" command which will tell you what run level you're currently booted into. On other systems you'll need to find the default run level setting in /etc/inittab-- "grep default /etc/inittab" usually works. The alternative is to just change the links in all of the rc*.d directories, just to be on the safe side.

Say you're on a Ubuntu system and you're booting into run level 2. You "cd /etc/rc2.d" and start messing with the links. You can see the services that are started at this run level with a simple "ls S*". If you want to make changes, just remember that the init program ignores any script that doesn't start with an "S" or a "K". So one way to prevent a service from being started at boot time is to just rename the link. There are lots of different conventions people use for this: some people change the "S" links to "K" links, others give them names like "aaa*" (sorts at the beginning of the directory), "zzz*" (sorts to the end), or ".NO* (hides links from normal "ls"). You can also just remove the links.

The only problem with messing with the link names directly is that you'll often find that vendor patches and software updates will often restore the original links in your rc*.d directories. So after updating your system it's a good idea to check your rc*.d directories to make sure your changes haven't been reverted.

To deal with this problem, most Unix variants have some sort of tool that manages boot time configuration of services. The tool typically manages some extra "meta-data" associated with each boot script that says which scripts should be started at the different run-levels. For example, Red Hat derived systems (RHEL, Fedora, CentOS, etc) use a command-line tool called chkconfig (originally developed for IRIX) that uses meta-data stored in special comments at the top of each boot script. Debian has update-rc.d and sysv-rc-conf which just rename "S" links to "K" links on your behalf (and vice versa). Solaris has the XML horror that is the Service Management Framework (SMF). You'll need to read the docs for your particular flavor of Unix.

Episode #56: Find the Missing JPEG

Hal is helpful:

As a way of giving back to the community, I occasionally drop in and answer questions on the Ubuntu forums. One of the users on the forum posed the following question:

I have about 1300 pictures that I am trying to organize.
They are numbered sequentially eg. ( 0001.jpg -> 1300.jpg )
The problem is that I seem to be missing some...

Is there a fast way to be able to scan the directory to see which ones I am missing? Other than to do it manually, which would take a long time.

A fast way to scan the directory? How about some command-line kung fu:

for i in $(seq -w 1 1300); do [ ! -f $i.jpg ] && echo $i.jpg; done

The main idiom that I think is important here is the use of the test operator ("[ ... ]") and the short-circuit "and" operation ("&&") as a quick-and-dirty "if" statement in the middle of the loop. Ed and I have both used this trick in various Episodes, but I don't think we've called it out explicitly. For simple conditional expressions, it sure saves a lot of typing over using a full-blown "if ... then ..." kind of construct.

I'm also making use of the seq command to produce a sequence of numbers. In particular, I'm using the "-w" option so that the smaller numbers are "padded with zeroes" so that they match the required file names. However, while seq is commonly found on Linux systems, you may not have it on other Unix platforms. Happily, bash includes a printf routine, so I could also write my loop as:

for ((i=1; $i <= 1300; i++)); do file=$(printf "%04d.jpg" $i); \
    [ ! -f $file ] && echo $file; done

Update from a Loyal Reader: Jeff Haemer came up with a nifty solution that uses the brace expansion feature in bash 4.0 and later (plus a clever exploitation of standard error output) to solve this problem with much less typing. You can read about it in his blog posting.

Now I have to confess one more thing. The other reason I picked this problem to talk about is that I'm pretty sure it's going to be one of those "easy for Unix, hard for Windows problems". Let's see if Ed can solve this problem without using four nested for loops, shall we?

Ed retorts:
Apparently our little rules have changed. Now, it seems that we get to impose constraints on our shell kung fu sparing partners, huh? Hal doesn't want four nested FOR loops. As if I didn't have enough constraints working in cmd.exe, Hal the sadist wants to impose more. Watch out, big guy. Perhaps next time, I'll suggest you solve a challenge without using any letter in the qwerty row of the keyboard. That should spice things up a bit. Of course, you'd probably use perl and just encode everything. But I digress.

One of the big frustrations of cmd.exe is its limitations on formulating output in a completely flexible fashion. Counting is easy, thanks to the FOR /L loop. But prepending zeros to shorter integers... that's not so easy. The Linux printf command, with its % notation, is far more flexible than what we've got in cmd.exe. The most obvious way to do this is that which Hal prohibits, namely four FOR /L loops. But, our dominatrix Hal says no to four nested FOR loops. What can we do?

Well, I've got a most delightful little kludge to create leading zeros, and it only requires one FOR /L loop counter plus a little substring action like I discussed in Episode #48: Parse-a-Palooza. Here is the result:

c:\> cmd.exe /v:on /c "for /l %i in (10001,1,11300) do @set name=%i & set 
     fullname=!name:~1,4!.jpg & dir !fullname! >nul 2>nul || echo !fullname! Missing"
0008.jpg  Missing
0907.jpg  Missing
1200.jpg  Missing

Here, I'm launching a cmd.exe with /v:on to perform delayed variable expansion. That'll let my variables inside my command change as the command runs. Then, I have cmd.exe run the command (/c) of a FOR /L loop. That'll be an incrementing counter. I use %i as the iterator variable, counting from 10001 to 11300 in steps of 1. "But," you might think, "You are ten thousand too high in your counts." "Ah..." I respond, "That extra 10,000 gives me my leading zeros, provided that I shave off the 1 in front." And, that's just what I do. In the body of my FOR loop, I store my current iterator variable value of %i in a variable called "name". Remember, you cannot perform substring operations on iterator variables themselves, so we squirrel away their results elsewhere. I then introduce another variable called fullname, which is the value of name itself (when referring to delay-expanded vars, we use !var! and not %var%), but with a substring operation of (~1, 4), which means that I want characters starting at an offset of 1 and printing four characters (in this case digits). With offset counting starting at 0, we are shaving off that leading 1 from our ten-thousand-to-high iterator. I throw the output of my dir command away (>nul) as well as its standard error (2>nul).

Then, I use the || operator, which I mentioned in Episode #47, to run the echo command only if the dir command fails (i.e., there is no such file). I display the name of the missing file, and the word "Missing".

There are many other ways to do this as well, such as using IF NOT EXIST [filename]. But, my initial approach used dir with the || to match more closely Hal'sl use of &&. The IF statement is far more efficient, though, because it doesn't require running the dir command and disposing of its standard output and standard error. So, we get better performance with:

c:\> cmd.exe /v:on /c "for /l %i in (10001,1,11300) do @set name=%i &
    set fullname=!name:~1,4!.jpg & IF NOT EXIST !fullname! echo !fullname! 
    Missing"

In the end, we have a method for generating leading zeros without using nested loops by instead relying on substring operations, all to make the rather unreasonable Mr. Pomeranz happy. :)

Episode #55: Fishing for Network Configs

Ed kicks it off:

Man, we've covered a lot of topics in our 54 episodes prior to this one. But, in our rush to get you the latest chocolate-covered command-line fu, occasionally we've missed some fundamentals. People write in with questions (which we love) about such items, inspiring a new episode. Back in May, we received a great question from Johnny C:

> I have a suggestion for command line kungfu.
> I need to be able to change my IP Address back and forth from DHCP
> where everything is dynamic to a dedicated IP address.
> I've worked with this for a while and my problems have been not able
> to update DNS on Windows

Ah... good one, sir! Let's face it: the built-in Windows GUI associated with network configuration changes is horrible... forcing numerous clicks through various screens to make even small tweaks. At least we don't have to live through the dreaded reboots of the Windows 95 era just to change IP addresses anymore.

On Windows, for manipulating network configs at the command line, netsh rocks, and it can do what you want, Johnny C, and much more. In fact, when I've got a lazy summer afternoon with nothing better to do, I fire up netsh (or the equally fun and interesting wmic command) and just explore, sometimes for hours on end. The netsh command (like wmic) can run in two modes: either as a little command interpreter of itself (by typing netsh and hitting Enter) lending itself to exploration, or as a single shot command of netsh followed by various options.

To get a glimpse of the capabilities of netsh, run the following:

C:\> netsh
netsh> ?

The following commands are available:

Commands in this context:
..             - Goes up one context level.
?              - Displays a list of commands.
abort          - Discards changes made while in offline mode.
add            - Adds a configuration entry to a list of entries.
advfirewall    - Changes to the `netsh advfirewall' context.
alias          - Adds an alias.
bridge         - Changes to the `netsh bridge' context.
bye            - Exits the program.
commit         - Commits changes made while in offline mode.
delete         - Deletes a configuration entry from a list of entries.
dhcpclient     - Changes to the `netsh dhcpclient' context.
dump           - Displays a configuration script.
exec           - Runs a script file.
exit           - Exits the program.
firewall       - Changes to the `netsh firewall' context.
help           - Displays a list of commands.
http           - Changes to the `netsh http' context.
interface      - Changes to the `netsh interface' context.
ipsec          - Changes to the `netsh ipsec' context.
lan            - Changes to the `netsh lan' context.
nap            - Changes to the `netsh nap' context.
netio          - Changes to the `netsh netio' context.
offline        - Sets the current mode to offline.
online         - Sets the current mode to online.
p2p            - Changes to the `netsh p2p' context.
popd           - Pops a context from the stack.
pushd          - Pushes current context on stack.
quit           - Exits the program.
ras            - Changes to the `netsh ras' context.
rpc            - Changes to the `netsh rpc' context.
set            - Updates configuration settings.
show           - Displays information.
unalias        - Deletes an alias.
winhttp        - Changes to the `netsh winhttp' context.
winsock        - Changes to the `netsh winsock' context.
wlan           - Changes to the `netsh wlan' context.

Nice! Lots of very useful stuff, including "interface" and "firewall" (the latter of which we discussed in Episode #30). There's also some really nifty settings for ipsec (on 2003 and later) and wlan (on Vista and later) contexts. To change to an individual context, just type its name (such as "interface") and then type ? at the netsh> prompt to get more info about it. You can then navigate down by entering follow-up commands and contexts, and then pop back up to earlier contexts entering a command of dot-dot (".."). I wish there was a "back" command instead of .., but I can cope. There's even a pushd and popd command for netsh contexts, rather similar to the pushd and popd for directories we discussed in Episode #52.

One of my most common uses of netsh is to change IP address settings of the machine. In the spirit of the cliche "Give a man a fish and feed him for a day... teach him to fish and feed him for life", let me show you how you can fish around inside of netsh.

We first invoke netsh and then move to the interface context:

C:\> netsh
netsh> interface
netsh interface> ?

Here, you can see options for various elements we can configure on the machine. Of particular interest to us now is ip (on XP and 2003) or ipv4 (on Vista and later). Happily, you can just type "ip" on Vista, and it will take you to the ipv4 context, so our netsh commands for changing addresses and such are compatible between various versions of our beloved Windows operating system.

netsh interface> ip
netsh interface ip> set ?

Now, we can get a sense of the various items we can set, including addresses, dns, and wins. But, wouldn't it be nice if Windows would give us examples of how to set each? Well, ask and ye shall receive:

netsh interface ip> set address ?
Usage: set address [name=]  [[source=]dhcp|static] [[address=][/] [[mask=]
] [[gateway=]|none [gwmetric=]] [[type=]unicast|anycast] [[subinterface=]
] [[store=]active|persistent]                             

Examples:

  set address name="Local Area Connection" source=dhcp
  set address "Local Area connection" static 10.0.0.9 255.0.0.0 10.0.0.1 1

If you'd like to get a list of all interfaces available on the machine, you could run (from the normal C:\> prompt, not within netsh):

C:\> netsh interface show interface

I know... it looks like it is redundantly repeating itself twice back to back, and it is. But, that's the command. Now, we know how to refer to our network interfaces for manipulating them.

Then, to set an IP address, we could just run the command:

C:\> netsh interface ip set address name="Local Area Connection"
static 10.10.10.10 255.255.255.0 10.10.10.1 1

This will set our IP address to 10.10.10.10, with a netmask of 255.255.255.0, a default gateway of 10.10.10.1, and a routing metric (number of hops to that gateway) of 1.

For DHCP, we simply run:

C:\> netsh interface ip set address name="Local Area Connection" source=dhcp

OK.... now to answer Johnny C's specific question, setting our primary DNS server:

C:\> netsh interface ip set dnsserver name="Local Area Connection"
static 10.10.10.85 primary

And, if you'd rather get that info from DHCP, you could use:

C:\> netsh interface ip set dnsserver name="Local Area Connection" source=dhcp

I frequently find myself changing between my laboratory network and my production network, which have completely different IP addressing schemes. To help make a quick switch between them, I don't use those one of those goofy network configurator GUIs, because, well, they are kind of tawdry. Instead, I've created two simple scripts that I keep on my desktop: test.bat and prod.bat. Each one contains two netsh commands. The first command sets my IP address, netmask, and default gatewy for either prod or test, and the second command sets my DNS server. When I want to invoke them, I simply run them with admin privs (based on either being logged in as admin, or right clicking and selecting "run as administrator").

Hal kicks it old school:

Where the Windows way is to have one big command that does everything, remember the Unix design religion is to have a bunch of small commands that do simple things and then combine them to produce the effect you want. It doesn't help that Unix systems have been networked devices since the earliest days of the Internet-- we've got multiple generations of command interfaces to deal with. But let me try to hit the high points.

Suppose our system is normally configured to use DHCP but we want to manually move it onto a new network with a static address assignment. Step one is to change your IP address with the ifconfig command:

# ifconfig eth0 10.10.10.1 netmask 255.255.255.0
# ifconfig eth0
eth0      Link encap:Ethernet  HWaddr 00:0C:29:18:C3:0D 
         inet addr:10.10.10.1  Bcast:10.10.10.255  Mask:255.255.255.0
         inet6 addr: fe80::20c:29ff:fe18:c30d/64 Scope:Link
         UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
         RX packets:55 errors:0 dropped:0 overruns:0 frame:0
         TX packets:158 errors:0 dropped:0 overruns:0 carrier:0
         collisions:0 txqueuelen:1000
         RX bytes:7542 (7.3 KiB)  TX bytes:32712 (31.9 KiB)
         Interrupt:18 Base address:0x2024

As you can see from the example above, you can also use ifconfig to display information about an interface's configuration ("ifconfig -a" will display the configuration information for all interfaces on the system).

However, changing the IP address with ifconfig doesn't have any impact on your routing table. You'll probably need to add a default route when you change the IP address:

# route add default gw 10.10.10.254
# netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
10.10.10.0      0.0.0.0         255.255.255.0   U         0 0          0 eth0
0.0.0.0         10.10.10.254    0.0.0.0         UG        0 0          0 eth0

I should note that as you move from on Unix distribution to another, the route command syntax tends to change in minor ways, rendering a command that is syntactically correct on one system completely useless on another. The above example is for Linux. Oh and by the way, if your DHCP configuration has created a default route for the network you used to be on, you can remove it with "route del default gw <ipaddr>".

The last thing you have to do is update your list of local DNS servers. This list is configured in /etc/resolv.conf. Most DHCP clients will simply overwrite the contents of this file with the name servers and local domain they learn from their DHCP servers, but you can also edit this file directly. A sample file might look like:

nameserver 10.10.10.100
search somedomain.com

Replace "somedomain.com" with the default domain you want the host to use for looking up unqualified host names. You can have multiple "nameserver" lines in your file for redundancy. However, I warn you that the timeout on the first lookup is long enough that your users will pick up the phone and call you to tell you the "network is down" before the system fails over to the next name server in the list.

The combination of ifconfig, route, and editing your resolv.conf file should be sufficient to get you manually moved onto a new network. The more interesting question is how to you revert back to using DHCP to configure your network interface? Assuming your machine is configured by default to use DHCP, the easiest thing is to just shut down and then reactivate your network interface. Of course the process for doing this is completely different for each Unix system you encounter. On most Linux systems, however, the following will work:

# ifdown eth0
# ifup eth0

Determining IP information for eth0... done.
# ifconfig eth0
eth0      Link encap:Ethernet  HWaddr 00:0C:29:18:C3:0D 
         inet addr:192.168.1.1  Bcast:192.168.1.255  Mask:255.255.255.0
         inet6 addr: fe80::20c:29ff:fe18:c30d/64 Scope:Link
         UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
         RX packets:56 errors:0 dropped:0 overruns:0 frame:0
         TX packets:228 errors:0 dropped:0 overruns:0 carrier:0
         collisions:0 txqueuelen:1000
         RX bytes:7884 (7.6 KiB)  TX bytes:44273 (43.2 KiB)
         Interrupt:18 Base address:0x2024

By the way if you're looking for your static network interface configuration information, you'll find it in /etc/sysconfig/network-scripts/ifcfg-eth0 on Red Hat systems (including CentOS and Fedora) and in /etc/network/interfaces on Debian systems. This is the place where you can set the default interface configuration parameters to be used when booting. Be aware, however, that on modern Ubuntu systems network configuration is under the control of NetworkManager by default-- a GUI-based network configuration tool very reminiscent of the Windows network configuration GUI. Helpful for people coming over from the Windows environment I guess, but kind of a pain for us old farts who are used to configuring network interfaces with vi.

Episode #54: chmod Squad

Hal says:

I was doing some work for a customer recently that involved a lot of manipulating file system permissions and ownerships and I realized that we've not yet had a Command Line Kung Fu Episode on this subject. That's just crazy! So let's address this shortcoming right here and now.

You're probably familiar with the chown command for changing the owner (and group owner, if desired) of a file or group of files. Add the "-R" (recursive) option, and you can set the ownerships on an entire directory structure:

# chown -R hal:users /home/hal

These days you typically have to be root to use the chown command. In the olden days of Unix, it was common to allow any user to use chown and let them "give away" their own files to other users. But this caused problems for systems that used disk quotas because it was a loophole that allowed malicious users to consume other users' quotas. And in the really old days the chown command wouldn't strip execute bits when files were chowned, thus allowing you to create set-UID binaries belonging to other users-- obviously a huge security problem.

The problem with having to chown everything as root, however, is that all too often I see administrators making a mistake like this:

# cd /home/hal
# chown -R hal:users .*     # *NEVER* DO THIS! DANGER!

Why is the above chown command so dangerous? The problem is that the glob ".*" matches the ".." link that points back to the parent directory of the current directory. So in the example above we're going to end up making the entire /home file system owned by "hal". This is actually incredibly difficult to recover from, short of restoring from a backup, because it's not safe to assume that every file in a user's home directory is going to be owned by that user.

Anyway, the safe way to chown a user's "dot-files" and directories is:

# chown -R hal:users .[^.]*     # SAFE

The "[^.]" means "match any character EXCEPT period", thus protecting you from matching the ".." link. Of course it would also skip files and directories named things like "..example", but you wouldn't expect to find these in a typical file system.

Having set the ownerships on a set of files, you can also control the access rights or permissions on those files. There are three categories of permissions-- the permissions for the primary owner of the file ("user" permissions), the group owner of the file ("group" permissions), and "everybody else" ("other" in Unix parlance). For each group you can allow "read" access (view the contents of a file or get a listing of a directory), "write" (modify the contents of a file or add, remove, and/or rename files in a directory), and/or "execute" privileges (execute a file as a program or access files in a directory). In "absolute" mode with the chmod command, we express these permissions as a vector of octal digits: "read" is 4, "write" is 2, and "execute" is 1. Here are some common examples:

$ chmod 755 /home/hal     # rwx for me, r+x for group and other
$ chmod 666 insecure      # rw for everybody, "world writable"
$ chmod 700 private.dir   # only I have access here

Because the default assumption for the Unix operating system is to create world-writable files and directories, we use the umask value to express which bits we want NOT to be set when new files are created. The common Unix umask default is "022" which means "don't set the write bits for group and other". Some sites enforce a default umask of "077", which requires you to explicitly use chown to all others to access your files ("discretionary access control" is the term usually bandied about here).

There's actually an optional leading fourth digit you can use with the chmod command, which covers the "set-UID" (4), "set-GID" (2), and "sticky-bit" settings (1). Here are some examples showing the typical permission settings for some common Unix programs and directories:

# chmod 4755 /bin/su              # set-UID
# chmod 2755 /usr/sbin/sendmail   # set-GID
# chmod 1777 /tmp                 # "sticky"

The "sticky-bit" is another interesting piece of Unix history. Back in the very old days when computers, disks, and even memory were vastly slower than they are today, there was a significant start-up cost with loading a program into memory before execution. For commonly-used programs like ls, the total overhead was enormous. So certain executables were marked with the "sticky-bit" as a signal to the kernel so that the program image would tend to "stick around" in memory so that the program didn't have to constantly be reloaded. Of course, in the age of shared libraries plus fast disks and memory this use has long since stopped having any value. Nowadays, the sticky bit is used as a marker on world-writable directories like /tmp and prevents anybody except the owner of the file from removing or renaming the file.

As you're probably aware, chmod also supports "symbolic" mode for expressing changes to the permissions of files:

$ chmod go-w insecure   # strip the write bits for group and other
$ chmod a+x myscript    # make file executable for everybody ("all")
$ chmod +x myscript     # same as above

The first part of the syntax is the groups to apply the permission change to: "u" for the primary user or owner of the file, "g" for the group owner, and "o" for other or everybody else. You can also use "a" to represent all groups, or just leave it off completely because "a" is the default. The next item in the symbolic description is a plus or a minus sign depending on whether you're "adding" or "subtracting" permission bits. Finally you specify the bit(s) you want to add or subtract.

Why is symbolic mode useful? Well, I recently needed to make an entire directory structure be only accessible by its owner. You can't really use absolute file modes for this, because while "chmod -R 600 ..." would work fine for regular files, it wouldn't do for the directories (directories must have "x" set to be usable). "chmod -R 700 ..." would be fine for directories, but not appropriate for regular files. You could hack something together with find, but it's much easier to just do:

# chmod -R go-rwx /some/directory

This strips all bits for the "group" and "other" categories, but leaves the current permissions for the owner of the file alone.

Unfortunately, I now have to turn things over to Ed. I say "unfortunately", because I'm going to have to endure his gloating about the higher level of permissions granularity implemented in NTFS as compared to the typical Unix file system. To this my only response is, "Oh yeah? Tell me about ownerships and permissions in FAT file systems, boyo."

Ed responds:
I thought we had an agreement to never bring up FAT. As far as I'm concerned, it never happened. NTFS has been with us since Day 1. Yeah... right.

Anyway, since Hal brought up FAT, I'll address it briefly. There is no concept of security with FAT file systems. Every user is God and can access everything. It's a relic from the days when DOS (and later Windows) machines were expected to be single-user systems not connected to, you know, a network or anything. The security model was to diligently implement no security model whatsoever. Mission accomplished!

But, then, there came NTFS. Ahhh... finally, a modern file system for Windows boxen. NTFS has the concept of ownership and even more fine-grained controls than our Unix brethren have. As Windows NT matured into Windows 2000 and then XP/2003, and then Vista/2008 and beyond, these fine-grained permissions and the ability to manipulate them at the command line has expanded significantly. In fact, we're now to the point where these features are simultaneously flexible enough to use yet complex enough to be overwhelming and almost unusable. That happens a lot in the Windows world.

From the command-line, we can see the ownership of a file or directory using the dir command with the /q option:

C:\> dir /q

Nice! Easy! Cool!

Yeah, but wait. Now things get ugly. If the hostname\username is particularly long, it will be truncated in the display, which allocates only 23 characters for the hostname backslash username. None of the other output formatting options of dir (/b, /w, /d, and /x) fix this, because they either override /q (leaving off the ownership info) or keep the truncation. This problem is livable, but still kind of annoying.

To change the owner of a file or directory, a la the Linux chown command, we can use the icacls command found in 2003 SP2, Vista, 2008, and Windows 7, as follows:

C:\> icacls [filename] /setowner [username]

To change the owner in a recursive fashion, use the /t option with icacls, because, as everyone knows the word "recursive" has no letter t in it. That makes it easy to remember, right? (Actually, I think they were going after the mnemonic for "tree", but a /s or even a /r would be more palatable). And, /c makes icacls continue despite an error with one or more files as it is processing through a /t or wildcard.

Now, look at that list of supported Windows versions for icacls... there's an important one missing from the list. Which one? Queue the Jeopardy music and pause. Sadly, it's our good friend, Windows XP. Unfortunately, I haven't found a way at the command line using only built-in tools to change owner in XP. You could mount the XP system's drive from a 2003/Vista/2008/7 box and run icacls, but that's not exactly using built-in tools, now, is it? You can copy the 2003 SP2 version of icacls.exe to Windows XP, and it'll work... but again, that's not exactly using built-in tools, and I have no idea whether that violates some sort of Windows license limitation. And, I don't want to know, so don't send me any mail about it.

The Vista version won't run on XP though. I've also found that icacls on 2003 (whether running in 2003 or... ahem... copied to XP) is quite buggy as well, often giving errors when trying to change owners. This 2003 icacls fail is a known issue documented by Microsoft, and they released a hotfix for it which is seldom installed. So, does this count as not using a built-in command? :)

To change ownership in the GUI on XP, you can apply a ridiculous process described by Microsoft here.

Now, XP does include the cacls command (not icacls), as does Win2K and all of the later versions of Windows. The cacls command lets you change permissions, the rough equivalent of the Linux chmod command. But, cacls will not let you change the owner.

The syntax for the cacls command lets us specify the file or directory name we want to change, followed by a bunch of potential options. We can grant access rights with /G [user:perm]. The perms supported at the command line are R (Read), W (Write), C (Change, sometimes referred to as "Modify" in Windows documentation), and F (Full Control). These rights are actually conglomerations of the very fine grained rights built-into NTFS, which are described here.

We can revoke these access rights with /R [user]. Note that you cannot revoke individual rights (R/W/C/F), but instead you revoke all of them at a given time for a user. Revocation is an all-or-nothing situation. The /E is used to edit the existing ACL, as opposed to the default, which replaces the ACL. We often want to use /E so that we don't blow away any access rights already there. There is also a /D [user] option in cacls, which explicitly denies the user access to the object, again on an all or nothing basis. These deny rights override any allow rights, thankfully.

With that overview under our belts, to frame the following fu, I'd like to mimic Hal's commands, mapping them into the Windows world to the extent we can.

We start with Hal's first command:

$ chmod 755 /home/hal     # rwx for me, r+x for group and other

In Windows, we can achieve the same thing with these three commands:

C:\> cacls [filename] /G [username]:F
C:\> cacls [filename] /E /G [groupname]:R
C:\> cacls [filename] /E /G Everyone:R

Note that the R here is a conglomerated Read, which includes reading and executing the file (again, those conglomerated rights are defined here). Also, in the first command of these three, we've not used /E, so we are blowing away all existing access rights to start out and adding full control for our username in the first command. Hal's assigning permissions absolutely, not relative to existing permissions, so we leave off the /E. In our follow-up command, though, we edit rights (/E) building on with a couple of extra commands to match roughly what Hal has done.

Next, our sparring buddy ran:

$ chmod 666 insecure      # rw for everybody, "world writable"

We can roughly mimic this with:

C:\> cacls [filename] /G [username]:F
C:\> cacls [filename] /E /G Everyone:C

Now, the execute capability is baked into both Full control (F) and Change (C), so we're not really removing execute capability here. With icacls (not cacls), we can access the fine-grained rights and make a file read-only with:

C:\> icacls [filename] /grant Everyone:RW

Remember, /G is for cacls, and /grant is for icacls. Consistency is a beautiful thing.

And then, Hal pulled this out of his ear:

$ chmod 700 private.dir   # only I have access here

In our comfy Windows world, we could run:

C:\> cacls [filename] /G [username]:F

Without the /E above, this snips off everyone else's access.

Hal then wowed us all with:

$ chmod go-w insecure   # strip the write bits for group and other

This one is a bear to do at the command-line in Windows, because the /R option is all or nothing when revoking permissions. We'd need to analyze the existing ACL first, and then manipulate it with a /E to build the ACL we want. It would require a script to do this by my estimation.

Hal then popped off:

$ chmod a+x myscript    # make file executable for everybody ("all")

Which we can do with:

C:\> cacls [filename] /E /G Everyone:R

Again, in cacls, R includes Read and Execute.

And, finally, Hal did:

# chmod -R go-rwx /some/directory

Which, mapped to our own personal insanity, is:

C:\> cacls [directory] /p [user]:F /t

In this one, we've used the /p to replace the existing ACLs for the given user name, which we are giving full control (F), in a recursive fashion (/t).

Be careful when playing with icacls and cacls, because they are a great way to very much hose up your system. The icacls command has an option to save ACLs into a file for later inspection or even restoring from:

C:\> icacls [file_or_dir] /save [aclfile]

Again, we have a /t or /c option here. The restore function is invoked with /restore. It should be noted that the aclfile holds only relative paths. Thus, if you are going to do a restore, you need to run the command in the same directory that you used for the icacls /save.

Episode #53: The Final Countdown

Ed starts:

My 40th birthday is coming up this Tuesday (the day that this episode will be posted), so I've been thinking about the passage of time a lot lately, with countdowns, hour glasses, and ticking clocks swirling through my thoughts. This all came to a head a couple of days ago, when I received an e-mail from a buddy of mine who runs capture the flag games. He wanted a countdown timer for his games, something to spice up players' experience in the game. But, instead of using some lame clock or stopwatch app, he asked me to provide him some command-line kung fu to do the trick. I came up with a solution that he liked, and it includes some fun twists and turns that I thought our readers here may find useful or at least enjoyable.

I relied heavily on FOR loops in the following command:

C:\> for /l %m in (1,-1,0) do @for /l %s in (59,-1,0) do @echo %m minutes %s seconds
 LEFT & ping -n 2 127.0.0.1 > nul
1 minutes 59 seconds LEFT
1 minutes 58 seconds LEFT
1 minutes 57 seconds LEFT
1 minutes 56 seconds LEFT
1 minutes 55 seconds LEFT

Here, I've made a countdown timer that will run for two minutes (starting at minute 1, and counting back 60 seconds from 59 to 0, then going to minute zero and counting back the same way). I start off my minute variable (%m), running it from 1 down to 0 in steps of -1. If you want a 10 minute counter, replace that 1 with a 9. My second variable (%s) runs from 59 down to 0, again in steps of -1. At each iteration through the loop, I print out how much time is left. Finally, I ping myself twice, which takes about* a second (the first ping happens immediately, the second happens about* 1 second later).

That was my starting point. But, I wanted to add some flair, adding a little audio and popping up a message on the screen at the end of each minute. So, I added:

C:\> (for /l %m in (1,-1,0) do @(for /l %s in (9,-1,0) do @(echo ONLY %m minutes %s seconds
 LEFT & ping -n 2 127.0.0.1>nul)) & start /max cmd.exe /c "echo ^G %m MINUTE^(s^)
 LEFT! & ping -n 6 127.0.0.1>nul") & echo ^G^G^G^G^G^G^G^G^G

Here, as each minute expires, I'm using the start command to run a program in a separate window (which start does by default... the /b option makes start run a program in the backgroudn of the same window as we discussed in Episode #23 on job control) maximized on the screen with the /max option. The start command allows me to kick off something else while my main loop keeps running, ticking off seconds into the next minute.

The program that the start command will launch is cmd.exe, which I'm asking to execute a command with the /c flag. The command run by cmd.exe will echo a CTRL-G, which makes the system beep, and then echoes the number of minutes left. We then wait for 5 seconds (by pinging localhost 6 times). Because I started the cmd.exe with a /c option, after its command finishes (5 seconds later), the window will disappear. The location of all those parens is vitally important here, so that we're properly grouping our commands together to notch off time.

To add even a little more flair, I added several CTRL-Gs after the timer is done to make the expiration of the full time more audible.

* OK.... yes, you are right. The countdown timer I've describe here isn't super accurate, because the 1-second rule on pings is an approximation. Also, some time will be consumed by the commands gluing this all together, making this stopwatch slower than it should be. Also, if the system is heavily loaded, that'll slow things down even more. But, to a first approximation, we've got a stopwatch here. For more accuracy, you could write a script that relies on the %time% variable we discussed in Episode #49.

Time for Hal:

Emulating Ed's loop is straightforward. I'll even add a command at the end to pop up a window when time runs out:

$ for ((m=1; $m >= 0; m--)); do for ((s=59; $s >= 0; s--)); do \
    echo $m minutes $s seconds to go; sleep 1; done; done; \
    xterm -e 'echo TIME IS UP!; bash'

Notice that I'm executing bash in the xterm after the echo command. Spawning an interactive shell here keeps the xterm window from closing as soon as it echoes "TIME IS UP!".

But Ed's example got me thinking about ways to have a more accurate clock. One idea that occurred to me was to just use watch on the date command:

$ watch -dt -n 1 date

This will give you a clock that updates every second, with some extra highlighting to show the positions in the time string updating (that's the "-d" option). But this isn't a count down timer, it's a "count up" timer.

Another idea I had was to use at or cron to pop up the warnings (see Episode #50 for more detail on at and cron). The only problem is that both at and cron are limited to 1 minute granularity, so they don't work really well as a count down timer. But if you know your capture the flag session is supposed to end at noon, you could always do something like:

$ echo xterm -display $DISPLAY -e \'echo TIME IS UP\!\; bash\' | at noon

Notice I had to use a bunch of backwhacks so that the echo command passes a syntactically correct command into at. More interestingly, I have to explicitly declare my local $DISPLAY in the xterm command. While at is normally careful to preserve your environment variable settings, it apparently intentionally skips $DISPLAY-- probably because there's no guarantee you'll actually be on that screen when you at job finally executes.

Episode #52: Prompts & Pushing It

Ed goes:

I was reading through our recent episodes the other day (a wonderful pastime activity that I encourage everyone to participate in). In Episode #49, Hal mentioned a nice option for incorporating the current time in the command prompt, a useful feature for forensics folks who want to record the time that they executed each command, as well as for general sysadmins and testers who want to see how long given activities take. In Episode #38, I mentioned how we could change our prompt by setting the environment variable called "prompt", but I didn't give a bunch of options there. Let's explore these options in more detail, starting with including the time in our prompt a la Hal.

First off, instead of using the "set prompt=[whatever]" command to set our prompt to various values, we can alternatively use the prompt command to do so. The advantage of the latter approach is that it allows us to see all the glorious options we can use in setting our prompt:

C:\> prompt /?

By default, we've got $P for the path and $G for the greater than sign. To mimic Hal's prompt from Episode #49, we could run:

C:\> prompt $C$T$F$G
( 8:02:19.57)> dir

That gives us a prompt of open paren ($C) followed by the time ($T) followed by close paren ($F) followed by a greater than sign.

For forensics guys, we may want to included the date as well, and remove the parens:

C:\> prompt $D$S$T$G
Mon 07/20/2009  8:04:36.85>

Note that the $S is a space.

Now, these changes are just temporary, applying only to the current cmd.exe and any new cmd.exe processes it starts. To make the changes permanent, we need to alter the PROMPT environment variable in the registry

Mon 07/20/2009  8:14:05.60>reg add "hklm\system\currentcontrolset\control\session manager\environment"
         /v prompt /t reg_expand_sz /d $D$S$T$G
The operation completed successfully.

After a reboot, your prompt will be changed for all users.

There is another interesting option we can add to our prompt via the $+ notation. This one will prepend our command prompt with a single plus sign for each directory we've pushed onto our directory stack via the pushd command. Wait a sec... before we get ahead of ourselves, let's look at the pushd and popd commands a bit.

When working in a given directory, sometimes you need to temporarily change into another directory or two, do a little work there, and then pop back into the original directory you were working in. The pushd and popd commands were created to help make such transitions smoother. When changing directories, instead of using the familiar cd command, you could run "pushd [dir]" as in:

Mon 07/20/2009  8:18:02.21> prompt $P$G
C:\Users\Ed> pushd c:\windows\system32
C:\Windows\System32>

This command will store your current directory in a stack in memory, and then change you to the other directory (in the case above, c:\Users\Ed will be stored on a stack, and you will change to c:\windows\system32). You can do stuff in that other directory, and even change to other directories beyond that. But, when your work is done there, you can go back to your original directory by simply running:

C:\Windows\System32> popd
C:\Users\Ed>

While you can push a bunch of directories on this directory stack and then pop them off in a LIFO operation, I find the pushd/popd commands most useful for just storing a single directory I know I'll have to pop back into in the near future, so I find myself often running:

C:\[wherever_I_am_working]> pushd .

Then, I do a little more work, and eventually change directories to where I need to work temporarily. When I'm ready to go back to where I was, I can simply popd. This technique is very helpful given that cmd.exe doesn't have the "cd -" option found in bash so that I can simply change into a previous directory I was in earlier.

With that background of pushd and popd under our belts, we can now see what the $+ does in our prompt -- It prepends one plus sign for each directory we've pushed.

C:\> prompt $+$P$G
C:\> pushd c:\windows\system32
+C:\Windows\System32> pushd c:\Users\
++C:\Users> pushd c:\temp
+++C:\temp> popd
++C:\Users> popd
+C:\Windows\System32> popd
C:\>

The little pluses can help you remember how many things you have pushed on your directory stack.

And, if you wanted to get really elaborate and simulate the behavior of "cd -" by implementing some simple scripts to use in place of cd, you could do the following:

C:\> echo @pushd %1 > c:\windows\system32\cdd.bat
C:\> echo @popd > c:\windows\system32\cd-.bat

Now, instead of using "cd" to change directories, you could always run "cdd" to do so (I use that last d to remind me that it's pushing the directory onto the directory stack). Your new cdd command will work just like the old one, but it will remember the directories you've changed from, pushing them on the directory stack. Then, to go back to where you were before, you could run "cd-" (no space). It looks like this in action:

C:\> cdd c:\users\ed
+C:\Users\Ed> cd-
C:\>

Whither Ed goest, Hal will go!

Ed, you say you want the date in the prompt in addition to the time? So be it:

$ export PS1='\D{%D %T}> '
07/20/09 10:02:50>

Unlike "\t" for the time code, there's no built-in escape sequence for including the date in the prompt. But bash does recognize "\D{...}" to insert an arbitrary set of strftime(3) escape sequences. This is very flexible, though not quite as terse. I could even add the day name to fully emulate Ed's Windows madness:

07/20/09 10:02:50> export PS1='\D{%a %D %T}> '
Mon 07/20/09 10:08:16>

You'll also notice that Unix allows us to specify spaces as actual spaces rather than "$S". What will those wacky Unix folks think of next?

Windows totally stole the pushd/popd idea from the Unix shell, and it works pretty much the same on both platforms:

$ export PS1='[\w]$ '
[~]$ pushd /tmp
/tmp ~
[/tmp]$ pushd /usr/local/bin
/usr/local/bin /tmp ~
[/usr/local/bin]$ popd
/tmp ~
[/tmp]$ popd
~

You'll notice that bash prints the directory stack after each pushd/popd operation, sort of as a reminder of where you are. You can also use the dirs command to dump the directory stack (in several different formats) or even clear the stack entirely:

[~]$ pushd /tmp
/tmp ~
[/tmp]$ pushd /usr/local/bin
/usr/local/bin /tmp ~
[/usr/local/bin]$ dirs         # default
/usr/local/bin /tmp ~
[/usr/local/bin]$ dirs -l      # expand ~ to full pathname
/usr/local/bin /tmp /home/hal
[/usr/local/bin]$ dirs -p      # one dir per line
/usr/local/bin
/tmp
~
[/usr/local/bin]$ dirs -c      # clear the stack
[/usr/local/bin]$ dirs
/usr/local/bin

You can even use popd to selectively remove elements from the directory stack:

[/opt]$ dirs
/opt /dev /usr/local/bin /tmp /usr /var /etc ~
[/opt]$ popd +2
/opt /dev /tmp /usr /var /etc ~

Notice that the elements of the directory list are numbered starting with zero (in C, arrays are numbered starting from zero and this convention was carried through to most Unix utilities), so "popd +2" removes the third element counting from the left. "popd -2" would remove the third element counting in from the right.

Unlike Windows, we don't have to create a script file to make a cdd command on our Unix systems. We can just make an alias:

[~]$ alias cdd=pushd
[~]$ cdd /tmp
/tmp ~
[/tmp]$

What is interesting is that there's no built-in Unix equivalent for the Windows "$+" prompting functionality. But since the bash shell does command substitution on $PS1 (if necessary) each time it emits a shell prompt, we can hack our own solution together:

[~]$ export PS1='`for ((i=1; $i < ${#DIRSTACK[*]}; i++)); do echo -n +; done`[\w]$ '
[~]$ pushd /etc
/etc ~
+[/etc]$ pushd /tmp
/tmp /etc ~
++[/tmp]$ pushd /usr
/usr /tmp /etc ~
+++[/tmp]$ popd
/usr /etc ~
++[/usr]$ popd
/etc ~
+[/etc]$ popd
~
[~]$

As you can see, I've added an expression in backticks at the front of the declaration for $PS1, so whatever text this sequence of commands results in will be incorporated into the shell prompt. Inside the backticks, I'm using a for loop to produce the necessary plus signs. To terminate the loop, I'm comparing against "${#DIRSTACK[*]}" which translates to "the number of elements in the $DIRSTACK array variable". $DIRSTACK is a highly magical environment variable that stores the elements of the directory stack that pushd and popd use.

You'll also notice that I'm starting the loop at offset 1 rather than offset 0. As you can see in Ed's Windows example, the "$+" element only counts the number of directories that have been explicitly pushed onto the stack with pushd. However, the bash $DIRSTACK variable always includes the current working directory. So when you pushd for the first time, the number of elements in $DIRSTACK is actually two: your original directory that you started in, plus the new directory you pushd-ed onto the stack. Anyway, I started my loop counter variable at 1 to more closely emulate the Windows behavior.

Episode #51: Leapin Lizards! ARP Stuff!

It's Hal's turn in the Bucket:

Mr. Bucket was pointing out the other day that we haven't had much to say about viewing and manipulating the ARP cache from the command line, and that's a darn shame. So let's remedy that little oversight right now.

The ARP cache is the system's local mapping of ethernet addresses to IP addresses of machines on the same LAN. On Unix systems, you can use "arp -a" to dump the cache or just dump information for a particular host:

$ arp -a
gw.deer-run.com (192.168.1.1) at 00:04:23:5f:20:98 [ether] on eth0
deer.deer-run.com (192.168.1.2) at 00:30:48:7b:22:2e [ether] on eth0
$ arp -a gw
gw.deer-run.com (192.168.1.1) at 00:04:23:5f:20:98 [ether] on eth0

As with most network administration commands on Unix, the "-n" flag suppresses the hostname information, in case you're having name resolution problems that are causing the command to hang:

$ arp -an
? (192.168.1.1) at 00:04:23:5f:20:98 [ether] on eth0
? (192.168.1.2) at 00:30:48:7b:22:2e [ether] on eth0

Occasionally monitoring your ARP cache can be interesting, because it can alert you when system hardware is changing on your network and also to ARP spoofing attacks. Here's a little bit of shell fu to watch for changes to a particular ARP entry:

$ while :; do c=`arp -a 192.168.1.1 | awk '{print $4}'`; \
       [ "X$c" == "X$p" ] && echo "$c (OK)" || echo "$p to $c (CHANGED)"; \
       p=$c; sleep 5; done
to 00:04:23:5f:20:98 (CHANGED)
00:04:23:5f:20:98 (OK)
00:04:23:5f:20:98 (OK)
^C

Notice the equality test where I'm putting an "X" in front of the value of each variable ("X$c" == "X$p")? That's a common trick for doing comparisons where one of the values might be null-- as in our case when we first enter the loop and $p is not set yet. If we just wrote, "[ $c == $p ]", the shell would generate a syntax error message during the first pass through the loop because it would interpret our fu as "[ 00:04:23:5f:20:98 == ]" which is clearly bogus. The extra "X" ensures that neither side of the comparison is ever empty.

Actually, the above example is a little bogus (though it includes plenty of tasty shell fu), because there's already a tool called Arpwatch that will monitor your ARP cache for changes-- expected or otherwise. I'll often run this tool on a couple of machines on critical networks, just to look for signs of hanky panky.

If you're running with root privileges, you can also use the arp command to manually manipulate your ARP cache. For example, "arp -d" deletes ARP entries. However, since ARP dynamically relearns the deleted entries, they don't tend to stay deleted for long:

# arp -d deer
# arp -a
gw.deer-run.com (192.168.1.1) at 00:04:23:5f:20:98 [ether] on eth0
deer.deer-run.com (192.168.1.2) at  on eth0
# ping -c 1 deer >/dev/null
# arp -a
gw.deer-run.com (192.168.1.1) at 00:04:23:5f:20:98 [ether] on eth0
deer.deer-run.com (192.168.1.2) at 00:30:48:7b:22:2e [ether] on eth0

The root account can also use "arp -s" to add static ARP assignments to the cache. For example, I was recently debugging a VPN problem for one of my customers and we suspected the Open Source VPN software we were running on our Unix system wasn't properly doing Proxy ARP for the remote clients. So we manually added an ARP entry for the client we were testing with that pointed to the MAC address of our gateway (note that you must do this on the gateway machine that is the owner of the specified MAC address):

# arp -s 192.168.1.202 00:04:23:5f:20:98 pub
# arp -an
? (192.168.1.2) at 00:30:48:7b:22:2e [ether] on eth0
? (192.168.1.202) at  PERM PUB on eth0

The extra "pub" argument means that the host should "publish" this static ARP entry-- that is, respond with our local MAC address when other hosts on the LAN make ARP requests for 192.168.1.202. In other words, our gateway should perform Proxy ARP for this address.

Since we did not specify the "temp" option in the above command, this static ARP entry will persist until we reboot the machine. If you specify "temp", then the entry will time out as normal when your system flushes its ARP cache. Generally, though, if you're specifying static ARP entries, you're doing it for a reason and you want them to stick around. For example, some highly secure sites will populate static ARP entries for all authorized hosts on their local LANs in order to thwart ARP spoofing attacks (static ARP entries will never be overwritten by dynamic ARP information learned from the network). Of course, the static ARP entries will need to be reloaded every time the system reboots, so you'll need to create a boot script to automatically reload the entries from a configuration file. And you'd need to update the configuration file every time a host was added or a NIC was changed on the LAN. It's a huge hassle and not normally done in practice.

That's about it for ARP on Unix. If memory serves, things aren't too different on Windows, but I bet Ed has some tasty fu for us anyway...

Ed responds to Hal's Bucket Screed:
As Hal anticipated, the arp command on Windows is very similar to the Linux version, but there are a few annoying deltas. To dump the ARP cache, we can run:

C:\> arp -a

Interface: 10.10.10.15 --- 0x10003
Internet Address      Physical Address      Type
10.10.10.45           00-0c-29-c2-9f-1b     dynamic
10.10.11.11           de-ad-be-ef-00-00     static
10.10.75.1            00-0c-29-a3-2c-8b     dynamic

There are a couple of noteworthy things here: First off, we don't see host names. The Windows arp command works exclusively with IP addresses and not names. Thus, we don't have a -n option, and cannot enter domain names of the machines at command line. If we want to zoom in on individual machines, we need to enter an IP address, not a name:

C:> arp -a 10.10.11.11

Interface: 10.10.10.15 --- 0x10003
Internet Address      Physical Address      Type
10.10.11.11           de-ad-be-ef-00-00     static

Next, note that we have that "Type" column, which tells us whether an entry is dynamic or static. This is handy, because we can list all static ARP entries using:

C:\> arp -a | find "static"

Next, note that the Windows arp command uses dashes between octets in the MAC address, and not colons. This is annoying, and I often type it wrong before I type it right.

Hal's ARP-monitoring command is fun and useful, so I tried to mimic its functionality almost identically on Windows, coming up with the following command (brace yourselves):

C:\> cmd.exe /v:on /c "for /l %i in (1,0,2) do @for /f "skip=3 tokens=2" %i in
('arp -a 10.10.10.45') do @set current=%i & (if X!current!==X!previous!
(echo %i ^(OK^)) else echo !previous! to %i ^(CHANGED^)) & set previous=%i
& ping -n 6 127.0.0.1 > nul)"

!previous! to 00-0c-29-c2-9f-1b (CHANGED)
00-0c-29-c2-9f-1b (OK)
00-0c-29-c2-9f-1b (OK)
00-0c-29-c2-9f-1b (OK)
00-0c-29-c2-9f-1b (OK)
00-0c-29-c2-9f-1b  to 01-02-03-04-05-06 (CHANGED)
01-02-03-04-05-06 (OK)
01-02-03-04-05-06 (OK)
01-02-03-04-05-06 (OK)

To analyze this command, let's work our way from the outside in. I'm invoking a cmd.exe with delayed environment variable expansion so that I can store and compare some variables inside my command. I then have a FOR /L loop that counts from 1 to 2 in steps of zero to keep this puppy running forever. I add a 5-second delay at the end by pinging 127.0.0.1 six times. Now, we get to the core of the command. I run a FOR /F loop to parse the output of my arp command, skipping 3 lines of output to zoom in on what I want, grabbing the second item (the MAC address), which I place into iterator variable %i. At each step through the loop, I set the variable "current" to my iterator variable so I can compare it against my previous MAC address for that ARP table entry with an IF statement. IF statements don't like to deal with FOR loop iterator variables, so I have to move my %i value into "current" for comparison. I then compare it with the previous value, using Hal's little X prepend trick so they are never empty strings. If they match, I print out the MAC address (%i) and the "(OK)" just like Hal does. I have to put ^'s in front of my parens so that echo knows to display them. If the current and previous MAC address from that arp entry do not match, I print out that the previous changed to %i followed by "(CHANGED)". Finally, I store the current value of %i as my "previous" variable and then wrap around. It's not elegant, but it gets the job done, and looks very similar to the output and functionality of Hal's command.

Lessee... what other stuff does Hal have for us?

Yup... we have "arp -d" to delete an ARP entry:

C:\> arp -d 10.10.10.45

And, we also have arp -s to plant a static ARP entry:

C:\> arp -s 10.10.10.45 01-02-03-04-05-06

Unfortunately, the Windows arp command does not have a "pub" option like the Linux command to implement proxy ARP functionality.

By the way, this whole discussion of the arp command was triggered by Byte Bucket's comment to Hal and me in our discussion of scheduling tasks. Byte (if I can be so informal) mentioned that on Windows, to define a static ARP entry, Microsoft recommends that you schedule a startup task, by saying, "To create permanent static ARP cache entries, place the appropriate arp commands in a batch file and use Scheduled Tasks to run the batch file at startup." This is a good idea if you ever want to hard code your arp tables to avoid ARP cache-poisoning attacks. Of course, in their recommendation, Microsoft never tells you how to actually do this. Well, never fear... Command Line Kung Fu is here!

We start by creating our batch file for the static ARP entry:

C:\> echo arp -s 10.10.10.45 aa-bb-cc-dd-ee-ff > C:\static_arp.bat

You can add more static arp entries into this file if you'd like, echoing in one arp command per line, appending to the file with >>. I'm storing it right in c:\ itself for easy spotting, although you may find that tucking it away somewhere you keep your admin scripts is a less clutterful approach. Then, we make that a startup task using the schtasks command, which we described in our last episode (#50):

C:\> schtasks /create /sc onstart /tr c:\static_arp.bat /tn static_arp

And there you have it! We've seen how to make a little arp-cache watcher in a single command, as well as how to implement static arp entries at system startup. Good topic, Hal!

UPDATE: Diligent reader Curt points out that in Windows Vista and 7, you can add a static ARP entry using the netsh command, with the following syntax:

C:\> netsh interface ipv4 add neighbors "Local Area Connection" 10.1.1.1 12-34-56-78-9a-bc

And, I'm delighted to report that this will survive a reboot, so no scheduled task startup script is required. However, this neighbors context for netsh only appears in Vista, 2008, and Win7. In XP or 2003, you can go with the schtasks method I outline above. Thanks for the input, Curt!