Tuesday, August 25, 2009

Episode #57: At Your Services

Ed embarks:

Sometimes, administrators and users need to alter the state of running services on a Windows box, especially those services that automatically start during system boot. Many users are familiar with the services control panel GUI, with its familiar "Startup Type" column and values of Manual, Disabled, and Automatic. The Automatic services, in particular, are those that startup when the system is booted, even if no one logs onto the machine.

As described in Episode #40, Ed's Heresy, you can launch the services control GUI at the command line using:

C:\> services.msc

Look through all those services and their startup types, and you'll see an awful lot of automatic ones. Unlike Linux, Windows doesn't have the concept of different run-levels which start different sets of services. Instead, all automatic services are... well... automatically started at bootup. Vista and later did introduce a new option for automatic services startup type, called "Automatic (Delayed start)", which causes Windows to activate a service after all of the initial boot activities are completed, making bootup faster. So, I guess we do have a bit of fine-grained differentiation in bootup services: those that are done during the boot itself [Startup Type= Automatic] and those that happen right afterward [Startup Type= Automatic (Delayed Start)]. Of course, "Manual" services are initiated by user action, such as clicking the "Start" button in the services.msc GUI.

But, speaking honestly and bluntly, why use the crappy services.msc GUI when we can interact with our services at the command line? One of the real gems of the Windows command line is the Service Control command, known as "sc" for short. In the olden days, we could use the "net start" command to get a list of running services. But, good old "net start" pales in comparison to the mighty sc. I only use "net start" when I'm on a Windows 2000 box that lacks the sc command from the Resource Kit. I'm happy to report that sc is included in XP Pro and later.

We can use sc to get a list of all services on the box, regardless of their state, by running:

C:\> sc query state= all

Please note that sc is finicky about spaces in its command options. You have to enter it this way: "state equals space all". If you omit the space, it doesn't work. If you put a space before the equals, it doesn't work. Annoying, to be sure, but once you get the hang of it, it's almost tolerable. In fact, that could be the new marketing line for Windows: "Once you get the hang of it, it's almost tolerable." Microsoft marketing reps can feel free to contact me if they'd like to license that line.

Anyway, we can get more detail about each service's state using:

C:\> sc queryex state= all

The queryex there gives us additional information, including the PID each service is running inside of. If you'd like to focus on a single service, looking at all of its details, you could run:

C:\> sc qc [SERVICE_NAME]

The qc stands for "query configuration", and shows all kinds of neat stuff, including the binary path and command line flags used to launch the executable.

That [SERVICE_NAME] option used in the sc command must be the "SERVICE_NAME" and not the "DISPLAY_NAME", both of which are shown in the "sc query" output. The former is the internal name of the command used within Windows, while the latter is a more human-friendly form of the name used in the GUI. Most Windows admins think in terms of DISPLAY_NAME, but the sc command thinks otherwise. To map the DISPLAY_NAME in your head to SERVICE_NAME, you could use WMIC to map one to the other:

C:\> wmic service where displayname="[DISPLAY_NAME]" get name

You can even use substring wildcards on DISPLAY_NAME here with:

C:\> wmic service where (displayname like "%something%") get name

OK, now, fully armed with the sc-friendly SERVICE_NAME, we can proceed. If you want to stop or start a service immediately, you can use this simple command:

C:\> sc [start|stop] [SERVICE_NAME]

Please note, however, if you stop a service whose type is "Automatic" or "Automatic (Delayed Start)", that service will come back the next time you reboot. To change that behavior, you'll have to alter the startup type of the service, making it either manual or disabled. To do so, you could use the sc command as follows:

C:\> sc config [SERVICE_NAME] start= demand

Note that the GUI refers to such services as "manual", but the sc command calls them "demand". Consistency, thy name is Windows.... NOT! (Hey! There's another potential marketing campaign!) Note that if the service is currently running, this change in its configuration won't stop it... we are merely changing its configuration, not its current state.

To configure a service to be disabled at next boot, you could run:

C:\> sc config [SERVICE_NAME] start= disabled

For Automatic, use "start= auto" and for Automatic (Delayed Start), use "start= delayed-auto".

Note that there are two other service start types: boot and system. These are for device drivers, and I wouldn't mess with them unless you first create a snapshot of your virtual machine. What? You are running Windows on _real_ hardware and not a VM? Yikes. Be very, very careful with your config!

Another nifty feature of the sc command is its ability to show us service dependencies, as follows:

C:\> sc enumdepend [SERVICE_NAME] [buffer_size]

Try this for the RPC Service as follows:

C:\> sc enumdepend rpcss

It'll show you the start of the list of services that depend on rpcss, but then complain that its default buffer for gathering this information is too small. You can specify a bigger buffer by putting an integer at the end for the output display buffer, such as 8092. How convenient it is that it allows you to specify your display buffer size. Yeah, right.

Another fine aspect of the sc command is that it can be used remotely, provided that you have admin-level SMB access of a remote system. Any of the sc commands listed above can be directed to a remote system by simply adding \\[IP_addr] right after the sc, as in:

C:\> sc \\[IP_addr] [other_options]

Oh, and one more thing... All those automatic services associated with the underlying operating system do start in a specific order that Microsoft has carefully planned. That order is stored in the registry as a list, and can be displayed using:

C:\> reg query hklm\system\currentcontrolset\control\servicegrouporder

The list looks hideous in the output of the reg command. If you'd like it to look a little prettier, you can open that registry key in regedit.

Hal wants to know where Ed gets off:

Oh rats. Welcome to another episode of "this would be simple if every flavor of Unix didn't do this a little bit differently". The first simplifying assumption I'm going to make is to throw the *BSD folks off the bus. Sorry guys, if you're reading this then you already know how boot services work on your OS and you don't need me to tell you.

For everybody else in the world, the boot service controls on your system have typically evolved from some knockoff version of the System V boot sequencing scheme. This means there's probably a directory on your system called /etc/init.d (or perhaps /etc/rc*/init.d) that contains a whole pile of scripts. Generally, each one of these scripts is responsible for starting and stopping a particular service.

One of the nice features of this system is that you can run the boot scripts manually to start and stop services. The scripts accept "start", "stop", and often the "restart" option as well:

# /etc/init.d/ssh stop
* Stopping OpenBSD Secure Shell server sshd [ OK ]
# /etc/init.d/ssh start
* Starting OpenBSD Secure Shell server sshd [ OK ]
# /etc/init.d/ssh restart
* Restarting OpenBSD Secure Shell server sshd [ OK ]

In general, using the /etc/init.d scripts is the preferred method for starting and stopping services-- as opposed to just killing them-- because the script may perform additional service-specific cleanup actions.

Now when the system is booting we need to be careful that services get started in the correct dependency order. For example, there's no point in starting network services like SSH and Apache until the network interfaces have been initialized. Boot sequencing is the job of the /etc/rc*.d (sometimes /etc/rc*/rc*.d) directories.

If you look into these directories, you'll find a bunch of files named Snn* and Knn*-- for example "S16ssh". These "files" are actually links back to the scripts in /etc/init.d. The numbers are used to make sure the scripts are run by the init process in the correct sequence, and there are usually gaps left in the numbering so that you can add your own scripts at appropriate points in the sequence. The leading "S" tells init to run the script with the "start" option to start the given sequence at boot time. "K" means kill or "stop".

So why are there lots of different rc*.d directories? The basic idea was that Unix systems were supposed to be able to boot to different "run levels", numbered 1-5, that enabled different levels of functionality. I'm old enough to remember a time when booting to run level 2 meant "multi-user mode" and run level 3 meant "multi-user mode plus network file sharing" (yes, I know, get me a walker and you kids stay off my lawn!). These days, the whole "run level" concept has gotten wildly confused. Many Linux systems use run level 3 for multi-user and run level 5 for "multi-user with GUI", but Ubuntu boots into run level 2 by default. Solaris boots using both the run level 2 and run level 3 scripts, which is just wacky.

The reason the whole run level question is relevant is that in order to enable or disable certain services from being started at boot time, you need to change the links in the appropriate rc*.d directory. To do that, you need to know what run level your system is booting into by default. Some systems have a "runlevel" command which will tell you what run level you're currently booted into. On other systems you'll need to find the default run level setting in /etc/inittab-- "grep default /etc/inittab" usually works. The alternative is to just change the links in all of the rc*.d directories, just to be on the safe side.

Say you're on a Ubuntu system and you're booting into run level 2. You "cd /etc/rc2.d" and start messing with the links. You can see the services that are started at this run level with a simple "ls S*". If you want to make changes, just remember that the init program ignores any script that doesn't start with an "S" or a "K". So one way to prevent a service from being started at boot time is to just rename the link. There are lots of different conventions people use for this: some people change the "S" links to "K" links, others give them names like "aaa*" (sorts at the beginning of the directory), "zzz*" (sorts to the end), or ".NO* (hides links from normal "ls"). You can also just remove the links.

The only problem with messing with the link names directly is that you'll often find that vendor patches and software updates will often restore the original links in your rc*.d directories. So after updating your system it's a good idea to check your rc*.d directories to make sure your changes haven't been reverted.

To deal with this problem, most Unix variants have some sort of tool that manages boot time configuration of services. The tool typically manages some extra "meta-data" associated with each boot script that says which scripts should be started at the different run-levels. For example, Red Hat derived systems (RHEL, Fedora, CentOS, etc) use a command-line tool called chkconfig (originally developed for IRIX) that uses meta-data stored in special comments at the top of each boot script. Debian has update-rc.d and sysv-rc-conf which just rename "S" links to "K" links on your behalf (and vice versa). Solaris has the XML horror that is the Service Management Framework (SMF). You'll need to read the docs for your particular flavor of Unix.