Similar to last week, this week's challenge comes from Tim's friend who is mentoring a CCDC team. The mentor was interested in creating some shell fu that lets them monitor all network connections in and out of a system and get information about the executable that's handling the local side of the connection. The kind of information they're looking for is the sort of thing you'd get from the output of "ls -l": permissions, ownership, file size, MAC times, etc.
Truthfully, I got a sinking feeling when I heard this request. We already established back in Episode #93 how nasty it can be to determine the path name of an executable from the output of lsof if you want to do it in a way that's portable across a wide number of Unix-like systems. But let's adapt what we learned in Episode #93 to get the executable names we're interested in:
# for pid in $(lsof -i -t); do
lsof -a -p $pid -d txt | awk '/txt/ {print $9}' | head -1;
done
/usr/sbin/avahi-daemon
/usr/sbin/sshd
/sbin/dhclient3
/usr/sbin/mysqld
/usr/sbin/cupsd
/opt/cisco/vpn/bin/vpnagentd
/usr/lib/apache2/mpm-worker/apache2
/usr/lib/apache2/mpm-worker/apache2
/usr/lib/apache2/mpm-worker/apache2
/usr/sbin/ntpd
/usr/lib/firefox-3.6.12/firefox-bin
/usr/bin/ssh
/usr/bin/ssh
In the for loop, "lsof -i -t" tells lsof to just print out the PIDs ("-t") of the processes that have active network connections ("lsof -i"). We then use the trick we developed in Episode #93 to get the binary name associated with each process ID.
Of course, you'll notice that there are multiple instances of executables like ssh and apache2, and we probably don't want to dump the same information multiple times. A little "sort -u" action will fix that right up:
# for pid in $(lsof -i -t); do
lsof -a -p $pid -d txt | awk '/txt/ {print $9}' | head -1;
done | sort -u | xargs ls -l
-rwsr-xr-x 1 root root 1719832 2010-02-17 16:17 /opt/cisco/vpn/bin/vpnagentd
-rwxr-xr-x 1 root root 443472 2010-01-26 20:35 /sbin/dhclient3
-rwxr-xr-x 1 root root 333464 2009-10-22 12:58 /usr/bin/ssh
-rwxr-xr-x 1 root root 478768 2010-08-16 10:42 /usr/lib/apache2/mpm-worker/apache2
-rwxr-xr-x 1 root root 51496 2010-10-27 06:37 /usr/lib/firefox-3.6.12/firefox-bin
-rwxr-xr-x 1 root root 119032 2010-09-22 11:03 /usr/sbin/avahi-daemon
-rwxr-xr-x 1 root root 416304 2010-11-02 11:24 /usr/sbin/cupsd
-rwxr-xr-x 1 root root 9943440 2010-11-09 21:19 /usr/sbin/mysqld
-rwxr-xr-x 1 root root 548976 2009-12-04 11:03 /usr/sbin/ntpd
-rwxr-xr-x 1 root root 441888 2009-10-22 12:58 /usr/sbin/sshd
Once I use "sort -u" to produce the unique list of executable names, I just pop that output into xargs to get a detailed file listing about each executable.
I would say that this meets the terms of the challenge, but the output left me rather unsatisfied. I'd really like to see exactly what network connections are associated with each of the above executables. So I decided to replace xargs with another loop:
# for pid in $(lsof -i -t); do
lsof -a -p $pid -d txt | awk '/txt/ {print $9}' | head -1;
done | sort -u |
while read exe; do
echo ===========;
ls -l $exe;
lsof -an -i -c $(basename $exe);
done
===========
-rwsr-xr-x 1 root root 1719832 2010-02-17 16:17 /opt/cisco/vpn/bin/vpnagentd
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
vpnagentd 2314 root 12u IPv4 8634 0t0 TCP 127.0.0.1:29754 (LISTEN)
===========
-rwxr-xr-x 1 root root 443472 2010-01-26 20:35 /sbin/dhclient3
===========
...
My new while loop reads each executable path and generates a little report-- first a separator line, then the output of "ls -l", and then some lsof output. In this case we have lsof dump the network information ("-i") related to the given command name ("-c"). However, the "-c" option only wants the "basename" of the commmand and not the full path name. The "-a" option says to join the "-i" and "-c" requirements with a logical "and" and "-n" suppresses mapping IP addresses to host names.
But what's up with the dhclient3 output? Why are we not seeing anything from lsof?
# lsof -an -i -c /dhcli/
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
dhclient 1725 root 5w IPv4 6266 0t0 UDP *:bootpc
Broadening our search a little bit by using the "/dhcli/" syntax to do a substring match, you can now see in the lsof output that the "command name" as far as lsof is concerned is "dhclient", and not "dhclient3". It turns out that on this system, /sbin/dhclient is a symbolic link to /sbin/dhclient3, so there's a disconnect between the executable name and the name that the program was invoked with.
Well that's a bother! But I can make this work:
# for pid in $(lsof -i -t); do
lsof -a -p $pid -d txt | awk '/txt/ {print $9,$1}' | head -1;
done | sort -u |
while read exe cmd; do
echo ==========;
ls -l $exe;
lsof -an -i -c $cmd;
done
==========
-rwsr-xr-x 1 root root 1719832 2010-02-17 16:17 /opt/cisco/vpn/bin/vpnagentd
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
vpnagentd 2314 root 12u IPv4 8634 0t0 TCP 127.0.0.1:29754 (LISTEN)
==========
-rwxr-xr-x 1 root root 443472 2010-01-26 20:35 /sbin/dhclient3
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
dhclient 1725 root 5w IPv4 6266 0t0 UDP *:bootpc
==========
...
If you look carefully, the awk statement in the first loop is now outputting the executable path followed by the command name as reported by lsof ("print $9,$1"). So now all my second loop has to do is read these two values into separate variables and call ls and lsof with the appropriate arguments. This actually saves me calling out to basename, so it's more efficient anyway (and probably what I should have done in the first place).
Whew! That one was all kinds of nasty! I wonder how Tim will fare this week?
Tim set himself up:
I can tell you right now, I didn't fare well on this one. I had initially suggested this topic, but it was such a pain and was borderline scripting. I wanted to nix it, but no, Hal wanted to torture me. Merry friggin' Christmas to you too Hal.
Let's start off with cmd. We have the netstat command and we can use it to see what executable is involved in creating each connection or listening port.
C:\> netstat -bn
Active Connections
Proto Local Address Foreign Address State PID
TCP 192.168.1.10:2870 11.22.33.44:443 ESTABLISHED 172
[ma.exe]
TCP 192.168.1.10:1420 99.88.77.66:80 CLOSE_WAIT 5408
[firefox.exe]
...
This is nice since it gives us the IP Addresses and ports in use, as well as the name of the executable. The problem is, it doesn't give us the full path. Since we don't have the full path we either have to search for the executable (big pain) or just go off of the name. Not a great solution since two executables can have the same name but be in different directories. We need a different approach, what if we use netstat in a For loop with tasklist?
C:\> for /F "tokens=5 skip=4" %i in ('netstat -ano') do @tasklist /V /FI "PID eq %i"
Image Name PID Session Name Session# Mem Usage Status User Name CPU Time Window Title
============ ==== ============ ======== ========= ======= ============================ ======== ============
svchost.exe 1840 Console 0 5,084 K Running NT AUTHORITY\NETWORK SERVICE 0:00:01 N/A
Image Name PID Session Name Session# Mem Usage Status User Name CPU Time Window Title
============ ==== ============ ======== ========= ======= ============================ ======== ============
ma.exe 172 Console 0 3,372 K Running NT AUTHORITY\SYSTEM 0:00:10 MicroAgent
Image Name PID Session Name Session# Mem Usage Status User Name CPU Time Window Title
============ ==== ============ ======== ========= ======= ============================ ======== ============
svchost.exe 1768 Console 0 5,608 K Running NT AUTHORITY\SYSTEM 0:00:00 N/A
Image Name PID Session Name Session# Mem Usage Status User Name CPU Time Window Title
============ ==== ============ ======== ========= ======= ============================ ======== ============
svchost.exe 1840 Console 0 5,084 K Running NT AUTHORITY\NETWORK SERVICE 0:00:01 N/A
...This is our good ol' For loop we've used a bunch of times. The loop takes the output of "netstat -ano", skips the four header lines and sets %i to the Process ID. We then use the Process ID with the tasklist command's filter to get the information on the process. The /V switch is used to get additional information on the process, but it still doesn't give us the full path. Bah, humbug. Cmd get's a lump of coal. Let's see if PowerShell can do anything better!
First off, PowerShell doesn't have a cmdlet that gives us a nice version of netstat. Off to a bad start already.
I use the Get-Netstat script (yep, I said script) to parse netstat output for use with PowerShell.
PS C:\> Get-Netstat | ft *
Protocol LocalAddress Localport RemoteAddress Remoteport State PID ProcessName
-------- ------------ --------- ------------- ---------- ----- --- -----------
TCP 0.0.0.0 135 0.0.0.0 0 LISTENING 1840 svchost
TCP 0.0.0.0 445 0.0.0.0 0 LISTENING 4 System
TCP 0.0.0.0 912 0.0.0.0 0 LISTENING 2420 vmware-authd
TCP 0.0.0.0 3389 0.0.0.0 0 LISTENING 1768 svchost
...
We can use the Add-Member cmdlet to extend this object to add a Path property.
PS C:\> Get-Netstat | % { Add-Member -InputObject $_ -MemberType NoteProperty -Name Path
-Value (Get-Process -Id $_.PID).Path -Force -PassThru }
Protocol : TCP
LocalAddress : 0.0.0.0
Localport : 912
RemoteAddress : 0.0.0.0
Remoteport : 0
State : LISTENING
PID : 2420
ProcessName : vmware-authd
Path : C:\Program Files\VMware\VMware Player\vmware-authd.exe
Protocol : TCP
LocalAddress : 0.0.0.0
Localport : 3389
RemoteAddress : 0.0.0.0
Remoteport : 0
State : LISTENING
PID : 1768
ProcessName : svchost
Path : C:\WINDOWS\system32\svchost.exe
...The Add-Member cmdlet takes the current object sent down the pipeline ($_) as its input object. We then use the MemberType switch to specify a NoteProperty (static value) along with the name and value. The Force option has to be used for some silly reason, or else PowerShell complains the property already exists (which it doesn't). Finally, the PassThru switch is used to send our object down the pipeline.
We can use this approach to add as many properties as we would like. Let's add the CreationTime from the executable.
PS C:\> Get-Netstat | % { Add-Member -InputObject $_ -MemberType NoteProperty -Name Path
-Value (Get-Process -Id $_.PID).Path -Force -PassThru } | % { Add-Member -InputObject $_
-MemberType NoteProperty -Name CreationTime -Value (ls $_.Path).CreationTime -Force -PassThru }
Protocol : TCP
LocalAddress : 0.0.0.0
Localport : 912
RemoteAddress : 0.0.0.0
Remoteport : 0
State : LISTENING
PID : 2420
ProcessName : vmware-authd
Path : C:\Program Files\VMware\VMware Player\vmware-authd.exe
CreationTime : 11/11/2010 11:11:11 PMAdding more properties makes our command significantly longer. But, it does have the added benefit of everything being an object. We can export this or do all sorts of filtering. However, this would obviously be better suited as a...<gulp>...script...