Many times on our little blog here, I've spoken appreciatively of some of my favorite Windows command-line friends, including wmic, netsh, and sc. Yes, I've fought the desire to personify them, but I can't help but think of them as buddies... Buddies with annoying quirks, a bad attitude, and flaws that sometimes let you down, but compadres nonetheless.
While spending some time over the holidays hanging out with these trustworthy sidekicks, something occurred to me: We haven't spent nearly enough time on this blog with the good ol' netstat command. Sure, we touched upon him a little bit in analyzing protocol statistics, checking out whether we've got a SYN flood, and even as an example of ways to get help from Windows commands. But, so far, I've truly let one of my most faithful and useful commands down by not giving him the spotlight he deserves. So, without further adieu, let's look at some really useful invocations of netstat:
Detecting when a scan reaches a given target box: So, Mike Poor and I were at a client's facilities conducting a large-scale vulnerability scan of thousands of hosts. Mike was running the scanning tool from the corporate headquarters, while I was located at another building sitting with the client in front of one of their Windows servers that was included in Mike's scan. The client and I wanted to know when Mike's scan would reach us. I asked the client if I could install a sniffer on their server, or connect my laptop to a span port on the switch so I could use my own sniffer to see when Mike's scan reached us. "No dice," said the client, who wasn't authorized to let us install any software or get access to the switch. I asked the client if we could logon to the Windows server and run a simple single netstat command that would show us when Mike's scan reached us, with approximately 1 second accuracy. He said, "Sure thing," so I ran:
C:\> netstat -na | find /i "Listening"
Here, I'm just checking to make sure there is at least one listening TCP port, piping netstat's output through the find command in a case-insensitive fashion (/i) because I didn't want to hold down the shift-lock key to type LISTENING. Sure enough, on this Windows server, I saw several listening ports, including TCP 445. When Mike's scanner reached us, it would open a connection to that port.
Then, I ran:
C:\> netstat -na 1 | find "[MikePoorIPaddr]"
That little space followed by a 1 means that Netstat will re-run approximately every 1 second. I scrape through its output looking for the string of Mike's address. If I see no output, there is no connection from Mike. When I start seeing output, I'll know that Mike's scan has reached me. Sure, this isn't perfect. It's quite possible that the scanning tool will make and drop connections so quickly that they will fall in between the 1-second interval we're working with here. However, that's quite unlikely. And, sure enough, we were able to determine when Mike's scan reached us, and finished with us. We then headed for lunch.
Detecting when a piece of malware starts interacting with the network: I was in my lab, fully clothed, analyzing some malware. After it finished some initialization actions, this little gem would start listening on TCP port 47145. Again, I wanted to know when it would start to listen, with 1-second accuracy... I ran:
C:\> netstat -na 1 | find ":47145"
I then wanted to go a little further, and find when the bad guy actually made a connection to this listener. For that, I executed:
C:\> netstat -na 1 | find ":47145" | find /i "Established"
Finding the mysterious cause of an ICMP Host Unreachable Message: A student who was taking my SANS 504 class told me that he was getting a mysterious ICMP Host Unreachable message coming from their border firewall back into their network a couple times per day, and he wanted to know what was triggering it. I asked him what the destination address of the packet was. It was one of their Windows 2003 servers. So, the Windows 2003 server was trying to send a message out through the firewall, but the firewall was responding that it didn't know how to reach the destination machine. The student wanted to know which program on the Win2K3 box was sending the original message. I had him run:
C:\> netstat -nao 1 | find "[DestIPaddr]"
As long as that program was trying to make a TCP connection, we'd see output from netstat for the half open connection, and could see the process ID number (the -o in netstat gives me the PID of the process using the port). We were then able to run the wmic command to get all kinds of info about that process:
C:\> wmic process where processid="[pid]" list full
Playing process whack-a-mole: I was playing a king-of-the-hill style capture the flag game, and my adversaries were trying to listen on a given TCP port on the target box, which I had already compromised. I wanted to simply kill any process that started on that port as quickly as possible. The netstat invocations listed above are quite nice, but they don't really help much for this need. You see, putting the integer after the netstat invocation makes netstat run continuously, showing us its output every 1 second. We can scrape through that output with the find command, but we cannot then run any other command after that, such as a taskkill. That's because netstat never stops running, so we cannot kickoff another command. For this, we'll have to wrap things in a FOR /L loop to get our continuity of operation, and in a FOR /F loop, to parse out the PID from our netstat output, as follows:
C:\> for /L %i in (1,0,2) do @for /f "tokens=5" %j in ('netstat -nao ^| find ^":47145^"')
do @taskkill /f /PID %j
SUCCESS: The process with PID 3152 has been terminated.
SUCCESS: The process with PID 2432 has been terminated.
SUCCESS: The process with PID 4068 has been terminated.Here, I've simply set up a FOR /L loop to count from 1 to 2 in steps of zero to implement a continuous loop. Then, at each iteration through the loop, I run a FOR /F loop set to parse the fifth column (the PID) from the output of a netstat -nao command, whose output itself is filtered by the find command looking for our port. Once I've got my PID, I then use taskkill to forcefully (/f) kill that PID. The evil listening process cannot listen for very long now.
Note that I've eliminated the 1 second delay -- this thing will run as fast as it can, useful in a CtF game as long as you aren't worried about being a performance hog. If you want a delay, we can simply add "& ping -n 2 127.0.0.1>nul" inside our FOR /F loop to ping ourselves twice (which takes about 1 second), as follows:
C:\> for /L %i in (1,0,2) do @(ping -n 2 127.0.0.1>nul & for /f "tokens=5"
%j in ('netstat -nao ^| find ^":47145^"') do @taskkill /f /PID %j)
Create a real-time network sentinel by adding a date and timestamp and dumping process info: Now, let's really get jiggy with it. Instead of killing processes or using the above commands in an interactive mode, wouldn't it be nice if we could run a command that would just watch a given TCP port, and when it finds something using that port, it displays the date/time and various vital aspects of that process? Of course it would!
But, we'd only want it to print information when a new processID starts to listen on the given port. That way, if one process starts and listens for a while, and then stops, and another starts up, we would capture on our output the time and details of each process that interacted with the port, a diligent sentinel gathering data for us. That would allow us to run a command that can watch for given network activity over night, and then come back to see what happened and when it happened later. I use this concept and the resulting command a lot during investigations. Check this out:
C:\> cmd.exe /v:on /c "set status=0 & for /l %i in (1,0,2) do @for /f "tokens=5"
%j in ('"netstat -nao ^| find ":47145""') do @if NOT !status!==%j echo !date!
!time! & wmic process where processid="%j" list full & set status=%j"
Sun 01/03/2010 5:36:47.79
CommandLine=nc -l -p 47145
CSName=FRED2
Description=nc.exe
ExecutablePath=C:\tools\netcat\nc.exe
ExecutionState=
Handle=3116
HandleCount=33
InstallDate=
KernelModeTime=0
MaximumWorkingSetSize=1413120
MinimumWorkingSetSize=204800
Name=nc.exe
OSName=Microsoft Windows XP Professional|C:\WINDOWS|\Device\Harddisk0\Partition1
OtherOperationCount=208
OtherTransferCount=124718
PageFaults=475
PageFileUsage=540672
ParentProcessId=3584
PeakPageFileUsage=540672
PeakVirtualSize=14987264
PeakWorkingSetSize=1953792
Priority=8
PrivatePageCount=540672
ProcessId=3116
QuotaNonPagedPoolUsage=2376
QuotaPagedPoolUsage=29780
QuotaPeakNonPagedPoolUsage=2376
QuotaPeakPagedPoolUsage=29924
ReadOperationCount=5
ReadTransferCount=15977
SessionId=0
Status=
TerminationDate=
ThreadCount=1
UserModeTime=300432
VirtualSize=14987264
WindowsVersion=5.1.2600
WorkingSetSize=1953792
WriteOperationCount=1
WriteTransferCount=72
In this command, I start by running a cmd.exe with /v:on to enable delayed variable expansion. I'll need that, because I need to store status to track if the PID using the port changes. I then set my initial status to zero, since no process should have that PID. I then start a FOR /L loop to run everything continuously. Then, I run a FOR /F loop to invoke my netstat command, piping its output through the find command to look for that port (47145, although you could also look for an IP address, as shown earlier in the article). I take the fifth column of output, the PID, and store it in iterator variable %j. Then, in the do clause of my FOR /F loop, I use an IF command to see if my status has changed (if the status variable is NOT the same as my PID, it has changed since the last iteration). When I detect such a change, I print out the date and time (note that !status!, !date!, and !time! have the exclamation points because I want their values to float over time with delayed variable expansion). I also run wmic to get full details of the given process. Finally, I set my status to the current PID. That way, I can check to see if it changes in the next go-round. This may look complicated, but it is darn useful. I'm simply dumping output to the screen here, but you could append it to a results file if you'd like by using >>results.txt at the end.
As you can see, there are a lot of fun things my little friend netstat can do using these building blocks.
Hal jumps in:
It's always been interesting to me how much
$ netstat -na | grep -i listen
Note that the output that Ed wants to match in this example is "LISTEN" in the Unix universe, not "LISTENING".
One important difference between Unix and Windows netstat is the way you get netstat to run continuously rather than producing one set of output and then exiting. With the Unix version, you need to use "-c" to specify continuous mode:
$ netstat -nac | grep <Mike's IP addr>
The interval for "-c" is fixed at one second-- you can't specify your own interval like you can with Windows. If you want to use a less frequent interval, you'll have to write your own loop and use "sleep", like so:
$ while :; do netstat -na | grep <Mike's IP addr>; sleep 5; done
Frankly, though, I think I'd just use our old friend the "watch" command instead of either "netstat -c" or the loop option:
$ watch -n 1 'netstat -na | grep <Mike's IP addr>'
The watch command is not only quicker to type than the loop, it allows us to easily choose our preferred monitoring interval, which we can't do with "netstat -c".
But really I wouldn't use netstat for this at all as long as I was on a machine that had the lsof command installed. With lsof, not only can I do this in less keystrokes, but I'll also get more information than I would from the netstat. lsof is *my* little friend and the friend of Unix admins everywhere.
The downside is that you usually have to be root to get complete output from lsof:
# watch -n 1 lsof -nPi @<Mike's IP addr>
"lsof -i @<ipaddr>" show you all connections related to the specified IP address. As with netstat, "-n" suppresses IP address to hostname mappings. However, while "netstat -n" also suppresses port number to port name mappings, lsof wants you to use the "-P" option to show numeric port numbers.
We can also use lsof to do something similar to Ed's port monitoring examples:
# watch -n 1 lsof -nPi :47145
Notice that you prefix port numbers with a colon when using lsof, while IP addresses are prefixed with "@" as we saw above. This allows us to stack IP address and port combinations to be even more specific. For example, suppose I was interested in SSH connections to a particular box, I could write:
# watch -n 1 lsof -nPi tcp@<ipaddr>:22
The above example also shows how to specify a particular protocol type (usually "tcp" or "udp") along with the IP address and port information. If I just wanted SSH connections to all IP addresses, I could do:
# watch -n 1 lsof -nPi tcp:22
Another advantage to using lsof instead of netstat is that lsof's "-t" option makes "process whack-a-mole" dead easy:
# while :; do kill -9 `lsof -t -i :47145`; done
As we discussed back in Episode #69, "lsof -t" just outputs the PIDs of the matching processes. This means it's easy to kill those processes by using backticks and the kill command as we're doing here.
Ed's final monitoring loop is interesting. Here's how I'd do something similar on a Linux machine:
# while :; do
pid=`lsof -ti :47145`
[[ "X$pid" == "X" || "X$pid" == "X$lastpid" ]] && continue
lastpid=$pid
echo -n '*** '
date
ps e --cols 1600 -fl -p $pid
lsof -p $pid
done
I'm first getting the PID of the process we're interested in using "lsof -t...". If there's currently no PID associated with this port, or if the current PID is the same as the previous time we ran the loop then we just start the loop all over again.
Otherwise I update the value of $lastpid and output some info about the new process. You could choose to use whatever commands you want, but here I start by throwing out some asterisks with "echo -n" so that it's easier to find the start of each section of output. Then I follow that up with the date command to get a timestamp. Since I used "echo -n", the date output will appear on the same line as the asterisks.
Then I use the ps command to dump some information about the process. "ps e" will dump any environment variables that are set in the process context. Since this output tends to be long, I also add the "--cols 1600" option to specify the screen width as a larger value so that the output doesn't get truncated (on narrower displays, the lines will simply wrap). I also use "-fl" to get the most detailed output possible. The "-p" option is used to specify the PID we're interested in. While the ps command will give us plenty of output, I'm also using lsof to dump information about all of the open files associated with the process.
It's worth noting that both Ed's loop and mine assume that only one process at a time is ever going to be associated with the port we're monitoring. But if the port is a back-door login port, you might actually have multiple connections to the port at the same time. I'd finesse this in bash by using an array to track multiple PIDs at the same time, but I think it would be next to impossible for Ed to deal with this in cmd.exe.
Tim leaps in:
Of course we can use netstat in PowerShell, but it would be mostly redundant (with the exception of the all of Ed's For Loops). Also, Hal seems upset about Windows "stealing" netstat so let's try the same thing in PowerShell without using netstat. Without netstat we will have to use the .NET framework available to us through PowerShell.
Here is the PowerShell and .Net equivalent of the first netstat command.
PS C:\> ([Net.NetworkInformation.IPGlobalProperties]::
GetIPGlobalProperties()).GetActiveTcpListeners()
AddressFamily Address Port
------------- ------- ----
InterNetwork 0.0.0.0 80
InterNetwork 0.0.0.0 135
InterNetwork 0.0.0.0 445
InterNetwork 192.168.1.5 139
...
Most of the information we are going to need is accessed via the IPGlobalProperties object's methods and properties. Since we are going to be using it a lot let's dump it into a variable so we can save keystrokes in future commands.
PS C:\> $a = [Net.NetworkInformation.IPGlobalProperties]::GetIPGlobalProperties()
Now we can run the first command again like this:
PS C:\> $a.GetActiveTcpListeners()
The name of this method is pretty self explanatory, it returns the active TCP listeners. Not much to talk about, so we'll move on.
The next task is to find out when Mike connects.
PS C:\> while (1) {$a.GetActiveTcpConnections() | ?
{$_.RemoteEndPoint.Address -eq "[MikePoorIPaddr]"} }We use a while loop that will run forever since "1" is always true (except for very small values of 1). Then we get the active TCP connections and look for Mike's IP address.
This will really hit the processor, so we probably should put in a delay of a at least a half a second.
PS C:\> while (1) {$a.GetActiveTcpConnections() | ?
{$_.RemoteEndPoint -like "[MikePoorIPaddr]:*";
sleep -milliseconds 500}}...or maybe two seconds
PS C:\> while (1) {$a.GetActiveTcpConnections() | ?
{$_.RemoteEndPoint -like "[MikePoorIPaddr]:*";
sleep 2}}So no we move on to the malware that Ed was analyzing. In the example, the port to be monitored is TCP 47145.
PS C:\> $a.GetActiveTcpListeners | ? {$_.Port -eq 47145}We get the active tcp listeners and then filter the results for connections where the local port is 47145.
Now we have the problem of killing the process. Without using netstat we don't have a way to determine the PID. It looks like we will have to cave in an use our old friend netstat (even though Hal is trying to poison this friendship).
PS C:\> netstat -ano | ? {$_ -like "*:47145 *"} |
% { $_ -match "\d+$"; stop-process $matches[0] }The output of netstat is filtered for connections on port 47145. Using a regular expression inside the ForEach-Object loop we look for a number at the end of the line. In regular-expression-land \d+ means at least one numeric digit and $ means it has to be at the end of the line. The number that we found is represented by the $matches object so we use the first and only match, $matches[0], with stop-process to kill it.
Ed's grand finale, the monitoring loop done in PowerShell.
PS C:\> while(1) {netstat -ano | ? {$_ -like "*:47145 *"} |
% {
$_ -match "\d+$";
Get-Process -id $matches[0] | Format-List *;
(Get-Process -id $matches[0]).WaitForExit()
}
}While it ain't pretty, I think we can safely say that the PowerShell version is the easiest to read of the bunch (finally), but I did add a few extra line breaks and spaces to make it even easier to read. How does it work?
First, we have our friendly infinite while loop. Nested inside the loop is our netstat command and our filter to search for port 47145. As before, we use the match operator to find the PID. Then we get all of the properties for the process. Piping the Get-Process command into Format-List * will display all the properties, not just the most commonly used properties. Finally we wait for the process to end and then our loop continues from the beginning. By using the WaitForExit method we know when that process dies and we can look for a new process. We don't have to worry about finding, keeping track of, and comparing the PID. It makes it much easier and cleaner. It also has the added benefit of being very light on system resources while it is waiting.