Episode #11 - Listing Files by Inode as a Proxy for Create Time

Hal Says:

One of the problems with classic Unix file systems (FFS, UFS, ext[23], etc) is that they don't track the creation time of files ("ctime" in Unix is the inode change time, not the creation time). However, forensically it's often very useful to know when a given file was created.

While there's no way to know the exact creation date of a file from file system metadata, you can use the assigned inode number of the file-- because inodes tend to be assigned sequentially-- as a proxy to figure out the relative creation dates of files in a directory:

$ ls -li /etc | sort -n
total 4468
1835010 drwxr-xr-x  5 root root      4096 Nov 23 10:04 lvm
1835011 drwxr-xr-x 10 root root      4096 Nov 23 10:04 sysconfig
1835013 drwxr-xr-x  8 root root      4096 Nov 23 10:01 X11
1835014 drwxr-xr-x  2 root root      4096 May 24  2008 rpm
1835018 -rw-r--r--  1 root root       435 Jul 14  2007 reader.conf
1835019 -rw-r--r--  1 root root       105 Jul 14  2007 modprobe.conf
...
1837339 -rw-r--r--  1 root root      2200 Jul 22  2008 passwd
1837348 -rw-r--r--  1 root root       814 Jul 22  2008 group
1867786 drwxr-xr-x  4 root root      4096 May 24  2008 gimp
1867804 drwxr-xr-x  2 root root      4096 Jul 14  2007 sane.d
1867868 drwxr-xr-x  7 root root      4096 Jul 22  2008 gdm
1867890 drwxr-xr-x  2 root root      4096 Jul 22  2008 setroubleshoot
1867906 drwxr-xr-x  3 root root      4096 Aug  8  2007 apt
1867925 drwxr-xr-x  3 root root      4096 Aug  8  2007 smart
1867929 drwxr-xr-x  5 root root      4096 Dec 11 14:24 raddb
1867954 drwxr-xr-x 10 root root      4096 Dec 15 09:03 vmware
1867972 drwxr-xr-x  2 root root      4096 Aug  8  2007 syslog-ng
1868042 drwxrwsr-x  2 root mailman   4096 Jul 22  2008 mailman
1868075 drwxr-x---  3 root root      4096 Jul 22  2008 audisp
1900546 drwxr-xr-x  2 root root      4096 Jul 22  2008 purple
1933364 drwxr-xr-x  2 root root      4096 Nov 23 14:08 vmware-vix
2293777 -rw-r--r--  1 root root    362031 Nov 23 14:04 services

At the top of the output you can see that the inodes are clustered tightly together, indicating these files were probably all created about the same time-- typically when the system was first installed. Towards the end of the output, however, you can see other "clusters" of inode numbers corresponding to groups of files that were created around the same time. In this case, these are mostly the configuration directories for software packages I added after the initial OS install.

Ed Responds:

"...A proxy to figure the relative creation dates of files"? Oh my... If I may indulge in a little trash talk, you'd think that a real operating system would have some better way of tracking file creation times than resorting to inode numbers.

Just to pick an alternative operating system at random off the top of my head, let's consider... um... Windows. Yeah, Windows.

Oh yes, we have file creation time, which can be displayed using the really obscure dir command.

In all seriousness, by default, the dir command displays file modification date and time. If you want it to display creation time, simply run it with the /tc option. The /t indicates you want to twiddle with the time field (yeah, it stands for "twiddle" ;). The options after it are c for creation date/time, a indicates last access, and w is for last written. For example:

$ dir /tc

Lot simpler than Hal's fu above, and it gets the job done.

Oh, and Hal wanted them sorted. Sadly, we don't have a numeric sort in Windows, just an alphanumeric one. But, that lament is for another day, because we can sort based on time stamp right within dir, as follows:

$ dir /tc /od

The /o indicates we want to provide a sort order, and we're sorting by date, oldest first. To reverse the order (newest first), use /o-d, with the minus reversing the date sort.

Episode #10 - Finding Names of Files Matching a String

Hal Says:

This is one of my favorite questions to ask job interviewees, so pay attention!

Question: How would you list the names of all files under a given directory that match a particular string? For example, list all files under /usr/include that reference the sockaddr_in structure.

Most interviewees' first approximations look like this:

$ find /usr/include -type f -exec grep sockaddr_in {} \;

The only problem is that this gives you the matching lines, but not the file names. So part of the trick is either (a) asking me if it's OK to look at the grep manual page or help text (which is really the response I'm looking for), or (b) just happening to know that "grep -l" lists the file names and not the matching lines:

$ find /usr/include -type f -exec grep -l sockaddr_in {} \;

The folks who really interest me, however, are the ones who also strike up a conversation about using xargs to be more efficient:

$ find /usr/include -type f | xargs grep -l sockaddr_in

How much faster is the xargs approach? Let's use the shell's built-in benchmarker and see:

$ time find /usr/include -type f -exec grep -l sockaddr_in {} \; >/dev/null

real 0m12.734s
user 0m2.097s
sys 0m10.713s
$ time find /usr/include -type f | xargs grep -l sockaddr_in >/dev/null

real 0m0.410s
user 0m0.108s
sys 0m0.344s

You really, really want to use "find ... | xargs ..." instead of "find ... -exec ..."

Paul Says:

That's an awesome tip! I immediately put this to good use when using Metasploit. One of the requests we most often get from students when using metasploit is a way to find the exploit for a particular vulnerability. Metasploit has built in a search feature, but grep is far more powerful and comprehensive. Since all of the modules and exploits within metasploit are just Ruby files, you can use the method above to seek out functionality in Metasploit:

find ./modules/ -type f | xargs grep -li 'ms08\_*' | grep -v ".svn"

The above command will find all modules that contain references to "ms08", indicating an exploit for a vulnerability released by Microsoft in 2008.

Ed throws in his two cents:

On Windows, we have two string search tools: find and findstr. The latter has many more options (including the ability to do regex). We can use it to answer Hal's interview question with the /m option to print only the file name. Why /m? I guess because "Name" has an "m" in it, and /n was already taken to tell findstr to print line numbers.

So, the results is:

C:\> findstr /d:[directory] /m [string] [files]

The [files] lets you specify what kind of files you want to look in, such as *.ini or *.txt. To look in any kind of file, just specify a *. Also, to make it recurse the directory you specify, add the /s option.

How about an example? Suppose you want to look in C:\windows and its subdirectories for all files that contain the string "mp3". You could run:

C:\> findstr /s /d:c:\windows /m mp3 *

Another useful feature of findstr is its ability to find files that contain only printable ASCII characters using the /p flag. That is, any file with unprintable, high-end ASCII sequences will be omitted from the output, letting you focus on things like txt, inf, and related simple text files often associated with configuration:

C:\> findstr /s /p /d:c:\windows /m mp3 *

Be careful with the /p, however. You may be telling findstr to leave out a file that is important to you simply because it has one high-end ASCII sequence somewhere in the file.

Also, thanks, Hal, for now making me lust after not only xargs, -exec, ``, head, tail, awk, sed, and watch. Now, I really want a real "time" command in Windows. And, no, I'm not talking about the goofy built-in Windows time command that shows you the time of day. I'm talking about seeing how long it took another command to run. Thank goodness for Cygwin!

Episode #9 - Stupid Shell Tricks: Display the Nth Line

Hal Says:

Here's a stupid little shell idiom. How do you print only the nth line of a file? There are only about a dozen ways to do this in the Unix shell, but the one programmed into my wetware is:

 $ head -<n> <file> | tail -1

Paul Responds:

I'm kind of partial to awk, awk is my friend, its quick, dirty, and powerful (and I seem to learn about new techniques all the time, which makes it fun!):

 $ awk 'FNR == 42' file

Also, I like this command because its shorter, and despite popular belief UNIX people don't really like to type :)

Ed adds a little Windows perspective:

Ahhh... what I wouldn't give for head or tail in Windows. That sounds like a new motto for a Pauldotcom T-shirt or bumper sticker.

You can get most of the functionality you are describing here in Windows using the following construct:

C:\> find /v /n "" <file> | findstr /b /L [<n>] 

This may look crazy, but without head or tail, we need to trick Windows into doing what we want, as usual. What I'm doing here is using the find command to prepend line numbers (/n) to lines in the file that do not (/v) contain the string "". As we saw in Episode #3, searching for lines that do not have nothing shows all lines. Thus, the first portion of this command is actually prepending line numbers (in the form of [N], with the brackets) to each line in the file and sending them to standard out. I then pipe the result to the findstr command. The /b option tells findstr to display lines that have the string we are searching for at the beginning of a line. That way we won't get accidental collisions if [<n>] shows up inside of the file anywhere. We'll just be looking for the [<n>] that the find command prepended. I use a /L to indicate a literal string match. Otherwise, the double bracket around the n will confuse findstr.

The output is almost just what we want. There is one downside, though. There will be a [n] prepended to our line. But, that's pretty close, no?

Well, if you insist, you can remove that [n] with a FOR /F loop to do some parsing, but that starts to get really ugly if you just want to see the contents of the line. Anyway, because I luv ya, here goes:

C:\> find /v /n "" <file> | findstr /b /L [<n>] > temp.txt &
for /F "delims=[] tokens=2" %i in (temp.txt) do @echo %i & del temp.txt

Told you it was ugly. But, when you only have FOR /F loops to parse, you sometimes have to do this kind of thing.

Episode #8: Netstat Protocol Stats

Ed Says:

On Windows, the netstat command has really got a lot of features that are useful for analyzing network behavior. Even without installing a sniffer, you can learn a good deal about network traffic with a stock Windows machine by running netstat with the -s flag to see statistics for all supported protocols. You can select an individual protocol's stats with -p [proto], with TCP, UDP, ICMP, and IP supported. On Vista, they also added IPv6, ICMPv6, TCPv6, and UDPv6.

C:\> netstat -s -p ip

That'll show you IPv4 stats including packets received and fragments created.

C:\> netstat -s -p tcp

This one shows the number of active opens and reset connections, among other things. Those stats are useful if you suspect some kinds of denial of service attacks.

Hal Comments:

The Linux "netstat -s" command will also dump statistics:

$ netstat -s
Ip:
115851638 total packets received
237 with invalid headers
0 forwarded
0 incoming packets discarded
115825742 incoming packets delivered
72675668 requests sent out
2914 reassemblies required
1457 packets reassembled ok
Icmp:
34672 ICMP messages received
18 input ICMP message failed.
...

Unfortunately, while there are command line options to dump just the TCP or just the UDP statistics, they don't work consistently across different Linux distributions. In some cases they even include other protocol statistics, like IP and ICMP stats, along with the TCP or UDP stats.

I did come up with a fairly gross hack for pulling out sections of the report for a specific protocol:

$ netstat -s | awk '/:/ { p = $1 }; (p ~ /^Tcp/) { print }'
Tcp:
64684 active connections openings
25587 passive connection openings
1043 failed connection attempts
236 connection resets received
15 connections established
114808177 segments received
71655514 segments send out
24271 segments retransmited
11 bad segments received.
2906 resets sent
TcpExt:
1640 invalid SYN cookies received
15 ICMP packets dropped because they were out-of-window
57520 TCP sockets finished time wait in fast timer
...

The first part of the awk expression matches on the ":" character in the "header" line of each protocol section and sets our magic "p" variable to the current protocol name. That value remains in "p" until we reach the next header, and so on. The second part of the awk expression does a regular expression match against the current value of "p" and prints the current line as long as "p" matches the protocol we're looking for. That gets us the header line itself, plus all of the following lines of output up until the next header.

Why is this so clunky? Basically, Unix commands are generally poor at "remembering context" across multiple lines, so you often end up with these sorts of hacked solutions.

Paul Says:

As byte_bucket mentioned, things work a bit differently on OS X. Hal's command above needs to have a lower case "tcp" in order to work in OS X:

$ netstat -s | awk '/:/ { p = $1 }; (p ~ /^tcp/) { print }'

Also the following command:

$ netstat -s -p tcp

Works great on both Linux and OS X. I find these commands very useful for network troubleshooting, especially given slow performance or high error counts on the network switch.

Episode #7 - Aborting a System Shutdown

Ed says:

Sometimes, when using a Windows box, Really Bad Things (TM) happen, forcing the system to shut down. For example, if someone exploits the system, and their exploit accidentally kills lsass.exe or services.exe, Windows is very unhappy. It pops up a dialog box expressing its discontent, telling you that it will reboot in 60 seconds.

But, suppose you don't want it to reboot that quickly? Maybe you need just a little more time to save a file, close something out, launch your retaliatory missiles, or whatever. Most of the time, you can abort a shutdown by running:

C:\> shutdown /a

Of course, without an lsass.exe or services.exe process, the box is pretty well hosed. But, this command can give you a little bit of extra time in event of dire emergencies, limping along with a machine that is only partly dead. You can then make the box reboot on your own time frame with the following command:

C:\> shutdown /r /t [N_seconds]

If you omit the /t, it'll reboot in 30 seconds. Use /t 0 to make it reboot now.

Hal Comments:

I've always hated the Unix shutdown command. I find the "write all" behavior more annoying than useful. I normally use "reboot", "halt", or "init 0" (stop and power down). That being said:

# shutdown -c           # cancels scheduled shutdown
# shutdown -r +1        # shut down and reboot in 1 minute
# shutdown -r 14:30     # shut down and reboot at 2:30pm

Interestingly, you can't schedule shutdowns with finer than one-minute granularity, though I suppose you could do something like:

# sleep 30; shutdown -r now

Paul Comments:

Interesting to note that the OS X shutdown command does not have the "-c" option allowing you to halt the shutdown.

Episode #6 -- Command-Line Ping Sweeper

Ed Says:

Here's a Windows command to do ping sweeps at the command line:

C:\> FOR /L %i in (1,1,255) do @ping -n 1 10.10.10.%i | find "Reply"

Here, I've got a FOR /L loop, which is a counter. My iterator variable is %i. It starts at 1, steps up by 1 through each iteration through the loop, going up to 255. I want to ping through a /24-sized subnet. I then turn off command echo (@), and ping each IP address once (-n 1). I scrape through the output using the find command, looking for "Reply". The find command is case sensitive, so I put in the cap-R in "Reply". Or, you could use /i to make the find case insensitive.

By the way, you can speed it up by adding "-w 100" to have a timeout of 100 milliseconds between each ping, rather than the normal.

(Note... I had "-t 100" here earlier, but fixed it for "-w 100". Thanks to @bolbroe for the catch. The fact is, I so often use -t with Windows ping to make it keep pinging a la Linux, it feels very natural to put -t in. But, the issue here is to make it wait, with -w, for 100 milliseconds.)

Hal Comments:

I have to admit that my first impulse here was to respond with "sudo apt-get install nmap". But Ed's going to be a stickler for our "built-in shell commands only" rule, so I guess I have to come up with something else.

Here's a fun approach that's very different from Ed's loop:

# ping -b -c 3 255.255.255.255 >/dev/null 2>&1; arp -an | awk '{print $2}'

ping the broadcast address a few times and then scrape out your ARP table to get the IP addresses of the responding hosts (the old "ARP shotgun" approach). The only problem is that this only works for hosts on your local LAN.

So I guess my final solution is a lot like Ed's:

$ for i in `seq 1 255`; do ping -c 1 10.10.10.$i | tr \\n ' ' | awk '/1 received/ {print $2}'; done

By the way, notice the "tr \\n ' '" hiding in the middle of that shell pipeline? The problem is that the ping command generally produces multi-line output and I need to confirm that the packet was received (last line of output) before printing the IP address I pinged (first line of output). So I'm using tr to convert the multi-line output into a single line that's easier to tokenize with awk. This is a useful little shell programming idiom for your toolkit.

Ed stirs the pot a little bit more:

I like your broadcast ping approach. Nifty! Unfortunately, modern Windows boxen don't respond to broadcast pings. Thus, your command will find Linux and other machines on your same subnet, but not the Windows boxes. I tested it in my lab, and found all my Linux machines happily telling me about their existence, but my super stealthified (NOT!) Windows boxes were silent. Thus, while the broadcast ping is a nifty alternative for some special edge cases (targets on same subnet, don't care to find Windows boxes), I think the sweeper is the better way to go.

Episode #5 - Simple Text Manipulation - Reverse DNS Records

Paul Says:

There are many times when I run commands to collect information, such as hostnames and IP addresses, and the output is, well, less than desirable. For example, lets say that you have a file called "lookups.txt" that contains the following:

207.251.16.10.in-addr.arpa domain name pointer server1.srv.mydomain.net.
208.251.16.10.in-addr.arpa domain name pointer server2.srv.mydomain.net.

The output is not easy to read, so I like to manipulate it such that I get a list of IPs and hostnames:

$ awk -F . '{print $4 "." $3 "." $2 "." $1 " " $6 "."$7"."$8"."$9}' lookups.txt | cut -d" " -f1,6
10.16.251.165 server1.srv.mydomain.net.
10.16.251.166 server1.srv.mydomain.net.

Hal Comments:

The problem with your awk expression, Paul, is that you're assuming that all of the fully-qualified hostnames are four levels deep. What if your file also contains lines like:

16.254.16.10.in-addr.arpa domain name pointer www.mydomain.net.
17.254.16.10.in-addr.arpa domain name pointer mydomain.com.

The awk doesn't choke and die, but you do end up with weird output:

$ awk -F . '{print $4 "." $3 "." $2 "." $1 " " $6 "."$7"."$8"."$9}' lookups.txt | cut -d" " -f1,6
10.16.251.207 server1.srv.mydomain.net
10.16.251.208 server2.srv.mydomain.net
10.16.254.16 www.mydomain.net.
10.16.254.17 mydomain.com..

Yuck! Frankly, this looks like a job for sed to me:

$ sed 's/\([0-9]*\)\.\([0-9]*\)\.\([0-9]*\)\.\([0-9]*\).in-addr.arpa domain name pointer\(.*\)\./\4.\3.\2.\1\5/' \
        lookups.txt
10.16.251.207 server1.srv.mydomain.net
10.16.251.208 server2.srv.mydomain.net
10.16.254.16 www.mydomain.net
10.16.254.17 mydomain.com

sed expressions like this end up looking like nasty thickets of backwhacks, because of all the "\( ... \)" expressions, but this approach allows us to re-order the octets of the IP address and remove all of the extra text in one fell swoop.

And, yes, a lot of people (including me) would probably use Perl instead of sed for this, because Perl's regular expression syntax allows for a much more compact command line. But Paul, Ed, and I have agreed to avoid diving into pure scripting languages like Perl.

Paul (aka Grasshopper) Says:

Yes, I was assuming a static hostname, and wrote it as a one off to quickly parse my particular output. I now see that sed is even more powerful than I thought! This will certainly be a nice addition to some of the command line one-liners I use a on regular basis. Many times when doing a penetration test you have to move information, such as IP addresses, between tools and this will make the job much easier.

Ed (aka Ed) Says:

I really do wish we had awk or sed on Windows. I know, I know... we can get them with Cygwin or other shells that we could add in. But, our ground rules here force us to rely on built-in commands. That means, to parse in Windows, we rely on FOR /F loops, which can parse files, strings, or the output of commands.

When I first saw Paul's post above, I came up with this pretty straight-forward approach:

C:\> FOR /F "tokens=1-4,10-14 delims=. " %a in (lookups.txt) do @echo %d.%c.%b.%a %e.%f.%g.%h

Here, I'm parsing the file, using iterator variables starting with %a (FOR /F will automatically allocate more vars while it parses) and delimiters of . and spaces (gotta have that space there, because the dot overrides default parsing on spaces). I tokenize my variables around the first four and tenth through fourteenth places in the line, the IP address and domain name. Then, I dump everything out in our desired order. Simple and effective.

But, Hal brings up an interesting twist. Like Paul's approach, mine also has those ugly variable number of periods at the end, because we can't always assume that the domain name has four elements. I thought about it for a while, trying to push my first FOR /F loop to deal with this, and it got real ugly, real fast. Lots of IF statements made it impractical. So, I came up with a simpler approach: embedded FOR /F loops, the outer one to parse the file, and the inner loop to parse a string from the outer loop's results. Here it is:

C:\> FOR /F "tokens=1-5" %a in (lookups.txt) do @(@FOR /F "tokens=1-4 delims=." %i in ("%a") do @echo %l.%k.%j.%i %e)

What's this mess? Well, I use my outer FOR loop to parse lookups.txt into five components, using the default delims of spaces. %a will contain the IP address, with dots and all. The fifth item (%e) is the domain name. Then, in my inner FOR loop, I parse the string %a, using delims of periods and a variable of %i. That'll drop each octet of our IP address into a variable, which we can echo out. Furthermore, it preserves our domain name as one chunk in %e, regardless of the number of entities it has in it. I then just echo the IP address (reversing the octets, of course) followed by the domain name. There's one small drawback here: I leave the trailing period at the end of every domain name. There's only one there, and it's there for all of them, unlike the earlier approach. Still, this is very workable, and keeps the command syntax almost typable. :)