Episode #4 - Listing Files and Their Sizes

Ed's Initial Command:

On Windows, we can list the size of all files on c:\ and their names:

C:\> for /r c:\ %i in (*) do @echo %~zi, %i

This can be useful to find files that are hogging up disk space so you can back them up or delete them. To dump it into a file, append >> filename.csv. You can then open it in a spreadsheet for searching or sorting.

Hal's Comments:

Of course Unix folks tend to use du for tracking down directories that are consuming large amounts of disk space, but if you really want to drill down on individual files:

# find / -type f -exec wc -c {} \;

How about sorting that list so it's easier to find the big files:

# find / -type f -exec wc -c {} \; | sort -n

Or how about just the top 100 largest files in descending order of size:

# find / -type f -exec wc -c {} \; | sort -nr | head -100

By the way, this is an interesting example of a case where I really want to use "find ... -exec ..." rather than "find ... | xargs ...". If I were to let xargs aggregate the calls to "wc -c" I'd end up having wc intersperse totals in the middle of my output. By calling wc individually on each file with find I get output that's much nicer for piping into other programs.

Episode #3 - Watching the File Count in a Directory

Ed Says:

On Windows, count the number of files in a directory every 5 sec:

 C:\> for /L %i in (1,0,2) do @dir /b | find /c /v "" & ping -n 6 127.0.0.1>nul 

I often use this one when doing analysis of an anti-malware solution, to take inventory of the number of specimens it's detected/nabbed/quarantined.

This command includes some useful constructs as well:

• The for /L loop, which counts from one to two in steps of zero keeps it running forever, acting kind of like the Linux "watch" command.
• The /b option makes the dir command drop the cruft from its output (., .., volume name, size, etc.) We want only one line per file, so we use /b.
  
• The find /c /v "" means to find, and count (/c) the number of lines that do not have (/v) nothing (""). Lines that don't have nothing are all lines. Even a blank line has a Carriage Return / Line Feed, so it gets counted.
• And, we ping ourselves 6 times to introduce a 5-second delay. First ping happens instantly, the remaining happen once per second. Bummer that there is no sleep in Windows. Vista does have the "timeout /t -[N]" command, but we want something that'll work on nearly all Windows boxen, so we ping.

Byte_bucket points out that dir, when used like this, doesn't show files that are marked with the system or read-only attributes, and he's right. He suggested replacing the dir /b command with the attrib command, which will show all files regardless of these attributes. However, call me old fashioned, but I like to see lists of files using the dir command. Thus, if I want to make sure that I see hidden, system, and read-only files, I use dir /a. The /a means show me files with a given set of attributes (which can be h for hidden, s for system, and r for read-only). If I don't provide an attribute list, it will show me files regardless of the attribute. Thus, the resulting command:

 C:\> for /L %i in (1,0,2) do @dir /b /a | find /c /v "" & ping -n 6 127.0.0.1>nul 

Thanks for the great point, Mr. Bucket. Or should I call you Byte? :)

Hal Comments:

*Yawn* Too easy, Ed:

$ while :; do ls | wc -l; sleep 5; done

This example does point out one of the more interesting properties of the ls command: ls is conscious of where its output is going. If the output is going to normal terminal output, then the ls command gives you multi-column output. But if the output is being piped into another command, it outputs one file per line (you can verify this by doing "ls | cat"). How can ls figure this out? Check out the manual page for isatty(3).

Paul's Comments:

A fellow Twitter (Sorry, I can't find the person's name) recommended the watch command, which you can run like this:

$ watch -n 5 'ls | wc -l'

This will first clear the screen, then display the command in the upper left hand corner, and the date/time the command was last run in the upper right hand corner, followed by the count. Its pretty neat, and seems to be included in most Linux distributions (however, in OS X I used MacPorts to install it with port install watch)

Episode #2 - Looking at the Config of Built-In Firewall

Ed's netsh Kung Fu:

On a Windows box, show all ports allowed through the built-in firewall:

C:\> netsh firewall show portopening

Show all programs allowed to communicate through the built-in firewall:

C:\> netsh firewall show allowedprogram

Show all configuration options of built-in firewall:

C:\> netsh firewall show config

Gosh, I love the netsh command!

Paul's Comments:

On Linux, list all iptables firewall rules in all chains:

# iptables -t nat -nL
# iptables -t mangle -nL
# iptables -t filter -nL
# iptables -t raw -nL

I find it important to list out all chains in all tables so there are no rules that "hide" from you. Also, the "-n" is important to avoid reverse lookups that could take a long time to run. It would be nice to run in this in one command...

Hal's Comments:

Paul, you mean one command like this?

# for type in nat mangle filter raw; do iptables -t $type -nL; done

OK, maybe it's not strictly speaking one command, but it's at least one command line. :-) I also like using the "-v" option when dumping my iptables chains so that I can see the number of times a given rule and/or chain was triggered.

Paul's Comments:

Yes, that's exactly what I need! Often times during a penetration test, or even during normal systems administration troubleshooting, I want to see ALL the iptables rules. In a penetration test this is an information gathering exercise, in addition to gaining an understanding of how to communicate with the host we've compromised. For systems administration I will often setup iptables rules, either in place of or in addition to a network-based firewall. This command is useful to see the rules in all tables and chains to hunt down any connectivity issues.

Episode #1 - Convert Dos To UNIX

Okay, so we'll start simple here at Command Line Kung Fu. I think many of us run into the problem where we get those funny Windows characters in our files (^M). These are of course Windows line breaks. The simple solution (which should work on OS X and Linux):

 tr '\r' '\n' < file.txt > file.txt

I just had to do this on a file that someone shared with me. Next maybe we'll look at how I chopped up and hacked that file with grep and cut :)

Paul Asadoorian
PaulDotCom

Davide Brini points out that there are a couple of problems with the solution above:

1. The output redirections here "... <file.txt >file.txt" are going to leave you with an empty file. That's because the ">file.txt" truncates the file before "<file.txt" reads any data from it. What you need to do instead is something like "... <file.txt >newfile.txt"



2. Paul's tr command results in double newlines for DOS-formatted files whose lines typically end with "\r\n". A more correct solution would be "sed 's/\r$//' file.txt >newfile.txt"

As Davide also points out, there's also the dos2unix utility which is an Open Source tool that's often found in Unix-like OSes.