Episode #181: Making Contact

Hal wanders back on stage
Whew! Sure is dusty in here!
Man, those were the days! It started with Ed jamming on Twitter and me heckling from the audience. Then Ed invited me up on stage (once we built the stage), and that was some pretty sweet kung fu. Then Tim joined the band, Ed left, and the miles, and the booze, and the groupies got to be too much.
But we still get fan mail! Here's one from superfan Josh Wright that came in just the other day:
I have a bunch of sub-directories, all of which have files of various names. I want to produce a list of directories that do not have a file starting with "ContactSheet-*.jpg".
I thought I would just use "find" with "-exec test":

find . -type d \! -exec test -e "{}/ContactSheet\*" \; -print

Unfortunately, "test" doesn't support globbing, so this always fails.
Here's a sample directory tree and files. I thought this might be an interesting CLKF topic.

$ ls
Adriana  Alessandra  Gisele  Heidi  Jelena  Kendall  Kim  Miranda
$ ls *
Adriana:
Adriana-1.jpg  Adriana-3.jpg  Adriana-5.jpg
Adriana-2.jpg  Adriana-4.jpg  Adriana-6.jpg

Alessandra:
Alessandra-1.jpg  Alessandra-4.jpg  ContactSheet-Alessandra.jpg
Alessandra-2.jpg  Alessandra-5.jpg
Alessandra-3.jpg  Alessandra-6.jpg

Gisele:
Gisele-1.jpg  Gisele-3.jpg  Gisele-5.jpg
Gisele-2.jpg  Gisele-4.jpg  Gisele-6.jpg

Heidi:
ContactSheet-Heidi.jpg  Heidi-2.jpg  Heidi-4.jpg  Heidi-6.jpg
Heidi-1.jpg             Heidi-3.jpg  Heidi-5.jpg

Jelena:
ContactSheet-Jelena.jpg  Jelena-2.jpg  Jelena-4.jpg  Jelena-6.jpg
Jelena-1.jpg             Jelena-3.jpg  Jelena-5.jpg

Kendall:
Kendall-1.jpg  Kendall-3.jpg  Kendall-5.jpg
Kendall-2.jpg  Kendall-4.jpg  Kendall-6.jpg

Kim:
ContactSheet-Kim.jpg  Kim-2.jpg  Kim-4.jpg  Kim-6.jpg
Kim-1.jpg             Kim-3.jpg  Kim-5.jpg

Miranda:
Miranda-1.jpg  Miranda-3.jpg  Miranda-5.jpg
Miranda-2.jpg  Miranda-4.jpg  Miranda-6.jpg

OK, Josh. I'm feeling you on this one. Maybe I can find some of the lost magic.
Because Josh started this riff with a "find" command, my brain went there first. But my solutions all ended up being variants on running two find commands-- get a list of the directories with "ContactSheet" files and "subtract" that from the list of all directories. Here's one of those solutons:

$ sort <(find * -type d) <(find * -name ContactSheet-\* | xargs dirname) | uniq -u
Adriana
Gisele
Kendall
Miranda

The first "find" gets me all the directory names. The second "find" gets all of the "ContactSheet" files, and then that output gets turned into a list of directory names with "xargs dirname". Then I use the "<(...)" construct to feed both lists of directories into the "sort" command. "uniq -u" gives me a list of the directories that only appear once-- which is the directories that do not have a "ContactSheet" file in them.
But I wasn't cool with running the two "find" commands-- especially when we might have a big set of directories. And then it hit me. Just like our CLKF jam is better when I had my friends Ed and Tim rocking out with me, we can make this solution better by combining our selection criteria into a single "find" command:

$ find * \( -type d -o -name ContactSheet\* \) | sed 's/\/ContactSheet.*//' | uniq -u
Adriana
Gisele
Kendall
Miranda

By itself, the "find" command gives me output like this:

$ find * \( -type d -o -name ContactSheet\* \)
Adriana
Alessandra
Alessandra/ContactSheet-Alessandra.jpg
Gisele
Heidi
Heidi/ContactSheet-Heidi.jpg
Jelena
Jelena/ContactSheet-Jelena.jpg
Kendall
Kim
Kim/ContactSheet-Kim.jpg
Miranda

Then I use "sed" to pick off the file name, and I end up with the directory list with the duplicate directory names already sorted together. That means I can just feed the results into "uniq -u" and everything is groovy!
Cool, man. That was cool. Now if only my friends Ed and Tim were here, that would be something else.

A Wild-Eyed Man Appears on Stage for the First Time Since December 2013

Wh-wh-where am I?  Why am I covered with dust?  Who are you people?  What's going on?

This all looks so familiar.  It reminds me of... of... those halcyon days of the early Command Line Kung Fu blog.  A strapping young Tim Medin throwing down some amazing shell tricks.  Master Hal Pomeranz busting out beautiful bash fu.  Wow... those were the days.  Where the heck have I been?

Oh wait... what?  You want me to solve Josh's dilemma using cmd.exe?  What, am I a trained monkey who dances for you when you turn the crank on the music box?  Oh I can hear the sounds now.  That lovely music box... with its beautiful tunes that are so so so hypnotizing... so hypnotizing...

...and then Ed starts to dance...

I originally thought, "Uh-oh... a challenge posed by Josh Wright and then smashed by Hal is gonna be an absolute pain in the neck in cmd.exe, the Ruprecht of shells."  But then, much to my amazement, the answer all came together in about 3 minutes.  Here ya go, Big Josh.

c:\> for /D /R %i in (*) do @dir /b %i | find "ContactSheet" > nul || echo %i

The logic works thusly:

I've got a "for" loop, iterating over directories (/D) in a recursive fashion (/R) with an iterator variable of %i which will hold the directory names.  I do this for everything in the current working directory and down (in (*)... although you could put a directory path in there to start at a different directory).  That'll spit out each directory. At each iteration through the loop, I do the following:

• Turn off the display of commands so it doesn't clutter the output (@)
• Get a directory listing of the current directory indicated by the iterator variable, %i.  I want this directory listing in bare form without the Volume Name and Size cruft but with full path (/b).  That'll spew all these directories and their contents on standard out.  Remember, I'm recursing using the /R in my for loop, so I don't need to use a /s in my dir command here.
• I take the output of the dir command and pipe it through the find command to look for the string "ContactSheet".  I throw out the output of the find command, because it's a bunch of cruft where it actually finds ContactSheet.
• But, if the find command FAILS to see the string "ContactSheet" (||), I want to display the path of the directory where it failed, so I echo %i.

Voilamundo!  There you go!  The output looks like this:

c:\tmp\test>for /D /r %i in (*) do @dir /b /s %i | find "ContactSheet" > nul || echo %i
c:\tmp\test\ChrisFall
c:\tmp\test\dan
c:\tmp\test\Fred\Fred2

I'd like to thank Josh for the most engaging challenge!  I'll now go back into my hibernative state.... Zzzzzzzzz...

...and then Tim grabs the mic...

What a fantastic throwback, a reunion of sorts. Like the return of Guns n' Roses, but more Welcome to the Jungle than Paradise City, it gets worse here everyday. We got everything you want honey, we know the commands. We are the people that can find whatever you may need.

Uh, sorry... I've missed the spotlight here. Let's get back to the commands. Here is a working command in long form and shortened form:

PS C:\Photos> Get-ChildItem -Directory | Where-Object { -not (Get-ChildItem -Path $_ -Filter ContactSheet*) }

PS C:\Photos> ls -di | ? { -not (ls $_ -Filter ContactSheet*) }

Let's take it day piece by day piece. If you want it you're gonna bleed but it's the price to pay. Well, actually, it isn't that difficult so no bleeding will be involved.

The first portion simply gets a list of directories in the current directory.

Next, we have a Where-Object filter that will pass the proper objects down the pipeline. In the filter we need to look for files in the directory passed down the pipeline ($_) containing files that start with ContactSheet. We simply invert the search with the -not operator.

With this command you can have everything you want but you better not take it from me... the other guys did some excellent work. We've missed ya'll and hopefully we will see you again. Now back into the cave to hibernate.

Episode #180: Open for the Holidays!

Not-so-Tiny Tim checks in with the ghost of Christmas present:

I know many of you have been sitting on Santa's lap wishing for more Command Line Kung Fu. Well, we've heard your pleas and are pushing one last Episode out before the New Year!

We come bearing a solution for a problem we've all encountered. Ever try to delete or modify a file and receive an error message that the file is in use? Of course you have! The real problem is trying to track down the user and/or process that has the file locked.

I have a solution for you on Windows, "openfiles". Well, sorta. This time of year I can't risk getting on Santa's bad side so let me add the disclaimer that it is only a partial solution. Here's what I mean, let's look for open files:

C:\> openfiles

INFO: The system global flag 'maintain objects list' needs
      to be enabled to see local opened files.
      See Openfiles /? for more information.


Files opened remotely via local share points:
---------------------------------------------

INFO: No shared open files found.

By default when we run this command it gives us an error that we haven't enabled the feature. Wouldn't it be nice if we could simply turn it on and then look at the open files. Yes, it would be nice...but no. You have to reboot. This present is starting to look a lot like a lump of coal. So you need know that you will encounter the problem before it happens so you can be ready for it. Bah-Humbug!

To enable "openfile" run this command:

C:\> openfile /local on

SUCCESS: The system global flag 'maintain objects list' is enabled.
         This will take effect after the system is restarted.

...then reboot.

Of course, now that we've rebooted the file will be unlocked, but we are prepared for next time. So next time when it happens we can run this command to see the results (note: if you don't specify a switch /query is implied):

C:\> openfiles /query

Files Opened Locally:
---------------------

ID    Process Name         Open File (Path\executable)                       
===== ==================== ==================================================
8     taskhostex.exe       C:\Windows\System32
224   taskhostex.exe       C:\Windows\System32\en-US\taskhostex.exe.mui
296   taskhostex.exe       C:\Windows\Registration\R00000000000d.clb
324   taskhostex.exe       C:\Windows\System32\en-US\MsCtfMonitor.dll.mui
752   taskhostex.exe       C:\Windows\System32\en-US\winmm.dll.mui
784   taskhostex.exe       C:\..\Local\Microsoft\Windows\WebCache\V01tmp.log
812   taskhostex.exe       C:\Windows\System32\en-US\wdmaud.drv.mui
...

Of course, this is a quite long list. You can use use "find" or "findstr" to filter the results, but be aware that long file names are truncated (see ID 784). You can get a full list by changing the format with "/fo LIST". However, the file name will be on a separate line from the owning process and neither "find" nor "findstr" support context.

Another oddity, is that there seems to be duplicate IDs.

C:\> openfiles /query | find "888"
888   chrome.exe           C:\Windows\Fonts\consola.ttf
888   Lenovo Transition.ex C:\..\Lenovo\Lenovo Transition\Gui\yo_btn_g3.png
888   vprintproxy.exe      C:\Windows\Registration\R00000000000d.clb

Different processes with different files, all with the same ID. This means that when you disconnect the open file you better be careful.

Speaking of disconnecting the files, we can do just that with the /disconnect switch. We can disconnect by ID (ill advised) with the /id switch. We can also disconnect all the files based on the user:

C:\> openfiles /disconnect /a jacobmarley

Or the file name:

C:\> openfiles /disconnect /op "C:\Users\tm\Desktop\wishlist.txt" /a *

Or even the directory:

C:\> openfiles /disconnect /op "C:\Users\tm\Desktop\" /a *

We can even run this against a remote system with the /s SERVERNAME option.

This command is far from perfect, but it is pretty cool.

Sadly, there is no built-in capability in PowerShell to do this same thing. With PowerShell v4 we get Get-SmbOpenFile and Close-SmbOpenFile, but they only work on files opened over the network, not on files opened locally.

Now it is time for Mr. Scrooge Pomeranz to ruin my day by using some really useful, built-in, and ENABLED features of Linux.

It's a Happy Holiday for Hal:

Awww, Tim got me the nicest present of all-- a super-easy Command-Line Kung Fu Episode to write!

This one's easy because Linux comes with lsof, a magical tool surely made by elves at the North Pole. I've talked about lsof in several other Episodes already but so far I've focused more on network and process-related queries than checking objects in the file system.

The simplest usage of lsof is checking which processes are using a single file:

# lsof /var/log/messages
COMMAND    PID USER   FD   TYPE DEVICE SIZE/OFF    NODE NAME
rsyslogd  1250 root    1w   REG    8,3 13779999 3146461 /var/log/messages
abrt-dump 5293 root    4r   REG    8,3 13779999 3146461 /var/log/messages

Here we've got two processes that have /var/log/messages open-- rsyslogd for writing (see the "1w" in the "FD" column, where the "w" means writing), and abrt-dump for reading ("4r", "r" for read-only).

You can use "lsof +d" to see all open files in a given directory:

# lsof +d /var/log
COMMAND    PID USER   FD   TYPE DEVICE SIZE/OFF    NODE NAME
rsyslogd  1250 root    1w   REG    8,3 14324534 3146461 /var/log/messages
rsyslogd  1250 root    2w   REG    8,3   175427 3146036 /var/log/cron
rsyslogd  1250 root    5w   REG    8,3  1644575 3146432 /var/log/maillog
rsyslogd  1250 root    6w   REG    8,3     2663 3146478 /var/log/secure
abrt-dump 5293 root    4r   REG    8,3 14324534 3146461 /var/log/messages

The funny thing about "lsof +d" is that it only shows you open files in the top-level directory, but not in any sub-directories. You have to use "lsof +D" for that:

# lsof +D /var/log
COMMAND     PID   USER   FD   TYPE DEVICE SIZE/OFF    NODE NAME
rsyslogd   1250   root    1w   REG    8,3 14324534 3146461 /var/log/messages
rsyslogd   1250   root    2w   REG    8,3   175427 3146036 /var/log/cron
rsyslogd   1250   root    5w   REG    8,3  1644575 3146432 /var/log/maillog
rsyslogd   1250   root    6w   REG    8,3     2663 3146478 /var/log/secure
httpd      3081 apache    2w   REG    8,3      586 3146430 /var/log/httpd/error_log
httpd      3081 apache   14w   REG    8,3        0 3147331 /var/log/httpd/access_log
...

Unix-like operating systems track open files on a per-partition basis. This leads to an interesting corner-case with lsof: if you run lsof on a partition boundary, you get a list of all open files under that partition:

# lsof /
COMMAND     PID      USER   FD   TYPE DEVICE SIZE/OFF     NODE NAME
init          1      root  cwd    DIR    8,3     4096        2 /
init          1      root  rtd    DIR    8,3     4096        2 /
init          1      root  txt    REG    8,3   150352 12845094 /sbin/init
init          1      root  DEL    REG    8,3           7340061 /lib64/libnss_files-2.12.so
init          1      root  DEL    REG    8,3           7340104 /lib64/libc-2.12.so
...
# lsof / | wc -l
3500

Unlike Windows, Linux doesn't really have a notion of disconnecting processes from individual files. If you want a process to release a file, you kill the process. lsof has a "-t" flag for terse output. In this mode, it only outputs the PIDs of the matching processes. This was designed to allow you to easily substitute the output of lsof as the arguments to the kill command. Here's the little trick I showed back in Episode 22 for forcibly unmounting a file system:

# umount /home
umount: /home: device is busy
# kill $(lsof -t /home)
# umount /home

Here we're exploiting the fact that /home is a partition mount point so lsof will list all processes with files open anywhere in the file system. Kill all those processes and Santa might leave coal in your stocking next year, but you'll be able to unmount the file system!

Episode #179: The Check is in the Mail

Tim mails one in:

Bob Meckle writes in:

I have recently come across a situation where it would be greatly beneficial to build a script to check revocation dates on certificates issued using a certain template, and send an email to our certificate staff letting them know which certificates will expire within the next 6 weeks. I am wondering if you guys have any tricks up your sleeve in regards to this situation?

Thanks for writing in Bob. This is actually quite simple on Windows. One of my favorite features of PowerShell is that dir (an alias for Get-ChildItem) can be used on pretty much any hierarchy, including the certificate store. After we have a list of the certificates we simply filter by the expiration date. Here is how we can find expiring certificates:

PS C:\> Get-ChildItem -Recurse cert: | Where-Object { $_.NotAfter -le (Get-Date).AddDays(42) -and $_.NotAfter -ge (Get-Date) }

We start off by getting a recursive list of the certificates. Then the results are piped into the Where-Object cmdlet to filter for the certificates that expire between today and six weeks (42 days) from now (inclusive). Additional filtering can be added by simply modifying the Where-Object filter, per Bob's original request. We can shorten the command using aliases and shortened parameter names, as well as store the output to be emailed.

PS C:\> $tomail = ls -r cert: | ? { $_.NotAfter -le (Get-Date).AddDays(42) -and $_.NotAfter -ge (Get-Date) }

Emailing is quite simple too. PowerShell version 4 adds a new cmdlet, Send-MailMessage. We can send the certificate information like this:

PS C:\> Send-MailMessage -To me@blah.com -From cert@blah.com -Subject "Expiring Certs" -Body $tomail

Pretty darn simple if you ask me. Hal, what do you have up your sleeve?

Hal relies on his network:

Tim, you're getting soft with all these cool PowerShell features. Why when I was your age... Oh, nevermind! Kids these days! Hmph!

If I'm feeling curmudgeonly, it's because this is far from a cake-walk in Linux. Obviously, I can't check a Windows certificate store remotely from a Linux machine. So I thought I'd focus on checking remote web site certificates (which could be Windows, Linux, or anything else) from my Linux command line.

The trick for checking a certificate is fairly well documented on the web:

$ echo | openssl s_client -connect www.google.com:443 2>/dev/null | 
     openssl x509 -noout -dates
notBefore=Jun  4 08:58:29 2014 GMT
notAfter=Sep  2 00:00:00 2014 GMT

We use the OpenSSL built-in "s_client" to connect to the target web server and dump the certificate information. The "2>/dev/null" drops some extraneous standard error logging. The leading "echo" piped into the standard input makes sure that we close the connection right after receiving the certificate info. Otherwise our command line will hang and never return to the shell prompt. We then use OpenSSL again to output the information we want from the downloaded certificate. Here we're just requesting the "-dates"-- "-noout" stops the openssl command from displaying the certificate itself.

However, there is much more information you can parse out. Here's a useful report that displays the certificate issuer, certificate name, and fingerprint in addition to the dates:

$ echo | openssl s_client -connect www.google.com:443 2>/dev/null | 
     openssl x509 -noout -issuer -subject -fingerprint -dates
issuer= /C=US/O=Google Inc/CN=Google Internet Authority G2
subject= /C=US/ST=California/L=Mountain View/O=Google Inc/CN=www.google.com
SHA1 Fingerprint=F4:E1:5F:EB:F1:9F:37:1A:29:DA:3A:74:BF:05:C5:AD:0A:FB:B1:95
notBefore=Jun  4 08:58:29 2014 GMT
notAfter=Sep  2 00:00:00 2014 GMT

If you want a complete dump, just use "... | openssl x509 -text".

But let's see if we can answer Bob's question, at least for a single web server. Of course, that's going to involve some date arithmetic, and the shell is clumsy at that. First I need to pull off just the "notAfter" date from my output:

$ echo | openssl s_client -connect www.google.com:443 2>/dev/null | 
     openssl x509 -noout -dates | tail -1 | cut -f2 -d=
Sep  2 00:00:00 2014 GMT

You can convert a time stamp string like this into a "Unix epoch" date with the GNU date command:

$ date +%s -d 'Sep  2 00:00:00 2014 GMT'
1409616000

To do it all in one command, I just use "$(...) for command output substitution:

$ date +%s -d "$(echo | openssl s_client -connect www.google.com:443 2>/dev/null | 
                     openssl x509 -noout -dates | tail -1 | cut -f2 -d=)"
1409616000

I can calculate the difference between this epoch date and the current epoch date ("date +%s") with some shell arithmetic and more command output substitution:

$ echo $(( $(date +%s -d "$(echo | openssl s_client -connect www.google.com:443 2>/dev/null | 
                                openssl x509 -noout -dates | tail -1 | cut -f2 -d=)") 
            - $(date +%s) ))
5452606

Phew! This is already getting pretty long-winded, but now I can check my result against Bob's 6 week threshold (that's 42 days or 3628800 seconds):

$ [[ $(( $(date +%s -d "$(echo | openssl s_client -connect www.google.com:443 2>/dev/null | 
                              openssl x509 -noout -dates | tail -1 | cut -f2 -d=)") - $(date +%s) )) 
    -gt 3628800 ]] && echo GOOD || echo EXPIRING
GOOD

Clearly this is straying all too close to the borders of Scriptistan, but if you wanted to check several web sites, you could add an outer loop:

$ for d in google.com sans.org facebook.com twitter.com; do 
    echo -ne $d\\t; 
    [[ $(( $(date +%s -d "$(echo | openssl s_client -connect www.$d:443 2>/dev/null | 
                            openssl x509 -noout -dates | tail -1 | cut -f2 -d=)") - $(date +%s) ))
       -gt 3628800 ]] && echo GOOD || echo EXPIRING; 
done
google.com	GOOD
sans.org	GOOD
facebook.com	GOOD
twitter.com	GOOD

See? It's all good, Bob!

Episode #178: Luhn-acy

Hal limbers up in the dojo

To maintain our fighting trim here in the Command Line Kung Fu dojo, we like to set little challenges for ourselves from time to time. Of course, we prefer it when our loyal readers send us ideas, so keep those emails coming! Really... please oh please oh please keep those emails coming... please, please, please... ahem, but I digress.

All of the data breaches in the news over the last year got me thinking about credit card numbers. As many of you are probably aware, credit card numbers have a check digit at the end to help validate the account number. The Luhn algorithm for computing this digit is moderately complicated and I wondered how much shell code it would take to compute these digits.

The Luhn algorithm is a right-to-left calculation, so it seemed like my first task was to peel off the last digit and be able to iterate across the remaining digits in reverse order:

$ for d in $(echo 123456789 | rev | cut -c2- | sed 's/\(.\)/\1 /g'); do echo $d; done
8
7
6
5
4
3
2
1

The "rev" utility flips the order of our digits, and then we just grab everything from the second digit onwards with "cut". We use a little "sed" action to break the digits up into a list we can iterate over.

Then I started thinking about how to do the "doubling" calculation on every other digit. I could have set up a shell function to calculate the doubling each time, but with only 10 possible outcomes, it seemed easier to just create an array with the appropriate values:

$ doubles=(0 2 4 6 8 1 3 5 7 9)
$ for d in $(echo 123456789 | rev | cut -c2- | sed 's/\(.\)/\1 /g'); do echo $d ${doubles[$d]}; done
8 7
7 5
6 3
5 1
4 8
3 6
2 4
1 2

Then I needed to output the "doubled" digit only every other digit, starting with the first from the right. That means a little modular arithmetic:

$ c=0
$ for d in $(echo 123456789 | rev | cut -c2- | sed 's/\(.\)/\1 /g'); do 
    echo $(( ++c % 2 ? ${doubles[$d]} : $d )); 
done
7
7
3
5
8
3
4
1

I've introduced a counting variable, "$c". Inside the loop, I'm evaluating a conditional expression to decide if I need to output the "double" of the digit or just the digit itself. There are several ways I could have handled this conditional operation in the shell, but having it in the mathematical "$((...))" construct is particularly useful when I want to calculate the total:

$ c=0; t=0; 
$ for d in $(echo 123456789 | rev | cut -c2- | sed 's/\(.\)/\1 /g'); do 
    t=$(( $t + (++c % 2 ? ${doubles[$d]} : $d) )); 
done
$ echo $t
38

We're basically done at this point. Instead of outputting the total, "$t", I need to do one more calculation to produce the Luhn digit:

$ c=0; t=0; 
$ for d in $(echo 123456789 | rev | cut -c2- | sed 's/\(.\)/\1 /g'); do 
    t=$(( $t + (++c % 2 ? ${doubles[$d]} : $d) ));
$ echo $(( ($t * 9) % 10 ))
2

Here's the whole thing in one line of shell code, including the array definition:

doubles=(0 2 4 6 8 1 3 5 7 9); 
c=0; t=0; 
for d in $(echo 123456789 | rev | cut -c2- | sed 's/\(.\)/\1 /g'); do 
    t=$(( $t + (++c % 2 ? ${doubles[$d]} : $d) )); 
done; 
echo $(( ($t * 9) % 10 ))

Even with all the extra whitespace, the whole thing fits in under 100 characters! Grand Master Ed would be proud.

I'm not even going to ask Tim to try and do this in CMD.EXE. Grand Master Ed could have handled it, but we'll let Tim use his PowerShell training wheels. I'm just wondering if he can do it so it still fits inside a Tweet...

Tim checks Hal's math

I'm not quite sure how Hal counts, but I when I copy/paste and then use Hal's own wc command I get 195 characters. It is less than *2*00 characters, not long enough to tweet.

Here is how we can accomplish the same task in PowerShell. I'm going to use a slightly different method than Hal. First, I'm going to use his lookup method as it is more terse then doing the extra match via if/then. In addition, I am going to extend his method a little to save a little space.

PS C:\> $lookup = @((0..9),(0,2,4,6,8,1,3,5,7,9));

This mutli-dimensional array contains a lookup for the number as well as the doubled number. That way I can index the value without an if statement to save space. Here is an example:

PS C:\> $isdoubled = $false
PS C:\> $lookup[$isdoubled][6]
6
PS C:\> $isdoubled = $true
PS C:\> $lookup[$isdoubled][7]
15

The shortest way to get each digit, from right to left, is by using regex (regular expression) match and working right to left. A longer way would be to use the string, convert it to a char array, then reverse it but that is long, ugly, and we need to use an additional variable.

The results are fed into a ForEach-Object loop. Before the objects (the digits) passed down the pipeline are handled we need to initialize a few variables, the total and the boolean $isdoubled variables in -Begin. Next, we add the digits up by accessing the items in our array as well as toggling the $isdoubled variable. Finally, we use the ForEach-Object's -End to output the final value of $sum.

PS C:\> ([regex]::Matches('123456789','.','RightToLeft')) | ForEach-Object 
-Begin { $sum = 0; $isdoubled = $false} -Process { $sum += $l[$isdoubled][[int]$_.value];  $d = -not $d } 
-End { $sum }

We can shorten the command to this to save space.

PS C:\> $l=@((0..9),(0,2,4,6,8,1,3,5,7,9));
([regex]::Matches('123456789','.','RightToLeft')) | %{ $s+=$l[$d][$_.value];$d=!$d} -b{$s=0;$d=0} -en{$s}

According to my math this is exactly 140 characters. I could trim another 2 by removing a few spaces too. It's tweetable!

I'll even throw in a bonus version for cmd.exe:

C:\> powershell -command "$l=@((0..9),(0,2,4,6,8,1,3,5,7,9));
([regex]::Matches("123456789",'.','RightToLeft')) | %{ $s+=$l[$d][$_.value];$d=!$d} -b{$s=0;$d=0} -en{$s}"

Ok, it is a bit of cheating, but it does run from CMD.

Hal gets a little help

I'm honestly not sure where my brain was at when I was counting characters in my solution. Shortening variable names and removing all extraneous whitespace, I can get my solution down to about 150 characters, but no smaller.

Happily, Tom Bonds wrote in with this cute little blob of awk which accomplishes the mission:

awk 'BEGIN{split("0246813579",d,""); for (i=split("12345678",a,"");i>0;i--) {t += ++x % 2 ? d[a[i]+1] : a[i];} print (t * 9) % 10}'

Here it is with a little more whitespace:

awk 'BEGIN{ split("0246813579",d,""); 
            for (i=split("12345678",a,"");i>0;i--) {
                t += ++x % 2 ? d[a[i]+1] : a[i];
            } 
            print (t * 9) % 10
          }'

Tom's getting a lot of leverage out of the "split()" operator and using his "for" loop to count backwards down the string. awk is automatically initializing his $t and $x variables to zero each time his program runs, whereas in the shell I have to explicitly set them to zero or the values from the last run will be used.

Anyway, Tom's original version definitely fits in a tweet! Good show, Tom!

Episode #177: There and Back Again

Hal finds some old mail

Way, way back after Episode #170 Tony Reusser sent us a follow-up query. If you recall, Episode #170 showed how to change files named "fileaa", "fileab", "fileac", etc to files named "file.001", "file.002", "file.003". Tony's question was how to go back the other way-- from "file.001" to "fileaa", "file.002" to "fileab", and so on.

Why would we want to do this? Heck, I don't know! Ask Tony. Maybe he just wants to torture us to build character. Well we here at Command Line Kung Fu fear no characters, though we may occasionally lose reader emails behind the refrigerator for several months.

The solution is a little scripty, but I did actually type it in on the command line:

    c=1
    for l1 in {a..z}; do 
        for l2 in {a..z}; do 
            printf -v ext %03d $(( c++ ))
            [[ -f file.$ext ]] && mv file.$ext file$l1$l2 || break 2
        done
    done

There are two nested loops here which work together to create our alphabetic file extensions. $l1 represents the first of the two letters, ranging from 'a' to 'z'. $l2 is the second letter, also ranging from 'a' to 'z'. Put them next to each other and you get "aa", "ab", "ac", etc.

Like Episode #170, I'm using a counter variable named $c to track the numeric file extension. So, for all you computer science nerds, this is a rather weird looping construct because I'm using three different loop control variables. And weird is how I roll.

Inside the loop, I re-use the code from Episode #170 to format $c as our three-digit file extension (saved in variable $ext) and auto-increment $c in the same expression. Then I check to see if "file.$ext" exists. If we have a "file.$ext", then we rename it to "file$l1$l2" ("fileaa", "fileab", etc). If "file.$ext" does not exist, then we've run out of "file.xxx" pieces and we can stop looping. "break 2" breaks out of both enclosing loops and terminates our command line.

And there you go. I hope Tim has as much fun as I did with this. I bet he'd have even more fun if I made him do it in CMD.EXE. Frankly all that PowerShell has made him a bit sloppy...

Tim mails this one in just in time from Abu Dhabi

Wow, CMD huh? That trip to Scriptistan must have made him crazy. I think a CMD version of this would violate the laws of physics.

I'm on the other side of the world looking for Nermal in Abu Dhabi. Technically, it is May 1 here, but since the publication date shows April I think this counts as our April post. At least, that is the story I'm sticking to.

Sadly, I too will have to enter Scriptistan. I have a visa for a long stay on this one. I'll start by using a function to convert a number to Base26. The basis of this function is taken from here. I modified the function so we could add leading A's to adjust the width.

function Convert-ToLetters ([parameter(Mandatory=$true,ValueFromPipeline=$true)][int] $Value, [int]$MinWidth=0) {
    $currVal = $Value
    if ($LeadingDigits -gt 0) { $currVal = $currVal + [int][Math]::Pow(26, $LeadingDigits) }
    $returnVal = '';
    while ($currVal -ge 26) {
        $returnVal = [char](($currVal) % 26 + 97) + $returnVal;
        $currVal =  [int][math]::Floor($currVal / 26)
    }
    $returnVal = [char](($currVal) + 64) + $returnVal;
     
    return $returnVal
}

This allows me to cheat greatly simplify the renaming process.

PS C:\> ls file.* | % { move $_ "file.$(Convert-ToLetters [int]$_.Extension.Substring(1) -MinWidth 3 )" }

This command will read the files starting with "file.", translate the extension in to Base26 (letters), and rename the file. The minimum width is configurable as well, so file.001 could be file.a, file.aa, file.aaa, etc. Also, this version will support more than 26^2 files.

Episode #176: Step Up to the WMIC

Tim grabs the mic:

Michael Behan writes in:

Perhaps you guys can make this one better. Haven’t put a ton of thought into it:

C:\> (echo HTTP/1.0 200 OK & wmic process list full /format:htable) | nc -l -p 3000

Then visit http://127.0.0.1:3000

This could of course be used to generate a lot more HTML reports via wmic that are quick to save from the browser. The downside is that in its current state is that the page can only be visited once. Adding something like /every:5 just pollutes the web page with mostly duplicate output.

Assuming you already have netcat (nc.exe) on the system the command above will work fine, but it will only work once. After the browser recieves the data the connection has been used and the command is done. To do this multiple times you must wrap it in an infinite For loop.

C:\> for /L %i in (1, 0, 2) do (echo HTTP/1.0 200 OK & wmic process list full /format:htable) | nc -l -p 3000

This will count from 1 to 2 and count by 0, which will never happen (except for very large values of 0). We could use the wmic command to request this information from the remote machine and view it in our browser. This method will authenticate to the remote machine instead of allowing anyone to access the information.

C:\> wmic /node:joelaptop process list full /format:htable > joelaptopprocesses.html && start joelaptopprocesses.html

This will use your current credentials to authenticate to the remote machine, request the remote process in html format, save it to a file, and finally open the file in your default viewer (likely your browser). If you need to use separate credentials you can specify /user:myusername and /password:myP@assw0rd.

Hal, your turn, and I want to see this in nice HTML format. :)

Hal throws up some jazz hands:

Wow. Tim seems a little grumpy. Maybe it's because he can make a simple web server on the command line but has no way to actually request data from it via the command line. Don't worry Little Tim, maybe someday...

Heck, maybe Tim's grumpy because of the dumb way he has to code infinite loops in CMD.EXE. This is a lot easier:

$ while :; do ps -ef | nc -l 3000; done

Frankly, most browsers will interpret this as "text/plain" by default and display the output correctly.

But the above loop got me thinking that we could actually stack multiple commands in sequence:

while :; do
    ps -ef | nc -l 3000
    netstat -anp | nc -l 3000
    df -h | nc -l 3000
    ...
done

Each connection will return the output of a different command until you eventually exhaust the list and start all over again with the first command.

OK, now let's deal with grumpy Tim's request for "nice HTML format". Nothing could be easier, my friends:

$ while :; do (echo '<pre>'; ps -ef; echo '</pre>') | nc -l 3000; done

Hey, it's accepted by every major browser I tested it with! And that's the way we do it downtown... (Hal drops the mic)

Episode #175: More Time! We Need More Time!

Tim leaps in

Every four years (or so) we get an extra day in February, leap year. When I was a kid this term confused me. Frogs leap, they leap over things. A leap year should be shorter! Obviously, I was wrong.

This extra day can give us extra time to complete tasks (e.g. write blog post), so we are going to use our shells to check if the current year is a leap year.

PS C:\> [DateTime]::IsLeapYear(2014)
False

Sadly, this year we do not have extra time. Let's confirm that this command does indeed work by checking a few other years.

PS C:\> [DateTime]::IsLeapYear(2012)
True
PS C:\> [DateTime]::IsLeapYear(2000)
True
PS C:\> [DateTime]::IsLeapYear(1900)
False

Wait a second! Something is wrong. The year 1900 is a multiple of 4, why is it not a leap year?

The sun does not take exactly 365.25 days to get around the sun, it is actually 365.242199 days. This means that if we always leaped every four years we would slowly get off course. So every 100 years we skip the leap year.

Now you are probably wondering why 2000 had a leap year. That is because it is actually the exception to the exception. Every 400 years we skip skipping the leap year. What a cool bit of trivia, huh?

Hal, how jump is your shell?

Hal jumps back

I should have insisted Tim do this one in CMD.EXE. Isn't is nice that PowerShell has a IsLeapYear() built-in? Back in my day, we didn't even have zeroes! We had to bend two ones together to make zeroes! Up hill! Both ways! In the snow!

Enough reminiscing. Let's make our own IsLeapYear function in the shell:

function IsLeapYear {
    year=${1:-$(date +%Y)};
    [[ $(($year % 400)) -eq 0 || ( $(($year % 4)) -eq 0 && $(($year % 100)) -ne 0 ) ]]
}

There's some fun stuff in this function. First we check to see if the function is called with an argument ("${1-..."). If so, then that's the year we'll check. Otherwise we check the current year, which is the value returned by "$(date +%Y)".

The other line of the function is the standard algorithm for figuring leap years. It's a leap year if the year is evenly divisible by 400, or divisible by 4 and not divisible by 100. Since shell functions return the value of the last command or expression executed, our function returns whether or not it's a leap year. Nice and easy, huh?

Now we can run some tests using our IsLeapYear function, just like Tim did:

$ IsLeapYear && echo Leaping lizards! || echo Arf, no
Arf, no
$ IsLeapYear 2012 && echo Leaping lizards! || echo Arf, no
Leaping lizards!
$ IsLeapYear 2000 && echo Leaping lizards! || echo Arf, no
Leaping lizards!
$ IsLeapYear 1900 && echo Leaping lizards! || echo Arf, no
Arf, no

Assuming the current year is not a Leap Year, we could even wrap a loop around IsLeapYear to figure out the next leap year:

$ y=$(date +%Y); while :; do IsLeapYear $((++y)) && break; done; echo $y
2016

We begin by initializing $y to the current year. Then we go into an infinte loop ("while :; do..."). Inside the loop we add one to $y and call IsLeapYear. If IsLeapYear returns true, then we "break" out of the loop. When the loop is all done, simply echo the last value of $y.

Stick that in your PowerShell pipe and smoke it, Tim!

Episode #174: Lightning Lockdown

Hal firewalls fast

Recently a client needed me to quickly set up an IP Tables firewall on a production server that was effectively open on the Internet. I knew very little about the machine, and we couldn't afford to break any of the production traffic to and from the box.

It occurred to me that a decent first approximation would be to simply look at the network services currently in use, and create a firewall based on that. The resulting policy would probably be a bit more loose than it needed to or should be, but it would be infinitely better than no firewall at all!

I went with lsof, because I found the output easier to parse than netstat:

# lsof -i -nlP | awk '{print $1, $8, $9}' | sort -u
COMMAND NODE NAME
httpd TCP *:80
named TCP 127.0.0.1:53
named TCP 127.0.0.1:953
named TCP [::1]:953
named TCP 150.123.32.3:53
named UDP 127.0.0.1:53
named UDP 150.123.32.3:53
ntpd UDP [::1]:123
ntpd UDP *:123
ntpd UDP 127.0.0.1:123
ntpd UDP 150.123.32.3:123
ntpd UDP [fe80::baac:6fff:fe8e:a0f1]:123
ntpd UDP [fe80::baac:6fff:fe8e:a0f2]:123
portreser UDP *:783
sendmail TCP 150.123.32.3:25
sendmail TCP 150.123.32.3:25->58.50.15.213:1526
sendmail TCP *:587
sshd TCP *:22
sshd TCP 150.123.32.3:22->121.28.56.2:39054

I could have left off the process name, but it helped me decide which ports were important to include in the new firewall rules. Honestly, the output above was good enough for me to quickly throw together some workable IP Tables rules. I simply saved the output to a text file and hacked things together with a text editor.

But maybe you only care about the port information:

# lsof -i -nlP | awk '{print $9, $8, $1}' | sed 's/.*://' | sort -u
123 UDP ntpd
1526 TCP sendmail
22 TCP sshd
25 TCP sendmail
39054 TCP sshd
53 TCP named
53 UDP named
587 TCP sendmail
783 UDP portreser
80 TCP httpd
953 TCP named
NAME NODE COMMAND

Note that I inverted the field output order, just to make my sed a little easier to write

If you wanted to go really crazy, you could even create and load the actual rules on the fly. I don't recommend this at all, but it will make Tim's life harder in the next section, so here goes:

lsof -i -nlP | tail -n +2 | awk '{print $9, $8}' | 
    sed 's/.*://' | sort -u | tr A-Z a-z | 
    while read port proto; do ufw allow $port/$proto; done

I added a "tail -n +2" to get rid of the header line. I also dropped the command name from my awk output. There's a new "tr A-Z a-z" in there to lower-case the protocol name. Finally we end with a loop that takes the port and protocol and uses the ufw command line interface to add the rules. You could do the same with the iptables command and its nasty syntax, but if you're on a Linux distro with UFW, I strongly urge you to use it!

So, Tim, I figure you can parse netstat output pretty easily. How about the command-line interface to the Windows firewall? Remember, adversity builds character...

Tim builds character

When I first saw this I thought, "Man, this is going to be easy with the new cmdlets in PowerShell v4!" There are a lot of new cmdlets available in PowerShell version 4, and both Windows 8.1 and Server 2012R2 ship with PowerShell version 4. In addition, PowerShell version 4 is available for Windows 7 SP1 (and later) and Windows Server 2008 R2 SP1 (and later).

The first cmdlet that will help us out here is Get-NetTCPConnection. According to the help page this cmdlet "gets current TCP connections. Use this cmdlet to view TCP connection properties such as local or remote IP address, local or remote port, and connection state." This is going to be great! But...

It doesn't mention the process ID or process name. Nooooo! This can't be. Let's look at all the properties of the output objects.

PS C:\> Get-NetTCPConnection | Format-List *

State                    : Established
AppliedSetting           : Internet
Caption                  :
Description              :
ElementName              :
InstanceID               : 192.168.1.167++445++10.11.22.33++49278
CommunicationStatus      :
DetailedStatus           :
HealthState              :
InstallDate              :
Name                     :
OperatingStatus          :
OperationalStatus        :
PrimaryStatus            :
Status                   :
StatusDescriptions       :
AvailableRequestedStates :
EnabledDefault           : 2
EnabledState             :
OtherEnabledState        :
RequestedState           : 5
TimeOfLastStateChange    :
TransitioningToState     : 12
AggregationBehavior      :
Directionality           :
LocalAddress             : 192.168.1.167
LocalPort                : 445
RemoteAddress            : 10.11.22.33
RemotePort               : 49278
PSComputerName           :
CimClass                 : ROOT/StandardCimv2:MSFT_NetTCPConnection
CimInstanceProperties    : {Caption, Description, ElementName, InstanceID...}
CimSystemProperties      : Microsoft.Management.Infrastructure.CimSystemProperties

Dang! This will get most of what we want (where "want" was defined by that Hal guy), but it won't get the process ID or the process name. So much for rubbing the new cmdlets in his face.

Let's forget about Hal for a second and get what we can with this cmdlet.

PS C:\> Get-NetTCPConnection | Select-Object LocalPort | Sort-Object -Unique LocalPort
LocalPort
---------
    135
    139
    445
   3587
   5357
  49152
  49153
  49154
  49155
  49156
  49157
  49164

This is helpful for getting a list of ports, but not useful for making decisions about what should be allowed. Also, we would need to run Get-NetUDPEndpoint to get the UDP connections. This is so close, yet so bloody far. We have to resort to the old school netstat command and the -b option to get the executable name. In episode 123 we needed parsed netstat output. I recommended the Get-Netstat script available at poshcode.org. Sadly, we are going to have to resort to that again. With this script we can quickly get the port, protocol, and process name.

PS C:\> .\get-netstat.ps1 | Select-Object ProcessName, Protocol, LocalPort | 
   Sort-Object -Unique LocalPort, Protocol, ProcessName

ProcessName   Protocol      Localport
-----------   --------      ---------
svchost       TCP           135
System        UDP           137
System        UDP           138
System        TCP           139
svchost       UDP           1900
svchost       UDP           3540
svchost       UDP           3544
svchost       TCP           3587
dasHost       UDP           3702
svchost       UDP           3702
System        TCP           445
svchost       UDP           4500
...

It should be pretty obvious that the port 137-149 and 445 should not be accessible from the internet. We can filter these ports out so that we don't allow these ports through the firewall.

PS C:\> ... | Where-Object { (135..139 + 445) -NotContains $_.LocalPort }
ProcessName   Protocol      Localport
-----------   --------      ---------
svchost       UDP           1900
svchost       UDP           3540
svchost       UDP           3544
svchost       TCP           3587
dasHost       UDP           3702
svchost       UDP           3702
svchost       UDP           4500
...

Now that we have the ports and protocols we can create new firewall rules using the new New-NetFirewallRule cmdlet. Yeah!

PS C:\> .\get-netstat.ps1 | Select-Object Protocol, LocalPort | Sort-Object -Unique * | 
 Where-Object { (135..139 + 445) -NotContains $_.LocalPort } | 
 ForEach-Object { New-NetFirewallRule -DisplayName AllowedByScript -Direction Outbound 
 -Action Allow  -LocalPort $_.LocalPort -Protocol $_.Protocol }
Name                  : {d15ca484-5d16-413f-8460-a29204ff06ed}
DisplayName           : AllowedByScript
Description           :
DisplayGroup          :
Group                 :
Enabled               : True
Profile               : Any
Platform              : {}
Direction             : Outbound
Action                : Allow
EdgeTraversalPolicy   : Block
LooseSourceMapping    : False
LocalOnlyMapping      : False
Owner                 :
PrimaryStatus         : OK
Status                : The rule was parsed successfully from the store. (65536)
EnforcementStatus     : NotApplicable
PolicyStoreSource     : PersistentStore
PolicyStoreSourceType : Local
...

These new firewall cmdlets really make things easier, but if you don't have PowerShellv4 you can still use the old netsh command to add the firewall rules. Also, the Get-Netstat will support older version of PowerShell as well, so this is nicely backwards compatible. All we need to do is replace the command inside the ForEach-Object cmdlet's script block.

PS C:\> ... | ForEach-Object { netsh advfirewall firewall add rule 
 name="AllowedByScript" dir=in action=allow protocol=$_.Protocol 
 localport=$_.LocalPort }

Episode #173: Tis the Season

Hal finds some cheer
From somewhere near the borders of scriptistan, we send you:

function t { 
    for ((i=0; $i < $1; i++)); do
        s=$((8-$i)); e=$((8+$i));
        for ((j=0; j <= $e; j++)); do [ $j -ge $s ] && echo -n '^' || echo -n ' '; done;
        echo;
    done
}
function T {
    for ((i=0; $i < $1; i++)); do
        for ((j=0; j < 10; j++)); do [ $j -ge 7 ] && echo -n '|' || echo -n ' '; done;
        echo;
    done
    echo
}
t 3; t 5; t 7; T 2; echo -e "Season's Greetings\n    from CLKF"

Ed comes in out of the cold:

Gosh, I missed you guys.  It's nice to be home with my CLKF family for the holidays.  I brought you a present:

c:\>cmd.exe /v:on /c "echo. & echo A Christmas present for you: & color 24 & 
echo. & echo     0x0& for /L %a in (1,1,11) do @(for /L %b in (1,1,10) do @ set /a
%b%2) & echo 1"& echo. & echo Merry Christmas!

Tim awaits the new year:

Happy New Year from within the borders of Scriptistan!

Function Draw-Circle {
    Param( $Radius, $XCenter, $YCenter )
    
    for ($x = -$Radius; $x -le $Radius ; $x++) {
        $y = [int]([math]::sqrt($Radius * $Radius - $x * $x))
        Set-CursorLocation -X ($XCenter + $x) -Y ($YCenter + $y)
        Write-Host "*" -ForegroundColor Blue -NoNewline
        Set-CursorLocation -X ($XCenter + $x) -Y ($YCenter - $y)
        Write-Host "*" -ForegroundColor Blue -NoNewline
    }
}

Function Draw-Hat {
    Param( $XCenter, $YTop, $Height, $Width, $BrimWidth )
    
    $left = Round($XCenter - ($Width / 2))
    $row = "#" * $Width
    for ($y = $YTop; $y -lt $YTop + $Height - 1; $y++) {
        Set-CursorLocation -X $left -Y $y
        Write-Host $row -ForegroundColor Black -NoNewline
    }
    
    Set-CursorLocation -X ($left - $BrimWidth) -Y ($YTop + $Height - 1)
    $row = "#" * ($Width + 2 * $BrimWidth)
    Write-Host $row -ForegroundColor Black -NoNewline
}

Function Set-CursorLocation {
    Param ( $x, $y )

    $pos = $Host.UI.RawUI.CursorPosition
    $pos.X = $x
    $pos.Y = $y
    $Host.UI.RawUI.CursorPosition = $pos
}

Function Round {
    Param ( $int )
    # Stupid banker's rounding
    return [Math]::Round( $int, [MidpointRounding]'AwayFromZero' )
}

Clear-Host
Write-Host "Happy New Year!"
Draw-Circle -Radius 4 -XCenter 10 -YCenter 8
Draw-Circle -Radius 5 -XCenter 10 -YCenter 17
Draw-Circle -Radius 7 -XCenter 10 -YCenter 29
Draw-Hat -XCenter 10 -YTop 2 -Height 5 -Width 7 -BrimWidth 2
Set-CursorLocation -X 0 -Y 38

Episode #172: Who said bigger is better?

Tim sweats the small stuff

Ted S. writes in:

"I have a number of batch scripts which turn a given input file into a configurable amount of versions, all of which will contain identical data content, but none of which, ideally, contain the same byte content. My problem is, how do I, using *only* XP+ cmd (no other scripting - PowerShell, jsh, wsh, &c), replace the original (optionally backed up) with the smallest of the myriad versions produced by the previous batch runs?"

This is pretty straight forward, but it depends on what we want to do with the files. I assumed that the larger files should be deleted since they are redundant. This will leave us with only the smallest file in the directory. Let's start off by listing all the files in the current directory and sort them by size.

C:\> dir /A-D /OS /b
file3.txt
file2.txt
file1.txt
file4.txt

Sorting the files, and only files, in the current directory by size is pretty easy. The "/A" option filters on the object's properties and directories are filtered out with "-D". Next, the "/O" option is used to sort and the "S" tells the command to sort putting the smallest files first. Finally, the "/b" is used to show the bare format.

At this point we have the files in the proper order and in a nice light format. We can now use a For loop to delete everything while skipping the first file.

C:\> for /F "tokens=* skip=1" %i in ('dir /A-D /OS /b') do @del %i

Here is the same functionality in PowerShell:

PS C:\> Get-ChildItem | Where-Object { -not $_.PSIsContainer } | Sort-Object -Property Length | Select-Object -Skip 1 | Remove-Item

This is mostly readable. The only exception is the "PSIsContainer". Directories are container objects but files are not, so we filter out the containers (directories). Here is the same command shortented using aliases and positional parameters:

PS C:\> ls | ? { !$_.PSIsContainer } | sort Length | select -skip 1 | rm

There you go Ted, and in PowerShell even though you didn't want it. Here comes Hal brining something even smaller you don't want.

Hal's is smaller than Tim's... but less sweaty

Tim, how many times do I have to tell you, smaller is better when it comes to command lines:

ls -Sr | tail -n +2 | xargs rm

It's actually not that different from Tim's PowerShell solution, except that my "ls" command has "-S" to sort by size as a built-in. We use the "-r" flag to reverse the sort, putting the smallest file first and skipping it with "tail -n +2".

If you're worried about spaces in the file names, we could tart this one up a bit more:

ls -Sr | tail -n +2 | tr \\n \\000 | xargs -0 rm

After I use "tail" to get rid of the first, smallest file, I use "tr" to convert the newlines to nulls. That allows me to use the "-0" flag to "xargs" to split the input on nulls, and preserves the spaces in the input file names.

What may be more interesting about this Episode is the command line I used to create and re-create my files for testing. First I made a text file with lines like this:

1 3
2 4
3 1
4 2

And then I whipped up a little loop action around the "dd" command:

$ while read file size; do 
      dd if=/dev/zero bs=4K count=$size of=file$file; 
  done <../input.txt 
3+0 records in
3+0 records out
12288 bytes (12 kB) copied, 6.1259e-05 s, 201 MB/s
4+0 records in
4+0 records out
16384 bytes (16 kB) copied, 0.000144856 s, 113 MB/s
1+0 records in
1+0 records out
4096 bytes (4.1 kB) copied, 3.4961e-05 s, 117 MB/s
2+0 records in
2+0 records out
8192 bytes (8.2 kB) copied, 4.3726e-05 s, 187 MB/s

Then I just had to re-run the loop whenever I wanted to re-create my test files after deleting them.

Episode #171: Flexibly Finding Firewall Phrases

Old Tim answers an old email

Patrick Hoerter writes in:
I have a large firewall configuration file that I am working with. It comes from that vendor that likes to prepend each product they sell with the same "well defended" name. Each configuration item inside it is multiple lines starting with "edit" and ending with "next". I'm trying to extract only the configuration items that are in some way tied to a specific port, in this case "port10".

Sample Data:

edit "port10"
        set vdom "root"
        set ip 192.168.1.54 255.255.255.248
        set allowaccess ping
        set type physical
        set sample-rate 400
        set description "Other Firewall"
        set alias "fw-outside"
        set sflow-sampler enable
   next
edit "192.168.0.0"
        set subnet 192.168.0.0 255.255.0.0
    next
    edit "10.0.0.0"
        set subnet 10.0.0.0 255.0.0.0
    next
    edit "172.16.0.0"
        set subnet 172.16.0.0 255.240.0.0
    next
  edit "vpn-CandC-1"
        set associated-interface "port10"
        set subnet 10.254.153.0 255.255.255.0
    next
    edit "vpn-CandC-2"
        set associated-interface "port10"
        set subnet 10.254.154.0 255.255.255.0
    next
    edit "vpn-CandC-3"
        set associated-interface "port10"
        set subnet 10.254.155.0 255.255.255.0
    next
   edit 92
        set srcintf "port10"
        set dstintf "port1"
            set srcaddr "vpn-CandC-1" "vpn-CandC-2" "vpn-CandC-3"            
            set dstaddr "all"            
        set action accept
        set schedule "always"
            set service "ANY"            
        set logtraffic enable
    next
 

Sample Results:

edit "port10"
        set vdom "root"
        set ip 192.168.1.54 255.255.255.248
        set allowaccess ping
        set type physical
        set sample-rate 400
        set description "Other Firewall"
        set alias "fw-outside"
        set sflow-sampler enable
   next
  edit "vpn-CandC-1"
        set associated-interface "port10"
        set subnet 10.254.153.0 255.255.255.0
    next
    edit "vpn-CandC-2"
        set associated-interface "port10"
        set subnet 10.254.154.0 255.255.255.0
    next
    edit "vpn-CandC-3"
        set associated-interface "port10"
        set subnet 10.254.155.0 255.255.255.0
    next
   edit 92
        set srcintf "port10"
        set dstintf "port1"
            set srcaddr "vpn-CandC-1" "vpn-CandC-2" "vpn-CandC-3"            
            set dstaddr "all"            
        set action accept
        set schedule "always"
            set service "ANY"            
        set logtraffic enable
    next

Patrick gave us the full text and the expected output. In short, he wants the text between "edit" and "next" if it contains the text "port10". To begin this task we need to first need get each of the edit/next chunks.

PS C:\> ((cat fw.txt) -join "`n") | select-string "(?s)edit.*?next" -AllMatches | 
 select -ExpandProperty matches

This command will read the entire file fw.txt and combine it into one string. Normally, each line is treated as a separate object, but we are going to join them into a big string using the newline (`n) to join each line. Now that the text is one big string we can use Select-String with a regular expression to find all the matches. The regular expression will find text across line breaks and allows for very flexible searches so we can find our edit/next chunks. Here is a break down of the pieces of the regular expression:

• (?s) - Use single line mode where the dot (.) will match any character, including a newline character. This allows us to match text across multiple lines.
• edit - the literal text "edit"
• .*? - find any text, but be lazy, not greedy. This means it should match the smallest chunks that will match the criteria.
• next - literal text next

Now that we have the chunks we use a Where-Object filter (alias ?) to find matching objects to pass down the pipeline.

PS C:\> ((cat .\fw.txt) -join "`n") | select-string "(?s)edit.*?next" -AllMatches | 
 select -ExpandProperty matches | ? { $_.Captures | Select-String "port10" }

Inside the Where-Object filter we can check the Value property to see if it contains the text "port10". The Value property is piped into Select-String to look for the text "port10", and if it contains "port10" it continues down the pipeline, if not, it is dropped.

At this point, we have the objects we want, so all we need to do is display the results by expanding the Value and displaying it again. The expansion means that it just displays the text and no data or metadata associated with the parent object. Here is what the final command looks like.

PS C:\> ((cat .\fw.txt) -join "`n") | select-string "(?s)edit.*?next" -AllMatches | 
 select -ExpandProperty matches | ? { $_.Value | Select-String "port10" } | 
 select -ExpandProperty Value

Not so bad, but I have a feeling it is going to be worse for my friend Hal.

Old Hal uses some old tricks

Oh sure, I know what Tim's thinking here. "It's multi-line matching, and the Unix shell is lousy at that. Hal's in trouble now. Mwhahaha. The Command-Line Kung Fu title will finally be mine! Mine! Do you hear me?!? MINE!"

Uh-huh. Well how about this, old friend:

awk -v RS=next -v ORS=next '/port10/' fw.txt

While we're doing multi-line matching here, the blocks of text have nice regular delimiters. That means I can change the awk "record separator" ("RS") from newline to the string "next" and gobble up entire chunks at a time.

After that, it's smooth sailing. I just use awk's pattern-matching operator to match the "port10" strings. Since I don't have an action defined, "{print}" is assumed and we output the matching blocks of text.

The only tricky part is that I have to remember to change the "output record separator" ("ORS") to be "next". Otherwise, awk will use its default ORS value, which is newline. That would give me output like:

$ awk -v RS=next '/port10/' fw.txt
edit "port10"
        set vdom "root"
        set ip 192.168.1.54 255.255.255.248
        set allowaccess ping
        set type physical
        set sample-rate 400
        set description "Other Firewall"
        set alias "fw-outside"
        set sflow-sampler enable
   

  edit "vpn-CandC-1"
        set associated-interface "port10"
        set subnet 10.254.153.0 255.255.255.0
    

    edit "vpn-CandC-2"
        set associated-interface "port10"
...

The "next" terminators get left out and we get extra lines in the output. But when ORS is set properly, we get exactly what we were after:

$ awk -v RS=next -v ORS=next '/port10/' fw.txt
edit "port10"
        set vdom "root"
        set ip 192.168.1.54 255.255.255.248
        set allowaccess ping
        set type physical
        set sample-rate 400
        set description "Other Firewall"
        set alias "fw-outside"
        set sflow-sampler enable
   next
  edit "vpn-CandC-1"
        set associated-interface "port10"
        set subnet 10.254.153.0 255.255.255.0
    next
    edit "vpn-CandC-2"
        set associated-interface "port10"
...

So that wasn't bad at all. Sorry about that Tim. Maybe next time, old buddy.

Episode #170: Fearless Forensic File Fu

Hal receives a cry for help

Fellow forensicator Craig was in a bit of a quandary. He had a forensic image in "split raw" format-- a complete forensic image broken up into small pieces. Unfortunately for him, the pieces were named "fileaa", "fileab", "fileac", and so on while his preferred tool wanted the files to be named "file.001", "file.002", "file.003", etc. Craig wanted to know if there was an easy way to rename the files, using either Linux or the Windows shell.

This one's not too hard in Linux, and in fact it's a lot like something we did way back in Episode #26:

c=1; 
for f in file*; do 
    printf -v ext %03d $(( c++ )); 
    mv $f ${f/%[a-z][a-z]/.$ext}; 
done

You could remove the newlines and make that one big long line, but I think it's a bit easier to read this way. First we initialize a counter variable $c to 1. Then we loop over each of the files in our split raw image.

The printf statement inside the loop formats $c as three digits, with however many leading zeroes are necessary ("%03d"). There are a couple of tricky bits in the printf though. First is we're assigning the output of printf to a variable $ext ("-v ext"). Second, we're doing a little arithmetic on $c at the same time and using the "++" operator to increment the value of $c each time through the loop-- that's the "$(( c++ ))" part.

Then we use mv to rename our file. I'm using the variable substitution operator like we did in Episode #26. The format again is "${var/pattern/substitution}" and here the "%" after the first slash means "match at the end of the string". So I'm replacing the last two letters in the file name with a dot followed by our $ext value. And that's exactly what Craig wanted!

All of the symbols in this solution make it appear like a little chunk of line noise, but it's nowhere near as ugly as Ed's CMD.EXE solution in Episode #26. Here's hoping Tim's Powershell solution is a bit more elegant.

Tim finishes before September ends!

Elegance where here we come!

Long Version:
PS C:\> $i=1; Get-ChildItem file?? | Sort-Object -Propery Name | 
  ForeEach-Object { MoveItem -Path $_ -Destination ("file.{0:D3}" -f $i++) }
Shortened Version:
PS C:\> ls file?? | sort name | % { move $_ -dest ("file.{0:D3}" -f $i++) }

We start off by initializing our counter variable ($i) to 1 just like Hal did. Next, we list all the files that start with "file" and are followed by exactly two characters (each ? matches exactly 1 character of any kind). The results are then sorted by the file name to ensure that the files are renamed in the correct order. The results are then fed into the ForEach-Object cmdlet (alias %).

The ForEach-Object loop will operate on each object (file) as it moves down the pipeline. One at a time, each file will be represented by the current pipeline object ($_). The Move-Item cmdlet (alias move) is used to rename a file; to move it to its new name. The source path is provided by the current object and the destination is determined using the format operator (-f) and our counter ($i). The format operator will print $i as a three digit number prefixed with leading zeros and "file.". The ++ after $i will increment the counter after it has been used.

That is much cleaner than Ed's example...and even cleaner than Hal's to boot!

Update:

Reader m_cnd writes in with a solution for CMD. vm

C:\> for /F "tokens=1,2 delims=:" %d in ('dir /on /b file* ^| 
findstr /n "file"') do for /F %x in ('set ext^=00%d^&^& 
cmd /v:on /c "echo !ext:~-3!"') do rename %e file.%x

Nice work!

Episode #169: Move Me Maybe

Tim checks the mailbag

Carlos IHaveNoLastName writes in asking for a way to move a directory to a new destination. That's easy, but the directory should only be moved if the the directory (at any depth) does NOT contain a file with a specific extenstion.

Here is an example of a sample directory structure:

SomeTopDir1
|-OtherDir1
|  |-File1
|  |-File2
|  |-File2
|-OtherDir2
   |-File1
   |-File.inprogress

SomeTopDir2
|-OtherDir1
|  |-File1
|  |-File2
|  |-File2
|-OtherDir2
   |-File1
   |-File2

In this example we should NOT move SomeTopDir1 because it contains a file with the string "inprogress". We should however move SomeTopDir2 because it contains no such file. In short, "inprogress" means leave it alone.

Executing this in PowerShell is quite easy. CMD is a pain, and I'll skip that crazy long command because it is a circus trick. Here is the command to do exactly what Carlos asked:

PS C:\jobsdir> Get-ChildItem | ? { $_.PSIsContainer } | 
    ? { -not ( Get-ChildItem $_ -Recurse -Filter *.inprogress ) } | 
    Move-Item -Destination \archive
 

This command with use Get-ChildItem to list the contents of the current directory. We first filter for Directories (Container objects) just in case there are files in the root of the directory that we don't want to move. Next, another Where-Object cmdlet (alias ?) is used to check all the sub-directories and look for a file matching "*.inprogress". The -Not operator inverts the match so that only directores with a "*.inprogress" file will be passed down the pipeline.

At this point we have the directories that do not contain this file. The results are then piped into Move-Item and the directories are moved to the \archive directory.

One of the other criteria that Mr. IHaveNoLastName requested is that the command must work on XP. Well it does, but only if you install PowerShell. Sadly, XP does not support PowerShell v3. With PowerShell v3's simplified syntax (and some additional aliases) we can shorten the command to this:

PS C:\jobsdir> ls | ? PSIsContainer | ? { -not ( ls $_ -r -fi *.inprogress ) } | 
    mv -d \archive
 

Thanks for an easy one Carlos! Hal, your turn. I suspect this is will be almost as easy for you (even though it won't work on XP).

Hal takes it easy

This one's quite do-able in the shell. But unlike Tim's solution, the most straightforward approach in Linux is a loop:

for i in *; do [ "$(find $i -type f -name \*.inprogress)" ] || mv $i /some/dest; done

The loop is over all of the directories in the current directory. Inside the loop we run a find command looking for "*.inprogress" files. If we find any, then the test operator ("[ ... ]") returns true and we don't do the mv command on the other side of the "||". If we find nothing, then the directory gets moved. Easy peasy

"But wait!", I hear you cry, "That was too easy. And besides, you're running a mv command for each individual directory!"

OK, fine. You want a single mv command? Here you go:

mv $(ls | grep -vf <(find * -type f -name \*.inprogress | cut -f1 -d/)) /some/dest

Happy now?

The best way to puzzle this one out is to start with the command in the innermost parentheses:

find * -type f -name \*.inprogress | cut -f1 -d/

The find command returns the pathnames of all of the *.inprogress files, and the cut command pulls off the top-level directory name. If there are multiple *.inprogress files in a single directory, we'll get multiple instances of the top-level directory name, but that doesn't really matter.

The "<( ... )" syntax takes the output of our find pipeline and lets it be treated as an input file for another command:

ls | grep -vf <( ... )

We take the output of ls and use "grep -v" to filter out directories we don't want. Normally "grep -f" takes a list of patterns from an input file, but in this case we use the "<( ... )" syntax to substitute our find output instead of a normal input file. So we suppress any directories that have a *.inprogress file in them. Anything left over is a directory without a *.inprogress file, which is precisely the set of directories we want to move.

So we wrap the complicated ls pipline up in "$(...)" so that the output-- the list of directories we want to move-- is substituted into the "mv $(...) /some/dest" command. And that gets us to where we want to be.

Or you could use the same idea, but with xargs:

ls | grep -vf <(find * -type f -name \*.inprogress | cut -f1 -d/) | xargs mv -d /some/dest

This looks a lot more like Tim's approach in Powershell. However, this makes use of the "mv -d /some/dest ..." syntax that's supported in the GNU version of the command, but not widely supported in other more traditional Unix distros.

Oh, and by the way, Tim, this all works fine under Windows XP if you'd just install Cygwin like I've been telling you to...

Update:

m_cnd wrote in again with a shortcut for CMD.EXE:

dir SomeTopDir /s /b | findstr /i /e ".extension" > nul || move SomeTopDir Destination

I use this trick all the time, so I feel bad that I missed it here. With his shortcut I can put together a For loop (our favorite, and only, text parser) to do the work.

C:\> for /F "tokens=*" %i in ('dir /b /AD') do dir "%i" /s /b /a-d "%i\*.extension" 1>nul 2>nul || move "%i" Destination

The Dir command with /AD will list directories and not files. We can then use the output to search and move if necessary. The Tokens and quotes are used in case the directory names contain spaces.

Thanks again m_cnd!

Episode #168: Scan On, You Crazy Command Line

Hal gets back to our roots

With one ear carefully tuned to cries of desperation from the Internet, it's no wonder I picked up on this plea from David Nides on Twitter:

Request today, we need 2 scan XX terabytes of data across 3k file shares 4any files that have not been MAC since 2012. Then move files to x.— David Nides (@DAVNADS) March 13, 2013

Whenever I see a request to scan for files based on a certain criteria and then copy them someplace else, I immediately think of the "find ... | cpio -pd ..." trick I've used in several other Episodes.

Happily, "find" has "-mtime", "-atime", and "-ctime" options we can use for identifying the files. But they all want their arguments to be in terms of number of days. So I need to calculate the number of days between today and the end of 2012. Let's do that via a little command-line kung fu, shall we? That will make this more fun.

$ days=$(( ($(date +%Y) - 2012)*365 + $(date +%j | sed 's/^0*//') ))
$ echo $days
447

Whoa nelly! What just happened there? Well, I'm doing math with the bash "$(( ... ))" operator and assigning the result to a variable called "days" so I can use it later. But what's all that line noise in the middle?

• "date +%Y" returns the current year. That's inside "$( ... )" so I can use the value in my calculations.
• I subtract 2012 from the current year to get the number of years since 2012 and multiply that by 365. Screw you, leap years!
• "date +%j" returns the current day of the year, a value from 001-365.
• Unfortunately the shell interprets values with leading zeroes as octal and errors out on values like "008" and "097". So I use a little sed to strip the leading zeroes.

Hey, I said it would be fun, not that it would necessarily be a good idea!

But now that I've got my "$days" value, the answer to David's original request couldn't be easier:

$ find /some/dir -mtime +$days -atime +$days -ctime +$days | cpio -pd /new/dir

The "find" command locates files whose MAC times are all greater than our "$days" value-- that's what the "+$days" syntax means. After that, it's just a matter of passing the found files off to "cpio". Calculating "$days" was the hard part.

My final solution was short enough that I tweeted it back to David. Which took me all the way back to the early days of Command-Line Kung Fu, when Ed Skoudis had hair would tweet cute little CMD.EXE hacks that he could barely fit into 140 characters. And I would respond with bash code that would barely line wrap. Ah, those were the days!

Of course, Tim was still in diapers then. But he's come so far, that precocious little rascal! Let's see what he has for us this time!

Tim gets an easy one!

Holy Guacamole! This is FINALLY an easy one! Robocopy makes this super easy *and* it plays well with leap years. I feel like it is my birthday and I can finally get out of these diapers.

PS C:\> robocopy \some\dir \new\dir /MINLAD (Get-Date).DayOfYear /MINAGE (Get-Date).DayOfYear /MOV

Simply specify the source and destination directories and use /MOV to move the files. MINLAD will ignore files that have been accessed in the past X days (LAD = Last Access Date), and MINAGE does the same based on the creation date. All we need is the number of days since the beginning of the year. Fortunately, getting that number is super easy in PowerShell (I have no pity for Hal).

All Date objects have the property DayOfYear which is (surprise, surprise) the number of days since the beginning of the year (Get-Member will show all the available properties and methods of an object). All we need is the current date, which we get Get-Date.

DONE! That's all folks! You can go home now. I know you expected a long complicated command, but we don't have one here. However, if you feel that you need to read more you can go back and read the episodes where we cover some other options available with robocopy.

This command is so easy, simple, and short I could even fit it into a tweet!

Episode #167: Big MAC

Hal checks into Twitter:

So there I was, browsing my Twitter timeline and a friend forwarded a link to Jeremy Ashkenas' github site. Jeremy created an alias for changing your MAC address to a random value. This is useful when you're on a public WiFi network that only gives you a small amount of free minutes. Since most of these services keep track by noting your MAC address, as long as you keep cycling you MAC, you can keep using the network for free.

Here's the core of Jeremy's alias:

sudo ifconfig en0 ether `openssl rand -hex 6 | sed "s/\(..\)/\1:/g; s/.$//"`

Note that the syntax of the ifconfig command varies a great deal between various OS versions. On my Linux machine, the syntax would be "sudo ifconfig wlan0 hw ether..."-- you need "hw ether" after the interface name and not just "ether".

Anyway, this seemed like a lot of code just to generate a random MAC address. Besides, what if you didn't have the openssl command installed on your Linux box? So I decided to try and figure out how to generate a random MAC address in fewer characters and using commonly built-in tools.

What does a MAC address look like? It's six pairs of digits with colons between. "Pairs of digits with colons between" immediately made me think of time values. And this works:

$ date +00:11:22:%T
00:11:22:11:23:08

Just print three pairs of fixed digits followed by "hh:mm:ss". I originally tried "date +%T:%T". But in my testing, the ifconfig command didn't always like the fake MAC addresses that were generated this way. So specifying the first few octets was the way to go.

The only problem is that this address really isn't all that random. If there were a lot of people on the same WiFi network all using this trick, MAC address collisions could happen pretty easily. Though if everybody chose their own personal sequence for the first three octets, you could make this a lot less likely.

The Linux date command lets you output a nine-digit nanoseconds value with "%N". I could combine that with a few leading digits to generate a pseudo-random sequence of 12 digits:

$ date +000%N
000801073504

But now we need to use the sed expression in Jeremy's original alias to put the colons in. Or do we?

$ sudo ifconfig wlan0 hw ether $(date +000%N)
$ ifconfig wlan0
wlan0     Link encap:Ethernet  HWaddr 00:02:80:12:43:53  
...

I admit that I was a little shocked when I tried this and it actually worked! I can't guarantee that it will work across all Unix-like operating systems, but it allows me to come up with a much shorter bit of fu compared to Jeremy's solution.

What if you were on a system that didn't have openssl installed and didn't have a date command that had nanosecond resolution? If your system has a /dev/urandom device (and most do) you could use the trick we used way back in Episode #85:

$ sudo ifconfig wlan0 hw ether 00$(head /dev/urandom | tr -dc a-f0-9 | cut -c1-10)
$ ifconfig wlan0
wlan0     Link encap:Ethernet  HWaddr 00:7a:5f:be:a2:ca
...

Again I'm using two literal zeroes at the front of the MAC address, so that I create addresses that don't cause ifconfig to error out on me.

The expression above is not very short, but at least it uses basic commands that will be available on pretty much any Unix-like OS. If your ifconfig needs colons between the octets, then you'll have to add a little sed like Jeremy did:

$ sudo ifconfig wlan0 hw ether \
    00$(head /dev/urandom | tr -dc a-f0-9 | sed 's/\(..\)/:\1/g;' | cut -c1-15)
$ ifconfig wlan0
wlan0     Link encap:Ethernet  HWaddr 00:d9:3e:0d:80:57  
...

Jeremy's sed is more complicated because he takes 12 digits and adds colons after each octet, but leaves a trailing colon at the end of the address. So he has a second substitution to drop the trailing colon. I'm using cut to trim off the extra output anyway, so I don't really need the extra sed substitution. Also, since I'm specifying the first octet outside of the "$(...)", my sed expression puts the colons in front of each octet.

So there you have it. There's a very short solution for my Linux box that has a date command with nanosecond resolution and a very forgiving ifconfig command. And a longer solution that should work on pretty much any Unix-like OS. But even my longest solution is surely going to look great compared to what Tim's going to have to deal with.

Tim wishes he hadn't checked into Twitter:

I'm so jealous of Hal. I think his entire command is shorter than the name of my interface. This command is painful, quite painful. I would very much suggest something like Technitium's Mac Address Changer, but since Hal set me up here we go...

To start of, we need to get the name of our target interface. Sadly, the names of the interfaces aren't as simply named as they are on a *nix box. Not only is the name 11 times longer, but it is not easy to type. If you run "ipconfig /all" you can find the name and copy/paste it. (By the way, I'm only going to use PowerShell here, the CMD.EXE version would be ugly^2).

PS C:\> $ifname = "Intel(R) 82574L Gigabit Network Connection"

The MAC address for each interface is stored somewhere in the registry under this even-less-easy-to-type Key:
HKLM:\SYSTEM\CurrentControlSet\Control\Class\{4D36E972-E325-11CE-BFC1-08002bE10318}\[Some 4 digit number]\

First, a bit of clarification. Many people (erroneously) refer to Keys as the name/value pairs, but those pairs are actually called Values. A key is the container object (similar to a directory). How about that for a little piece of trivia?

With PowerShell we can use Get-ChildItem (alias dir, ls, gci) to list all the keys and then Get-ItemProperty (alias gp) to list the DriverDesc values. A simple Where-Object filter (alias where, ?) will find the key we need.

PS C:\> Get-ChildItem HKLM:\SYSTEM\CurrentControlSet\Control\Class\`{4D36E972-E325-
 11CE-BFC1-08002bE10318`}\[0-9]*\ | Get-ItemProperty -Name DriverDesc | 
 ? DriverDesc -eq "Intel(R) 82574L Gigabit Network Connection"
DriverDesc   : Intel(R) 82574L Gigabit Network Connection
PSPath       : Microsoft.PowerShell.Core\Registry::HKEY_LOCAL_MACHINE\SY...0318}\0010
PSParentPath : Microsoft.PowerShell.Core\Registry::HKEY_LOCAL_MACHINE\SY...0318}
PSChildName  : 0010
PSProvider   : Microsoft.PowerShell.Core\Registry

Note: the curly braces ({}) need to be prefixed with a back tick (`) so they are not interpreted as a script block.

So now we have the Key for our target network interface. Next, we need to generate a random MAC address. Fortunately, Windows does not requires the use of colons (or dots) in the MAC address. This is nice as it makes our command a little easier to read (a very very little, but we'll take any win we can). The acceptable values are between 000000000000 and fffffffffffe (ffffffffffff is the broadcast address and should be avoided). This is the range between 0 and 2^48-2 ([Math]::Pow(2,8*6)-2 = 281474976710654). The random number is then formatted as a 12 digit hex number.

PS C:\> [String]::Format("{0:x12}", (Get-Random -Minimum 0 -Maximum 281474976710655))
16db434bed4e
PS C:\> [String]::Format("{0:x12}", (Get-Random -Minimum 0 -Maximum 281474976710655))
a31bfae1296d

We have a random MAC address value and we know the Key, now we need to put those two pieces together to actually change the MAC address. The New-ItemProperty cmdlet will create the value if it doesn't exist and the -Force option will overwrite it if it already exists. This results in the final version of our ugly command. We could shorten the command a little (very little) bit, but this is the way it's mother loves it, so we'll leave it alone.

PS C:\> ls HKLM:\SYSTEM\CurrentControlSet\Control\Class\`{4D36E972-E325-11CE-BFC1-
 08002bE10318`}\0*\ | Get-ItemProperty -Name DriverDesc | ? DriverDesc -eq 
 "Intel(R) 82574L Gigabit Network Connection" | New-ItemProperty -Name 
 NetworkAddress -Value ([String]::Format("{0:x12}", (Get-Random -Minimum 0 
 -Maximum 281474976710655))) -PropertyType String -Force

You would think that after all of this mess we would be good to go, but you would be wrong. As with most things Windows, you could reboot the system to have this take affect, but that's no fun. We can accomplish the same goal by disabling and enabling the connection. This syntax isn't too bad, but we need to use a different long name here.

PS C:\> netsh set interface name="Wired Ethernet Connection" admin=DISABLED
PS C:\> netsh set interface name="Wired Ethernet Connection" admin=ENABLED

At this point you should be running with the new MAC address.

And now you can see why I recommend a better tool to do this...and why I envy Hal.

EDIT:
Andres Elliku wrote in and reminded me of the new NetAdapter cmdlets in version 3. Here is his response.

This is directed mainly to Tim as a suggestion to decrease his pain. :) (Tim's comment: for this I'm thankful!)

Powershell has included at least since version 2.0 the NetAdapter module. This means that in Powershell you could set the mac aadress with something like:

PS C:\> Set-NetAdapter -Name "Wi-Fi" -MacAddress ([String]::Format("{0:x12}", 
(Get-Random -Minimum 0 -Maximum 281474976710655))) | Restart-NetAdapter

NB! The adapter name might vary, but usually they are still pretty short.

The shorter interface names is one of my favorite features of Windows 8 and Windows 2012. Also, with these cmdlets we don't need the name if the device (Intel blah blah blah) but the newly shortened interface name. Great stuff Andres. Thanks for writing in! -Tim

EDIT 2:

@PowerShellGuy tweeted an even shorted version using the format operator and built-in byte conversion:

PS C:\> Set-NetAdapter "wi-fi" -mac ("{0:x12}" -f (get-random -max (256tb-1))) | 
Restart-NetAdapter

Well done for really shortening the command -Tim

Episode #166: Ping A Little Log For Me

We've been away for a while because, frankly, we ran out of material. In the meantime we tried to come up with some new ideas and there have had a few requests, but sadly they were all redundant, became scripts, or both. We've been looking long and hard for Fu that works in this format, and we've finally found it!

Nathan Sweaney wrote in with a great idea! It isn't a script, it isn't redundant, and it is quite useful. Three of the four of the criteria that makes a great episode (the fourth being beer fetching or beer opening). To top it off Nathan wrote the CMD.EXE portion himself. Thanks Nathan!

--Tim

Nathan Sweaney writes in:

Ping Network Monitor

Occasionally we have issues in the field where we think a customer's device is occasionally losing a connection, but we're not sure if, or when, or for how long. We need a log of when the connection is dropping so that we can compare to the customer's reports of issues. Sure there are fancy network monitoring tools that can help, but we're in a hurry with no budget.

In Linux this would be easy, but these are Windows boxen. So I hacked together the following one-liner for our techs to use in the field.

This command will ping an IP address once every second and when it doesn't get a response, it will log the time-stamp in a text file. Then we can compare those time-stamps to failure reports from the customer.

To use it, simply change the IP address 8.8.8.8 near the beginning to whatever IP we need to monitor. Then open a command prompt, CD into the directory you want the log file created, and run the command.

C:\> cmd.exe /v:on /c "FOR /L %i in (1,0,2) do @ping -n 1 8.8.8.8 |
find "Request timed out">NUL && (echo !date! !time! >> PingFail.txt) &
ping -n 2 127.0.0.1>NUL"

So let's dissect this. It's mostly just a combination of examples Ed has mentioned in the past.

First, we're using the "cmd.exe /v:on /c" command to allow for delayed environment variable expansion. Ed has explained in the past why that lets us do flexible variable parsing. This command wraps everything else.

The next layer of our onion is an infinite "FOR /L" loop that Ed mentioned WAY back. We're counting from 1 to 2 in steps of 0 so that our command will continue running until we manually stop it.

Inside of our FOR loop is where we really get to the meat. We've basically got 4 steps:

1) First we see @ping -n 1 8.8.8.8. The @ symbol says to hide the echo of the command to the screen. The switch (-n 1) says to only ping the IP once. And of course 8.8.8.8 is the address we want to ping.

2) Next we pipe the results of our ping into the FIND command and search for "Request timed out" to see if the ping failed. The last part of that >NUL says to dump the output from this command into NUL, because we don't really need to see it.

3) Now we get fancy. The && says to only run this command if the previous command succeeded. In other words, if our FIND command finds the text, which means our ping failed, then we run this command. And we've enclosed this command in parenthesis contain it as a single command. We need to use the "cmd.exe /v:on /c" command at the beginning to allow for delayed environment variable expansion; that way our time & date changes each iteration. So %date% and %time% becomes !date! and !time!.

And finally we're redirecting our output to a file called PingFail.txt. We use the >> operator append each new entry rather than overwrite with just >.

4) And finally we're on to the last step. As mentioned before, the & says to run the next command no matter what has already happened. This command simply pings localhost with (-n 2) which will give us a one-second delay. The first ping happens immediately, and the second ping happens after one second. This slows down our original ping back in step 1 which would otherwise fire off like a machine gun as fast as the FOR loop can go. Lastly, we're redirecting the output with >NUL because we don't care to see it.

WOW. I said it was convoluted. But it works, and it's rather simple to use.

Tim finds a letter in the mail slot:

Wow, it has been a while since we've dusted off the ol' kung fu for a blog post. I've missed it and I know Hal as too. In fact, he hasn't showered since our last episode. True story. This was his silent (but deadly) protest against our lack of ideas and usable suggestions. The Northwest can breathe a sigh of relief (in the now fresher air) now that we are back for this episode. I for one, missed the blog. Back to the Fu...

Nathan wrote in with his idea to log Ping failures. What a great idea for a quick and dirty network monitor. Thanks to CMD.EXE he's got a bit of funkyness to his command. Fortunately, we can be a little smoother with our approach.

PS C:\> for (;;) { Start-Sleep 2; ping -n 1 8.8.8.8 >$null; if(-not $?) { Get-Date -Format s | Tee-
Object mylog.txt -Append }}
2013-03-12T12:34:56

We start off with an infinite loop using the For loop but without any loop control structures. Without these structures there is nothing to limit the loop, and it will run forever...it will be UNSTOPPABLE! MWAAAAAHAHAHAHAHA! <cough> <cough> Sorry about that, it's been a while.

Inside our infinite loop we sleep for a few seconds. We could do it at the end, but for some reason I get inconsistent results when I do that. I have no idea why, and I've tried troubleshooting it for hours. That's OK, a pre-command nap never killed anyone.

After our brief nap, we do the ping. The results are sent into the garbage can that is the $NULL variable. Following this command we check the error state of the previous command by checking the value of $?. This variable is True if the previous command ran without error, if there was an error the the command is False. The If Statement is used to branch our logic based on this value. If it is False, the ping failed, and we need to log the error.

Inside our branch we get the current date with Get-Date (duh!) and change the format to the sortable format. We could use any format, but the OCD part of me likes this format. The formatted date is piped into the Tee-Object command which will append the date to a file as well as output to our console.

Notice we used the For loop here instead of a While loop. I did this to save single character. We can save a few more characters by using this command using aliases, shortened parameter names, and a little magic in our For loop.

PS C:\> for (;;sleep 2) {ping -n 1 8.8.8.8 >$null; if(-not $?) { date -f s | tee mylog.txt -a }}

I moved the Start-Sleep (alias sleep) cmdlet inside the For loop control. The For loops looks like this:

for ( variable initialization; condition; variable update ) {
  Code to execute while the condition is true
}

The variable initialization is run once before our loop starts. The condition is checked every time through the loop to see if we should continue the loop. We have no variable we care to initialize, and we want the loop to run forever so we don't use a condition. The variable update piece is executed after each time through the loop, and this we can use. Instead of modifying a variable used in the loop, we take a lovely two second nap. This gives us our nice delay between each ping.

There you have the long awaited PowerShell version of this command. It is better than CMD.EXE, but there is no nice way to use the short-circuit operator && or || operators to make this command more efficient. Don't tell Hal, but I'm really jealous of the way his shell can be used to complete this task. I'm jealous of his terseness...and his full head of hair.

Hal washes clean

Let's be clear. The only thing that smells around here is the Windows shells. Have to use ping to put a sleep in your loop? Sleep that works at the start of the loop but not the end? What kind of Mickey Mouse operating system is that?

The Linux solution doesn't look a lot different from the Windows solutions:

while :; do ping -c 1 -W 1 8.8.8.8 >/dev/null || date; sleep 1; done

"while :; do ... done" is the most convenient way of doing an infinite loop in the shell. The ping command uses the "-c 1" option to only send a single ping and "-W 1" to only wait one second for the response. We send the ping output to /dev/null so that it doesn't clutter the output of our loop. Whenever the ping fails, it returns false and we end up running the date command on the right-hand side to output a timestamp. The last thing in the loop is a sleep for one second. And yes, Tim, it actually works at the end of the loop in my shell.

Well that was easy. Hmmm, I don't want to embarrass Nathan and Tim by making my part of the Episode too short. How about we make the output of the date command a little nicer:

while :; do ping -c 1 -W 1 8.8.8.8 >/dev/null || date '+%F %T'; sleep 1; done

"%F" is the "full" ANSI-style date format "2013-03-12" and "%T" is the time in 24-hour notation. So we get "2013-03-12 04:56:22" instead of the default "Tue Mar 12 04:56:22 EST 2013"

Oh, you want to save the output in a file as well as having it show up in your terminal window? No problemo:

while :; do ping -c 1 -W 1 8.8.8.8 >/dev/null || date; sleep 1; done | tee mylog.txt

Hooray for tee!

Well I can't tart this up any more to save Tim and Nathan's fragile egos. So I'm outta here to go find a shower.