Episode #181: Making Contact

Hal wanders back on stage
Whew! Sure is dusty in here!
Man, those were the days! It started with Ed jamming on Twitter and me heckling from the audience. Then Ed invited me up on stage (once we built the stage), and that was some pretty sweet kung fu. Then Tim joined the band, Ed left, and the miles, and the booze, and the groupies got to be too much.
But we still get fan mail! Here's one from superfan Josh Wright that came in just the other day:
I have a bunch of sub-directories, all of which have files of various names. I want to produce a list of directories that do not have a file starting with "ContactSheet-*.jpg".
I thought I would just use "find" with "-exec test":

find . -type d \! -exec test -e "{}/ContactSheet\*" \; -print

Unfortunately, "test" doesn't support globbing, so this always fails.
Here's a sample directory tree and files. I thought this might be an interesting CLKF topic.

$ ls
Adriana  Alessandra  Gisele  Heidi  Jelena  Kendall  Kim  Miranda
$ ls *
Adriana:
Adriana-1.jpg  Adriana-3.jpg  Adriana-5.jpg
Adriana-2.jpg  Adriana-4.jpg  Adriana-6.jpg

Alessandra:
Alessandra-1.jpg  Alessandra-4.jpg  ContactSheet-Alessandra.jpg
Alessandra-2.jpg  Alessandra-5.jpg
Alessandra-3.jpg  Alessandra-6.jpg

Gisele:
Gisele-1.jpg  Gisele-3.jpg  Gisele-5.jpg
Gisele-2.jpg  Gisele-4.jpg  Gisele-6.jpg

Heidi:
ContactSheet-Heidi.jpg  Heidi-2.jpg  Heidi-4.jpg  Heidi-6.jpg
Heidi-1.jpg             Heidi-3.jpg  Heidi-5.jpg

Jelena:
ContactSheet-Jelena.jpg  Jelena-2.jpg  Jelena-4.jpg  Jelena-6.jpg
Jelena-1.jpg             Jelena-3.jpg  Jelena-5.jpg

Kendall:
Kendall-1.jpg  Kendall-3.jpg  Kendall-5.jpg
Kendall-2.jpg  Kendall-4.jpg  Kendall-6.jpg

Kim:
ContactSheet-Kim.jpg  Kim-2.jpg  Kim-4.jpg  Kim-6.jpg
Kim-1.jpg             Kim-3.jpg  Kim-5.jpg

Miranda:
Miranda-1.jpg  Miranda-3.jpg  Miranda-5.jpg
Miranda-2.jpg  Miranda-4.jpg  Miranda-6.jpg

OK, Josh. I'm feeling you on this one. Maybe I can find some of the lost magic.
Because Josh started this riff with a "find" command, my brain went there first. But my solutions all ended up being variants on running two find commands-- get a list of the directories with "ContactSheet" files and "subtract" that from the list of all directories. Here's one of those solutons:

$ sort <(find * -type d) <(find * -name ContactSheet-\* | xargs dirname) | uniq -u
Adriana
Gisele
Kendall
Miranda

The first "find" gets me all the directory names. The second "find" gets all of the "ContactSheet" files, and then that output gets turned into a list of directory names with "xargs dirname". Then I use the "<(...)" construct to feed both lists of directories into the "sort" command. "uniq -u" gives me a list of the directories that only appear once-- which is the directories that do not have a "ContactSheet" file in them.
But I wasn't cool with running the two "find" commands-- especially when we might have a big set of directories. And then it hit me. Just like our CLKF jam is better when I had my friends Ed and Tim rocking out with me, we can make this solution better by combining our selection criteria into a single "find" command:

$ find * \( -type d -o -name ContactSheet\* \) | sed 's/\/ContactSheet.*//' | uniq -u
Adriana
Gisele
Kendall
Miranda

By itself, the "find" command gives me output like this:

$ find * \( -type d -o -name ContactSheet\* \)
Adriana
Alessandra
Alessandra/ContactSheet-Alessandra.jpg
Gisele
Heidi
Heidi/ContactSheet-Heidi.jpg
Jelena
Jelena/ContactSheet-Jelena.jpg
Kendall
Kim
Kim/ContactSheet-Kim.jpg
Miranda

Then I use "sed" to pick off the file name, and I end up with the directory list with the duplicate directory names already sorted together. That means I can just feed the results into "uniq -u" and everything is groovy!
Cool, man. That was cool. Now if only my friends Ed and Tim were here, that would be something else.

A Wild-Eyed Man Appears on Stage for the First Time Since December 2013

Wh-wh-where am I?  Why am I covered with dust?  Who are you people?  What's going on?

This all looks so familiar.  It reminds me of... of... those halcyon days of the early Command Line Kung Fu blog.  A strapping young Tim Medin throwing down some amazing shell tricks.  Master Hal Pomeranz busting out beautiful bash fu.  Wow... those were the days.  Where the heck have I been?

Oh wait... what?  You want me to solve Josh's dilemma using cmd.exe?  What, am I a trained monkey who dances for you when you turn the crank on the music box?  Oh I can hear the sounds now.  That lovely music box... with its beautiful tunes that are so so so hypnotizing... so hypnotizing...

...and then Ed starts to dance...

I originally thought, "Uh-oh... a challenge posed by Josh Wright and then smashed by Hal is gonna be an absolute pain in the neck in cmd.exe, the Ruprecht of shells."  But then, much to my amazement, the answer all came together in about 3 minutes.  Here ya go, Big Josh.

c:\> for /D /R %i in (*) do @dir /b %i | find "ContactSheet" > nul || echo %i

The logic works thusly:

I've got a "for" loop, iterating over directories (/D) in a recursive fashion (/R) with an iterator variable of %i which will hold the directory names.  I do this for everything in the current working directory and down (in (*)... although you could put a directory path in there to start at a different directory).  That'll spit out each directory. At each iteration through the loop, I do the following:

• Turn off the display of commands so it doesn't clutter the output (@)
• Get a directory listing of the current directory indicated by the iterator variable, %i.  I want this directory listing in bare form without the Volume Name and Size cruft but with full path (/b).  That'll spew all these directories and their contents on standard out.  Remember, I'm recursing using the /R in my for loop, so I don't need to use a /s in my dir command here.
• I take the output of the dir command and pipe it through the find command to look for the string "ContactSheet".  I throw out the output of the find command, because it's a bunch of cruft where it actually finds ContactSheet.
• But, if the find command FAILS to see the string "ContactSheet" (||), I want to display the path of the directory where it failed, so I echo %i.

Voilamundo!  There you go!  The output looks like this:

c:\tmp\test>for /D /r %i in (*) do @dir /b /s %i | find "ContactSheet" > nul || echo %i
c:\tmp\test\ChrisFall
c:\tmp\test\dan
c:\tmp\test\Fred\Fred2

I'd like to thank Josh for the most engaging challenge!  I'll now go back into my hibernative state.... Zzzzzzzzz...

...and then Tim grabs the mic...

What a fantastic throwback, a reunion of sorts. Like the return of Guns n' Roses, but more Welcome to the Jungle than Paradise City, it gets worse here everyday. We got everything you want honey, we know the commands. We are the people that can find whatever you may need.

Uh, sorry... I've missed the spotlight here. Let's get back to the commands. Here is a working command in long form and shortened form:

PS C:\Photos> Get-ChildItem -Directory | Where-Object { -not (Get-ChildItem -Path $_ -Filter ContactSheet*) }

PS C:\Photos> ls -di | ? { -not (ls $_ -Filter ContactSheet*) }

Let's take it day piece by day piece. If you want it you're gonna bleed but it's the price to pay. Well, actually, it isn't that difficult so no bleeding will be involved.

The first portion simply gets a list of directories in the current directory.

Next, we have a Where-Object filter that will pass the proper objects down the pipeline. In the filter we need to look for files in the directory passed down the pipeline ($_) containing files that start with ContactSheet. We simply invert the search with the -not operator.

With this command you can have everything you want but you better not take it from me... the other guys did some excellent work. We've missed ya'll and hopefully we will see you again. Now back into the cave to hibernate.

Episode #180: Open for the Holidays!

Not-so-Tiny Tim checks in with the ghost of Christmas present:

I know many of you have been sitting on Santa's lap wishing for more Command Line Kung Fu. Well, we've heard your pleas and are pushing one last Episode out before the New Year!

We come bearing a solution for a problem we've all encountered. Ever try to delete or modify a file and receive an error message that the file is in use? Of course you have! The real problem is trying to track down the user and/or process that has the file locked.

I have a solution for you on Windows, "openfiles". Well, sorta. This time of year I can't risk getting on Santa's bad side so let me add the disclaimer that it is only a partial solution. Here's what I mean, let's look for open files:

C:\> openfiles

INFO: The system global flag 'maintain objects list' needs
      to be enabled to see local opened files.
      See Openfiles /? for more information.


Files opened remotely via local share points:
---------------------------------------------

INFO: No shared open files found.

By default when we run this command it gives us an error that we haven't enabled the feature. Wouldn't it be nice if we could simply turn it on and then look at the open files. Yes, it would be nice...but no. You have to reboot. This present is starting to look a lot like a lump of coal. So you need know that you will encounter the problem before it happens so you can be ready for it. Bah-Humbug!

To enable "openfile" run this command:

C:\> openfile /local on

SUCCESS: The system global flag 'maintain objects list' is enabled.
         This will take effect after the system is restarted.

...then reboot.

Of course, now that we've rebooted the file will be unlocked, but we are prepared for next time. So next time when it happens we can run this command to see the results (note: if you don't specify a switch /query is implied):

C:\> openfiles /query

Files Opened Locally:
---------------------

ID    Process Name         Open File (Path\executable)                       
===== ==================== ==================================================
8     taskhostex.exe       C:\Windows\System32
224   taskhostex.exe       C:\Windows\System32\en-US\taskhostex.exe.mui
296   taskhostex.exe       C:\Windows\Registration\R00000000000d.clb
324   taskhostex.exe       C:\Windows\System32\en-US\MsCtfMonitor.dll.mui
752   taskhostex.exe       C:\Windows\System32\en-US\winmm.dll.mui
784   taskhostex.exe       C:\..\Local\Microsoft\Windows\WebCache\V01tmp.log
812   taskhostex.exe       C:\Windows\System32\en-US\wdmaud.drv.mui
...

Of course, this is a quite long list. You can use use "find" or "findstr" to filter the results, but be aware that long file names are truncated (see ID 784). You can get a full list by changing the format with "/fo LIST". However, the file name will be on a separate line from the owning process and neither "find" nor "findstr" support context.

Another oddity, is that there seems to be duplicate IDs.

C:\> openfiles /query | find "888"
888   chrome.exe           C:\Windows\Fonts\consola.ttf
888   Lenovo Transition.ex C:\..\Lenovo\Lenovo Transition\Gui\yo_btn_g3.png
888   vprintproxy.exe      C:\Windows\Registration\R00000000000d.clb

Different processes with different files, all with the same ID. This means that when you disconnect the open file you better be careful.

Speaking of disconnecting the files, we can do just that with the /disconnect switch. We can disconnect by ID (ill advised) with the /id switch. We can also disconnect all the files based on the user:

C:\> openfiles /disconnect /a jacobmarley

Or the file name:

C:\> openfiles /disconnect /op "C:\Users\tm\Desktop\wishlist.txt" /a *

Or even the directory:

C:\> openfiles /disconnect /op "C:\Users\tm\Desktop\" /a *

We can even run this against a remote system with the /s SERVERNAME option.

This command is far from perfect, but it is pretty cool.

Sadly, there is no built-in capability in PowerShell to do this same thing. With PowerShell v4 we get Get-SmbOpenFile and Close-SmbOpenFile, but they only work on files opened over the network, not on files opened locally.

Now it is time for Mr. Scrooge Pomeranz to ruin my day by using some really useful, built-in, and ENABLED features of Linux.

It's a Happy Holiday for Hal:

Awww, Tim got me the nicest present of all-- a super-easy Command-Line Kung Fu Episode to write!

This one's easy because Linux comes with lsof, a magical tool surely made by elves at the North Pole. I've talked about lsof in several other Episodes already but so far I've focused more on network and process-related queries than checking objects in the file system.

The simplest usage of lsof is checking which processes are using a single file:

# lsof /var/log/messages
COMMAND    PID USER   FD   TYPE DEVICE SIZE/OFF    NODE NAME
rsyslogd  1250 root    1w   REG    8,3 13779999 3146461 /var/log/messages
abrt-dump 5293 root    4r   REG    8,3 13779999 3146461 /var/log/messages

Here we've got two processes that have /var/log/messages open-- rsyslogd for writing (see the "1w" in the "FD" column, where the "w" means writing), and abrt-dump for reading ("4r", "r" for read-only).

You can use "lsof +d" to see all open files in a given directory:

# lsof +d /var/log
COMMAND    PID USER   FD   TYPE DEVICE SIZE/OFF    NODE NAME
rsyslogd  1250 root    1w   REG    8,3 14324534 3146461 /var/log/messages
rsyslogd  1250 root    2w   REG    8,3   175427 3146036 /var/log/cron
rsyslogd  1250 root    5w   REG    8,3  1644575 3146432 /var/log/maillog
rsyslogd  1250 root    6w   REG    8,3     2663 3146478 /var/log/secure
abrt-dump 5293 root    4r   REG    8,3 14324534 3146461 /var/log/messages

The funny thing about "lsof +d" is that it only shows you open files in the top-level directory, but not in any sub-directories. You have to use "lsof +D" for that:

# lsof +D /var/log
COMMAND     PID   USER   FD   TYPE DEVICE SIZE/OFF    NODE NAME
rsyslogd   1250   root    1w   REG    8,3 14324534 3146461 /var/log/messages
rsyslogd   1250   root    2w   REG    8,3   175427 3146036 /var/log/cron
rsyslogd   1250   root    5w   REG    8,3  1644575 3146432 /var/log/maillog
rsyslogd   1250   root    6w   REG    8,3     2663 3146478 /var/log/secure
httpd      3081 apache    2w   REG    8,3      586 3146430 /var/log/httpd/error_log
httpd      3081 apache   14w   REG    8,3        0 3147331 /var/log/httpd/access_log
...

Unix-like operating systems track open files on a per-partition basis. This leads to an interesting corner-case with lsof: if you run lsof on a partition boundary, you get a list of all open files under that partition:

# lsof /
COMMAND     PID      USER   FD   TYPE DEVICE SIZE/OFF     NODE NAME
init          1      root  cwd    DIR    8,3     4096        2 /
init          1      root  rtd    DIR    8,3     4096        2 /
init          1      root  txt    REG    8,3   150352 12845094 /sbin/init
init          1      root  DEL    REG    8,3           7340061 /lib64/libnss_files-2.12.so
init          1      root  DEL    REG    8,3           7340104 /lib64/libc-2.12.so
...
# lsof / | wc -l
3500

Unlike Windows, Linux doesn't really have a notion of disconnecting processes from individual files. If you want a process to release a file, you kill the process. lsof has a "-t" flag for terse output. In this mode, it only outputs the PIDs of the matching processes. This was designed to allow you to easily substitute the output of lsof as the arguments to the kill command. Here's the little trick I showed back in Episode 22 for forcibly unmounting a file system:

# umount /home
umount: /home: device is busy
# kill $(lsof -t /home)
# umount /home

Here we're exploiting the fact that /home is a partition mount point so lsof will list all processes with files open anywhere in the file system. Kill all those processes and Santa might leave coal in your stocking next year, but you'll be able to unmount the file system!

Episode #179: The Check is in the Mail

Tim mails one in:

Bob Meckle writes in:

I have recently come across a situation where it would be greatly beneficial to build a script to check revocation dates on certificates issued using a certain template, and send an email to our certificate staff letting them know which certificates will expire within the next 6 weeks. I am wondering if you guys have any tricks up your sleeve in regards to this situation?

Thanks for writing in Bob. This is actually quite simple on Windows. One of my favorite features of PowerShell is that dir (an alias for Get-ChildItem) can be used on pretty much any hierarchy, including the certificate store. After we have a list of the certificates we simply filter by the expiration date. Here is how we can find expiring certificates:

PS C:\> Get-ChildItem -Recurse cert: | Where-Object { $_.NotAfter -le (Get-Date).AddDays(42) -and $_.NotAfter -ge (Get-Date) }

We start off by getting a recursive list of the certificates. Then the results are piped into the Where-Object cmdlet to filter for the certificates that expire between today and six weeks (42 days) from now (inclusive). Additional filtering can be added by simply modifying the Where-Object filter, per Bob's original request. We can shorten the command using aliases and shortened parameter names, as well as store the output to be emailed.

PS C:\> $tomail = ls -r cert: | ? { $_.NotAfter -le (Get-Date).AddDays(42) -and $_.NotAfter -ge (Get-Date) }

Emailing is quite simple too. PowerShell version 4 adds a new cmdlet, Send-MailMessage. We can send the certificate information like this:

PS C:\> Send-MailMessage -To me@blah.com -From cert@blah.com -Subject "Expiring Certs" -Body $tomail

Pretty darn simple if you ask me. Hal, what do you have up your sleeve?

Hal relies on his network:

Tim, you're getting soft with all these cool PowerShell features. Why when I was your age... Oh, nevermind! Kids these days! Hmph!

If I'm feeling curmudgeonly, it's because this is far from a cake-walk in Linux. Obviously, I can't check a Windows certificate store remotely from a Linux machine. So I thought I'd focus on checking remote web site certificates (which could be Windows, Linux, or anything else) from my Linux command line.

The trick for checking a certificate is fairly well documented on the web:

$ echo | openssl s_client -connect www.google.com:443 2>/dev/null | 
     openssl x509 -noout -dates
notBefore=Jun  4 08:58:29 2014 GMT
notAfter=Sep  2 00:00:00 2014 GMT

We use the OpenSSL built-in "s_client" to connect to the target web server and dump the certificate information. The "2>/dev/null" drops some extraneous standard error logging. The leading "echo" piped into the standard input makes sure that we close the connection right after receiving the certificate info. Otherwise our command line will hang and never return to the shell prompt. We then use OpenSSL again to output the information we want from the downloaded certificate. Here we're just requesting the "-dates"-- "-noout" stops the openssl command from displaying the certificate itself.

However, there is much more information you can parse out. Here's a useful report that displays the certificate issuer, certificate name, and fingerprint in addition to the dates:

$ echo | openssl s_client -connect www.google.com:443 2>/dev/null | 
     openssl x509 -noout -issuer -subject -fingerprint -dates
issuer= /C=US/O=Google Inc/CN=Google Internet Authority G2
subject= /C=US/ST=California/L=Mountain View/O=Google Inc/CN=www.google.com
SHA1 Fingerprint=F4:E1:5F:EB:F1:9F:37:1A:29:DA:3A:74:BF:05:C5:AD:0A:FB:B1:95
notBefore=Jun  4 08:58:29 2014 GMT
notAfter=Sep  2 00:00:00 2014 GMT

If you want a complete dump, just use "... | openssl x509 -text".

But let's see if we can answer Bob's question, at least for a single web server. Of course, that's going to involve some date arithmetic, and the shell is clumsy at that. First I need to pull off just the "notAfter" date from my output:

$ echo | openssl s_client -connect www.google.com:443 2>/dev/null | 
     openssl x509 -noout -dates | tail -1 | cut -f2 -d=
Sep  2 00:00:00 2014 GMT

You can convert a time stamp string like this into a "Unix epoch" date with the GNU date command:

$ date +%s -d 'Sep  2 00:00:00 2014 GMT'
1409616000

To do it all in one command, I just use "$(...) for command output substitution:

$ date +%s -d "$(echo | openssl s_client -connect www.google.com:443 2>/dev/null | 
                     openssl x509 -noout -dates | tail -1 | cut -f2 -d=)"
1409616000

I can calculate the difference between this epoch date and the current epoch date ("date +%s") with some shell arithmetic and more command output substitution:

$ echo $(( $(date +%s -d "$(echo | openssl s_client -connect www.google.com:443 2>/dev/null | 
                                openssl x509 -noout -dates | tail -1 | cut -f2 -d=)") 
            - $(date +%s) ))
5452606

Phew! This is already getting pretty long-winded, but now I can check my result against Bob's 6 week threshold (that's 42 days or 3628800 seconds):

$ [[ $(( $(date +%s -d "$(echo | openssl s_client -connect www.google.com:443 2>/dev/null | 
                              openssl x509 -noout -dates | tail -1 | cut -f2 -d=)") - $(date +%s) )) 
    -gt 3628800 ]] && echo GOOD || echo EXPIRING
GOOD

Clearly this is straying all too close to the borders of Scriptistan, but if you wanted to check several web sites, you could add an outer loop:

$ for d in google.com sans.org facebook.com twitter.com; do 
    echo -ne $d\\t; 
    [[ $(( $(date +%s -d "$(echo | openssl s_client -connect www.$d:443 2>/dev/null | 
                            openssl x509 -noout -dates | tail -1 | cut -f2 -d=)") - $(date +%s) ))
       -gt 3628800 ]] && echo GOOD || echo EXPIRING; 
done
google.com	GOOD
sans.org	GOOD
facebook.com	GOOD
twitter.com	GOOD

See? It's all good, Bob!

Episode #178: Luhn-acy

Hal limbers up in the dojo

To maintain our fighting trim here in the Command Line Kung Fu dojo, we like to set little challenges for ourselves from time to time. Of course, we prefer it when our loyal readers send us ideas, so keep those emails coming! Really... please oh please oh please keep those emails coming... please, please, please... ahem, but I digress.

All of the data breaches in the news over the last year got me thinking about credit card numbers. As many of you are probably aware, credit card numbers have a check digit at the end to help validate the account number. The Luhn algorithm for computing this digit is moderately complicated and I wondered how much shell code it would take to compute these digits.

The Luhn algorithm is a right-to-left calculation, so it seemed like my first task was to peel off the last digit and be able to iterate across the remaining digits in reverse order:

$ for d in $(echo 123456789 | rev | cut -c2- | sed 's/\(.\)/\1 /g'); do echo $d; done
8
7
6
5
4
3
2
1

The "rev" utility flips the order of our digits, and then we just grab everything from the second digit onwards with "cut". We use a little "sed" action to break the digits up into a list we can iterate over.

Then I started thinking about how to do the "doubling" calculation on every other digit. I could have set up a shell function to calculate the doubling each time, but with only 10 possible outcomes, it seemed easier to just create an array with the appropriate values:

$ doubles=(0 2 4 6 8 1 3 5 7 9)
$ for d in $(echo 123456789 | rev | cut -c2- | sed 's/\(.\)/\1 /g'); do echo $d ${doubles[$d]}; done
8 7
7 5
6 3
5 1
4 8
3 6
2 4
1 2

Then I needed to output the "doubled" digit only every other digit, starting with the first from the right. That means a little modular arithmetic:

$ c=0
$ for d in $(echo 123456789 | rev | cut -c2- | sed 's/\(.\)/\1 /g'); do 
    echo $(( ++c % 2 ? ${doubles[$d]} : $d )); 
done
7
7
3
5
8
3
4
1

I've introduced a counting variable, "$c". Inside the loop, I'm evaluating a conditional expression to decide if I need to output the "double" of the digit or just the digit itself. There are several ways I could have handled this conditional operation in the shell, but having it in the mathematical "$((...))" construct is particularly useful when I want to calculate the total:

$ c=0; t=0; 
$ for d in $(echo 123456789 | rev | cut -c2- | sed 's/\(.\)/\1 /g'); do 
    t=$(( $t + (++c % 2 ? ${doubles[$d]} : $d) )); 
done
$ echo $t
38

We're basically done at this point. Instead of outputting the total, "$t", I need to do one more calculation to produce the Luhn digit:

$ c=0; t=0; 
$ for d in $(echo 123456789 | rev | cut -c2- | sed 's/\(.\)/\1 /g'); do 
    t=$(( $t + (++c % 2 ? ${doubles[$d]} : $d) ));
$ echo $(( ($t * 9) % 10 ))
2

Here's the whole thing in one line of shell code, including the array definition:

doubles=(0 2 4 6 8 1 3 5 7 9); 
c=0; t=0; 
for d in $(echo 123456789 | rev | cut -c2- | sed 's/\(.\)/\1 /g'); do 
    t=$(( $t + (++c % 2 ? ${doubles[$d]} : $d) )); 
done; 
echo $(( ($t * 9) % 10 ))

Even with all the extra whitespace, the whole thing fits in under 100 characters! Grand Master Ed would be proud.

I'm not even going to ask Tim to try and do this in CMD.EXE. Grand Master Ed could have handled it, but we'll let Tim use his PowerShell training wheels. I'm just wondering if he can do it so it still fits inside a Tweet...

Tim checks Hal's math

I'm not quite sure how Hal counts, but I when I copy/paste and then use Hal's own wc command I get 195 characters. It is less than *2*00 characters, not long enough to tweet.

Here is how we can accomplish the same task in PowerShell. I'm going to use a slightly different method than Hal. First, I'm going to use his lookup method as it is more terse then doing the extra match via if/then. In addition, I am going to extend his method a little to save a little space.

PS C:\> $lookup = @((0..9),(0,2,4,6,8,1,3,5,7,9));

This mutli-dimensional array contains a lookup for the number as well as the doubled number. That way I can index the value without an if statement to save space. Here is an example:

PS C:\> $isdoubled = $false
PS C:\> $lookup[$isdoubled][6]
6
PS C:\> $isdoubled = $true
PS C:\> $lookup[$isdoubled][7]
15

The shortest way to get each digit, from right to left, is by using regex (regular expression) match and working right to left. A longer way would be to use the string, convert it to a char array, then reverse it but that is long, ugly, and we need to use an additional variable.

The results are fed into a ForEach-Object loop. Before the objects (the digits) passed down the pipeline are handled we need to initialize a few variables, the total and the boolean $isdoubled variables in -Begin. Next, we add the digits up by accessing the items in our array as well as toggling the $isdoubled variable. Finally, we use the ForEach-Object's -End to output the final value of $sum.

PS C:\> ([regex]::Matches('123456789','.','RightToLeft')) | ForEach-Object 
-Begin { $sum = 0; $isdoubled = $false} -Process { $sum += $l[$isdoubled][[int]$_.value];  $d = -not $d } 
-End { $sum }

We can shorten the command to this to save space.

PS C:\> $l=@((0..9),(0,2,4,6,8,1,3,5,7,9));
([regex]::Matches('123456789','.','RightToLeft')) | %{ $s+=$l[$d][$_.value];$d=!$d} -b{$s=0;$d=0} -en{$s}

According to my math this is exactly 140 characters. I could trim another 2 by removing a few spaces too. It's tweetable!

I'll even throw in a bonus version for cmd.exe:

C:\> powershell -command "$l=@((0..9),(0,2,4,6,8,1,3,5,7,9));
([regex]::Matches("123456789",'.','RightToLeft')) | %{ $s+=$l[$d][$_.value];$d=!$d} -b{$s=0;$d=0} -en{$s}"

Ok, it is a bit of cheating, but it does run from CMD.

Hal gets a little help

I'm honestly not sure where my brain was at when I was counting characters in my solution. Shortening variable names and removing all extraneous whitespace, I can get my solution down to about 150 characters, but no smaller.

Happily, Tom Bonds wrote in with this cute little blob of awk which accomplishes the mission:

awk 'BEGIN{split("0246813579",d,""); for (i=split("12345678",a,"");i>0;i--) {t += ++x % 2 ? d[a[i]+1] : a[i];} print (t * 9) % 10}'

Here it is with a little more whitespace:

awk 'BEGIN{ split("0246813579",d,""); 
            for (i=split("12345678",a,"");i>0;i--) {
                t += ++x % 2 ? d[a[i]+1] : a[i];
            } 
            print (t * 9) % 10
          }'

Tom's getting a lot of leverage out of the "split()" operator and using his "for" loop to count backwards down the string. awk is automatically initializing his $t and $x variables to zero each time his program runs, whereas in the shell I have to explicitly set them to zero or the values from the last run will be used.

Anyway, Tom's original version definitely fits in a tweet! Good show, Tom!

Episode #177: There and Back Again

Hal finds some old mail

Way, way back after Episode #170 Tony Reusser sent us a follow-up query. If you recall, Episode #170 showed how to change files named "fileaa", "fileab", "fileac", etc to files named "file.001", "file.002", "file.003". Tony's question was how to go back the other way-- from "file.001" to "fileaa", "file.002" to "fileab", and so on.

Why would we want to do this? Heck, I don't know! Ask Tony. Maybe he just wants to torture us to build character. Well we here at Command Line Kung Fu fear no characters, though we may occasionally lose reader emails behind the refrigerator for several months.

The solution is a little scripty, but I did actually type it in on the command line:

    c=1
    for l1 in {a..z}; do 
        for l2 in {a..z}; do 
            printf -v ext %03d $(( c++ ))
            [[ -f file.$ext ]] && mv file.$ext file$l1$l2 || break 2
        done
    done

There are two nested loops here which work together to create our alphabetic file extensions. $l1 represents the first of the two letters, ranging from 'a' to 'z'. $l2 is the second letter, also ranging from 'a' to 'z'. Put them next to each other and you get "aa", "ab", "ac", etc.

Like Episode #170, I'm using a counter variable named $c to track the numeric file extension. So, for all you computer science nerds, this is a rather weird looping construct because I'm using three different loop control variables. And weird is how I roll.

Inside the loop, I re-use the code from Episode #170 to format $c as our three-digit file extension (saved in variable $ext) and auto-increment $c in the same expression. Then I check to see if "file.$ext" exists. If we have a "file.$ext", then we rename it to "file$l1$l2" ("fileaa", "fileab", etc). If "file.$ext" does not exist, then we've run out of "file.xxx" pieces and we can stop looping. "break 2" breaks out of both enclosing loops and terminates our command line.

And there you go. I hope Tim has as much fun as I did with this. I bet he'd have even more fun if I made him do it in CMD.EXE. Frankly all that PowerShell has made him a bit sloppy...

Tim mails this one in just in time from Abu Dhabi

Wow, CMD huh? That trip to Scriptistan must have made him crazy. I think a CMD version of this would violate the laws of physics.

I'm on the other side of the world looking for Nermal in Abu Dhabi. Technically, it is May 1 here, but since the publication date shows April I think this counts as our April post. At least, that is the story I'm sticking to.

Sadly, I too will have to enter Scriptistan. I have a visa for a long stay on this one. I'll start by using a function to convert a number to Base26. The basis of this function is taken from here. I modified the function so we could add leading A's to adjust the width.

function Convert-ToLetters ([parameter(Mandatory=$true,ValueFromPipeline=$true)][int] $Value, [int]$MinWidth=0) {
    $currVal = $Value
    if ($LeadingDigits -gt 0) { $currVal = $currVal + [int][Math]::Pow(26, $LeadingDigits) }
    $returnVal = '';
    while ($currVal -ge 26) {
        $returnVal = [char](($currVal) % 26 + 97) + $returnVal;
        $currVal =  [int][math]::Floor($currVal / 26)
    }
    $returnVal = [char](($currVal) + 64) + $returnVal;
     
    return $returnVal
}

This allows me to cheat greatly simplify the renaming process.

PS C:\> ls file.* | % { move $_ "file.$(Convert-ToLetters [int]$_.Extension.Substring(1) -MinWidth 3 )" }

This command will read the files starting with "file.", translate the extension in to Base26 (letters), and rename the file. The minimum width is configurable as well, so file.001 could be file.a, file.aa, file.aaa, etc. Also, this version will support more than 26^2 files.

Episode #176: Step Up to the WMIC

Tim grabs the mic:

Michael Behan writes in:

Perhaps you guys can make this one better. Haven’t put a ton of thought into it:

C:\> (echo HTTP/1.0 200 OK & wmic process list full /format:htable) | nc -l -p 3000

Then visit http://127.0.0.1:3000

This could of course be used to generate a lot more HTML reports via wmic that are quick to save from the browser. The downside is that in its current state is that the page can only be visited once. Adding something like /every:5 just pollutes the web page with mostly duplicate output.

Assuming you already have netcat (nc.exe) on the system the command above will work fine, but it will only work once. After the browser recieves the data the connection has been used and the command is done. To do this multiple times you must wrap it in an infinite For loop.

C:\> for /L %i in (1, 0, 2) do (echo HTTP/1.0 200 OK & wmic process list full /format:htable) | nc -l -p 3000

This will count from 1 to 2 and count by 0, which will never happen (except for very large values of 0). We could use the wmic command to request this information from the remote machine and view it in our browser. This method will authenticate to the remote machine instead of allowing anyone to access the information.

C:\> wmic /node:joelaptop process list full /format:htable > joelaptopprocesses.html && start joelaptopprocesses.html

This will use your current credentials to authenticate to the remote machine, request the remote process in html format, save it to a file, and finally open the file in your default viewer (likely your browser). If you need to use separate credentials you can specify /user:myusername and /password:myP@assw0rd.

Hal, your turn, and I want to see this in nice HTML format. :)

Hal throws up some jazz hands:

Wow. Tim seems a little grumpy. Maybe it's because he can make a simple web server on the command line but has no way to actually request data from it via the command line. Don't worry Little Tim, maybe someday...

Heck, maybe Tim's grumpy because of the dumb way he has to code infinite loops in CMD.EXE. This is a lot easier:

$ while :; do ps -ef | nc -l 3000; done

Frankly, most browsers will interpret this as "text/plain" by default and display the output correctly.

But the above loop got me thinking that we could actually stack multiple commands in sequence:

while :; do
    ps -ef | nc -l 3000
    netstat -anp | nc -l 3000
    df -h | nc -l 3000
    ...
done

Each connection will return the output of a different command until you eventually exhaust the list and start all over again with the first command.

OK, now let's deal with grumpy Tim's request for "nice HTML format". Nothing could be easier, my friends:

$ while :; do (echo '<pre>'; ps -ef; echo '</pre>') | nc -l 3000; done

Hey, it's accepted by every major browser I tested it with! And that's the way we do it downtown... (Hal drops the mic)

Episode #175: More Time! We Need More Time!

Tim leaps in

Every four years (or so) we get an extra day in February, leap year. When I was a kid this term confused me. Frogs leap, they leap over things. A leap year should be shorter! Obviously, I was wrong.

This extra day can give us extra time to complete tasks (e.g. write blog post), so we are going to use our shells to check if the current year is a leap year.

PS C:\> [DateTime]::IsLeapYear(2014)
False

Sadly, this year we do not have extra time. Let's confirm that this command does indeed work by checking a few other years.

PS C:\> [DateTime]::IsLeapYear(2012)
True
PS C:\> [DateTime]::IsLeapYear(2000)
True
PS C:\> [DateTime]::IsLeapYear(1900)
False

Wait a second! Something is wrong. The year 1900 is a multiple of 4, why is it not a leap year?

The sun does not take exactly 365.25 days to get around the sun, it is actually 365.242199 days. This means that if we always leaped every four years we would slowly get off course. So every 100 years we skip the leap year.

Now you are probably wondering why 2000 had a leap year. That is because it is actually the exception to the exception. Every 400 years we skip skipping the leap year. What a cool bit of trivia, huh?

Hal, how jump is your shell?

Hal jumps back

I should have insisted Tim do this one in CMD.EXE. Isn't is nice that PowerShell has a IsLeapYear() built-in? Back in my day, we didn't even have zeroes! We had to bend two ones together to make zeroes! Up hill! Both ways! In the snow!

Enough reminiscing. Let's make our own IsLeapYear function in the shell:

function IsLeapYear {
    year=${1:-$(date +%Y)};
    [[ $(($year % 400)) -eq 0 || ( $(($year % 4)) -eq 0 && $(($year % 100)) -ne 0 ) ]]
}

There's some fun stuff in this function. First we check to see if the function is called with an argument ("${1-..."). If so, then that's the year we'll check. Otherwise we check the current year, which is the value returned by "$(date +%Y)".

The other line of the function is the standard algorithm for figuring leap years. It's a leap year if the year is evenly divisible by 400, or divisible by 4 and not divisible by 100. Since shell functions return the value of the last command or expression executed, our function returns whether or not it's a leap year. Nice and easy, huh?

Now we can run some tests using our IsLeapYear function, just like Tim did:

$ IsLeapYear && echo Leaping lizards! || echo Arf, no
Arf, no
$ IsLeapYear 2012 && echo Leaping lizards! || echo Arf, no
Leaping lizards!
$ IsLeapYear 2000 && echo Leaping lizards! || echo Arf, no
Leaping lizards!
$ IsLeapYear 1900 && echo Leaping lizards! || echo Arf, no
Arf, no

Assuming the current year is not a Leap Year, we could even wrap a loop around IsLeapYear to figure out the next leap year:

$ y=$(date +%Y); while :; do IsLeapYear $((++y)) && break; done; echo $y
2016

We begin by initializing $y to the current year. Then we go into an infinte loop ("while :; do..."). Inside the loop we add one to $y and call IsLeapYear. If IsLeapYear returns true, then we "break" out of the loop. When the loop is all done, simply echo the last value of $y.

Stick that in your PowerShell pipe and smoke it, Tim!