Tuesday, May 10, 2011

Episode #146: Hard CIDR

Tim is the mailman:

Most of our episode ideas come from the mailbag, and we can never have too many. Usually, we usually have too few, so keep sending in ideas. This episode was inspired by the modern art masterpiece sent in by Matt Graeber. His "piece" reads a file, expands a list of CIDR formatted IP addresses, and does a reverse DNS lookup against each IP address.

C:\> cmd.exe /v:on /c "FOR /F "tokens=1-5 delims=./" %G IN (ips.txt) DO
@(set network=%G.%H.%I.& set /a fourth="%J+1" 1^>nul & set mask=%K&
set /a end="(2 ^<^< (32-!mask!-1))-2+!fourth!" 1>nul &
echo !network!!fourth!-!end!& echo ****************** &
FOR /L %O IN (!fourth!,1,!end!) DO @(for /F "tokens=2" %P in
('nslookup !network!%O server 2^>nul ^| find "Name:"') do
@echo !network!%O %P & ping -n 3 127.0.0.1>nul))" 2>nul


That is some serious cmd madness. Since it is such beautiful art, we will leave it to the reader to behold and interpret for himself, but we will go over a more simplified version.

It should be noted, that all the commands presented today only work with 24+ bit masks. Using built in tools to expand the CIDR format for a mask less than 24 is significantly more work because we have to manipulate the third (and possibly second or first) octet(s) as well. We'll use the file below as our sample input file:

192.168.1.32/27
10.10.10.128/26
172.16.0.0/24


We'll simplify Matt's command so it only expands and outputs the IP addresses. From there, you can replace the echo command with the command of your choice. This CIDR expansion is very handy for wraping commands that only take a single IP as input.

The simplified version of the CIDR expander is:

C:\> cmd.exe /v:on /c "FOR /F "tokens=1-5 delims=./" %G IN (ips.txt) DO
@(set network=%G.%H.%I.& set /a fourth="%J" 1^>nul & set mask=%K&
set /a end="(2 ^<^< (32-!mask!-1))-1+!fourth!" 1>nul &
FOR /L %O IN (!fourth!,1,!end!) DO @echo !network!%O)" 2>nul


192.168.1.32
192.168.1.33
...
192.168.1.62
192.168.1.63
10.10.10.128
10.10.10.129
...
10.10.10.190
10.10.10.191
172.16.0.0
172.16.0.1
...
172.16.0.254
172.16.0.255


The command begins by enabling delayed variable expansion (explained at the end of Episode #46). The For Loop then reads our file and splits the text on each period and forward slash, so we have variables that hold each octet (%G-%J) as well as the number of bits in the subnet mask (%K). Next, a bit of cmd math is done to calculate the last IP address in the range. Finally, another For loop is used to output all of the IP addresses. Of course, we could do something else with the IP addresses instead of just displaying them, like ping or nslookup.

The math used to calculate the upper bound of our network uses a Logical Shift Left (<<). This operator shifts each bit to the left one location. So 2 (00000010) left shifted by 1 is 4 (00000100) and 2 left shifted twice is 8 (000010000). This operator is used to replace the nonexistent 2^x command.

PowerShell:

The PowerShell command to expand the addresses ranges is:

PS C:\> gc ips.txt | % {
$a=$_.split('./');
$a[3]..($a[3]-1+[Math]::Pow(2,32-$a[4])) | % {
echo ($a[0],$a[1],$a[2],$_ -join ".")
}
}


We start by reading our file using the Get-Content cmdlet (aliases gc and cat). Each line is piped into a ForEach-Object loop to be parsed and manipulated.

Inside our loop we split each string into an array of substrings, using period and forward slash as delimiters. We then end up with $a[0] through $a[3] holding the first through fourth octets and $a[5] is the mask.

The range operator (..) is use to generate a list of numbers between our lower bound (last octet) and the upper bound (end of network range). Again, we use a little math to determine the size of the range. In short, the size of the range is 2^(32-masklen). There isn't a native exponent operator, so we use the built in .NET Math class to do our power function.

The output of the range is fed into another ForEach-Object cmdlet. Inside this cmdlet's scriptblock is where the IP address is reassembled and output.

Now that we have the output, we can easily feed it to another command. For example, say we wanted to ping each of the IP addresses:

 PS C:\> ... | % { ping $_ }


No off to go find some hard cider to make this pain go away...

Hal is the eggman

Tim and I usually work up our pieces of the blog independently and then jam them together using a few insults as connective tissue. It's interesting that in this case my thought process developed along a parallel course with Tim's transition from CMD.EXE to Powershell.

My initial attempt looked a lot like Matt and Tim's CMD.EXE solutions:

while read line; do 
exp=${line/*\//};
base=${line/\/*/};
lastoct=${base/*./};
net=${base/%.$lastoct/};
for ((i=0; $i < $(( 2 ** (32-$exp) )); i++)); do
oct=$(($lastoct + $i));
echo $net.$oct;
done;
done <input

The outer while loop reads the input file line-by-line. Then I exploit the bash variable substitution operator ("${var/pattern/sub}") to pull the address and exponent apart, and to strip the last octet from the address. After that it's just a for loop to create the list of addresses. That little "echo" statement in the middle of the inner loop is the whole purpose of the exercise-- the rest is just grisly parsing and setup.

So I wasn't really happy with my solution. But then it occurred to me that I don't have to do the parsing. The shell will do it for me!

IFS=./
while read o1 o2 o3 o4 exp; do
for ((i=o4; $i < $(( $o4 + 2 ** (32-$exp) )); i++)); do
echo $o1.$o2.$o3.$i;
done;
done <input

The trick is to set IFS so that the shell automatically tokenizes each line as we "read" it in the while loop (essentially we're accomplishing the same thing that Tim does with his Powershell split() operator). If IFS is set to "./" then each octet and the exponent is broken out into a different variable and I don't have to do any parsing myself. This is a very useful technique when you're dealing with any "strongly delimited" input-- like /etc/passwd, /etc/shadow, and so on.

You'll also notice that I cleaned up my loop control a little bit so that I save myself a "$((...))" operation inside the inner loop. Rather than having the loop control variable run from 0..2**(32-exp) and then adding the value to the last octet inside the loop, it's easier to start the loop with the last octet value and increment from there.

By the way, some of you may be wondering why I didn't use the built-in range operator in bash and make my for loop be something like:

for $i in {$o4..$(($o4 + 2**(32-$exp) - 1))} ...

I actually tried this approach first. But it turns out that bash only lets you use literal values-- integers or characters-- in a "{a..z}" operation. This actually strikes me as something of a bug.

In any event, that's why I ended up opting for the C-style for loop instead. I could have used something like "$( seq $o4 $(($o4 + 2**(32-$exp) - 1)) )", but why call an external program when you don't have to?

So goo goo g'joob and have a great week everybody!

Tuesday, May 3, 2011

Episode #145: A Date to Copy

Tim checks the mail (again):

One of our readers writes in asking how to generate "a list of locations of doc files which are created or modified after month and year 01 /2011." He then wants to copy all those files to another directory. He specifically stated he wanted it to work on XP and without PowerShell.

Well, let me get on my soap box for a second. PowerShell v1 and v2 are available for XP. I can't recommend enough that people install PowerShell. It will make your Windows administration and command line experience easier and much less painful. Case in point, it took me a decent amount of time to write the cmd.exe portion of this episode, which includes all the trial and error. However, the PowerShell portion is so straight forward I wrote that portion off the top of my head. So if you are on a Windows box without PowerShell, go install it before continuing. Ok, enough ranting, back to our regularly scheduled programming...

One nice thing is that we are only looking for files based on year, so that makes our time parsing much easier in cmd.exe. If we had to parse based on month and year it would be exponentially harder with this antiquated shell.

We can use a number of the techniques taken from Episode #29. Specifically, the For loop with the ~f (full file path). We can use ~t with our variable to get the modified timestamp, but there isn't a modifier to give us the creation time, so we'll have to take a slightly different approach.

First, we start off with a For loop that will find all the .doc files on the system.

C:\> for /r %i in (*.doc) do @echo %~fi

C:\Doc\file1.doc
C:\Doc\file2.doc
...


We have all the .doc files, so now we take a giant leap to our final command:

C:\> for /r %i in (*.doc) do (@dir /tw %~fi | find "/2011" > NUL ||
@dir /tc %~fi | find "/2011" > NUL) && copy %~fi C:\DocsFrom2011


We use two Dir commands with different time (/t) options. The first, /tw, displays the Last Write Time. The second, /tc, displays the Creation Time. Each of these Dir commands is piped into a Find command that looks for the year portion of the date "/2011". All that is wrapped in a bit of magic logic that is used to determine if the file should be copied or not. The magic explained...

We use some Logical ORs (||) and a Logical And (&&) to do the magic. The cmd.exe shell uses short circuit logical operators, but what does that mean? In short, if we wanted to evaluate A && B the second expression (B) is only evaluated when the result is not fully determined by A. For example, given A && B, when A is false there is no reason to evaluate B since the Logical And will then be false regardless of the value of B. Enough of the Math/Logic lesson, back to our command...

The logic in our command reduces to: (Last Write Time Matches OR CreationDate Matches) AND copy. Given our short circuit logical operators, if neither the Creation or Last Write Times match, we don't do the copy. But if either matches, copy.

Now on to the much easier, PowerShell.

PowerShell

Not only is the PowerShell version is much easier, and much more robust, as it can actually compare dates. That means we could just as easily look for files created after any arbitrary date, not just January 1.

PS C:\> Get-ChildItem -Recurse -Include *.doc | Where-Object {
$_.LastWriteTime -ge "1/1/2011" -or $_.CreationTime -ge "1/1/2011" } |
Copy-Item -Destination C:\DocsFrom2011


The command can be shortened using built in aliases and shortened parameter names:

PS C:\> ls -r -i *.doc | ? { $_.LastWriteTime -ge "1/1/2011" -or $_.CreationTime -ge "1/1/2011" } |
cp -d C:\DocsFrom2011


This command does a recursive directory listing and pipes the results into a Where-Object filter. The filter only passes objects where the relevant timestamps are greater than, or equal to, Jan 1, 2011. All the objects that make it through the filter are sent to Copy-Item to be copied to our target folder.

The PowerShell version is included, even though this portion wasn't requested by our reader. I guess that makes me only half wanted. Since Hal's shell isn't supported on XP I guess he isn't wanted either, but here he is anyway.

Hal checks in

Not wanted? Don't go projecting your insecurities onto me now. Just repeat this daily affirmation, "CMD.EXE is good enough, and gosh darn it people just love XP!" Besides, you can always install Cygwin to help you over the rough spots.

This one really is similar to Episode #29-- we just have to do something with the file names once we pick them out. For that I think I'll bust out the cpio magic as I did back in Episode #115:

# touch -t 201101010000 /tmp/marker
# cd /some/source/dir
# find . -depth \( -name \*.doc -o -name \*.docx \) -newer /tmp/marker |
cpio -pd /path/to/target/dir

5854 blocks

First I use touch to create a file with the earliest date for files which we're interested in. I use this with "find ... -newer" to find all files that have been modified since this timestamp. Unfortunately traditional Unix file systems still don't track creation time on files (unless you've upgraded to EXT4), so last modified time is all we have to go on.

Our loyal reader specified that they were only interested in doc files, but I decided to make my life harder by looking for both the old style *.doc files and the newer *.docx format. Notice that the Windows guy in our little CommandLineKungFu partnership didn't think to look for both file extensions.

Notice that I'm also using the "-depth" option with find to work around any possible issues with read-only directory permissions when cpio is creating the parallel directory structure in the target directory. See the discussion in Episode #115 for more details.

So there you go, a solution to a new problem created by putting two earlier solutions together. And that's pretty much the Unix command-line religion anyway. And that's why Unix is cool. And, gosh darn it, people like it!

Tuesday, April 26, 2011

Episode #144: What the Hex?

Tim has a sugar high

Recently, I've been going through some exploit development tutorials. One of the exploits was for a file format vulnerability. I created my malicious file and saved it on my Windows machine, and I wanted to take a quick peek at the file and confirm that I had written the correct information. I wanted to take a look at the bytes, but not all the bytes are printable. Of course there are a million ways to view this file, but I decided to try PowerShell since I didn't have any other tools installed in this clean install of Windows.

PS C:\> get-content -encoding byte -totalcount 1000 shellcode.bin
80
83
81
82
86
87
85
156
...


This uses the good ol' Get-Content cmdlet. It reads raw bytes by setting the Encoding parameter to Byte. I used the TotalCount parameter since I only want to see the first 1000 bytes.

Of course, the output isn't nice. First, the output is in decimal form instead of hex. We can quickly fix that:

PS C:\> get-content -encoding byte -totalcount 1000 shellcode.bin | % { "{0:X2} " -f $_ }
50
53
51
52
56
57
55
9C
...


We use the ForEach-Object cmdlet (alias %) to format each byte as hex. Now to fix the second output option, the vertical display.

PS C:\> (get-content -encoding byte -totalcount 1000 shellcode.bin | % { "{0:X2}" -f $_ }) -join " "
50 53 51 52 56 57 55 9C ...


Simply wrapping the entire command in paranthesis and using the Join operator effectively rotates our output.

If you want a different format you can modify the format string. Say we wanted it with a prepended '\x'.

PS C:\> (get-content -encoding byte -totalcount 1000 shellcode.bin | % { "\x{0:X2}" -f $_ }) -join " "
\x50 \x53 \x51 \x52 \x56 \x57 \x55 \x9C ...


Not to bad huh? Of course, this is super simple for Hal. <Sigh>

Hal is still Bali Hai

Thanks for taking it easy on me as I'm coming down off my vacation, Tim. How's that competition snow shoveling working out for you?

As Tim points out, this one is easy for Unix because (being a programmers' operating system) we have built-in hex dumpers. The old standby is the "od", or "octal dump" command. Despite the name, od can output in a variety of formats, including hex:

$ od -N 48 -x /bin/bash
0000000 457f 464c 0102 0001 0000 0000 0000 0000
0000020 0002 003e 0001 0000 f210 0041 0000 0000
0000040 0040 0000 0000 0000 3a80 000e 0000 0000
0000060

Here I'm using the "-N" option to only dump out the first 48 bytes of the file, and "-x" specifies that I want the output in hex format. If I wanted the output grouped as single bytes rather than pairs, I could use a slightly different option:

$ od -N 48 -t x1 /bin/bash
0000000 7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00
0000020 02 00 3e 00 01 00 00 00 10 f2 41 00 00 00 00 00
0000040 40 00 00 00 00 00 00 00 80 3a 0e 00 00 00 00 00
0000060

The "-t" option lets you specify the output format more completely. The first letter denotes the type of output you want: "x" for hex, "o" for octal, "a" and "c" for character types, "f" for floating point, and so on. The number following the letter lets you choose the size groups you want in the output.

But since I generally only ever want hex dumps, I usually use xxd instead of od:

$ xxd -l 48 -g 1 /bin/bash
0000000: 7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00 .ELF............
0000010: 02 00 3e 00 01 00 00 00 10 f2 41 00 00 00 00 00 ..>.......A.....
0000020: 40 00 00 00 00 00 00 00 80 3a 0e 00 00 00 00 00 @........:......

xxd is a pure hex dumper (although it can dump bits if you use "-b") so we don't need a special option to output hexadecimal. Use "-l" to specify how many bytes you want to read, and "-g" to specify the number of bytes for grouping the output. Like od, the default output grouping is 2 bytes, but I specified 1 byte output in the command above.

If you don't want all the extra frippery that the normal xxd output gives you-- like byte offsets and ASCII translation-- you can use the "-p" (plain) option to just get the raw bytes:

$ xxd -p -l 48 /bin/bash
7f454c4602010100000000000000000002003e000100000010f241000000
00004000000000000000803a0e0000000000

This format is particularly useful when you want to pipe the output into something else. For example, I could now use sed to produce the "\x" notation similar to Tim's last example:

$ xxd -p -l 48 /bin/bash | sed 's/\(..\)/\\x\1 /g'
\x7f \x45 \x4c \x46 \x02 \x01 \x01 \x00 \x00 \x00 \x00 \x00 \x00 \x00 \x00 \x00 \x02 ...
\x00 \x00 \x40 \x00 \x00 \x00 \x00 \x00 \x00 \x00 \x80 \x3a \x0e \x00 \x00 \x00 \x00 \x00

Now if I could only figure out a way to use the shell to dump myself as quickly from one side of the Pacific to the other, that would be quite a feat! Oh well, perhaps the long flight will inspire me with some new shell fu for next week.

Tuesday, April 19, 2011

Episode #143: Unicode to Shellcode

Hal checks in from Bali

Just a quick one this week. After finishing up with my posse of hard-rocking malware-crazed weasels from SANS Bali, I'm due for a little R&R and, hey, did I mention I'm in Bali? But sun, sand, surf, and sky will not prevent me from providing our faithful readers with their weekly dose of Command Line Kung Fu. Besides, teaching Lenny Zeltser's excellent Reverse Engineering Malware class always gives me ideas for new Fu.

This week's challenge is a fairly common one for malware analysts. Many times you have Javascript code from web pages, Microsoft Office documents, or PDF files that contain shellcode payloads in various different obfuscated forms. One encoding type is two-byte, little-endian unicode represented by strings such as %u8806. Today's sample goes that one better though, by introducing extra whitespace and punctuation like so:

'%u5350%u5', '251%u5756%', 'u9c55%u00', 'e8%u0', '000%u5d00', ...

Lenny came up with some tasty Perl code to extract the necessary bytes and reformat them into hex notation such as "\x06\x88" (and from there they can be converted to an executable to analyze with a tool such as shellcode2exe.py). While I'm a big fan of Perl, it's against the rules of our little blog experiment. So I started wondering what the sed solution would look like.

Wonder no more:

$ sed 's/[^0-9A-Fa-f]//g; 
s/\(..\)\(..\)/\\x\2\\x\1/g' shellcode.uni >shellcode.hex

The first substitution gets rid of anything that isn't a hex digit. Not only does this clean up the whitespace, quotes, and commas, it even eliminates all those "%u"s for us. Next we need to grab each byte-- represented by two hex digits-- and put a "\x" in front of it. The tricky part is that the original unicode was in little-endian format, so we must swap each pair of bytes as we work through the string of characters. That's why you see me grabbing two bytes at a time and outputting them with "...\2...\1..." on the right-hand side of the substitution.

Well that's all from my little corner of the South Pacific. I must go now as I hear something calling me...

Tim checks in from a spot where the @$%#$%^ snow is falling:

The approach is the same with PowerShell: 1) read the file, 2) remove characters that don't represent hex, and 3) swap the pairs of characters while adding a "\x".

PS C:\> Get-Content shellcode.uni |
ForEach-Object { $_ -replace '[^0-9A-F]', '' } |
ForEach-Object { $_ -replace '(..)(..)', '\x$2\x$1' }


\x50\x53\x51\x52\x56\x57\x55\x9c


Notice the command above uses single quotes. That is because PowerShell will expand any strings inside double quotes before our Replace method has a chance to do any replacing. This means that PowerShell would try to convert $1 into a variable and not pass the literal string to the Replace method. If you happened to use double quotes you would get output like this:

PS C:\> Get-Content shellcode.uni |
ForEach-Object { $_ -replace '[^0-9A-F]', '' } |
ForEach-Object { $_ -replace '(..)(..)', "\x$2\x$1" }


\x\x\x\x\x\x\x\x\x\x...


A longer explanation can be found toward the end of Episode #126.

Now we have the output we want, but let's shorten up the command by using aliases:
PS C:\> gc shellcode.uni |
% { $_ -replace '[^0-9A-F]', '' } |
% { $_ -replace '(..)(..)', '\x$2\x$1' }


We can also remove the ForEach-Object cmdlets and shorten it further:

PS C:\> ((gc shellcode.uni) -replace '[^0-9A-F]', '') -replace '(..)(..)', '\x$2\x$1'


We have one minor problem, if shellcode.uni contains line breaks then each line will be read separately and the breaks won't be removed. If there are multiple lines of text in the file then Get-Content will return an array of objects instead of a multiline string.

PS C:\> (gc shellcode.uni).getType()

IsPublic IsSerial Name BaseType
-------- -------- ---- --------
True True Object[] System.Array


We can fix this by converting it to a string.

PS C:\> ([string](gc shellcode.uni)).getType()

IsPublic IsSerial Name BaseType
-------- -------- ---- --------
True True String System.Object


Our robust shortened version of the command looks like this:

PS C:\> ([string](gc shellcode.uni) -replace '[^0-9A-F]', '') -replace '(..)(..)', '\x$2\x$1'


The final piece is to output our results, and we can use the Out-File cmdlet to do it. However, the default output encoding for PowerShell is unicode which isn't what a disassembler is expecting since it isn't the shellcode we want. We have to tell PowerShell to use ASCII by using the -Encoding parameter.

PS C:\> ... | Out-File shellcode.hex -Encoding ASCII


This our final command looks like this:

PS C:\> ([string](gc shellcode.uni) -replace '[^0-9A-F]', '') -replace '(..)(..)', '\x$2\x$1' |
Out-File shellcode.hex -en ASCII


Well that's all from my frozen corner of the Minnesota. I must go now as I hear something calling me...

Tuesday, April 12, 2011

Episode #142: XML in Shell

Hal gets email from an old friend

We got another email from Ryan Tuthill. We'd been counting the days until his next missive, and suddenly, like the promise of Spring, there it was in our inboxes.

Ryan had been doing a little web surfing and had come across this article for creating a rotating slide show of images on your desktop in Ubuntu. Now I have to admit that my first thought upon reading that article was, "Holy carp! Why didn't they just set up a cron job to change the image on a regular basis instead of mucking around with all that XML nonsense?" But Ryan's question was perhaps more interesting: "Why are they entering all that XML by hand when there ought to be a command line which could generate it for you?"

So let's look at the problem Ryan has set for us. Aside from a header and a footer which we can generate by hand, we need to create a series of XML blocks like this:

<static>
<duration>1795.0</duration>
<file>/some/path/to/img1.png</file>
</static>

<transition>
<duration>5.0</duration>
<from>/some/path/to/img1.png</from>
<to>/some/path/to/img2.png</to>
</transition>

The values that are changing are the image file names in each block.

So it seems we need a list of files and a loop. I think I'll take a cue from our last Episode and just generate my list of files like this:

$ find ~/Documents/My\ Pictures -type f | egrep '\.(gif|jpg|png)$'
/home/hal/Documents/My Pictures/aus-pics/2007-12-06/2007 12 06 Australia 001.jpg
/home/hal/Documents/My Pictures/aus-pics/2007-12-06/2007 12 06 Australia 012.jpg
/home/hal/Documents/My Pictures/aus-pics/2007-12-06/2007 12 06 Australia 006.jpg
,,,

Now I could do an output substitution here with "$(...)", but I'm worried about the size of my input, so I think I'll just pipe things into a nice little while loop:

$ find ~/Documents/My\ Pictures -type f | egrep '\.(gif|jpg|png)$' |
while read file; do
echo "<transition><duration>5.0</duration><from>$prev</from><to>$file</to></transition>";
echo "<static><duration>1795.0</duration><file>$file</file></static>";
prev=$file;
done

<transition><duration>5.0</duration><from></from><to>...001.jpg</to></transition>
<static><duration>1795.0</duration><file>...001.jpg</file></static>
<transition><duration>5.0</duration><from>...001.jpg</from><to>...012.jpg</to></transition>
<static><duration>1795.0</duration><file>...012.jpg</file></static>
<transition><duration>5.0</duration><from>...012.jpg</from><to>...006.jpg</to></transition>
<static><duration>1795.0</duration><file>...006.jpg</file></static>
...

Hey, the XML doesn't have to be beautiful, it just has to be correct.

What my loop does here is track two variables: $file which is the next picture we'll be doing a <transition> block to, and $prev which is the last picture I did a <static> block for. But I cheat a little bit on the first time through the loop. I know I'm not going to have $prev set at this point, so my first <transition> block is going to be erroneous. But I go ahead and dump it out anyway, and follow it up with a (correct) <static> block for the first file name. After I've "primed the pump" so to speak, the <transition> and <static> blocks for the rest of the input will be just fine.

So the first line of output is junk. But we can get rid of the junk line easily with a little post-processing:

$ find ~/Documents/My\ Pictures -type f | egrep '\.(gif|jpg|png)$' |
while read file; do
echo "<transition><duration>5.0</duration><from>$prev</from><to>$file</to></transition>";
echo "<static><duration>1795.0</duration><file>$file</file></static>";
prev=$file;
done | tail -n +2 >slideshow.xml

$ head slideshow.xml
<static><duration>1795.0</duration><file>...001.jpg</file></static>
<transition><duration>5.0</duration><from>...001.jpg</from><to>...012.jpg</to></transition>
<static><duration>1795.0</duration><file>...012.jpg</file></static>
<transition><duration>5.0</duration><from>...012.jpg</from><to>...006.jpg</to></transition>
<static><duration>1795.0</duration><file>...006.jpg</file></static>
...

A little "tail -n +2" after the while loop to scrape off the first line and we're golden. And then all we have to do is redirect our output into a file. You can edit this file by hand and add the necessary additional XML tags per the original article.

So the trick here is to be willing to make a "mistake" at the beginning of your input in order to simplify your loop. You can always clean up after yourself later. And speaking of cleaning up after your own mistakes, here's Tim who's had plenty of practice in that area...

Tim never cleans up after himself

Hal's XML "handling" is just text with extra >'s and <'s thrown in. A few months ago I asked about doing some XML parsing, and his archaic shell couldn't handle it. So I'll play along and create some XML his way.

In brief, we take the same approach as Hal and start off by finding all of the images we want:

PS C:\Users\tm\Pictures> Get-ChildItem -Recurse -Include *.jpg,*.png,*.gif

C:\Users\tm\Pictures\aus-pics\2007-12-06\2007 12 06 Australia 001.jpg
C:\Users\tm\Pictures\aus-pics\2007-12-06\2007 12 06 Australia 002.jpg
C:\Users\tm\Pictures\aus-pics\2007-12-06\2007 12 06 Australia 003.jpg
...


Here is the same command, but using a Get-ChildItem alias and shortened parameter names:

PS C:\Users\tm\Pictures> ls -r -i *.jpg,*.png,*.gif


This gives us all our file objects. From here the quickest way to create our output is to do exactly what Hal did, generate the output, plus mistake, and then clean up.

PS C:\Users\tm\pictures> ls -r -i *.jpg,*.png,*.gif | % {
echo "<transition><duration>5.0</duration><from>$prev</from><to>$_</to></transition>";
echo "n<static><duration>1795.0</duration><file>$_</file></static>"; $prev = $_
} | select -Skip 1


We use Echo (alias of Write-Output) to output each line. The current pipeline object ($_) represents our iterator as we go through our ForEach-Object (alias %) loop. We then use Select-Object (alias select) with the Skip parameter to skip (duh) the first "mistake" line.

Outputting text is fine since we are creating the XML from scratch. Hal would be crying if we actually needed to manipulate or parse XML. We can treat XML as just another object and access the various elements and properties.

PS C:\> $xml = [xml](Get-Content file.xml)
PS C:\> $xml.objects.static
duration file
-------- ----
1795.0 2007 12 06 Australia 001.jpg
1795.0 2007 12 06 Australia 002.jpg
1795.0 2007 12 06 Australia 003.jpg


Oh, if only Hal and his shell from the 1900's could keep up.

Tuesday, April 5, 2011

Episode #141: Bonus Fu

Tim reads the fine print

The mail we received last week included a second part. The sender, Mr Thomas the Anonymous, asked:

Bonus question: we have random files all over the place create by Excel/Word that were exactly eight characters in length and have no file extension such as D0F5KLM3. What's the syntax for use for looking? If I do a "DIR ??????" on my root drive, it appears to be showing all directory names six characters in length and shorter, when I am only looking for those that are exactly six characters in length?

One important piece that Tom the Masked One missed is the /A:-D option to filter out results that have the directory attribute set. This will leave us with only files. Here is the resulting command with the bare format (/B) and recursive (/S) options:

C:\> dir /b /s /a:-d ??????
C:\Program Files\Wireshark\manuf
C:\Program Files\Wireshark\etc\gtk-2.0\gtkrc
C:\Program Files\Wireshark\help\toc
...


The output isn't quite what we wanted, as we are getting files that are shorter than 6 characters.

As it turns out, the "?" wildcard character represents any character...including no character. That means ?????? will match a, aa, aaa, aaaa, aaaaa, and aaaaaa. However, since you can't have "no character" at the beginning (??.txt won't match a.txt) or in the middle (a??a.txt won't match aa.txt) of a string, this special case only occurs when the question mark is at the end of the search string.

We can use our initial command to narrow down our search, but we'll need to use a more aggressive filter to get exactly what we want. FindStr's regular expression filtering to the rescue.

C:\> dir /b /s /a:-d ?????? | findstr /E
"\\[A-Z0-9][A-Z0-9][A-Z0-9][A-Z0-9][A-Z0-9][A-Z0-9]"


C:\Users\theawesomeone\Desktop\randomdir\D0F5KL
...


The /E option matches the string at the end of the line. The regular expression looks for a backslash (requires an escape character, which is a backslash), followed by 6 characters of A-Z and/or 0-9. Unfortunately, this command is a bit verbose since FindStr doesn't allow us to use [A-Z0-9]{6} to specify 6 A-Z and/or 0-9 characters. To do that we have to use...

PowerShell

Unlike CMD.EXE, PowerShell's version of the "?" wildcard requires a character, so it will not match 5 character file names. The first pass of our command is:

PS C:\> Get-ChildItem -Recurse -Include ??????


This command will do a recursive listing and only return objects with a 6 character name; however, the results still include directories and 6 character file names that include a dot. We'll have to do more filtering further down the pipeline. We could do it all further down the pipeline, but the earlier we do the filter the faster the command will be.

Here is our final command:

PS C:\> Get-ChildItem -Recurse -Include ?????? | 
Where-Object { -not $_.PSIsContainer -and $_.Name -match "[A-Z0-9]{6}" }


This further filters our results for objects that are not containers (files) and objects whose name match our regular expression. This regular expression looks for 6 character names that only contain A-Z and/or 0-9.

Per usual, we can shorten up this command significantly.

PS C:\> ls -r -i ?????? | 
? { !$_.PSIsContainer -and $_.Name -match "[A-Z0-9]{6}" }


Now Tom the Hunter of Office Files can clean up some space.

Looks like I scored some bonus points, and in two categories even. Hal, any scoring in *n?x land?

Hal is blue

Given that I'm on the road and only seeing my wife about once every other week, there's not much scoring going on at all. And thanks for rubbing salt into that particular wound, partner.

I'm also a little confused about the challenge. Mystery Tom starts out saying he's looking for eight character file names, and then faster than you can say "original Blade Runner theatrical release", we're suddenly looking for six character file names. Well, let's continue with six character file names because it makes our examples shorter. I'm sure you'll be able to figure out how to do eight character file names without much trouble.

In Unix-land, the "?" actually matches exactly one character. So we can do:

$ find testing/ -type f -name '??????'
[...]
testing/Mac-PropertyList-1.33/t/dict.t
testing/Mac-PropertyList-1.33/t/time.t
testing/Mac-PropertyList-1.33/t/load.t
testing/Mac-PropertyList-1.33/README
testing/Mac-PropertyList-1.33/examples/README
[...]

The only problem is that "?" matches any character, including dots and other punctuation. If we really only want to match files containing upper-case letters and numbers, then we have to use an expression like "[A-Z0-9]". And just like CMD.EXE, find won't let us do something like "[A-Z0-9]{8}". So we're left with:

$ find testing/ -type f -name '[A-Z0-9][A-Z0-9][A-Z0-9][A-Z0-9][A-Z0-9][A-Z0-9]'
[...]
testing/Mac-PropertyList-1.33/README
testing/Mac-PropertyList-1.33/examples/README
[...]

But rather than doing all that typing, let's just use egrep instead:

$ find testing/ -type f | egrep '/[A-Z0-9]{6}$'
[...]
testing/Mac-PropertyList-1.33/README
testing/Mac-PropertyList-1.33/examples/README
[...]

Notice that I'm using "/" at the front of the regex, and "$" at the end to ensure that what I'm matching against is only the final file name component of each line.

So there are my two (OK, three) solutions. Score!

Tuesday, March 29, 2011

Episode #140: Folder Foulups

Tim checks the mailbag this time

Tom He-Who-Shall-Not-Be-Named "Doe" writes in:

We have a network share and while users do not have rights to create folders at the root of the drive, they have the ability to accidentally start dragging a copy of a root folder to some sub-location in a directory where they have modify rights, thus creating incomplete (and unnecessary) copies of root directories. I never get word of this (surprise!).

Can you put together a script that I can run periodically which will dynamically inventory the top-level directories (currently 28) on this network share and looking for sub-folders with matching names *** all the way down *** underneath those top-level dirs? Yes, I realize this could run for hours, but I will run it against a DR box that has DFSR on it, so I won't impact performance for the users.


He also included a sample directory structure:

DIR1
-----> FolderA
-----> FolderB
-----> FolderC
DIR2
----->FolderD
DIR3
----->DIR1
----------> FolderA
----------> FolderB
----------> FolderC
----->FolderE


Per our conversation with Tom I'm-on-the-run-from-the-mob Doe, all we need to do is match a subdirectory name with the name of one of the root directories. Here is the rather simple command to do what we need:

PS C:\MyShare> $a = ls | % {$_.Name}; ls -r * | 
? { $_.PSIsContainer -and $a -contains $_.Name } | select fullname


FullName
--------
C:\MyShare\DIR3\DIR1


The are two parts to our command. First, we load a variable ($a) with the names of each of our root directories. We have to use the ForEach-Object cmdlet to extract the name in a scalar, string format.

The second portion is our search. We start off with a recursive directory listing and send the objects into a Where-Object (alias ?) filter. Our filter ensures that we are only looking at directory objects (Containers). It also checks to see if the Name of the current object is contained in our collection of directory names. If both criteria are satisfied the results are passed down the pipeline. Finally, we output just the full path (FullName) of our object.

There is one tiny little nugget of information that I didn't mention, but if you looked really close you might have wondered why I added the asterisk with ls -r. Why? Well let's see what happens when we don't use it:

PS C:\MyShare> $a = ls | % {$_.Name}; ls -r | 
? { $_.PSIsContainer -and $a -contains $_.Name } | select fullname


FullName
--------
C:\MyShare\DIR1
C:\MyShare\DIR2
C:\MyShare\DIR3
C:\MyShare\DIR3\DIR1


For some reason the command "ls -r" includes the directory listing for the current directory but "ls -r *" does not. It is a cool little trick, but I have no idea why it works. Here is a better demonstration using the Compare-Object cmdlet:

PS C:\MyShare> Compare-Object (ls -r) (ls -r *)

InputObject SideIndicator
----------- -------------
DIR1 <=
DIR2 <=
DIR3 <=


Weird, but it works.

CMD.EXE

While we can't take the same approach with CMD.EXE, we can accomplish this task with the following command:

C:\MyShare> for /F "tokens=*" %a in ('dir /B /S /A:D') do
@for /F "tokens=*" %r in ('dir /B /A:D') do @echo %a| findstr "\\%r$"

C:\MyShare\DIR1
C:\MyShare\DIR2
C:\MyShare\DIR3
C:\MyShare\DIR3\DIR1


Our command wraps a For loop around a bare format (/B), recursive (/S) directory listing that only looks for directories (/A:D). The variable %a contains the full path of our directory. We then use another For loop to get the names (not full path) of the directories in our root folder. It should be noted that the bare format (/B) is different depending on whether the recursive option (/S) is used. The format is the full format when used with /S, and just the filename without /S.

We then use Echo to output %a (full directory path), and use Findstr to search for %r (a root folder name). The regular expression ensures the name is a full match, and that the match is at the end of the string. To do this we make sure that our name is between a backslash and the end of line ($). The backslash is the regular expression escape character so it must be escaped with another backslash.

Unfortunately, this command is going to be slooooooow. For each new directory we traverse, we will have to do a directory listing of our root directory. This means we will probably do thousands of extra directory listings that we don't need to do with PowerShell.

We can filter out our initial directories by adding an extra Findstr command to the end to filter out anything that is two levels deep. This additional filter looks like this:

 ... | findstr /V "C:\\MyShare\\[^\\]*$"
C:\MyShare\DIR3\DIR1


The /V only prints lines that do not contain a match. Our search string matches the first portion of our path ("C:\MyShare\"), then any character that isn't a backslash, and finally the end of line character. The means that C:\MyShare\Blah will be filtered out, but C:\MyShare\Blah\Blah will be displayed.

It may not be fast, but at least its [mostly] easy. Now, let's see if Hal fast and easy.

Hal wonders if Tim is projecting a bit

Personally I prefer to think of it as "efficient and morally flexible", but have it your own way. As far as the challenge goes, Unix makes this both fast and, well... easy is in the eye of the beholder I guess.

There is a simple find command for solving this challenge:

$ find */* -type d \( -name DIR1 -o -name DIR2 -o -name DIR3 \)
DIR3/DIR1

Notice that I'm doing "find */* ..." so that I start my search at the level below the top. Otherwise I'd get matches on the top-level directories themselves, and that wouldn't be helpful.

This command works great as long as there are only three directories and their names don't change. But Mystery Tom's challenge specifies that there are actually 28 directories, which is way more than I'd want to type by hand. So it would be cool if there was a way to create the list of "-name DIR1 -o -name DIR2 ..." automatically.

It's actually pretty straightforward to do this:

$ ls | tr \\n ' ' | sed 's/ $//; s/ / -o -name /g'
DIR1 -o -name DIR2 -o -name DIR3

I start with the list of top-level directories, which the ls command will output with newlines after each name. So I use tr to transform the newlines to spaces, which leaves me with a trailing space in place of the last newline. But since I'm feeding the whole thing into sed anyway, I first have sed remove the trailing space and then replace all of the remaining spaces with " -o -name ".

The only problem is that I have no "-name" at the beginning of my output, but I can fix that up when I do the output substitution into the original find command:

$ find */* -type d \( -name $(ls | tr \\n ' ' | sed 's/ $//; s/ / -o -name /g') \)
DIR3/DIR1

All I did here was take our ls pipeline and throw it inside "$(...)"-- the command output substitution operation-- in place of the "DIR1 -o -name DIR2 -o -name DIR3" that I hand-entered in the original find command. Since we needed the extra "-name" at the front, I just put that in manually before the "$(...)". So now we have a command that will work for any number of directories and even work when the directory names change.

And it's much sexier than either of Tim's solutions.