Episode #174: Lightning Lockdown

Hal firewalls fast

Recently a client needed me to quickly set up an IP Tables firewall on a production server that was effectively open on the Internet. I knew very little about the machine, and we couldn't afford to break any of the production traffic to and from the box.

It occurred to me that a decent first approximation would be to simply look at the network services currently in use, and create a firewall based on that. The resulting policy would probably be a bit more loose than it needed to or should be, but it would be infinitely better than no firewall at all!

I went with lsof, because I found the output easier to parse than netstat:

# lsof -i -nlP | awk '{print $1, $8, $9}' | sort -u
COMMAND NODE NAME
httpd TCP *:80
named TCP 127.0.0.1:53
named TCP 127.0.0.1:953
named TCP [::1]:953
named TCP 150.123.32.3:53
named UDP 127.0.0.1:53
named UDP 150.123.32.3:53
ntpd UDP [::1]:123
ntpd UDP *:123
ntpd UDP 127.0.0.1:123
ntpd UDP 150.123.32.3:123
ntpd UDP [fe80::baac:6fff:fe8e:a0f1]:123
ntpd UDP [fe80::baac:6fff:fe8e:a0f2]:123
portreser UDP *:783
sendmail TCP 150.123.32.3:25
sendmail TCP 150.123.32.3:25->58.50.15.213:1526
sendmail TCP *:587
sshd TCP *:22
sshd TCP 150.123.32.3:22->121.28.56.2:39054

I could have left off the process name, but it helped me decide which ports were important to include in the new firewall rules. Honestly, the output above was good enough for me to quickly throw together some workable IP Tables rules. I simply saved the output to a text file and hacked things together with a text editor.

But maybe you only care about the port information:

# lsof -i -nlP | awk '{print $9, $8, $1}' | sed 's/.*://' | sort -u
123 UDP ntpd
1526 TCP sendmail
22 TCP sshd
25 TCP sendmail
39054 TCP sshd
53 TCP named
53 UDP named
587 TCP sendmail
783 UDP portreser
80 TCP httpd
953 TCP named
NAME NODE COMMAND

Note that I inverted the field output order, just to make my sed a little easier to write

If you wanted to go really crazy, you could even create and load the actual rules on the fly. I don't recommend this at all, but it will make Tim's life harder in the next section, so here goes:

lsof -i -nlP | tail -n +2 | awk '{print $9, $8}' | 
    sed 's/.*://' | sort -u | tr A-Z a-z | 
    while read port proto; do ufw allow $port/$proto; done

I added a "tail -n +2" to get rid of the header line. I also dropped the command name from my awk output. There's a new "tr A-Z a-z" in there to lower-case the protocol name. Finally we end with a loop that takes the port and protocol and uses the ufw command line interface to add the rules. You could do the same with the iptables command and its nasty syntax, but if you're on a Linux distro with UFW, I strongly urge you to use it!

So, Tim, I figure you can parse netstat output pretty easily. How about the command-line interface to the Windows firewall? Remember, adversity builds character...

Tim builds character

When I first saw this I thought, "Man, this is going to be easy with the new cmdlets in PowerShell v4!" There are a lot of new cmdlets available in PowerShell version 4, and both Windows 8.1 and Server 2012R2 ship with PowerShell version 4. In addition, PowerShell version 4 is available for Windows 7 SP1 (and later) and Windows Server 2008 R2 SP1 (and later).

The first cmdlet that will help us out here is Get-NetTCPConnection. According to the help page this cmdlet "gets current TCP connections. Use this cmdlet to view TCP connection properties such as local or remote IP address, local or remote port, and connection state." This is going to be great! But...

It doesn't mention the process ID or process name. Nooooo! This can't be. Let's look at all the properties of the output objects.

PS C:\> Get-NetTCPConnection | Format-List *

State                    : Established
AppliedSetting           : Internet
Caption                  :
Description              :
ElementName              :
InstanceID               : 192.168.1.167++445++10.11.22.33++49278
CommunicationStatus      :
DetailedStatus           :
HealthState              :
InstallDate              :
Name                     :
OperatingStatus          :
OperationalStatus        :
PrimaryStatus            :
Status                   :
StatusDescriptions       :
AvailableRequestedStates :
EnabledDefault           : 2
EnabledState             :
OtherEnabledState        :
RequestedState           : 5
TimeOfLastStateChange    :
TransitioningToState     : 12
AggregationBehavior      :
Directionality           :
LocalAddress             : 192.168.1.167
LocalPort                : 445
RemoteAddress            : 10.11.22.33
RemotePort               : 49278
PSComputerName           :
CimClass                 : ROOT/StandardCimv2:MSFT_NetTCPConnection
CimInstanceProperties    : {Caption, Description, ElementName, InstanceID...}
CimSystemProperties      : Microsoft.Management.Infrastructure.CimSystemProperties

Dang! This will get most of what we want (where "want" was defined by that Hal guy), but it won't get the process ID or the process name. So much for rubbing the new cmdlets in his face.

Let's forget about Hal for a second and get what we can with this cmdlet.

PS C:\> Get-NetTCPConnection | Select-Object LocalPort | Sort-Object -Unique LocalPort
LocalPort
---------
    135
    139
    445
   3587
   5357
  49152
  49153
  49154
  49155
  49156
  49157
  49164

This is helpful for getting a list of ports, but not useful for making decisions about what should be allowed. Also, we would need to run Get-NetUDPEndpoint to get the UDP connections. This is so close, yet so bloody far. We have to resort to the old school netstat command and the -b option to get the executable name. In episode 123 we needed parsed netstat output. I recommended the Get-Netstat script available at poshcode.org. Sadly, we are going to have to resort to that again. With this script we can quickly get the port, protocol, and process name.

PS C:\> .\get-netstat.ps1 | Select-Object ProcessName, Protocol, LocalPort | 
   Sort-Object -Unique LocalPort, Protocol, ProcessName

ProcessName   Protocol      Localport
-----------   --------      ---------
svchost       TCP           135
System        UDP           137
System        UDP           138
System        TCP           139
svchost       UDP           1900
svchost       UDP           3540
svchost       UDP           3544
svchost       TCP           3587
dasHost       UDP           3702
svchost       UDP           3702
System        TCP           445
svchost       UDP           4500
...

It should be pretty obvious that the port 137-149 and 445 should not be accessible from the internet. We can filter these ports out so that we don't allow these ports through the firewall.

PS C:\> ... | Where-Object { (135..139 + 445) -NotContains $_.LocalPort }
ProcessName   Protocol      Localport
-----------   --------      ---------
svchost       UDP           1900
svchost       UDP           3540
svchost       UDP           3544
svchost       TCP           3587
dasHost       UDP           3702
svchost       UDP           3702
svchost       UDP           4500
...

Now that we have the ports and protocols we can create new firewall rules using the new New-NetFirewallRule cmdlet. Yeah!

PS C:\> .\get-netstat.ps1 | Select-Object Protocol, LocalPort | Sort-Object -Unique * | 
 Where-Object { (135..139 + 445) -NotContains $_.LocalPort } | 
 ForEach-Object { New-NetFirewallRule -DisplayName AllowedByScript -Direction Outbound 
 -Action Allow  -LocalPort $_.LocalPort -Protocol $_.Protocol }
Name                  : {d15ca484-5d16-413f-8460-a29204ff06ed}
DisplayName           : AllowedByScript
Description           :
DisplayGroup          :
Group                 :
Enabled               : True
Profile               : Any
Platform              : {}
Direction             : Outbound
Action                : Allow
EdgeTraversalPolicy   : Block
LooseSourceMapping    : False
LocalOnlyMapping      : False
Owner                 :
PrimaryStatus         : OK
Status                : The rule was parsed successfully from the store. (65536)
EnforcementStatus     : NotApplicable
PolicyStoreSource     : PersistentStore
PolicyStoreSourceType : Local
...

These new firewall cmdlets really make things easier, but if you don't have PowerShellv4 you can still use the old netsh command to add the firewall rules. Also, the Get-Netstat will support older version of PowerShell as well, so this is nicely backwards compatible. All we need to do is replace the command inside the ForEach-Object cmdlet's script block.

PS C:\> ... | ForEach-Object { netsh advfirewall firewall add rule 
 name="AllowedByScript" dir=in action=allow protocol=$_.Protocol 
 localport=$_.LocalPort }

Episode #173: Tis the Season

Hal finds some cheer
From somewhere near the borders of scriptistan, we send you:

function t { 
    for ((i=0; $i < $1; i++)); do
        s=$((8-$i)); e=$((8+$i));
        for ((j=0; j <= $e; j++)); do [ $j -ge $s ] && echo -n '^' || echo -n ' '; done;
        echo;
    done
}
function T {
    for ((i=0; $i < $1; i++)); do
        for ((j=0; j < 10; j++)); do [ $j -ge 7 ] && echo -n '|' || echo -n ' '; done;
        echo;
    done
    echo
}
t 3; t 5; t 7; T 2; echo -e "Season's Greetings\n    from CLKF"

Ed comes in out of the cold:

Gosh, I missed you guys.  It's nice to be home with my CLKF family for the holidays.  I brought you a present:

c:\>cmd.exe /v:on /c "echo. & echo A Christmas present for you: & color 24 & 
echo. & echo     0x0& for /L %a in (1,1,11) do @(for /L %b in (1,1,10) do @ set /a
%b%2) & echo 1"& echo. & echo Merry Christmas!

Tim awaits the new year:

Happy New Year from within the borders of Scriptistan!

Function Draw-Circle {
    Param( $Radius, $XCenter, $YCenter )
    
    for ($x = -$Radius; $x -le $Radius ; $x++) {
        $y = [int]([math]::sqrt($Radius * $Radius - $x * $x))
        Set-CursorLocation -X ($XCenter + $x) -Y ($YCenter + $y)
        Write-Host "*" -ForegroundColor Blue -NoNewline
        Set-CursorLocation -X ($XCenter + $x) -Y ($YCenter - $y)
        Write-Host "*" -ForegroundColor Blue -NoNewline
    }
}

Function Draw-Hat {
    Param( $XCenter, $YTop, $Height, $Width, $BrimWidth )
    
    $left = Round($XCenter - ($Width / 2))
    $row = "#" * $Width
    for ($y = $YTop; $y -lt $YTop + $Height - 1; $y++) {
        Set-CursorLocation -X $left -Y $y
        Write-Host $row -ForegroundColor Black -NoNewline
    }
    
    Set-CursorLocation -X ($left - $BrimWidth) -Y ($YTop + $Height - 1)
    $row = "#" * ($Width + 2 * $BrimWidth)
    Write-Host $row -ForegroundColor Black -NoNewline
}

Function Set-CursorLocation {
    Param ( $x, $y )

    $pos = $Host.UI.RawUI.CursorPosition
    $pos.X = $x
    $pos.Y = $y
    $Host.UI.RawUI.CursorPosition = $pos
}

Function Round {
    Param ( $int )
    # Stupid banker's rounding
    return [Math]::Round( $int, [MidpointRounding]'AwayFromZero' )
}

Clear-Host
Write-Host "Happy New Year!"
Draw-Circle -Radius 4 -XCenter 10 -YCenter 8
Draw-Circle -Radius 5 -XCenter 10 -YCenter 17
Draw-Circle -Radius 7 -XCenter 10 -YCenter 29
Draw-Hat -XCenter 10 -YTop 2 -Height 5 -Width 7 -BrimWidth 2
Set-CursorLocation -X 0 -Y 38

Episode #172: Who said bigger is better?

Tim sweats the small stuff

Ted S. writes in:

"I have a number of batch scripts which turn a given input file into a configurable amount of versions, all of which will contain identical data content, but none of which, ideally, contain the same byte content. My problem is, how do I, using *only* XP+ cmd (no other scripting - PowerShell, jsh, wsh, &c), replace the original (optionally backed up) with the smallest of the myriad versions produced by the previous batch runs?"

This is pretty straight forward, but it depends on what we want to do with the files. I assumed that the larger files should be deleted since they are redundant. This will leave us with only the smallest file in the directory. Let's start off by listing all the files in the current directory and sort them by size.

C:\> dir /A-D /OS /b
file3.txt
file2.txt
file1.txt
file4.txt

Sorting the files, and only files, in the current directory by size is pretty easy. The "/A" option filters on the object's properties and directories are filtered out with "-D". Next, the "/O" option is used to sort and the "S" tells the command to sort putting the smallest files first. Finally, the "/b" is used to show the bare format.

At this point we have the files in the proper order and in a nice light format. We can now use a For loop to delete everything while skipping the first file.

C:\> for /F "tokens=* skip=1" %i in ('dir /A-D /OS /b') do @del %i

Here is the same functionality in PowerShell:

PS C:\> Get-ChildItem | Where-Object { -not $_.PSIsContainer } | Sort-Object -Property Length | Select-Object -Skip 1 | Remove-Item

This is mostly readable. The only exception is the "PSIsContainer". Directories are container objects but files are not, so we filter out the containers (directories). Here is the same command shortented using aliases and positional parameters:

PS C:\> ls | ? { !$_.PSIsContainer } | sort Length | select -skip 1 | rm

There you go Ted, and in PowerShell even though you didn't want it. Here comes Hal brining something even smaller you don't want.

Hal's is smaller than Tim's... but less sweaty

Tim, how many times do I have to tell you, smaller is better when it comes to command lines:

ls -Sr | tail -n +2 | xargs rm

It's actually not that different from Tim's PowerShell solution, except that my "ls" command has "-S" to sort by size as a built-in. We use the "-r" flag to reverse the sort, putting the smallest file first and skipping it with "tail -n +2".

If you're worried about spaces in the file names, we could tart this one up a bit more:

ls -Sr | tail -n +2 | tr \\n \\000 | xargs -0 rm

After I use "tail" to get rid of the first, smallest file, I use "tr" to convert the newlines to nulls. That allows me to use the "-0" flag to "xargs" to split the input on nulls, and preserves the spaces in the input file names.

What may be more interesting about this Episode is the command line I used to create and re-create my files for testing. First I made a text file with lines like this:

1 3
2 4
3 1
4 2

And then I whipped up a little loop action around the "dd" command:

$ while read file size; do 
      dd if=/dev/zero bs=4K count=$size of=file$file; 
  done <../input.txt 
3+0 records in
3+0 records out
12288 bytes (12 kB) copied, 6.1259e-05 s, 201 MB/s
4+0 records in
4+0 records out
16384 bytes (16 kB) copied, 0.000144856 s, 113 MB/s
1+0 records in
1+0 records out
4096 bytes (4.1 kB) copied, 3.4961e-05 s, 117 MB/s
2+0 records in
2+0 records out
8192 bytes (8.2 kB) copied, 4.3726e-05 s, 187 MB/s

Then I just had to re-run the loop whenever I wanted to re-create my test files after deleting them.

Episode #171: Flexibly Finding Firewall Phrases

Old Tim answers an old email

Patrick Hoerter writes in:
I have a large firewall configuration file that I am working with. It comes from that vendor that likes to prepend each product they sell with the same "well defended" name. Each configuration item inside it is multiple lines starting with "edit" and ending with "next". I'm trying to extract only the configuration items that are in some way tied to a specific port, in this case "port10".

Sample Data:

edit "port10"
        set vdom "root"
        set ip 192.168.1.54 255.255.255.248
        set allowaccess ping
        set type physical
        set sample-rate 400
        set description "Other Firewall"
        set alias "fw-outside"
        set sflow-sampler enable
   next
edit "192.168.0.0"
        set subnet 192.168.0.0 255.255.0.0
    next
    edit "10.0.0.0"
        set subnet 10.0.0.0 255.0.0.0
    next
    edit "172.16.0.0"
        set subnet 172.16.0.0 255.240.0.0
    next
  edit "vpn-CandC-1"
        set associated-interface "port10"
        set subnet 10.254.153.0 255.255.255.0
    next
    edit "vpn-CandC-2"
        set associated-interface "port10"
        set subnet 10.254.154.0 255.255.255.0
    next
    edit "vpn-CandC-3"
        set associated-interface "port10"
        set subnet 10.254.155.0 255.255.255.0
    next
   edit 92
        set srcintf "port10"
        set dstintf "port1"
            set srcaddr "vpn-CandC-1" "vpn-CandC-2" "vpn-CandC-3"            
            set dstaddr "all"            
        set action accept
        set schedule "always"
            set service "ANY"            
        set logtraffic enable
    next
 

Sample Results:

edit "port10"
        set vdom "root"
        set ip 192.168.1.54 255.255.255.248
        set allowaccess ping
        set type physical
        set sample-rate 400
        set description "Other Firewall"
        set alias "fw-outside"
        set sflow-sampler enable
   next
  edit "vpn-CandC-1"
        set associated-interface "port10"
        set subnet 10.254.153.0 255.255.255.0
    next
    edit "vpn-CandC-2"
        set associated-interface "port10"
        set subnet 10.254.154.0 255.255.255.0
    next
    edit "vpn-CandC-3"
        set associated-interface "port10"
        set subnet 10.254.155.0 255.255.255.0
    next
   edit 92
        set srcintf "port10"
        set dstintf "port1"
            set srcaddr "vpn-CandC-1" "vpn-CandC-2" "vpn-CandC-3"            
            set dstaddr "all"            
        set action accept
        set schedule "always"
            set service "ANY"            
        set logtraffic enable
    next

Patrick gave us the full text and the expected output. In short, he wants the text between "edit" and "next" if it contains the text "port10". To begin this task we need to first need get each of the edit/next chunks.

PS C:\> ((cat fw.txt) -join "`n") | select-string "(?s)edit.*?next" -AllMatches | 
 select -ExpandProperty matches

This command will read the entire file fw.txt and combine it into one string. Normally, each line is treated as a separate object, but we are going to join them into a big string using the newline (`n) to join each line. Now that the text is one big string we can use Select-String with a regular expression to find all the matches. The regular expression will find text across line breaks and allows for very flexible searches so we can find our edit/next chunks. Here is a break down of the pieces of the regular expression:

• (?s) - Use single line mode where the dot (.) will match any character, including a newline character. This allows us to match text across multiple lines.
• edit - the literal text "edit"
• .*? - find any text, but be lazy, not greedy. This means it should match the smallest chunks that will match the criteria.
• next - literal text next

Now that we have the chunks we use a Where-Object filter (alias ?) to find matching objects to pass down the pipeline.

PS C:\> ((cat .\fw.txt) -join "`n") | select-string "(?s)edit.*?next" -AllMatches | 
 select -ExpandProperty matches | ? { $_.Captures | Select-String "port10" }

Inside the Where-Object filter we can check the Value property to see if it contains the text "port10". The Value property is piped into Select-String to look for the text "port10", and if it contains "port10" it continues down the pipeline, if not, it is dropped.

At this point, we have the objects we want, so all we need to do is display the results by expanding the Value and displaying it again. The expansion means that it just displays the text and no data or metadata associated with the parent object. Here is what the final command looks like.

PS C:\> ((cat .\fw.txt) -join "`n") | select-string "(?s)edit.*?next" -AllMatches | 
 select -ExpandProperty matches | ? { $_.Value | Select-String "port10" } | 
 select -ExpandProperty Value

Not so bad, but I have a feeling it is going to be worse for my friend Hal.

Old Hal uses some old tricks

Oh sure, I know what Tim's thinking here. "It's multi-line matching, and the Unix shell is lousy at that. Hal's in trouble now. Mwhahaha. The Command-Line Kung Fu title will finally be mine! Mine! Do you hear me?!? MINE!"

Uh-huh. Well how about this, old friend:

awk -v RS=next -v ORS=next '/port10/' fw.txt

While we're doing multi-line matching here, the blocks of text have nice regular delimiters. That means I can change the awk "record separator" ("RS") from newline to the string "next" and gobble up entire chunks at a time.

After that, it's smooth sailing. I just use awk's pattern-matching operator to match the "port10" strings. Since I don't have an action defined, "{print}" is assumed and we output the matching blocks of text.

The only tricky part is that I have to remember to change the "output record separator" ("ORS") to be "next". Otherwise, awk will use its default ORS value, which is newline. That would give me output like:

$ awk -v RS=next '/port10/' fw.txt
edit "port10"
        set vdom "root"
        set ip 192.168.1.54 255.255.255.248
        set allowaccess ping
        set type physical
        set sample-rate 400
        set description "Other Firewall"
        set alias "fw-outside"
        set sflow-sampler enable
   

  edit "vpn-CandC-1"
        set associated-interface "port10"
        set subnet 10.254.153.0 255.255.255.0
    

    edit "vpn-CandC-2"
        set associated-interface "port10"
...

The "next" terminators get left out and we get extra lines in the output. But when ORS is set properly, we get exactly what we were after:

$ awk -v RS=next -v ORS=next '/port10/' fw.txt
edit "port10"
        set vdom "root"
        set ip 192.168.1.54 255.255.255.248
        set allowaccess ping
        set type physical
        set sample-rate 400
        set description "Other Firewall"
        set alias "fw-outside"
        set sflow-sampler enable
   next
  edit "vpn-CandC-1"
        set associated-interface "port10"
        set subnet 10.254.153.0 255.255.255.0
    next
    edit "vpn-CandC-2"
        set associated-interface "port10"
...

So that wasn't bad at all. Sorry about that Tim. Maybe next time, old buddy.

Episode #170: Fearless Forensic File Fu

Hal receives a cry for help

Fellow forensicator Craig was in a bit of a quandary. He had a forensic image in "split raw" format-- a complete forensic image broken up into small pieces. Unfortunately for him, the pieces were named "fileaa", "fileab", "fileac", and so on while his preferred tool wanted the files to be named "file.001", "file.002", "file.003", etc. Craig wanted to know if there was an easy way to rename the files, using either Linux or the Windows shell.

This one's not too hard in Linux, and in fact it's a lot like something we did way back in Episode #26:

c=1; 
for f in file*; do 
    printf -v ext %03d $(( c++ )); 
    mv $f ${f/%[a-z][a-z]/.$ext}; 
done

You could remove the newlines and make that one big long line, but I think it's a bit easier to read this way. First we initialize a counter variable $c to 1. Then we loop over each of the files in our split raw image.

The printf statement inside the loop formats $c as three digits, with however many leading zeroes are necessary ("%03d"). There are a couple of tricky bits in the printf though. First is we're assigning the output of printf to a variable $ext ("-v ext"). Second, we're doing a little arithmetic on $c at the same time and using the "++" operator to increment the value of $c each time through the loop-- that's the "$(( c++ ))" part.

Then we use mv to rename our file. I'm using the variable substitution operator like we did in Episode #26. The format again is "${var/pattern/substitution}" and here the "%" after the first slash means "match at the end of the string". So I'm replacing the last two letters in the file name with a dot followed by our $ext value. And that's exactly what Craig wanted!

All of the symbols in this solution make it appear like a little chunk of line noise, but it's nowhere near as ugly as Ed's CMD.EXE solution in Episode #26. Here's hoping Tim's Powershell solution is a bit more elegant.

Tim finishes before September ends!

Elegance where here we come!

Long Version:
PS C:\> $i=1; Get-ChildItem file?? | Sort-Object -Propery Name | 
  ForeEach-Object { MoveItem -Path $_ -Destination ("file.{0:D3}" -f $i++) }
Shortened Version:
PS C:\> ls file?? | sort name | % { move $_ -dest ("file.{0:D3}" -f $i++) }

We start off by initializing our counter variable ($i) to 1 just like Hal did. Next, we list all the files that start with "file" and are followed by exactly two characters (each ? matches exactly 1 character of any kind). The results are then sorted by the file name to ensure that the files are renamed in the correct order. The results are then fed into the ForEach-Object cmdlet (alias %).

The ForEach-Object loop will operate on each object (file) as it moves down the pipeline. One at a time, each file will be represented by the current pipeline object ($_). The Move-Item cmdlet (alias move) is used to rename a file; to move it to its new name. The source path is provided by the current object and the destination is determined using the format operator (-f) and our counter ($i). The format operator will print $i as a three digit number prefixed with leading zeros and "file.". The ++ after $i will increment the counter after it has been used.

That is much cleaner than Ed's example...and even cleaner than Hal's to boot!

Update:

Reader m_cnd writes in with a solution for CMD. vm

C:\> for /F "tokens=1,2 delims=:" %d in ('dir /on /b file* ^| 
findstr /n "file"') do for /F %x in ('set ext^=00%d^&^& 
cmd /v:on /c "echo !ext:~-3!"') do rename %e file.%x

Nice work!

Episode #169: Move Me Maybe

Tim checks the mailbag

Carlos IHaveNoLastName writes in asking for a way to move a directory to a new destination. That's easy, but the directory should only be moved if the the directory (at any depth) does NOT contain a file with a specific extenstion.

Here is an example of a sample directory structure:

SomeTopDir1
|-OtherDir1
|  |-File1
|  |-File2
|  |-File2
|-OtherDir2
   |-File1
   |-File.inprogress

SomeTopDir2
|-OtherDir1
|  |-File1
|  |-File2
|  |-File2
|-OtherDir2
   |-File1
   |-File2

In this example we should NOT move SomeTopDir1 because it contains a file with the string "inprogress". We should however move SomeTopDir2 because it contains no such file. In short, "inprogress" means leave it alone.

Executing this in PowerShell is quite easy. CMD is a pain, and I'll skip that crazy long command because it is a circus trick. Here is the command to do exactly what Carlos asked:

PS C:\jobsdir> Get-ChildItem | ? { $_.PSIsContainer } | 
    ? { -not ( Get-ChildItem $_ -Recurse -Filter *.inprogress ) } | 
    Move-Item -Destination \archive
 

This command with use Get-ChildItem to list the contents of the current directory. We first filter for Directories (Container objects) just in case there are files in the root of the directory that we don't want to move. Next, another Where-Object cmdlet (alias ?) is used to check all the sub-directories and look for a file matching "*.inprogress". The -Not operator inverts the match so that only directores with a "*.inprogress" file will be passed down the pipeline.

At this point we have the directories that do not contain this file. The results are then piped into Move-Item and the directories are moved to the \archive directory.

One of the other criteria that Mr. IHaveNoLastName requested is that the command must work on XP. Well it does, but only if you install PowerShell. Sadly, XP does not support PowerShell v3. With PowerShell v3's simplified syntax (and some additional aliases) we can shorten the command to this:

PS C:\jobsdir> ls | ? PSIsContainer | ? { -not ( ls $_ -r -fi *.inprogress ) } | 
    mv -d \archive
 

Thanks for an easy one Carlos! Hal, your turn. I suspect this is will be almost as easy for you (even though it won't work on XP).

Hal takes it easy

This one's quite do-able in the shell. But unlike Tim's solution, the most straightforward approach in Linux is a loop:

for i in *; do [ "$(find $i -type f -name \*.inprogress)" ] || mv $i /some/dest; done

The loop is over all of the directories in the current directory. Inside the loop we run a find command looking for "*.inprogress" files. If we find any, then the test operator ("[ ... ]") returns true and we don't do the mv command on the other side of the "||". If we find nothing, then the directory gets moved. Easy peasy

"But wait!", I hear you cry, "That was too easy. And besides, you're running a mv command for each individual directory!"

OK, fine. You want a single mv command? Here you go:

mv $(ls | grep -vf <(find * -type f -name \*.inprogress | cut -f1 -d/)) /some/dest

Happy now?

The best way to puzzle this one out is to start with the command in the innermost parentheses:

find * -type f -name \*.inprogress | cut -f1 -d/

The find command returns the pathnames of all of the *.inprogress files, and the cut command pulls off the top-level directory name. If there are multiple *.inprogress files in a single directory, we'll get multiple instances of the top-level directory name, but that doesn't really matter.

The "<( ... )" syntax takes the output of our find pipeline and lets it be treated as an input file for another command:

ls | grep -vf <( ... )

We take the output of ls and use "grep -v" to filter out directories we don't want. Normally "grep -f" takes a list of patterns from an input file, but in this case we use the "<( ... )" syntax to substitute our find output instead of a normal input file. So we suppress any directories that have a *.inprogress file in them. Anything left over is a directory without a *.inprogress file, which is precisely the set of directories we want to move.

So we wrap the complicated ls pipline up in "$(...)" so that the output-- the list of directories we want to move-- is substituted into the "mv $(...) /some/dest" command. And that gets us to where we want to be.

Or you could use the same idea, but with xargs:

ls | grep -vf <(find * -type f -name \*.inprogress | cut -f1 -d/) | xargs mv -d /some/dest

This looks a lot more like Tim's approach in Powershell. However, this makes use of the "mv -d /some/dest ..." syntax that's supported in the GNU version of the command, but not widely supported in other more traditional Unix distros.

Oh, and by the way, Tim, this all works fine under Windows XP if you'd just install Cygwin like I've been telling you to...

Update:

m_cnd wrote in again with a shortcut for CMD.EXE:

dir SomeTopDir /s /b | findstr /i /e ".extension" > nul || move SomeTopDir Destination

I use this trick all the time, so I feel bad that I missed it here. With his shortcut I can put together a For loop (our favorite, and only, text parser) to do the work.

C:\> for /F "tokens=*" %i in ('dir /b /AD') do dir "%i" /s /b /a-d "%i\*.extension" 1>nul 2>nul || move "%i" Destination

The Dir command with /AD will list directories and not files. We can then use the output to search and move if necessary. The Tokens and quotes are used in case the directory names contain spaces.

Thanks again m_cnd!

Episode #168: Scan On, You Crazy Command Line

Hal gets back to our roots

With one ear carefully tuned to cries of desperation from the Internet, it's no wonder I picked up on this plea from David Nides on Twitter:

Request today, we need 2 scan XX terabytes of data across 3k file shares 4any files that have not been MAC since 2012. Then move files to x.— David Nides (@DAVNADS) March 13, 2013

Whenever I see a request to scan for files based on a certain criteria and then copy them someplace else, I immediately think of the "find ... | cpio -pd ..." trick I've used in several other Episodes.

Happily, "find" has "-mtime", "-atime", and "-ctime" options we can use for identifying the files. But they all want their arguments to be in terms of number of days. So I need to calculate the number of days between today and the end of 2012. Let's do that via a little command-line kung fu, shall we? That will make this more fun.

$ days=$(( ($(date +%Y) - 2012)*365 + $(date +%j | sed 's/^0*//') ))
$ echo $days
447

Whoa nelly! What just happened there? Well, I'm doing math with the bash "$(( ... ))" operator and assigning the result to a variable called "days" so I can use it later. But what's all that line noise in the middle?

• "date +%Y" returns the current year. That's inside "$( ... )" so I can use the value in my calculations.
• I subtract 2012 from the current year to get the number of years since 2012 and multiply that by 365. Screw you, leap years!
• "date +%j" returns the current day of the year, a value from 001-365.
• Unfortunately the shell interprets values with leading zeroes as octal and errors out on values like "008" and "097". So I use a little sed to strip the leading zeroes.

Hey, I said it would be fun, not that it would necessarily be a good idea!

But now that I've got my "$days" value, the answer to David's original request couldn't be easier:

$ find /some/dir -mtime +$days -atime +$days -ctime +$days | cpio -pd /new/dir

The "find" command locates files whose MAC times are all greater than our "$days" value-- that's what the "+$days" syntax means. After that, it's just a matter of passing the found files off to "cpio". Calculating "$days" was the hard part.

My final solution was short enough that I tweeted it back to David. Which took me all the way back to the early days of Command-Line Kung Fu, when Ed Skoudis had hair would tweet cute little CMD.EXE hacks that he could barely fit into 140 characters. And I would respond with bash code that would barely line wrap. Ah, those were the days!

Of course, Tim was still in diapers then. But he's come so far, that precocious little rascal! Let's see what he has for us this time!

Tim gets an easy one!

Holy Guacamole! This is FINALLY an easy one! Robocopy makes this super easy *and* it plays well with leap years. I feel like it is my birthday and I can finally get out of these diapers.

PS C:\> robocopy \some\dir \new\dir /MINLAD (Get-Date).DayOfYear /MINAGE (Get-Date).DayOfYear /MOV

Simply specify the source and destination directories and use /MOV to move the files. MINLAD will ignore files that have been accessed in the past X days (LAD = Last Access Date), and MINAGE does the same based on the creation date. All we need is the number of days since the beginning of the year. Fortunately, getting that number is super easy in PowerShell (I have no pity for Hal).

All Date objects have the property DayOfYear which is (surprise, surprise) the number of days since the beginning of the year (Get-Member will show all the available properties and methods of an object). All we need is the current date, which we get Get-Date.

DONE! That's all folks! You can go home now. I know you expected a long complicated command, but we don't have one here. However, if you feel that you need to read more you can go back and read the episodes where we cover some other options available with robocopy.

This command is so easy, simple, and short I could even fit it into a tweet!