Episode #112: Sensitivity Training

Tim get's PO'ed:

Jason Jarvis sends a question via the Post Office:

I need to conduct folder permissions audit on folders with specific names and then check to make sure that a specific group is explicitly denied.

I produced some PowerShell code to do that and was fairly happy:

PS C:\> Get-Childitem -path S: -recurse -include *classified*,*sensitive*,restricted*
-exclude *notsensitive* | where { $_.Attributes -match "d" } | Get-Acl | where {
$_.AccessToString -notmatch "DOMAIN\\GROUP" } | select PSPath, AccessToString |
export-csv outputfilename.csv

This works a treat until I realized that there are 140 remote locations where I don't have PowerShell installed.

It is a nice little bit of fu. The command starts off by getting a recursive directory listing of the S drive and looking for objects with classified, sensitive, or restricted in the name, but not containing the word notsensitive. The objects are filtered for directories by using Where-Object and checking the Attributes of the object. I usually filter for directories by checking the PSIsContainer property since it is faster, but it isn't a huge difference (~ 10%).

The objects are piped into Get-Acl and subsequently into another Where-Object filter, which only passes objects that do *not* have permissions defined for our group. Finally, the results are exported to a CSV.

The AccessToString property converts the ACL to a nice string which contains the principal, allow or deny, and the right (full, read, write, etc). It looks like this:

AccessToString: Domain\Group Deny FullControl
                BUILTIN\Administrators Allow  FullControl
                NT AUTHORITY\SYSTEM Allow  FullControl
                BUILTIN\Users Allow  ReadAndExecute, Synchronize
                NT AUTHORITY\Authenticated Users Allow  Modify, Synchronize

We can do a few things to shorten the original command by using aliases, but the biggest suggestion would be to change the filter on AccessToString. We want to make sure the group in question isn't just explicitly defined, but it also denied. Here is the shortened bit with the fix.

PS C:\> ls S: -r -include *classified*,*sensitive*,restricted* -exclude *notsensitive* |
? { $_.PSIsContainer } | Get-Acl | ? { $_.AccessToString -notmatch "DOMAIN\\GROUP Deny" } |
select PSPath, AccessToString | export-csv outputfilename.csv

Now how do we do this in cmd.exe?

Let's build up to the final command, and start off by getting a list of all the directories with the matching names.

C:\> dir S: /a:d /s /b | findstr /i "classified$ sensitive$ restricted$" |
findstr /i /v "notsensitive$"

We use dir with a few switches to get directories on the S drive. The /a:d switch just returns directories, the /s switch enables recursion, and the /b switch gives us the full path and not the header info. We then use Findstr to filter for directories with our keywords. The spaces in the search string are treated as logical ORs. Finally, we use Find with the /v switch to filter out anything that contains the word notsensitive. Edit: We also use the /i switch to make the search case insensitive.

To parse it, we'll have to use our friend, the For loop.

C:\> for /F %i in ('dir S: /a:d /s /b ^| findstr /i "classified$ sensitive$ restricted$" ^|
findstr /v /i "notsensitive$"') do @echo %i

S:\sensitive
S:\classified
...

Using the For loop we have the variable %i that contains our directory, but how do we check permissions? We use cacls, and to do our searching we have to know what the output looks like.

C:\>  cacls S:\
S:\sensitive domain\group:(OI)(CI)N
             BUILTIN\Administrators:(ID)F
             BUILTIN\Administrators:(OI)(CI)(IO)(ID)F
             NT AUTHORITY\SYSTEM:(ID)F
             ...

The N at the end of the line means No Access (or access denied). So we need to search for the group name and N at the end of the line. We can do it like this:

C:\>  cacls S:\ | findstr /i "domain\\group:.*)N"
S:\sensitive domain\group:(OI)(CI)N

The FindStr command uses a regular expression to perform our search. Now, let's put it all together:

C:\> for /F %i in ('dir /a:d /s /b ^| findstr /i "classified$ sensitive$ restricted$" ^|
  findstr /i /v "notsensitive$"') do @cacls sensitive | findstr /i 
  "domain\\group:.*)N domain\\group:.*DENY" > nul || @echo %i >> outputfilename.txt

Edit, the output of cacls seems to depends on the version of windows. The regular expression used with findstr needs to catch both "domain\group:(OI)(CI)N" and "FN2131\user123:(OI)(CI)(DENY)".

We use a little trick with the Logical OR (||). If the first command is successful, then the second command won't be executed. Likewise, if the first command is not successful, then the second command executes. So, if there is no output from our cacls/find, meaning nothing was found, then it executes the next command. In our case, the second command writes the variable %i to a file. We are then left with a file that contains all of the directories.

Let's see if Hal can wade his way through his sensitive and non-sensitive sides.

Hal gets wacky

Hey, man, I am one sensitive m------f-----!

There are probably lots of ways to solve this challenge in the Unix shell, but I decided to try and answer the question using only a single find command. The result looks quite complicated, but it's really not too bad:

find . -type d \
   \( \( -group users -a -perm -0010 \) -o -perm -0001 -o \( -prune -false \) \) \
   \( \( -name '*classified*' -o -name '*restricted*' -o -name '*sensitive*' \)  \
      -a ! -name '*nonsensitive*' \) \
   -ls

If you take this command apart into pieces, you'll see that there are four major chunks:

1. First there's the basic "find . -type d" bit that descends from the current directory looking for any sub-directories.



2. Next we check the permissions on the directory. For the directory to be accessible, the directory must be owned by the group in question (I'm using group "users" in this example) AND the group execute bit must be set ("-group users -a -perm -0010"). The directory can also be accessed if the execute bit is set for "everybody" ("-perm -0001").
   
   If neither of these conditions is true, then there's no point in checking any of the directories beneath the inaccessible directory, because members of our group won't be able to reach any of the subdirectories below this point. So we use "-prune" to prevent find from going further. However, "-prune" always returns "true", which means find would actually output information about this directory. But we don't want that because we've already determined that this directory is not accessible. So the idiom "-prune -false" prunes the directory structure but returns false so that we don't output information about the top-level directory.



3. The next complicated looking logical operation filters out the directory names. Per Jason's request, we're matching directory names that contain "classified", "restricted", or "sensitive" and then explicitly excluding directories with the word "nonsensitive".



4. Finally we use the "-ls" operator to output a detailed listing of our matching directories. If you really need a CSV file, I'd recommend using GNU find so that you can use the "-printf" operator to dump the output in whatever format you require. But I'll leave that as an exercise for the reader.

It's interesting to note that the order of the two main logical blocks is important-- you'll get the wrong answer if you check names before permissions. Consider what would happen if we had a directory called "foo" that was mode 700 and owned by root that contained a subdirectory "sensitive" that was mode 755. In my example above, the find command hits the "foo" directory, determines that it is inaccessible, and immediately prunes the search at that point.

If we were to check names first, however, then the directory name "foo" would not match any of our keywords. So that clause would evaluate to "false" and the permissions on the directory "foo" would never be checked. We'd never get to the "-prune" statement! This means that find would happily descend into the "foo" directory and erroneously report that "foo/sensitive" was accessible, when in fact it's not because "foo" is closed.

Thanks for your interesting problem this week, Jason! Keep those cards and letters coming!

Episode #111: What's in a Name Server?

Hal is feeling puckish

Loyal reader Matt Raspberry writes:

Recently I needed to ensure that all the configured dns servers of a server had reverse DNS setup correctly:

for i in `awk '/^nameserver/ {print $2}' /etc/resolv.conf`; do host $i; done;

Honestly that's probably the way I would have solved the same challenge. However, once I consulted my command-line muse, the following alternate solution occurred to me:

awk '/^nameserver/ {print $2}' /etc/resolv.conf | xargs -L 1 host

Normally the xargs command would take as many input lines as possible and stack them all up as arguments in a single host command. But host doesn't react well to multiple arguments and you'll get erroneous output.

But "xargs -L <n>" lets you specify a maximum number of input lines to use for each command invocation. In the example above, we're telling xargs to take one line at a time and call host on each individual input. So basically we're swapping out Matt's original for loop for an xargs command. This saves a little bit of typing, but it's in no way a real optimization.

Honestly, I picked this one out of the mailbag for two reasons. First, Matt did all the work for me this week (thanks Matt!), which is nice because I'm going on vacation. Second, it's going to be a lot more work for Tim. Let's watch the carnage, shall we? What fools these mortals be!

Tim brings the carnage:

This isn't pretty, but WMI rarely (read never) is. WMI is the only way to get information on the DNS servers in use. Ugh! But let's start off easy and get a list of the DNS servers in use.

PS C:\> PS C:\> gwmi Win32_NetworkAdapterConfiguration  | ? { $_.DNSServerSearchOrder } |
  select DNSServerSearchOrder

DNSServerSearchOrder
--------------------
{208.67.222.222, 208.67.220.220}

The Get-WmiObject cmdlet (alias gwmi) is used to get the configuration of all network adapters. The results are piped into the Where-Object cmdlet (alias ?) to filter for objects where the DNSServerSearchOrder property has a value. Select-Object (alias select) is used to return only the property we want.

One problem, the result is an array of IP Addresses, but, we can easily deal with it by using the ExpandProperty switch with the Select-Object cmdlet.

PS C:\> gwmi Win32_NetworkAdapterConfiguration  | ? { $_.DNSServerSearchOrder } |
  select -expand DNSServerSearchOrder

208.67.222.222
208.67.220.220

The ExpandProperty switch will explode the property in question, so we get two strings instead of one array of strings. Doesn't this sound like the same thing? Not if you think of how the objects are going to be sent down the pipeline. We need individual strings to do our lookup, not an array of strings.

Now that we have the IP Addresses of our DNS serves, we can do the lookup.

PS C:\> gwmi Win32_NetworkAdapterConfiguration  | ? { $_.DNSServerSearchOrder } |
  select -expand DNSServerSearchOrder | % { nslookup $_ }

So there you have it, carnage. But what about the carnage we can cause with old school cmd.exe? Time to go Freddy Krueger on this thing.

Let's get the DNS servers.

C:\> wmic nicconfig get dnsserversearchorder | find ","

{"208.67.222.222", "208.67.220.220"}

It returns a pseudo array of the DNS servers. That part is easy, but as you all know, parsing in windows is not easy. When all you have is a hammer crappy For loop, the world starts to look like a nail.

Here is the most simplistic way to parse this:

C:\> for /f "tokens=1-3 delims={}, " %a in
  ('wmic nicconfig get dnsserversearchor ^| find ","')
  do @nslookup %a & @nslookup %b 2>gt;nul & @nslookup %c 2>gt;nul

We use the handy-dandy For loop to split the output. The results are split using the delimiters left brace ({), right brace (}), comma, and space. We only return the first three tokens, since more is pretty rare in Windows. We then output the results of each nslookup and send any errors to nul.

Without some really really really really ugly cmd.exe fu, that no one would actually use, we can't parse the list and guarantee that we get the full list. In our case, we don't know if there was a 4th DNS server configured. We could change the number of tokens and use all 26 variables (a-z), but what if there were a 27th server? It can be done. In fact, Ed has done some crazy fu like that, but we are talking Quentin Tarantino level carnage, and that is just ridiculous.

Episode #110: Insert Title Here

In this corner, Tim, the disputed PowerShell Heavyweight Champion of the World:

What is in a title? Titles are important and they can be really useful. An aptly named window can be useful for screen shots, managing multiple sessions, and for basic organization. The default PowerShell title of Administrator: C:\WINDOWS\system32\WindowsPowerShell\v1.0\powershell.exe can be really useless. So how do we rename the window to something more useful?

PS C:\> (Get-Host).UI.RawUI.WindowTitle = "Shell to Take over the World"

Boom! Now we have a window that is distinct from our other windows, and clearly displays its use for world domination. So how does the command work?

Get-Host returns an object that represents the host. By accessing the UI, RawUI, and WindowTitle property hierarchy we can modify the title. Not straight-forward, but rather simple. As for the actual plans to take over the world, those aren't simple, nor are they for sale.

And in the red trunks, Hal is undisputedly heavy

Oh sure, namby-pamby Powershell can have an elegant API for changing the window title, but in Unix we prefer things a little more raw:

$ echo -e "\e]0;im in ur windoz, changin ur titlz\a"

Yep, that's the highly intuitive sequence "<Esc>-right bracket-zero-semicolon-<title>-<BEL>" to set your window title to "<title>". Since this obviously doesn't roll trippingly off the fingers, I generally create the following shell function in my .bashrc:

function ct { echo -e "\e]0;$*\a"; }

See the "$*" in the middle of all the line noise? That takes all of the "arguments" to the function (anything we type on the command line after the function name, "ct") and uses them as the title for your window. So now you can change titles in your windows by typing:

$ ct im in ur windoz, changin ur titlz

And that's a whole lot nicer.

Another cool place to use this escape sequence is in your shell prompt. Yes, I know we've done several Episodes at this point that deal with wacky shell prompt games, but this is a fun hack:

$ export PS1='\e]0;\u@\h: $PWD\a\h\$ '
elk$ 

Here we've embedded our title-setting escape sequence into the PS1 variable that sets up our command-line prompt. We're using it to set the window title to "\u@\h: $PWD", where \u is expanded by the shell as our username, \h is the unqualified hostname, and $PWD is our current working directory. So now we can easily look at the window titles and know exactly who and where we are. This is useful for picking out the one window we want if we have a stack of minimized terminal windows. Notice we also have "\h\$ " after the escape sequence which will actually set the visible shell prompt we see in our terminal window.

Whee!

Mike Cardosa slips something a little extra into his gloves, cmd.exe

Only a true Unix guy would use the word "elegant" to describe the PowerShell method for setting the window title. I do not think that word means what you think that it means.

It turns out that the cmd.exe developers failed to achieve the level of obscurity that we've come to expect when dealing with this humble shell.

Ladies and gentlemen, introducing quite possibly the most intuitive command available in cmd.exe:

C:\> title h@xor at work

That's it! We've managed to change the window title using the simple 'title' command. Just feed it any string you'd like and you are good to go. Including environment variables is also startlingly simple:

C:\> title netcat listener on %computername%

Look at that. The window title - including environment variable - is set without using a single cryptic command line switch or for loop.

Fortunately for sys admins (and certain bloggers), this sort of simplicity is the exception rather than the rule, so the world still needs those with a mastery of obscure commands.

Episode #109: The $PATH Less Taken

Hal is in a reflective mood:

I was perusing SHELLdorado the other day and came across a tip from glong-at-openwave-dot-com for printing the elements of your $PATH with each directory on a separate line:

$ IFS=':'; for i in $PATH; do echo $i; done
/bin
/usr/bin
/usr/X11R6/bin
/usr/local/bin
/sbin
/usr/sbin
/usr/local/sbin
/usr/games
/home/hal/bin

It's an interesting example of using IFS to break up your input on something other than whitespace, but in this particular case there are obviously more terse ways to accomplish the same thing:

$ echo $PATH | sed 's/:/\n/g'
/bin
/usr/bin
/usr/X11R6/bin
/usr/local/bin
/sbin
/usr/sbin
/usr/local/sbin
/usr/games
/home/hal/bin

Or to be even more terse:

$ echo $PATH | tr : \\n
/bin
...

The important piece of advice here is that if you're just exchanging one character for another (or even one set of characters for another), then tr is probably the quickest way for you to accomplish your mission. sed is obviously a superset of this functionality, but you have to use a more complex operator to do the same thing.

I think this example also nicely brings home the fact that both sed and tr (as well as awk and many other Unix input-processing primitives) are implicit loops over their input. The for loop from the first example has been subsumed by the functionality of sed and tr, but you still pay the performance cost of reading the entire input. So if you have a multi-stage pipeline that has several sed, tr, and/or awk commands in it, you might try to look at ways to combine operations in order to reduce the number of times you have to read your input.

Tim takes the high road:

In PowerShell we can do the same thing, and just as easily.

PS C:\> $env:path -replace ";","`n"
C:\WINDOWS\system32
C:\WINDOWS
C:\WINDOWS\System32\Wbem
C:\WINDOWS\system32\WindowsPowerShell\v1.0

Just as the registry and the filesystem have a provider, environment variables have their own too. The providers allow PowerShell to access the objects using a common set of cmdlets, like Get-ChildItem (alias gci, ls, and dir). Just as we can list the contents of a drive by typing gci c: we can list all the environment variables with gci env:.

To access a specific variable we use $env: followed by the variable name.

PS C:\> $env:path
C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\system32\WindowsPowerShell\v1.0

Once we get the variable's value, we just replace the semicolon with the new line character (`n) by using the replace operator.

Bonus! Guest CMD.EXE solution:

We've all been missing Ed's CMD.EXE wisdom, but loyal reader Vince has moved in before the body is even cold and sent us this little tidbit:

C:\> for %i in ("%PATH:;=" "%") do @echo %i
"C:\Program Files\Perl\site\bin"
"C:\Program Files\Perl\bin"
"C:\WINDOWS\system32"
"C:\WINDOWS"

Thanks, Vince!

Episode #108: Acess List Listing

Hal's turn in the mailbag

Loyal reader Rick Miner sent us an interesting challenge recently. He's got several dozen Cisco IOS and PIX configuration files containing access-list rules. He'd like to have an easy way to audit the access-lists across all the files and see which rules are common to all files and where rules might be missing from some files.

Basically we're looking through the files for lines that start with "access-list", like this one:

access-list 1 deny   any log

However, access-lists can also contain comment lines (indicated by the keyword "remark"), and we don't care about these:

access-list 1 remark This is a comment

We also want to be careful to ignore any extra spaces that may have been introduced for readability. So the following two lines should be treated as the same:

access-list 1 deny   any log
access-list 1 deny any log

Rick sent us a sample access-list, which he'd sanitized with generic IP addresses, etc. I created a directory with a few slightly modified versions of his original sample-- giving me 5 different files to test with.

Now I love challenges like this, because they always allow me to construct some really fun pipelines. Here's my solution:

$ grep -h ^access-list rules0* | grep -v remark | sed 's/  */ /g' | 
    sort | uniq -c | sort -nr
      5 access-list 4 permit any
...
      4 access-list 1 deny any log
...

First I use grep to pull the access-list lines out of my sample files (named rules01, rules02, ...). Normally when you run grep against multiple files it will prepend the file name to each matching line, but I don't want that because I plan on feeding the output to sort and uniq later. So I use the "-h" option with grep to suppress the file names.

Next we have a little cleanup action. I use a second grep command to strip out all the "remark" lines. The output then goes to sed to replace instances of multiple spaces with a single space. Note that the sed substitution is "s/<space><space>*/<space>/g", though it's a little difficult to read in this format.

Finally we have to process our output to answer Rick's question. We sort the lines and then use "uniq -c" to count the number of occurrences of each rule. The second sort gives us a descending numeric sort the lines using the number of instances of each rule as the sort criteria. Since I'm working with five sample files, rules like "access-list 4 permit any" must appear in each file (assuming no duplicate rules, which seems unlikely). On the other hand, "access-list 1 deny any log" appears to be missing from one file.

But which file is our rule missing from? One way to answer this question is to look for the files where the rule is present:

$ grep -l 'access-list 1 deny any log' rules0*'

Wait a minute! What just happened here? We should have gotten four matching files! Well remember how we canonicalized the lines by converting multiple spaces to a single space? Let's try making our rule a bit more flexible:

$ grep -l 'access-list *1 *deny *any *log' rules0*'
rules01
rules02
rules03
rules05

That's better! We use the stars to match any number of spaces and we find our rules. "grep -l" (that's an "el" not a "one") means just display the matching file names and not the matching lines so that we can easily see that "rules04" is the file missing the rule.

But what if you were Rick with dozens of files to sort through. It wouldn't necessarily be clear which files weren't included in the output from our grep command. It would be nicer if we could output the names of the files that were missing the rule, rather than listing the files that included the rule. Easier done than said:

$ sort <(ls) <(grep -l 'access-list *1 *deny *any *log' rules0*) | uniq -u
rules04

"<(...)" is an output substitution that allows you to insert the output of a command in a spot where you would normally expect to use a filename. Here I'm using sort to merge the output of ls, which gives me a list of all files in the directory, with our previous command for selecting the files that contain the rule we're interested in. "uniq -u" gives you the lines that only appear once in the output (the unique lines). Of course these are the files that appear in the ls output but which are not matched by our grep expression, and thus they're the files that don't contain the rule that we're looking for. And that's the answer we wanted.

You can do so much with sort and uniq on the Unix command line. They're some of my absolute favorite utilities. I've laid off the sort and uniq action because Windows CMD.EXE didn't have anything like them and it always made Ed grumpy when I pulled them out of my tool chest. But now that we've murdered Ed and buried him in a shallow grave out backbid farewell to Ed, I get to bring out more of my favorites. Still, I fear this may be another one of those "character building" Episodes for Tim. Let's watch, shall we?

Tim's got plenty of character:

Alright Hal, you used to push Ed around, but you are going to have a slightly tougher time pushing me around. And not just because I wear sticky shoes.

This is going to be easy. Ready to watch, Hal?

PS C:\> ls rules* | ? { -not (Select-String "access-list *1 *deny *any *log" $_) }

    Directory: C:\temp

Mode                LastWriteTime     Length Name
----                -------------     ------ ----
-a---         8/16/2010   9:44 PM       1409 rules05.txt

Files whose name begin with "rules" are piped into our filter. The Where-Object filter (alias ?) uses a logical Not in conjunction with Select-String to find files that do not contain our search string. The search string used is the same as that used by Hal.

Now to crank it up a few notches...

But what if we have a file containing our gold standard, and we wanted to compare it against all of our config files to find ones that don't comply with our standard. Your wish is my command (unless your with involves a water buffalo, a nine iron, and some peanut butter).

PS C:\> cat gold.txt | select-string -NotMatch '^ *$' | % { $_ -replace "\s+", "\s+" } | % {
  $a = $_;
  ls rules* | ? { -not (select-string $a $_) }
} | get-unique

    Directory: C:\temp

Mode                LastWriteTime     Length Name
----                -------------     ------ ----
-a---         8/16/2010   9:44 PM       1409 rules02.txt
-a---         8/16/2010   9:44 PM       1337 rules05.txt

In the command above, the first line of our command gets each non-blank line of our gold config, and changes and spaces into \s+ for use in our search string. The \s+ is the regular expression equivalent of "one or more spaces". Now that we have generated our search string, lets search each file like we did earlier. Finally, we use the Get-Unique cmdlet to remove duplicates.

Hal, you may have buried Ed, but you haven't killed me off...yet.

Episode #107: Email for Natural File Enhancement

Tim pretends he is Cliff Claven:

Another dive into the mailbag this week. Jonathon English writes in asking how to find files that are larger than 10Mb, determine the owner, and send him/her an email.

Let's start off with the easy part, getting the owner of files bigger than 10MB.

PS C:\> ls -r | ? { $_.Length -ge 10485760 } | get-acl | select path, owner

Path                                                      Owner
----                                                      -----
Microsoft.PowerShell.Core\FileSystem::C:\bigfile.txt      DOMAIN\user1
Microsoft.PowerShell.Core\FileSystem::C:\biggerfile.txt   DOMAIN\user2
Microsoft.PowerShell.Core\FileSystem::C:\biggestfile.txt  DOMAIN\user3

The command starts off with the Get-ChildObject (Alias dir, ls). The -r[ecurse] option is used to search all subdirectories. Files are filtered with Where-Object cmdlet (alias ?) to return only files that are 10MB or larger. The results are piped into Get-Acl, which gets the owner. Finally, the attributes owner and path are selected. As you may notice, the path looks a little funny. The Get-Acl cmdlet works with multiple providers, and the path returned by the cmdlet includes the provider.

The path is a bit ugly, so it would be good to clean it up - but how?" There are a few ways to do it. Here is the easiest and the quickest.

PC C:\> ls -r | ? { $_.Length -ge 10485760 } |
  select FullName, @{Name="Owner";Expression={(Get-Acl $_).owner }}

FullName            Owner
--------            -----
C:\bigfile.txt      DOMAIN\user1
C:\biggerfile.txt   DOMAIN\user2
C:\biggestfile.txt  DOMAIN\user3

Here the Select-Object cmdlet (alias select) is used with a calculated property to get the file's owner.

So we have the info, now to email it. Some of this will depend on the security settings used by your mail server(s) and you may have to customize this yourself. Also, we may cross into scriptland and violate rule #1. Ed isn't around, so no one tell him.

PS C:\> $SmtpClient = new-object system.net.mail.smtpClient
PS C:\> $SmtpClient.Host = "smtpserver.domain.com"
PS C:\> ls -r | ? { $_.Length -ge 10485760 } | select FullName, @{Name="Owner";Expression={(Get-Acl $_).owner } } | % {
  $MailMessage = New-Object system.net.mail.mailmessage
  $mailmessage.from = "big brother"
  $mailmessage.To.add($_.Owner -replace "DOMAIN\","" + "mydomain.com")
  $mailmessage.Subject = "Your big file won't copy"
  $mailmessage.IsBodyHtml = 1
  $mailmessage.Body = "This file is too big and won't be copied:" + $_.Path
  $smtpclient.Credentials = [System.Net.CredentialCache]::DefaultNetworkCredentials
  $smtpclient.Send($mailmessage)
}

These are the same commands as used above. The only difference is we have ported some .NET code to send the email. In the code above, I assumed that the username was part of the email address or alias. If that isn't the case, we would have to lookup the user's email address. To easily lookup the email address we would need the Exchange snap-in. (Ok, not a default install, but pretty common. And if I am going to violate one rule, I may as well break them all).

PS C:\> Get-User tim | select WindowsEmailAddress

WindowsEmailAddress
-------------------
Tim.Medin@commandlinekungfu.com

Here is the rewritten command using the Exchange cmdlet.

PS C:\> ls -r \\sharepoint\data\it\general | ? { $_.Length -ge 10485760 } |
  select FullName, @{Name="Owner";Expression={(Get-User (Get-Acl $_).owner).WindowsEmailAddress } }

FullName            Owner
--------            -----
C:\bigfile.txt      user1@domain.com
C:\biggerfile.txt   user2@domain.com
C:\biggestfile.txt  user3@domain.com

Here is how it would look with our...ahem...script...

PS C:\> $SmtpClient = new-object system.net.mail.smtpClient
PS C:\> $SmtpClient.Host = "smtpserver.domain.com"
PS C:\> ls -r \\sharepoint\data\it\general | ? { $_.Length -ge 10485760 }
  | select FullName, @{Name="Owner";Expression={(Get-User (Get-Acl $_).owner).WindowsEmailAddress } } % {
  $MailMessage = New-Object system.net.mail.mailmessage
  $mailmessage.from = "big brother"
  $mailmessage.To.add($_.Owner)
  $mailmessage.Subject = "Your big file won't copy"
  $mailmessage.IsBodyHtml = 1
  $mailmessage.Body = "This file is too big and won't be copied:" + $_.Path
  $smtpclient.Credentials = [System.Net.CredentialCache]::DefaultNetworkCredentials
  $smtpclient.Send($mailmessage)
}

So that is how it looks. Let's see if Hal is manly enough to bend some rules.

Hal thinks hard

Wait, the guy who brought up "Natural Enhancement" is questioning my manhood? Hey, sport, in Unix-land we can actually send email from the command line without resorting to writing a script.

Actually, I did end up relaxing one of my own personal rules for this week's Episode. Generally I try very hard to come up with a solution that's a single command-line. It may be a really long pipeline and spawn lots of subshells, but you should only have to hit the "Go" button once. But this time, it was frankly easier to come up with a clear solution that uses two primary command lines. Sue me.

First is the task of identifying large files and sorting them by user. I'm going to do this work in a temporary directory just to make my life easier:

# mkdir /tmp/emails
# cd /tmp/emails
# find / -size +10000000 -ls | awk '{ print $5, $0 }' | 
    while read user line; do echo $line >>$user; done

The find command uses the same expression we saw back in Episode 102-- find all files whose size is greater than 10,000,000 (bytes by default). I've added the "-ls" action so that instead of just the file names we get a detailed listing ala "ls -dils". This output gets piped into awk, where I pull out the username in field number five and re-output the line with the username duplicated at the front of the line. The modified lines then go into the while loop where I pull the username off the front of each line and append the remainder of the line (the "find ... -ls" output) to a file named for the user. So at the end of the command, I've got the detailed listing information for all the big files on the system broken out into separate files by user.

That's actually the hard part. Sending email to the users is easy (assuming outgoing email on your machine is configured properly):

# for user in *; do cat $user | mailx -s 'Large File Report' $user; done

All we need to do is shove the contents of each file into the standard input of a mailx process with the username specified as the recipient. The "-s" option even lets us specify a subject line for the message.

Note that I used "cat ... | mailx ..." here rather than "mailx ... <$user" because I wanted to make it easier to add a canned message to the top of the report in addition to the find output. For example, if you had the boilerplate in /tmp/mymsg, then you would use a command line like:

# for user in *; do cat /tmp/mymsg $user | mailx -s 'Large File Report' $user; done

I actually am feeling a little... well let's just say "impotent"... for having to do this in two commands. There's probably some crazy output redirection stunt I could have done to pack this into a single command line, but my solution accomplishes the task and is straightforward to type and understand.

If anybody feels man enough to accept the challenge of coming up with a single command version, mail your answers to suggestions[at]commandlinekungfu[dot]com. I will choose the "best" solutions using some completely arbitrary set of criteria that I haven't developed yet and post them here on the blog. Prizes will probably not be awarded.

Episode #106: Epoch FAIL!

Hal goes to camp:

I recently had the opportunity to teach at the US Cyber Challenge Camp. The campers were awesome and we had tons of geeky fun.

Among other topics, I covered securing Linux systems, including some work with SELinux. One of the problems you run into when dealing with SELinux is reading the SELinux-related messages in the system audit.log file. Here's a sample entry:

# egrep '^type=(AVC|SELINUX)' /var/log/audit/audit.log | tail -1
type=AVC msg=audit(1279670222.302:199): avc:  denied  { name_bind } for  pid=18232 comm="httpd"
src=31337 scontext=user_u:system_r:httpd_t:s0 tcontext=system_u:object_r:port_t:s0 tclass=tcp_socket

The audit.log files live under /var/log/audit, and the SELinux entries start with either "type=AVC" or "type=SELINUX". I use "tail -1" above to pull out just the last line of output.

The first thing you might notice is that, unlike most Unix log files, the audit.log lines don't appear to contain a timestamp. In fact, there is a timestamp on each line-- it's just in Unix epoch time, or seconds since Jan 1, 1970. The big number inside the "msg=audit(...)" section is the epoch time value (so "1279670222" in the example above). Back in Episode #77, Adrian Shaw showed us how to convert this value into a human-readable timestamp:

# date -d @1279670222
Tue Jul 20 16:57:02 PDT 2010

One of the campers challenged me to come up with some Command-Line Kung Fu to automatically translate the epoch time values and rewrite each line with the human-readable time format at the beginning of the line. It took me a little while to figure out a solution, but I eventually ended up with:

# egrep '^type=(AVC|SELINUX)' /var/log/audit/audit.log |
  while read line; do
     time=`echo $line | sed 's/.*audit(\([0-9]*\).*/\1/'`;
     echo `date -d @$time` $line;
  done
[...]
Tue Jul 20 16:57:02 PDT 2010 type=AVC msg=audit(1279670222.302:199): avc: denied { name_bind }
for pid=18232 comm="httpd" src=31337 scontext=user_u:system_r:httpd_t:s0 
tcontext=system_u:object_r:port_t:s0 tclass=tcp_socket

I decided to use a while loop to read the input line-by-line because it gave me more flexibility to run multiple commands against each line. First I use sed to filter out the epoch time value and store it in the variable $time. The sed expression is a substitution. The LHS matches the entire line, but specifically matches the epoch time value as a sub-expression which we delimit with "\(...\)". Read the regex on the LHS as "any amount of stuff (`.*'), then `audit(' followed by zero or more digits (`[0-9]*'), followed by a bunch more stuff (`.*')". We replace everything we matched on the LHS with the epoch time value we matched in the subexpression ("\1"). Thus the output of the sed expression will be just the epoch time value.

Once we extract the epoch time value, we can call "date -d @$time". All we have to do at that point is output the result of the date command followed by the original input line which has been hanging out in $line all this time. Whee!

While this is really useful on Linux systems, there's no real equivalent in the Windows environment. But I sent Tim a copy of a Linux audit.log file to see what he could do with it in Powershell. This is a situation you might legitimately have to deal with if you were analyzing a Linux system using a Windows forensics tool. Let's see how good Tim's parsing skills are, shall we?

Tim pitches a tent:

I find it rather ironic that the Security Enhanced Linux's audit log seems to be lacking an enhancement that would be useful for auditing. Now, let's dive into some level 1 parsing!

PS C:\> Select-String "[0-9]{10}" audit.log | select @{Name="Epoch";
  Expression={$_.Matches[0].Value}}, line | fl

Epoch : 1279670222
Line  : type=AVC msg=audit(1279670222.302:199): avc: denied { name_bind } for
        pid=18232 comm="httpd" src=            31337 scontext=user_u:system_r:
        httpd_t:s0 tcontext=system_u:object_r:port_t:s0 tclass=tcp_socket

We start off using Select-String to both read the file and find the epoch timestamp. The results are piped into Select-Object (alias select), which is used to display the line as well as create a new epoch property.

There are now two properties, the epoch stamp and the line. Let's crank it up a notch and reformat the timestamp. Get ready for level 2 parsing!

PS C:\> Select-String "[0-9]{10}" audit.log | select @{Name="TimeStamp";
  Expression={(Get-Date 1/1/1970).AddSeconds($_.Matches[0].Value)}}, line | fl

TimeStamp : 7/20/2010 11:57:02 PM
Line      : type=AVC msg=audit(1279670222.302:199): avc: denied { name_bind } for
            pid=18232 comm="httpd" src=31337 scontext=user_u:system_r:
            httpd_t:s0 tcontext=system_u:object_r:port_t:s0 tclass=tcp_socket

In Level 2, we do the same thing as Level 1, except we reformat the timestamp. How do we rewrite the timestamp? Epoch time is just the number of seconds since 1/1/1970. So all we need to do is the math.

(Get-Date 1/1/1970).AddSeconds($_.Matches[0].Value)

Ok, so we have the results we are looking for, and we could leave it here. But Hal filtered for lines containing AVC or SELINUX, and we can't let Hal beat us. Get ready for Level 3!

PS C:\> Select-String "[0-9]{10}" audit.log | select @{Name="TimeStamp";
  Expression={(Get-Date 1/1/1970).AddSeconds($_.Matches[0].Value)}}, line |
  ? { $_.Line -match "type=(AVC|SELINUX)" }

Nothing fancy here, just added filtering. One minor problem, Select-String is being used for parsing and then the line is searched again by the Where-Object (alias ?) filter. The redundant searching will slow things down, so let's speed it up. Get ready for Level 4!

PS C:\> Select-String "type=(AVC|SELINUX) msg=audit\(([0-9]+)" audit.log |
  select @{Name="TimeStamp"; Expression={(Get-Date 1/1/1970).AddSeconds($_.Matches[0].Groups[2].Value)}}, line

The regular expression is juiced up to do the filtering and find the timestamp. The juiced regex returns multiple groups and must be accesses differently (as shown above).

Group 0: type=AVC msg=audit(1279670222
Group 1: AVC
Group 2: 1279670222

That is how you parse the audit.log with PowerShell. I hope it wasn't too painful parsing my parsing.