Command Line Kung Fu: Episode #171: Flexibly Finding Firewall Phrases

Old Tim answers an old email

Patrick Hoerter writes in:
I have a large firewall configuration file that I am working with. It comes from that vendor that likes to prepend each product they sell with the same "well defended" name. Each configuration item inside it is multiple lines starting with "edit" and ending with "next". I'm trying to extract only the configuration items that are in some way tied to a specific port, in this case "port10".

Sample Data:

edit "port10"
        set vdom "root"
        set ip 192.168.1.54 255.255.255.248
        set allowaccess ping
        set type physical
        set sample-rate 400
        set description "Other Firewall"
        set alias "fw-outside"
        set sflow-sampler enable
   next
edit "192.168.0.0"
        set subnet 192.168.0.0 255.255.0.0
    next
    edit "10.0.0.0"
        set subnet 10.0.0.0 255.0.0.0
    next
    edit "172.16.0.0"
        set subnet 172.16.0.0 255.240.0.0
    next
  edit "vpn-CandC-1"
        set associated-interface "port10"
        set subnet 10.254.153.0 255.255.255.0
    next
    edit "vpn-CandC-2"
        set associated-interface "port10"
        set subnet 10.254.154.0 255.255.255.0
    next
    edit "vpn-CandC-3"
        set associated-interface "port10"
        set subnet 10.254.155.0 255.255.255.0
    next
   edit 92
        set srcintf "port10"
        set dstintf "port1"
            set srcaddr "vpn-CandC-1" "vpn-CandC-2" "vpn-CandC-3"            
            set dstaddr "all"            
        set action accept
        set schedule "always"
            set service "ANY"            
        set logtraffic enable
    next

Sample Results:

edit "port10"
        set vdom "root"
        set ip 192.168.1.54 255.255.255.248
        set allowaccess ping
        set type physical
        set sample-rate 400
        set description "Other Firewall"
        set alias "fw-outside"
        set sflow-sampler enable
   next
  edit "vpn-CandC-1"
        set associated-interface "port10"
        set subnet 10.254.153.0 255.255.255.0
    next
    edit "vpn-CandC-2"
        set associated-interface "port10"
        set subnet 10.254.154.0 255.255.255.0
    next
    edit "vpn-CandC-3"
        set associated-interface "port10"
        set subnet 10.254.155.0 255.255.255.0
    next
   edit 92
        set srcintf "port10"
        set dstintf "port1"
            set srcaddr "vpn-CandC-1" "vpn-CandC-2" "vpn-CandC-3"            
            set dstaddr "all"            
        set action accept
        set schedule "always"
            set service "ANY"            
        set logtraffic enable
    next

Patrick gave us the full text and the expected output. In short, he wants the text between "edit" and "next" if it contains the text "port10". To begin this task we need to first need get each of the edit/next chunks.

PS C:\> ((cat fw.txt) -join "`n") | select-string "(?s)edit.*?next" -AllMatches | 
 select -ExpandProperty matches

This command will read the entire file fw.txt and combine it into one string. Normally, each line is treated as a separate object, but we are going to join them into a big string using the newline (`n) to join each line. Now that the text is one big string we can use Select-String with a regular expression to find all the matches. The regular expression will find text across line breaks and allows for very flexible searches so we can find our edit/next chunks. Here is a break down of the pieces of the regular expression:

(?s) - Use single line mode where the dot (.) will match any character, including a newline character. This allows us to match text across multiple lines.
edit - the literal text "edit"
.*? - find any text, but be lazy, not greedy. This means it should match the smallest chunks that will match the criteria.
next - literal text next

Now that we have the chunks we use a Where-Object filter (alias ?) to find matching objects to pass down the pipeline.

PS C:\> ((cat .\fw.txt) -join "`n") | select-string "(?s)edit.*?next" -AllMatches | 
 select -ExpandProperty matches | ? { $_.Captures | Select-String "port10" }

Inside the Where-Object filter we can check the Value property to see if it contains the text "port10". The Value property is piped into Select-String to look for the text "port10", and if it contains "port10" it continues down the pipeline, if not, it is dropped.

At this point, we have the objects we want, so all we need to do is display the results by expanding the Value and displaying it again. The expansion means that it just displays the text and no data or metadata associated with the parent object. Here is what the final command looks like.

PS C:\> ((cat .\fw.txt) -join "`n") | select-string "(?s)edit.*?next" -AllMatches | 
 select -ExpandProperty matches | ? { $_.Value | Select-String "port10" } | 
 select -ExpandProperty Value

Not so bad, but I have a feeling it is going to be worse for my friend Hal.

Old Hal uses some old tricks

Oh sure, I know what Tim's thinking here. "It's multi-line matching, and the Unix shell is lousy at that. Hal's in trouble now. Mwhahaha. The Command-Line Kung Fu title will finally be mine! Mine! Do you hear me?!? MINE!"

Uh-huh. Well how about this, old friend:

awk -v RS=next -v ORS=next '/port10/' fw.txt

While we're doing multi-line matching here, the blocks of text have nice regular delimiters. That means I can change the awk "record separator" ("RS") from newline to the string "next" and gobble up entire chunks at a time.

After that, it's smooth sailing. I just use awk's pattern-matching operator to match the "port10" strings. Since I don't have an action defined, "{print}" is assumed and we output the matching blocks of text.

The only tricky part is that I have to remember to change the "output record separator" ("ORS") to be "next". Otherwise, awk will use its default ORS value, which is newline. That would give me output like:

$ awk -v RS=next '/port10/' fw.txt
edit "port10"
        set vdom "root"
        set ip 192.168.1.54 255.255.255.248
        set allowaccess ping
        set type physical
        set sample-rate 400
        set description "Other Firewall"
        set alias "fw-outside"
        set sflow-sampler enable
   

  edit "vpn-CandC-1"
        set associated-interface "port10"
        set subnet 10.254.153.0 255.255.255.0
    

    edit "vpn-CandC-2"
        set associated-interface "port10"
...

The "next" terminators get left out and we get extra lines in the output. But when ORS is set properly, we get exactly what we were after:

$ awk -v RS=next -v ORS=next '/port10/' fw.txt
edit "port10"
        set vdom "root"
        set ip 192.168.1.54 255.255.255.248
        set allowaccess ping
        set type physical
        set sample-rate 400
        set description "Other Firewall"
        set alias "fw-outside"
        set sflow-sampler enable
   next
  edit "vpn-CandC-1"
        set associated-interface "port10"
        set subnet 10.254.153.0 255.255.255.0
    next
    edit "vpn-CandC-2"
        set associated-interface "port10"
...

So that wasn't bad at all. Sorry about that Tim. Maybe next time, old buddy.

Command Line Kung Fu

Pages

Tuesday, October 8, 2013

Episode #171: Flexibly Finding Firewall Phrases