Tuesday, October 5, 2010

Episode #115: Shadowy Directories

[A PLEA FROM THE BLOGGERS: Hey, if you think it's easy coming up with over 100 ideas for Command-Line Kung Fu Episodes... we really, really want to hear from you! Send us you ideas for things you'd like to see covered in future Episodes! Because we're lazy and we need you to program the blog for us we're running out of ideas here...]

Hal's got a lot on his mind

Lately I've been thinking a lot about directories.

Loyal reader Josh Olson wrote in with an idea for an Episode that involved making a copy of one directory hierarchy under a different directory. The idea is that you just copy the directory structure-- but not any of the files-- from the original directory.

Josh's solution involved a "for" loop, but I can go that one better with a little xargs action:

$ cd /some/path/to/source/dir
$ find * -type d -print0 | (cd /path/to/dest/dir; xargs -0 mkdir)

In the above example we're taking the output from find and piping it into a subshell that first moves over to our target directory and then invokes mkdir via xargs to create the parallel directory structure. In many ways this is similar to the traditional "tar cp" idiom that I mentioned back in Episode #73, except that here I'm just making directories, not copying the directory contents.

Of course, my solution here involves "find ... -print0" piped into "xargs -0", which is great if you live an environment that supports these options for null-terminated I/O. But this is not portable across all Unix-like operating systems. If you want portability, you need to go "old school":

$ find * -depth -type d | cpio -pd /path/to/dest/dir
0 blocks

Yes, that's right. I'm busting out the old cpio command to amaze and terrify you. In this case I'm using "-p" ("pass mode"), which reads in a list of files and/or directories from the standard input and recreates the same file/directory structure under the specified destination directory. The "-d" option tells cpio to create directories as necessary. As a bonus, cpio preserves the original file ownerships and permissions during the copy process.

And this can actually be a problem-- for example when you're copying a directory which has read-only permissions for you. That's why I've added the "-depth" option to the find command, so that cpio is told about the "deepest" objects in the tree first. With "-d", it will make the directory structure to hold these objects and copy them into place. However, cpio will not set the restrictive permissions on the parent directory until it reads the parent directory name in the input.

See what happens when I try the above command without the "-depth" option:

$ find * -type d -ls
656249 4 dr-xr-xr-x 5 hal hal 4096 Oct 2 09:26 read-only
656253 4 drwxr-xr-x 2 hal hal 4096 Oct 2 09:26 read-only/dir3
656251 4 drwxr-xr-x 2 hal hal 4096 Oct 2 09:26 read-only/dir1
656252 4 drwxr-xr-x 2 hal hal 4096 Oct 2 09:26 read-only/dir2
$ find * -type d | cpio -pd ../dest
cpio: ../dest/read-only/dir3: Cannot stat: Permission denied
cpio: ../dest/read-only/dir1: Cannot stat: Permission denied
cpio: ../dest/read-only/dir2: Cannot stat: Permission denied
0 blocks
$ find ../dest/* -type d -ls
656254 4 dr-xr-xr-x 2 hal hal 4096 Oct 2 11:05 ../dest/read-only

Without "-depth", the directory "read-only" appears first in the output before its sub-directories. So the cpio command makes this directory first and sets it to be mode 555, just like the original directory. But then when cpio goes to make the subdirectories that come next in the find output, it doesn't have the write permissions it needs and it fails. So the moral of the story here is always use "find -depth ..." when piping your input into cpio.

I'm curious to see what Tim's got for us this week. It's possible that this is one of those cases where Windows actually makes this task easier to do than in the Unix shell...

Tim's been on the road

Finally, an episode that is really easy in Windows. We can do this with XCopy, which stands for EXTREEEEEEME COPY! Actually, it stands for eXtended Copy, but for all intents and purposes it is EXTREEEEEEME (because it is really easy)!

C:\> xcopy originaldir mynewdir /T /E /I


The /T option creates the directory structure and does not copy the files, but it does not copy empty directories or subdirectories. To copy the empty directories we need to use the /E option. And the /I option, well, that is a weird option...

If we don't use the /I option xcopy isn't sure if our destination is a file or directory, so we get this prompt:

C:\> xcopy originaldir mynewdir /T /E
Does mynewdir specify a file name
or directory name on the target
(F = file, D = directory)? D


If you select F, then the copy obviously doesn't work. What a weird setting.

Hal copied the permissions too, and so can we. All that needs to be added is the /O switch to copy ownership and ACL information.

Wow, that was easy. Like really easy. And xcopy works in PowerShell and cmd.exe. That makes it even more EXTREEEEEEME!