Lately I've found myself having to make lots of file systems. This is mostly due to forensic work, where I'm either sanitizing hard drives and rebuilding file systems on them or I'm creating test file systems for research. Either way, I'm spending lots of time fiddling with file systems at the command line.
Way back in Episode 32 we talked about how to use dd to overwrite a disk device with zeroes:
# dd if=/dev/zero of=/dev/sdd bs=1M
dd: writing `/dev/sdd': No space left on device
992+0 records in
991+0 records out
1039663104 bytes (1.0 GB) copied, 299.834 s, 3.5 MB/s
Of course, this leaves you with an invalid partition table. Happily, the GNU parted utility makes short work of creating a new MS-DOS style disk label and adding a partition:
# parted /dev/sdd print
Error: /dev/sdd: unrecognised disk label
# parted /dev/sdd mklabel msdos
Information: You may need to update /etc/fstab.
# parted /dev/sdd mkpart primary 1 1G
Information: You may need to update /etc/fstab.
# parted /dev/sdd print
Model: LEXAR JUMPDRIVE SPORT (scsi)
Disk /dev/sdd: 1040MB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Number Start End Size Type File system Flags
1 32.3kB 1036MB 1036MB primary
At this point we need to create a file system in our new partition. You actually can use parted to create file systems, but even the parted manual page suggests that you use an external program instead. In Linux, this would be mkfs, which allows you to choose between several different kinds of file systems.
Since this is a small USB key, you might want to just create a FAT file system on it to make it easy to share files between your Linux box and other, less flexible operating systems:
# mkfs -t vfat -F 32 /dev/sdd1
We're using the FAT-specific "-F" option to specify the FAT cluster address size-- here we're creating a FAT32 file system. For each file system type, mkfs has a number of special options specific to that file system. You'll need to read the appropriate manual page to see them all: "man mkfs.vfat" in this case.
If I didn't want my co-authors to be able to easily see the files on this USB stick, I could create an EXT file system instead:
# mkfs -t ext2 /dev/sdd1
mke2fs 1.41.9 (22-Aug-2009)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
63360 inodes, 253015 blocks
12650 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=260046848
8 block groups
32768 blocks per group, 32768 fragments per group
7920 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376
Writing inode tables: done
Writing superblocks and filesystem accounting information: done
This filesystem will be automatically checked every 30 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.
Here I'm creating an "ext2" file system because I didn't want to waste space on a file system journal, but you of course have the option of creating "ext3" and even "ext4" file systems if you want.
If you want to make NTFS file systems, you may have to download an additional package. For example, on my Ubuntu laptop I had to "sudo apt-get ntfsprogs". Once that's done, making NTFS volumes is a snap:
# mkfs -t ntfs -Q /dev/sdd1
Cluster size has been automatically set to 4096 bytes.
Creating NTFS volume structures.
mkntfs completed successfully. Have a nice day.
When creating NTFS volumes, you definitely want to use the "-Q" (quick) option. If you leave off the "-Q" then the mkfs.ntfs program overwrites the device with zeroes and performs a bad block check before creating your file system. This takes a really long time, particularly on large drives, and is also unnecessary in this case since we previously overwrote the drive with zeroes using dd.
It's interesting to note that you don't actually have to have a physical disk device to test file systems. mkfs will (grudgingly) create file systems on non-device files:
# dd if=/dev/zero of=testfs bs=1M count=4096
4096+0 records in
4096+0 records out
4294967296 bytes (4.3 GB) copied, 69.6688 s, 61.6 MB/s
# mkfs -t ntfs -Q -F testfs
testfs is not a block device.
mkntfs forced anyway.
[...]
# mount -o loop,show_sys_files testfs /mnt/test
# ls /mnt/test
$AttrDef $Bitmap $Extend $MFTMirr $UpCase
$BadClus $Boot $LogFile $Secure $Volume
Here I'm first using dd to make a file called "testfs" that contains 4GB of zeroes. Then I call mkfs on the file, using the "-F" (force) option so that it won't exit with an error when I tell it to operate on a non-device file. Though the command whines a lot, it does finally produce a working NTFS file system that can be mounted using a loopback mount.
Of course I can create EXT and FAT file systems in a similar fashion. However, the "-F" option for mkfs.vfat is used to specify the cluster address size. It turns out that you don't need a "force" option when making FAT file systems in non-device files-- the mkfs.vfat will create file systems without complaint regardless of the type of file it is pointed at. For EXT file systems, you can use "-F" if you want. However, if you leave the option off, you'll get a "are you sure?" prompt when running the command against a non-device file (as opposed to mkfs.ntfs which simply bombs out with an error). They say that "the wonderful thing about standards is that there are so many to choose from", but I really wish Linux could rationalize the various mkfs command-line interfaces a bit more.
In any event, being able to create file systems in raw disk files is a real boon when you want to test file system behavior without actually having to commandeer a physical disk drive from someplace. But I think I'd better stop there-- I'm already feeling the hatred and jealousy emanating from my Windows brethren. Let's see what Tim can cook up this week.
Tim was relaxing this weekend for his birthday
This week's episode is pretty easy, but only because there aren't a lot of options. Besides, why would you want to create raw disk files or test file system behavior without searching for a physical disk, connectors, power, ...
No, I'm not jealous. I have everything I need. I don't need all those options. Windows is good enough, smart enough, and doggone it, people like it!
The "streamlined" command in Windows is the good ol' Format command.
C:\> format d:In Vista and later, the format command writes zeros to the entire disk when a full format is performed. In XP and earlier, the format command does not zero the disk. To zero the disk with XP you have to use the diskpart utility.
WARNING, ALL DATA ON NON-REMOVABLE DISK
DRIVE D: WILL BE LOST!
Proceed with Format (Y/N)? y
C:\> diskpartThe clean all command within diskpart zeros the entire disk. One benefit of using clean all is that it actually zeros the disk and doesn't create the MFT. We usually want one though, so Format will suffice.
Microsoft DiskPart version 6.1.7600
Copyright (C) 1999-2008 Microsoft Corporation.
On computer: MYMACHINE
DISKPART> list disk
Disk ### Status Size Free Dyn Gpt
-------- ------------- ------- ------- --- ---
Disk 0 Online 149 GB 0 B
Disk 1 Online 149 GB 0 B
DISKPART> select disk 1
Disk 1 is now the selected disk.
DISKPART> clean all
Format can be used to specify the file system too. We don't have all the
C:\> format e: /FS:NTFSBesides the size restriction, one of the biggest problems with the FAT file system is that it provides no security features. If a user has access to the disk then they have full access to the disk, i.e. there is no way to give a user read access and deny write access to a directory. NTFS allows finer control of ACLs, or even ACLs at all.
C:\> format f: /FS:FAT32
So how do we convert a FAT drive to NTFS? But of course, by using the convert command:
C:\> convert f: /FS:NTFSThe FS switch is required even though the only option is NTFS.
That's about it. Not a lot here these week, and no PowerShell either. There aren't any new cmdlets in PowerShell that provide any additional functionality.