Resizing QEMU images with Win7 installed under Linux

Here's another Windows 7 post. I started with a 16GB virtual image for Windows and almost immediately ran out of space. It turns out that it's not that hard to resize your partition; I found a very helpful forum post on the subject written a few years ago, and a couple things have changed for Windows 7, so I'll post an updated method here.

First, I'm working from the poster's second set of instructions. He claims they're more dangerous, but they're really not. You never have to work on your only copy of your data; in fact, by the end you'll have three working copies of the drive. It's probably a good idea not to delete the extras until then, but if you are crunched for disk space that's fine. Make sure you have the following commands available: qemu-img, ntfsresize, hexedit. They all should be available in packages supplied by your distro. I'm starting from an image called win7.img, using an intermediate raw image called win7.raw, and ending with an image called win7.qcow2. Obviously, you should use whatever names you're comfortable with.

Before you start, be sure you shut down Windows normally last time you used it. Otherwise ntfsresize will refuse to touch the data.

First, your disk image is probably in qcow2 format. You need to convert it to "raw" format, which is just a flat readout of the data. Check the format:

$ qemu-img info win7.img

If necessary, convert to raw; this will take several minutes (where I have qcow2 you should put the format of your image):

$ qemu-img convert -f qcow2 win7.img -O raw win7.raw

Now expand the raw image by writing to a large offset in the file. This should run very quickly on most Linux filesystems, and not consume much extra space. The bs parameter specifies the "block size" for writes (in bytes) and the seek parameter the number of blocks to skip. So I will write 83886080 × 512 = 42949672960 bytes (40 GiB) in.

$ dd if=/dev/zero of=winb.raw bs=512 count=0 seek=83886080
0+0 records in
0+0 records out
0 bytes (0 B) copied, 1.2997e-05 s, 0.0 kB/s

Become root and use Linux's loopback device to operate on the raw image as if it were a real hard drive.

$ sudo su
(enter password)
# losetup /dev/loop0 win7.raw

Now you'll want to check out the fake disk geometry and edit the partition table using fdisk. You'll want to switch fdisk to display in units of sectors, print the partition table, delete the main partition, and remake a larger one in its place. Your input is in bold.

# fdisk /dev/loop0

The number of cylinders for this disk is set to 5221.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): u
Changing display/entry units to sectors

Command (m for help): p

Disk /dev/loop0: 42.9 GB, 42949672960 bytes
255 heads, 63 sectors/track, 5221 cylinders, total 83886080 sectors
Units = sectors of 1 * 512 = 512 bytes
Disk identifier: 0x23830da9

      Device Boot      Start         End      Blocks   Id  System
/dev/loop0p1   *        2048      206847      102400    7  HPFS/NTFS
Partition 1 does not end on cylinder boundary.                      
/dev/loop0p2          206848    31455231    15624192    7  HPFS/NTFS

Windows 7 has created two NTFS partitions on my disk. The second one is the one we're interested in. Also note the disk geometry: 255 heads. If it doesn't say 255 heads, and your disk is large, it should say 255 heads. Obviously the disk geometry is completely made up, and I'm not really sure why Linux's loopback device should be expected to fake the same geometry as Qemu... comments on the forum suggest this value should always be 255. Anyway, on to editing the partition table entry. The number you use for the first sector must be the same as in the original partition table; for me that's 206848.

Command (m for help): d
Partition number (1-4): 2

Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 2
First sector (63-83886079, default 63): 206848
Last sector, +sectors or +size{K,M,G} (206848-83886079, default 83886079):
Using default value 83886079

Command (m for help): t
Partition number (1-4): 2
Hex code (type L to list codes): 7
Changed system type of partition 2 to 7 (HPFS/NTFS)

Command (m for help): p

Disk /dev/loop0: 42.9 GB, 42949672960 bytes
255 heads, 63 sectors/track, 5221 cylinders, total 83886080 sectors
Units = sectors of 1 * 512 = 512 bytes
Disk identifier: 0x23830da9

      Device Boot      Start         End      Blocks   Id  System
/dev/loop0p1   *        2048      206847      102400    7  HPFS/NTFS
Partition 1 does not end on cylinder boundary.                      
/dev/loop0p2          206848    83886079    41839616    7  HPFS/NTFS

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: Re-reading the partition table failed with error 22: Invalid argument.
The kernel still uses the old table. The new table will be used at
the next reboot or after you run partprobe(8) or kpartx(8)
Syncing disks.

Now re-connect the partition table and check that it worked.

# losetup -d /dev/loop0
# losetup /dev/loop0 win7.raw
# fdisk -ul /dev/loop0

Disk /dev/loop0: 42.9 GB, 42949672960 bytes
255 heads, 63 sectors/track, 5221 cylinders, total 83886080 sectors
Units = sectors of 1 * 512 = 512 bytes
Disk identifier: 0x23830da9

      Device Boot      Start         End      Blocks   Id  System
/dev/loop0p1   *        2048      206847      102400    7  HPFS/NTFS
Partition 1 does not end on cylinder boundary.                      
/dev/loop0p2          206848    83886079    41839616    7  HPFS/NTFS

# losetup -d /dev/loop0

Now, especially if you're resizing from a much smaller drive, we should check that each of the NTFS partitions has the correct number of heads stored in it (the forum post has an explanation of why this is important; to make a long story short, Windows won't start if it's wrong). To find the address of each partition, take its start value from fdisk and multiply by the sector size of 512 bytes. Then convert to hex.

# bc
2048*512
1048576 (this is the start of the first partition in bytes
206848*512
105906176 (this is the start of the second partition in bytes)
quit
# printf "%x %x\n" 1048576 105906176
100000 6500000 (these are the hexadecimal addresses)

Open win7.raw in hexedit; hit return to enter an address, enter the first address, hit return again. Now you're at the start of the first NTFS partition. At an offset 0x1A bytes from that is the head count. Navigate to that byte (if the original address was 0x100000 you'll be at 0x10001A). Make sure it's FF (the hex value of 255). If not, type over that byte with FF. Then do the exact same thing for the second address. Hit ctrl-x to exit, confirming changes if necessary.

Now you can use ntfsresize to actually resize the Windows partition. You have to connect the loopback device to the start of the partition you're resizing this time; this is the decimal version of the second offset you found in the last step. Before doing anything with ntfsresize we check that everything is OK using the -n flag.

# losetup -o105906176 /dev/loop0 win7.raw
# ntfsresize -n /dev/loop0
ntfsresize v2.0.0 (libntfs 10:0:0)
Device name : /dev/loop0
NTFS volume version: 3.1
Cluster size : 4096 bytes
Current volume size: 15999169024 bytes (16000 MB)
Current device size: 42843766784 bytes (42844 MB)
New volume size : 42843763200 bytes (42844 MB)
Checking filesystem consistency ...
100.00 percent completed
Accounting clusters ...
Space in use : 15875 MB (99.2%)
Collecting resizing constraints ...
Schedule chkdsk for NTFS consistency check at Windows boot time ...
Resetting $LogFile ... (this might take a while)
Updating $BadClust file ...
Updating $Bitmap file ...
Updating Boot record ...
The read-only test run ended successfully.

If your output is something like that you're ready to go. The default action is to expand the partition to its largest size, so you only need to run:

# ntfsresize /dev/loop0
Device name : /dev/loop0
NTFS volume version: 3.1
Cluster size : 4096 bytes
Current volume size: 15999169024 bytes (16000 MB)
Current device size: 42843766784 bytes (42844 MB)
New volume size : 42843763200 bytes (42844 MB)
Checking filesystem consistency ...
100.00 percent completed
Accounting clusters ...
Space in use : 15875 MB (99.2%)
Collecting resizing constraints ...
WARNING: Every sanity check passed and only the dangerous operations left.
Make sure that important data has been backed up! Power outage or computer
crash may result major data loss!
Are you sure you want to proceed (y/[n])? y
Schedule chkdsk for NTFS consistency check at Windows boot time ...
Resetting $LogFile ... (this might take a while)
Updating $BadClust file ...
Updating $Bitmap file ...
Updating Boot record ...
Syncing device ...
Successfully resized NTFS on device '/dev/loop0'.

Hopefully your result is something like the above. Now you can disconnect the loopback device and end the sudo session.

# losetup -d /dev/loop0
# exit

You should try out the new image by starting Qemu/KVM the way you usually would, except substituting the name of the new image for that of the old one. Windows will check the disk (because ntfsresize told it to), and everything should come out OK. If not, you still have the original disk image. If everything works you can delete your original image and go on -- if everything is broken you should delete the raw image, figure out what went wrong, and start over.

Now you probably don't want to keep the image around as a raw; qcow2 takes up less space on the host system and has a lot of nice features. You can easily convert it to qcow2 like this:

$ qemu-img convert -f raw win7.raw -O qcow2 win7.qcow2

Make sure the new qcow2 image works. If so, delete the raw image and you're done.

Thanks to poster IntuitiveNipple on the QEMU forum for writing a nice explanation of the bit-hacking necessary on NTFS, and the overall procedure for resizing images with Windows installed.

EDIT: I edited this post to fix table formatting in fdisk output.

The Windows 7 Chronicles: GNU Patch, mt.exe, and the horror of UAC

Real quick one here. patch is a nice little command-line utility. GNU provides a fine version, and there's a Windows build available through the GnuWin32 project. When you try to run it on Windows 7 you get UAC prompts. Why? It turns out that Windows guesses, based on the name of the program, that it's going to patch application files, and goes ahead and requests elevated permissions for it that it doesn't really need. This was discovered a long time ago and a bug was filed against GnuWin, because it's clearly their responsibility to code around Microsoft's incredibly stupid heuristics*.

*in·cred·i·bly stu·pid heu·ris·tics, n: Heuristics that happen to be wrong in my case.

The bug report lists a manifest that you can embed into patch.exe to fix the problem. It looks like this:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<assembly xmlns="urn:schemas-microsoft-com:asm.v1" manifestVersion="1.0">
<trustInfo xmlns="urn:schemas-microsoft-com:asm.v3">
<security>
<requestedPrivileges>
<requestedExecutionLevel level="asInvoker" uiAccess="false"/>
</requestedPrivileges>
</security>
</trustInfo>
</assembly>

Embedding it took me a while, but it's ultimately not that hard. First copy the manifest into a file somewhere. You have to use a tool called mt.exe, which is part of the Windows SDK. It's probably not in your path, but it's in the path of a Visual Studio Command Prompt. So open up one of those with admin permissions (under Win7: Start->All Programs->Microsoft Visual C++ 200x->Visual Studio Tools; right-click on Visual Studio 200x Command Prompt, select Run as administrator...). Then cd '\Program Files (x86)\GnuWin32\bin' (or wherever you've installed it) and mt -manifest yourfile -outputresource:patch.exe;1. Why ;1 at the end? Because Microsoft says so.

QEMU/KVM, Windows 7, blurry text

My mom got me a copy of Windows 7 for Christmas so I could run it in virtualization under Linux for development purposes (Audacity, mostly). That's been somewhat successful. I'm using KVM/QEMU and none of the audio devices that QEMU emulates have 64-bit Win7 drivers. Maybe I'll look at writing an emulation layer for a more modern sound card (it might be slightly easier to write a new 64-bit Windows driver for one of the cards QEMU already emulates, but it might be easier to get an addition to QEMU distributed widely). Anyway. I'm mostly writing this because I just solved a smaller, odder problem: the text in Windows just looked horrible! I generally prefer my text on Kubuntu to text on Windows*, but I didn't remember it being bad on Windows.

* I think people come to prefer the fonts and rendering style of the systems they use every day. I mostly use Kubuntu, and I find text easier to read there than anywhere else. It could just be that I like the font shapes better. FreeType (X11) and ClearType (Windows) look pretty similar given the same fonts, although there are some things FreeType can't do with hinting information because of patents. I find Mac OS X's font rendering ugly and hard to read, but that's probably just because it's more different and I'm not used to it.

So these days QEMU's virtual VGA window can be resized. When you do this it just scales everything in the window. Unfortunately this makes anti-aliased, sub-pixel rendered text (ClearType) and deliberately sharp graphics (Diesel Sweeties) look really bad. KDE lets you set a window to an exact numerical size, which should fix the problem, but it doesn't; the size includes window decorations. Lame.

So I found a little program called xdotool that, among many other things, lets you resize windows from the command line by client size. Rad. It's in the Ubuntu repos, even. But setting the client size to exactly Windows' resolution didn't help.

So I took some screenshots, blew 'em up in the GIMP, and found that the window size was one pixel too large in each dimension. With my Windows resolution set to 1152x864 I actually need the QEMU window's client size to be 1151x863. I'm not sure if this is a weird problem with Windows, QEMU, or X11... but it's weird. And now that it's fixed, everything looks OK in virtualized Windows.