Wednesday, February 17, 2010

distcc, the Macintosh, the Linux box

Today I decided to get distcc working between Jess' computer and mine. There went my day...

I think my situation is fairly common. I have a Linux box that's pretty fast and Jess has a Mac laptop that's not quite as fast. I sometimes have to build Audacity on Jess' computer, and it would be nice if my computer could help with the compiling. Unfortunately this is not as easy as it should be. Here are some of the problems:

1. Apple's version of distcc isn't compatible with the normal version (Apple added some stuff to the protocol). It also has client-side restrictions on the names of compilers you can use, which are configurable, but require the compiler to be in the same location on the client and server. So you should build a normal version of distcc from the source package to install at /usr/local.

2. distcc 3.1 won't build if you just ./configure && make && make install because of a tricky 3-way incompatibility involving GCC, Python, and universal binaries.

3. Xcode likes to invoke gcc with the -x option to specify language. distcc refuses to distribute tasks using -x (Apple's version accepts it, but since it uses a different protocol we can't use it).

And how I did this.

First you need to get a couple of cross-compiling tools build on the Linux side. The good news is that you don't have to have everything for distcc, just as, gcc, and g++. For as you have to find an odcctools package somewhere. There's one here that should work. Then you need Apple's GCC package, matching the one on your Mac. Use gcc --version to get the release and build numbers and find the package on Apple's site.

To build GCC and as instructions #0-4 on this page should work (adjusting for correct target, version numbers, and sources — opendarwin.org no longer exists). It lists cc1plusobj at one point instead of cc1objplus — other than that, fine. Jess has an Intel Mac with MacOS 10.5.8 and XCode 3.1.4, for which the target is i686-apple-darwin9, GCC version number 4.0.1, and GCC build number 5493. So I had with binary names like i686-apple-darwin9-gcc-4.0.1; there is a compiler with the same name on the Mac in the PATH (this must be the case for distcc to work). Heed the directions! You will probably get compile errors, and they won't matter.

Next you need to build a normal version of distcc 3.1 for Mac. To get around the universal binary weirdness you need a non-universal install of Python. Grab a Python 2.6.x source package and the standard ./configure --prefix=$HOME/foo && make && make install cycle will give you a non-universal. $HOME/foo/python2.6. So you can cd back to the distcc source and do PYTHON=$HOME/foo/python2.6 ./configure --prefix=/usr/local && make && make install. It should build and install correctly. Once that's done you don't need that extra Python binary, so you can nuke $HOME/foo.

OK. Now time to make some little connections. First, on the Linux side, you need distcc to be able to find these new compilers. On Deb/Ubuntu you can set the search path by setting PATH in /etc/default/distcc. You should also set up which network interfaces to listen on in that file.

On the Mac side, as our reference notes in step #5, it's nice to have little shell scripts to run distcc. He puts them in /usr/local/bin, you can put them anywhere... the important thing is that his scripts don't work for me; I need $@ instead of @!. Maybe he has csh and I have bash or something. Anyway, for the g++, if your /bin/sh works like mine, you'll want:
#!/bin/sh
/usr/local/bin/distcc <your $TARGET>-g++-<your GCC version> -msse2 "$@"

Then, to account for XCode's use of gcc -x c++ instead of g++, you'll need to get a little fancy in the gcc script (this obviously won't cover all cases but it's been good enough for me so far):

#!/bin/sh
if [ "$1" = "-x" ] && [ "$2" = "c++" ]
then
shift 2
/usr/local/bin/g++ "$@" # this should point to your g++ script, wherever it is
exit
fi

/usr/local/bin/distcc <your $TARGET>-gcc-<your GCC version> -msse2 "$@"

Now you're almost ready. On the Mac, put localhost your_server in ~/.distcc/hosts (reverse the order if you want to go to the server first) and give it a test run (run your gcc script on something simple). tail -f /var/log/distccd.log on the Linux box to make sure the jobs are making it over. For distcc to distribute you need to use the -c flag (i.e. /usr/local/bin/gcc -c hello.c); otherwise, because there's a linking step included in the compile, it must be done locally. You can do /usr/local/bin/gcc -c hello.c && gcc -o hello hello.o && ./hello for a nice test.

Then in your XCode projects you can set the CC variable to the location of your gcc script, and CXX to the location of your g++ script. There might be a better way to do this, I'm far from an XCode expert. Since this bypasses XCode's normal way of working with distcc, it won't know to add extra parallel build steps. Fix this with: defaults write com.apple.Xcode PBXNumberOfParallelBuildSubtasks 6. Or whatever number works best for you there.

I'm pretty sure this is everything essential I did (I went down lots of dead-ends, including trying to use Apple's distcc). And now distcc works!

To be honest, some of the ways Apple modifies Free tools rub me the wrong way. I appreciate that it's a lot easier to interoperate with their tools than, say, Microsoft's, but Microsoft actually wrote their own compilers; they can do it however they want. Apple used GCC and distcc in such a way that interop takes way more work than it ought to. I guess that's why the Linux box is mine, and the Mac is not.

Wednesday, January 27, 2010

Resizing QEMU images with Win7 installed under Linux

Here's another Windows 7 post. I started with a 16GB virtual image for Windows and almost immediately ran out of space. It turns out that it's not that hard to resize your partition; I found a very helpful forum post on the subject written a few years ago, and a couple things have changed for Windows 7, so I'll post an updated method here.

First, I'm working from the poster's second set of instructions. He claims they're more dangerous, but they're really not. You never have to work on your only copy of your data; in fact, by the end you'll have three working copies of the drive. It's probably a good idea not to delete the extras until then, but if you are crunched for disk space that's fine. Make sure you have the following commands available: qemu-img, ntfsresize, hexedit. They all should be available in packages supplied by your distro. I'm starting from an image called win7.img, using an intermediate raw image called win7.raw, and ending with an image called win7.qcow2. Obviously, you should use whatever names you're comfortable with.

Before you start, be sure you shut down Windows normally last time you used it. Otherwise ntfsresize will refuse to touch the data.

First, your disk image is probably in qcow2 format. You need to convert it to "raw" format, which is just a flat readout of the data. Check the format:

$ qemu-img info win7.img

If necessary, convert to raw; this will take several minutes (where I have qcow2 you should put the format of your image):

$ qemu-img convert -f qcow2 win7.img -O raw win7.raw

Now expand the raw image by writing to a large offset in the file. This should run very quickly on most Linux filesystems, and not consume much extra space. The bs parameter specifies the "block size" for writes (in bytes) and the seek parameter the number of blocks to skip. So I will write 83886080 × 512 = 42949672960 bytes (40 GiB) in.

$ dd if=/dev/zero of=winb.raw bs=512 count=0 seek=83886080
0+0 records in
0+0 records out
0 bytes (0 B) copied, 1.2997e-05 s, 0.0 kB/s


Become root and use Linux's loopback device to operate on the raw image as if it were a real hard drive.

$ sudo su
(enter password)
# losetup /dev/loop0 win7.raw


Now you'll want to check out the fake disk geometry and edit the partition table using fdisk. You'll want to switch fdisk to display in units of sectors, print the partition table, delete the main partition, and remake a larger one in its place. Your input is in bold.

# fdisk /dev/loop0

The number of cylinders for this disk is set to 5221.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): u
Changing display/entry units to sectors

Command (m for help): p

Disk /dev/loop0: 42.9 GB, 42949672960 bytes
255 heads, 63 sectors/track, 5221 cylinders, total 83886080 sectors
Units = sectors of 1 * 512 = 512 bytes
Disk identifier: 0x23830da9


      Device Boot      Start         End      Blocks   Id  System
/dev/loop0p1 * 2048 206847 102400 7 HPFS/NTFS
Partition 1 does not end on cylinder boundary.
/dev/loop0p2 206848 31455231 15624192 7 HPFS/NTFS

Windows 7 has created two NTFS partitions on my disk. The second one is the one we're interested in. Also note the disk geometry: 255 heads. If it doesn't say 255 heads, and your disk is large, it should say 255 heads. Obviously the disk geometry is completely made up, and I'm not really sure why Linux's loopback device should be expected to fake the same geometry as Qemu... comments on the forum suggest this value should always be 255. Anyway, on to editing the partition table entry. The number you use for the first sector must be the same as in the original partition table; for me that's 206848.

Command (m for help): d
Partition number (1-4): 2

Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 2
First sector (63-83886079, default 63): 206848
Last sector, +sectors or +size{K,M,G} (206848-83886079, default 83886079):
Using default value 83886079

Command (m for help): t
Partition number (1-4): 2
Hex code (type L to list codes): 7
Changed system type of partition 2 to 7 (HPFS/NTFS)

Command (m for help): p

Disk /dev/loop0: 42.9 GB, 42949672960 bytes
255 heads, 63 sectors/track, 5221 cylinders, total 83886080 sectors
Units = sectors of 1 * 512 = 512 bytes
Disk identifier: 0x23830da9


      Device Boot      Start         End      Blocks   Id  System
/dev/loop0p1 * 2048 206847 102400 7 HPFS/NTFS
Partition 1 does not end on cylinder boundary.
/dev/loop0p2 206848 83886079 41839616 7 HPFS/NTFS

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: Re-reading the partition table failed with error 22: Invalid argument.
The kernel still uses the old table. The new table will be used at
the next reboot or after you run partprobe(8) or kpartx(8)
Syncing disks.


Now re-connect the partition table and check that it worked.

# losetup -d /dev/loop0
# losetup /dev/loop0 win7.raw
# fdisk -ul /dev/loop0

Disk /dev/loop0: 42.9 GB, 42949672960 bytes
255 heads, 63 sectors/track, 5221 cylinders, total 83886080 sectors
Units = sectors of 1 * 512 = 512 bytes
Disk identifier: 0x23830da9


      Device Boot      Start         End      Blocks   Id  System
/dev/loop0p1 * 2048 206847 102400 7 HPFS/NTFS
Partition 1 does not end on cylinder boundary.
/dev/loop0p2 206848 83886079 41839616 7 HPFS/NTFS

# losetup -d /dev/loop0

Now, especially if you're resizing from a much smaller drive, we should check that each of the NTFS partitions has the correct number of heads stored in it (the forum post has an explanation of why this is important; to make a long story short, Windows won't start if it's wrong). To find the address of each partition, take its start value from fdisk and multiply by the sector size of 512 bytes. Then convert to hex.

# bc
2048*512
1048576
(this is the start of the first partition in bytes
206848*512
105906176
(this is the start of the second partition in bytes)
quit
# printf "%x %x\n" 1048576 105906176
100000 6500000
(these are the hexadecimal addresses)

Open win7.raw in hexedit; hit return to enter an address, enter the first address, hit return again. Now you're at the start of the first NTFS partition. At an offset 0x1A bytes from that is the head count. Navigate to that byte (if the original address was 0x100000 you'll be at 0x10001A). Make sure it's FF (the hex value of 255). If not, type over that byte with FF. Then do the exact same thing for the second address. Hit ctrl-x to exit, confirming changes if necessary.

Now you can use ntfsresize to actually resize the Windows partition. You have to connect the loopback device to the start of the partition you're resizing this time; this is the decimal version of the second offset you found in the last step. Before doing anything with ntfsresize we check that everything is OK using the -n flag.

# losetup -o105906176 /dev/loop0 win7.raw
# ntfsresize -n /dev/loop0
ntfsresize v2.0.0 (libntfs 10:0:0)
Device name : /dev/loop0
NTFS volume version: 3.1
Cluster size : 4096 bytes
Current volume size: 15999169024 bytes (16000 MB)
Current device size: 42843766784 bytes (42844 MB)
New volume size : 42843763200 bytes (42844 MB)
Checking filesystem consistency ...
100.00 percent completed
Accounting clusters ...
Space in use : 15875 MB (99.2%)
Collecting resizing constraints ...
Schedule chkdsk for NTFS consistency check at Windows boot time ...
Resetting $LogFile ... (this might take a while)
Updating $BadClust file ...
Updating $Bitmap file ...
Updating Boot record ...
The read-only test run ended successfully.


If your output is something like that you're ready to go. The default action is to expand the partition to its largest size, so you only need to run:

# ntfsresize /dev/loop0
Device name : /dev/loop0
NTFS volume version: 3.1
Cluster size : 4096 bytes
Current volume size: 15999169024 bytes (16000 MB)
Current device size: 42843766784 bytes (42844 MB)
New volume size : 42843763200 bytes (42844 MB)
Checking filesystem consistency ...
100.00 percent completed
Accounting clusters ...
Space in use : 15875 MB (99.2%)
Collecting resizing constraints ...
WARNING: Every sanity check passed and only the dangerous operations left.
Make sure that important data has been backed up! Power outage or computer
crash may result major data loss!
Are you sure you want to proceed (y/[n])? y
Schedule chkdsk for NTFS consistency check at Windows boot time ...
Resetting $LogFile ... (this might take a while)
Updating $BadClust file ...
Updating $Bitmap file ...
Updating Boot record ...
Syncing device ...
Successfully resized NTFS on device '/dev/loop0'.


Hopefully your result is something like the above. Now you can disconnect the loopback device and end the sudo session.

# losetup -d /dev/loop0
# exit


You should try out the new image by starting Qemu/KVM the way you usually would, except substituting the name of the new image for that of the old one. Windows will check the disk (because ntfsresize told it to), and everything should come out OK. If not, you still have the original disk image. If everything works you can delete your original image and go on -- if everything is broken you should delete the raw image, figure out what went wrong, and start over.

Now you probably don't want to keep the image around as a raw; qcow2 takes up less space on the host system and has a lot of nice features. You can easily convert it to qcow2 like this:

$ qemu-img convert -f raw win7.raw -O qcow2 win7.qcow2

Make sure the new qcow2 image works. If so, delete the raw image and you're done.

Thanks to poster IntuitiveNipple on the QEMU forum for writing a nice explanation of the bit-hacking necessary on NTFS, and the overall procedure for resizing images with Windows installed.

EDIT: I edited this post to fix table formatting in fdisk output.

Monday, January 18, 2010

The Windows 7 Chronicles: GNU Patch, mt.exe, and the horror of UAC

Real quick one here. patch is a nice little command-line utility. GNU provides a fine version, and there's a Windows build available through the GnuWin32 project. When you try to run it on Windows 7 you get UAC prompts. Why? It turns out that Windows guesses, based on the name of the program, that it's going to patch application files, and goes ahead and requests elevated permissions for it that it doesn't really need. This was discovered a long time ago and a bug was filed against GnuWin, because it's clearly their responsibility to code around Microsoft's incredibly stupid heuristics*.

*in·cred·i·bly stu·pid heu·ris·tics, n: Heuristics that happen to be wrong in my case.

The bug report lists a manifest that you can embed into patch.exe to fix the problem. It looks like this:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<assembly xmlns="urn:schemas-microsoft-com:asm.v1" manifestVersion="1.0">
<trustInfo xmlns="urn:schemas-microsoft-com:asm.v3">
<security>
<requestedPrivileges>
<requestedExecutionLevel level="asInvoker" uiAccess="false"/>
</requestedPrivileges>
</security>
</trustInfo>
</assembly>

Embedding it took me a while, but it's ultimately not that hard. First copy the manifest into a file somewhere. You have to use a tool called mt.exe, which is part of the Windows SDK. It's probably not in your path, but it's in the path of a Visual Studio Command Prompt. So open up one of those with admin permissions (under Win7: Start->All Programs->Microsoft Visual C++ 200x->Visual Studio Tools; right-click on Visual Studio 200x Command Prompt, select Run as administrator...). Then cd '\Program Files (x86)\GnuWin32\bin' (or wherever you've installed it) and mt -manifest yourfile -outputresource:patch.exe;1. Why ;1 at the end? Because Microsoft says so.

Wednesday, January 6, 2010

QEMU/KVM, Windows 7, blurry text

My mom got me a copy of Windows 7 for Christmas so I could run it in virtualization under Linux for development purposes (Audacity, mostly). That's been somewhat successful. I'm using KVM/QEMU and none of the audio devices that QEMU emulates have 64-bit Win7 drivers. Maybe I'll look at writing an emulation layer for a more modern sound card (it might be slightly easier to write a new 64-bit Windows driver for one of the cards QEMU already emulates, but it might be easier to get an addition to QEMU distributed widely). Anyway. I'm mostly writing this because I just solved a smaller, odder problem: the text in Windows just looked horrible! I generally prefer my text on Kubuntu to text on Windows*, but I didn't remember it being bad on Windows.

* I think people come to prefer the fonts and rendering style of the systems they use every day. I mostly use Kubuntu, and I find text easier to read there than anywhere else. It could just be that I like the font shapes better. FreeType (X11) and ClearType (Windows) look pretty similar given the same fonts, although there are some things FreeType can't do with hinting information because of patents. I find Mac OS X's font rendering ugly and hard to read, but that's probably just because it's more different and I'm not used to it.

So these days QEMU's virtual VGA window can be resized. When you do this it just scales everything in the window. Unfortunately this makes anti-aliased, sub-pixel rendered text (ClearType) and deliberately sharp graphics (Diesel Sweeties) look really bad. KDE lets you set a window to an exact numerical size, which should fix the problem, but it doesn't; the size includes window decorations. Lame.

So I found a little program called xdotool that, among many other things, lets you resize windows from the command line by client size. Rad. It's in the Ubuntu repos, even. But setting the client size to exactly Windows' resolution didn't help.

So I took some screenshots, blew 'em up in the GIMP, and found that the window size was one pixel too large in each dimension. With my Windows resolution set to 1152x864 I actually need the QEMU window's client size to be 1151x863. I'm not sure if this is a weird problem with Windows, QEMU, or X11... but it's weird. And now that it's fixed, everything looks OK in virtualized Windows.