The Problem

The stock Python 2.6 shipped with OS X 10.6 Snow Leopard has a readline module linked to libedit, the BSD alternative to the GPL’ed readline. The readline module, if you are not aware, is (among other things) responsible for keeping command history in the IPython interpreter. This causes command history in the IPython 0.10 interpreter to behave in very odd ways. When backtracking through the command history buffer using the up-arrow key, for example, the previous command is only partially recalled, and appears completely scrambled. Indents, too, seem off — in a whitespace-sensitive language like Python, this is annoying. (See first figure)

IPython command interpreter is broken when using libedit with command history

Fixing IPython’s bugs are beyond my ability. While I certainly don’t want to delve into the quagmire that is GPL vs BSD licensing, I do understand why Apple would want to avoid the viral nature of the GPL and ship libedit instead. However, using a genuine Readline library is going to be the best recourse for this problem. I already have a copy of readline compiled and ready to go, and just need a new version of readline.so, the library that links Python to readline.

The easy solution

Sifting through my records, I came across a SelfSolved problem record from my good friend Hannes who had issues with his IPython command history.

The solution: sudo easy_install readline, which uses setuptools to install a precompiled package of readline.so statically linked to genuine GNU readline. Restart your IPython console and everything should work. (See second figure)

IPython with readline

The hard solution

Being the inquisitive sort, I also wondered how I would be able to reproduce this work from scratch. readline.so ships with the Python source package, but surely I would not be required to compile a whole new copy of Python for one measly module library?

I documented this process in SelfSolved again: building readline.so for Python. At some point I should write an interface between SelfSolved and Wordpress so that I don’t have to reproduce a lot of my work here manually.

Compiling readline.so

This is actually fairly easy.

1. Get a copy of the Python source code. In OS X 10.6, it ships with Python 2.6.1.
2. Unpack it and go into its directory. You should find a Modules subdirectory. In it is readline.c, the source file for readline.so.
3. Compile the source file. The appropriate incantation is:
   gcc -O2 -arch x86_64 -arch i386 -shared -o readline.so readline.c -I/usr/local/include -I/System/Library/Frameworks/Python.framework/Versions/2.6/include/python2.6 -L/usr/local/lib -lreadline -ltermcap -framework Python

   where the -arch flags should be whatever processors you wish to support, the -I arguments should point to the directories that contain header files for the readline library and the Python framework, and the -L argument should point to the path for the readline library. Use whatever optimization flags you feel comfortable with, instead of -O2, if you wish.

Replacing readline.so

So now we have a readline.so that’s properly linked to readline.dylib. The thornier question is how to override the system-provided readline.so. The system version is located at /System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-dynload/readline.so, and the naive would simply overwrite it with their new readline.so. This is a bad idea.

As I have mentioned in the past, overwriting system libraries in OS X is an unhealthy thing to do. The problem is that Apple furnishes no official package management system — anything you personally change is considered fair game for the next official system update. On the next system update, if the Python component is affected by the update, the Apple updater will happily clobber your compiled files with its own, leaving you suddenly back at square one. You don’t know how many times I’ve had to recompile emacs (for X11 support) on OS X 10.4 because of this little annoyance. Leave things in the /System/Library directory hierarchy alone, for your own sanity.

However, in this case /System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-dynload comes ahead of the user-modifiable /Library/Python/2.6/site-packages directory on Python’s sys.path. So if you just drop readline.so into site-packages, the system version still takes priority.

There are a few ways to do this. For one, you can create a sitecustomize.py in /Library/Python2.6/site-packages. In this file, arbitrary Python statements can be written, and the interpreter will automatically execute them at runtime. So, you can add a sys.path = ["/dir/here"] + sys.path statement and point it to a directory containing your readline.so file. Alternatively, you can abuse the technique used in the easy_install.pth file. It turns out that if you ever used easy_install, directories pointed to by the easy_install.pth file takes priority over the system paths. They use an interesting way to accomplish this, which you can copy. Or, you can just insert your directory containing readline.so into easy_install.pth. In any case, this will force the readline-based readline.so to take precedence over the libedit-based readline.so, without overwriting anything.

Discussion

So for any sane person, the easy solution should be enough. For the rest, the hard solution is an interesting exploration of how some of Python’s built-in modules can be compiled and inserted individually.

Seagate did in fact issue a firmware update — SD1A — that supposedly addressed this issue, but of course, there’s one catch: you can’t install the firmware through an external drive enclosure. In communication with Seagate support, a representative confirmed that for those of us without a desktop tower that has a SATA bay, we’re hosed:

Unfortunately, due to the nature of firmware updates and the way external drives work, the firmware update program cannot directly communicate with the drive in the manner it needs to in order to be able to upload the new firmware to the drive. It must be plugged into an internal SATA controller in order to update the drive.

Fair enough. That makes technical sense — but of course, it doesn’t work for me. I asked whether they would handle a mail-in repair, given that I have no easy access to such a desktop. The answer, of course, is No.

I have to find a desktop, open it up, jam this baby in (possibly in place of the existing drive if there’s only one bay), update the firmware, and put everything back together. Sadly, most of my friends who still own desktops would not trust me that far.

Half a year passes, and I finally find a sucker good friend who’s gullible awesome enough for me to try this procedure on his machine. The fellow owns a nice if aging Dell Precision T5400, which comes with two SATA bays (so I don’t have to inflict undue harm onto the existing system). Since this thing can run two drives at once, I can use the first method (a Windows-based firmware updater), though I burned a boot CD for the second method just in case. I popped in the drive, fired up Windows XP, downloaded the Windows-based Firmware Update Utility, double-clicked, and thought it was the (triumphant) end. In fact, it took 3 hours of my life to find out just how deep this rabbit hole goes.

Problem 1: The lying updater

The firmware updater will give a bunch of scary warnings and then reboot the machine. It will automatically reboot to a Seagate Loader screen, which attempts to apply the patch to all eligible SATA drives. To its credit, it’ll skip the non-qualifying (i.e. non-Seagate, non-Barracuda, etc.) drives, but it’ll still try them out first. At the end of the process, it will report “firmware downloaded” and “SUCCESSFUL” or some variant thereof, and automatically reboot back into Windows.

At this point, I advise you to use the SeaTools utility to verify that the firmware update actually applied. Despite its claims, if you were on a stock setup Dell T5400 (or perhaps other models as well), this will prove that the updater is a lying scumbag. And in fact, this particular drive still reported firmware SD15, the broken one.

Problem 2: The broken Boot CD

To save both me and my gracious host (who’s starting to suspect my computer-fixin’ skills now) some time, I decided to try the boot CD method, rather than pounding my head trying to see why the updater was lying. I downloaded the boot CD from the same Seagate Support site above, burned it to disk, and tried it out.

The result is a new SelfSolved posting: SelfSolved #59: getFatBlock error when upgrading Seagate Barracuda 7200.11 firmware. In essence:

The FreeDOS boot CD reports a number of ” error reading partition table drive 01 sector 0 ” errors. This is followed by ” get Fatblock failed:0×000000e8 ” or some variant of ” getFatBlock failed : ” The FreeDOS boot process appears to stall at this stage, and does not continue to the firmware flasher program.

That was lovely.

The Solution

I chased some red herrings. I came across postings about failures in various FreeDOS-based Seagate tools. One such post mentioned that it took a long time for the boot disc to get over the “error reading partition table” errors, but I waited forever (well, 15 minutes) and the boot process did appear to be frozen / stalled. I reformatted the drive via diskpart clean, thinking that the getFatBlock and error reading partition errors were related to a non-MBR partition table (I had it set to GPT). I should have realized, of course, that the errors were completely unrelated to filesystems, despite the “fat block” to which it refers.

The actual solution is deceptively simple — the boot disc & flasher appears to handle AHCI-based SATA mode badly. The Dell I was using was set to AHCI mode, out of the three possible Legacy, AHCI, and RAID options for SATA. Apparently the boot disc simply doesn’t handle this mode correctly on the Dell machine (and may also be related to why the Windows-based updater lies). When the machine switches on, use F12 to enter the boot menu, and select Setup to enter the BIOS. Then, on the list of Drive Options, skip past the SATA drives and down to SATA options. Pick the Legacy option to use ATA mode, instead of AHCI. Once this is done, the boot disc will function correctly, and the updated firmware will be applied without incident. Remember to switch the mode back to AHCI — it’s default for reason, no doubt.

The “error reading partition” messages were completely red herrings. They appear whether you are in the right SATA mode or not, and does not appear to affect the operation of the firmware updater or the boot process. It should not take very long to get to the flasher on this particular setup, so don’t wait on that message too long — it’s a good sign something’s not quite right.

In the end, I did recover my $100 hard drive, and the confidence of my peer in my mad hardware skillz (actually, quite non-existent).

Discussion

In the end, I’m quite appalled at Seagate. This sort of failure shouldn’t have happened, of course. Once it did, Seagate should have offered to take back and replace broken drives — the data I had on there was non-critical. I would have been perfectly willing to pay shipping costs to get a fixed replacement through mail-in service. I should not have been forced to search my social network for a person willing to let me tear his desktop computer apart, for a dubious and unsure firmware update procedure that fails mysteriously. I spent an additional 3 hours tracing mystery failures, for which the error messages were rather useless. Without my trusty iPhone and access to the Internet, I would not been able to solve this problem. How should I have known what “getFatBlock failed” means?

This little episode has convinced me to never buy a Seagate drive again — I simply cannot afford the time and energy for these sort of firmware upgrade adventures. While I was looking for a desktop to tear apart, I bought a Western Digital Caviar Black 1TB drive instead. Another $100, but at least I had a scratch drive for my work.

The moral of the story: Seagate, you are the worst storage vendor I’ve had to work with so far. I hope this record is not broken in the future.

After compiling Subversion 1.6.2 from source on OS X 10.5 Leopard, the compilation is apparently successful, but svn dies when it tries to connect to the network for the first time. Crash log reports that symbols are missing from libneon.dylib.

Crash report from shell:

dyld: lazy symbol binding failed: Symbol not found: _ne_set_connect_timeout
Referenced from: /usr/local/lib/libsvn_ra_neon-1.0.dylib
Expected in: dynamic lookup

dyld: Symbol not found: _ne_set_connect_timeout
Referenced from: /usr/local/lib/libsvn_ra_neon-1.0.dylib
Expected in: dynamic lookup


Check out the places that I googled and my final solution writeup … at SelfSolved #49: Subversion 1.6.2 explodes on first network access.

The problem is very similar to a previous compilation issue I solved for PHP. In essence, the -L library search path passed to GCC at compilation time has /usr/lib in front of everything else. This means whatever library path you might have given to it at configure time, it’ll always look for the library in /usr/lib first, picking up the old system libneon in the process. Since the bad libneon dynamically linked, the problem doesn’t manifest itself until runtime — and only at runtime with network access involved.

As with the PHP issue, change the very first -L/usr/lib to -L/usr/local/lib (or wherever your newer libneon is located), and it’ll link correctly.

Out of curiosity, I checked MacPorts first. The MacPorts solution of disabling libneon version checking is odd — it also works, but I dunno if it’s linking to the right thing or not.

A couple of my iPhone projects require a decent HTML/XHTML parser. On OS X, Cocoa ships with NSXMLDocument, which includes dirty HTML parsing functionality from libtidy. Unfortunately, NSXMLDocument is not part of the actual iPhone 2.2 SDK (though it is part of the 2.2 Simulator — so it’ll compile just fine at dev time but break when deploying — a big gotcha if you never tested against a real iPhone).

NSXMLParser is a part of the iPhone SDK…This is not a reasonable alternative.

Check out my writeup at SelfSolved #42: HTML or XHTML Parser for iPhone SDK 2.x

Finally, all out of all the potential alternatives I found (all referenced at the SelfSolved writeup — including one that requires a license fee to use), this one seems to be the most promising and requires the least amount of pain (read: interaction with the libxml C API — god knows I’ve done enough of that while building prototypes at Yahoo! Research Berkeley)

One thing that has been irritating me lately is that there’s a cosmetic error MenuMeters 1.3 that causes negative values to appear in the VM Statistics section of the memory stats display. For example, the page faults value can roll over INT_MAX to report -1,800,000 page faults, when I’ve used the same OS X session for a long time without rebooting.

Since MenuMeters is GPL’ed, a quick lookthrough at its codebase reveals the problem. The details of this problem and solution is currently documented as #32 MenuMeters Memory Meter reports negative page faults at SelfSolved, a new web application I’ve written to keep track of these things.

More details to follow.

In the case of SSH, even if you google for this, you’ll run through thousands of examples of using ssh as a SOCKS server, but not through one as a SOCKS client. There are some convoluted solutions, but none of them I can use directly on an OS X 10.5 machine.

TSocks: the solution…if it were that easy

Now, tsocks is a nifty little tool to transparently divert network calls through a SOCKS 4 or SOCKS 5 proxy. This allows even non-SOCKS-aware applications to function through a SOCKS server.

Unfortunately it is very old, unmaintained code (1.8 beta 5 was released in 2002). It doesn’t compile cleanly on OS X due to this, nor will it compile under GCC 4.x. Further, it won’t work out of the box either if you do manage to compile it. The problem is that it relies on the Linux-only LD_PRELOAD functionality to use a shared library to hijack network system calls. This mechanism is called DYLD_INSERT_LIBRARIES on OS X and only works if DYLD_FORCE_FLAT_NAMESPACES is active.

Getting a working tsocks: MacPorts

There is an easy way to get tsocks. MacPorts ships a ported tsocks package. If you use MacPorts, sudo port install tsocks should do it.

Unfortunately on several machines I don’t use MacPorts, and don’t want to pull down an entire third-party package manager with its own library tree on each of these boxes. So I have do to this the hard way.

Getting a working tsocks: rolling my own

First to notice is that there are two tsocks distributions. One is the original tsocks 1.8b5, last updated in the first half of this decade. To make it work, follow the instructions provided by Marc Abramowitz in 2006. Note that his patch is actually located at his new domain address instead of the old, linked one.

The MacPorts distribution, on the other hand, is based on R. Garcia’s patched tsocks distribution, incorporating some modernization and new features by the Tor team. This distribution is numbered 1.8.x, with the last being 1.8.4. Unfortunately it is also no longer maintained, as the Tor devs forked this into a custom version to use with the Tor network only. Unfortunate, but for now, it still compiles, and works a bit better than the 2002 original.

To roll your own tsocks via source out of the MacPorts distribution, you will want the patches from the MacPorts repository. An outline of the compilation procedure:

1. Download tsocks 1.8.4 from the author’s page
2. Download all the patches from the MacPorts repository
3. Concatenate all of the patches together:

   cat patch-* > tsocks.osx.patch

4. Put the concatenated tsocks.osx.patch file into the tsocks source directory. Apply the patches:

   patch -p0 < tsocks.osx.patch

5. Regenerate the configure script:

   autoreconf

6. Configure the package:

   ./configure --prefix=/usr/local --bindir=/usr/local/bin --mandir=/usr/local/man --sysconfdir=/etc --libdir=/usr/local/lib

7. Install the library and binaries:

   sudo make install

8. Install the conf file:

   sudo cp ./tsocks.conf.complex.example /etc/tsocks.conf

9. Edit the conf file. Make sure that if you’re not using tor, that you write in the conf file

   tordns_enable = false

Configuring tsocks

The complex configuration file example should have explained all of the features to be set. For my configuration:

Some important settings:

• local – this setting, in the format of IP/netmask can be repeated several times, each time to exclude a set of IPs from being diverted to the SOCKS server. For obvious reasons, your SOCKS server will have to exist in one of these excluded IP ranges – otherwise you will never even reach your proxy server.
• server and server_port – these should point to the IP address and port of your SOCKS server
• server_type – tsocks defaults to SOCKS4 mode. You may wish to set it to 5 for SOCKS5 usage.
• tordns_enable – this needs to be set as false if you don’t use Tor.

Using tsocks

Once this is set up, simply prefixing the network command you want to run with tsocks will force a diversion through the proxy connection. For example:

tsocks ssh example.com

The same can be applied to Subversion.

tsocks svn update

will force the svn client to act through the proxy set in tsocks.conf.

SOCKS on localhost

Note that SOCKS services on 127.0.0.1 has a minor gotcha. Sometimes, you are able to SSH into a remote machine, and use that connection as your SOCKS server. This is described in my post about using SSH as a pseudo-VPN, which describes the -D switch. My use case here is that once you do this, all further local SSH connections to other machines should be diverted through the first SSH. For example, I’d like to do:

my-machine$ ssh -D 40000 gateway.example.com # establish a SOCKS server on localhost:40000 to the gateway host

and then:

my-machine$ ssh lan-1.example.com # access the protected lan-1 machine through the SOCKS, which will see me as gateway.example.com 

This is very doable in the tsocks setup if you set tsocks.conf:

server = 127.0.0.1/255.255.255.255
server_port = 40000

and then:

my-machine$ ssh -D 40000 gateway.example.com
my-machine$ tsocks ssh lan-1.example.com

This is the gotcha: make sure the netmask is set correctly to 255.255.255.255. Otherwise tsocks will die with a cryptic:

IP (127.0.0.1) & SUBNET (0.0.0.0) != IP on line 22 in configuration file, ignored

It is apparently fairly sensitive about the subnet mask setup to conform to exact standards.

With this tsocks setup, you won’t have to create special VPNs to lock a LAN machine behind a gateway. As long as you can SSH into the gateway machine from your local machine, you can access the resources behind it with any application on your local machine via tsocks. Nifty, huh?

Undefined symbols for architecture i386:
  "_xmlTextReaderSchemaValidate", referenced from:
      _zim_xmlreader_setSchema in php_xmlreader.o
  "_xmlTextReaderSetup", referenced from:
      _zim_xmlreader_XML in php_xmlreader.o
ld: symbol(s) not found for architecture i386
collect2: ld returned 1 exit status
Undefined symbols for architecture x86_64:
  "_xmlTextReaderSchemaValidate", referenced from:
      _zim_xmlreader_setSchema in php_xmlreader.o
  "_xmlTextReaderSetup", referenced from:
      _zim_xmlreader_XML in php_xmlreader.o
ld: symbol(s) not found for architecture x86_64

The MacPorts folks have encounted similar issues in ticket 15891, but WONTFIX‘ed the issue. Apparently the PHP devs are also punting on the problem.

The immediate cause is that you have multiple versions of some shared libraries. For example, in the case above, I have two libxml versions — one in /usr/lib, and another in /usr/local/lib. This is because I do not want to overwrite the Apple-provided libxml version, but still needed new features provided in later libxml versions. The arrangement works fine in every other software compile except this one, so I investigated further.

The root of the problem

Despite the developers’ airy dismissal of the issue, the underlying problem is indeed that the Makefile generated by PHP at configure time is slightly broken. In Makefile and Makefile.global, you’re going to see this line:

libs/libphp$(PHP_MAJOR_VERSION).bundle: $(PHP_GLOBAL_OBJS) $(PHP_SAPI_OBJS)
        $(CC) $(MH_BUNDLE_FLAGS) $(CFLAGS_CLEAN) $(EXTRA_CFLAGS) $(LDFLAGS) $(EXTRA_LDFLAGS) $(PHP_GLOBAL_OBJS:.lo=.o) $(PHP_SAPI_OBJS:.lo=.o) $(PHP_FRAMEWORKS) $(EXTRA_LIBS) $(ZEND_EXTRA_LIBS) -o $@ && cp $@ libs/libphp$(PHP_MAJOR_VERSION).so

where $MH_BUNDLE_FLAGS is usually defined as something like

MH_BUNDLE_FLAGS = -bundle -bundle_loader /usr/sbin/httpd -L/usr/lib \
 -L/usr/lib -laprutil-1 -lsqlite3 -lexpat -liconv -L/usr/lib -lapr-1 -lpthread

The problem is that this hardcodes the search paths for linking shared libraries. GCC searches for shared libraries to link in the order of the provided -L paths. In this case, MH_BUNDLE_FLAGS is expanded immediately after $CC — so the load order is:

1. /usr/lib
2. /usr/lib (these are redundant, and so will probably be collapsed into one path)
3. …every other custom library path you specify

Now you see the issue. No matter what your library paths are set to, the PHP compilation system will insist that whatever shared libraries in /usr/lib take precedence. Therefore, even if you specified that another version (say, libxml.dylib in /usr/local/lib) should be used instead, the invocation to link against -lxml2 will search in /usr/lib first. And since it finds the old version, which may be missing a number of symbols, the compilation blows up right there.

Evidence

And indeed, if you look at the (rather long and massive) compilation/link command right before it fails, you’ll see:

gcc -bundle -bundle_loader /usr/sbin/httpd -L/usr/lib -L/usr/lib \
-laprutil-1 -lsqlite3 -lexpat  -liconv -L/usr/lib -lapr-1 -lpthread -O2 -I/usr/include -DZTS   \
-arch i386 -arch x86_64 -L/usr/local/lib ... 

emphasis mine, where /usr/local/lib might be /opt/lib or whatever custom path you provided to configure.

Solutions

The trivial solution is to manually invoke that last line of compilation, but swapping the -L load paths.

gcc -bundle -bundle_loader /usr/sbin/httpd -L/usr/local/lib -L/usr/lib \
-L/usr/lib -laprutil-1 -lsqlite3 -lexpat  -liconv -L/usr/lib -lapr-1 -lpthread -O2 -I/usr/include -DZTS   \
-arch i386 -arch x86_64  ... 

This is easy to do and takes just a second.

Another possible solution is to patch the Makefile, such that MH_BUNDLE_FLAGS comes later in the compilation line:

libs/libphp$(PHP_MAJOR_VERSION).bundle: $(PHP_GLOBAL_OBJS) $(PHP_SAPI_OBJS)
        $(CC) $(CFLAGS_CLEAN) $(EXTRA_CFLAGS) $(LDFLAGS) $(EXTRA_LDFLAGS) $(PHP_GLOBAL_OBJS:.lo=.o) $(PHP_SAPI_OBJS:.lo=.o) $(PHP_FRAMEWORKS) $(EXTRA_LIBS) $(ZEND_EXTRA_LIBS) $(MH_BUNDLE_FLAGS) -o $@ && cp $@ libs/libphp$(PHP_MAJOR_VERSION).so

This will force your library paths to be searched before /usr/lib, thus resolving the link problem.

update 7/18/09
An anonymous reader mentions that you could also specify the right libxml by full path, instead of letting it use -lxml. Basically, in the last compilation line, you would remove any mentions of -lxml and replace that with the full path to your library e.g. /usr/local/lib/libxml.dylib. In fact, this is probably the way that has the least possible side-effects, since you aren’t changing the search order for any other libraries.

Discussion

This is not the first time that PHP core developers have refused to fix a compilation issue that is arguably preventable through actual testing under different installation scenarios. This is an “edgier” edge case than the tidy.h issue, but still should be fairly noticeable for a substantial number of people.

The “You should only have one library installed” argument is, to be honest, unnecessarily arrogant (sadly, not as a rare a problem as you’d like in some open source development projects ). On OS X, due to the lack of official Apple package management systems, no one should be overwriting system default libraries — down that way lies insanity, especially at every system or security update. And yet, this build system is obviously broken any time there is a substantial difference between user-installed libraries and system libraries. This bad behavior is especially egregious, because the configure command allows you specify your own library path — misleading users into thinking that the path they specified would be obeyed at compile time. If you only intend for the system library to be used and no other, perhaps the configure script should auto-detect this on OS X and disable that command-line option. Basic user interface design should apply even to command-line interfaces.

Note that changing link ordering may have some unforseen consequences, since the devs obviously never tested this path. For example, you should make sure the dynamic libraries are loaded in the right order at runtime. On OS X, the load path is typically hard-coded into the dylib, so usually there won’t be a problem — but there may be edge cases. Test your build (and any PHP extensions you built) before using it in production!

Establishing a connection

To start with, the usual process is to telnet to a POP3 server port, usually on TCP port 110. This would be very simple:
telnet mail.example.com 110

Nowadays, though, most POP3 servers are secured via SSL, usually sitting on port 995. If you try to use telnet on an SSL-only POP3 server, you’ll either get an error “Command is not valid in this state”, such as:

Trying 127.0.0.1...
Connected to mail.example.com.
+OK The Microsoft Exchange POP3 service is ready.
USER yiming
-ERR Command is not valid in this state.

or you’ll get a rather brusque brushoff

Trying 10.0.1.202...
Connected to mail2.example.com.
Escape character is '^]'.
USER yiming
Connection closed by foreign host.

When this is encountered, OpenSSL’s s_client should be used instead to perform the necessary SSL negotiations.

openssl s_client -connect mail.example.com:995

or

openssl s_client -crlf -connect mail.example.com:110 -starttls pop3

The second incantation is typically used for Microsoft Exchange servers. Note the -crlf option, which tells s_client to send \r\n line endings. If the wrong line ending is used for a server, the symptom is that the server will not respond to any commands. It will only sit there and wait for further input, while you are staring at a blank responses or blank lines in your telnet session.

Authentication

Having established a connection, it is now necessary to authenticate as a POP3 user. In the simplest case, plain text authentication is used. In this case, the command USER [username] is used to establish the username, and PASS [password] is used to establish the password in plaintext. (Since the connection is under SSL encryption, presumably this plaintext won’t matter).

+OK Server ready
USER yiming
+OK
PASS foobar
+OK Logged in.

Server interactions

Several commands are useful here.

• LIST – lists the messages available in the user’s account, returning a status message and list with each row containing a message number and the size of that message in bytes
• STAT – returns a status message, the number of messages in the mailbox, and the size of the mailbox in bytes
• RETR [message_num] – returns the message identified by the message number, which is the same as the message number shown in the LIST command output
• TOP [message_num] [n] – returns the top n lines of the message denoted by message number.

When finished, the QUIT command will end the session.

Conclusion

For other POP3 commands, such as commands marking deletion of a message, refer to RFC 1939, the canonical document defining the Post Office Protocol Version 3 ( POP3 ). At some point, if the commands to be tested become complicated, it may be more efficient use of time to install a mail client such as alpine.

See also my previous post on chatting with HTTP / HTTPS servers.

What’s less new is the sudo path inheritance issue — this one’s been around. Ubuntu’s sudo hard-codes its PATH variable at compile-time with a --secure-path option. I’m sure this sounded like a good idea to the security goon who decided to fix this at fsckin’ COMPILE TIME with no way to override it in sudoers, or at runtime with -E after an env_reset. The policy may have been reasonable when it was set on a typical Debian stable server (where software is basically left to fossilize over decades), but certainly not on a constantly changing desktop distro. You can’t even sudo to any /opt/bin binaries! Read the Ubuntu bug report on sudo not preserving PATH.

Long story short, after a lot of experiments looking for workarounds (that won’t eventually take years off my life, one sudo command at a time), I decided to cut the Gordian knot and recompile sudo. Since I didn’t want to roll this from source (and incur all the maintenance hassle of removing/updating the software later on), this meant figuring out compiling source packages with dpkg — oh joy.

Debian source package compilation: the general process

It’s surprisingly non-painful compared with my RPM experience. The long way around:

1. cd into a temp or source-keeping directory in your user account
2. retrieve the source package: apt-get source [packagename]
3. grab missing build dependencies: sudo apt-get build-dep [packagename]
4. cd into the directory created for the package in your pwd (you can safely ignore the original tarball and the patch file, which have been untarred and applied for you already, respectively). Make edits to the source as needed.
5. If you need to change configure options for the source package, look in the file debian/rules in the source directory
6. when satisfied, build the binary package by issuing this incantation in the $PWD ( you’ll need the fakeroot package if you don’t already have it ):
   dpkg-buildpackage -rfakeroot -uc -b
7. The completed .deb packages are placed in the parent directory, one level up from the source directory. cd back up one level.
8. install: sudo dpkg -i [packagename].deb

If you’re screwing around with sudo, you will want to have a sudo tty session open before installing your replacement package, in case you screw up everything and lock yourself out.

A shortcut is potentially available using the -b switch to apt-get when you grab from source. However, I needed to look through configuration files and source code, so I took the long way around.

The easiest way to fix the sudo secure_path issue is to remove the --with-secure-path configuration option in debian/rules, in two places in that file. If you do this, pay attention to your $PATH and make sure they are sane (for example: it shouldn’t contain a globally writeable directory), as it will be inherited in sudo shells. In sudo 1.7, there is a runtime secure_path option for the sudoers file, so that would be the ideal, non-annoying solution to this issue.

Hard-coding the sudo PATH at compile-time tilts heavily toward security in the security/usability tradeoff — YMMV, but I find it entirely not worth it on a desktop distribution.

echo $?

Given the nature of this variable (no one indexes text like ‘$?’), it’s annoyingly hard to Google for.

There appears to be some sort of limitation on screen colors when using ARD 3.2.2 to control an older Mac remotely. The symptom of this is a black screen when you attempt to Observe or Control the remote machine. Unfortunately, this same symptom usually pears when you have a blocked network port (ARD uses TCP and UDP ports 3283 and 5900), so it may be confusing as to which is the issue.

After verifying all network settings and router port forwardings are set up correctly, you might try this if you have an older Mac as the target: move the color slider on the top-right corner of your ARD admin panel to a lower value, and then try to reconnect.

The story is that I was trying to remote control a G4 dual 500 ( Mystic ) from a MacBook Pro (early 2008). This used to work until recently, when I had nothing but a black screen. Keyboard commands still worked (I can blindly log in from the loginwindow), though mouse movements did not pass through to the old Mac.

After fruitlessly chasing network issues with my AirPort router, the last post at an Apple Discussions thread pointed me to the right direction. Once I used the color control on the ARD application to lower the color depth by 1 notch, the next connection worked just fine, with the screen showing up and behaving normally.

Now this poor old G4 tower is running 10.4.11 with an ancient Rage 128 Pro graphics card, but it handles its 17inch screen just fine at “million of colors” color depth when sitting in front of it. Very odd how it just stopped working at that depth over ARD.

01/24/2010 – DocPreview updated for 64-bit Safari and 10.6. Still seems to work.
—

DocPreview is a lightweight WebKit browser plugin I wrote to display a text-only preview of Microsoft Word .doc documents, inline and within Safari 3.x (and, apparently, 4.x) on both Intel and PowerPC Macs. This behavior is much like the functionality provided by the PPC-only Word Browser Plugin from Schubert|IT. DocPreview, of course, is a universal binary plug-in — since Word Browser’s lack of Intel support is really why I wrote this plug-in.

DocPreview only supports WebKit-based browsers that can use .webplugin files. I am currently unable to support other browsers, since DocPreview is built against the WebKit API instead of the Netscape plug-in API. Frankly, the Netscape API is a mess, and I haven’t had the time to figure it out. A NSAPI guru who could point me in the right direction would be much appreciated.

Download
DocPreview.zip (10.5+) – v0.1
DocPreview.zip (10.4) – v0.1

This is a proof-of-concept release. It passes the worked-for-me test — as in, I’m using this plug-in daily. While I believe it functions correctly, I make no guarantees as to stability and usefulness, and am not liable for blowing up your browser or any harm that might befall you through your adventurous use of this plug-in. Although, I do want this thing to work for everyone, so please leave me a note here if it doesn’t work.

(UPDATE 02/24/2009) – I see a few downloads already in my server logs, but no comments at all on this post. Can I assume it works for all of you out there? Or was it so broken that no one’s browser lived to tell the tale?

DocPreview features & limitations:

• Universal binary support, for use on both PPC and Intel Macs. No Windows Safari support — and plus, there are already good solutions for inline doc browsing on the Windows side. If you’re on PPC, I suggest you use Word Browser Plugin instead unless you desperately need find-on-page using Safari’s built-in facilities with Word docs.
• Tested extensively on Safari 4.0b1, 3.2.x, and somewhat with Shiira 2.x. May also work on earlier Safari versions, but I simply do not have the ability to test the plugin against them.
• Uses OS X’s internal engine for opening and processing Word files. DocPreview performs as well (or as badly, depending on your opinion) as OS X itself for the same task.
• Supported on OS X 10.5 for .doc, .docx, and .odt files. On OS X 10.5, the plugin can parse Microsoft’s new OpenXML (.docx) and OO.org’s ODF Text (.odt) documents. On 10.4, the plugin will still work, but only for .doc files.
• Support for full document view mode or embedded mode (if the page author uses object and embed tags, like with Flash objects — although, who actually does that with .doc files? )
• In full document view mode, DocPreview uses Safari’s built-in Find and Text Zoom abilities. Any command that can be run on a normal Safari webpage can be performed on a DocPreview rendering.
• In embedded mode, Find on Page and Text Zoom support are implemented separately. I’m still thinking about how I can hook into Safari’s built-in system from within an embedded plugin. Since no one ever uses embedded .doc files anyway, the point is fairly moot for most users.

To install, drag DocPreview.webplugin into
/Users/<your name>/Library/Internet Plug-ins
or
/Library/Internet Plug-ins.

Screenshots of the plug-in in operation can be seen to the right of this post. The first pair of images are .doc files in full document mode in the browser and in Word 2008. The second pair of images are two documents embedded inline using object tags, and one of them shown in Word 2008. As you can see, the conversion fidelity is fairly decent — this is the same level of fidelity that you would have gotten by importing Word data into TextEdit, or using textutil on the command-line.

DocPreview serves the same purpose as Word Browser. It’s intended as a quick preview (much like how Google indexes the text in .doc files), so you don’t blindly download any Word files that actually don’t interest you.

( As an aside, I cannot understand people who want to disable built-in PDF support in Safari :p . That feature has singlehandedly improved my productivity/research output by a magnitude. )

In any case, this is a quick hack, since I expect Apple to support .doc previews with the next version(s) of Safari, very soon. MobileSafari on the iPhone already provides .doc viewing support (and .xls, if I remember correctly). On Windows, MS provides read support for .doc files within the browser. DocPreview is just a temporary solution for Safari users until official Apple support (or better yet, better Microsoft Office integration) arrives.

If the thing doesn’t work for you, let me know via the comment thread. If it does work, let me know too Any suggestions and comments welcome.

Except, of course, the Apple AirPort and AirPort Express routers don’t actually have this feature. This is understandable, considering that MAC addresses are supposed to be globally unique so as to avoid network problems — allowing the user to change it willy-nilly will probably not serve that goal. Nevertheless, it is quite annoying in this case, as you won’t be able to change IP with an AirPort router using the MAC cloning method.

There’s one trick that might work for you here, though. A few ISPs’ DHCP servers are set up to accept DHCP Client IDs — an optional field that identifies a DHCP client. While they default to use MAC address as a client identifier by default, they will treat you as a different DHCP client for IP assignment if you start using or change a Client ID. All that is required to obtain a new IP address, then, is to add a Client ID or change it. There is an interface for this in the AirPort Utility, as seen in the screenshot. This is far easier than trying to clone a MAC address on AirPort routers.

Not all ISPs support use of the DHCP Client ID, so this may not work for everyone. Since there are uniqueness requirements involved with these client IDs, it is a good way to screw up DHCP assignments if two clients claim the same ID. If your ISP does support this, make sure to pick a unique client ID.

This trick appears to work for Cox Cable at my current location. It does not appear to work for Comcast.

The symptoms

Google has a nifty query suggestion feature that is turned on by default on its homepage search box. Whenever I typed in a phrase query (e.g. +"query suggestion" +"Google features") with the suggestion feature turned on, the browser crashed with a SIGSEGV around 30% of the time.

Excerpt from the crash log:

Exception Type:  EXC_BAD_ACCESS (SIGSEGV)
Exception Codes: KERN_INVALID_ADDRESS at 0x000000001bdca240
Crashed Thread:  0

Thread 0 Crashed:
0   ???                            0x16619e75 0 + 375496309
1   com.apple.WebCore              0x94325ea0 WebCore::AutoTableLayout::fullRecalc() + 704
2   com.apple.WebCore              0x9432581a WebCore::AutoTableLayout::calcPrefWidths(int&, int&) + 26
3   com.apple.WebCore              0x943252b8 WebCore::RenderTable::calcPrefWidths() + 56
4   com.apple.WebCore              0x9431c04b WebCore::RenderBox::minPrefWidth() const + 27
5   com.apple.WebCore              0x9432507c WebCore::RenderTable::calcWidth() + 124
6   com.apple.WebCore              0x943241a8 WebCore::RenderTable::layout() + 392
...

In the remainder of the cases, when it does not crash immediately, a JavaScript error is logged to the browser error console (to access, go to Develop -> Show Error Console)
SyntaxError: Invalid regular expression: nothing to repeat
http://www.google.com/extern_js/f/CgJlbhICdXMrMAc4AiwrMAo4EywrMA44AywrMBg4Ayw/nMD0sKnpeG0.js (line 21)
for every letter that I type into the search box. During this time, no query suggestion is made.

Diagnostics

• I have never used an InputManager or “plug-in” to Safari
• The same crash does NOT happen under a fresh new user account created for diagnostic purposes
• Clearing the browser cache, temp files, hidden cache files ( getconf DARWIN_USER_CACHE_DIR ), etc. did not help.
• Deleting Safari preferences did not help.

Solution

After applying a divide-and-conquer strategy to the entire ~/Library directory (not made any easier by Finder’s obstinate resistance to my attempt to move subdirectories within the Library directory, despite having the appropriate permissions — had to drop to Terminal for this), I traced it to the ~/Library/Cookies directory. Moving away the Cookies.plist file contained within cured the crash, the lack of query suggestions, and the Javascript error. More specifically, deleting all Google-related cookies within the Cookies file also accomplished the same thing.

Remarks

Some combination of a bad cookie and bad regexes appears to have triggered a crash bug in this version of WebKit / WebCore. You wouldn’t think a bad cookie could take down a browser. But apparently it does.

I dearly hope this is not a potential buffer overflow or other security problem within WebKit.

A redacted transcript:

HELO example.com
250 Hello
MAIL FROM: foo@example.com
555 5.5.2 Syntax error.
MAIL FROM: <foo@example.com>
250 2.1.0 OK
RCPT TO: bar@example.com
555 5.5.2 Syntax error.
RCPT TO: <bar@example.com&gt
250 2.1.5 OK
DATA

....

I’m sure this is intended so that clients that send “Joe Schmoe <joe@example.com>” are correctly interpreted. I’m not sure if the RFC asks for this strict enforcement (on my admittedly cursory reading), though and yet a fair number of servers are rejecting the alternate address-only syntax. Wacky.

This post is my attempt to catalog all the event stills and how to obtain them in game. The events are listed below in order of their appearance on the View Event Stills page. In general, you only need to view the Event Still once. That means that you can start a game, obtain the Events you want, without actually completing or even saving the game once. Even if you abandon that game later, once you’ve viewed the event, it counts toward your completion of collecting all 30.

Note that I will not suggest any unreasonably contrived means to get events, even if that might be faster, because I consider these tactics basically cheating. For example, I will not suggest that you send your sovereign out to die deliberately to get the Sovereign Death event, even if you might be able to do that. You may also create officers with birth and death dates to your exact specification, to achieve some of these events, but that, too, is basically cheating. Most of the fun in these events is trying to obtain them fairly while playing a game — it adds an additional dimension of strategic planning. If you’re just going to cheat, stop reading this and go download the fully unlocked sysfile that’s floating around the Web.

Page 1

• Plague – when a city receives a Plague event, causing all sorts of havoc. You will likely get at least one in any scenario.
• Locusts – when a city receives a Locusts event, decreasing food production in that city. You will likely get at least one in any scenario.
• Plenty – when a city receives a Plenty event during harvest season, increasing food production in that city. You will likely get at least one in any scenario.
• Bandits – when bandits appear in your territories, as the name suggests. The easiest way to obtain this still is to start the Yellow Turban Rebellion scenario. During the story intro sequence, the Yellow Turbans will rise against the Han court, and this will unlock the event still.
• Coronation – when an Emperor is crowned, or when you declare support for the Emperor after capturing the capital (Luoyang, Chang’an, or Xu Chang, depending on the scenario). Easily achievable in most scenarios if you’re winning.
• Unity: Male – when you unite China under a male sovereign. This event will be unlocked when you win a scenario with a male sovereign. I prefer Liu Bei, but YMMV
• Unity: Female – notes not yet available
• Downfall – when one of the forces in the game is eliminated. At least one ruler will fall in any game, so this is trivial to obtain. You can also get this by losing your city in Tutorial 3, Going into Battle.
• Independence – when a prefect or viceroy declares independence and forms his or her own force in a game. In Tutorial 6, Capturing Cheng Du, there is a good chance that Jian Ning will declare independence under the prefect Yong Kai, but it is not guaranteed. In the 190 scenario, the free cities in Jiangdong may declare independence under their prefects within the first two years. In any scenario, foment an independence movement in an enemy city via the strategy Collaborate. Alternatively, wait until one of your district viceroys’ or city prefects’ loyalty falls below a certain threshold, at which point an AI force is likely to foment a revolt in that prefecture or district. Finally, when you’re winning a full scenario, at the very end of the game one of your officers will ask you to investigate another officer for treason. If you choose to do so, that officer will be proven innocent, but will revolt anyway due to your distrust of him. This revolt will trigger an Independence event.
• Omen – This event is displayed at the start of the year, when your strategist warns that one of your officers is about to die (of natural causes, not in battle). If you play through a game with historical age settings, quite a few of your officers will die. Pick a middle scenario, when there are a lot of older officers around.
• Sovereign Death – when your own sovereign dies of natural causes. The easiest way to get this one, short of deliberate regicide on your part, is to use Tao Qian in 194, Rival Warlords. He will most likely die within the next two years, giving you this event.

Page 2

• Officer Death : Male – When one of your male officers die. This is relatively trivial to get – one will come within the year, after an Omen, for example. Your officer may also be killed by a cavalry charge or in a duel, if officer deaths are on.
• Officer Death : Female – When one of your female officers die. This is a little more annoying, given the sparsity of female officers. Still, at least one should die in any game. Your officer may also be killed by a cavalry charge or in a duel.
• Funeral – When any sovereign (except yours) dies of natural causes — that is, of illness or old age, not in battle. In the 194 scenarios, Tao Qian will die within the next two years. Play the scenario as a ruler other than him, and you will trigger this event still.
• 2 Sworn Brothers – when you arrange for two male officers to swear brotherhood with each other. Officers with sworn brothers on the same force will not defect, and will obtain bonuses when both are assigned to the same unit. You’ll need two officers of the male gender and at least 500 in Deeds each. You will also need 500 technique points (TPs). At that point, Sovereign -> Mediate will allow you to mediate sworn brotherhood between these two officers and unlock this event.
• 2 Sworn Sisters – when you arrange for two female officers to swear sisterhood with each other. Officers with sworn sisters on the same force will not defect, and will obtain bonuses when both are assigned to the same unit. You’ll need two officers of the female persuasion and at least 500 in Deeds each. You will also need 500 technique points (TPs). At that point, Sovereign -> Mediate will allow you to mediate sworn sisterhood between these two officers and unlock this event.
• Male Officer Appears – Triggers at the start of a new year, when a male officer (who is related to one of your current officers) arrives to offer his services to your ruler. For example, as Liu Bei in 211 Liu Bei in Shu scenario, you will likely receive Zhang Bao – Zhang Fei’s son, in 212. Note that your sovereign should be within a city for this event to occur.
• Marriage – when you marry one of your officers to another – married officers will not defect, and obtain bonuses when assigned to the same unit. You’ll need two officers of opposite gender, at least 500 in Deeds, and unmarried status. You will also need as 500 technique points (TPs). At that point, Sovereign -> Mediate will allow you to conduct a marriage between these officers and unlock this event.
• Fem. Officer Appears – Triggers at the start of a new year, when a female officer (who is related to one of your current officers) arrives to offer her services to your ruler. This one is fairly tricky, because there aren’t that many female officers. In general, play one of the middle scenarios as a prominent family. For example, if you play as Sun Quan in 211, you will get a female officer from the Sun family in 215.
• Resignation – when an officer resigns from your force due to low loyalty. This is also a hard one to get if you are a decent player, since you’d never let an officer’s loyalty drop low enough. Note that if your sovereign dies and your officers resign, you will not receive this event still. You must have this officer resign through low loyalty. Officers with loyalty less than 70 will tend to resign, and they usually do so on Jan of a new year. Best way to do this may be to play as Dong Zhuo in 190. Use your overwhelming starting forces to take down Cao Cao, then hire his wife Bian Shi. Use Bian Shi to hire Cao Cao. Dong Zhuo and Cao Cao are disliked officers, so Cao Cao’s loyalty will start in the 70s and drop fast. However, since you have his wife in your service, he will not be hired away by another sovereign. At the start of the next year, Cao Cao will most likely resign from your force.
• Decree – when an imperial decree arrives. For example, when the Emperor promotes your sovereign to the rank of Duke, he will use this kind of decree. Since to win a game, you’ll likely surpass the rank of Duke, this event is trivial to unlock if you win a game.
• Orders – when a different type of imperial order arrives. If you start 190 Cao Cao's Ascent, the story intro sequence to the scenario involves Cao Cao’s forged imperial edict arriving at Yuan Shao’s city. This will grant you the event still.

Page 3

• Fallen Castle – when a city falls to your forces. You are likely to receive this in Tutorial 6, Capturing Cheng Du…when you capture Cheng Du. Otherwise, take any city or base, and the event is yours. Trivial.
• Conversation – when one of your civil officers head off to search for a friend in the area and obtains his or her services after a debate. This seems to occur randomly, when your sovereign is in the same city as certain civil officers on his force. It is the analogous event to “Search”, which applies only to military (high WAR) officers. The event represents the “conversation” between your officer and his friend. This obviously only triggers when there are hidden officers to find in a city — so start an early scenario with lots of hidden officers for a better shot at the event.
• Coalition – pick up this event when the coalition against Dong Zhuo forms in 190 Cao Cao's Ascent, at the very beginning during the story intro sequence. Otherwise, play as a strong army (say, Cao Cao in 207), and eventually two of your weaker enemies will merge to form a coalition against you.
• Search – when one of your military officers head off to search for a famous hero in the area and obtains his or her services after a duel. This seems to occur either randomly or scripted to resemble randomness. This obviously only triggers when there are hidden officers to find in a city — so start an early scenario with lots of hidden officers for a better shot at the event. The 190 scenario with Cao Cao in Xu Chang or Chen Liu should be an easy way to get this event, as you recruit various notables like Dian Wei or Xu Chu.
• Secret Meeting – when you meet a hidden character for a philosophical discussion. This one is ridiculously obscure and difficult to obtain, to the effect that many people do not know how to obtain it. Forum posts and such suggest that you’d receive this when the AI foments unrest with one of your viceroys or prefects (one of the in-game log messages say “your prefect blah is meeting someone secretly”. However, that is a red herring. One post correctly identifies that the Secret Meeting event is awarded for meeting a hidden character such as Xu Shao during a Search. As Cao Cao in 190 Cao Cao's Ascent, if you keep Searching the area around Chen Liu or Ru Nan, against your strategist’s advice that nothing would be found, you will eventually encounter Xu Shao, who will do a physiogamy reading for you, unlocking this event. One of the early scenarios, such as Rise of Dong Zhuo, is preferred for obtaining this event. Just keep Searching your city, a lot, and fruitlessly most of the time. You’ll get this eventually ( took me under two game years of on-and-off searching in Chen Liu while holding off Dong Zhuo’s army ). A comment from Lance Lidner reports that using Kong Zhou in Rise of Dong Zhuo works best, as Xu Shao appears in Ru Nan at that time. Kong Zhou is also out of immediate reach from most hostile factions in that scenario, so you can afford to take your time.
• Appoint – when the emperor issues a new title to you. This occurs when you’ve captured enough cities to reach the next rank, and that your sovereign is currently in a city (as opposed to being on campaign). As Liu Bei in the 211 Liu Bei in Shu scenario, taking Xiang Yang (via Jiang Ling – you’ve got Zhuge Liang, Pang Tong, Zhang Fei, Liu Bei, Sun Shang Xiang, Zhao Yun … that’s plenty to take the city from Cao Ren early on) and Yong An will bump you up to Field Marshal, and Emperor Xian will award you the title and this event. As Cao Cao in 190 Cao Cao's Ascent, taking the free cities of Xu Chang and Xin Ye will grant the title of Lt. Governor, which once again easily unlocks this event.
• 3 Sworn Brothers – when you arrange for three male officers to swear brotherhood, much like the historical Oath of the Peach Garden of Liu Bei, Guan Yu, and Zhang Fei. Officers with sworn brothers on the same force will not defect, and will obtain bonuses when two or more are assigned to the same unit. You’ll need three officers of the male gender and at least 500 in Deeds each. You will also need 500 technique points (TPs). At that point, Sovereign -> Mediate will allow you to mediate sworn brotherhood between these officers and unlock this event.
• 3 Sworn Sisters – when you arrange for three female officers to swear sisterhood. Officers with sworn sisters on the same force will not defect, and will obtain bonuses when two or more are assigned to the same unit. You’ll need three officers of the female persuasion and at least 500 in Deeds each. You will also need 500 technique points (TPs). At that point, Sovereign -> Mediate will allow you to mediate sworn sisterhood between these officers and unlock this event. The hardest part of this is rounding up 3 female officers to begin with.

ActiveRecord::StatementInvalid: Mysql::Error: Column 'attr2' cannot be null: INSERT INTO `foo` (`attr1`, `attr2`) ... VALUES ('1', NULL)

where attr2 is a varchar (or t.string, in Rails lingo) and set to not null default '' (or, in other words, :null=>false, :default=>''). Strangely enough, instead of the default value of ”, ActiveRecord was setting the value to nil instead, which translates into a NULL. Since the schema explicitly forbids NULLs on that column, the statement explodes.

After an hour of poking around and hacking up a spike solution, it turns out a plugin was to blame. I’d pulled in the foreign_key_migrations plugin (a highly recommended add-on) to automatically install foreign key constraints (in this day and age, the foremost web framework still can’t automatically handle FOREIGN KEY constraints, the most basic tool for ensuring data integrity in relational databases, for its migrations? Bah!).

This plugin has a dependency on redhillonrails_core, which has a known bug: Incorrectly overwrites mysql empty-string default with nil for string/text/binary types.

The bug is apparently not being worked on as of the time of this writing. The dev doesn’t consider this a bug, as he claims that “[he considers] empty strings to be semantically identical to NULL”.

This position, unfortunately, is not supported by the SQL standard. Wikipedia has a section on common SQL NULL mistakes documenting some of the potential problems involved in making such an assumption. Philip Greenspun has further notes on this. Using NULLs trigger the all the arcane annoyances of three-valued logic, and you must be prepared to consider True, False, and NULL values as comparison outcomes. Someone not very versed in three-valued logic can easily cause a number of subtle mistakes trying to compare values.

To workaround this bug, you will need to comment out the initializer function in

vendor/plugins/redhillonrails_core/lib/red_hill_consulting/core/active_record/connection_adapters/mysql_column.rb

Alternatively, you can delete the file, and remove relevant references to it..

If you agree with dev’s position that NULLs are “semantically identical” to empty strings, remember to pay attention when you formulate your SQL queries (and when Rails formulates those queries) — your results may not be what you expect, if implemented naively. Get your three-valued logic truth tables out

/bin/sh /tmp/subversion-1.5.3/apr/libtool --silent --mode=compile gcc-4.2 -Os -arch i386 -arch x86_64 -DHAVE_CONFIG_H -DDARWIN -DSIGPROCMASK_SETS_THREAD_MASK -no-cpp-precomp -I./include -I/tmp/subversion-1.5.3/apr/include/arch/unix -I./include/arch/unix -I/tmp/subversion-1.5.3/apr/include/arch/unix -I/tmp/subversion-1.5.3/apr/include -o strings/apr_snprintf.lo -c strings/apr_snprintf.c && touch strings/apr_snprintf.lo
strings/apr_snprintf.c: In function ‘conv_os_thread_t’:
strings/apr_snprintf.c:500: error: duplicate case value
strings/apr_snprintf.c:498: error: previously used here
strings/apr_snprintf.c: In function ‘conv_os_thread_t_hex’:
strings/apr_snprintf.c:671: error: duplicate case value
strings/apr_snprintf.c:669: error: previously used here

This will most likely happen when you are configured to build a dual 32-bit / 64-bit universal binary, whether it be ppc / ppc64, or i386 / x86_64, or any permutation thereof. This ticket over at MacPorts documents a particular instance of this problem, with no apparent solution.

The symptom is easy to explain. Somehow, two case labels in the relevant switch statement in strings/apr_snprintf.c:500:

switch(sizeof(u.tid)) {
    case sizeof(apr_int32_t):
        return conv_10(u.u32, TRUE, &is_negative, buf_end, len);
    case sizeof(apr_int64_t):
        return conv_10_quad(u.u64, TRUE, &is_negative, buf_end, len);
    default:
        /* not implemented; stick 0 in the buffer */
        return conv_10(0, TRUE, &is_negative, buf_end, len);
    }

have evaluated to the same value. In particular, it believes that sizeof(apr_int32_t) and sizeof(apr_int64_t) are the same value. As we all know in C, you cannot have two identical case labels in the same switch statement. However, the root of the problem is a bit more subtle.

In $SRCDIR/include/apr.h, you’re likely to see this fragment of code.

typedef  long       apr_int64_t;
typedef  unsigned long  apr_uint64_t;

Notice that it has typdef’ed apr_int64_t as a long and apr_uint64_t as unsigned long. This is because at configure time, the script detected that long values are 64-bit on this system, so it assigned the apache 64-bit types to longs. However, this only holds true for half of the compilation – because you are building a universal binary for a 32-bit architecture as well. Remember that in 32-bit GCC on OS X, longs are 32-bit rather than 64-bit. Your run-of-the-mill autoconf script, done by a non-OS X programmer, isn’t going to be able to detect this subtlety – if the 64-bit part worked, it’ll keep thinking longs are 64-bit, end of story – and happily generate the incorrect typedef expressions. When you apply sizeof to these types in apr_snprintf.c, both evaluate to 4 bytes under 32-bit compilation, thus blowing up the compile run.

To truly fix the root of the problem requires rewriting the autoconf script to detect Mac OS X and its universal binary building, which can potentially throw quadruple architectures at the same compilation script. However, a quick hack to make this particular problem go away is to change apr.h such that:

typedef  long long       apr_int64_t;
typedef  unsigned long long apr_uint64_t;

Now that we ensure in either 32-bit or 64-bit compilation, apr_int64_t and apr_uint64_t are always typedef’ed to appropriate, guaranteed 64-bit types. The compilation of APR (and Subversion) will proceed normally.

Note that long long is not an standard C type. As a GCC extension, this fix is a kludge. A kludge that works (for me), though.

UPDATE:
There may also be an issue with sizeof definitions that may cause the library to crash. In particular, there may be occurrences of

#define APR_SIZEOF_VOIDP 8

that were generated by configure. To fix this, you will need to remove the define and have the compiler check for 64-bit at compile-time:

#ifndef __LP64__
    #define APR_SIZEOF_VOIDP 4
#else
    #define APR_SIZEOF_VOIDP 8
#endif

In general, any predefined sizeofs need to be changed. I am not sure why the APR developers do hard-coded defines like this, given that the point of having sizeof() calls is to avoid such issues.

Problem 1: transcoding to mp3 insists on using 64kbits/s or 128kbits/s bitrate

So a simple task is to transcode an audio file of some arbitrary format to MP3. In my case, I only wanted a 30-second piece of the original file, converted to MP3. The original was in 256kbits/s, and I decided to use the same bitrate for the output, just for kicks. Reading the man page, -b is for video bitrate, and -ab is for audio bitrate. So I executed:

wrong:  ffmpeg -ss 00:00:30.00 -t 25 -i foo.mp3 -ab 256 foo-new.mp3

The output, of course, was this:

Input #0, mp3, from 'foo.mp3':
...
mdb:109, lastbuf:0 skipping granule 0
size= 393kB time=25.2 bitrate= 128.0kbits/s
video:0kB audio:393kB global headers:0kB muxing overhead 0.007950%


128 kbits/s. That was…not what I wanted.

Turns out, as the ffmpeg man page hints but does not specify clearly, a “k” is required to label the units of the new bitrate. As in:

right:  ffmpeg -ss 00:00:30.00 -t 25 -i foo.mp3 -ab 256k foo-new.mp3

This time, the output was:
Input #0, mp3, from 'foo.mp3':
...
mdb:109, lastbuf:0 skipping granule 0
size= 782kB time=25.0 bitrate= 256.0kbits/s
video:0kB audio:782kB global headers:0kB muxing overhead 0.003996%

Much better. It’s very strange how that if I don’t give it the ending “k” but do give it a higher value than 64, it always bumps up the bitrate to 128kbits/s, from the default of 64k (but not to the number I actually wanted).

Problem 2: cutting media file without re-encoding

So ffmpeg can be used to cut a media file, without reencoding the media stream. You simply pass the raw copy codec to the -acodec (or the -vcodec for video) at encode time. For my MP3, I thought it was a pretty trivial problem, so I issued:

wrong: ffmpeg -i bar.mp3 -ss 00:00:30.00 -t 25 -acodec copy bar-new.mp3

This seemed to process correctly, except it created a 555-byte empty file with no content in it. What’s even weirder, if you issued:

wrong: ffmpeg -i bar.mp3 -ss 00:00:10.00 -t 25 -acodec copy bar-new.mp3

That is, -ss 00:00:10.00 to seek to the 10th second, and -t 25 to record 25 seconds worth of audio. Strangely enough, the output file had 15 seconds of audio, the subtraction of 25 by 10. Curiouser and curiouser. Now,

wrong: ffmpeg -i bar.mp3 -ss 00:00:10.00 -t 25 -acodec mp3 bar-new.mp3

does in fact create a 25-second MP3 file. Only, of course, a re-encoded one at 64kbits/s. Something that we are not looking for. But strange.

A post on ffmpeg-devel finally provided enough hints to clue me into the problem. Unlike many command line apps, the order of arguments passed to ffmpeg seem to be silently significant. The correct incantation is:

right: ffmpeg -ss 00:00:30.00 -t 25 -i bar.mp3 -acodec copy bar-new.mp3

Note the switched order for the arguments -i and the args -ss and -t, where the -i must follow the other two. Now, the desired 25-second file, cut from the original, is correctly produced.

When arguments are missing some arbitrary text or in the wrong order, ffmpeg doesn’t sanity-check or warn you of these…it just silently proceeds and does some very strange things. Things that make you scratch your head and wonder, “wtf? did I mistype an argument somewhere?”

Ah, and of course, you make me have to track down all these little idiosyncrasies, wading through blog posts and mailing lists and forums. Agh.

You may get this error while trying to compile Subversion 1.5.2:
ld: Undefined symbols:
_svn_fs_txn_root_base_revision referenced from libsvn expected to be defined in libsvn
_svn_fs_change_txn_props referenced from libsvn expected to be defined in libsvn
_svn_fs_get_mergeinfo referenced from libsvn expected to be defined in libsvn
_svn_fs_recover referenced from libsvn expected to be defined in libsvn
_svn_fs_upgrade referenced from libsvn expected to be defined in libsvn
_svn_fs_node_origin_rev referenced from libsvn expected to be defined in libsvn
/usr/bin/libtool: internal link edit command failed
make: *** [subversion/libsvn_ra_local/libsvn_ra_local-1.la] Error 1

Turns out there may be a bug in the libtool script shipped with the source tarball, under certain circumstances with OS X 10.4, that causes it to fail to link. Ah, libtool, you make my life so much harder sometimes.

If you copy /usr/bin/glibtool over $SRCDIR/libtool (that is, into the directory for the subversion source code, replacing the one that is placed there by the package itself), the package should compile with no further complaints. make test also shows success on all tests, so this seems a satisfactory solution.

UPDATE 1/31/2009
I’ve been informed in the comments that MacPorts may rely on source compilation from tarball, and thus have this issue. If you’re having this issue with the MacPorts SVN package, please check out the advice in the comment section from farkinga, who has additional notes.

This error has nothing to do with the particular code you’re compiling. Turns out GNU Make 3.80 has a bug, in which it sometimes sends a ridiculous requested block size to malloc(), which then fails to allocate the space.

The solution is to upgrade to GNUMake 3.81. Particularly pernicious on OS X 10.4 and Xcode 2.4.1.

So recently I was trying to interact with an avi file with a video stream encoded in H.264 and an audio stream encoded in MP3. To my surprise, my trusty Perian 1.1 failed to even play the video and gave me a black screen (though the audio played just fine). With some Google-fu, it was found in a forum post that the SVN version of Perian fixed this error.

So compile my own Perian 1.1.1b1 from SVN, right? No big deal. Heh. And thus begins our story. (Note that this procedure is accurate as of the time of writing. OSS projects change rapidly as bugfixes are checked in. You may not have this problem at all. But it’s still useful as a reference in the future, in case similar problems arise).

Problem 1: AVCodecContext.bits_per_sample

The first problem you hit is probably:
/tmp/perian-orig/ff_private.c:181: error: ‘struct AVCodecContext’ has no member named ‘bits_per_sample’
…
/tmp/perian-orig/FFissionCodec/FFissionDecoder.cpp:289: error: ‘struct AVCodecContext’ has no member named ‘bits_per_sample’

Turns out that the upstream project FFMPEG, who is responsible for libavcodec, has very recently changed the API in one of its latest revisions. bits_per_sample has become
bits_per_coded_sample in revision 15262. Because Perian’s SVN does an external SVN checkout of the HEAD of libavcodec, it pulled these API-breaking changes without considering whether that breaks its own compilation or not.

The fix is simple enough: global search and replace the member variable to bits_per_coded_sample.

Problem 2: Xcode’s unnatural fondness for dynamic libraries

At this point, libavcodec compiles, so you think you’re home free. For most users, that is probably the case. For some of us, though, there’s one more hurdle.

While compiling Perian.component, you might get something like:
ld warning: in /Developer/SDKs/MacOSX10.4u.sdk/usr/local/lib/libavcodec.dylib, file is not of required architecture
Undefined symbols:
"_avcodec_close", referenced from:
…and so on

Closer examination reveals that at some point, you might have compiled FFMPEG or MPlayer or any of the many media projects that depend on FFMPEG libraries (for me, that was transcode, whose own compilation idiosyncracies I documented in an earlier post). When you installed these libraries, they went into your library search path. Note the Perian compilation line immediately preceding the compile failure:
/Developer/usr/bin/g++ ... -read_only_relocs suppress -lavcodec -lavformat -lavutil -lebml -lmatroska ... /tmp/perian/UniversalDetector/build/Release/libuniversaldetector.a -lbz2 -o /tmp/perian/build/Deployment/Perian.component/Contents/MacOS/Perian

As you can see, the -lavcodec linker argument will cause the linker go on the search path and try to find the library. It of course first finds your pre-installed library in /usr/local/lib or some other system lib path, and tries to use it. If you compiled that library as Intel only or PPC only and you tried to build a universal Perian, that would obvious fail and generate an architecture error. If you, like me, compiled only for x86_64 or ppc64, that would also trigger the same architecture problem (as Perian builds as x86 32-bit).

This problem stumped a user “vs”, in this Perian-discuss Google groups thread. Notice how the devs didn’t get the root of the problem, and answered snarkily and defensively to the user bug report.

The devs aren’t wrong, though. Didn’t we just see a statically compiled target, libavcodec.a, in the target list for the Perian Xcode project? Why yes, yes we did! In fact, there are static targets for all of the -lav* libraries in the Xcode project. So why did the linker go off and use -lavcodec rather than, say, $PERIAN_DIR/build/Universal/libavcodec.a, as instructed?

The workaround

In short: Xcode is screwing things up. Xcode 3.1 has a distinct (some say, unnatural) preference for dynamic libraries. When you add a library to the “Link Binary With Library” listing under Targets, it simply adds a -l<libname> to the link line to pass to ld at link time. When this happens, it will look for lib<libname>.dylib (in this case, libavcodec.dylib, libavformat.dylib, libavutil.dylib, etc in your library path. If it finds one, it will use that one rather than the static library you provided in the Xcode project file. Obviously, this ends up linking the wrong library, perhaps with the wrong architecture or the wrong version. Compilation explodes.

The workarounds are not very clean. There seems to be no good way to instruct Xcode to forget its dynamic library fetish and use the static versions you provide. You could remove the dylibs from the system library path entirely — when the linker fails to find the .dylib version, it will fall back to the .a.

Alternatively, you can explicitly instruct it to use static linking as an environment flag and remove the built-in Xcode linker relationships:

1. Right-click on your Target (in this case, Perian) and Get Info to bring up the Target Info box.
2. Go to the Build tab
3. Select your Build configuration – in this case, probably Deployment
4. Find or locate OTHER_LD_FLAGS, also called “Other Linker Flags”
5. Manually add the paths to the .a files in this field, one at a time. If you don’t know the path, right-click -> Get Info on the desired libraries under Link Binary With Library, and it should show you a field called Full Path on the General tab. That’s the path to use. Should look something like /tmp/perian/build/Universal/libavcodec.a
6. Do the same for all offending libraries – this may include libavcodec, libavformat, libavutil, libmatroska, etc.
7. Remove the libraries you added to Other Linker Flags from your Xcode’s Target -> Target Name -> Link Binary With Library section, to stop Xcode from auto-generating broken linker flags. This is important.

If done correctly, this should no longer find the wrong type of the library. Your compilation of Perian should proceed as normal.

The Moral of the Story

Xcode has some weird idiosyncratic ways of building its projects. It’s useful to poke at things and find out about these behaviors. Its unnatural preference for dynamic libraries over your explicit instructions is disturbing, though.

Don’t attack your users when they pose a problem, and drop the attitude of: ooo, we’re doing this for free, so fsck off, users. In both problems, the exact same attitude surfaced amongst a few of the devs. Arrogance (“we don’t have the responsibility to do anything for you”) and snarky dismissal (‘Well, there’s your problem. We don’t build libavcodec.dylib nor do we place it in /usr/local/lib.’) helps no one. No matter what you thought your build script did, the linker is heading off toward /usr/local/lib, and you better damn well find out why.

It’s obvious that there’s something peculiar going on with the link process, and the user tried to make it clear multiple times (“You missed my point. We build a .a file and we don’t put it in /usr/local/lib.”), but no one seems to pay attention. Following that thread, the user is increasingly frustrated and expresses that, and now the dev’s hammer of arrogance comes down hard ( “You looked like a prick here, to be honest.” — really? not your obtuse-ness or your blindness to a user need?)

When you have a “works-for-me” situation, it’s especially important to communicate effectively with the user. In most cases, you have no idea why it blows up for the user and not for you. Admitting it is fine — could be a very localized issue for that user, or could reveal a subtle systemic problem in your code – either way, it’s useful to have that established. I realize we’re all very busy people, but at the end of the day it’s still your project. Either acknowledge you have no idea what’s going on and solicit help and cooperation with the user who’s having this issue, or man up, put on your detective hat, and fix your own damn problems. The rest of us are busy with our own issues.

The Free Software culture seems to breed this sort of contempt for the user in some devs. Might be a paper in analyzing how corrosive this can be.

UPDATED: I certainly do not mean this last section as a criticism for all Free Software devs. The vast majority of people I’ve talked to in the various communities are really knowledgeable, helpful folks. There are just a few people in some cases, though, who seem to think that if the user can’t make the software work, it’s somehow the user’s fault. It’s a joint problem to be solved together, not “you’re an idiot if you can’t make it work”.

UPDATED 2 (9-29-2008): Perian 1.1.1 has been released. This should fix the original problem with H.264 and .avi files that prompted this post.


tcmemcpy.c:30:missing or invalid immediate expression `0b111' taken as 0
tcmemcpy.c:30:suffix or operands invalid for `and'
tcmemcpy.c:42:missing or invalid immediate expression `0b1000' taken as 0
tcmemcpy.c:42:suffix or operands invalid for `test'
tcmemcpy.c:52:missing or invalid immediate expression `0b1111' taken as 0


The problem here appears to be that the code in tcmemcpy.c under x86 and x86_64 relies on some inline assembly features not supported by all compiler/assemblers. Namely, the use of 0b (binary immediate) operands in assembly instructions is not apparently supported during compilation under Apple gcc and gcc-4.2.

Setting the CCAS flag did not seem to help. A quick fix, then, is to convert all binary immediates in the asm instructions to hex or decimal.

Thankfully, these operands only existed in tcmemcpy, so there’s not too much work. Look for 0bxxxx immediates and convert them to their hex or decimal equivalent. For example, 0b111 is 0×7, 0b1000 is 0×8, 0b1111 is 0xf. Thus, a line like:
and $0b111, %%eax # ... which is the number of bytes to copyn

becomes
and $0x7, %%eax # ... which is the number of bytes to copyn

Once these operands are modified, the compiler should no longer complain about invalid immediate expressions.

In any case, the latest CVS version of transcode should have resolved these problems already. The workaround should only be necessary if you still must compile the release tarball for 1.0.5.

Updated: There is a second potential problem if you were using ffmpeg-SVN and transcode 1.0.5, in which its hard-coded include lookup for avcodec.h always looks for ffmpeg/avcodec.h. The SVN version of ffmpeg has moved these headers to libavcodec/avcodec.h. A path patch for transcode 1.0.5 and ffmpeg is available, which basically boils down to changing the #include headers and the configure.in file to look for the new path.

Updated: A third problem has now cropped up, with the ffmpeg API change that moved AVCodecContext’s bits_per_sample to bits_per_coded_sample. A global search and replace seems to work for now. See this also in my post on compiling Perian.

If you’ve been having problems compiling various Unix packages on OS X 10.5.4, and that your compile run fails mysteriously with something like:

i686-apple-darwin9-gcc: /usr/X11/lib/libpng12.0.26.0.dylib: No such file or directory

One strange yet very likely explanation: your libtool archive file /usr/X11/lib/libpng12.la is lying about the location of the shared library for libpng12 — namely, that a file called /usr/X11/lib/libpng12.0.26.0.dylib exists and should be used for linking against libpng12. However, if you actually look in /usr/X11/lib, no such file exists – perhaps you might have libpng.0.24.0.dylib, but not .26. Therefore, packages that make use of this incorrect libtool archive metadata are suitably confused, causing the compiler to bail out when trying to link against this non-existent file.

Since libtool archive (.la) files are text-based, you can open it up in emacs. The quick and dirty fix to this is to simply change the offending library_names and the current and age properties to the correct numbers. In my case, the libpng sitting in /usr/X11/lib was .24, and so I string-replaced all the values in those three properties from 26 to 24. The compilation then proceeded normally.

The long term solution, of course, is to track down what put a wrong .la file there in the first place. I suspect Xcode 3.1 and Mac OS X SDK 10.5, which shipped with the latest iPhone SDK.

UPDATED 9/8/2008:
In the comments section, I’ve been informed that the X11SDK package in Xcode 3.1 in the culprit. Thanks Anonymous!

Of course, it was not obvious where Catan was trying to save its games – finding out that missing directory was the culprit took a bit of investigation. I took a wild stab at the start by creating a “save” directory in its own program files directory. No such luck. Time to bring out the big guns.

A number of ways could have been used, but one is to use the awesome Sysinternals tool Process Monitor, or Procmon.exe. It tracks events and calls from a process, such as filesystem accesses, and has advanced filtering capability to organize and show only the events of interest to a debugging human.

With ProcMon, I simply filtered on the Catan process and tried to save a game as foo. Then, viewing the event log (screenshot 1), it was obvious that CreateFile calls to create foo.sav failed, with the exact target path specified. A quick Windows Explorer excursion confirms that the path does not exist. Creating that directory, of course, solved the savegame problem.

The moral of the story is that ProcMon is a fine tool for tracking mysterious interactions between an application and a system. For something like failing to make a saved game (in this narrow gamig context) or various system-related errors in general (especially when you lack the source code to debug in depth), it sometimes pays to examine the exact sequence of calls and events that led up to the failure. The solution could be very trivial, if you only knew what and where things failed.

This has bothered me for a very long time. In essence, Office 2008 maps Command-G, the customary Mac OS keyboard shortcut for “Find Next” (in essence, to find the next instance of a string match and part of the set of standard Find/Replace [All] functions in any good text editor / word processor), to “Go To”.

To its credit, Word allows customization of its keyboard shortcuts. Unfortunately, I just could not locate the “FindNext” keyboard shortcut in Tools -> Customize Keyboard to apply customizations to. Turns out it’s actually named “RepeatFind”, as opposed to “EditFindNext”, which you’d expect when the “Find…” command is simply “EditFind”.

Further, it turns out that wouldn’t have mattered anyway. Microsoft overrides the keyboard shortcut selection behavior in View -> Customize Toolbars and Menus. It does so in a ridiculously arcane way, by which the menu name “&Go To” fixes the shortcut key to Command-G. Therefore, no matter what shortcut is set in Customize Keyboard, Command-G will always map to Go To and not Find Next. ARGH.

Even knowing this, it was not obvious how to edit the name either (screenshot). Apparently the solution is to use the fake menu bars that pop up after selecting Customize Toolbars and Menus, and then right-click or Control-click the Go To menu item. That will pop up an edit box with the name of the menu item, at which point you can remove the gratuitous &, such that Command-G will map correctly based on Customize Keyboard.

However, disabling hardware acceleration under Vista is apparently easier said than done. Instead of using the Personalization -> Display Settings control panel (as one might think to do based on Windows XP experience), the correct solution is to use the DirectX SDK and dxcpl.exe, the DirectX Control Panel (located within the SDK distribution under Utilities\bin\[cpu_arch]\. From within this control panel, pick DirectDraw on the upper tab bar. Amongst the various configuration options available on that tab, the only one you care about is the box to turn on or turn off hardware acceleration. Turn that off (temporarily, of course) and you’re good to go.

The Context

Baldur’s Gate 1 performs surprisingly well under Windows Vista, despite being a venerable (some might say, ‘ancient’) 10-year old RPG. Unfortunately it’s plagued by a number of graphics glitches when running on NVidia cards. In essence, a number of items and sprites (items on the belt, the timepiece to the lower left corner, birds flying overhead, for example) will be surrounded by black outlines. Further, on your character paper doll in the Inventory screen, giant black boxes obscure much of the figure. In some cases, the ‘fog of war’ on the unexplored regions of a map will be rectangular black boxes, rather than the ‘foggy’ darkness you’re used to. These glitches are widely experienced.

For this very annoying problem, two workarounds are available.

1. Trade color depth for correctness

The prevalent strategy, as noted in a forum post at Spellhold Studios, is to switch on Software Transparent BLT and use 16-bit color depth. This apparently routes around whatever strange bug NVidia managed to introduce in their graphics acceleration layer.

This method works just fine, but was not ideal for me. The game runs at 640 x 480, and is already quite pixelated when scaled up to full screen. 16-bit vs 32-bit color is somewhat noticeable, once at that scale.

2. Trade performance for correctness

Here’s another classic trade-off. Since the problem is obviously arising from the DirectX layer and its interaction with NVidia graphics hardware (boot into safe mode and run BG1 to verify this), another solution is to just kick NVidia out of the loop by disabling hardware acceleration for DirectDraw. This is feasible if you’re doing this on a fast machine (and if you’re using a series-8 card in that box, I’d assume it’s pretty fast anyway) – after all, BG1 is a 10-year old game. Your Core 2 Duo or quad-core Xeon can use some exercise anyway.

Of Hardware Acceleration Controls

In the glory days of XP, this simply meant right-click on Desktop -> Properties -> Settings -> Advanced -> Troubleshoot -> Hardware Acceleration slider (dear god that’s convoluted). In Vista, the analogous experience is right-click on Desktop -> Personalize -> Display Settings -> Advanced Settings -> Troubleshoot -> Change Settings (I see you still haven’t hired that user experience designer, Microsoft).

The problem now is that if you try this with your NVidia card and Vista, you’ll just be staring at a disabled ‘Change Settings’ button and a terse message: 'Your current display driver does not allow changes to be made to hardware acceleration settings.' If you also try the old standby dxdiag, you might be surprised to know that the Disable buttons have been removed from the DirectX Features box on the Display tab. Thanks, NVidia and Microsoft. Apparently they really don’t want us changing these settings.

But we don’t really want to change much – just the DirectX technologies, and in fact, just the 2D-based DirectDraw (since 3D is more or less irrelevant for BG1), and only for a short while. Enter the DirectX SDK.

The SDK is meant for developing and debugging DirectX-based programs, but it comes with a fair suite of nifty utilities, one of which being the DirectX Control Panel: dxcpl.exe. Download the 400+ MB SDK and grab the control panel in Utilities\bin (starting from where you installed the SDK to). Make sure that if you are on x64 (x86_64) that you also use the x86 architecture switches for BG1 (see comments for this post for more details).

In the control panel, use the DirectDraw tab – in the set of checkboxes, uncheck the “Use Hardware Acceleration” box. Fire up BG1 and see the non-black-boxed goodness of 1998 graphics (screenshot 2).

If you were disabling acceleration for another reason, this control panel should work for you too – pick the appropriate tab and have at it. Do not forget to turn acceleration back on after the session. You probably do not want unaccelerated graphics performance in your normal, non-glitchy apps.

Turning off hardware accelerated DirectDraw avoids the BG1 black-box bug. You’ll have to assess for yourself which is more expendable: color depth (trivial to change directly from BG1’s Options configuration) or graphics performance (more difficult to tweak, but perhaps compensated by CPU performance).

DirectX SDK vs Settings slider

In any case, the DirectX control panel is a somewhat useful trick to know in general, especially when faced with Vista’s obstinate insistence on not letting you change graphics acceleration settings. The control panel provides all the functionality that the old XP Settings slider would have give you – except in a much more technical interface. In fact, the old slider more or less tweaked settings in the Direct3D and DirectDraw tabs, except in a coarse-grained, all-or-nothing kind of way. Here at least you have fine-grained control on most of the detailed options in each panel.

Still, such a pain.

Know that tweaking these settings are done at your own risk (NVidia and MS obviously are against it), and may or may not work at all depending on your setup, your driver version, and pure luck. On the plus side, if you ever need to write some DirectX apps, the SDK is now just a few clicks and SDK path text fields away from Visual Studio 2005/2008, so the 400 MB bandwidth and disk space isn’t completely wasted. Hopefully.

UPDATE 9/8/2008
It’s come to my attention that some people still have problems after applying this fix – namely, there are cursor trails on menu screens. I cannot reproduce this issue locally on my 8600M GT, but it’s possible that there are new problems introduced with newer series-8 cards and drivers. If this is the case, I’d recommend using the first workaround — that is, using Software BLT and 16-bit color, rather than the DirectDraw workaround.

UPDATE 7/15/2009
There is now a Nvidia driver .dll patcher available for BG1-era Infinity Engine games at Spellhold Studios. They also explain what the underlying issue is and what the DLL wrapper does to work around the bug. I have not personally tested this fix. It does install a new graphics DLL that overrides existing calls, so it is theoretically possible that it may introduce other issues, but I am told by others that it works quite well. Check it out — it saves you a lot of trouble of ticking DirectDraw boxes on and off.

If you have SSH access to any other machine on that local network, you can use the forwarding feature and the SOCKS 4 or 5 protocol to get to the server from your home box. The connection is mediated and forwarded by the machine on the network that you can reach, and to the firewalled server, you appear to be this internal machine.

The appropriate incantation is simply:
ssh -D port_num ssh_hostname

where port_num is a local port number (I like 50000, but any non-privileged port would be good)

Then, simply point your system or browser (in Firefox, for example, this would be in Preferences/Options -> Advanced -> Network -> Settings ) to use a SOCKS proxy at localhost, port port_num. Now accesses from that browser will be proxied through the ssh_hostname machine to the actual remote_host.

The context is that there was an application server that I had to reach from my home machine. The application server sits on machine R, which is restricted to an organization internal network I. There is no VPN service for I. SSH to machine H was available, which is also in I and is reachable from the public Internet. For small things, I could run commands from H, but it would have been really helpful to reach R directly from my home development box. I could use X11 forwarding to get an xterm for various tools there, but the overhead is huge. The administrator of network I has yet to grant me external access.

With this trick, just SSH’ed into machine H, set up the proxy port via -D, set up my browser to proxy through the local port, and easily accessed R from home. Nifty.

If you happen to have SSH access to a number of servers (as I seem to have for some reason…), this same trick can be used as a way to rotate through them fairly quickly. Just log out of your existing connection and ssh into a new host with the -D switch. This allows you to test various network apps from a number of different machines.

This is documented via Microsoft KB article 276228. Microsoft’s proposed fix, in abstract: “don’t do that”.

So I was trying my hand at some fancy AJAX to dynamically populate a select dropdown menu based on a previous select menu. An xmlhttprequest is triggered onChange of the first menu. We hit up an API, parse and transform the response into this HTML fragment consisting of <option> tags and values, and assign to the innerHTML property. Simple and quick. Works on Safari, Firefox…and of course, not on IE 6.

Some time wasted later, an alert() on the innerHTML property shows that in fact, the string there is malformed. The first <option> start tag is truncated from the innerHTML. No wonder it doesn’t work; it’s malformed. And it wasn’t, to start with, when my transformation finished and delivered the final string.

The apparent solution is to follow Microsoft’s advice on that page and do something else:
- assign to the options collection
- workaround using outerHTML

I’m sure there’s a way to jury rig it so that IE is fooled into concat’ing an “<option>” back onto the innerHTML, and disabling this workaround for other user agents. But that’s rather inelegant, no? I opted to use the options collection, which seems a reasonable (if slightly more complicated) method than just a simple assignment to innerHTML.

The KB article lists this as a problem with IE 5, but it recurs in IE 6. Does it persist in IE 7?

So one of my web pages renders great in Safari and Firefox, but in IE 6, it is a completely blank page, devoid of content. Puzzled, I ran it through the W3C validator – no problem at all. Selected a View Source in IE, and noted that the entire HTML output looked OK.

Eventually I narrowed down the problem to a <script> tag in the markup. Namely, a <script type="text/javascript" ... src="foo.js" /> kind of tag. IE rendered the page when I removed the tag, and goes blank when I put it back. Curiously enough, I hadn’t actually invoked any functions from that .js file, so it was definitely not any code I was executing. Replacing the .js file with a dummy .js file also triggered the blank page. Changing or omitting the other attributes did not help.

The problem is fairly obvious now. When I close the tag in HTML style, with an actual </script> tag, IE proceeds to render just fine.

The obvious conclusion is that IE is buggy, but that may not necessarily true (well, in this one instance anyway). Despite most pages’ “compliance” with XHTML, DOCTYPE’ed and all, most web servers still serve these “XHTML” files as mimetype text/html instead of the recommended application/xhtml+xml. This is pragmatic, since IE 6 doesn’t even bother to render application/xhtml+xml, and user agents are required to stop rendering upon encountering non-valid markup (imagine the chaos that would cause).

However, it seems this might introduce a cause for the gotcha. Interpreted in actual text/html mode, one might imagine that to a HTML parser, <script .... /> doesn’t really appear to close the <script> tag at all – in fact, it might merely look like a rather malformed start script tag and no end tag. If I were a dumbly compliant parser+renderer, I might just start walking down the response string looking for that mythical end to this start tag. And end up rendering nothing. Of course, if I were a slightly smarter parser, I would look for a DOCType, but then I’d contradict the server’s mimetype, and down that road lies even more madness.

Nevertheless, the solution, when staring at a blank page in IE when the markup seems fine, is to check your script tags, if any.

I’m no expert at this soup of SGML/HTML/XHTML/XML standards thing, so the above is just my random opinion plus some observations. Still, it seems that MS should patch this particular problem, since it’s fairly non-obvious (many people, I’d surmise, would use the this kind of shorthand close tag in an XHTML file, especially since it validates fine) and upsets the status quo compromise of incremental Web standards compliance through browser compliance modes, content negotiation, and occasionally bad mimetype service. But of course, that’s never going to happen.

Update: I’ve been made aware in the comments section that the same issue occurs in IE 7. Just great.