This is the last straw. I’ve been a loyal MS Entourage / MS Outlook user since the days of Outlook Express for Mac and Office 2001. But at this point, this software has actively impeded my communications with my friends and colleagues. We’re done.

The Problem

Here’s a really simple illustration of the problem, from the receiver’s end:

See how the URL, which was composed as one plaintext line, gets split up into two lines?

Here is another example, purely from the editor UI (and not even being sent yet). I start with a perfectly good reply saved as a draft:

I make a small wording change and resave:

See that third line? Thanks to the hard line breaks inserted by Outlook (even at composition stage), the line wrap has been mangled. This draft has to be re-wrapped manually, by the tedious process of deleting the newline-based hard line breaks from every line following in the paragraph. That was a short paragraph. Imagine doing that in a long paragraph, from the first line.

To add insult to injury, there is not even a “re-wrap” functionality in the editor, to at least solve this user-interface level problem (as opposed to the protocol level problem). Obviously no one at Microsoft sends plaintext emails anymore.

The Issue

Back when email was first devised, servers didn’t have a lot of memory, and people had pretty tiny terminals with fixed line widths and not a whole lot of processing power to deal with it. The Internet standards for email messages http://www.ietf.org/rfc/rfc2822.txt, RFC2822 Section 2.1.1, defines recommendations for email body text transferred over SMTP:

There are two limits that this standard places on the number of
characters in a line. Each line of characters MUST be no more than
998 characters, and SHOULD be no more than 78 characters, excluding
the CRLF.

The 998 character limit is due to limitations in many implementations
which send, receive, or store Internet Message Format messages that
simply cannot handle more than 998 characters on a line. Receiving
implementations would do well to handle an arbitrarily large number
of characters in a line for robustness sake…

The more conservative 78 character recommendation is to accommodate
the many implementations of user interfaces that display these
messages which may truncate, or disastrously wrap, the display of
more than 78 characters per line…

…it is encumbant upon implementations which display messages
to handle an arbitrarily large number of characters in a line
(certainly at least up to the 998 character limit) for the sake of
robustness.

Basically, the SMTP server can count on messages that come in 80 characters per line (and always less than 1000 characters per line), and email clients can trust that they only have to render up to the 78th column of text. This limitation is hardly useful in the modern age, but persists since it’s part of the standard. And it’s a fine, conservative design model. But now we write some pretty long lines without linebreaking ourselves, so something magical has to happen in the email client itself, like Outlook 2011.

The naive solution, of course, is to slap arbitrary line breaks into the user’s email message at every 78 characters, which is what ye olde email clients from yesteryears did (and Outlook 2011 still does). It’s a matter of personal preference whether this is a reasonable solution. Proponents argue that the email will “always look the same” on all devices, including those limited to 78 chars per line.

I (and many others), on the other hand, think the spirit of the RFC is to allow the actual handling client to decide where to break lines. With the exception of source code, it is almost always better for the email client to use the full width of their display, however many characters that might be. Even in the case of source code, it should also not be mangled by the insertion of arbitrary line breaks in them — what if newlines are meaningful in this language, and the author used more than 78 characters per line? The example with the URI is illustrative of this problem — the URI got an arbitrary newline in the middle, destroying its meaning. Users who copy-paste the two lines will end up getting a 404, due to that stupid inserted newline in the middle of it. This should not be allowed to happen.

Because this naive solution was not perfect, an extension was proposed as RFC 2646. In the email header, if the Content-type was set to have the format=flowed parameter, such as

Content-type: text/plain; charset=US-ASCII; format=flowed

(emphasis mine), the sending and receiving email clients are allowed to reflow the text based on user linebreaks. It follows some simple reflowing rules, but in short it will preserve user-inserted hard line breaks while adjusting the rest of the message for the proper line length while the message is “on the wire”, and recombining the lines on receipt and display. Modern email clients like Thunderbird, designed for user comfort and the generous system limitations of the year 2011, implement this standard.

Guess what header Outlook 2011 sends?

Content-type: text/plain; charset="US-ASCII"

Not even an option to change that behavior. It does not appear that Outlook 2011 deals with any of this. It just inserts some line breaks and calls it a day.

An alternative, implemented by Apple’s Mail.app, is to send messages with the Content-Transfer-Encoding header set to “quoted-printable”, as per RFC 2045. In this model, soft line breaks are sent explicitly with the character “=” representing it, breaking at the usual 70-odd character column. On the receiving end, the client processes this character as a no-op and concats the line back together for display.

Outlook doesn’t do that either. It just wants to mangle your emails.

Conclusion

The world moved on and adopted HTML emails, which doesn’t have this newline problem. For those of us who do think HTML emails are an atrocity to be used sparingly, if at all, the idiosyncrasies of plaintext email have to be addressed. Outlook 2011 appears to do even worse than Entourage 2008 at this problem, by not dealing with it at all. And apparently getting a bunch of Microsoft “MVPs” on their forums to cloud the issue with promises of support and unrelated commentary.

Given the sad state of email clients on the Mac, I believe Thunderbird is now my only option for sane plaintext messaging.

So I went about looking for a way to switch this otherwise beautiful weapon, The Edge of Night, into a longsword. The similarly named Edge of Song and Glory is the ideal target for the transformation.

There is not much modding possible for DA2 — the game is very much locked down to drive DLC sales. Nevertheless, the base game itself is at least unencrypted; however, there is no official mod editor support. The DLCs, however, are encrypted — you cannot even view the data contained within these files. The following modding method requires that you decrypt the DLC’s designeritems.erf file first, then extracting the Edge of Night’s item file for editing. This only works for PC versions of DA2.

Note that there is already a mod floating around for this. There is a good thing and a bad thing about this mod. The good thing is that it uses a less questionable method that does not need to decrypt the ERF file. On the other hand, because of the limitations of such a method, that mod doesn’t change the item icon or the item description. It will still look like an axe and call itself an axe in the character’s inventory screen — only the actual gameplay model is changed to a longsword. Some people would be satisfied with that; I wasn’t.

Decrypting designeritems.erf

DLC items are in found in ${DRAGON AGE 2 INSTALLDIR}/addins. In this case, it is the DA2_MTX_WAR addin. The ERF file is
${DRAGON AGE 2 INSTALLDIR}/addins/da2_mtx_war/core/data/designeritems.erf, and the UTI file is [dd9ea6bb335cfa89].uti.

Make a backup copy of the da2_mtx_war directory.

To decrypt the ERF file, you need the decrypt key and a decrypter. There is a tool called “slurpda2.exe”, usually distributed with the deda2drm decrypter package. I will not explain how to obtain this package, but it’s pretty easily found. If you legitimately own this game, slurpda2 will give you a list of all the decrypt keys, one for each DLC package. Use the right key (have to do trial-and-error here, as it doesn’t tell you which key belongs to which package) and deda2drm by cd’ing to the deda2drm directory and executing from the command prompt:

deda2drm.exe DA2_MTX_WAR [key]

The output will tell you whether this succeeded. If so, this will decrypt the ERF.

Extracting the UTI

A tool like PyGFF will open the decrypted ERF file. Look for the dd9ea6bb335cfa89 hash key and extract that UTI. For reference, the UTI should contain an item with the tag war100im_warrior_1h_sword. Save the UTI file as war100im_warrior_1h_sword.uti (though technically you can name the filename anything you want, as long as it ends in uti). As you can see by the name, even the original designers thought it should be a sword and not an axe :p

Editing the UTI

Using a tool like TlkEdit, open the UTI file. Change the ModelVariation field to 123 (which is the Edge of Song and Glory, as listed in 2da.rim’s item_variations.gda file). I believe 132 is Glandivalis / Vigilance, if you prefer that look. Change the Icon field to “ico_1h_sword_5.dds”. Save the UTI file.

Here you can choose to restore the original encrypted copy of da2_mtx_war or not. If so, this reverses the decryption so it doesn’t confuse DA2 when you open it again. Alternatively, you can edit the manifest.xml file in the decrypted copy of da2_mtx_war to tell it to skip the authorization requirement by setting RequiresAuthorization=0. I left mine unencrypted, but I think this gets into legal gray territory, and the UTI mod should work even with the original encrypted version.

Drop the final modded uti file into packages/core/override in either the DA2 install directory or the user directory in Documents/Bioware (create the override directory if needed).

Result

I have a copy of the modded UTI file, but I think distributing it might violate the EULA. Oh well. If you can’t figure out the instructions, leave a comment here (with your email address) and I’ll try to help.

The features I’m looking for:

• POP3/IMAP + SSL support
• Full-text search, or indexable by OS X’s Spotlight
• Multiple account support
• Archival — if I delete an email account, it should not wipe out all emails from local storage that belonged to this account, especially if said account was POP3
• Plaintext composition — I’m taking the side of format=flowed in plaintext composition. Manually formatting hard line breaks at char 78 is an insane holdover from a bygone age when dumb ASCII terminals were still the primary user interface. At least allow this as an option, or support the quoted-printable content type and soft line-breaks during composition.
• Filtering rules — Basic filters that let me decide where to put messages based on mail headers and subject is enough
• mbox import/export — I need to be able to import mail from my previous client. Similarly, in the scenario that this app is no longer supported, I need the option of moving to another client.
• Bonus: auto-bcc to an arbitrary email address — I’ve never solved my problem with archiving sent mail. All you really need to do is auto-populate the BCC field on any “New Mail” composition window with this address. That’s all I really need to be happy here.

Amazingly enough, there is not a single Mac mail client that fulfills all of these basic conditions. Especially egregious problems for the top three mail clients:

• Microsoft Outlook Mac 2011 — mangles all outgoing plaintext emails by inserting hard line breaks. No support for flowed plaintext, despite promises to the contrary, and no support for quoted-printable content-type and soft line breaks. No provision for auto-bcc.
• Apple Mail (Snow Leopard) — deleting an old email account deletes ALL mail belonging to this account, even for downloaded POP3 mail. WTF. Can only auto-bcc “myself”, which is a fixed email address corresponding to the sending account; if you allow autobcc, why not let the user pick the email to autobcc to?
• Mozilla Thunderbird 8 — deleting an account deletes all downloaded mail belonging to this account, unless messages are stored to the “local folder” rather than its inbox abstractions.

For fear of accidental data loss, I’ve stuck to the Microsoft offering, despite its incredible inability to keep plaintext mail intact. The newer social clients are all Thunderbird-based and leave little else worth examining — they seem far more interested in integrating ever more social network APIs than to support more basic email features. The rest of my friends are all sworn Gmail webmail users and think I’m a strange luddite for even considering native clients. I’ll leave the webmail vs native client debate for fear of going on a page-long rant; suffice to say that the user experience between the two options is not comparable at all.

It almost makes me want to start writing a new email client for myself, or hire someone to start some skeleton code at least, because obviously no one else is going to scratch this itch of mine. That kind of spare time and money, however, would probably never come unless I manage to actually sell a company or two (rather than just founding unsuccessful ones).

This note is not about trouncing the Harvester with level 35 ubergeared supermen. Any halfwit can do that — just go get the gear, follow any YouTube min-maxing guide for DAO, and start whacking away. You can do this with any properly geared character. This note is about how to complete Golems of Amgarrak on Nightmare difficulty with a newly created, level 20 mage, with stock gear and sneaky tactics. This is not to claim it’s the best way to accomplish the feat — just the way I managed it. If you have better ideas about how to use a newly created level 20 character to win, contribute them in the comments.

This note only applies to DA:O as played on the PC. On some console-based versions, the pet bronto cannot be controlled, making the final battle even more annoying than usual.

Background

If you don’t know yet, completing Golems of Amgarrak on Nightmare (or Hard, actually) difficulty awards all of the other characters in other campaigns (even existing campaigns’ characters) some really nifty items — but only nifty for lower level characters. This includes the amulet The High Regard of House Dace, which gives +6 cunning so Daveth can open up all those annoying chests in Ostagar, and The Reaper’s Cudgel, which is worth well over 300 sovereigns. The early cash allows you to buy the Blood Dragon Armor, the Spellward, and the Arcane Knowledge and Physical skill tomes from Bodahn immediately after Lothering before his store restocks; you can then buy another copy of these items after the restock.

With my time constraints, I will most likely play DAO only once. So I’d like to have these items in my Origins campaign, without having to complete the game first to get an absurdly well geared character to do Amgarrak. I thought I can just start a new Amgarrak campaign with a new character, and finish it to get my gear. Boy, was I wrong. People call this the hardest DLC for Origins, and for good reason. I played a warrior and simply could not win an initial fight against 4 elite golems, and had to delete, restart the campaign, and rethink my strategy.

General Setup

Despite claims, the module is not impossible, even with a new level 20 character. It is simply very poorly balanced for new characters, and forces you to proceed carefully and methodically. This is Dragon Age, not Diablo — as a new character, you cannot just rush into every battle, aggro’ing all the mobs at once. Separate mobs from each other, killing them one at a time, withdraw backwards to avoid being surrounded, pause frequently to issue new orders as the situation requires and don’t rely on the Combat Tactics scripts, and you should have very little difficulty dealing with most creatures in the module.

To win, I strongly suggest playing a mage. In fact, the best character for completing the DLC with is probably the mage, regardless of level. The main reason: there are no merchants, not nearly enough healing poultices, and most unfortunate of all, no other playable mage in the entire module. The objective is to preserve all potions and consumables until the final battle with the Harvester, which means the party will need to rely on mage healing spells.

The party at the start consists of a duel-wielding rogue, a bronto (with a charge attack and a Taunt), and the PC. The party at the end adds a sword-and-shield warrior (without Taunt!) and a Runic Golem with some typical golem abilities, plus a weak Group Heal, a Cleansing Aura, and Chain Lightning/Tempest spells. All of these characters are poorly configured with mediocre equipment, and the party only has one respec tome (Memoirs of the Grey Wardens).

Save the tome for the rogue. He’s going to be the main DPS character against the Harvester, but only if he’s an archer and standing far away from the action.

Attributes and Specializations

If starting this without having beaten Awakening, then the mage is restricted to three base specializations: Spirit Healer, Arcane Warrior, and Shapeshifter. Since Shapeshifter is worthless, the two specializations are decided already. If you have started Awakening, you might consider the Battle Mage instead of Arcane Warrior for additional utility, but Spirit Healer is an absolute must as a new character.

As a mage, 60+ magic is necessary for spells to have decent effect. The rest should be spent in willpower to pad out the mana supply. Usually, when you build a Mage for Origins or Awakening, you don’t want to put any points in Willpower. However, remember here, there are very few, very precious lyrium potions and no way to buy more; there are no special attribute bonuses that boost the mana pool for free; and the PC’s starting equipment is subpar and does not provide a large mana boost (and again, there is no way to buy better equipment). If you don’t take additional Willpower, you will not be able to cast during the long endgame battle against the Harvester, and that would be very unfortunate.

The mage should have maxed Combat Training to ensure that spells aren’t interrupted by ordinary attacks. Survival is a plus for the nature resistances, and the first level Vitality and Mana skills from Awakening. This is a pure combat module, with no ingredients for crafting and a grand total of two conversations — no need for Coercion or any of the crafting skills.

Talents

One might be tempted to build an armored arcane warrior, since this is a popular “I win” character build.

This would be … unfortunate. Normally, the arcane warrior is very overpowered: all the gear of a tank and the firepower of a mage. In this module, with a new character, you have neither of these advantages. The small module does not offer a merchant or any sort of decent armor, and a stock character does not have enough willpower to have a deep mana pool after putting up the arcane warrior defensive abilities. Wearing a robe, swinging a Warden’s Companion longsword with a mediocre shield, while not being able to cast healing spells — this is a recipe for disaster. Further, Brogan, the warrior in the party, is configured as a tank — a poor tank, but he can’t be respec’ed as a DPS due to lack of tomes, while the PC can be a DPS. Therefore, playing a support + nuker mage is probably a better choice. Some of the advice that follows will assume a mage with an assortment of crowd control and attack spells.

As for specific talents, since it’s a stock character, there aren’t that many points available. Here is my advice:

• The Healing tree up to Regeneration is a must. As Spirit Healer, also take at minimum Group Healing, because both the golem enemies and the Harvester at the end have area-of-effect attack abilities that can damage an entire party. Remember that this character will be the only healer until the Runic Golem is fully upgraded, and even so, the Golem is a terrible healer. These healing spells are basically critical to finishing the module, and is why stock level 20 Warriors or Rogues have a hard time surviving.
• Make sure to get Force Field, for several possible reasons. Crushing Prison is nice, but only if there are points for it.
• It would be wise to take Mana Clash, at the top of the Mana Drain tree. In several battles (unfortunately, not the end battle), the party will face several spellcasters along with hard-hitting warriors. Mana Clash will immediately remove the casters from the scene, which trivializes the encounter.
• For me, I took Tempest and Blizzard (from the Lightning and Cold trees), since I already have Spell Might from the Mana Drain tree. This sets up the Storm of the Century combination that was useful in one instance, but this is optional. I strongly recommend Cone of Cold in any case as it is an absurdly overpowered spell with a freezing effect that applies even to elites and bosses.
• If taking Storm of the Century, or not taking Mana Clash, then Glyph of Neutralization is desirable. As for the other glyphs, since the main enemies are almost always elite or boss level golems and undead, they are often resisted — except the Paralysis Explosion combination, if set up correctly.
• I did not use any other spells. If you have a favorite set, use the remaining points to take them. Remember that practically all enemies worth using mana on are Elite or above, so some spells that have minimum effect on elites are not worth taking.

Remember that only one party member will get a chance to respec, and the rogue should use it to reconfigure as a survival-minded DEX-based archer. This is because this underpowered party simply cannot stand toe-to-toe against any of the major enemies, and will rely on kiting (and not even simple kiting either, given the setup of the final battle) and off-tanking to survive many fights.

Jerrik, the Archer

Jerrik will be the main non-magical DPS. Unfortunately, he is setup as a frontline fighter. Because the tank is terrible, and can’t even manage threat effectively (no Taunt!), plus both enemy Golems and the Harvester have area-of-effect attacks to damage everyone in their immediate vicinity, this is a bad configuration. He must respec into a DEX-based DPS archer (enough STR to wear his armor, then all DEX), with Master Archer or Scattershot, Arrow of Slaying, and Feign Death (for the many times when the tank loses threat and the enemy rushes the archer). Optionally, he could take a few cunning and take Stealth to lose threat, though the final battle involves an elite boss and hordes of minions.

Major Battles

On Nightmare difficulty, in general, proceed slowly and cautiously, withdraw backwards into chokepoints when needed, and there should be minimal pain. In particular, when faced with multiple spellcasters, hit Mana Clash to remove most (if not all), and then rush them. Play this as a normal RPG and use the pause key frequently to strategize and adjust tactics. Try not to use any consumables until the final fight. For a general walkthrough, follow Gamebanshee’s Amgarrak guide.

There are two battles that takes some strategy to pull off, however. Some people are stumped here and can’t proceed further. They’re not hard, but you can’t use the normal tank-and-spank tactic with a level 20 Warden and a crappy party.

The ancient hall of golems

As soon as the party enters Amgarrak, there are three directions to go, two of which are sealed off at the start. In the third direction, there is a room with four stationary golems. Two of them are called Watchers, the other two are elites called Sentinels. When the party proceeds far enough into the room, these four will activate.

This is one of the toughest fights because the party does not have its warrior tank yet. Further, these four golems will open the battle by hurling rocks — which deals heavy area-of-effect damage and knocks down any character beside the Runic Golem. They will then follow by using Slam on the incapacitated character, and then Hurl Rock again, basically going into a stunlock until the character is dead.

Careful pulling is impossible, because all four activate at the same time. However, there is one good way to deal with these golems: The Hallway of Doom.

There is a chokepoint hallway going into the hall of golems. The party could fight the golems one-on-one at the doorway, but they’ll jut use Slam to knock the blocker out of the way and rush into the room. Then again, they won’t be able to do that …if the blocker is force-fielded.

Have the party hold position. Use the Runic Golem to trigger the four enemy golems, and then run back to the door. At the doorway, use the Runic Golem to block the door and force field the golem before he gets Slammed. Have the ranged DPS open fire. Force Field your own golem, because the enemy golems are elite and will have shorter Force Field durations.

For even better effect, throw down a Blizzard to slow the enemy as the Runic Golem approaches. Then, trigger Storm of the Century using Tempest, and then Force Field your Runic Golem before it gets seriously damaged. Now the enemy will stand in the massive electrical storm, be drained of stamina by the lightning storm’s side effect (so they can’t Hurl Rock or Slam anymore), and run around uselessly. When the Force Field expires, withdraw the Runic Golem and slaughter the heavily injured enemies (or heal the Golem until you’re able to Force Field again — the enemy should no longer have enough stamina for another Slam).

Make sure the mage is standing off to the side as the Runic Golem runs toward the party, because the enemy could see the mage and interrupt the casting with a hurled rock. If you cast the Force Field too late, the enemy will Slam the blocker out of the way, which will probably cause a party wipe.

I really wish I had two mages in this module, which would be able to keep this little trick going forever. Alas, that is not to be.

NOTE: If you didn’t take Storm of the Century Force Field, or if you can’t get the blocking trick to work, there is another way. You need to have respec’ed Jerrik with at least Combat Stealth. Have the party hold position far away from the action, have Jerrik walk into the hallway and trigger the golems. When the golems activate, Stealth immediately and walk away. Using Jerrik, carefully pull the now activated golems, one or two at a time, and kill them individually as they follow you down the hallway. This is arguably easier, if less impressive, than the hallway blockade trick.

The green room

Again, a horde of golems.

This time, the party needs to flip a switch in the green phase, which triggers an encounter with four boss golems and two elite golems. If you didn’t feel enough hurled rocks and full-bodied rock slams in that previous golem encounter, this one will surely give enough. Even with Brogan the warrior, no one in a level 20 party can tank through that. There is a doorway chokepoint, but boss golems have huge resistances, and even a full Storm of the Century will only take off 1/3 of their hitpoints on Nightmare. The blocker golem won’t survive too long after the Force Field expires; certainly not long enough to get another Force Field up. Once again, they all activate at the same time, so you can’t pull one away by itself…or can you?

This one needs an underhanded trick. Remember, there was a room next to the starting chamber of the Amgarrak level with a blue and a purple switch. Both of these switches will throw the party out-of-phase with the golems, meaning that they disappear from the map. When this happens, the hostility chase trigger is also reset. If you can somehow kite the golems to other rooms, then run to the switch to go into blue or purple phase, walk away while phased out, and then switch back into normal phase when you’re far away enough…these golems will lose sight of you and stand still. Furthermore, now they’re isolated from each other, and you can pull each one independently instead of triggering all six at once! This means you can pull each golem, kill it, heal, and repeat.

So the strategy is as follows: after you hit the green room switch to activate the golems, immediately run back out of the room, down the stairs, and toward the starting chamber. The faster you run, the more the enemy golems aren’t going to keep up, and will be strung out along the way. As you arrive in the switch chamber, switch phase. All the golems disappear as you go out of phase with them.

Walk back to the green room, switch phase again, rob the chests if you need to, and return to normal phase. Slowly go down the stairs again, pull each golem one at a time along the way, kill them one by one until you arrive at the switch room again. Problem solved.

The Final Battle

This is the fight that few people seem to win as a poorly geared level 20. This is because they are going toe-to-toe with the Harvester flesh golem. There are two major problems with this.

1. The Harvester hits for 90+ damage, and crits for 100+ damage on Brogan or the tank, due to the crappy equipment setup of the stock party. There are not enough healing poultices or spells to keep the tank alive at that rate.
2. There is a second problem — the Harvester in the Flesh Golem phase periodically summons a skeleton, either with his Vile Disembowelment skill, or they just wander into the room. As the flesh golem takes damage, the skeletons spawn faster. The skeleton is usually elite, which hits really hard, or boss, which can seriously damage even the tank. This isn’t even the main problem — the real problem is that the skeletons go after the DPS’ers. Because Brogan has terrible threat management, the skeletons will wander into the room, go immediately for the caster or the archer, use Riposte to stun them, and hit for 50+ on leather and even higher on cloth. If the mage and/or the golem goes down, it’s a guaranteed wipe.

Since the tank can’t hold up, the only strategy is to kite the Harvester around the room. There is plenty of space to maneuver, but there is not enough DPS with a stock party to take it down quickly enough. Jerrik hits for 80 to 100 per arrow on the Harvester, and crits for a bit higher. The mage hits at around 30 or 40 with the staff and more with spells if not resisted, but she can’t cast too many attack spells as healing is very needed and there are about 10 to 20 lyrium potions in all. The rest of the party is fairly useless in terms of DPS, since they’re melee and can’t get a good hit in on the Harvester as it walks around. The room is going to be filled with skeletons. They all have stun skills. If the kiter take a stun or two, he will get swarmed and probably die.

Preparation

The party needs to set some basic tactics.

• Brogan needs to take the least powerful health poultice at 25% health or less
• Jerrik needs to set typical archer tactics, and feign death at health < 10%
• The PC’s mage needs to drink a least powerful lyrium potion at 25% mana, cast Regeneration on an ally when the ally’s health is < 50%
• The Runic Golem should turn off his Hurl Rock and Quake tactics, and cast Group Heal when an ally’s health < 50%

Have the Runic Golem set Flame Weapons. When the battle begins, coat weapons with the highest level poison you have available.

Phase One

In Phase One, the Harvester will jump inside a flesh golem and attack with Vile Disembowelment as an area-of-effect attack, which spawns a new skeleton.

As the battle begins, set the party to Hold Position, and manually move the party members into position. The mage and the archer should go stand by the switch that appears, but far enough apart that they don’t get nailed by the same Harvester attack at the same time. Brogan needs to stand close by as the off-tank. The Runic Golem should stand at the center of room with his Cleansing Aura. The bronto (which actually has Taunt and runs faster than any other character in combat) should attack the Harvester, throw down a Taunt, and then start kiting the Harvester around the Runic Golem’s aura radius.

If a normal skeleton appears, Brogan should go tank it immediately, and the mage can help freeze it with a Cone of Cold. If an elite or boss skeleton appears, or if there are more than 3 skeletons, have the mage pull the switch immediately to destroy all the skeletons. Jerrik should always focus on the Harvester. If either the archer or the mage is under attack, have the bronto run over and Taunt, then run the mage and/or archer away. The skeleton should lose interest and focus on the bronto. If not, Brogan needs to pick it up and tank it while the DPS’ers run far away.

Heal the bronto constantly, as it’ll inevitably get hit by the Harvester and its skeletons. If the bronto is stunned, or if it is killed, Brogan must pick up threat and resume kiting immediately until Jerrik is able to resummon the bronto. Switch back once the bronto is up again.

Keep an eye on the archer and mage while you’re kiting with the bronto. Make sure no skeleton has wandered into the room and started attacking them. Pause a lot if you need to.

If this is executed correctly, eventually the Harvester will fall. When its health gets very low, try to avoid hitting the switch and save it for just before Phase Two. The switch will not appear again at that point, and there may be several skeletons that need killing at the end.

There is a bit of luck involved. If the switch is hit too early, and elite skeletons emerge while the switch is inactive, then this is probably bad. If it’s a boss skeleton, you might consider using Brogan to kite it around until the switch is available again. Controlling two kiters is very difficult, so pause a lot.

Phase Two

When the flesh golem is killed, the Harvester will jump out and attack. If the party made it this far, and the switch is still available, then the battle is more or less won. The Harvester lizard-form is far, far weaker than its previous flesh golem body. It hits for about 60 on Brogan, but hits slowly and misses a lot, so it can be tanked. It also disengages a lot to channel a skeleton summoning spell, so it doesn’t have a lot of DPS to hit the party with. The best news: skeletons will no longer wander into the room in large packs — at this point, the Harvester has to summon one with his spell.

When the game screen comes back from the cutscene, hit the switch one last time to clear out all remaining skeletons if it’s still available. If not, resume the kiting strategy and quickly remove the remaining skeletons. Then, have the party resume normal attack stance from hold position and focus fire on the Harvester’s lizard form. If it uses the jump ability, cast Crushing Prison (it won’t resist nearly as much now) or the Runic Golem’s Slam attack to slow it down while the melee attackers catch up. It can also be frozen solid with Cone of Cold, though it can’t be shattered obviously. Jerrik and the mage are still going to be the main DPS. If the Harvester finishes channeling a normal or elite skeleton, tank it with Brogan and blast it down quickly. If it’s a boss skeleton, Force Field it or have the bronto kite it around the room while the remaining party members continue to blast the Harvester. It is only a matter of time before the Harvester dies.

The Grim Reaper achievement should hit the screen after a short cutscene. Woot!

The first time I did this, it took me three tries and about an hour of playing, but is definitely a reproducible strategy on Nightmare difficulty. If playing on Hard, this is actually really easy, as most skeletons will be normal skeletons and can be frozen and shattered. It does take a lot of micromanagement, but since you can pause the game and issue new orders, this management shouldn’t be too hard — kinds of reminds of me Baldur’s Gate 1, where I auto-paused after every combat round and barely took any damage.

It’s odd, as Ph.D. students (and former startup founders) can hold forth via very long and extensive blog posts, especially on technical topics they care about (and sometimes on topics that they don’t). It may speak to my lack of aptitude for Ph.D. work; I hope not.

When I implemented SelfSolved, I created a sitemap for search engine crawlers as a matter of course, but did not do so immediately. Lots of content were present on the site before I completed the sitemap feature. It turns out that search engines like the GOOG seem to ignore certain content in the sitemap — so much for indexing all the world’s knowledge. I thought the crawler would get around to it eventually, but it doesn’t appear to be the case after nearly a year.

It’s interesting, because there doesn’t to be any rules. For example, “turning caching off completely in Pylons” has a very short problem statement and solution. Other posts, however, are fairly normal: “SVN directory is viewable through Apache” is by all accounts a normal sized SelfSolved problem, with references and a full solution writeup. As of this post, neither are indexed.

As soon as I wrote this blog post on this WordPress installation, the post became available in GOOG’s main index. If I search on the phrases above, I get an entry back immediately. But for a site like SelfSolved, these indexes aren’t available for days or weeks, even though SelfSolved publishes an Atom feed of its content as well, and pings the same GOOG notification URIs. This does suggest that if you build your site on a well-used CMS, rather than custom-designed software, it will get you into indexes in almost real-time vs in days. If you run a for-profit content-driven site (which SelfSolved is not), that could mean a fair bit of money.

It’s no wonder that SEO consultancies are flourishing. There are certain hidden rules with crawling that go beyond the simple “create good content” and “get a sitemap” guidelines that the GOOG wants you to believe.

The Problem

The stock Python 2.6 shipped with OS X 10.6 Snow Leopard has a readline module linked to libedit, the BSD alternative to the GPL’ed readline. The readline module, if you are not aware, is (among other things) responsible for keeping command history in the IPython interpreter. This causes command history in the IPython 0.10 interpreter to behave in very odd ways. When backtracking through the command history buffer using the up-arrow key, for example, the previous command is only partially recalled, and appears completely scrambled. Indents, too, seem off — in a whitespace-sensitive language like Python, this is annoying. (See first figure)

IPython command interpreter is broken when using libedit with command history

Fixing IPython’s bugs are beyond my ability. While I certainly don’t want to delve into the quagmire that is GPL vs BSD licensing, I do understand why Apple would want to avoid the viral nature of the GPL and ship libedit instead. However, using a genuine Readline library is going to be the best recourse for this problem. I already have a copy of readline compiled and ready to go, and just need a new version of readline.so, the library that links Python to readline.

The easy solution

Sifting through my records, I came across a SelfSolved problem record from my good friend Hannes who had issues with his IPython command history.

The solution: sudo easy_install readline, which uses setuptools to install a precompiled package of readline.so statically linked to genuine GNU readline. Restart your IPython console and everything should work. (See second figure)

IPython with readline

The hard solution

Being the inquisitive sort, I also wondered how I would be able to reproduce this work from scratch. readline.so ships with the Python source package, but surely I would not be required to compile a whole new copy of Python for one measly module library?

I documented this process in SelfSolved again: building readline.so for Python. At some point I should write an interface between SelfSolved and WordPress so that I don’t have to reproduce a lot of my work here manually.

Compiling readline.so

This is actually fairly easy.

1. Get a copy of the Python source code. In OS X 10.6, it ships with Python 2.6.1.
2. Unpack it and go into its directory. You should find a Modules subdirectory. In it is readline.c, the source file for readline.so.
3. Compile the source file. The appropriate incantation is:
   gcc -O2 -arch x86_64 -arch i386 -shared -o readline.so readline.c -I/usr/local/include -I/System/Library/Frameworks/Python.framework/Versions/2.6/include/python2.6 -L/usr/local/lib -lreadline -ltermcap -framework Python

   where the -arch flags should be whatever processors you wish to support, the -I arguments should point to the directories that contain header files for the readline library and the Python framework, and the -L argument should point to the path for the readline library. Use whatever optimization flags you feel comfortable with, instead of -O2, if you wish.

Replacing readline.so

So now we have a readline.so that’s properly linked to readline.dylib. The thornier question is how to override the system-provided readline.so. The system version is located at /System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-dynload/readline.so, and the naive would simply overwrite it with their new readline.so. This is a bad idea.

As I have mentioned in the past, overwriting system libraries in OS X is an unhealthy thing to do. The problem is that Apple furnishes no official package management system — anything you personally change is considered fair game for the next official system update. On the next system update, if the Python component is affected by the update, the Apple updater will happily clobber your compiled files with its own, leaving you suddenly back at square one. You don’t know how many times I’ve had to recompile emacs (for X11 support) on OS X 10.4 because of this little annoyance. Leave things in the /System/Library directory hierarchy alone, for your own sanity.

However, in this case /System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-dynload comes ahead of the user-modifiable /Library/Python/2.6/site-packages directory on Python’s sys.path. So if you just drop readline.so into site-packages, the system version still takes priority.

There are a few ways to do this. For one, you can create a sitecustomize.py in /Library/Python2.6/site-packages. In this file, arbitrary Python statements can be written, and the interpreter will automatically execute them at runtime. So, you can add a sys.path = ["/dir/here"] + sys.path statement and point it to a directory containing your readline.so file. Alternatively, you can abuse the technique used in the easy_install.pth file. It turns out that if you ever used easy_install, directories pointed to by the easy_install.pth file takes priority over the system paths. They use an interesting way to accomplish this, which you can copy. Or, you can just insert your directory containing readline.so into easy_install.pth. In any case, this will force the readline-based readline.so to take precedence over the libedit-based readline.so, without overwriting anything.

Discussion

So for any sane person, the easy solution should be enough. For the rest, the hard solution is an interesting exploration of how some of Python’s built-in modules can be compiled and inserted individually.

Seagate did in fact issue a firmware update — SD1A — that supposedly addressed this issue, but of course, there’s one catch: you can’t install the firmware through an external drive enclosure. In communication with Seagate support, a representative confirmed that for those of us without a desktop tower that has a SATA bay, we’re hosed:

Unfortunately, due to the nature of firmware updates and the way external drives work, the firmware update program cannot directly communicate with the drive in the manner it needs to in order to be able to upload the new firmware to the drive. It must be plugged into an internal SATA controller in order to update the drive.

Fair enough. That makes technical sense — but of course, it doesn’t work for me. I asked whether they would handle a mail-in repair, given that I have no easy access to such a desktop. The answer, of course, is No.

I have to find a desktop, open it up, jam this baby in (possibly in place of the existing drive if there’s only one bay), update the firmware, and put everything back together. Sadly, most of my friends who still own desktops would not trust me that far.

Half a year passes, and I finally find a sucker good friend who’s gullible awesome enough for me to try this procedure on his machine. The fellow owns a nice if aging Dell Precision T5400, which comes with two SATA bays (so I don’t have to inflict undue harm onto the existing system). Since this thing can run two drives at once, I can use the first method (a Windows-based firmware updater), though I burned a boot CD for the second method just in case. I popped in the drive, fired up Windows XP, downloaded the Windows-based Firmware Update Utility, double-clicked, and thought it was the (triumphant) end. In fact, it took 3 hours of my life to find out just how deep this rabbit hole goes.

Problem 1: The lying updater

The firmware updater will give a bunch of scary warnings and then reboot the machine. It will automatically reboot to a Seagate Loader screen, which attempts to apply the patch to all eligible SATA drives. To its credit, it’ll skip the non-qualifying (i.e. non-Seagate, non-Barracuda, etc.) drives, but it’ll still try them out first. At the end of the process, it will report “firmware downloaded” and “SUCCESSFUL” or some variant thereof, and automatically reboot back into Windows.

At this point, I advise you to use the SeaTools utility to verify that the firmware update actually applied. Despite its claims, if you were on a stock setup Dell T5400 (or perhaps other models as well), this will prove that the updater is a lying scumbag. And in fact, this particular drive still reported firmware SD15, the broken one.

Problem 2: The broken Boot CD

To save both me and my gracious host (who’s starting to suspect my computer-fixin’ skills now) some time, I decided to try the boot CD method, rather than pounding my head trying to see why the updater was lying. I downloaded the boot CD from the same Seagate Support site above, burned it to disk, and tried it out.

The result is a new SelfSolved posting: SelfSolved #59: getFatBlock error when upgrading Seagate Barracuda 7200.11 firmware. In essence:

The FreeDOS boot CD reports a number of ” error reading partition table drive 01 sector 0 ” errors. This is followed by ” get Fatblock failed:0x000000e8 ” or some variant of ” getFatBlock failed : ” The FreeDOS boot process appears to stall at this stage, and does not continue to the firmware flasher program.

That was lovely.

The Solution

I chased some red herrings. I came across postings about failures in various FreeDOS-based Seagate tools. One such post mentioned that it took a long time for the boot disc to get over the “error reading partition table” errors, but I waited forever (well, 15 minutes) and the boot process did appear to be frozen / stalled. I reformatted the drive via diskpart clean, thinking that the getFatBlock and error reading partition errors were related to a non-MBR partition table (I had it set to GPT). I should have realized, of course, that the errors were completely unrelated to filesystems, despite the “fat block” to which it refers.

The actual solution is deceptively simple — the boot disc & flasher appears to handle AHCI-based SATA mode badly. The Dell I was using was set to AHCI mode, out of the three possible Legacy, AHCI, and RAID options for SATA. Apparently the boot disc simply doesn’t handle this mode correctly on the Dell machine (and may also be related to why the Windows-based updater lies). When the machine switches on, use F12 to enter the boot menu, and select Setup to enter the BIOS. Then, on the list of Drive Options, skip past the SATA drives and down to SATA options. Pick the Legacy option to use ATA mode, instead of AHCI. Once this is done, the boot disc will function correctly, and the updated firmware will be applied without incident. Remember to switch the mode back to AHCI — it’s default for reason, no doubt.

The “error reading partition” messages were completely red herrings. They appear whether you are in the right SATA mode or not, and does not appear to affect the operation of the firmware updater or the boot process. It should not take very long to get to the flasher on this particular setup, so don’t wait on that message too long — it’s a good sign something’s not quite right.

In the end, I did recover my $100 hard drive, and the confidence of my peer in my mad hardware skillz (actually, quite non-existent).

Discussion

In the end, I’m quite appalled at Seagate. This sort of failure shouldn’t have happened, of course. Once it did, Seagate should have offered to take back and replace broken drives — the data I had on there was non-critical. I would have been perfectly willing to pay shipping costs to get a fixed replacement through mail-in service. I should not have been forced to search my social network for a person willing to let me tear his desktop computer apart, for a dubious and unsure firmware update procedure that fails mysteriously. I spent an additional 3 hours tracing mystery failures, for which the error messages were rather useless. Without my trusty iPhone and access to the Internet, I would not been able to solve this problem. How should I have known what “getFatBlock failed” means?

This little episode has convinced me to never buy a Seagate drive again — I simply cannot afford the time and energy for these sort of firmware upgrade adventures. While I was looking for a desktop to tear apart, I bought a Western Digital Caviar Black 1TB drive instead. Another $100, but at least I had a scratch drive for my work.

The moral of the story: Seagate, you are the worst storage vendor I’ve had to work with so far. I hope this record is not broken in the future.

After compiling Subversion 1.6.2 from source on OS X 10.5 Leopard, the compilation is apparently successful, but svn dies when it tries to connect to the network for the first time. Crash log reports that symbols are missing from libneon.dylib.

Crash report from shell:

dyld: lazy symbol binding failed: Symbol not found: _ne_set_connect_timeout
Referenced from: /usr/local/lib/libsvn_ra_neon-1.0.dylib
Expected in: dynamic lookup

dyld: Symbol not found: _ne_set_connect_timeout
Referenced from: /usr/local/lib/libsvn_ra_neon-1.0.dylib
Expected in: dynamic lookup


Check out the places that I googled and my final solution writeup … at SelfSolved #49: Subversion 1.6.2 explodes on first network access.

The problem is very similar to a previous compilation issue I solved for PHP. In essence, the -L library search path passed to GCC at compilation time has /usr/lib in front of everything else. This means whatever library path you might have given to it at configure time, it’ll always look for the library in /usr/lib first, picking up the old system libneon in the process. Since the bad libneon dynamically linked, the problem doesn’t manifest itself until runtime — and only at runtime with network access involved.

As with the PHP issue, change the very first -L/usr/lib to -L/usr/local/lib (or wherever your newer libneon is located), and it’ll link correctly.

Out of curiosity, I checked MacPorts first. The MacPorts solution of disabling libneon version checking is odd — it also works, but I dunno if it’s linking to the right thing or not.

A couple of my iPhone projects require a decent HTML/XHTML parser. On OS X, Cocoa ships with NSXMLDocument, which includes dirty HTML parsing functionality from libtidy. Unfortunately, NSXMLDocument is not part of the actual iPhone 2.2 SDK (though it is part of the 2.2 Simulator — so it’ll compile just fine at dev time but break when deploying — a big gotcha if you never tested against a real iPhone).

NSXMLParser is a part of the iPhone SDK…This is not a reasonable alternative.

Check out my writeup at SelfSolved #42: HTML or XHTML Parser for iPhone SDK 2.x

Finally, all out of all the potential alternatives I found (all referenced at the SelfSolved writeup — including one that requires a license fee to use), this one seems to be the most promising and requires the least amount of pain (read: interaction with the libxml C API — god knows I’ve done enough of that while building prototypes at Yahoo! Research Berkeley)

I gave this some thought, and perhaps one could say that blowing up Vulcan is a symbolic “fuck you” to the core values of Trek, to wit: “Logic, reason, humanism–fuck these things! Do what feels right regardless of the consequences. Or you’ll get sucked into a black hole.” Vulcan, I think, represented a key component of the Federation’s philosophy, and now they expect us to accept its destruction so they could thrill some casual viewers?

To my mind, the interesting tension in the Trek universe prior to this travesty resulted from Federation principles being tested against competing internal and external forces. However, in Trek: Rebooted land, there’s one hand clapping–it’s all barbarism all the time; there were no discernible Federation principles in the first place. You know, in Star Trek VI, Spock said “Logic is the beginning of wisdom, not the end.” New version: “Fuck all that shit man, party on the Enterprise!” Logic is apparently a trifle to be jettisoned whenever your adrenaline glands act up.

I think this has become a fairly general problem in what is considered “mainstream” sci-fi — the shows that “ordinary people” would accept. The so-called more real vision of the future espoused by these things can roughly be generalized into “darker, edgier, younger”. Reality, as it seems, must be full of angst, petty problems, and on the occasion, explosions.

A far cry from when the thoughtful visions of TNG garnered mainstream viewers by the millions.

There’s a quote attributed to AI pioneer Marvin Minsky:

“General fiction is pretty much about ways that people get into problems and screw their lives up. Science fiction is about everything else.”

The point of science fiction is to imagine what is possible beyond the confining frameworks of everyday reality and drudgery. It is supposed to illustrate what humanity is capable of and what they should aspire toward.

TNG, for example, aimed to explore a post-scarcity human society where the driving force is no longer personal economic gain. Of course it would not be “real”, so much as matter replication and warp drive are also “not real”. An average man of today would have to have to stretch his imagination to understand the motivations of those characters and their society — but that’s precisely the point. If we all spun in place forever, mired in the minutiae of what is (rather than what should and will be), there would never be any progress — only more of the same.

This is also part of a rather annoying trend in popular culture to treat technology as something to be feared, and how curiosity should be punished. How many recent sci-fi successes can be boiled down to: “Don’t research/develop technology X, because it’ll just destroy all humans/blow up the world/nuke us and all of our colonies!”. It is trivially easy to spin a cautionary tale of fear for the new and unknown, to play to the most primitive parts of the human mind — such fearmongering undoubtedly holds TV ratings and sells movie tickets. It takes a true visionary to look into what might be the next step for human progress, and instead of fearing the unknown, embrace it.

The Original Series inspired a generation of young people to become scientists and engineers. The Next Generation taught the value of deliberation, diplomacy, and teamwork, creating a vision of a humanity pulling away from the basest parts of its past — and moving onto bigger and better things. While they are still quite entertaining, they are part of that grand tradition of science fiction, opening the minds of ordinary people and pushing them, if only a little bit, toward exploring “final frontiers”.

What do the latest and leading “sci-fi” do for that vision?

One thing that has been irritating me lately is that there’s a cosmetic error MenuMeters 1.3 that causes negative values to appear in the VM Statistics section of the memory stats display. For example, the page faults value can roll over INT_MAX to report -1,800,000 page faults, when I’ve used the same OS X session for a long time without rebooting.

Since MenuMeters is GPL’ed, a quick lookthrough at its codebase reveals the problem. The details of this problem and solution is currently documented as #32 MenuMeters Memory Meter reports negative page faults at SelfSolved, a new web application I’ve written to keep track of these things.

More details to follow.

One persistent problem that I run into is that I need to access certain network resources through a SOCKS proxy server. This is all well and good if they are web resources — Safari, Firefox, etc. support SOCKS proxies quite well. However, I also need, for example, SSH and Subversion access to some resources. SOCKS support is woefully inadequate or nonexistent in these tools.

In the case of SSH, even if you google for this, you’ll run through thousands of examples of using ssh as a SOCKS server, but not through one as a SOCKS client. There are some convoluted solutions, but none of them I can use directly on an OS X 10.5 machine.

TSocks: the solution…if it were that easy

Now, tsocks is a nifty little tool to transparently divert network calls through a SOCKS 4 or SOCKS 5 proxy. This allows even non-SOCKS-aware applications to function through a SOCKS server.

Unfortunately it is very old, unmaintained code (1.8 beta 5 was released in 2002). It doesn’t compile cleanly on OS X due to this, nor will it compile under GCC 4.x. Further, it won’t work out of the box either if you do manage to compile it. The problem is that it relies on the Linux-only LD_PRELOAD functionality to use a shared library to hijack network system calls. This mechanism is called DYLD_INSERT_LIBRARIES on OS X and only works if DYLD_FORCE_FLAT_NAMESPACES is active.

Getting a working tsocks: MacPorts

There is an easy way to get tsocks. MacPorts ships a ported tsocks package. If you use MacPorts, sudo port install tsocks should do it.

Unfortunately on several machines I don’t use MacPorts, and don’t want to pull down an entire third-party package manager with its own library tree on each of these boxes. So I have do to this the hard way.

Getting a working tsocks: rolling my own

First to notice is that there are two tsocks distributions. One is the original tsocks 1.8b5, last updated in the first half of this decade. To make it work, follow the instructions provided by Marc Abramowitz in 2006. Note that his patch is actually located at his new domain address instead of the old, linked one.

The MacPorts distribution, on the other hand, is based on R. Garcia’s patched tsocks distribution, incorporating some modernization and new features by the Tor team. This distribution is numbered 1.8.x, with the last being 1.8.4. Unfortunately it is also no longer maintained, as the Tor devs forked this into a custom version to use with the Tor network only. Unfortunate, but for now, it still compiles, and works a bit better than the 2002 original.

To roll your own tsocks via source out of the MacPorts distribution, you will want the patches from the MacPorts repository. An outline of the compilation procedure:

1. Download tsocks 1.8.4 from the author’s page
2. Download all the patches from the MacPorts repository
3. Concatenate all of the patches together:

   cat patch-* > tsocks.osx.patch

4. Put the concatenated tsocks.osx.patch file into the tsocks source directory. Apply the patches:

   patch -p0 < tsocks.osx.patch

5. Regenerate the configure script:

   autoreconf

6. Configure the package:

   ./configure --prefix=/usr/local --bindir=/usr/local/bin --mandir=/usr/local/man --sysconfdir=/etc --libdir=/usr/local/lib

7. Install the library and binaries:

   sudo make install

8. Install the conf file:

   sudo cp ./tsocks.conf.complex.example /etc/tsocks.conf

9. Edit the conf file. Make sure that if you’re not using tor, that you write in the conf file

   tordns_enable = false

Configuring tsocks

The complex configuration file example should have explained all of the features to be set. For my configuration:

Some important settings:

• local – this setting, in the format of IP/netmask can be repeated several times, each time to exclude a set of IPs from being diverted to the SOCKS server. For obvious reasons, your SOCKS server will have to exist in one of these excluded IP ranges – otherwise you will never even reach your proxy server.
• server and server_port – these should point to the IP address and port of your SOCKS server
• server_type – tsocks defaults to SOCKS4 mode. You may wish to set it to 5 for SOCKS5 usage.
• tordns_enable – this needs to be set as false if you don’t use Tor.

Using tsocks

Once this is set up, simply prefixing the network command you want to run with tsocks will force a diversion through the proxy connection. For example:

tsocks ssh example.com

The same can be applied to Subversion.

tsocks svn update

will force the svn client to act through the proxy set in tsocks.conf.

SOCKS on localhost

Note that SOCKS services on 127.0.0.1 has a minor gotcha. Sometimes, you are able to SSH into a remote machine, and use that connection as your SOCKS server. This is described in my post about using SSH as a pseudo-VPN, which describes the -D switch. My use case here is that once you do this, all further local SSH connections to other machines should be diverted through the first SSH. For example, I’d like to do:

my-machine$ ssh -D 40000 gateway.example.com # establish a SOCKS server on localhost:40000 to the gateway host

and then:

my-machine$ ssh lan-1.example.com # access the protected lan-1 machine through the SOCKS, which will see me as gateway.example.com 

This is very doable in the tsocks setup if you set tsocks.conf:

server = 127.0.0.1/255.255.255.255
server_port = 40000

and then:

my-machine$ ssh -D 40000 gateway.example.com
my-machine$ tsocks ssh lan-1.example.com

This is the gotcha: make sure the netmask is set correctly to 255.255.255.255. Otherwise tsocks will die with a cryptic:

IP (127.0.0.1) & SUBNET (0.0.0.0) != IP on line 22 in configuration file, ignored

It is apparently fairly sensitive about the subnet mask setup to conform to exact standards.

With this tsocks setup, you won’t have to create special VPNs to lock a LAN machine behind a gateway. As long as you can SSH into the gateway machine from your local machine, you can access the resources behind it with any application on your local machine via tsocks. Nifty, huh?

Undefined symbols for architecture i386:
  "_xmlTextReaderSchemaValidate", referenced from:
      _zim_xmlreader_setSchema in php_xmlreader.o
  "_xmlTextReaderSetup", referenced from:
      _zim_xmlreader_XML in php_xmlreader.o
ld: symbol(s) not found for architecture i386
collect2: ld returned 1 exit status
Undefined symbols for architecture x86_64:
  "_xmlTextReaderSchemaValidate", referenced from:
      _zim_xmlreader_setSchema in php_xmlreader.o
  "_xmlTextReaderSetup", referenced from:
      _zim_xmlreader_XML in php_xmlreader.o
ld: symbol(s) not found for architecture x86_64

This doesn’t only happen with libxml. If you’ve installed any extra updated libraries, like iconv or tidy or any library that has significant symbol changes between versions, it’ll die in similar ways. The MacPorts folks have encounted similar issues in ticket 15891, but WONTFIX‘ed the issue. Apparently the PHP devs are also punting on the problem.

The immediate cause is that you have multiple versions of some shared libraries. For example, in the case above, I have two libxml versions — one in /usr/lib, and another in /usr/local/lib. This is because I do not want to overwrite the Apple-provided libxml version, but still needed new features provided in later libxml versions. The arrangement works fine in every other software compile except this one, so I investigated further.

The root of the problem

Despite the developers’ airy dismissal of the issue, the underlying problem is indeed that the Makefile generated by PHP at configure time is slightly broken. In Makefile and Makefile.global, you’re going to see this line:

libs/libphp$(PHP_MAJOR_VERSION).bundle: $(PHP_GLOBAL_OBJS) $(PHP_SAPI_OBJS)
        $(CC) $(MH_BUNDLE_FLAGS) $(CFLAGS_CLEAN) $(EXTRA_CFLAGS) $(LDFLAGS) $(EXTRA_LDFLAGS) $(PHP_GLOBAL_OBJS:.lo=.o) $(PHP_SAPI_OBJS:.lo=.o) $(PHP_FRAMEWORKS) $(EXTRA_LIBS) $(ZEND_EXTRA_LIBS) -o $@ && cp $@ libs/libphp$(PHP_MAJOR_VERSION).so

where $MH_BUNDLE_FLAGS is usually defined as something like

MH_BUNDLE_FLAGS = -bundle -bundle_loader /usr/sbin/httpd -L/usr/lib \
 -L/usr/lib -laprutil-1 -lsqlite3 -lexpat -liconv -L/usr/lib -lapr-1 -lpthread

The problem is that this hardcodes the search paths for linking shared libraries. GCC searches for shared libraries to link in the order of the provided -L paths. In this case, MH_BUNDLE_FLAGS is expanded immediately after $CC — so the load order is:

1. /usr/lib
2. /usr/lib (these are redundant, and so will probably be collapsed into one path)
3. …every other custom library path you specify

Now you see the issue. No matter what your library paths are set to, the PHP compilation system will insist that whatever shared libraries in /usr/lib take precedence. Therefore, even if you specified that another version (say, libxml.dylib in /usr/local/lib) should be used instead, the invocation to link against -lxml2 will search in /usr/lib first. And since it finds the old version, which may be missing a number of symbols, the compilation blows up right there.

Evidence

And indeed, if you look at the (rather long and massive) compilation/link command right before it fails, you’ll see:

gcc -bundle -bundle_loader /usr/sbin/httpd -L/usr/lib -L/usr/lib \
-laprutil-1 -lsqlite3 -lexpat  -liconv -L/usr/lib -lapr-1 -lpthread -O2 -I/usr/include -DZTS   \
-arch i386 -arch x86_64 -L/usr/local/lib ... 

emphasis mine, where /usr/local/lib might be /opt/lib or whatever custom path you provided to configure.

Solutions

The trivial solution is to manually invoke that last line of compilation, but swapping the -L load paths.

gcc -bundle -bundle_loader /usr/sbin/httpd -L/usr/local/lib -L/usr/lib \
-L/usr/lib -laprutil-1 -lsqlite3 -lexpat  -liconv -L/usr/lib -lapr-1 -lpthread -O2 -I/usr/include -DZTS   \
-arch i386 -arch x86_64  ... 

This is easy to do and takes just a second.

Another possible solution is to patch the Makefile, such that MH_BUNDLE_FLAGS comes later in the compilation line:

libs/libphp$(PHP_MAJOR_VERSION).bundle: $(PHP_GLOBAL_OBJS) $(PHP_SAPI_OBJS)
        $(CC) $(CFLAGS_CLEAN) $(EXTRA_CFLAGS) $(LDFLAGS) $(EXTRA_LDFLAGS) $(PHP_GLOBAL_OBJS:.lo=.o) $(PHP_SAPI_OBJS:.lo=.o) $(PHP_FRAMEWORKS) $(EXTRA_LIBS) $(ZEND_EXTRA_LIBS) $(MH_BUNDLE_FLAGS) -o $@ && cp $@ libs/libphp$(PHP_MAJOR_VERSION).so

This will force your library paths to be searched before /usr/lib, thus resolving the link problem.

update 7/18/09
An anonymous reader mentions that you could also specify the right libxml by full path, instead of letting it use -lxml. Basically, in the last compilation line, you would remove any mentions of -lxml and replace that with the full path to your library e.g. /usr/local/lib/libxml.dylib. In fact, this is probably the way that has the least possible side-effects, since you aren’t changing the search order for any other libraries.

Discussion

This is not the first time that PHP core developers have refused to fix a compilation issue that is arguably preventable through actual testing under different installation scenarios. This is an “edgier” edge case than the tidy.h issue, but still should be fairly noticeable for a substantial number of people.

The “You should only have one library installed” argument is, to be honest, unnecessarily arrogant (sadly, not as a rare a problem as you’d like in some open source development projects ). I understand that it’s an open source project, and no self-respecting software engineer likes to use time on project plumbing / build systems rather than work on the product. However, on OS X, due to the lack of official Apple package management systems, no one should be overwriting system default libraries — down that way lies insanity, especially at every system or security update. PHP’s build system is obviously broken any time there is a substantial difference between user-installed libraries and system libraries. This bad behavior is especially egregious, because the configure command allows you specify your own library path — misleading users into thinking that the path they specified would be obeyed at compile time. If you only intend for the system library to be used and no other, perhaps the configure script should auto-detect this on OS X and disable that command-line option. Basic user interface design should apply even to command-line interfaces.

Note that changing link ordering may have some unforseen consequences, since the devs obviously never tested this path. For example, you should make sure the dynamic libraries are loaded in the right order at runtime. On OS X, the load path is typically hard-coded into the dylib, so usually there won’t be a problem — but there may be edge cases. Test your build (and any PHP extensions you built) before using it in production!

Establishing a connection

To start with, the usual process is to telnet to a POP3 server port, usually on TCP port 110. This would be very simple:
telnet mail.example.com 110

Nowadays, though, most POP3 servers are secured via SSL, usually sitting on port 995. If you try to use telnet on an SSL-only POP3 server, you’ll either get an error “Command is not valid in this state”, such as:

Trying 127.0.0.1...
Connected to mail.example.com.
+OK The Microsoft Exchange POP3 service is ready.
USER yiming
-ERR Command is not valid in this state.

or you’ll get a rather brusque brushoff

Trying 10.0.1.202...
Connected to mail2.example.com.
Escape character is '^]'.
USER yiming
Connection closed by foreign host.

When this is encountered, OpenSSL’s s_client should be used instead to perform the necessary SSL negotiations.

openssl s_client -connect mail.example.com:995

or

openssl s_client -crlf -connect mail.example.com:110 -starttls pop3

The second incantation is typically used for Microsoft Exchange servers. Note the -crlf option, which tells s_client to send \r\n line endings. If the wrong line ending is used for a server, the symptom is that the server will not respond to any commands. It will only sit there and wait for further input, while you are staring at a blank responses or blank lines in your telnet session.

Authentication

Having established a connection, it is now necessary to authenticate as a POP3 user. In the simplest case, plain text authentication is used. In this case, the command USER [username] is used to establish the username, and PASS [password] is used to establish the password in plaintext. (Since the connection is under SSL encryption, presumably this plaintext won’t matter).

+OK Server ready
USER yiming
+OK
PASS foobar
+OK Logged in.

Server interactions

Several commands are useful here.

• LIST – lists the messages available in the user’s account, returning a status message and list with each row containing a message number and the size of that message in bytes
• STAT – returns a status message, the number of messages in the mailbox, and the size of the mailbox in bytes
• RETR [message_num] – returns the message identified by the message number, which is the same as the message number shown in the LIST command output
• TOP [message_num] [n] – returns the top n lines of the message denoted by message number.

When finished, the QUIT command will end the session.

Conclusion

For other POP3 commands, such as commands marking deletion of a message, refer to RFC 1939, the canonical document defining the Post Office Protocol Version 3 ( POP3 ). At some point, if the commands to be tested become complicated, it may be more efficient use of time to install a mail client such as alpine.

See also my previous post on chatting with HTTP / HTTPS servers.

What’s less new is the sudo path inheritance issue — this one’s been around. Ubuntu’s sudo hard-codes its PATH variable at compile-time with a --secure-path option. I’m sure this sounded like a good idea to the security goon who decided to fix this at fsckin’ COMPILE TIME with no way to override it in sudoers, or at runtime with -E after an env_reset. The policy may have been reasonable when it was set on a typical Debian stable server (where software is basically left to fossilize over decades), but certainly not on a constantly changing desktop distro. You can’t even sudo to any /opt/bin binaries! Read the Ubuntu bug report on sudo not preserving PATH.

Long story short, after a lot of experiments looking for workarounds (that won’t eventually take years off my life, one sudo command at a time), I decided to cut the Gordian knot and recompile sudo. Since I didn’t want to roll this from source (and incur all the maintenance hassle of removing/updating the software later on), this meant figuring out compiling source packages with dpkg — oh joy.

Debian source package compilation: the general process

It’s surprisingly non-painful compared with my RPM experience. The long way around:

1. cd into a temp or source-keeping directory in your user account
2. retrieve the source package: apt-get source [packagename]
3. grab missing build dependencies: sudo apt-get build-dep [packagename]
4. cd into the directory created for the package in your pwd (you can safely ignore the original tarball and the patch file, which have been untarred and applied for you already, respectively). Make edits to the source as needed.
5. If you need to change configure options for the source package, look in the file debian/rules in the source directory
6. when satisfied, build the binary package by issuing this incantation in the $PWD ( you’ll need the fakeroot package if you don’t already have it ):
   dpkg-buildpackage -rfakeroot -uc -b
   Use -nc if you mess up and need to continue a build.
7. The completed .deb packages are placed in the parent directory, one level up from the source directory. cd back up one level.
8. install: sudo dpkg -i [packagename].deb

If you’re screwing around with sudo, you will want to have a sudo tty session open before installing your replacement package, in case you screw up everything and lock yourself out.

A shortcut is potentially available using the -b switch to apt-get when you grab from source. However, I needed to look through configuration files and source code, so I took the long way around.

The easiest way to fix the sudo secure_path issue is to remove the --with-secure-path configuration option in debian/rules, in two places in that file. If you do this, pay attention to your $PATH and make sure they are sane (for example: it shouldn’t contain a globally writeable directory), as it will be inherited in sudo shells. In sudo 1.7, there is a runtime secure_path option for the sudoers file, so that would be the ideal, non-annoying solution to this issue.

Hard-coding the sudo PATH at compile-time tilts heavily toward security in the security/usability tradeoff — YMMV, but I find it entirely not worth it on a desktop distribution.

echo $?

Given the nature of this variable (no one indexes text like ‘$?’), it’s annoyingly hard to Google for.

There appears to be some sort of limitation on screen colors when using ARD 3.2.2 to control an older Mac remotely. The symptom of this is a black screen when you attempt to Observe or Control the remote machine. Unfortunately, this same symptom usually appears when you have a blocked network port (ARD uses TCP and UDP ports 3283 and 5900), so it may be confusing as to which is the issue.

After verifying all network settings and router port forwardings are set up correctly, you might try this if you have an older Mac as the target: move the color slider on the top-right corner of your ARD admin panel to a lower value, and then try to reconnect.

The story is that I was trying to remote control a G4 dual 500 ( Mystic ) from a MacBook Pro (early 2008). This used to work until recently, when I had nothing but a black screen. Keyboard commands still worked (I can blindly log in from the loginwindow), though mouse movements did not pass through to the old Mac.

After fruitlessly chasing network issues with my AirPort router, the last post at an Apple Discussions thread pointed me to the right direction. Once I used the color control on the ARD application to lower the color depth by 1 notch, the next connection worked just fine, with the screen showing up and behaving normally.

Now this poor old G4 tower is running 10.4.11 with an ancient Rage 128 Pro graphics card, but it handles its 17inch screen just fine at “million of colors” color depth when sitting in front of it. Very odd how it just stopped working at that depth over ARD.

Only NPAPI plugins are apparently allowed from 5.1 forward. This API does not give native access to the browser window as before; therefore the old methods of converting .doc files to HTML no longer works. Because I still need Word preview capability for myself, I have a few ideas on how to make DocPreview work for 5.1+. However, this requires a complete re-write of the entire code. Apple obviously already has .doc preview capability on iOS but hasn’t shared that with us on OS X. I’m not sure it’ll be worth it to make these modifications, only for Apple to release .doc preview as a native capability to Safari a few months later.

Again, any NPAPI experts willing to work with me on this issue — please contact me.

12/1/2010 – As promised, DocPreview is now open sourced at Google Code.

01/24/2010 – DocPreview updated for 64-bit Safari and 10.6. Still seems to work.

early 2009 – Schubert|IT’s Word Browser Plugin has been made a Universal binary. Therefore, DocPreview is no longer needed, as I originally wrote this plugin to fill in the gap when Word Browser Plugin was PPC-only. Word Browser Plugin offers a more advanced user interface than DocPreview. You should try it out first. I will continue to tweak DocPreview, more as a challenge to myself. I will probably open source the package — at soon as I fix a hack or two and no longer feel ashamed of my code.

—

DocPreview is a lightweight WebKit browser plugin I wrote to display a text-only preview of Microsoft Word .doc documents, inline and within Safari 3.x (and, apparently, 4.x) on both Intel and PowerPC Macs. This behavior is much like the functionality provided by the PPC-only Word Browser Plugin from Schubert|IT. DocPreview, of course, is a universal binary plug-in — since Word Browser’s lack of Intel support is/was really why I wrote this plug-in.

DocPreview only supports WebKit-based browsers that can use .webplugin files. Therefore, this includes Safari, Shiira, etc., but not Chrome (as of the time of this update). I am currently unable to support other browsers, since DocPreview is built against the WebKit API instead of the Netscape plug-in API. Frankly, the Netscape API is a mess, and I haven’t had the time to figure it out. A NSAPI guru who could point me in the right direction would be much appreciated.

Download
DocPreview.zip (Safari 5.0 and below on 10.5, 10.6) – v0.1
DocPreview.zip (10.4) – v0.1

This is a work-in-progress. It passes the “dogfooding” test; as in, I’m using this plug-in daily (eating my own dogfood, as they say). While I believe it functions correctly — at least on my machines, I make no guarantees as to stability and usefulness, and am not liable for blowing up your browser or any harm that might befall you through your adventurous use of this plug-in. I do want this thing to work for everyone, so please leave me a note here if it doesn’t work.

DocPreview features & limitations:

• Universal binary support, for use on both PPC and Intel Macs. No Windows Safari support — and plus, there are already good solutions for inline doc browsing on the Windows side. If you’re on PPC, I suggest you use Word Browser Plugin instead unless you desperately need find-on-page using Safari’s built-in facilities with Word docs.
• Tested extensively on Safari 4.x, 3.x, and somewhat with Shiira 2.x. May also work on earlier Safari versions, but I simply do not have the ability to test the plugin against them. Does not work on Chrome, but I’m sure there are better solutions using Chrome extensions.
• Uses OS X’s internal engine for opening and processing Word files. DocPreview performs as well (or as badly, depending on your opinion) as OS X itself for the same task.
• Supported on OS X 10.5 for .doc, .docx, and .odt files. On OS X 10.5, the plugin can parse Microsoft’s new OpenXML (.docx) and OO.org‘s ODF Text (.odt) documents. On 10.4, the plugin will still work, but only for .doc files.
• Support for full document view mode or embedded mode (if the page author uses object and embed tags, like with Flash objects — although, who actually does that with .doc files? )
• In full document view mode, DocPreview uses Safari’s built-in Find and Text Zoom abilities. Any command that can be run on a normal Safari webpage can be performed on a DocPreview rendering.
• In embedded mode, Find on Page and Text Zoom support are implemented separately. I’m still thinking about how I can hook into Safari’s built-in system from within an embedded plugin. Since no one ever uses embedded .doc files anyway, the point is fairly moot for most users.

To install, drag DocPreview.webplugin into
/Users/<your name>/Library/Internet Plug-ins
or
/Library/Internet Plug-ins.

Screenshots of the plug-in in operation can be seen to the right of this post. The first pair of images are .doc files in full document mode in the browser and in Word 2008. The second pair of images are two documents embedded inline using object tags, and one of them shown in Word 2008. As you can see, the conversion fidelity is fairly decent — this is the same level of fidelity that you would have gotten by importing Word data into TextEdit, or using textutil on the command-line.

DocPreview serves the same purpose as Word Browser. It’s intended as a quick preview (much like how Google indexes the text in .doc files), so you don’t blindly download any Word files that actually don’t interest you.

( As an aside, I cannot understand people who want to disable built-in PDF support in Safari :p . That feature has singlehandedly improved my productivity/research output by a magnitude. )

I expect Apple to support .doc previews with the next version(s) of Safari, very soon (ok, so not in 4.x, but surely in 5.x. Anyone on the Apple Safari team reading this: come on, this feature is trivial — even I can do it ). MobileSafari on the iPhone already provides .doc viewing support (and .xls, if I remember correctly). On Windows, MS provides read support for .doc files within the browser. DocPreview is a temporary solution for Safari users until official Apple support (or better yet, better Microsoft Office integration) arrives.

If the thing doesn’t work for you, let me know via the comment thread. If it does work, let me know too. Any suggestions and comments welcome.

Except, of course, the Apple AirPort and AirPort Express routers don’t actually have this feature. This is understandable, considering that MAC addresses are supposed to be globally unique so as to avoid network problems — allowing the user to change it willy-nilly will probably not serve that goal. Nevertheless, it is quite annoying in this case, as you won’t be able to change IP with an AirPort router using the MAC cloning method.

There’s one trick that might work for you here, though. A few ISPs’ DHCP servers are set up to accept DHCP Client IDs — an optional field that identifies a DHCP client. While they default to use MAC address as a client identifier by default, they will treat you as a different DHCP client for IP assignment if you start using or change a Client ID. All that is required to obtain a new IP address, then, is to add a Client ID or change it. There is an interface for this in the AirPort Utility, as seen in the screenshot. This is far easier than trying to clone a MAC address on AirPort routers.

Not all ISPs support use of the DHCP Client ID, so this may not work for everyone. Since there are uniqueness requirements involved with these client IDs, it is a good way to screw up DHCP assignments if two clients claim the same ID. If your ISP does support this, make sure to pick a unique client ID.

This trick appears to work for Cox Cable at my current location. It does not appear to work for Comcast.

In a bizarre case of digital food poisoning, I experienced a series of mysterious, persistent, reproducible crashes with Safari 3.2.1 this morning, traceable to a bad Google cookie.

The symptoms

Google has a nifty query suggestion feature that is turned on by default on its homepage search box. Whenever I typed in a phrase query (e.g. +"query suggestion" +"Google features") with the suggestion feature turned on, the browser crashed with a SIGSEGV around 30% of the time.

Excerpt from the crash log:

Exception Type:  EXC_BAD_ACCESS (SIGSEGV)
Exception Codes: KERN_INVALID_ADDRESS at 0x000000001bdca240
Crashed Thread:  0

Thread 0 Crashed:
0   ???                            0x16619e75 0 + 375496309
1   com.apple.WebCore              0x94325ea0 WebCore::AutoTableLayout::fullRecalc() + 704
2   com.apple.WebCore              0x9432581a WebCore::AutoTableLayout::calcPrefWidths(int&, int&) + 26
3   com.apple.WebCore              0x943252b8 WebCore::RenderTable::calcPrefWidths() + 56
4   com.apple.WebCore              0x9431c04b WebCore::RenderBox::minPrefWidth() const + 27
5   com.apple.WebCore              0x9432507c WebCore::RenderTable::calcWidth() + 124
6   com.apple.WebCore              0x943241a8 WebCore::RenderTable::layout() + 392
...

In the remainder of the cases, when it does not crash immediately, a JavaScript error is logged to the browser error console (to access, go to Develop -> Show Error Console)
SyntaxError: Invalid regular expression: nothing to repeat
http://www.google.com/extern_js/f/CgJlbhICdXMrMAc4AiwrMAo4EywrMA44AywrMBg4Ayw/nMD0sKnpeG0.js (line 21)
for every letter that I type into the search box. During this time, no query suggestion is made.

Diagnostics

• I have never used an InputManager or “plug-in” to Safari
• The same crash does NOT happen under a fresh new user account created for diagnostic purposes
• Clearing the browser cache, temp files, hidden cache files ( getconf DARWIN_USER_CACHE_DIR ), etc. did not help.
• Deleting Safari preferences did not help.

Solution

After applying a divide-and-conquer strategy to the entire ~/Library directory (not made any easier by Finder’s obstinate resistance to my attempt to move subdirectories within the Library directory, despite having the appropriate permissions — had to drop to Terminal for this), I traced it to the ~/Library/Cookies directory. Moving away the Cookies.plist file contained within cured the crash, the lack of query suggestions, and the Javascript error. More specifically, deleting all Google-related cookies within the Cookies file also accomplished the same thing.

Remarks

Some combination of a bad cookie and bad regexes appears to have triggered a crash bug in this version of WebKit / WebCore. You wouldn’t think a bad cookie could take down a browser. But apparently it does.

I dearly hope this is not a potential buffer overflow or other security problem within WebKit.

A redacted transcript:

HELO example.com
250 Hello
MAIL FROM: foo@example.com
555 5.5.2 Syntax error.
MAIL FROM: <foo@example.com>
250 2.1.0 OK
RCPT TO: bar@example.com
555 5.5.2 Syntax error.
RCPT TO: <bar@example.com&gt
250 2.1.5 OK
DATA

....

I’m sure this is intended so that clients that send “Joe Schmoe <joe@example.com>” are correctly interpreted. I’m not sure if the RFC asks for this strict enforcement (on my admittedly cursory reading), though and yet a fair number of servers are rejecting the alternate address-only syntax. Wacky.

This post is my attempt to catalog all the event stills and how to obtain them in game. The events are listed below in order of their appearance on the View Event Stills page. In general, you only need to view the Event Still once. That means that you can start a game, obtain the Events you want, without actually completing or even saving the game once. Even if you abandon that game later, once you’ve viewed the event, it counts toward your completion of collecting all 30.

Note that I will not suggest any unreasonably contrived means to get events, even if that might be faster, because I consider these tactics basically cheating. For example, I will not suggest that you send your sovereign out to die deliberately to get the Sovereign Death event, even if you might be able to do that. You may also create officers with birth and death dates to your exact specification, to achieve some of these events, but that, too, is basically cheating. Most of the fun in these events is trying to obtain them fairly while playing a game — it adds an additional dimension of strategic planning. If you’re just going to cheat, stop reading this and go download the fully unlocked sysfile that’s floating around the Web.

Page 1

• Plague – when a city receives a Plague event, causing all sorts of havoc. You will likely get at least one in any scenario.
• Locusts – when a city receives a Locusts event, decreasing food production in that city. You will likely get at least one in any scenario.
• Plenty – when a city receives a Plenty event during harvest season, increasing food production in that city. You will likely get at least one in any scenario.
• Bandits – when bandits appear in your territories, as the name suggests. The easiest way to obtain this still is to start the Yellow Turban Rebellion scenario. During the story intro sequence, the Yellow Turbans will rise against the Han court, and this will unlock the event still.
• Coronation – when an Emperor is crowned, or when you declare support for the Emperor after capturing the capital (Luoyang, Chang’an, or Xu Chang, depending on the scenario). Easily achievable in most scenarios if you’re winning.
• Unity: Male – when you unite China under a male sovereign. This event will be unlocked when you win a scenario with a male sovereign. I prefer Liu Bei, but YMMV
• Unity: Female – notes not yet available
• Downfall – when one of the forces in the game is eliminated. At least one ruler will fall in any game, so this is trivial to obtain. You can also get this by losing your city in Tutorial 3, Going into Battle.
• Independence – when a prefect or viceroy declares independence and forms his or her own force in a game. In Tutorial 6, Capturing Cheng Du, there is a good chance that Jian Ning will declare independence under the prefect Yong Kai, but it is not guaranteed. In the 190 scenario, the free cities in Jiangdong may declare independence under their prefects within the first two years. In any scenario, foment an independence movement in an enemy city via the strategy Collaborate. Alternatively, wait until one of your district viceroys’ or city prefects’ loyalty falls below a certain threshold, at which point an AI force is likely to foment a revolt in that prefecture or district. Finally, when you’re winning a full scenario, at the very end of the game one of your officers will ask you to investigate another officer for treason. If you choose to do so, that officer will be proven innocent, but will revolt anyway due to your distrust of him. This revolt will trigger an Independence event.
• Omen – This event is displayed at the start of the year, when your strategist warns that one of your officers is about to die (of natural causes, not in battle). If you play through a game with historical age settings, quite a few of your officers will die. Pick a middle scenario, when there are a lot of older officers around.
• Sovereign Death – when your own sovereign dies of natural causes. The easiest way to get this one, short of deliberate regicide on your part, is to use Tao Qian in 194, Rival Warlords. He will most likely die within the next two years, giving you this event.

Page 2

• Officer Death : Male – When one of your male officers die. This is relatively trivial to get – one will come within the year, after an Omen, for example. Your officer may also be killed by a cavalry charge or in a duel, if officer deaths are on.
• Officer Death : Female – When one of your female officers die. This is a little more annoying, given the sparsity of female officers. Still, at least one should die in any game. Your officer may also be killed by a cavalry charge or in a duel.
• Funeral – When any sovereign (except yours) dies of natural causes — that is, of illness or old age, not in battle. In the 194 scenarios, Tao Qian will die within the next two years. Play the scenario as a ruler other than him, and you will trigger this event still.
• 2 Sworn Brothers – when you arrange for two male officers to swear brotherhood with each other. Officers with sworn brothers on the same force will not defect, and will obtain bonuses when both are assigned to the same unit. You’ll need two officers of the male gender and at least 500 in Deeds each. You will also need 500 technique points (TPs). At that point, Sovereign -> Mediate will allow you to mediate sworn brotherhood between these two officers and unlock this event.
• 2 Sworn Sisters – when you arrange for two female officers to swear sisterhood with each other. Officers with sworn sisters on the same force will not defect, and will obtain bonuses when both are assigned to the same unit. You’ll need two officers of the female persuasion and at least 500 in Deeds each. You will also need 500 technique points (TPs). At that point, Sovereign -> Mediate will allow you to mediate sworn sisterhood between these two officers and unlock this event.
• Male Officer Appears – Triggers at the start of a new year, when a male officer (who is related to one of your current officers) arrives to offer his services to your ruler. For example, as Liu Bei in 211 Liu Bei in Shu scenario, you will likely receive Zhang Bao – Zhang Fei’s son, in 212. Note that your sovereign should be within a city for this event to occur.
• Marriage – when you marry one of your officers to another – married officers will not defect, and obtain bonuses when assigned to the same unit. You’ll need two officers of opposite gender, at least 500 in Deeds, and unmarried status. You will also need as 500 technique points (TPs). At that point, Sovereign -> Mediate will allow you to conduct a marriage between these officers and unlock this event.
• Fem. Officer Appears – Triggers at the start of a new year, when a female officer (who is related to one of your current officers) arrives to offer her services to your ruler. This one is fairly tricky, because there aren’t that many female officers. In general, play one of the middle scenarios as a prominent family. For example, if you play as Sun Quan in 211, you will get a female officer from the Sun family in 215.
• Resignation – when an officer resigns from your force due to low loyalty. This is also a hard one to get if you are a decent player, since you’d never let an officer’s loyalty drop low enough. Note that if your sovereign dies and your officers resign, you will not receive this event still. You must have this officer resign through low loyalty. Officers with loyalty less than 70 will tend to resign, and they usually do so on Jan of a new year. Best way to do this may be to play as Dong Zhuo in 190. Use your overwhelming starting forces to take down Cao Cao, then hire his wife Bian Shi. Use Bian Shi to hire Cao Cao. Dong Zhuo and Cao Cao are disliked officers, so Cao Cao’s loyalty will start in the 70s and drop fast. However, since you have his wife in your service, he will not be hired away by another sovereign. At the start of the next year, Cao Cao will most likely resign from your force.
• Decree – when an imperial decree arrives. For example, when the Emperor promotes your sovereign to the rank of Duke, he will use this kind of decree. Since to win a game, you’ll likely surpass the rank of Duke, this event is trivial to unlock if you win a game.
• Orders – when a different type of imperial order arrives. If you start 190 Cao Cao's Ascent, the story intro sequence to the scenario involves Cao Cao’s forged imperial edict arriving at Yuan Shao’s city. This will grant you the event still.

Page 3

• Fallen Castle – when a city falls to your forces. You are likely to receive this in Tutorial 6, Capturing Cheng Du…when you capture Cheng Du. Otherwise, take any city or base, and the event is yours. Trivial.
• Conversation – when one of your civil officers head off to search for a friend in the area and obtains his or her services after a debate. This seems to occur randomly, when your sovereign is in the same city as certain civil officers on his force. It is the analogous event to “Search”, which applies only to military (high WAR) officers. The event represents the “conversation” between your officer and his friend. This obviously only triggers when there are hidden officers to find in a city — so start an early scenario with lots of hidden officers for a better shot at the event.
• Coalition – pick up this event when the coalition against Dong Zhuo forms in 190 Cao Cao's Ascent, at the very beginning during the story intro sequence. Otherwise, play as a strong army (say, Cao Cao in 207), and eventually two of your weaker enemies will merge to form a coalition against you.
• Search – when one of your military officers head off to search for a famous hero in the area and obtains his or her services after a duel. This seems to occur either randomly or scripted to resemble randomness. This obviously only triggers when there are hidden officers to find in a city — so start an early scenario with lots of hidden officers for a better shot at the event. The 190 scenario with Cao Cao in Xu Chang or Chen Liu should be an easy way to get this event, as you recruit various notables like Dian Wei or Xu Chu.
• Secret Meeting – when you meet a hidden character for a philosophical discussion. This one is ridiculously obscure and difficult to obtain, to the effect that many people do not know how to obtain it. Forum posts and such suggest that you’d receive this when the AI foments unrest with one of your viceroys or prefects (one of the in-game log messages say “your prefect blah is meeting someone secretly”. However, that is a red herring. One post correctly identifies that the Secret Meeting event is awarded for meeting a hidden character such as Xu Shao during a Search. As Cao Cao in 190 Cao Cao's Ascent, if you keep Searching the area around Chen Liu or Ru Nan, against your strategist’s advice that nothing would be found, you will eventually encounter Xu Shao, who will do a physiogamy reading for you, unlocking this event. One of the early scenarios, such as Rise of Dong Zhuo, is preferred for obtaining this event. Just keep Searching your city, a lot, and fruitlessly most of the time. You’ll get this eventually ( took me under two game years of on-and-off searching in Chen Liu while holding off Dong Zhuo’s army ). A comment from Lance Lidner reports that using Kong Zhou in Rise of Dong Zhuo works best, as Xu Shao appears in Ru Nan at that time. Kong Zhou is also out of immediate reach from most hostile factions in that scenario, so you can afford to take your time.
• Appoint – when the emperor issues a new title to you. This occurs when you’ve captured enough cities to reach the next rank, and that your sovereign is currently in a city (as opposed to being on campaign). As Liu Bei in the 211 Liu Bei in Shu scenario, taking Xiang Yang (via Jiang Ling – you’ve got Zhuge Liang, Pang Tong, Zhang Fei, Liu Bei, Sun Shang Xiang, Zhao Yun … that’s plenty to take the city from Cao Ren early on) and Yong An will bump you up to Field Marshal, and Emperor Xian will award you the title and this event. As Cao Cao in 190 Cao Cao's Ascent, taking the free cities of Xu Chang and Xin Ye will grant the title of Lt. Governor, which once again easily unlocks this event.
• 3 Sworn Brothers – when you arrange for three male officers to swear brotherhood, much like the historical Oath of the Peach Garden of Liu Bei, Guan Yu, and Zhang Fei. Officers with sworn brothers on the same force will not defect, and will obtain bonuses when two or more are assigned to the same unit. You’ll need three officers of the male gender and at least 500 in Deeds each. You will also need 500 technique points (TPs). At that point, Sovereign -> Mediate will allow you to mediate sworn brotherhood between these officers and unlock this event.
• 3 Sworn Sisters – when you arrange for three female officers to swear sisterhood. Officers with sworn sisters on the same force will not defect, and will obtain bonuses when two or more are assigned to the same unit. You’ll need three officers of the female persuasion and at least 500 in Deeds each. You will also need 500 technique points (TPs). At that point, Sovereign -> Mediate will allow you to mediate sworn sisterhood between these officers and unlock this event. The hardest part of this is rounding up 3 female officers to begin with.

ActiveRecord::StatementInvalid: Mysql::Error: Column 'attr2' cannot be null: INSERT INTO `foo` (`attr1`, `attr2`) ... VALUES ('1', NULL)

where attr2 is a varchar (or t.string, in Rails lingo) and set to not null default '' (or, in other words, :null=>false, :default=>''). Strangely enough, instead of the default value of ”, ActiveRecord was setting the value to nil instead, which translates into a NULL. Since the schema explicitly forbids NULLs on that column, the statement explodes.

After an hour of poking around and hacking up a spike solution, it turns out a plugin was to blame. I’d pulled in the foreign_key_migrations plugin (a highly recommended add-on) to automatically install foreign key constraints (in this day and age, the foremost web framework still can’t automatically handle FOREIGN KEY constraints, the most basic tool for ensuring data integrity in relational databases, for its migrations? Bah!).

This plugin has a dependency on redhillonrails_core, which has a known bug: Incorrectly overwrites mysql empty-string default with nil for string/text/binary types.

The bug is apparently not being worked on as of the time of this writing. The dev doesn’t consider this a bug, as he claims that “[he considers] empty strings to be semantically identical to NULL”.

This position, unfortunately, is not supported by the SQL standard. Wikipedia has a section on common SQL NULL mistakes documenting some of the potential problems involved in making such an assumption. Philip Greenspun has further notes on this. Using NULLs trigger the all the arcane annoyances of three-valued logic, and you must be prepared to consider True, False, and NULL values as comparison outcomes. Someone not very versed in three-valued logic can easily cause a number of subtle mistakes trying to compare values.

To workaround this bug, you will need to comment out the initializer function in

vendor/plugins/redhillonrails_core/lib/red_hill_consulting/core/active_record/connection_adapters/mysql_column.rb

Alternatively, you can delete the file, and remove relevant references to it..

If you agree with dev’s position that NULLs are “semantically identical” to empty strings, remember to pay attention when you formulate your SQL queries (and when Rails formulates those queries) — your results may not be what you expect, if implemented naively. Get your three-valued logic truth tables out

/bin/sh /tmp/subversion-1.5.3/apr/libtool --silent --mode=compile gcc-4.2 -Os -arch i386 -arch x86_64 -DHAVE_CONFIG_H -DDARWIN -DSIGPROCMASK_SETS_THREAD_MASK -no-cpp-precomp -I./include -I/tmp/subversion-1.5.3/apr/include/arch/unix -I./include/arch/unix -I/tmp/subversion-1.5.3/apr/include/arch/unix -I/tmp/subversion-1.5.3/apr/include -o strings/apr_snprintf.lo -c strings/apr_snprintf.c && touch strings/apr_snprintf.lo
strings/apr_snprintf.c: In function ‘conv_os_thread_t’:
strings/apr_snprintf.c:500: error: duplicate case value
strings/apr_snprintf.c:498: error: previously used here
strings/apr_snprintf.c: In function ‘conv_os_thread_t_hex’:
strings/apr_snprintf.c:671: error: duplicate case value
strings/apr_snprintf.c:669: error: previously used here

This will most likely happen when you are configured to build a dual 32-bit / 64-bit universal binary, whether it be ppc / ppc64, or i386 / x86_64, or any permutation thereof. This ticket over at MacPorts documents a particular instance of this problem, with no apparent solution.

The symptom is easy to explain. Somehow, two case labels in the relevant switch statement in strings/apr_snprintf.c:500:

switch(sizeof(u.tid)) {
    case sizeof(apr_int32_t):
        return conv_10(u.u32, TRUE, &is_negative, buf_end, len);
    case sizeof(apr_int64_t):
        return conv_10_quad(u.u64, TRUE, &is_negative, buf_end, len);
    default:
        /* not implemented; stick 0 in the buffer */
        return conv_10(0, TRUE, &is_negative, buf_end, len);
    }

have evaluated to the same value. In particular, it believes that sizeof(apr_int32_t) and sizeof(apr_int64_t) are the same value. As we all know in C, you cannot have two identical case labels in the same switch statement. However, the root of the problem is a bit more subtle.

In $SRCDIR/include/apr.h, you’re likely to see this fragment of code.

typedef  long       apr_int64_t;
typedef  unsigned long  apr_uint64_t;

Notice that it has typdef’ed apr_int64_t as a long and apr_uint64_t as unsigned long. This is because at configure time, the script detected that long values are 64-bit on this system, so it assigned the apache 64-bit types to longs. However, this only holds true for half of the compilation – because you are building a universal binary for a 32-bit architecture as well. Remember that in 32-bit GCC on OS X, longs are 32-bit rather than 64-bit. Your run-of-the-mill autoconf script, done by a non-OS X programmer, isn’t going to be able to detect this subtlety – if the 64-bit part worked, it’ll keep thinking longs are 64-bit, end of story – and happily generate the incorrect typedef expressions. When you apply sizeof to these types in apr_snprintf.c, both evaluate to 4 bytes under 32-bit compilation, thus blowing up the compile run.

To truly fix the root of the problem requires rewriting the autoconf script to detect Mac OS X and its universal binary building, which can potentially throw quadruple architectures at the same compilation script. However, a quick hack to make this particular problem go away is to change apr.h such that:

typedef  long long       apr_int64_t;
typedef  unsigned long long apr_uint64_t;

Now that we ensure in either 32-bit or 64-bit compilation, apr_int64_t and apr_uint64_t are always typedef’ed to appropriate, guaranteed 64-bit types. The compilation of APR (and Subversion) will proceed normally.

Note that long long is not an standard C type. As a GCC extension, this fix is a kludge. A kludge that works (for me), though.

UPDATE:
There may also be an issue with sizeof definitions that may cause the library to crash. In particular, there may be occurrences of

#define APR_SIZEOF_VOIDP 8

that were generated by configure. To fix this, you will need to remove the define and have the compiler check for 64-bit at compile-time:

#ifndef __LP64__
    #define APR_SIZEOF_VOIDP 4
#else
    #define APR_SIZEOF_VOIDP 8
#endif

In general, any predefined sizeofs need to be changed. I am not sure why the APR developers do hard-coded defines like this, given that the point of having sizeof() calls is to avoid such issues.

Problem 1: transcoding to mp3 insists on using 64kbits/s or 128kbits/s bitrate

So a simple task is to transcode an audio file of some arbitrary format to MP3. In my case, I only wanted a 30-second piece of the original file, converted to MP3. The original was in 256kbits/s, and I decided to use the same bitrate for the output, just for kicks. Reading the man page, -b is for video bitrate, and -ab is for audio bitrate. So I executed:

wrong:  ffmpeg -ss 00:00:30.00 -t 25 -i foo.mp3 -ab 256 foo-new.mp3

The output, of course, was this:

Input #0, mp3, from 'foo.mp3':
...
mdb:109, lastbuf:0 skipping granule 0
size= 393kB time=25.2 bitrate= 128.0kbits/s
video:0kB audio:393kB global headers:0kB muxing overhead 0.007950%


128 kbits/s. That was…not what I wanted.

Turns out, as the ffmpeg man page hints but does not specify clearly, a “k” is required to label the units of the new bitrate. As in:

right:  ffmpeg -ss 00:00:30.00 -t 25 -i foo.mp3 -ab 256k foo-new.mp3

This time, the output was:
Input #0, mp3, from 'foo.mp3':
...
mdb:109, lastbuf:0 skipping granule 0
size= 782kB time=25.0 bitrate= 256.0kbits/s
video:0kB audio:782kB global headers:0kB muxing overhead 0.003996%

Much better. It’s very strange how that if I don’t give it the ending “k” but do give it a higher value than 64, it always bumps up the bitrate to 128kbits/s, from the default of 64k (but not to the number I actually wanted).

Problem 2: cutting media file without re-encoding

So ffmpeg can be used to cut a media file, without reencoding the media stream. You simply pass the raw copy codec to the -acodec (or the -vcodec for video) at encode time. For my MP3, I thought it was a pretty trivial problem, so I issued:

wrong: ffmpeg -i bar.mp3 -ss 00:00:30.00 -t 25 -acodec copy bar-new.mp3

This seemed to process correctly, except it created a 555-byte empty file with no content in it. What’s even weirder, if you issued:

wrong: ffmpeg -i bar.mp3 -ss 00:00:10.00 -t 25 -acodec copy bar-new.mp3

That is, -ss 00:00:10.00 to seek to the 10th second, and -t 25 to record 25 seconds worth of audio. Strangely enough, the output file had 15 seconds of audio, the subtraction of 25 by 10. Curiouser and curiouser. Now,

wrong: ffmpeg -i bar.mp3 -ss 00:00:10.00 -t 25 -acodec mp3 bar-new.mp3

does in fact create a 25-second MP3 file. Only, of course, a re-encoded one at 64kbits/s. Something that we are not looking for. But strange.

A post on ffmpeg-devel finally provided enough hints to clue me into the problem. Unlike many command line apps, the order of arguments passed to ffmpeg seem to be silently significant. The correct incantation is:

right: ffmpeg -ss 00:00:30.00 -t 25 -i bar.mp3 -acodec copy bar-new.mp3

Note the switched order for the arguments -i and the args -ss and -t, where the -i must follow the other two. Now, the desired 25-second file, cut from the original, is correctly produced.

When arguments are missing some arbitrary text or in the wrong order, ffmpeg doesn’t sanity-check or warn you of these…it just silently proceeds and does some very strange things. Things that make you scratch your head and wonder, “wtf? did I mistype an argument somewhere?”

Ah, and of course, you make me have to track down all these little idiosyncrasies, wading through blog posts and mailing lists and forums. Agh.

You may get this error while trying to compile Subversion 1.5.2:
ld: Undefined symbols:
_svn_fs_txn_root_base_revision referenced from libsvn expected to be defined in libsvn
_svn_fs_change_txn_props referenced from libsvn expected to be defined in libsvn
_svn_fs_get_mergeinfo referenced from libsvn expected to be defined in libsvn
_svn_fs_recover referenced from libsvn expected to be defined in libsvn
_svn_fs_upgrade referenced from libsvn expected to be defined in libsvn
_svn_fs_node_origin_rev referenced from libsvn expected to be defined in libsvn
/usr/bin/libtool: internal link edit command failed
make: *** [subversion/libsvn_ra_local/libsvn_ra_local-1.la] Error 1

Turns out there may be a bug in the libtool script shipped with the source tarball, under certain circumstances with OS X 10.4, that causes it to fail to link. Ah, libtool, you make my life so much harder sometimes.

If you copy /usr/bin/glibtool over $SRCDIR/libtool (that is, into the directory for the subversion source code, replacing the one that is placed there by the package itself), the package should compile with no further complaints. make test also shows success on all tests, so this seems a satisfactory solution.

UPDATE 1/31/2009
I’ve been informed in the comments that MacPorts may rely on source compilation from tarball, and thus have this issue. If you’re having this issue with the MacPorts SVN package, please check out the advice in the comment section from farkinga, who has additional notes.

This error has nothing to do with the particular code you’re compiling. Turns out GNU Make 3.80 has a bug, in which it sometimes sends a ridiculous requested block size to malloc(), which then fails to allocate the space.

The solution is to upgrade to GNUMake 3.81. Particularly pernicious on OS X 10.4 and Xcode 2.4.1.

So recently I was trying to interact with an avi file with a video stream encoded in H.264 and an audio stream encoded in MP3. To my surprise, my trusty Perian 1.1 failed to even play the video and gave me a black screen (though the audio played just fine). With some Google-fu, it was found in a forum post that the SVN version of Perian fixed this error.

So compile my own Perian 1.1.1b1 from SVN, right? No big deal. Heh. And thus begins our story. (Note that this procedure is accurate as of the time of writing. OSS projects change rapidly as bugfixes are checked in. You may not have this problem at all. But it’s still useful as a reference in the future, in case similar problems arise).

Problem 1: AVCodecContext.bits_per_sample

The first problem you hit is probably:
/tmp/perian-orig/ff_private.c:181: error: ‘struct AVCodecContext’ has no member named ‘bits_per_sample’
…
/tmp/perian-orig/FFissionCodec/FFissionDecoder.cpp:289: error: ‘struct AVCodecContext’ has no member named ‘bits_per_sample’

Turns out that the upstream project FFMPEG, who is responsible for libavcodec, has very recently changed the API in one of its latest revisions. bits_per_sample has become
bits_per_coded_sample in revision 15262. Because Perian’s SVN does an external SVN checkout of the HEAD of libavcodec, it pulled these API-breaking changes without considering whether that breaks its own compilation or not.

The fix is simple enough: global search and replace the member variable to bits_per_coded_sample.

Problem 2: Xcode’s unnatural fondness for dynamic libraries

At this point, libavcodec compiles, so you think you’re home free. For most users, that is probably the case. For some of us, though, there’s one more hurdle.

While compiling Perian.component, you might get something like:
ld warning: in /Developer/SDKs/MacOSX10.4u.sdk/usr/local/lib/libavcodec.dylib, file is not of required architecture
Undefined symbols:
"_avcodec_close", referenced from:
…and so on

Closer examination reveals that at some point, you might have compiled FFMPEG or MPlayer or any of the many media projects that depend on FFMPEG libraries (for me, that was transcode, whose own compilation idiosyncracies I documented in an earlier post). When you installed these libraries, they went into your library search path. Note the Perian compilation line immediately preceding the compile failure:
/Developer/usr/bin/g++ ... -read_only_relocs suppress -lavcodec -lavformat -lavutil -lebml -lmatroska ... /tmp/perian/UniversalDetector/build/Release/libuniversaldetector.a -lbz2 -o /tmp/perian/build/Deployment/Perian.component/Contents/MacOS/Perian

As you can see, the -lavcodec linker argument will cause the linker go on the search path and try to find the library. It of course first finds your pre-installed library in /usr/local/lib or some other system lib path, and tries to use it. If you compiled that library as Intel only or PPC only and you tried to build a universal Perian, that would obvious fail and generate an architecture error. If you, like me, compiled only for x86_64 or ppc64, that would also trigger the same architecture problem (as Perian builds as x86 32-bit).

This problem stumped a user “vs”, in this Perian-discuss Google groups thread. Notice how the devs didn’t get the root of the problem, and answered snarkily and defensively to the user bug report.

The devs aren’t wrong, though. Didn’t we just see a statically compiled target, libavcodec.a, in the target list for the Perian Xcode project? Why yes, yes we did! In fact, there are static targets for all of the -lav* libraries in the Xcode project. So why did the linker go off and use -lavcodec rather than, say, $PERIAN_DIR/build/Universal/libavcodec.a, as instructed?

The workaround

In short: Xcode is screwing things up. Xcode 3.1 has a distinct (some say, unnatural) preference for dynamic libraries. When you add a library to the “Link Binary With Library” listing under Targets, it simply adds a -l<libname> to the link line to pass to ld at link time. When this happens, it will look for lib<libname>.dylib (in this case, libavcodec.dylib, libavformat.dylib, libavutil.dylib, etc in your library path. If it finds one, it will use that one rather than the static library you provided in the Xcode project file. Obviously, this ends up linking the wrong library, perhaps with the wrong architecture or the wrong version. Compilation explodes.

The workarounds are not very clean. There seems to be no good way to instruct Xcode to forget its dynamic library fetish and use the static versions you provide. You could remove the dylibs from the system library path entirely — when the linker fails to find the .dylib version, it will fall back to the .a.

Alternatively, you can explicitly instruct it to use static linking as an environment flag and remove the built-in Xcode linker relationships:

1. Right-click on your Target (in this case, Perian) and Get Info to bring up the Target Info box.
2. Go to the Build tab
3. Select your Build configuration – in this case, probably Deployment
4. Find or locate OTHER_LD_FLAGS, also called “Other Linker Flags”
5. Manually add the paths to the .a files in this field, one at a time. If you don’t know the path, right-click -> Get Info on the desired libraries under Link Binary With Library, and it should show you a field called Full Path on the General tab. That’s the path to use. Should look something like /tmp/perian/build/Universal/libavcodec.a
6. Do the same for all offending libraries – this may include libavcodec, libavformat, libavutil, libmatroska, etc.
7. Remove the libraries you added to Other Linker Flags from your Xcode’s Target -> Target Name -> Link Binary With Library section, to stop Xcode from auto-generating broken linker flags. This is important.

If done correctly, this should no longer find the wrong type of the library. Your compilation of Perian should proceed as normal.

The Moral of the Story

Xcode has some weird idiosyncratic ways of building its projects. It’s useful to poke at things and find out about these behaviors. Its unnatural preference for dynamic libraries over your explicit instructions is disturbing, though.

Don’t attack your users when they pose a problem, and drop the attitude of: ooo, we’re doing this for free, so fsck off, users. In both problems, the exact same attitude surfaced amongst a few of the devs. Arrogance (“we don’t have the responsibility to do anything for you”) and snarky dismissal (‘Well, there’s your problem. We don’t build libavcodec.dylib nor do we place it in /usr/local/lib.’) helps no one. No matter what you thought your build script did, the linker is heading off toward /usr/local/lib, and you better damn well find out why.

It’s obvious that there’s something peculiar going on with the link process, and the user tried to make it clear multiple times (“You missed my point. We build a .a file and we don’t put it in /usr/local/lib.”), but no one seems to pay attention. Following that thread, the user is increasingly frustrated and expresses that, and now the dev’s hammer of arrogance comes down hard ( “You looked like a prick here, to be honest.” — really? not your obtuse-ness or your blindness to a user need?)

When you have a “works-for-me” situation, it’s especially important to communicate effectively with the user. In most cases, you have no idea why it blows up for the user and not for you. Admitting it is fine — could be a very localized issue for that user, or could reveal a subtle systemic problem in your code – either way, it’s useful to have that established. I realize we’re all very busy people, but at the end of the day it’s still your project. Either acknowledge you have no idea what’s going on and solicit help and cooperation with the user who’s having this issue, or man up, put on your detective hat, and fix your own damn problems. The rest of us are busy with our own issues.

The Free Software culture seems to breed this sort of contempt for the user in some devs. Might be a paper in analyzing how corrosive this can be.

UPDATED: I certainly do not mean this last section as a criticism for all Free Software devs. The vast majority of people I’ve talked to in the various communities are really knowledgeable, helpful folks. There are just a few people in some cases, though, who seem to think that if the user can’t make the software work, it’s somehow the user’s fault. It’s a joint problem to be solved together, not “you’re an idiot if you can’t make it work”.

UPDATED 2 (9-29-2008): Perian 1.1.1 has been released. This should fix the original problem with H.264 and .avi files that prompted this post.


tcmemcpy.c:30:missing or invalid immediate expression `0b111' taken as 0
tcmemcpy.c:30:suffix or operands invalid for `and'
tcmemcpy.c:42:missing or invalid immediate expression `0b1000' taken as 0
tcmemcpy.c:42:suffix or operands invalid for `test'
tcmemcpy.c:52:missing or invalid immediate expression `0b1111' taken as 0


The problem here appears to be that the code in tcmemcpy.c under x86 and x86_64 relies on some inline assembly features not supported by all compiler/assemblers. Namely, the use of 0b (binary immediate) operands in assembly instructions is not apparently supported during compilation under Apple gcc and gcc-4.2.

Setting the CCAS flag did not seem to help. A quick fix, then, is to convert all binary immediates in the asm instructions to hex or decimal.

Thankfully, these operands only existed in tcmemcpy, so there’s not too much work. Look for 0bxxxx immediates and convert them to their hex or decimal equivalent. For example, 0b111 is 0×7, 0b1000 is 0×8, 0b1111 is 0xf. Thus, a line like:
and $0b111, %%eax # ... which is the number of bytes to copyn

becomes
and $0x7, %%eax # ... which is the number of bytes to copyn

Once these operands are modified, the compiler should no longer complain about invalid immediate expressions.

In any case, the latest CVS version of transcode should have resolved these problems already. The workaround should only be necessary if you still must compile the release tarball for 1.0.5.

Updated: There is a second potential problem if you were using ffmpeg-SVN and transcode 1.0.5, in which its hard-coded include lookup for avcodec.h always looks for ffmpeg/avcodec.h. The SVN version of ffmpeg has moved these headers to libavcodec/avcodec.h. A path patch for transcode 1.0.5 and ffmpeg is available, which basically boils down to changing the #include headers and the configure.in file to look for the new path.

Updated: A third problem has now cropped up, with the ffmpeg API change that moved AVCodecContext’s bits_per_sample to bits_per_coded_sample. A global search and replace seems to work for now. See this also in my post on compiling Perian.

If you’ve been having problems compiling various Unix packages on OS X 10.5.4, and that your compile run fails mysteriously with something like:

i686-apple-darwin9-gcc: /usr/X11/lib/libpng12.0.26.0.dylib: No such file or directory

One strange yet very likely explanation: your libtool archive file /usr/X11/lib/libpng12.la is lying about the location of the shared library for libpng12 — namely, that a file called /usr/X11/lib/libpng12.0.26.0.dylib exists and should be used for linking against libpng12. However, if you actually look in /usr/X11/lib, no such file exists – perhaps you might have libpng.0.24.0.dylib, but not .26. Therefore, packages that make use of this incorrect libtool archive metadata are suitably confused, causing the compiler to bail out when trying to link against this non-existent file.

Since libtool archive (.la) files are text-based, you can open it up in emacs. The quick and dirty fix to this is to simply change the offending library_names and the current and age properties to the correct numbers. In my case, the libpng sitting in /usr/X11/lib was .24, and so I string-replaced all the values in those three properties from 26 to 24. The compilation then proceeded normally.

The long term solution, of course, is to track down what put a wrong .la file there in the first place. I suspect Xcode 3.1 and Mac OS X SDK 10.5, which shipped with the latest iPhone SDK.

UPDATED 9/8/2008:
In the comments section, I’ve been informed that the X11SDK package in Xcode 3.1 in the culprit. Thanks Anonymous!