Python multiprocessing code crashes on OS X under IPython

While working on a project on OS X 10.10.4 Yosemite, some innocuous Python 2.7.6 code using the multiprocessing module (via the concurrent.futures module) was crashing when run from the IPython interpreter, but works just fine when executed via shell directly.


Some nice simple test code. It pulls some data from two Web servers, via the requests module, concurrently using a multi-process worker pool.

from multiprocessing import Pool
import concurrent.futures
import requests

TEST_URI = ["", ""]

def http_get(uri):
    return requests.get(uri)

def test():
    p = Pool(5)
    print(, TEST_URI))

def futures_test():
    with concurrent.futures.ProcessPoolExecutor() as executor:
        print [response for response in, TEST_URI)]

if __name__ == '__main__':

When run as $ python, everything works beautifully. When test() or futures_test() was called from an IPython shell however, a segmentation fault occurs:

Exception Type:        EXC_BAD_ACCESS (SIGSEGV)
Exception Codes:       KERN_INVALID_ADDRESS at 0x0000000000000110


Application Specific Information:
crashed on child side of fork pre-exec

Thread 0 Crashed:: Dispatch queue:
0   libdispatch.dylib             	0x00007fff9772c16f _dispatch_async_f_slow + 395
1      	0x00007fff92e91541 _CFPrefsWithDaemonConnection + 305
2      	0x00007fff92e60ac6 __80-[CFPrefsSearchListSource generationCountFromListOfSources:count:allowFetching:]_block_invoke + 150
3      	0x00007fff92e609d2 -[CFPrefsSearchListSource generationCountFromListOfSources:count:allowFetching:] + 258
4      	0x00007fff92d1cea5 -[CFPrefsSearchListSource alreadylocked_copyDictionary] + 133
5      	0x00007fff92d17dba -[CFPrefsSearchListSource alreadylocked_copyValueForKey:] + 42
6      	0x00007fff92e9211c ___CFPreferencesCopyAppValueWithContainer_block_invoke + 60
7      	0x00007fff92e5f979 +[CFPrefsSearchListSource withSearchListForIdentifier:container:perform:] + 729
8      	0x00007fff92e92097 _CFPreferencesCopyAppValueWithContainer + 183
9 	0x00007fff96be8db4 SCDynamicStoreCopyProxiesWithOptions + 153
10                   	0x000000010e07f915 0x10e07f000 + 2325
11  org.python.python             	0x000000010d89a9ed PyEval_EvalFrameEx + 14935
12  org.python.python             	0x000000010d89d60e 0x10d813000 + 566798

The same code works just fine under Ubuntu Linux, from IPython or otherwise.


It turns out that there is some kind of edge case bug going on between IPython, multiprocessing, and OS X. Namely, a segfault seems to trigger from within OS X’s Grand Central Dispatch architecture, when multiprocessing is forking a new process to access the network, under IPython.

Note these lines in the crash log:

0   libdispatch.dylib             	0x00007fff9772c16f _dispatch_async_f_slow + 395
10                   	0x000000010e07f915 0x10e07f000 + 2325

Something in (a part of Python’s lib-dynload) is not happy with libdispatch. calls out to OS X’s SystemConfiguration framework for network proxy information. The quickest workaround for this bug, then, is to disable this proxy check. IPython seems to respect the customary no_proxy rule to skip proxy checks (at least IPython 4 does), so:

$ env no_proxy='*' ipython

and then

$ env no_proxy='*' ipython
Python 2.7.6 (default, Sep  9 2014, 15:04:36) 
Type "copyright", "credits" or "license" for more information.

IPython 4.0.0-b1 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: import mp_test

In [2]: mp_test.test()
[<Response [200]>, <Response [200]>]

works just fine. No segfault.

Of course if you actually need this proxy functionality, you would be out of luck.

Setting up a Gmail POP3 account for on 10.10.3 beta

The last beta of OS X has apparently removed the option to create POP3-type email accounts for Gmail in When you type in a Gmail account, IMAP is setup automatically. Presumably because OAuth doesn’t work all that well with it.

There (currently) is a workaround.

  • Turn off/unplug your Internet connection. This will force Mail to not verify the settings that you are giving it.
  • Open -> File Menu -> Preferences -> Accounts tab
  • Hit the + button to Add Other Mail Account
  • In the email address field in the following box, enter something that IS NOT Enter some random email.
  • will now inform you that manual setup is required. That’s exactly what we wanted in the first place, thank you very much. In the next screen, you will get to pick IMAP vs POP. Pick the POP tab and enter the correct Gmail POP server information and Gmail outbound SMTP server in the following screens, for your account. Make sure to use SSL and port 995 if appropriate
  • Use the application-specific password in the password field if you have two-factor auth setup.
  • Once setup is complete and you are thrown back into the preference pane, change the “Email Address” field back to the correct email address
  • Under “Advanced”, untick “automatically detect and maintain account settings”
  • Quit Turn Internet connection back on
  • When you re-open, the POP account should be operational

Hooray for we remaining holdouts, still using obsolete and dubiously secure technology. Because when I want mail on my local machine, I mean I want mail on my local machine, The Cloud(tm) be damned.

Note they might still change this depending on feedback received. I would advise giving such feedback if you care at all. stuck at fetching mail on OS X 10.10.3

If is stuck at “Fetching Mail….” with Gmail POP accounts after upgrading to OS X 10.10.3’s beta, be aware that Apple added OAuth support to Which is nice, except in the current beta, they don’t warn you to go set up OAuth permissions for every account. will choke on fetching mail, without letting you know the actual problem. This is possibly POP3-specific, as I didn’t notice IMAP having this issue.

In the connection log:

[kCFStreamSocketSecurityLevelTLSv1_0] -- -- port:995 -- socket:0xxxxxx -- thread:0xxxxx
-ERR invalid SASL argument ......

The solution here is to simply set up Google OAuth via System Preferences -> Internet Accounts. It should automatically prompt you for OAuth permission to your Google Account when you click on the affected account in the prefpane. Once that is done, will fetch Gmail via POP3 again.

I’ve also unticked “automatically detect and maintain account settings” under Advanced, in’s accounts panel, though this might just be superstition rather than actual prevention of issues. I’m just old school and uncomfortable with the thought of “automagic” happening in my mail server settings.

If you’ve already given up and swapped to IMAP, but would like to come back to the Dark Side of email retrieval technology, see also:
How to setup a Gmail POP3 account from scratch on 10.10.3 beta

Selfsolved reference:
#127: stuck at fetching mail on new OS X

I really need to rebuild Selfsolved. Someday, when I’m not trying to write.

restoring Safari preferences from backup files in OS X Mavericks

Recently I had the misfortune of having to restore some Safari settings from backup, on OS X 10.9 Mavericks. I have done this many times before on older OS X versions, without incident — simply pull the various preference files such as from backup and replace the damaged/unwanted ones. Takes all of 2 minutes, and years ago, I had already wrote a shell script to do exactly that.

It turns out that after Mavericks, Safari is incredibly resistant to conventional methods of preference file backup and restoration. Considering that preferences in OS X have always been stored in XML-based preference lists, you would think (as in previous operating systems from 10.0 to 10.7) taking the relevant preference files and replacing the unwanted new ones in ~/Library/Preferences would be enough. But no, an incredible amount of effort is now required for a simple task:

  1. unhiding the Library directory, because clearly we’re all Windows babies who cannot be trusted to see where application preferences are stored
  2. replacing the actual preference files, scattered around the system in ~/Library/Safari, ~/Library/Preferences, ~/Library/Caches, and hopefully remember to have turned off iCloud, or otherwise see changes being clobbered by iCloud sync (or, worse yet, having experimental changes reflected all across a network of Mac and iDevices)
  3. resetting the preference cache daemon, cfprefsd so that preference changes can be reflected in a running system. This where a lot of people get stuck in general, judging by Google results; when they replace preference files and find that their changes aren’t being reflected, it leads them into a wild goose chase for “hidden preference files” for Safari, when the answer lies in a simply yet utterly non-obvious background daemon.
  4. restoring the list of installed extensions — which, incredibly, is NOT stored in the deceptively named, but in the login.keychain.


Had some issues where certain websites were behaving differently under private browsing mode than normal browsing mode. I deduced there was some kind of corrupted stored state, whether it was a cookie or localstorage issue. Had the brilliant idea of setting Safari preferences aside, thus resetting Safari to factory state, and then divide-and-conquer by restoring parts of the settings until the problem recurs. I’ve done this many times before.

First I turned off iCloud sync, having been bitten by sync propagation of experimental changes in the past. This is pretty important if you don’t want to blow up Safari bookmarks (at the very least) across all Apple-manufactured, iCloud-compatible devices. I then removed ~/Library/, ~/Library/, ~/Library/Safari, and ~/Library/Caches/Safari, ~/Library/Cookies. After resetting to confirm some issues have disappeared, I moved some files from backup to original locations. Imagine my surprise when nothing became restored, and all my Safari extensions (installed from the extension store or custom-built by me) disappeared.


Increasingly desperate, I started to trace filesystem accesses using fs_usage. It showed nothing out of the ordinary. 30 mins of reviewing useless forum posts later, I pieced together a multi-stage solution. It turns out there were two separate obstacles.

Preference caching

Presumably to save energy, OS X Mavericks caches application preferences (in RAM?) using a daemon called cfprefsd. Instead of applications pulling their preferences from XML files on disk at launch, it requests this from the daemon instead. The defaults command has been modified to operate with this daemon, so if you had been working with preferences from the command line (as I have been), the changes have been transparent.

However if the preference files are changed or edited directly, this change is not propagated to the preference cache daemon. When the app is opened again, the cached version takes precedence, and is re-written out to disk, clobbering the restored versions.

This does not mean there are hidden Safari preferences somewhere that you haven’t found, though you might think this at first. When Safari is reset manually from the filesystem, or if the plist files are edited, cfprefsd must be reset as well.

There exists a cfprefsd daemon for every logged in user, running under that user’s privileges, as well as a root-owned one. Safari preferences are stored under the user domain, so the user-specific daemon is the one that needs a reset when files change. Can also quit the process from Activity Viewer, or killall cfprefsd. A login-logout cycle would also reset the user-specific daemon.

Extension list caching

Having done this reset with the backup files in place, most preferences will be restored on next launch, *except* the list of extensions you had installed previously. That will remain empty. Even though all the extensions and their settings have been restored to ~/Library/Safari.

For a long time I traced ~/Library/, and wondered why it wasn’t being read.

An Apple discussion forum post (shockingly enough) gave a vital clue. There exists an “application password” in the login keychain titled “Safari Extensions List”. Whether it is merely a cryptographic key, or the actual list of extensions, is unknown, but that is the critical preference to restore extensions. Having reset extensions by moving them away, this preference is apparently emptied out. The entire login keychain, being an encrypted document, has to be restored to a corresponding previous version to restore access to previous extensions. Without this, all extensions would have to be reinstalled manually (and get a new copy of the extension file stored into ~/Library/Safari/Extensions, instead of the previous version being reused).


Given recent focus on energy consumption, I can understand preference caching. However, it’s not that hard to track filesystem changes (the Time Machine/Spotlight APIs explicitly do this!) and reload appropriate preferences when they are changed on disk. It would show respect for power users and developers who might need to interact with the preferences system in a more convenient way.

But stuffing extension lists in an obscure corner of a password keychain? What sense does that make? Are my list of extensions (not actual extension data or settings, mind you — those are in plaintext on the disk for anyone to copy and look at) such privileged information that it has to reside alongside my login password? Why can’t you just read the list of extensions, oh I don’t know, from the list of extension files installed into the Extensions directory? Wouldn’t that be a lot more reasonable?

A Twitter timeline to Atom feed proxy

So Twitter is retiring version 1.0 of its API in March 2013. In its ongoing quest to become (more) evil, Twitter has decided that open syndication standards like Atom are no longer worth supporting. This, in addition to the gratuitously byzantine OAuth system (even for 2-legged auth between my own client and Twitter itself), makes consuming Twitter content anywhere else except on Twitter (and its official apps) an increasingly annoying task. An intended effect, perhaps.

I’m one of the few holdouts who believe current Web standards work just fine. I consume a lot of content from the Web in my native RSS client, as part of my ordinary daily workflow, without using 50 different apps and dozens of notifications. This includes my Twitter home timeline (where I follow just under 70 people/orgs of interest), which under API v1.0 was provided as a simple, standard RSS feed, like other open streams of data on the Web.

I’d been hacking on a Python-based Twitter JSON API to Atom feed proxy for some time, but became increasingly disillusioned with the stupidity of the API — and that it takes two libraries and tons of code to even get the OAuth dance started.

So I thought, in this entire WWW, there must be someone else as annoyed by Twitter’s obstinacy as I am. Sure enough, Russell Beattie developed a single-file PHP script that accesses the authenticated Twitter stream and outputs an Atom feed.

I abandoned my Python code, forked that codebase, and made some minor modifications to suit my personal needs. This patched version is available here:

All you need to do to use this script is to create your own Twitter app over at (I called mine “TimelineProxy”), create an OAuth token, and fill in the blanks in the script. Since the new API also has a 100,000 user per application limit, it’s probably best for every user to have his own proxy app with its own token, instead of relying on a central one.

There are some minor differences between my version and the upstream original. Basically this version uses full php tags instead of the less-well-supported short tags, and replaces shortened URLs with their full original URLs. I also take advantage of the html content type in the Atom entry to allow links in the entry text, so in most feed readers any links are “clickable”. Finally, this version returns proper HTTP error codes instead of 200 OK in case of Twitter API errors (like when you hit the rather draconian rate limits on each OAuth token).

However, it preserves the simplicity of the original, which is that you can drop this in the web directory of any PHP-enabled web server (no need for root access or installation of any libraries) and enable your own timeline proxy.

I want to eventually modify this script so that a single server can host any arbitrary user’s timeline as an Atom proxy, provided they give the proper auth tokens. This means having to deal with the OAuth dance at some point. Ugh.

In any case, this disturbing “enclosure movement” of the open Web — taking previously free streams of information and fencing them into walled gardens of content — is a trend that should be opposed whenever possible. My thanks to Mr. Beattie for making the original script.

Outlook 2011 for Mac still adding arbitrary line breaks into plaintext emails

Outlook 2011 on Mac OS X, v14.1.3, for whatever reason, still does not properly support “format=flowed” content-type or “quoted-printable” extensions for plaintext emails. This causes plaintext emails to be sent as mangled messes, full of arbitrarily inserted linebreaks. This appears to be a regression from Entourage, as far as I recall, which never handled plaintext quite this badly, and this is also despite Microsoft’s promises to have “implemented format=flowed”.

This is the last straw. I’ve been a loyal MS Entourage / MS Outlook user since the days of Outlook Express for Mac and Office 2001. But at this point, this software has actively impeded my communications with my friends and colleagues. We’re done.

The Problem

Here’s a really simple illustration of the problem, from the receiver’s end:

See how the URL, which was composed as one plaintext line, gets split up into two lines?

Here is another example, purely from the editor UI (and not even being sent yet). I start with a perfectly good reply saved as a draft:

I make a small wording change and resave:

See that third line? Thanks to the hard line breaks inserted by Outlook (even at composition stage), the line wrap has been mangled. This draft has to be re-wrapped manually, by the tedious process of deleting the newline-based hard line breaks from every line following in the paragraph. That was a short paragraph. Imagine doing that in a long paragraph, from the first line.

To add insult to injury, there is not even a “re-wrap” functionality in the editor, to at least solve this user-interface level problem (as opposed to the protocol level problem). Obviously no one at Microsoft sends plaintext emails anymore.

The Issue

Back when email was first devised, servers didn’t have a lot of memory, and people had pretty tiny terminals with fixed line widths and not a whole lot of processing power to deal with it. The Internet standards for email messages, RFC2822 Section 2.1.1, defines recommendations for email body text transferred over SMTP:

There are two limits that this standard places on the number of
characters in a line. Each line of characters MUST be no more than
998 characters, and SHOULD be no more than 78 characters, excluding
the CRLF.

The 998 character limit is due to limitations in many implementations
which send, receive, or store Internet Message Format messages that
simply cannot handle more than 998 characters on a line. Receiving
implementations would do well to handle an arbitrarily large number
of characters in a line for robustness sake…

The more conservative 78 character recommendation is to accommodate
the many implementations of user interfaces that display these
messages which may truncate, or disastrously wrap, the display of
more than 78 characters per line…

…it is encumbant upon implementations which display messages
to handle an arbitrarily large number of characters in a line
(certainly at least up to the 998 character limit) for the sake of

Basically, the SMTP server can count on messages that come in 80 characters per line (and always less than 1000 characters per line), and email clients can trust that they only have to render up to the 78th column of text. This limitation is hardly useful in the modern age, but persists since it’s part of the standard. And it’s a fine, conservative design model. But now we write some pretty long lines without linebreaking ourselves, so something magical has to happen in the email client itself, like Outlook 2011.

The naive solution, of course, is to slap arbitrary line breaks into the user’s email message at every 78 characters, which is what ye olde email clients (looking at you, pine — how did I ever put up with you…) from yesteryears did (and Outlook 2011 still does). It’s a matter of personal preference whether this is a reasonable solution. Proponents argue that the email will “always look the same” on all devices, including those limited to 78 chars per line.

I (and many others), on the other hand, think the spirit of the RFC is to allow the actual handling client to decide where to break lines. With the exception of source code, it is almost always better for the email client to use the full width of their display, however many characters that might be. Even in the case of source code, it should also not be mangled by the insertion of arbitrary line breaks in them — what if newlines are meaningful in this language, and the author used more than 78 characters per line? The example with the URI is illustrative of this problem — the URI got an arbitrary newline in the middle, destroying its meaning. Users who copy-paste the two lines will end up getting a 404, due to that stupid inserted newline in the middle of it. This should not be allowed to happen.

Because this naive solution was not perfect, an extension was proposed as RFC 2646. This format of email is characterized by the content-type:

Content-type: text/plain; charset=US-ASCII; format=flowed

In format=flowed emails, the sending and receiving email clients are allowed to reflow the text based on user linebreaks. It follows some simple reflowing rules, but in short it will preserve user-inserted hard line breaks while adjusting the rest of the message for the proper line length while the message is “on the wire”, and recombining the lines on receipt and display. Modern email clients like Thunderbird, designed for user comfort and the generous system limitations of the year 2011, implement this standard.

Guess what format Outlook 2011 sends?

Content-type: text/plain; charset="US-ASCII"

Not even an option to change that behavior. It does not appear that Outlook 2011 deals with any of this. It just inserts some line breaks and calls it a day.

An alternative, implemented by Apple’s, is to send messages with the Content-Transfer-Encoding header set to “quoted-printable”, as per RFC 2045. In this model, soft line breaks are sent explicitly with the character “=” representing it, breaking at the usual 70-odd character column. On the receiving end, the client processes this character as a no-op and concats the line back together for display.

Outlook doesn’t do that either. It just wants to mangle your emails.


The world moved on and adopted HTML emails, which doesn’t have this newline problem. For those of us who do think HTML emails are an atrocity to be used sparingly, if at all, the idiosyncrasies of plaintext email have to be addressed. Outlook 2011 appears to do even worse than Entourage 2008 at this problem, by not dealing with it at all. And apparently getting a bunch of Microsoft “MVPs” on their forums to cloud the issue with promises of support and unrelated commentary.

Given the sad state of email clients on the Mac, I believe Thunderbird is now my only option for sane plaintext messaging.

Dragon Age 2: modding the Edge of Night axe into a longsword

Having a bit of fun with Dragon Age 2 in my spare time. The selection of longswords in Dragon Age 2 leaves much to be desired. Call me old-fashioned, but when I play fantasy RPGs, my warriors are always sword-and-shield (munchkin min-maxing be damned), because…that’s how heroic fantasy warriors are supposed to be. As TvTropes succinctly put it, Heroes Prefer Swords. I actually shelled out money for the Warrior Item Pack DLC hoping to find a better longsword. Unfortunately, the only decent one-handed weapon in the pack is an axe. An axe?! What are we, woodcutting barbarians?

So I went about looking for a way to switch this otherwise beautiful weapon, The Edge of Night, into a longsword. The similarly named Edge of Song and Glory is the ideal target for the transformation.

There is not much modding possible for DA2 — the game is very much locked down to drive DLC sales. Nevertheless, the base game itself is at least unencrypted; however, there is no official mod editor support. The DLCs, however, are encrypted — you cannot even view the data contained within these files. The following modding method requires that you decrypt the DLC’s designeritems.erf file first, then extracting the Edge of Night’s item file for editing. This only works for PC versions of DA2.

Note that there is already a mod floating around for this. There is a good thing and a bad thing about this mod. The good thing is that it uses a less questionable method that does not need to decrypt the ERF file. On the other hand, because of the limitations of such a method, that mod doesn’t change the item icon or the item description. It will still look like an axe and call itself an axe in the character’s inventory screen — only the actual gameplay model is changed to a longsword. Some people would be satisfied with that; I wasn’t.

Decrypting designeritems.erf

DLC items are in found in ${DRAGON AGE 2 INSTALLDIR}/addins. In this case, it is the DA2_MTX_WAR addin. The ERF file is
${DRAGON AGE 2 INSTALLDIR}/addins/da2_mtx_war/core/data/designeritems.erf, and the UTI file is [dd9ea6bb335cfa89].uti.

Make a backup copy of the da2_mtx_war directory.

To decrypt the ERF file, you need the decrypt key and a decrypter. There is a tool called “slurpda2.exe”, usually distributed with the deda2drm decrypter package. I will not explain how to obtain this package, but it’s pretty easily found. If you legitimately own this game, slurpda2 will give you a list of all the decrypt keys, one for each DLC package. Use the right key (have to do trial-and-error here, as it doesn’t tell you which key belongs to which package) and deda2drm by cd’ing to the deda2drm directory and executing from the command prompt:

deda2drm.exe DA2_MTX_WAR [key]

The output will tell you whether this succeeded. If so, this will decrypt the ERF.

Extracting the UTI

A tool like PyGFF will open the decrypted ERF file. Look for the dd9ea6bb335cfa89 hash key and extract that UTI. For reference, the UTI should contain an item with the tag war100im_warrior_1h_sword. Save the UTI file as war100im_warrior_1h_sword.uti (though technically you can name the filename anything you want, as long as it ends in uti). As you can see by the name, even the original designers thought it should be a sword and not an axe :p

Editing the UTI

Using a tool like TlkEdit, open the UTI file. Change the ModelVariation field to 123 (which is the Edge of Song and Glory, as listed in 2da.rim’s item_variations.gda file). I believe 132 is Glandivalis / Vigilance, if you prefer that look. Change the Icon field to “”. Save the UTI file.

Here you can choose to restore the original encrypted copy of da2_mtx_war or not. If so, this reverses the decryption so it doesn’t confuse DA2 when you open it again. Alternatively, you can edit the manifest.xml file in the decrypted copy of da2_mtx_war to tell it to skip the authorization requirement by setting RequiresAuthorization=0. I left mine unencrypted, but I think this gets into legal gray territory, and the UTI mod should work even with the original encrypted version.

Drop the final modded uti file into packages/core/override in either the DA2 install directory or the user directory in Documents/Bioware (create the override directory if needed).


Edge of Night sword The Edge of Night sword in action

Isn’t that pretty? Wielding a sword, as a hero should. One of these days I’ll mod the TLK file to change the occurrences of “axe” to sword in its Codex file, and this change would be perfect.

I have a copy of the modded UTI file, but I think distributing it might violate the EULA. Oh well. If you can’t figure out the instructions, leave a comment here (with your email address) and I’ll try to help.

A wishlist for a native Mac email client

As the tech-savvy sort, I often do some things that the typical consumer might not care about. However, it is strange that I simply cannot find a native Mac email client on the market that fulfills what I consider to be very basic features for decent email management. While everyone is gushing over the latest social network to be jammed into an email client, I just want my email to work in a sane way.

The features I’m looking for:

  • POP3/IMAP + SSL support
  • Full-text search, or indexable by OS X’s Spotlight
  • Multiple account support
  • Archival — if I delete an email account, it should not wipe out all emails from local storage that belonged to this account, especially if said account was POP3
  • Plaintext composition — I’m taking the side of format=flowed in plaintext composition. Manually formatting hard line breaks at char 78 is an insane holdover from a bygone age when dumb ASCII terminals were still the primary user interface. At least allow this as an option, or support the quoted-printable content type and soft line-breaks during composition.
  • Filtering rules — Basic filters that let me decide where to put messages based on mail headers and subject is enough
  • mbox import/export — I need to be able to import mail from my previous client. Similarly, in the scenario that this app is no longer supported, I need the option of moving to another client.
  • Bonus: auto-bcc to an arbitrary email address — I’ve never solved my problem with archiving sent mail. All you really need to do is auto-populate the BCC field on any “New Mail” composition window with this address. That’s all I really need to be happy here.

Amazingly enough, there is not a single Mac mail client that fulfills all of these basic conditions. Especially egregious problems for the top three mail clients:

  • Microsoft Outlook Mac 2011 — mangles all outgoing plaintext emails by inserting hard line breaks. No support for flowed plaintext, despite promises to the contrary, and no support for quoted-printable content-type and soft line breaks. No provision for auto-bcc.
  • Apple Mail (Snow Leopard) — deleting an old email account deletes ALL mail belonging to this account, even for downloaded POP3 mail. WTF. Can only auto-bcc “myself”, which is a fixed email address corresponding to the sending account; if you allow autobcc, why not let the user pick the email to autobcc to?
  • Mozilla Thunderbird 8 — deleting an account deletes all downloaded mail belonging to this account, unless messages are stored to the “local folder” rather than its inbox abstractions.

For fear of accidental data loss, I’ve stuck to the Microsoft offering, despite its incredible inability to keep plaintext mail intact. The newer social clients are all Thunderbird-based and leave little else worth examining — they seem far more interested in integrating ever more social network APIs than to support more basic email features. The rest of my friends are all sworn Gmail webmail users and think I’m a strange luddite for even considering native clients. I’ll leave the webmail vs native client debate for fear of going on a page-long rant; suffice to say that the user experience between the two options is not comparable at all.

It almost makes me want to start writing a new email client for myself, or hire someone to start some skeleton code at least, because obviously no one else is going to scratch this itch of mine. That kind of spare time and money, however, would probably never come unless I manage to actually sell a company or two (rather than just founding unsuccessful ones).

A Brief Guide to Golems of Amgarrak with a New Character

So I’ve been playing Dragon Age: Origins in the few spare hours I get every week. Having never finished Baldur’s Gate 2, this is redemption for my neglected gamer soul, I think. In general, it’s pretty easy. However, several of the most annoying battles in all of DAO are from the Golems of Amgarrak DLC. So far, all of the FAQs and guides I’ve seen (including videos on YouTube) are about absurdly min-maxed level 35 characters imported from Origins and Awakening, with elite gear, hundreds of poultices and potions, and the most overpowered skillsets.

This note is not about trouncing the Harvester with level 35 ubergeared supermen. Any halfwit can do that — just go get the gear, follow any YouTube min-maxing guide for DAO, and start whacking away. (My personal favorite: 100% dodge build: nothing can land a hit on you with physical attacks. Eat that, Harvester. But I digress). You can do this DLC easily with any properly geared character. This note is about how to complete Golems of Amgarrak on Nightmare difficulty with a newly created, level 20 mage, with stock gear and sneaky tactics. This is not to claim it’s the best way to accomplish the feat — just the way I managed it. If you have better ideas about how to use a newly created level 20 character to win, contribute them in the comments.

This note only applies to DA:O as played on the PC. On some console-based versions, the pet bronto cannot be controlled, making the final battle even more annoying than usual.


If you don’t know yet, completing Golems of Amgarrak on Nightmare (or Hard, actually) difficulty awards all of the other characters in other campaigns (even existing campaigns’ characters) some really nifty items — but only nifty for lower level characters. This includes the amulet The High Regard of House Dace, which gives +6 cunning so Daveth can open up all those annoying chests in Ostagar, and The Reaper’s Cudgel, which is worth well over 300 sovereigns. The early cash allows you to buy the Blood Dragon Armor, the Spellward, and the Arcane Knowledge and Physical skill tomes from Bodahn immediately after Lothering before his store restocks; you can then buy another copy of these items after the restock.

With my time constraints, I will most likely play DAO only once. So I’d like to have these items in my Origins campaign, without having to complete the game first to get an absurdly well geared character to do Amgarrak. I thought I can just start a new Amgarrak campaign with a new character, and finish it to get my gear. Boy, was I wrong. People call this the hardest DLC for Origins, and for good reason. I played a warrior and simply could not win an initial fight against 4 elite golems, and had to delete, restart the campaign, and rethink my strategy.

General Setup

Despite claims, the module is not impossible, even with a new level 20 character. It is simply very poorly balanced for new characters, and forces you to proceed carefully and methodically. This is Dragon Age, not Diablo — as a new character, you cannot just rush into every battle, aggro’ing all the mobs at once. Separate mobs from each other, killing them one at a time, withdraw backwards to avoid being surrounded, pause frequently to issue new orders as the situation requires and don’t rely on the Combat Tactics scripts, and you should have very little difficulty dealing with most creatures in the module.

To win, I strongly suggest playing a mage. In fact, the best character for completing the DLC with is probably the mage, regardless of level. The main reason: there are no merchants, not nearly enough healing poultices, and most unfortunate of all, no other playable mage in the entire module. The objective is to preserve all potions and consumables until the final battle with the Harvester, which means the party will need to rely on mage healing spells.

The party at the start consists of a duel-wielding rogue, a bronto (with a charge attack and a Taunt), and the PC. The party at the end adds a sword-and-shield warrior (without Taunt!) and a Runic Golem with some typical golem abilities, plus a weak Group Heal, a Cleansing Aura, and Chain Lightning/Tempest spells. All of these characters are poorly configured with mediocre equipment, and the party only has one respec tome (Memoirs of the Grey Wardens).

Save the tome for the rogue. He’s going to be the main DPS character against the Harvester, but only if he’s an archer and standing far away from the action.

Attributes and Specializations

If starting this without having beaten Awakening, then the mage is restricted to three base specializations: Spirit Healer, Arcane Warrior, and Shapeshifter. Since Shapeshifter is worthless, the two specializations are decided already. If you have started Awakening, you might consider the Battle Mage instead of Arcane Warrior for additional utility, but Spirit Healer is an absolute must as a new character.

As a mage, 60+ magic is necessary for spells to have decent effect. The rest should be spent in willpower to pad out the mana supply. Usually, when you build a Mage for Origins or Awakening, you don’t want to put any points in Willpower. However, remember here, there are very few, very precious lyrium potions and no way to buy more; there are no special attribute bonuses that boost the mana pool for free; and the PC’s starting equipment is subpar and does not provide a large mana boost (and again, there is no way to buy better equipment). If you don’t take additional Willpower, you will not be able to cast during the long endgame battle against the Harvester, and that would be very unfortunate.

The mage should have maxed Combat Training to ensure that spells aren’t interrupted by ordinary attacks. Survival is a plus for the nature resistances, and the first level Vitality and Mana skills from Awakening. This is a pure combat module, with no ingredients for crafting and a grand total of two conversations — no need for Coercion or any of the crafting skills.


One might be tempted to build an armored arcane warrior, since this is a popular “I win” character build.

This would be … unfortunate. Normally, the arcane warrior is very overpowered: all the gear of a tank and the firepower of a mage. In this module, with a new character, you have neither of these advantages. The small module does not offer a merchant or any sort of decent armor, and a stock character does not have enough willpower to have a deep mana pool after putting up the arcane warrior defensive abilities. Wearing a robe, swinging a Warden’s Companion longsword with a mediocre shield, while not being able to cast healing spells — this is a recipe for disaster. Further, Brogan, the warrior in the party, is configured as a tank — a poor tank, but he can’t be respec’ed as a DPS due to lack of tomes, while the PC can be a DPS. Therefore, playing a support + nuker mage is probably a better choice. Some of the advice that follows will assume a mage with an assortment of crowd control and attack spells.

As for specific talents, since it’s a stock character, there aren’t that many points available. Here is my advice:

  • The Healing tree up to Regeneration is a must. As Spirit Healer, also take at minimum Group Healing, because both the golem enemies and the Harvester at the end have area-of-effect attack abilities that can damage an entire party. Remember that this character will be the only healer until the Runic Golem is fully upgraded, and even so, the Golem is a terrible healer. These healing spells are basically critical to finishing the module, and is why stock level 20 Warriors or Rogues have a hard time surviving.
  • Make sure to get Force Field, for several possible reasons. Crushing Prison is nice, but only if there are points for it.
  • It would be wise to take Mana Clash, at the top of the Mana Drain tree. In several battles (unfortunately, not the end battle), the party will face several spellcasters along with hard-hitting warriors. Mana Clash will immediately remove the casters from the scene, which trivializes the encounter.
  • For me, I took Tempest and Blizzard (from the Lightning and Cold trees), since I already have Spell Might from the Mana Drain tree. This sets up the Storm of the Century combination that was useful in one instance, but this is optional. I strongly recommend Cone of Cold in any case as it is an absurdly overpowered spell with a freezing effect that applies even to elites and bosses.
  • If taking Storm of the Century, or not taking Mana Clash, then Glyph of Neutralization is desirable. As for the other glyphs, since the main enemies are almost always elite or boss level golems and undead, they are often resisted — except the Paralysis Explosion combination, if set up correctly.
  • I did not use any other spells. If you have a favorite set, use the remaining points to take them. Remember that practically all enemies worth using mana on are Elite or above, so some spells that have minimum effect on elites are not worth taking.

Remember that only one party member will get a chance to respec, and the rogue should use it to reconfigure as a survival-minded DEX-based archer. This is because this underpowered party simply cannot stand toe-to-toe against any of the major enemies, and will rely on kiting (and not even simple kiting either, given the setup of the final battle) and off-tanking to survive many fights.

Jerrik, the Archer

Jerrik will be the main non-magical DPS. Unfortunately, he is setup as a frontline fighter. Because the tank is terrible, and can’t even manage threat effectively (no Taunt!), plus both enemy Golems and the Harvester have area-of-effect attacks to damage everyone in their immediate vicinity, this is a bad configuration. He must respec into a DEX-based DPS archer (enough STR to wear his armor, then all DEX), with Master Archer or Scattershot, Arrow of Slaying, and Feign Death (for the many times when the tank loses threat and the enemy rushes the archer). Optionally, he could take a few cunning and take Stealth to lose threat, though the final battle involves an elite boss and hordes of minions.

Major Battles

On Nightmare difficulty, in general, proceed slowly and cautiously, withdraw backwards into chokepoints when needed, and there should be minimal pain. In particular, when faced with multiple spellcasters, hit Mana Clash to remove most (if not all), and then rush them. Play this as a normal RPG and use the pause key frequently to strategize and adjust tactics. Try not to use any consumables until the final fight. For a general walkthrough, follow Gamebanshee’s Amgarrak guide.

There are two battles that takes some strategy to pull off, however. Some people are stumped here and can’t proceed further. They’re not hard, but you can’t use the normal tank-and-spank tactic with a level 20 Warden and a crappy party.

The ancient hall of golems

As soon as the party enters Amgarrak, there are three directions to go, two of which are sealed off at the start. In the third direction, there is a room with four stationary golems. Two of them are called Watchers, the other two are elites called Sentinels. When the party proceeds far enough into the room, these four will activate.

This is one of the toughest fights because the party does not have its warrior tank yet. Further, these four golems will open the battle by hurling rocks — which deals heavy area-of-effect damage and knocks down any character beside the Runic Golem. They will then follow by using Slam on the incapacitated character, and then Hurl Rock again, basically going into a stunlock until the character is dead.

Careful pulling is impossible, because all four activate at the same time. However, there is one good way to deal with these golems: The Hallway of Doom.

There is a chokepoint hallway going into the hall of golems. The party could fight the golems one-on-one at the doorway, but they’ll jut use Slam to knock the blocker out of the way and rush into the room. Then again, they won’t be able to do that …if the blocker is force-fielded.

Have the party hold position. Use the Runic Golem to trigger the four enemy golems, and then run back to the door. At the doorway, use the Runic Golem to block the door and force field the golem before he gets Slammed. Have the ranged DPS open fire. Force Field your own golem, because the enemy golems are elite and will have shorter Force Field durations.

For even better effect, throw down a Blizzard to slow the enemy as the Runic Golem approaches. Then, trigger Storm of the Century using Tempest, and then Force Field your Runic Golem before it gets seriously damaged. Now the enemy will stand in the massive electrical storm, be drained of stamina by the lightning storm’s side effect (so they can’t Hurl Rock or Slam anymore), and run around uselessly. When the Force Field expires, withdraw the Runic Golem and slaughter the heavily injured enemies (or heal the Golem until you’re able to Force Field again — the enemy should no longer have enough stamina for another Slam).

Make sure the mage is standing off to the side as the Runic Golem runs toward the party, because the enemy could see the mage and interrupt the casting with a hurled rock. If you cast the Force Field too late, the enemy will Slam the blocker out of the way, which will probably cause a party wipe.

I really wish I had two mages in this module, which would be able to keep this little trick going forever. Alas, that is not to be.

NOTE: If you didn’t take Storm of the Century Force Field, or if you can’t get the blocking trick to work, there is another way. You need to have respec’ed Jerrik with at least Combat Stealth. Have the party hold position far away from the action, have Jerrik walk into the hallway and trigger the golems. When the golems activate, Stealth immediately and walk away. Using Jerrik, carefully pull the now activated golems, one or two at a time, and kill them individually as they follow you down the hallway. This is arguably easier, if less impressive, than the hallway blockade trick.

The green room

Again, a horde of golems.

all golems hostile in the golem room
This time, the party needs to flip a switch in the green phase, which triggers an encounter with four boss golems and two elite golems. If you didn’t feel enough hurled rocks and full-bodied rock slams in that previous golem encounter, this one will surely give enough. Even with Brogan the warrior, no one in a level 20 party can tank through that. There is a doorway chokepoint, but boss golems have huge resistances, and even a full Storm of the Century will only take off 1/3 of their hitpoints on Nightmare. The blocker golem won’t survive too long after the Force Field expires; certainly not long enough to get another Force Field up. Once again, they all activate at the same time, so you can’t pull one away by itself…or can you?

This one needs an underhanded trick. Remember, there was a room next to the starting chamber of the Amgarrak level with a blue and a purple switch. Both of these switches will throw the party out-of-phase with the golems, meaning that they disappear from the map. When this happens, the hostility chase trigger is also reset. If you can somehow kite the golems to other rooms, then run to the switch to go into blue or purple phase, walk away while phased out, and then switch back into normal phase when you’re far away enough…these golems will lose sight of you and stand still. Furthermore, now they’re isolated from each other, and you can pull each one independently instead of triggering all six at once! This means you can pull each golem, kill it, heal, and repeat.
Run back into this room and hit a switch to avoid the chasing golems
So the strategy is as follows: after you hit the green room switch to activate the golems, immediately run back out of the room, down the stairs, and toward the starting chamber. The faster you run, the more the enemy golems aren’t going to keep up, and will be strung out along the way. As you arrive in the switch chamber, switch phase. All the golems disappear as you go out of phase with them.

Walk back to the green room, switch phase again, rob the chests if you need to, and return to normal phase. Slowly go down the stairs again, pull each golem one at a time along the way, kill them one by one until you arrive at the switch room again. Problem solved.

The Final Battle

This is the fight that few people seem to win as a poorly geared level 20. This is because they are going toe-to-toe with the Harvester flesh golem. There are two major problems with this.

  1. The Harvester hits for 90+ damage, and crits for 100+ damage on Brogan or the tank, due to the crappy equipment setup of the stock party. There are not enough healing poultices or spells to keep the tank alive at that rate.
  2. There is a second problem — the Harvester in the Flesh Golem phase periodically summons a skeleton, either with his Vile Disembowelment skill, or they just wander into the room. As the flesh golem takes damage, the skeletons spawn faster. The skeleton is usually elite, which hits really hard, or boss, which can seriously damage even the tank. This isn’t even the main problem — the real problem is that the skeletons go after the DPS’ers. Because Brogan has terrible threat management, the skeletons will wander into the room, go immediately for the caster or the archer, use Riposte to stun them, and hit for 50+ on leather and even higher on cloth. If the mage and/or the golem goes down, it’s a guaranteed wipe.

Since the tank can’t hold up, the only strategy is to kite the Harvester around the room. There is plenty of space to maneuver, but there is not enough DPS with a stock party to take it down quickly enough. Jerrik hits for 80 to 100 per arrow on the Harvester, and crits for a bit higher. The mage hits at around 30 or 40 with the staff and more with spells if not resisted, but she can’t cast too many attack spells as healing is very needed and there are about 10 to 20 lyrium potions in all. The rest of the party is fairly useless in terms of DPS, since they’re melee and can’t get a good hit in on the Harvester as it walks around. The room is going to be filled with skeletons. They all have stun skills. If the kiter take a stun or two, he will get swarmed and probably die.


The party needs to set some basic tactics.

  • Brogan needs to take the least powerful health poultice at 25% health or less
  • Jerrik needs to set typical archer tactics, and feign death at health < 10%
  • The PC’s mage needs to drink a least powerful lyrium potion at 25% mana, cast Regeneration on an ally when the ally’s health is < 50%
  • The Runic Golem should turn off his Hurl Rock and Quake tactics, and cast Group Heal when an ally’s health < 50%

Have the Runic Golem set Flame Weapons. When the battle begins, coat weapons with the highest level poison you have available.

Phase One

In Phase One, the Harvester will jump inside a flesh golem and open with a Vile Disembowelment, an area-of-effect attack that immediately spawns a new skeleton.

The initial setup against the Harvester

As the fight begins, set the party to Hold Position, and manually move the party members into position. The mage and the archer should go stand by the switch that appears, but far enough apart that they don’t get nailed by the same AOE attack at the same time. Brogan needs to stand close by as the off-tank. The Runic Golem should stand at the center of room with his Cleansing Aura. The bronto (which actually has Taunt and runs faster than any other character in combat) should attack the Harvester, throw down a Taunt, and then start kiting the Harvester around the Runic Golem’s aura radius.

If a normal skeleton appears, Brogan should go tank it immediately, and the mage can help freeze it with a Cone of Cold. If an elite or boss skeleton appears, or if there are more than 3 skeletons, have the mage pull the switch immediately to destroy all the skeletons. Jerrik should always focus on the Harvester. If either the archer or the mage is under attack, have the bronto run over and Taunt, then run the mage and/or archer away. The skeleton should lose interest and focus on the bronto. If not, Brogan needs to pick it up and tank it while the DPS’ers run far away.

Heal the bronto constantly, as it’ll inevitably get hit by the Harvester and its skeletons. If the bronto is stunned, or if it is killed, Brogan must pick up threat and resume kiting immediately until Jerrik is able to resummon the bronto. Switch back once the bronto is up again.

(If this is starting to feel like playing a MMO 5-man dungeon boss encounter, except you have to control every party member by yourself, well…yes. Use that Pause function. The Pause key is the one major difference here that lets you control the multiple things going on in this encounter.)

Keep an eye on the archer and mage while you’re kiting with the bronto. Make sure no skeleton has wandered into the room and started attacking them. Pause a lot if you need to.
Nearing the end of the First Phase of the Harvester Fight
If this is executed correctly, eventually the Harvester will fall. When its health gets very low, try to avoid hitting the switch and save it for just before Phase Two. The switch will not appear again at that point, and there may be several skeletons that need killing at the end.

There is a bit of luck involved. If the switch is hit too early, and elite skeletons emerge while the switch is inactive, then this is probably bad. If it’s a boss skeleton, you might consider using Brogan to kite it around until the switch is available again. Controlling two kiters is very difficult, so pause a lot.

Phase Two

When the flesh golem is killed, the Harvester will jump out and attack. If the party made it this far, and the switch is still available, then the battle is more or less won. The Harvester lizard-form is far, far weaker than its previous flesh golem body. It hits for about 60 on Brogan, but hits slowly and misses a lot, so it can be tanked. It also disengages a lot to channel a skeleton summoning spell, so it doesn’t have a lot of DPS to hit the party with. The best news: skeletons will no longer wander into the room in large packs — at this point, the Harvester has to summon one with his spell.

When the game screen comes back from the cutscene, hit the switch one last time to clear out all remaining skeletons if it’s still available. If not, resume the kiting strategy and quickly remove the remaining skeletons. Then, have the party resume normal attack stance from hold position and focus fire on the Harvester’s lizard form. If it uses the jump ability, cast Crushing Prison (it won’t resist nearly as much now) or the Runic Golem’s Slam attack to slow it down while the melee attackers catch up. It can also be frozen solid with Cone of Cold, though it can’t be shattered obviously. Jerrik and the mage are still going to be the main DPS. If the Harvester finishes channeling a normal or elite skeleton, tank it with Brogan and blast it down quickly. If it’s a boss skeleton, Force Field it or have the bronto kite it around the room while the remaining party members continue to blast the Harvester. It is only a matter of time before the Harvester dies.

The Grim Reaper achievement should hit the screen after a short cutscene. Woot!

The first time I did this, it took me three tries and about an hour of playing, but is definitely a reproducible strategy on Nightmare difficulty. If playing on Hard, this is actually really easy, as most skeletons will be normal skeletons and can be frozen and shattered. It does take a lot of micromanagement, but since you can pause the game and issue new orders, this management shouldn’t be too hard — kinds of reminds of me Baldur’s Gate 1, where I auto-paused after every combat round and barely took any damage.

As a final word: the builds and strategies outlined in this note are *not* how you should typically play DAO. Given an entire campaign to work with, the builds here are suboptimal. These builds are designed only to complete this specific module, within the constraints that it imposes.

musings on self and search

So as you may be aware, I developed SelfSolved in order to avoid cluttering up this blog with “I fixed this!” posts. I’m trying to get into a habit of writing at a higher conceptual level, instead of this repository being just yet another technical problems blog. It turns out that without writing about random things that I fixed, I have little else to say. When I wrote up problems for this blog, I had monthly posts. Now I’ve gone 3 months without writing anything.

It’s odd, as Ph.D. students (and former startup founders) can hold forth via very long and extensive blog posts, especially on technical topics they care about (and sometimes on topics that they don’t). It may speak to my lack of aptitude for Ph.D. work; I hope not.

When I implemented SelfSolved, I created a sitemap for search engine crawlers as a matter of course, but did not do so immediately. Lots of content were present on the site before I completed the sitemap feature. It turns out that search engines like the GOOG seem to ignore certain content in the sitemap — so much for indexing all the world’s knowledge. I thought the crawler would get around to it eventually, but it doesn’t appear to be the case after nearly a year.

It’s interesting, because there doesn’t to be any rules. For example, “turning caching off completely in Pylons” has a very short problem statement and solution. Other posts, however, are fairly normal: “SVN directory is viewable through Apache” is by all accounts a normal sized SelfSolved problem, with references and a full solution writeup. As of this post, neither are indexed.

As soon as I wrote this blog post on this WordPress installation, the post became available in GOOG’s main index. If I search on the phrases above, I get an entry back immediately. But for a site like SelfSolved, these indexes aren’t available for days or weeks, even though SelfSolved publishes an Atom feed of its content as well, and pings the same GOOG notification URIs. This does suggest that if you build your site on a well-used CMS, rather than custom-designed software, it will get you into indexes in almost real-time vs in days. If you run a for-profit content-driven site (which SelfSolved is not), that could mean a fair bit of money.

It’s no wonder that SEO consultancies are flourishing. There are certain hidden rules with crawling that go beyond the simple “create good content” and “get a sitemap” guidelines that the GOOG wants you to believe.