check progress of photoanalysisd

If you’ve just installed Mac OS Sierra and now see photoanalysisd sucking 100% to 200% CPU power, this process is doing some kind of face detection + object / image recognition / indexing on your Photos library.

photoanalysisd progress
photoanalysisd progress

Open Photos.app, select People on the left sidebar, and see how far it’s gotten in this task. If you have a very large Photos library, this might take a while.

As a bonus, while Photos.app is open, photoanalysisd is suspended, allowing your laptop fans a bit of a rest.

It’s all for a good cause. After indexing, you’d be able to type “ocean” in the search box and get all your photos with the ocean in it, without having to tediously tag all your photos yourself. Magical, huh? ( Not really. If this is doing what I think it’s doing, well, it’s the kind of problem computer vision researchers have been tackling – and solving – for years already.).

As an aside, for consumer-grade apps like Photos, arguments have been made to do computationally intensive image analysis in The Cloud(tm) instead of on client machines. The tradeoff is fairly obvious. In exchange for the privacy benefit of Apple not uploading color histograms of all your photos to its own cloud servers (something I’m sure Google wouldn’t bat an eyelash at doing), you’ll have to pay the cost of doing this analysis yourself, with your own measly consumer/mobile-grade CPUs (which isn’t ideal if you want to get work done at the same time that this analysis is going on). Overall, the user experience probably could have been handled better, especially given the extensive public beta period that Sierra received. The current opaque process violates some core UX principles: giving users (at least the feeling of) control, and giving the user clear reasons to trust in the importance of the task being performed. Apple should have known that eating 100% CPU while people are actively working with their machines is immediately noticeable, and should have 1) let the user know that image analysis is about to happen, and 2) given them the option of choosing when and how much CPU to devote to this task.

Setting up OpenSSH Server on Windows 10 Anniversary Update

UPDATE 2017-12-22

Microsoft now ships a real SSH server (and client!) with Windows Fall Creators Update, no Cygwin or Linux subsystem required. Finally.

This post remains for historical context only.

Microsoft has finally landed its anniversary update for Windows 10. Among all the random useless features, is an actual Ubuntu Linux subsystem within Windows, with the ridiculously silly name “Bash on Ubuntu on Windows”. Goodbye Cygwin?

One of the first things I wanted to try was to setup the SSH server, so I can remote-login from my real box. Getting one up was actually fairly easy, if you can deal with a few problems with weird, weird red-herring error messages.

The problem

To recap, installing the OpenSSH server is as easy as popping open a bash shell (I assume you figured out how to get the Linux subsystem installed already and popped open Ubuntu )

sudo apt-get install openssh-server

If you try to now start the server with

sudo service ssh start

It’ll respond with

initctl: Unable to connect to Upstart: Failed to connect to socket /com/ubuntu/upstart: Connection refused

but subsequently

Starting OpenBSD Secure Shell server sshd [OK]

When you now try to connect in from a remote host, there are two outcomes:

  1. There is no SSH server on port 22.
  2. There is an SSH server on port 22, but it responds only to your Windows password and not your Ubuntu Linux user password. If login is successful, it drops you into a DOS prompt instead of a bash prompt. Trying to run bash within the prompt generates a response:
    "Error: 0x80070005".

Diagnosis

There are multiple underlying problems here.

  1. First, it’s important to note that there exists a separate Windows SSH server (separate from the OpenSSH server from the Ubuntu subsystem installation) now on port 22. If you telnet (yes, telnet) into port 22, and the host greeting is SSH-2.0-MS_1.100, you’ve run into the Microsoft SSH implementation (hence the MS part in the greeting).

    I’m not sure what this server is supposed to do (and, in fact, starting SSH servers on standard ports without explicitly telling the user seems like a potential security problem to me, in addition to being kind of jerk-ish). I do know it is launched when you reboot with Developer Mode on. You know, Developer Mode, the mode you have to turn on to run Bash on Ubuntu on Windows. This server is occupying port 22, so you cannot launch another SSH server to listen on this port. This is the problem that is causing the silent failure of sshd to start up, not a broken Upstart or any of that nonsense the error message being displayed is referring to.

  2. Apparently the Ubuntu compatibility layer on Windows does not implement chroot or one of the related system calls needed for OpenSSH privilege separation at the time of this post.
  3. If you launched bash from DOS cmd.exe, instead of the Bash on Ubuntu on Windows shortcut, /mnt/c/Users/[Windows Username] (the directory you start with in DOS) is not your Linux home directory. .ssh config files left in this directory will not configure your OpenSSH server.

Symptoms & Solutions

  1. To resolve the problem of conflicting SSH servers, set your OpenSSH server on a different port : one other than 22. This can be done by editing /etc/ssh/sshd_config and changing the Port configuration. Be sure to open this new port in your Windows Firewall for inbound connections, as this firewall configuration will not be automatically done for you. I set mine to 60022.
  2. To enable login, also change UsePrivilegeSeparation in the same config file to No. Failure to do so will cause the server to respond with Connection reset by nnn.nnn.nnn.nnn
  3. To enable public key auth and set other configurations, the Linux subsystem home directory is, as in normal Ubuntu, /home/<yourname>.

Having done this, now you can start or restart in the bash shell

sudo service ssh restart

And connect from remote host with your Ubuntu user credentials and/or public key:
$ ssh -p 60022 10.0.1.208
...
Welcome to Ubuntu 14.04.5 LTS (GNU/Linux 3.4.0+ x86_64)

(And if you telnet, the host message should be something like
SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.7
)

Not paying OnTheHub to redownload Windows ISOs

If you’re looking to re-download Microsoft Windows 10 Education edition ISOs and have a valid key from the university licensing program already, grab the education edition ISO images directly from Microsoft instead of paying OnTheHub/Kivuto protection money.

Context

UC Berkeley, like many other universities, has a volume licensing deal with Microsoft for its operating system products. In particular, it offers free downloads of Windows 10 Education edition to all current students.

The irritating thing about this is that they offer this deal through a shady vendor called OnTheHub or Kivuto. Instead of allowing the ability to re-download the ISO image, it holds you hostage if you want to access the image file again after 60 days.

When you first download, it offers you a discounted “extended access guarantee”:

To ensure that your download and/or key(s) remain accessible to you, you can extend your coverage to 24 months with the Extended Access Guarantee for $4.95. With this service, Kivuto will back up your download and/or key(s) on their servers, allowing you to access this information at any time under the “Your Account” section of the WebStore.

If you declined to take advantage of this oh-so-generous offer, and find yourself needing the ISO again after 60 days:

Access Guarantee Retrieval (60 days)
Purchase this service if you wish to recover your download(s) and/or key(s) after access has expired.
You will gain another 60 days of access to any expired product in your order.Learn more
$11.95

In my case, I backed up the wrong ISO — the generic install ISO, instead of the education edition ISO. The product key Kivuto issues you is for the Education edition, so the generic ISO you can get online won’t let you install with the key. It’ll report an error of “The product key entered does not match any of the Windows images” at install time.

So when it came time to reinstall, suddenly I’m faced with the OnTheHub protection racket.

Solution

For Windows 10, at least, you don’t have to pay the protection money. Education edition ISOs are available directly from Microsoft after product key verification. The download page is rather well-hidden, under the “more options” link from the software download page.

Unsurprisingly, OnTheHub makes no mention of this download source.

I wonder how much money OnTheHub makes off people who didn’t realize there was a free, official source for Windows 10 ISO images. Bandwidth costs money, sure, but you are a *education software download vendor* in 2015, with a target market of underpaid students and faculty. Nickel-and-diming poor student users for software downloads, in this era of cloud-driven computing, is an obsolete and despicable business practice.

Python multiprocessing code crashes on OS X under IPython

While working on a project on OS X 10.10.4 Yosemite, some innocuous Python 2.7.6 code using the multiprocessing module (via the concurrent.futures module) was crashing when run from the IPython interpreter, but works just fine when executed via shell directly.

Issue

Some nice simple test code. It pulls some data from two Web servers, via the requests module, concurrently using a multi-process worker pool.

from multiprocessing import Pool
import concurrent.futures
import requests

TEST_URI = ["http://example.com", "http://yimingliu.com"]

def http_get(uri):
    return requests.get(uri)

def test():
    p = Pool(5)
    print(p.map(http_get, TEST_URI))

def futures_test():
    with concurrent.futures.ProcessPoolExecutor() as executor:
        print [response for response in executor.map(http_get, TEST_URI)]

if __name__ == '__main__':
    test()

When run as $ python mp_test.py, everything works beautifully. When test() or futures_test() was called from an IPython shell however, a segmentation fault occurs:

Exception Type:        EXC_BAD_ACCESS (SIGSEGV)
Exception Codes:       KERN_INVALID_ADDRESS at 0x0000000000000110

...

Application Specific Information:
crashed on child side of fork pre-exec

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0   libdispatch.dylib             	0x00007fff9772c16f _dispatch_async_f_slow + 395
1   com.apple.CoreFoundation      	0x00007fff92e91541 _CFPrefsWithDaemonConnection + 305
2   com.apple.CoreFoundation      	0x00007fff92e60ac6 __80-[CFPrefsSearchListSource generationCountFromListOfSources:count:allowFetching:]_block_invoke + 150
3   com.apple.CoreFoundation      	0x00007fff92e609d2 -[CFPrefsSearchListSource generationCountFromListOfSources:count:allowFetching:] + 258
4   com.apple.CoreFoundation      	0x00007fff92d1cea5 -[CFPrefsSearchListSource alreadylocked_copyDictionary] + 133
5   com.apple.CoreFoundation      	0x00007fff92d17dba -[CFPrefsSearchListSource alreadylocked_copyValueForKey:] + 42
6   com.apple.CoreFoundation      	0x00007fff92e9211c ___CFPreferencesCopyAppValueWithContainer_block_invoke + 60
7   com.apple.CoreFoundation      	0x00007fff92e5f979 +[CFPrefsSearchListSource withSearchListForIdentifier:container:perform:] + 729
8   com.apple.CoreFoundation      	0x00007fff92e92097 _CFPreferencesCopyAppValueWithContainer + 183
9   com.apple.SystemConfiguration 	0x00007fff96be8db4 SCDynamicStoreCopyProxiesWithOptions + 153
10  _scproxy.so                   	0x000000010e07f915 0x10e07f000 + 2325
11  org.python.python             	0x000000010d89a9ed PyEval_EvalFrameEx + 14935
12  org.python.python             	0x000000010d89d60e 0x10d813000 + 566798
...

The same code works just fine under Ubuntu Linux, from IPython or otherwise.

Workaround

It turns out that there is some kind of edge case bug going on between IPython, multiprocessing, and OS X. Namely, a segfault seems to trigger from within OS X’s Grand Central Dispatch architecture, when multiprocessing is forking a new process to access the network, under IPython.

Note these lines in the crash log:

0   libdispatch.dylib             	0x00007fff9772c16f _dispatch_async_f_slow + 395
...
10  _scproxy.so                   	0x000000010e07f915 0x10e07f000 + 2325

Something in _scproxy.so (a part of Python’s lib-dynload) is not happy with libdispatch.

_scproxy.so calls out to OS X’s SystemConfiguration framework for network proxy information. The quickest workaround for this bug, then, is to disable this proxy check. IPython seems to respect the customary no_proxy rule to skip proxy checks (at least IPython 4 does), so:

$ env no_proxy='*' ipython

and then

$ env no_proxy='*' ipython
Python 2.7.6 (default, Sep  9 2014, 15:04:36) 
Type "copyright", "credits" or "license" for more information.

IPython 4.0.0-b1 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: import mp_test

In [2]: mp_test.test()
[<Response [200]>, <Response [200]>]

works just fine. No segfault.

Of course if you actually need this proxy functionality, you would be out of luck.

Setting up a Gmail POP3 account for Mail.app on 10.10.3 beta

The last beta of OS X has apparently removed the option to create POP3-type email accounts for Gmail in Mail.app. When you type in a Gmail account, IMAP is setup automatically. Presumably because OAuth doesn’t work all that well with it.

There (currently) is a workaround.

  • Turn off/unplug your Internet connection. This will force Mail to not verify the settings that you are giving it.
  • Open Mail.app -> File Menu -> Preferences -> Accounts tab
  • Hit the + button to Add Other Mail Account
  • In the email address field in the following box, enter something that IS NOT @gmail.com. Enter some random email.
  • Mail.app will now inform you that manual setup is required. That’s exactly what we wanted in the first place, thank you very much. In the next screen, you will get to pick IMAP vs POP. Pick the POP tab and enter the correct Gmail POP server information and Gmail outbound SMTP server in the following screens, for your account. Make sure to use SSL and port 995 if appropriate
  • Use the application-specific password in the password field if you have two-factor auth setup.
  • Once setup is complete and you are thrown back into the preference pane, change the “Email Address” field back to the correct email address @gmail.com
  • Under “Advanced”, untick “automatically detect and maintain account settings”
  • Quit Mail.app. Turn Internet connection back on
  • When you re-open mail.app, the POP account should be operational

Hooray for we remaining holdouts, still using obsolete and dubiously secure technology. Because when I want mail on my local machine, I mean I want mail on my local machine, The Cloud(tm) be damned.

Note they might still change this depending on feedback received. I would advise giving such feedback if you care at all.

Mail.app stuck at fetching mail on OS X 10.10.3

If Mail.app is stuck at “Fetching Mail….” with Gmail POP accounts after upgrading to OS X 10.10.3’s beta, be aware that Apple added OAuth support to Mail.app. Which is nice, except in the current beta, they don’t warn you to go set up OAuth permissions for every account. Mail.app will choke on fetching mail, without letting you know the actual problem. This is possibly POP3-specific, as I didn’t notice IMAP having this issue.

In the connection log:

[kCFStreamSocketSecurityLevelTLSv1_0] -- host:pop.gmail.com -- port:995 -- socket:0xxxxxx -- thread:0xxxxx
-ERR invalid SASL argument ......

The solution here is to simply set up Google OAuth via System Preferences -> Internet Accounts. It should automatically prompt you for OAuth permission to your Google Account when you click on the affected account in the prefpane. Once that is done, Mail.app will fetch Gmail via POP3 again.

I’ve also unticked “automatically detect and maintain account settings” under Advanced, in Mail.app’s accounts panel, though this might just be superstition rather than actual prevention of issues. I’m just old school and uncomfortable with the thought of “automagic” happening in my mail server settings.

If you’ve already given up and swapped to IMAP, but would like to come back to the Dark Side of email retrieval technology, see also:
How to setup a Gmail POP3 account from scratch on 10.10.3 beta

Selfsolved reference:
#127: Mail.app stuck at fetching mail on new OS X

I really need to rebuild Selfsolved. Someday, when I’m not trying to write.

restoring Safari preferences from backup files in OS X Mavericks

Recently I had the misfortune of having to restore some Safari settings from backup, on OS X 10.9 Mavericks. I have done this many times before on older OS X versions, without incident — simply pull the various preference files such as com.apple.Safari.plist from backup and replace the damaged/unwanted ones. Takes all of 2 minutes, and years ago, I had already wrote a shell script to do exactly that.

It turns out that after Mavericks, Safari is incredibly resistant to conventional methods of preference file backup and restoration. Considering that preferences in OS X have always been stored in XML-based preference lists, you would think (as in previous operating systems from 10.0 to 10.7) taking the relevant preference files and replacing the unwanted new ones in ~/Library/Preferences would be enough. But no, an incredible amount of effort is now required for a simple task:

  1. unhiding the Library directory, because clearly we’re all Windows babies who cannot be trusted to see where application preferences are stored
  2. replacing the actual preference files, scattered around the system in ~/Library/Safari, ~/Library/Preferences, ~/Library/Caches, and hopefully remember to have turned off iCloud, or otherwise see changes being clobbered by iCloud sync (or, worse yet, having experimental changes reflected all across a network of Mac and iDevices)
  3. resetting the preference cache daemon, cfprefsd so that preference changes can be reflected in a running system. This where a lot of people get stuck in general, judging by Google results; when they replace preference files and find that their changes aren’t being reflected, it leads them into a wild goose chase for “hidden preference files” for Safari, when the answer lies in a simply yet utterly non-obvious background daemon.
  4. restoring the list of installed extensions — which, incredibly, is NOT stored in the deceptively named com.apple.Safari.Extensions.plist, but in the login.keychain.

Context

Had some issues where certain websites were behaving differently under private browsing mode than normal browsing mode. I deduced there was some kind of corrupted stored state, whether it was a cookie or localstorage issue. Had the brilliant idea of setting Safari preferences aside, thus resetting Safari to factory state, and then divide-and-conquer by restoring parts of the settings until the problem recurs. I’ve done this many times before.

First I turned off iCloud sync, having been bitten by sync propagation of experimental changes in the past. This is pretty important if you don’t want to blow up Safari bookmarks (at the very least) across all Apple-manufactured, iCloud-compatible devices. I then removed ~/Library/com.apple.Safari.plist, ~/Library/com.apple.Safari.Extensions.plist, ~/Library/Safari, and ~/Library/Caches/Safari, ~/Library/Cookies. After resetting to confirm some issues have disappeared, I moved some files from backup to original locations. Imagine my surprise when nothing became restored, and all my Safari extensions (installed from the extension store or custom-built by me) disappeared.

Process

Increasingly desperate, I started to trace filesystem accesses using fs_usage. It showed nothing out of the ordinary. 30 mins of reviewing useless forum posts later, I pieced together a multi-stage solution. It turns out there were two separate obstacles.

Preference caching

Presumably to save energy, OS X Mavericks caches application preferences (in RAM?) using a daemon called cfprefsd. Instead of applications pulling their preferences from XML files on disk at launch, it requests this from the daemon instead. The defaults command has been modified to operate with this daemon, so if you had been working with preferences from the command line (as I have been), the changes have been transparent.

However if the preference files are changed or edited directly, this change is not propagated to the preference cache daemon. When the app is opened again, the cached version takes precedence, and is re-written out to disk, clobbering the restored versions.

This does not mean there are hidden Safari preferences somewhere that you haven’t found, though you might think this at first. When Safari is reset manually from the filesystem, or if the plist files are edited, cfprefsd must be reset as well.

There exists a cfprefsd daemon for every logged in user, running under that user’s privileges, as well as a root-owned one. Safari preferences are stored under the user domain, so the user-specific daemon is the one that needs a reset when files change. Can also quit the process from Activity Viewer, or killall cfprefsd. A login-logout cycle would also reset the user-specific daemon.

Extension list caching

Having done this reset with the backup files in place, most preferences will be restored on next launch, *except* the list of extensions you had installed previously. That will remain empty. Even though all the extensions and their settings have been restored to ~/Library/Safari.

For a long time I traced ~/Library/com.apple.Safari.Extensions.plist, and wondered why it wasn’t being read.

An Apple discussion forum post (shockingly enough) gave a vital clue. There exists an “application password” in the login keychain titled “Safari Extensions List”. Whether it is merely a cryptographic key, or the actual list of extensions, is unknown, but that is the critical preference to restore extensions. Having reset extensions by moving them away, this preference is apparently emptied out. The entire login keychain, being an encrypted document, has to be restored to a corresponding previous version to restore access to previous extensions. Without this, all extensions would have to be reinstalled manually (and get a new copy of the extension file stored into ~/Library/Safari/Extensions, instead of the previous version being reused).

Discussion

Given recent focus on energy consumption, I can understand preference caching. However, it’s not that hard to track filesystem changes (the Time Machine/Spotlight APIs explicitly do this!) and reload appropriate preferences when they are changed on disk. It would show respect for power users and developers who might need to interact with the preferences system in a more convenient way.

But stuffing extension lists in an obscure corner of a password keychain? What sense does that make? Are my list of extensions (not actual extension data or settings, mind you — those are in plaintext on the disk for anyone to copy and look at) such privileged information that it has to reside alongside my login password? Why can’t you just read the list of extensions, oh I don’t know, from the list of extension files installed into the Extensions directory? Wouldn’t that be a lot more reasonable?

A Twitter timeline to Atom feed proxy

So Twitter is retiring version 1.0 of its API in March 2013. In its ongoing quest to become (more) evil, Twitter has decided that open syndication standards like Atom are no longer worth supporting. This, in addition to the gratuitously byzantine OAuth system (even for 2-legged auth between my own client and Twitter itself), makes consuming Twitter content anywhere else except on Twitter (and its official apps) an increasingly annoying task. An intended effect, perhaps.

I’m one of the few holdouts who believe current Web standards work just fine. I consume a lot of content from the Web in my native RSS client, as part of my ordinary daily workflow, without using 50 different apps and dozens of notifications. This includes my Twitter home timeline (where I follow just under 70 people/orgs of interest), which under API v1.0 was provided as a simple, standard RSS feed, like other open streams of data on the Web.

I’d been hacking on a Python-based Twitter JSON API to Atom feed proxy for some time, but became increasingly disillusioned with the stupidity of the API — and that it takes two libraries and tons of code to even get the OAuth dance started.

So I thought, in this entire WWW, there must be someone else as annoyed by Twitter’s obstinacy as I am. Sure enough, Russell Beattie developed a single-file PHP script that accesses the authenticated Twitter stream and outputs an Atom feed.

I abandoned my Python code, forked that codebase, and made some minor modifications to suit my personal needs. This patched version is available here:

https://gist.github.com/yimingliu/4735445

All you need to do to use this script is to create your own Twitter app over at https://dev.twitter.com/apps (I called mine “TimelineProxy”), create an OAuth token, and fill in the blanks in the script. Since the new API also has a 100,000 user per application limit, it’s probably best for every user to have his own proxy app with its own token, instead of relying on a central one.

There are some minor differences between my version and the upstream original. Basically this version uses full php tags instead of the less-well-supported short tags, and replaces t.co shortened URLs with their full original URLs. I also take advantage of the html content type in the Atom entry to allow links in the entry text, so in most feed readers any links are “clickable”. Finally, this version returns proper HTTP error codes instead of 200 OK in case of Twitter API errors (like when you hit the rather draconian rate limits on each OAuth token).

However, it preserves the simplicity of the original, which is that you can drop this in the web directory of any PHP-enabled web server (no need for root access or installation of any libraries) and enable your own timeline proxy.

I want to eventually modify this script so that a single server can host any arbitrary user’s timeline as an Atom proxy, provided they give the proper auth tokens. This means having to deal with the OAuth dance at some point. Ugh.

In any case, this disturbing “enclosure movement” of the open Web — taking previously free streams of information and fencing them into walled gardens of content — is a trend that should be opposed whenever possible. My thanks to Mr. Beattie for making the original script.

Outlook 2011 for Mac still adding arbitrary line breaks into plaintext emails

Outlook 2011 on Mac OS X, v14.1.3, for whatever reason, still does not properly support “format=flowed” content-type or “quoted-printable” extensions for plaintext emails. This causes plaintext emails to be sent as mangled messes, full of arbitrarily inserted linebreaks. This appears to be a regression from Entourage, as far as I recall, which never handled plaintext quite this badly, and this is also despite Microsoft’s promises to have “implemented format=flowed”.

This is the last straw. I’ve been a loyal MS Entourage / MS Outlook user since the days of Outlook Express for Mac and Office 2001. But at this point, this software has actively impeded my communications with my friends and colleagues. We’re done.

The Problem

Here’s a really simple illustration of the problem, from the receiver’s end:

See how the URL, which was composed as one plaintext line, gets split up into two lines?

Here is another example, purely from the editor UI (and not even being sent yet). I start with a perfectly good reply saved as a draft:

I make a small wording change and resave:

See that third line? Thanks to the hard line breaks inserted by Outlook (even at composition stage), the line wrap has been mangled. This draft has to be re-wrapped manually, by the tedious process of deleting the newline-based hard line breaks from every line following in the paragraph. That was a short paragraph. Imagine doing that in a long paragraph, from the first line.

To add insult to injury, there is not even a “re-wrap” functionality in the editor, to at least solve this user-interface level problem (as opposed to the protocol level problem). Obviously no one at Microsoft sends plaintext emails anymore.

The Issue

Back when email was first devised, servers didn’t have a lot of memory, and people had pretty tiny terminals with fixed line widths and not a whole lot of processing power to deal with it. The Internet standards for email messages http://www.ietf.org/rfc/rfc2822.txt, RFC2822 Section 2.1.1, defines recommendations for email body text transferred over SMTP:

There are two limits that this standard places on the number of
characters in a line. Each line of characters MUST be no more than
998 characters, and SHOULD be no more than 78 characters, excluding
the CRLF.

The 998 character limit is due to limitations in many implementations
which send, receive, or store Internet Message Format messages that
simply cannot handle more than 998 characters on a line. Receiving
implementations would do well to handle an arbitrarily large number
of characters in a line for robustness sake…

The more conservative 78 character recommendation is to accommodate
the many implementations of user interfaces that display these
messages which may truncate, or disastrously wrap, the display of
more than 78 characters per line…

…it is encumbant upon implementations which display messages
to handle an arbitrarily large number of characters in a line
(certainly at least up to the 998 character limit) for the sake of
robustness.

Basically, the SMTP server can count on messages that come in 80 characters per line (and always less than 1000 characters per line), and email clients can trust that they only have to render up to the 78th column of text. This limitation is hardly useful in the modern age, but persists since it’s part of the standard. And it’s a fine, conservative design model. But now we write some pretty long lines without linebreaking ourselves, so something magical has to happen in the email client itself, like Outlook 2011.

The naive solution, of course, is to slap arbitrary line breaks into the user’s email message at every 78 characters, which is what ye olde email clients (looking at you, pine — how did I ever put up with you…) from yesteryears did (and Outlook 2011 still does). It’s a matter of personal preference whether this is a reasonable solution. Proponents argue that the email will “always look the same” on all devices, including those limited to 78 chars per line.

I (and many others), on the other hand, think the spirit of the RFC is to allow the actual handling client to decide where to break lines. With the exception of source code, it is almost always better for the email client to use the full width of their display, however many characters that might be. Even in the case of source code, it should also not be mangled by the insertion of arbitrary line breaks in them — what if newlines are meaningful in this language, and the author used more than 78 characters per line? The example with the URI is illustrative of this problem — the URI got an arbitrary newline in the middle, destroying its meaning. Users who copy-paste the two lines will end up getting a 404, due to that stupid inserted newline in the middle of it. This should not be allowed to happen.

Because this naive solution was not perfect, an extension was proposed as RFC 2646. This format of email is characterized by the content-type:

Content-type: text/plain; charset=US-ASCII; format=flowed

In format=flowed emails, the sending and receiving email clients are allowed to reflow the text based on user linebreaks. It follows some simple reflowing rules, but in short it will preserve user-inserted hard line breaks while adjusting the rest of the message for the proper line length while the message is “on the wire”, and recombining the lines on receipt and display. Modern email clients like Thunderbird, designed for user comfort and the generous system limitations of the year 2011, implement this standard.

Guess what format Outlook 2011 sends?

Content-type: text/plain; charset="US-ASCII"

Not even an option to change that behavior. It does not appear that Outlook 2011 deals with any of this. It just inserts some line breaks and calls it a day.

An alternative, implemented by Apple’s Mail.app, is to send messages with the Content-Transfer-Encoding header set to “quoted-printable”, as per RFC 2045. In this model, soft line breaks are sent explicitly with the character “=” representing it, breaking at the usual 70-odd character column. On the receiving end, the client processes this character as a no-op and concats the line back together for display.

Outlook doesn’t do that either. It just wants to mangle your emails.

Conclusion

The world moved on and adopted HTML emails, which doesn’t have this newline problem. For those of us who do think HTML emails are an atrocity to be used sparingly, if at all, the idiosyncrasies of plaintext email have to be addressed. Outlook 2011 appears to do even worse than Entourage 2008 at this problem, by not dealing with it at all. And apparently getting a bunch of Microsoft “MVPs” on their forums to cloud the issue with promises of support and unrelated commentary.

Given the sad state of email clients on the Mac, I believe Thunderbird is now my only option for sane plaintext messaging.

Dragon Age 2: modding the Edge of Night axe into a longsword

Having a bit of fun with Dragon Age 2 in my spare time. The selection of longswords in Dragon Age 2 leaves much to be desired. Call me old-fashioned, but when I play fantasy RPGs, my warriors are always sword-and-shield (munchkin min-maxing be damned), because…that’s how heroic fantasy warriors are supposed to be. As TvTropes succinctly put it, Heroes Prefer Swords. I actually shelled out money for the Warrior Item Pack DLC hoping to find a better longsword. Unfortunately, the only decent one-handed weapon in the pack is an axe. An axe?! What are we, woodcutting barbarians?

So I went about looking for a way to switch this otherwise beautiful weapon, The Edge of Night, into a longsword. The similarly named Edge of Song and Glory is the ideal target for the transformation.

There is not much modding possible for DA2 — the game is very much locked down to drive DLC sales. Nevertheless, the base game itself is at least unencrypted; however, there is no official mod editor support. The DLCs, however, are encrypted — you cannot even view the data contained within these files. The following modding method requires that you decrypt the DLC’s designeritems.erf file first, then extracting the Edge of Night’s item file for editing. This only works for PC versions of DA2.

Note that there is already a mod floating around for this. There is a good thing and a bad thing about this mod. The good thing is that it uses a less questionable method that does not need to decrypt the ERF file. On the other hand, because of the limitations of such a method, that mod doesn’t change the item icon or the item description. It will still look like an axe and call itself an axe in the character’s inventory screen — only the actual gameplay model is changed to a longsword. Some people would be satisfied with that; I wasn’t.

Decrypting designeritems.erf

DLC items are in found in ${DRAGON AGE 2 INSTALLDIR}/addins. In this case, it is the DA2_MTX_WAR addin. The ERF file is
${DRAGON AGE 2 INSTALLDIR}/addins/da2_mtx_war/core/data/designeritems.erf, and the UTI file is [dd9ea6bb335cfa89].uti.

Make a backup copy of the da2_mtx_war directory.

To decrypt the ERF file, you need the decrypt key and a decrypter. There is a tool called “slurpda2.exe”, usually distributed with the deda2drm decrypter package. I will not explain how to obtain this package, but it’s pretty easily found. If you legitimately own this game, slurpda2 will give you a list of all the decrypt keys, one for each DLC package. Use the right key (have to do trial-and-error here, as it doesn’t tell you which key belongs to which package) and deda2drm by cd’ing to the deda2drm directory and executing from the command prompt:

deda2drm.exe DA2_MTX_WAR [key]

The output will tell you whether this succeeded. If so, this will decrypt the ERF.

Extracting the UTI

A tool like PyGFF will open the decrypted ERF file. Look for the dd9ea6bb335cfa89 hash key and extract that UTI. For reference, the UTI should contain an item with the tag war100im_warrior_1h_sword. Save the UTI file as war100im_warrior_1h_sword.uti (though technically you can name the filename anything you want, as long as it ends in uti). As you can see by the name, even the original designers thought it should be a sword and not an axe :p

Editing the UTI

Using a tool like TlkEdit, open the UTI file. Change the ModelVariation field to 123 (which is the Edge of Song and Glory, as listed in 2da.rim’s item_variations.gda file). I believe 132 is Glandivalis / Vigilance, if you prefer that look. Change the Icon field to “ico_1h_sword_5.dds”. Save the UTI file.

Here you can choose to restore the original encrypted copy of da2_mtx_war or not. If so, this reverses the decryption so it doesn’t confuse DA2 when you open it again. Alternatively, you can edit the manifest.xml file in the decrypted copy of da2_mtx_war to tell it to skip the authorization requirement by setting RequiresAuthorization=0. I left mine unencrypted, but I think this gets into legal gray territory, and the UTI mod should work even with the original encrypted version.

Drop the final modded uti file into packages/core/override in either the DA2 install directory or the user directory in Documents/Bioware (create the override directory if needed).

Result

Edge of Night sword The Edge of Night sword in action

Isn’t that pretty? Wielding a sword, as a hero should. One of these days I’ll mod the TLK file to change the occurrences of “axe” to sword in its Codex file, and this change would be perfect.

I have a copy of the modded UTI file, but I think distributing it might violate the EULA. Oh well. If you can’t figure out the instructions, leave a comment here (with your email address) and I’ll try to help.