restoring Safari preferences from backup files in OS X Mavericks

Recently I had the misfortune of having to restore some Safari settings from backup, on OS X 10.9 Mavericks. I have done this many times before on older OS X versions, without incident — simply pull the various preference files such as com.apple.Safari.plist from backup and replace the damaged/unwanted ones. Takes all of 2 minutes, and years ago, I had already wrote a shell script to do exactly that.

It turns out that after Mavericks, Safari is incredibly resistant to conventional methods of preference file backup and restoration. Considering that preferences in OS X have always been stored in XML-based preference lists, you would think (as in previous operating systems from 10.0 to 10.7) taking the relevant preference files and replacing the unwanted new ones in ~/Library/Preferences would be enough. But no, an incredible amount of effort is now required for a simple task:

  1. unhiding the Library directory, because clearly we’re all Windows babies who cannot be trusted to see where application preferences are stored
  2. replacing the actual preference files, scattered around the system in ~/Library/Safari, ~/Library/Preferences, ~/Library/Caches, and hopefully remember to have turned off iCloud, or otherwise see changes being clobbered by iCloud sync (or, worse yet, having experimental changes reflected all across a network of Mac and iDevices)
  3. resetting the preference cache daemon, cfprefsd so that preference changes can be reflected in a running system. This where a lot of people get stuck in general, judging by Google results; when they replace preference files and find that their changes aren’t being reflected, it leads them into a wild goose chase for “hidden preference files” for Safari, when the answer lies in a simply yet utterly non-obvious background daemon.
  4. restoring the list of installed extensions — which, incredibly, is NOT stored in the deceptively named com.apple.Safari.Extensions.plist, but in the login.keychain.

Context

Had some issues where certain websites were behaving differently under private browsing mode than normal browsing mode. I deduced there was some kind of corrupted stored state, whether it was a cookie or localstorage issue. Had the brilliant idea of setting Safari preferences aside, thus resetting Safari to factory state, and then divide-and-conquer by restoring parts of the settings until the problem recurs. I’ve done this many times before.

First I turned off iCloud sync, having been bitten by sync propagation of experimental changes in the past. This is pretty important if you don’t want to blow up Safari bookmarks (at the very least) across all Apple-manufactured, iCloud-compatible devices. I then removed ~/Library/com.apple.Safari.plist, ~/Library/com.apple.Safari.Extensions.plist, ~/Library/Safari, and ~/Library/Caches/Safari, ~/Library/Cookies. After resetting to confirm some issues have disappeared, I moved some files from backup to original locations. Imagine my surprise when nothing became restored, and all my Safari extensions (installed from the extension store or custom-built by me) disappeared.

Process

Increasingly desperate, I started to trace filesystem accesses using fs_usage. It showed nothing out of the ordinary. 30 mins of reviewing useless forum posts later, I pieced together a multi-stage solution. It turns out there were two separate obstacles.

Preference caching

Presumably to save energy, OS X Mavericks caches application preferences (in RAM?) using a daemon called cfprefsd. Instead of applications pulling their preferences from XML files on disk at launch, it requests this from the daemon instead. The defaults command has been modified to operate with this daemon, so if you had been working with preferences from the command line (as I have been), the changes have been transparent.

However if the preference files are changed or edited directly, this change is not propagated to the preference cache daemon. When the app is opened again, the cached version takes precedence, and is re-written out to disk, clobbering the restored versions.

This does not mean there are hidden Safari preferences somewhere that you haven’t found, though you might think this at first. When Safari is reset manually from the filesystem, or if the plist files are edited, cfprefsd must be reset as well.

There exists a cfprefsd daemon for every logged in user, running under that user’s privileges, as well as a root-owned one. Safari preferences are stored under the user domain, so the user-specific daemon is the one that needs a reset when files change. Can also quit the process from Activity Viewer, or killall cfprefsd. A login-logout cycle would also reset the user-specific daemon.

Extension list caching

Having done this reset with the backup files in place, most preferences will be restored on next launch, *except* the list of extensions you had installed previously. That will remain empty. Even though all the extensions and their settings have been restored to ~/Library/Safari.

For a long time I traced ~/Library/com.apple.Safari.Extensions.plist, and wondered why it wasn’t being read.

An Apple discussion forum post (shockingly enough) gave a vital clue. There exists an “application password” in the login keychain titled “Safari Extensions List”. Whether it is merely a cryptographic key, or the actual list of extensions, is unknown, but that is the critical preference to restore extensions. Having reset extensions by moving them away, this preference is apparently emptied out. The entire login keychain, being an encrypted document, has to be restored to a corresponding previous version to restore access to previous extensions. Without this, all extensions would have to be reinstalled manually (and get a new copy of the extension file stored into ~/Library/Safari/Extensions, instead of the previous version being reused).

Discussion

Given recent focus on energy consumption, I can understand preference caching. However, it’s not that hard to track filesystem changes (the Time Machine/Spotlight APIs explicitly do this!) and reload appropriate preferences when they are changed on disk. It would show respect for power users and developers who might need to interact with the preferences system in a more convenient way.

But stuffing extension lists in an obscure corner of a password keychain? What sense does that make? Are my list of extensions (not actual extension data or settings, mind you — those are in plaintext on the disk for anyone to copy and look at) such privileged information that it has to reside alongside my login password? Why can’t you just read the list of extensions, oh I don’t know, from the list of extension files installed into the Extensions directory? Wouldn’t that be a lot more reasonable?

download servers and the Web Developer extension


One nifty thing that the Web Developer extension for Firefox can do is live HTML editing, on the currently loaded page. The feature is activated via the toolbar, under the Miscellaneous button, via the item “Edit HTML”. It pops up a text box containing the current page’s HTML. Edit to your heart’s content, and hit the Apply button (the blue-with-green-arrow button beside the search box – not exactly the most obvious icon for “Apply”, but that’s a UI critique for another time). The current loaded web page will reflect your changes.

Obviously it will stick around only until you load some other page, since you are not actually editing the web page on the remote server itself. So how is this useful?

So MegaShares is one of those sketchy file hosting and download sites, akin to Rapidshare, MegaUpload, etc. I had a problem here where some some files are served from storage machine #21, which was apparently overloaded or just not configured right – it would start the download fine, but the download gradually stalls before completion. Wacky. There appears to be some redundancy, however, and I wondered if I can grab the file from another server by changing the machine number in the URL.

Unfortunately, as most of these places do, they prohibit direct access to a file without going through their UI, so I can’t just take the download URL, change the machine number, and pop it in the browser. I assumed they were checking referrers, so I spoofed the REFERER field. No luck.

You can see where I’m going with this. Enter the Web Developer extension. Used the Edit HTML feature to change the URL on the page directly, and clicked through the changed link. Success! Their script accepts this action, and the download starts from machine #3. Whatever referrer check or scripting magic that they use to enforce their no-direct-access policy is still intact, since the rest of the page has not changed.

Obviously this is a specific example – if there were no storage redundancy at MegaShare, this trick would have been useless. Nevertheless, it demonstrates the power of live-editing a loaded page, in your browser. Extensions like Greasemonkey is the pinnacle of this kind of editing, but for a once-off adjustment, one doesn’t really need the power of a full scripting environment like that.

Not quite a real Read/Write Web, but an interesting trick to keep in mind.

Windows IE 6 ignores text/plain mimetype

A fairly border-case scenario that probably rarely comes up, but appears to be another gotcha. So apparently IE 6 for Windows, on occasion, decides it knows better than the web server what format a file is. Instead of using the mimetype supplied by the web server, as all good browsers tend to do, IE performs some heuristics on the file and overrides the mimetype with its own guess. The type text/plain is one such stupid circumstance.

Annoyingly, IE will insist on downloading plaintext files in some cases, instead of rendering it in browser. This usually occurs if a script is attempting to generate a “text/plain” document on the fly, but can also happen under other circumstances if the IE hard-coded heuristics comes up with a different result than the server-proclaimed mimetype.

A client-side workaround for text/plain is possible. You’d need to edit the Windows Registry (oh joy). In HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Internet Settings, add the DWORD key IsTextPlainHonored and set value to 0x1. This will make IE behave correctly for text/plain mimetypes. This solution comes per the MS Knowledgebase article, “Text/Plain” Content-Type Header Field Is Ignored. There are also some further explanations on how mimetypes are resolved in the MSDN article, on mimetype detection in IE.

Unfortunately, this is not a solution if this behavior comes up in a web-based tool for external use – as every client machine registry will have to be thus modified. This change may also carry security implications (actually, I’m completely guessing here, because I don’t quite see why the IE team decided to “not honor” mimetypes for text/plain…).

The context:

A PHP script in a project I maintain pulls a text file from a remote location, and then prints it to the browser as Content-type: text/plain. A hack to be sure, but simple enough to get the job done. This works out fine in Firefox, etc, but not in Windows IE. IE insists that this is a PHP script file that must be downloaded. Of course, once downloaded, you can fire up Notepad and see that it’s bloody plaintext. Firefox et al will render it in browser as expected.

In this case, the script was only used for internal testing, so I switched all the test machines to honor plaintext mimetypes. A longer term workaround would probably involve porting the output to XML instead.

Fixing FilePlanet’s stupidity on the Mac


Lately I haven’t been able to download files from fileplanet.com via my Mac. It’s inane, because downloads apparently requires an ActiveX control. I’m appalled at the utter stupidity of excluding all non-Windows platform users from your download service, just to set up a download queue. Can’t you put up a Flash control instead? Just as shiny and unusable, but actually compatible with other operating systems.

It gets better. The good news – the designer had some foresight to set up a fallback mechanism, to use plain old HTML queue. The bad news – it simply presents you with a 403 Forbidden when clicked.

As it turns out, I found a post that contained a possible solution. Actually, that post is a bit unnecessarily complicated. Apparently, they’re blocking all browsers without a Windows user agent. On the fallback solution that was supposed to work for all platforms. Argh.

Until Fileplanet addresses this issue (which could be tomorrow. or never), the simplest solution (that worked for me) was to switch my user agent in Firefox (via the aforementioned and highly recommended User Agent Switcher) to a Windows browser (try the default Opera XP user agent). Then, click on the fallback queuing link, and it should kick you into the download page.

Note that the User Agent option in Safari (at least, v2.0.x) Debug menu will not work straight up. Believe me, as a primarily Safari user, I’ve tried hard to make this work. Because Fileplanet pops up a new window when download is selected, and the Debug menu setting only sets it for the active window. As the download window is a pop-up, you do not get a chance to intervene and change the User Agent code before FilePlanet denies you access to it. So for now, Firefox + User Agent Switcher is the solution. If you have a browser (or Safari, in the future) which allows the fake user agent setting to persist across windows spawned from the initial window, that browser would work too.

UPDATE 3/27/2009:
Feedback in the comments section reports that this is still a problem for many users. Appalling. It’s been 2 years.