Creating Safari App Extensions and porting old Safari extensions

With the release of Safari 12 and macOS Mojave, Safari has taken a separate path from every other browser in terms of extension support, by creating a new “Safari app extension” model and deprecating old Javascript-based extensions. Like Google Chrome, Mozilla Firefox, and Microsoft Edge, old style extensions have been deactivated (with a scary and very disingenuous warning message that legacy extensions “unsafe”). However, while all other browsers have moved to support the growing WebExtensions standard, making browser extensions easy to generate across browser platforms. Safari is using its own weird and different extension model for its new Safari App Extension system where:
– the “global” part of the extension must be written in native code (Objective-C or Swift, instead of Javascript)
– the injected code remains in Javascript
– all embedded into a mostly useless macOS container app that magically installs an extension (sometimes) into the browser when first run.
– the container app (with its extension embedded) is distributed on the Mac App Store, as opposed to the old Safari Extension Gallery.

Leaving aside my personal opinion about this weird hybrid extension model and its effects on the already dwindling Safari extension ecosystem, this document constitutes my notes on porting a traditional JS-based Safari Extension to a Safari App Extension.

Disclaimer: This note pertains to Safari 12 and Xcode 10.1 only. Future versions may drastically change the extension model, and I may or may not update this document as things change. Also, I am merely an enthusiast, not a professional macOS app developer. I may very well be wrong in many, many things. Corrections and constructive suggestions welcome in the comments.

The Basics

How to create a Safari app extension

In the good old days, you went into Safari’s Develop menu and hit Show Extension Builder. In the brave new world of Safari App Extensions, extensions are made in Xcode, Apple’s all-purpose IDE.

In Xcode 10.0, making an app extension was a really confusing process. You had to create a generic Cocoa macOS app, then File -> New -> Target and pick Safari Extension from the list of app extension targets.

In Xcode 10.1 Apple has made it slightly less troublesome. In New Project, you can select “Safari Extension App” directly as one of the project templates.

This does the exact same thing as the manual process in Xcode 10.0. It creates 2 targets:

  1. The containing app — the containing app doesn’t have much functionality unless you are also making a macOS app alongside it. It effectively is just a delivery vehicle for the extension — when the app is first run, the extension is placed in Safari’s list of extensions, for the user to activate.
  2. The extension — this target contains most of the actual code that will run in Safari’s context.

What are all these boilerplate files

The templates for the containing app are largely irrelevant for stand-alone Safari Extensions. For me, I am building Safari extensions that stand on their own. If you are building an app that happens to offer a Safari extension as a side effect, that purpose is out of scope for this document, but you may find something useful in here for building the extension itself.

The meat of a Safari App Extension lives in the extension target. I am using an Objective-C project in these examples, but the Swift project looks very similar.

  • SafariExtensionHandler — this class is the delegate that gets called by Safari to do things. The main functionality of your extension are invoked from here. In porting old extensions, what used to live in global.js now needs to be in this file. Obviously, such code need to be ported from Javscript into Objective-C or Swift. This SafariExtensionHandler object gets created and destroyed frequently during Safari’s lifetime, so don’t expect to put global initializers here without wrapping them in dispatch_once or making a custom singleton class.
  • SafariExtensionViewController — this class is perhaps more appropriately called SafariExtensionToolbarPopoverController. This is the ViewController class that is invoked if your extension’s toolbar button (singular — you are only allowed 1 toolbar button, ever) is clicked. As of Safari 12, this class only used to show a popover from the toolbar button. The template code creates a singleton class for this controller.
  • SafariExtensionViewController.xib — the Interface Builder file for the toolbar popover. If you’ve done any Cocoa or iOS programming at all, this is effectively the same thing as any nib file. In short, you can lay out your user interface for the toolbar popover in this xib file, and load it during runtime (as opposed to having to programmatically create each UI element).
  • Info.plist — This file should be familiar to anyone who made legacy Safari extensions. It is an Apple plist file (obviously) that largely contains the same kind of extension metadata, like what domains the extension is allowed to access, what content script should be injected. The template project’s Info.plist does not contain all the possible keys, like SFSafariStylesheet (injected stylesheet) or SFSafariContextMenu (context menus), etc. There is also no convenient list of all possible keys, though you can eventually find all of them by reading through Apple’s documentation on Safari App Extension property list keys.
  • script.js — this is the content script (what used to be called the injected script in legacy Safari extensions), a Javascript file that is injected into every allowed domain. Javascript code that used to be in injected scripts need to be migrated here. There no longer seems to be a distinction between start scripts and end scripts — if you need the injected script to be an end script, the template code demonstrates that you can wrap it in a callback from the DOMContentLoaded event.
  • ToolbarItemIcon.pdf — the name is self explanatory. This is the icon that shows up in the toolbar if you have a toolbar item.
  • .entitlements – Safari App Extensions need to be sandboxed. If you try to turn off the sandbox, the extension will no longer appear in Safari. The entitlements file allows you to grant specific sandbox permissions for the extension. I haven’t played with this much, but I assume this works like any other sandboxed app. It also allows you to specify an App Group. This is also a relatively advanced macOS / iOS dev topic that is out of scope for this document, but in short, if you put your extension and the containing app in the same App Group, they can share UserDefaults preferences and other system-provided storage space. Otherwise, the containing app and the extension will maintain their own separate preference and storage space.

How to debug

When you click “Run” in the Xcode project for an Safari Extension app, it builds both the extension and the containing app, and then runs the containing app. This should install the extension into Safari, but by default the extension is disabled until the user manually activates it.

During development, you can instead run the App Extension target itself, by switching the scheme from the XCode toolbar.

If you run the App Extension itself, a menu pops up asking for which browser to run it with (which can be Safari Technology Preview as well). It will then temporarily install the extension for the duration of the debug session. If run this way, you can also see any NSLog debugging print statements in XCode’s console, as well as use the built-in debugger to step through native code.

Architectural difference: global.js -> SafariExtensionHandler

Legacy Safari extensions were pure Javascript code, split into a global.html page that ran in the background in its own context, and injected script files that were injected into the context of every accessible web page. Only global.html/global.js could interact with the larger Safari extension environment, and only the injected scripts could interact with the currently loaded web pages. These two separate worlds communicated with each other using message dispatches – global.html/global.js would push data to the injected scripts through safari.application.activeBrowserWindow.activeTab.page.dispatchMessage, while listening on safari.application.addEventListener for any messages coming back. Injected scripts would push data to the global page using the tab proxy safari.self.tab.dispatchMessage, and listen to messages from the global page using safari.self.addEventListener.

Safari App Extensions have a similar architecture. However, it has torn out the global.js/global.html portion of this, and replaced it with native code implemented in SafariExtensionHandler.

  • From SafariExtensionHandler:
    • Listen to messages from content scripts by implementing the method in SafariExtensionHandler: (void)messageReceivedWithName:(NSString *)messageName fromPage:(SFSafariPage *)page userInfo:(NSDictionary *)userInfo. The template project provides a skeleton for you. It ends up looking something like:
      - (void)messageReceivedWithName:(NSString *)messageName fromPage:(SFSafariPage *)page userInfo:(NSDictionary *)userInfo
      {
          // This method will be called when a content script provided by your extension calls safari.extension.dispatchMessage("message").
          [page getPagePropertiesWithCompletionHandler:^(SFSafariPageProperties *properties)
           {
               // NSLog(@"The extension received a message (%@) from a script injected into (%@) with userInfo (%@)", messageName, properties.url, userInfo);
               if ([messageName isEqualToString:@"command-from-content-script"]) {
                   // do things
                   NSDictionary *results = @{@"result" : @"Hello world", @"date" : [NSDate date]};
                   [page dispatchMessageToScriptWithName:@"command-done" userInfo:results];
               }
           }
           ];
      }
      
    • Dispatch messages to content scripts using: [page dispatchMessageToScriptWithName:@"messageNameHere" userInfo:....]where page is a SFSafariPage object that is either provided during a delegate callback, or retrievable using the SFSafariApplication class method + (void)getActiveWindowWithCompletionHandler:(void (^)(SFSafariWindow * _Nullable activeWindow))completionHandler;
      - (void)contextMenuItemSelectedWithCommand:(NSString *)command
                                          inPage:(SFSafariPage *)page userInfo:(NSDictionary *)userInfo
      {
          if ([command isEqualToString:@"contextmenu_1"]) {
              NSLog(@"contextmenu_1 invoked");
              [page dispatchMessageToScriptWithName:@"contextmenu_1_invoked" userInfo:@{@"result" : @"context_menu_1 works!"}];
          }
      }
      
  • From content script:
    • Listen to messages from the extension using safari.self.addEventListener, same as before.
    • Dispatch messages the global extension using: safari.extension.dispatchMessage

Injecting content scripts and stylesheets

In legacy Safari extensions, the extension builder had a nice UI to add scripts and stylesheets for injection. For app extensions, plist XML will have to be specified manually (or using the built-in plist editor in Xcode)

        <key>SFSafariContentScript</key>
        <array>
	    <dict>
		 <key>Script</key>
		 <string>script.js</string>
	    </dict>
        </array>
        <key>SFSafariStyleSheet</key>
        <array>
            <dict>
                <key>Style Sheet</key>
                <string>injected.css</string>
            </dict>
        </array>

By using Open As -> Source Code, you can edit the XML directly if messing around with plist editor is not your thing. However, plists are a specific format, with arrays and dict representations that are annoying to write raw XML with, if not using a plist-aware editor.

Context menus

Context menus have only changed a little from the legacy Safari extension architecture. You still specify context menus in the Info.plist, and they still interact largely the same way as in legacy extensions.

As of this post’s date, Apple’s porting guide states that “In a Safari App extension, there’s no validation of contextual menu items before the menu is displayed.” This is actually wrong. The API does provide for validation of contextual menu items before the menu is displayed.

In SafariExtensionHandler, implementing the validateContextMenuItemWithCommand method will let the extension decide whether to show or hide a context menu item, and whether to change its menu text depending on conditions.

- (void)validateContextMenuItemWithCommand:(NSString *)command
                                    inPage:(SFSafariPage *)page
                                  userInfo:(NSDictionary *)userInfo
                         validationHandler:(void (^)(BOOL shouldHide, NSString *text))validationHandler
{
    if ([command isEqualToString:@"Test1"]) {
        validationHandler(NO, @"Test1 nondefault text");
    } else if ([command isEqualToString:@"Test2"]) {
        validationHandler(YES, nil);
    }
}

As it suggests, validationHandler returns NO for shouldHide if the menu item should be displayed, or YES if it should be hidden. The text parameter allows for the menu text to be changed dynamically during validation.

What if we need to change which menu to show depending on if something exists on the web page or not? In the injected content script, listening to the “contextmenu” event and implementing a handler will allow you to populate userInfo dictionary with data, and validateContextMenuItemWithCommand will be called with that data.



function handleContextMenu(event)
{
    var data = {};
    // do some work and populate data
    safari.extension.setContextMenuEventUserInfo(event, data);

}

document.addEventListener("DOMContentLoaded", function(event) {
    document.addEventListener("contextmenu", handleContextMenu, false);

});

Handling the actual context command, once invoked by the user, requires implementing the method - (void)contextMenuItemSelectedWithCommand:(NSString *)command
inPage:(SFSafariPage *)page userInfo:(NSDictionary *)userInfo

The code is again very similar to how validation is done.

Toolbar: Buttons and Popovers

Safari App Extensions appear to be limited to a single toolbar button. Safari App Extensions cannot provide full toolbars (called “extension bars”, previously), like legacy extensions could.

The toolbar button can be configured to trigger one of two actions: Command which invokes a command immediately, or Popover which creates the popover view as provided by SafariExtensionViewController. The “menu” action that legacy Safari Extensions had, which simply created a menu of commands to choose from, has been removed. See the guide on Toolbar Item Keys.

To respond to a toolbar button click, implement - (void)toolbarItemClickedInWindow:(SFSafariWindow *)window in SafariExtensionHandler.

To show a popover on click, implement - (SFSafariExtensionViewController *)popoverViewController. This should be part of the boilerplate code generated by XCode.

Popovers are no longer HTML pages. They are now native AppKit views, created the same way as any macOS app’s views; legacy popovers would have to be ported to native code, or hacked using a WebView. The UI controls available to the popover are native macOS controls (NSButton, NSTextField, etc.), and must be created and styled using Interface Builder (or programmatically). These UI elements can be wired up to the SafariExtensionViewController singleton, which can provide the delegate methods to respond as the controls are changed.

References

Safari App Extension Guide

Safari App Extension Property List Keys

Converting a Legacy Safari Extension to a Safari App Extension

IKEv2 VPN on Ubuntu – IKE authentication credentials are unacceptable

I used these straightforward strongswan IKEv2 VPN setup instructions to set up a IKEv2 VPN on my Ubuntu server.

However, instead of self-generating my own certificate authority and having to deal with manually trusting this untrusted CA on every device I have to use VPN on, I decided that since I had letsencrypt in standalone mode set up on my server already (and the vpn subdomain properly covered under the certificate), I might as well use that certificate instead. I’m sure I’m committing half a dozen security sins, but it saves me time that I used to spend googling “how to trust certificate authority on [device name]”.

Setting up letsencrypt certificates with strongswan-based VPN is out of scope for this post, but in short, the difference is minimal. Instead of using the fake CA’s .pem, just symlink the letsencrypt certificate and key to /etc/ipsec.d/certs and /etc/ipsec.d/private. Remember to add a post_hook to /etc/letsencrypt/renewal/ to reboot strongswan after certificate renewal.

This worked great on macOS High Sierra and iOS 11. However, Windows 10 (Fall Creators) refused to connect to the VPN, stating that “IKE authentication credentials are unacceptable”.

The top google results for this were highly misleading for this particular context. It’s not about subjectAltName or Server Authentication flags or whatever.

It turns out that unlike macOS or iOS, Windows 10 wasn’t processing the full set of root and intermediate CA certificates, even though I symlinked the fullchain.pem to /etc/ipsec.d/certs. After throwing a symlink from /etc/letsencrypt/live/[domain]/chain.pem to /etc/ipsec.d/cacerts, Windows finally relented and let me connect to the VPN.

YMMV, because judging by the Google results, there are a lot of sources of error that yield the exact same, mildly useless “IKE authentication credentials are unacceptable” error message. However, this particular error is non-obvious, and worth looking at if you’re running into the same issue.

Safari Technology Preview crossed out in macOS 10.13 High Sierra

If you’ve just upgraded to macOS 10.13 High Sierra from 10.12 and are wondering why Safari Technology Preview is crossed out with a white 🚫 prohibited/cross mark and unable to be launched, turns out there is a separate Safari Tech Preview version for High Sierra. Confusingly, the App Store / updater does not automatically upgrade to this version when you install High Sierra, so the old version left over in /Applications is just…sitting there, broken.

Go back to the Apple Developer downloads page for Safari, grab the “Safari Technology Preview for macOS High Sierra” installer, and install it over the existing Safari Technology Preview. The app should then launch.

Resolving endless Apple Pay add card loop after Time Machine restore

If you recently had your Macbook Pro (Touch Bar) repaired (possibly with a logic board replacement), and restored from Time Machine backup, you might find yourself unable to use Apple Pay on Mac OS 10.12.5. The system will report “Apple Pay is already configured on this disk for another Mac” and ask you to “Reset Apple Pay and Add Card”. If you try to do so by authorizing it using fingerprint or password, it will immediately drop you back to the original “Apple Pay is already configured on this disk for another Mac” prompt, going back into this cycle ad infinitum.

The issue is that there is an Apple Pay cache at /private/var/db/applepay/ on the system that has been invalidated, but it seems to be unable to delete this cache properly. It will keep trying to refresh this cached data, and fail to do so.

There is a workaround for this. Obligatory warning: THIS IS MESSING WITH SYSTEM FILES. DO NOT EXECUTE ANY OF THIS IF YOU ARE NOT SURE WHAT YOU ARE DOING AND DO NOT HAVE A BACKUP OF YOUR DATA. TYPING THE COMMANDS WRONG MAY CAUSE SERIOUS ISSUES WITH YOUR MAC.

I used this process myself to good success under 10.12.5, on my Macbook Pro. YMMV if you decide to try this on any other version of Mac OS.

To fix this endless loop, you need to first clear out all the files (but not the folders) inside /private/var/db/applepay/. Open Terminal.app and enter the following commands:

# get a root shell
sudo -s
# move the stale files away
mv /private/var/db/applepay/Library/Caches/* ~/.Trash/
mv /private/var/db/applepay/Library/Preferences/* ~/.Trash/
# kill the related cache servers
pkill seld; pkill nfcd;

Then:
– Wait a few seconds for the relevant servers to boot themselves up again. Then, go back to System Preferences, hit Add Card…

– It will fail the first time with a mysterious error. That’s fine. Hit Add Card again…

– On this second try, it will say “Apple Pay is already configured on this disk for another Mac”.

– When you hit “Reset Apple Pay and Add Card” for the final time, it will actually break past the loop, and you will get to re-enter your Apple Pay card information without further issue.

It’s a relatively easy fix, so I imagine the next OS version will have addressed this problem in a more elegant way.

check progress of photoanalysisd

If you’ve just installed Mac OS Sierra and now see photoanalysisd sucking 100% to 200% CPU power, this process is doing some kind of face detection + object / image recognition / indexing on your Photos library.

photoanalysisd progress

photoanalysisd progress

Open Photos.app, select People on the left sidebar, and see how far it’s gotten in this task. If you have a very large Photos library, this might take a while.

As a bonus, while Photos.app is open, photoanalysisd is suspended, allowing your laptop fans a bit of a rest.

It’s all for a good cause. After indexing, you’d be able to type “ocean” in the search box and get all your photos with the ocean in it, without having to tediously tag all your photos yourself. Magical, huh? ( Not really. If this is doing what I think it’s doing, well, it’s the kind of problem computer vision researchers have been tackling – and solving – for years already.).

As an aside, for consumer-grade apps like Photos, arguments have been made to do computationally intensive image analysis in The Cloud(tm) instead of on client machines. The tradeoff is fairly obvious. In exchange for the privacy benefit of Apple not uploading color histograms of all your photos to its own cloud servers (something I’m sure Google wouldn’t bat an eyelash at doing), you’ll have to pay the cost of doing this analysis yourself, with your own measly consumer/mobile-grade CPUs (which isn’t ideal if you want to get work done at the same time that this analysis is going on). Overall, the user experience probably could have been handled better, especially given the extensive public beta period that Sierra received. The current opaque process violates some core UX principles: giving users (at least the feeling of) control, and giving the user clear reasons to trust in the importance of the task being performed. Apple should have known that eating 100% CPU while people are actively working with their machines is immediately noticeable, and should have 1) let the user know that image analysis is about to happen, and 2) given them the option of choosing when and how much CPU to devote to this task.

Setting up OpenSSH Server on Windows 10 Anniversary Update

UPDATE 2017-12-22

Microsoft now ships a real SSH server (and client!) with Windows Fall Creators Update, no Cygwin or Linux subsystem required. Finally.

This post remains for historical context only.

Microsoft has finally landed its anniversary update for Windows 10. Among all the random useless features, is an actual Ubuntu Linux subsystem within Windows, with the ridiculously silly name “Bash on Ubuntu on Windows”. Goodbye Cygwin?

One of the first things I wanted to try was to setup the SSH server, so I can remote-login from my real box. Getting one up was actually fairly easy, if you can deal with a few problems with weird, weird red-herring error messages.

The problem

To recap, installing the OpenSSH server is as easy as popping open a bash shell (I assume you figured out how to get the Linux subsystem installed already and popped open Ubuntu )

sudo apt-get install openssh-server

If you try to now start the server with

sudo service ssh start

It’ll respond with

initctl: Unable to connect to Upstart: Failed to connect to socket /com/ubuntu/upstart: Connection refused

but subsequently

Starting OpenBSD Secure Shell server sshd [OK]

When you now try to connect in from a remote host, there are two outcomes:

  1. There is no SSH server on port 22.
  2. There is an SSH server on port 22, but it responds only to your Windows password and not your Ubuntu Linux user password. If login is successful, it drops you into a DOS prompt instead of a bash prompt. Trying to run bash within the prompt generates a response:
    "Error: 0x80070005".

Diagnosis

There are multiple underlying problems here.

  1. First, it’s important to note that there exists a separate Windows SSH server (separate from the OpenSSH server from the Ubuntu subsystem installation) now on port 22. If you telnet (yes, telnet) into port 22, and the host greeting is SSH-2.0-MS_1.100, you’ve run into the Microsoft SSH implementation (hence the MS part in the greeting).

    I’m not sure what this server is supposed to do (and, in fact, starting SSH servers on standard ports without explicitly telling the user seems like a potential security problem to me, in addition to being kind of jerk-ish). I do know it is launched when you reboot with Developer Mode on. You know, Developer Mode, the mode you have to turn on to run Bash on Ubuntu on Windows. This server is occupying port 22, so you cannot launch another SSH server to listen on this port. This is the problem that is causing the silent failure of sshd to start up, not a broken Upstart or any of that nonsense the error message being displayed is referring to.

  2. Apparently the Ubuntu compatibility layer on Windows does not implement chroot or one of the related system calls needed for OpenSSH privilege separation at the time of this post.
  3. If you launched bash from DOS cmd.exe, instead of the Bash on Ubuntu on Windows shortcut, /mnt/c/Users/[Windows Username] (the directory you start with in DOS) is not your Linux home directory. .ssh config files left in this directory will not configure your OpenSSH server.

Symptoms & Solutions

  1. To resolve the problem of conflicting SSH servers, set your OpenSSH server on a different port : one other than 22. This can be done by editing /etc/ssh/sshd_config and changing the Port configuration. Be sure to open this new port in your Windows Firewall for inbound connections, as this firewall configuration will not be automatically done for you. I set mine to 60022.
  2. To enable login, also change UsePrivilegeSeparation in the same config file to No. Failure to do so will cause the server to respond with Connection reset by nnn.nnn.nnn.nnn
  3. To enable public key auth and set other configurations, the Linux subsystem home directory is, as in normal Ubuntu, /home/<yourname>.

Having done this, now you can start or restart in the bash shell

sudo service ssh restart

And connect from remote host with your Ubuntu user credentials and/or public key:
$ ssh -p 60022 10.0.1.208
...
Welcome to Ubuntu 14.04.5 LTS (GNU/Linux 3.4.0+ x86_64)

(And if you telnet, the host message should be something like
SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.7
)

Not paying OnTheHub to redownload Windows ISOs

If you’re looking to re-download Microsoft Windows 10 Education edition ISOs and have a valid key from the university licensing program already, grab the education edition ISO images directly from Microsoft instead of paying OnTheHub/Kivuto protection money.

Context

UC Berkeley, like many other universities, has a volume licensing deal with Microsoft for its operating system products. In particular, it offers free downloads of Windows 10 Education edition to all current students.

The irritating thing about this is that they offer this deal through a shady vendor called OnTheHub or Kivuto. Instead of allowing the ability to re-download the ISO image, it holds you hostage if you want to access the image file again after 60 days.

When you first download, it offers you a discounted “extended access guarantee”:

To ensure that your download and/or key(s) remain accessible to you, you can extend your coverage to 24 months with the Extended Access Guarantee for $4.95. With this service, Kivuto will back up your download and/or key(s) on their servers, allowing you to access this information at any time under the “Your Account” section of the WebStore.

If you declined to take advantage of this oh-so-generous offer, and find yourself needing the ISO again after 60 days:

Access Guarantee Retrieval (60 days)
Purchase this service if you wish to recover your download(s) and/or key(s) after access has expired.
You will gain another 60 days of access to any expired product in your order.Learn more
$11.95

In my case, I backed up the wrong ISO — the generic install ISO, instead of the education edition ISO. The product key Kivuto issues you is for the Education edition, so the generic ISO you can get online won’t let you install with the key. It’ll report an error of “The product key entered does not match any of the Windows images” at install time.

So when it came time to reinstall, suddenly I’m faced with the OnTheHub protection racket.

Solution

For Windows 10, at least, you don’t have to pay the protection money. Education edition ISOs are available directly from Microsoft after product key verification. The download page is rather well-hidden, under the “more options” link from the software download page.

Unsurprisingly, OnTheHub makes no mention of this download source.

I wonder how much money OnTheHub makes off people who didn’t realize there was a free, official source for Windows 10 ISO images. Bandwidth costs money, sure, but you are a *education software download vendor* in 2015, with a target market of underpaid students and faculty. Nickel-and-diming poor student users for software downloads, in this era of cloud-driven computing, is an obsolete and despicable business practice.

Python multiprocessing code crashes on OS X under IPython

While working on a project on OS X 10.10.4 Yosemite, some innocuous Python 2.7.6 code using the multiprocessing module (via the concurrent.futures module) was crashing when run from the IPython interpreter, but works just fine when executed via shell directly.

Issue

Some nice simple test code. It pulls some data from two Web servers, via the requests module, concurrently using a multi-process worker pool.

from multiprocessing import Pool
import concurrent.futures
import requests

TEST_URI = ["http://example.com", "http://yimingliu.com"]

def http_get(uri):
    return requests.get(uri)

def test():
    p = Pool(5)
    print(p.map(http_get, TEST_URI))

def futures_test():
    with concurrent.futures.ProcessPoolExecutor() as executor:
        print [response for response in executor.map(http_get, TEST_URI)]

if __name__ == '__main__':
    test()

When run as $ python mp_test.py, everything works beautifully. When test() or futures_test() was called from an IPython shell however, a segmentation fault occurs:

Exception Type:        EXC_BAD_ACCESS (SIGSEGV)
Exception Codes:       KERN_INVALID_ADDRESS at 0x0000000000000110

...

Application Specific Information:
crashed on child side of fork pre-exec

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0   libdispatch.dylib             	0x00007fff9772c16f _dispatch_async_f_slow + 395
1   com.apple.CoreFoundation      	0x00007fff92e91541 _CFPrefsWithDaemonConnection + 305
2   com.apple.CoreFoundation      	0x00007fff92e60ac6 __80-[CFPrefsSearchListSource generationCountFromListOfSources:count:allowFetching:]_block_invoke + 150
3   com.apple.CoreFoundation      	0x00007fff92e609d2 -[CFPrefsSearchListSource generationCountFromListOfSources:count:allowFetching:] + 258
4   com.apple.CoreFoundation      	0x00007fff92d1cea5 -[CFPrefsSearchListSource alreadylocked_copyDictionary] + 133
5   com.apple.CoreFoundation      	0x00007fff92d17dba -[CFPrefsSearchListSource alreadylocked_copyValueForKey:] + 42
6   com.apple.CoreFoundation      	0x00007fff92e9211c ___CFPreferencesCopyAppValueWithContainer_block_invoke + 60
7   com.apple.CoreFoundation      	0x00007fff92e5f979 +[CFPrefsSearchListSource withSearchListForIdentifier:container:perform:] + 729
8   com.apple.CoreFoundation      	0x00007fff92e92097 _CFPreferencesCopyAppValueWithContainer + 183
9   com.apple.SystemConfiguration 	0x00007fff96be8db4 SCDynamicStoreCopyProxiesWithOptions + 153
10  _scproxy.so                   	0x000000010e07f915 0x10e07f000 + 2325
11  org.python.python             	0x000000010d89a9ed PyEval_EvalFrameEx + 14935
12  org.python.python             	0x000000010d89d60e 0x10d813000 + 566798
...

The same code works just fine under Ubuntu Linux, from IPython or otherwise.

Workaround

It turns out that there is some kind of edge case bug going on between IPython, multiprocessing, and OS X. Namely, a segfault seems to trigger from within OS X’s Grand Central Dispatch architecture, when multiprocessing is forking a new process to access the network, under IPython.

Note these lines in the crash log:

0   libdispatch.dylib             	0x00007fff9772c16f _dispatch_async_f_slow + 395
...
10  _scproxy.so                   	0x000000010e07f915 0x10e07f000 + 2325

Something in _scproxy.so (a part of Python’s lib-dynload) is not happy with libdispatch.

_scproxy.so calls out to OS X’s SystemConfiguration framework for network proxy information. The quickest workaround for this bug, then, is to disable this proxy check. IPython seems to respect the customary no_proxy rule to skip proxy checks (at least IPython 4 does), so:

$ env no_proxy='*' ipython

and then

$ env no_proxy='*' ipython
Python 2.7.6 (default, Sep  9 2014, 15:04:36) 
Type "copyright", "credits" or "license" for more information.

IPython 4.0.0-b1 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: import mp_test

In [2]: mp_test.test()
[<Response [200]>, <Response [200]>]

works just fine. No segfault.

Of course if you actually need this proxy functionality, you would be out of luck.

Setting up a Gmail POP3 account for Mail.app on 10.10.3 beta

The last beta of OS X has apparently removed the option to create POP3-type email accounts for Gmail in Mail.app. When you type in a Gmail account, IMAP is setup automatically. Presumably because OAuth doesn’t work all that well with it.

There (currently) is a workaround.

  • Turn off/unplug your Internet connection. This will force Mail to not verify the settings that you are giving it.
  • Open Mail.app -> File Menu -> Preferences -> Accounts tab
  • Hit the + button to Add Other Mail Account
  • In the email address field in the following box, enter something that IS NOT @gmail.com. Enter some random email.
  • Mail.app will now inform you that manual setup is required. That’s exactly what we wanted in the first place, thank you very much. In the next screen, you will get to pick IMAP vs POP. Pick the POP tab and enter the correct Gmail POP server information and Gmail outbound SMTP server in the following screens, for your account. Make sure to use SSL and port 995 if appropriate
  • Use the application-specific password in the password field if you have two-factor auth setup.
  • Once setup is complete and you are thrown back into the preference pane, change the “Email Address” field back to the correct email address @gmail.com
  • Under “Advanced”, untick “automatically detect and maintain account settings”
  • Quit Mail.app. Turn Internet connection back on
  • When you re-open mail.app, the POP account should be operational

Hooray for we remaining holdouts, still using obsolete and dubiously secure technology. Because when I want mail on my local machine, I mean I want mail on my local machine, The Cloud(tm) be damned.

Note they might still change this depending on feedback received. I would advise giving such feedback if you care at all.

Mail.app stuck at fetching mail on OS X 10.10.3

If Mail.app is stuck at “Fetching Mail….” with Gmail POP accounts after upgrading to OS X 10.10.3’s beta, be aware that Apple added OAuth support to Mail.app. Which is nice, except in the current beta, they don’t warn you to go set up OAuth permissions for every account. Mail.app will choke on fetching mail, without letting you know the actual problem. This is possibly POP3-specific, as I didn’t notice IMAP having this issue.

In the connection log:

[kCFStreamSocketSecurityLevelTLSv1_0] -- host:pop.gmail.com -- port:995 -- socket:0xxxxxx -- thread:0xxxxx
-ERR invalid SASL argument ......

The solution here is to simply set up Google OAuth via System Preferences -> Internet Accounts. It should automatically prompt you for OAuth permission to your Google Account when you click on the affected account in the prefpane. Once that is done, Mail.app will fetch Gmail via POP3 again.

I’ve also unticked “automatically detect and maintain account settings” under Advanced, in Mail.app’s accounts panel, though this might just be superstition rather than actual prevention of issues. I’m just old school and uncomfortable with the thought of “automagic” happening in my mail server settings.

If you’ve already given up and swapped to IMAP, but would like to come back to the Dark Side of email retrieval technology, see also:
How to setup a Gmail POP3 account from scratch on 10.10.3 beta

Selfsolved reference:
#127: Mail.app stuck at fetching mail on new OS X

I really need to rebuild Selfsolved. Someday, when I’m not trying to write.