Python multiprocessing code crashes on OS X under IPython

While working on a project on OS X 10.10.4 Yosemite, some innocuous Python 2.7.6 code using the multiprocessing module (via the concurrent.futures module) was crashing when run from the IPython interpreter, but works just fine when executed via shell directly.

Issue

Some nice simple test code. It pulls some data from two Web servers, via the requests module, concurrently using a multi-process worker pool.

from multiprocessing import Pool
import concurrent.futures
import requests

TEST_URI = ["http://example.com", "http://yimingliu.com"]

def http_get(uri):
    return requests.get(uri)

def test():
    p = Pool(5)
    print(p.map(http_get, TEST_URI))

def futures_test():
    with concurrent.futures.ProcessPoolExecutor() as executor:
        print [response for response in executor.map(http_get, TEST_URI)]

if __name__ == '__main__':
    test()

When run as $ python mp_test.py, everything works beautifully. When test() or futures_test() was called from an IPython shell however, a segmentation fault occurs:

Exception Type:        EXC_BAD_ACCESS (SIGSEGV)
Exception Codes:       KERN_INVALID_ADDRESS at 0x0000000000000110

...

Application Specific Information:
crashed on child side of fork pre-exec

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0   libdispatch.dylib             	0x00007fff9772c16f _dispatch_async_f_slow + 395
1   com.apple.CoreFoundation      	0x00007fff92e91541 _CFPrefsWithDaemonConnection + 305
2   com.apple.CoreFoundation      	0x00007fff92e60ac6 __80-[CFPrefsSearchListSource generationCountFromListOfSources:count:allowFetching:]_block_invoke + 150
3   com.apple.CoreFoundation      	0x00007fff92e609d2 -[CFPrefsSearchListSource generationCountFromListOfSources:count:allowFetching:] + 258
4   com.apple.CoreFoundation      	0x00007fff92d1cea5 -[CFPrefsSearchListSource alreadylocked_copyDictionary] + 133
5   com.apple.CoreFoundation      	0x00007fff92d17dba -[CFPrefsSearchListSource alreadylocked_copyValueForKey:] + 42
6   com.apple.CoreFoundation      	0x00007fff92e9211c ___CFPreferencesCopyAppValueWithContainer_block_invoke + 60
7   com.apple.CoreFoundation      	0x00007fff92e5f979 +[CFPrefsSearchListSource withSearchListForIdentifier:container:perform:] + 729
8   com.apple.CoreFoundation      	0x00007fff92e92097 _CFPreferencesCopyAppValueWithContainer + 183
9   com.apple.SystemConfiguration 	0x00007fff96be8db4 SCDynamicStoreCopyProxiesWithOptions + 153
10  _scproxy.so                   	0x000000010e07f915 0x10e07f000 + 2325
11  org.python.python             	0x000000010d89a9ed PyEval_EvalFrameEx + 14935
12  org.python.python             	0x000000010d89d60e 0x10d813000 + 566798
...

The same code works just fine under Ubuntu Linux, from IPython or otherwise.

Workaround

It turns out that there is some kind of edge case bug going on between IPython, multiprocessing, and OS X. Namely, a segfault seems to trigger from within OS X’s Grand Central Dispatch architecture, when multiprocessing is forking a new process to access the network, under IPython.

Note these lines in the crash log:

0   libdispatch.dylib             	0x00007fff9772c16f _dispatch_async_f_slow + 395
...
10  _scproxy.so                   	0x000000010e07f915 0x10e07f000 + 2325

Something in _scproxy.so (a part of Python’s lib-dynload) is not happy with libdispatch.

_scproxy.so calls out to OS X’s SystemConfiguration framework for network proxy information. The quickest workaround for this bug, then, is to disable this proxy check. IPython seems to respect the customary no_proxy rule to skip proxy checks (at least IPython 4 does), so:

$ env no_proxy='*' ipython

and then

$ env no_proxy='*' ipython
Python 2.7.6 (default, Sep  9 2014, 15:04:36) 
Type "copyright", "credits" or "license" for more information.

IPython 4.0.0-b1 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: import mp_test

In [2]: mp_test.test()
[<Response [200]>, <Response [200]>]

works just fine. No segfault.

Of course if you actually need this proxy functionality, you would be out of luck.