While working on a project on OS X 10.10.4 Yosemite, some innocuous Python 2.7.6 code using the multiprocessing module (via the concurrent.futures module) was crashing when run from the IPython interpreter, but works just fine when executed via shell directly.
Issue
Some nice simple test code. It pulls some data from two Web servers, via the requests module, concurrently using a multi-process worker pool.
from multiprocessing import Pool
import concurrent.futures
import requests
TEST_URI = ["http://example.com", "http://yimingliu.com"]
def http_get(uri):
return requests.get(uri)
def test():
p = Pool(5)
print(p.map(http_get, TEST_URI))
def futures_test():
with concurrent.futures.ProcessPoolExecutor() as executor:
print [response for response in executor.map(http_get, TEST_URI)]
if __name__ == '__main__':
test()
When run as $ python mp_test.py, everything works beautifully. When test() or futures_test() was called from an IPython shell however, a segmentation fault occurs:
Exception Type: EXC_BAD_ACCESS (SIGSEGV) Exception Codes: KERN_INVALID_ADDRESS at 0x0000000000000110 ... Application Specific Information: crashed on child side of fork pre-exec Thread 0 Crashed:: Dispatch queue: com.apple.main-thread 0 libdispatch.dylib 0x00007fff9772c16f _dispatch_async_f_slow + 395 1 com.apple.CoreFoundation 0x00007fff92e91541 _CFPrefsWithDaemonConnection + 305 2 com.apple.CoreFoundation 0x00007fff92e60ac6 __80-[CFPrefsSearchListSource generationCountFromListOfSources:count:allowFetching:]_block_invoke + 150 3 com.apple.CoreFoundation 0x00007fff92e609d2 -[CFPrefsSearchListSource generationCountFromListOfSources:count:allowFetching:] + 258 4 com.apple.CoreFoundation 0x00007fff92d1cea5 -[CFPrefsSearchListSource alreadylocked_copyDictionary] + 133 5 com.apple.CoreFoundation 0x00007fff92d17dba -[CFPrefsSearchListSource alreadylocked_copyValueForKey:] + 42 6 com.apple.CoreFoundation 0x00007fff92e9211c ___CFPreferencesCopyAppValueWithContainer_block_invoke + 60 7 com.apple.CoreFoundation 0x00007fff92e5f979 +[CFPrefsSearchListSource withSearchListForIdentifier:container:perform:] + 729 8 com.apple.CoreFoundation 0x00007fff92e92097 _CFPreferencesCopyAppValueWithContainer + 183 9 com.apple.SystemConfiguration 0x00007fff96be8db4 SCDynamicStoreCopyProxiesWithOptions + 153 10 _scproxy.so 0x000000010e07f915 0x10e07f000 + 2325 11 org.python.python 0x000000010d89a9ed PyEval_EvalFrameEx + 14935 12 org.python.python 0x000000010d89d60e 0x10d813000 + 566798 ...
The same code works just fine under Ubuntu Linux, from IPython or otherwise.
Workaround
It turns out that there is some kind of edge case bug going on between IPython, multiprocessing, and OS X. Namely, a segfault seems to trigger from within OS X’s Grand Central Dispatch architecture, when multiprocessing is forking a new process to access the network, under IPython.
Note these lines in the crash log:
0 libdispatch.dylib 0x00007fff9772c16f _dispatch_async_f_slow + 395 ... 10 _scproxy.so 0x000000010e07f915 0x10e07f000 + 2325
Something in _scproxy.so (a part of Python’s lib-dynload) is not happy with libdispatch.
_scproxy.so calls out to OS X’s SystemConfiguration framework for network proxy information. The quickest workaround for this bug, then, is to disable this proxy check. IPython seems to respect the customary no_proxy rule to skip proxy checks (at least IPython 4 does), so:
$ env no_proxy='*' ipython
and then
$ env no_proxy='*' ipython Python 2.7.6 (default, Sep 9 2014, 15:04:36) Type "copyright", "credits" or "license" for more information. IPython 4.0.0-b1 -- An enhanced Interactive Python. ? -> Introduction and overview of IPython's features. %quickref -> Quick reference. help -> Python's own help system. object? -> Details about 'object', use 'object??' for extra details. In [1]: import mp_test In [2]: mp_test.test() [<Response [200]>, <Response [200]>]
works just fine. No segfault.
Of course if you actually need this proxy functionality, you would be out of luck.