So while compiling PHP 5.2.8 on OS X 10.5, you might run into something like:
Undefined symbols for architecture i386:
"_xmlTextReaderSchemaValidate", referenced from:
_zim_xmlreader_setSchema in php_xmlreader.o
"_xmlTextReaderSetup", referenced from:
_zim_xmlreader_XML in php_xmlreader.o
ld: symbol(s) not found for architecture i386
collect2: ld returned 1 exit status
Undefined symbols for architecture x86_64:
"_xmlTextReaderSchemaValidate", referenced from:
_zim_xmlreader_setSchema in php_xmlreader.o
"_xmlTextReaderSetup", referenced from:
_zim_xmlreader_XML in php_xmlreader.o
ld: symbol(s) not found for architecture x86_64
This doesn’t only happen with libxml. If you’ve installed any extra updated libraries, like iconv or tidy or any library that has significant symbol changes between versions, it’ll die in similar ways. The MacPorts folks have encounted similar issues in ticket 15891, but WONTFIX‘ed the issue. Apparently the PHP devs are also punting on the problem.
The immediate cause is that you have multiple versions of some shared libraries. For example, in the case above, I have two libxml versions — one in /usr/lib, and another in /usr/local/lib. This is because I do not want to overwrite the Apple-provided libxml version, but still needed new features provided in later libxml versions. The arrangement works fine in every other software compile except this one, so I investigated further.
The root of the problem
Despite the developers’ airy dismissal of the issue, the underlying problem is indeed that the Makefile generated by PHP at configure time is slightly broken. In Makefile and Makefile.global, you’re going to see this line:
libs/libphp$(PHP_MAJOR_VERSION).bundle: $(PHP_GLOBAL_OBJS) $(PHP_SAPI_OBJS)
$(CC) $(MH_BUNDLE_FLAGS) $(CFLAGS_CLEAN) $(EXTRA_CFLAGS) $(LDFLAGS) $(EXTRA_LDFLAGS) $(PHP_GLOBAL_OBJS:.lo=.o) $(PHP_SAPI_OBJS:.lo=.o) $(PHP_FRAMEWORKS) $(EXTRA_LIBS) $(ZEND_EXTRA_LIBS) -o $@ && cp $@ libs/libphp$(PHP_MAJOR_VERSION).so
where $MH_BUNDLE_FLAGS is usually defined as something like
MH_BUNDLE_FLAGS = -bundle -bundle_loader /usr/sbin/httpd -L/usr/lib \
-L/usr/lib -laprutil-1 -lsqlite3 -lexpat -liconv -L/usr/lib -lapr-1 -lpthreadThe problem is that this hardcodes the search paths for linking shared libraries. GCC searches for shared libraries to link in the order of the provided -L paths. In this case, MH_BUNDLE_FLAGS is expanded immediately after $CC — so the load order is:
- /usr/lib
- /usr/lib (these are redundant, and so will probably be collapsed into one path)
- …every other custom library path you specify
Now you see the issue. No matter what your library paths are set to, the PHP compilation system will insist that whatever shared libraries in /usr/lib take precedence. Therefore, even if you specified that another version (say, libxml.dylib in /usr/local/lib) should be used instead, the invocation to link against -lxml2 will search in /usr/lib first. And since it finds the old version, which may be missing a number of symbols, the compilation blows up right there.
Evidence
And indeed, if you look at the (rather long and massive) compilation/link command right before it fails, you’ll see:
gcc -bundle -bundle_loader /usr/sbin/httpd -L/usr/lib -L/usr/lib \
-laprutil-1 -lsqlite3 -lexpat -liconv -L/usr/lib -lapr-1 -lpthread -O2 -I/usr/include -DZTS \
-arch i386 -arch x86_64 -L/usr/local/lib ... emphasis mine, where /usr/local/lib might be /opt/lib or whatever custom path you provided to configure.
Solutions
The trivial solution is to manually invoke that last line of compilation, but swapping the -L load paths.
gcc -bundle -bundle_loader /usr/sbin/httpd -L/usr/local/lib -L/usr/lib \
-L/usr/lib -laprutil-1 -lsqlite3 -lexpat -liconv -L/usr/lib -lapr-1 -lpthread -O2 -I/usr/include -DZTS \
-arch i386 -arch x86_64 ... This is easy to do and takes just a second.
Another possible solution is to patch the Makefile, such that MH_BUNDLE_FLAGS comes later in the compilation line:
libs/libphp$(PHP_MAJOR_VERSION).bundle: $(PHP_GLOBAL_OBJS) $(PHP_SAPI_OBJS)
$(CC) $(CFLAGS_CLEAN) $(EXTRA_CFLAGS) $(LDFLAGS) $(EXTRA_LDFLAGS) $(PHP_GLOBAL_OBJS:.lo=.o) $(PHP_SAPI_OBJS:.lo=.o) $(PHP_FRAMEWORKS) $(EXTRA_LIBS) $(ZEND_EXTRA_LIBS) $(MH_BUNDLE_FLAGS) -o $@ && cp $@ libs/libphp$(PHP_MAJOR_VERSION).so
This will force your library paths to be searched before /usr/lib, thus resolving the link problem.
update 7/18/09
An anonymous reader mentions that you could also specify the right libxml by full path, instead of letting it use -lxml. Basically, in the last compilation line, you would remove any mentions of -lxml and replace that with the full path to your library e.g. /usr/local/lib/libxml.dylib. In fact, this is probably the way that has the least possible side-effects, since you aren’t changing the search order for any other libraries.
Discussion
This is not the first time that PHP core developers have refused to fix a compilation issue that is arguably preventable through actual testing under different installation scenarios. This is an “edgier” edge case than the tidy.h issue, but still should be fairly noticeable for a substantial number of people.
The “You should only have one library installed” argument is, to be honest, unnecessarily arrogant (sadly, not as a rare a problem as you’d like in some open source development projects ). I understand that it’s an open source project, and no self-respecting software engineer likes to use time on project plumbing / build systems rather than work on the product. However, on OS X, due to the lack of official Apple package management systems, no one should be overwriting system default libraries — down that way lies insanity, especially at every system or security update. PHP’s build system is obviously broken any time there is a substantial difference between user-installed libraries and system libraries. This bad behavior is especially egregious, because the configure command allows you specify your own library path — misleading users into thinking that the path they specified would be obeyed at compile time. If you only intend for the system library to be used and no other, perhaps the configure script should auto-detect this on OS X and disable that command-line option. Basic user interface design should apply even to command-line interfaces.
Note that changing link ordering may have some unforseen consequences, since the devs obviously never tested this path. For example, you should make sure the dynamic libraries are loaded in the right order at runtime. On OS X, the load path is typically hard-coded into the dylib, so usually there won’t be a problem — but there may be edge cases. Test your build (and any PHP extensions you built) before using it in production!
Thanx so much for this yiming, had some lack of knowledge concerning makefiles ;
well, still have, but it goes better for php compiling now;
Thankz a lot mate… really thanks!
Wonderful, thanks!
You seem to have plenty of time writing arrogant blog posts. How about being productive and provide patches instead? Or is that too hard?
Ah, because it’s quite productive to be attacking your users anonymously when they reveal a potential shortcoming? I investigated this issue simply because its occurrence cost me personal productivity — which I could have been using on my day job or my hobby projects. The fact that others have reported the bug, and it has been declared invalid, suggests that even if I did put down time, effort, and expense to learn the intricacies of a build system, it would probably not be accepted anyway.
This entire blog has consisted of “workarounds for problems”, which I hope will be useful to others. However, at the end of the day, I neither enjoy nor get paid to fix compilation bugs in these packages, nor does my company depend upon any of these packages I’ve had trouble with. I have no particular incentive to learn the codebase to a project every time I run into a build problem; as you said, I’d rather be productive :p
Great Post! and you are analysis is spot on!
Thanks for the in depth review of this issue. In retrospect, I should have looked at what the command line was obviously telling me, but since the configuration script allowed me to specify my library path I didn’t think that was being overlooked. I kept second guessing my install steps and not the php config script!
Has this been submitted as a patch to the php.net folks?
Yeah, I had the same inclination. Initially I was convinced that I’d somehow mis-compiled libxml and spent 15 minutes recompiling libxml 🙂
My fix is just a quick way to get past the problem, so I didn’t submit a patch. To really fix the underlying problem, the build system should try to link the library by full path when you specify your custom library path (gcc … /usr/local/lib/libxml.dylib) rather than using directory search ( as in gcc … -lxml ) This way, it will be sure to get the right library that you specifically told it to use, rather than always searching and finding the one living in the first -L path.
The point seems moot though — from the PHP bugzilla, they didn’t seem terribly interested in fixing the problem anyway (“you should only have one library!”)
Oh my goodness, thank you _so_ much; I’d been driven right round the bend by this. Your solution (and your analysis of the arrogance on the part of the
PHP developers) is, as has already been said, spot on.
Firstly, Thank you for the extremely clear explanation of what’s going on. I figure if I found my way to this post someone else with my problem will too so I wanted to comment how this helped me. I’m running mac os 10.5.8 with apache 2.2 and trying to install php 5.3.1 with curl. It was giving me undefined symbols errors because it couldn’t find iconv. I googled and found my way here. I tried this solution but it wasn’t working. So I tried decompressing my php source download again and running ./configure… from a fresh directory and it worked. I know some of you may be laughing at me or thinking that there was some other fancy command line thingy I could do to refresh the directory but if it gave me trouble it may give someone else trouble so I’m posting it here!
Further reading on this topic I found here:
http://bugs.php.net/bug.php?id=48195
http://jspr.tndy.me/2009/07/php-5-3-iconv-osx-symbols-missing-_libiconv/
and those two links at the top of this post
I found that compiling my own libxml2 into a non system directory then using the –with-libxml-dir= compile flag fixed my problems.
Good to know. The devs must have fixed it in more recent versions of the compilation system. Furthermore, on OS X 10.6, the system libxml version shipped is recent enough that the problem doesn’t crop up again — but you may still be linking whatever is in /usr/lib. When in doubt, use otool -L to check which library you linked against.
At the time this post was written, –with-libxml-dir had no effect, because -L/usr/lib was hardcoded to appear in front of anything you specified at link time. As long as there was a incompatible version of libxml in /usr/lib, you won’t be linking against the custom-compiled version.
Towering heaps of thanks for this info. My first issues were with iconv. Then came openssl. And then libxml. I tried dangerous things like renaming the library and header files, but it just became a mess. Modifying the makefile as recommended fixed everything. I now have PostgreSQL support on my MacBookPro, something which should be standard. I agree that PHP is at fault for this, but I wouldn’t expect them to fix it, since it isn’t really a ‘bug’. More like a feature that inconveniences many Mac users. Anyway, thanks again for your solution, I appreciate you taking the time so I don’t have to.
Kuddo’s on the information sharing! It was becoming a headache problem for me. It is a shame this info is not available / better findable on php.net
For reference: I was encountering problems while compiling php 5.3.2 on snow leopard (10.6.2). Ended up with pointing configure to MacPorts version of inconv (–with-iconv=/opt/local/) and flipping the order of the gcc line by putting MH_BUNDLE_FLAGS at the end as you instructed.
Undefined symbols:
“_iconv_close”, referenced from:
__php_iconv_strlen in iconv.o
Many thanks again!
Thanks for going to the trouble of posting this, and for explaining it so clearly. I was just about to give up on this one! Confirm it works for locally built iconv, PHP 5.3.2, OSX 10.6.3. Cheers
Thanks a lot.
I went thru a lot of work and re-compiling in order to solve a similar issue I have with iconv.
It looks like the folks at php prefer to highlight your blog than to fix the makefile generation, since I just downloaded the latest version and my macosx is up-to-date.
Unfortunately, because I think in the end it will cost less to fix the makefile generation than to answer to bug submissions (as they diligently do).