Subversion – The Sarth Repository

Subversion 1.6.2 runtime error on network access on OS X 10.5

Posted on May 27, 2009May 27, 2009 by yiming

A new SelfSolved solution is up for perusal. The problem I tried to solve:

After compiling Subversion 1.6.2 from source on OS X 10.5 Leopard, the compilation is apparently successful, but svn dies when it tries to connect to the network for the first time. Crash log reports that symbols are missing from libneon.dylib.

Crash report from shell:
dyld: lazy symbol binding failed: Symbol not found: _ne_set_connect_timeout Referenced from: /usr/local/lib/libsvn_ra_neon-1.0.dylib Expected in: dynamic lookup
dyld: Symbol not found: _ne_set_connect_timeout Referenced from: /usr/local/lib/libsvn_ra_neon-1.0.dylib Expected in: dynamic lookup

Check out the places that I googled and my final solution writeup … at SelfSolved #49: Subversion 1.6.2 explodes on first network access.

The problem is very similar to a previous compilation issue I solved for PHP. In essence, the -L library search path passed to GCC at compilation time has /usr/lib in front of everything else. This means whatever library path you might have given to it at configure time, it’ll always look for the library in /usr/lib first, picking up the old system libneon in the process. Since the bad libneon dynamically linked, the problem doesn’t manifest itself until runtime — and only at runtime with network access involved.

As with the PHP issue, change the very first -L/usr/lib to -L/usr/local/lib (or wherever your newer libneon is located), and it’ll link correctly.

Out of curiosity, I checked MacPorts first. The MacPorts solution of disabling libneon version checking is odd — it also works, but I dunno if it’s linking to the right thing or not.

SSH, Subversion through SOCKS proxy on Mac OS X

Posted on March 5, 2009April 4, 2012 by yiming

UPDATE Apr 2, 2012
Due to the complete lack of updates for tsocks, I recommend the use of proxychains over tsocks. It accomplishes the same thing but works out of the box.

One persistent problem that I run into is that I need to access certain network resources through a SOCKS proxy server. This is all well and good if they are web resources — Safari, Firefox, etc. support SOCKS proxies quite well. However, I also need, for example, SSH and Subversion access to some resources. SOCKS support is woefully inadequate or nonexistent in these tools.

In the case of SSH, even if you google for this, you’ll run through thousands of examples of using ssh as a SOCKS server, but not through one as a SOCKS client. There are some convoluted solutions, but none of them I can use directly on an OS X 10.5 machine.

TSocks: the solution…if it were that easy

Now, tsocks is a nifty little tool to transparently divert network calls through a SOCKS 4 or SOCKS 5 proxy. This allows even non-SOCKS-aware applications to function through a SOCKS server.

Unfortunately it is very old, unmaintained code (1.8 beta 5 was released in 2002). It doesn’t compile cleanly on OS X due to this, nor will it compile under GCC 4.x. Further, it won’t work out of the box either if you do manage to compile it. The problem is that it relies on the Linux-only LD_PRELOAD functionality to use a shared library to hijack network system calls. This mechanism is called DYLD_INSERT_LIBRARIES on OS X and only works if DYLD_FORCE_FLAT_NAMESPACES is active.

Getting a working tsocks: MacPorts

There is an easy way to get tsocks. MacPorts ships a ported tsocks package. If you use MacPorts, sudo port install tsocks should do it.

Unfortunately on several machines I don’t use MacPorts, and don’t want to pull down an entire third-party package manager with its own library tree on each of these boxes. So I have do to this the hard way.

Getting a working tsocks: rolling my own

First to notice is that there are two tsocks distributions. One is the original tsocks 1.8b5, last updated in the first half of this decade. To make it work, follow the instructions provided by Marc Abramowitz in 2006. Note that his patch is actually located at his new domain address instead of the old, linked one.

The MacPorts distribution, on the other hand, is based on R. Garcia’s patched tsocks distribution, incorporating some modernization and new features by the Tor team. This distribution is numbered 1.8.x, with the last being 1.8.4. Unfortunately it is also no longer maintained, as the Tor devs forked this into a custom version to use with the Tor network only. Unfortunate, but for now, it still compiles, and works a bit better than the 2002 original.

To roll your own tsocks via source out of the MacPorts distribution, you will want the patches from the MacPorts repository. An outline of the compilation procedure:

Download tsocks 1.8.4 from the author’s page
Download all the patches from the MacPorts repository
Concatenate all of the patches together:
```
cat patch-* > tsocks.osx.patch
```
Put the concatenated tsocks.osx.patch file into the tsocks source directory. Apply the patches:
```
patch -p0 < tsocks.osx.patch
```
Regenerate the configure script:
```
autoreconf
```

Configure the package:

./configure --prefix=/usr/local --bindir=/usr/local/bin --mandir=/usr/local/man --sysconfdir=/etc --libdir=/usr/local/lib

Install the library and binaries:
```
sudo make install
```

Install the conf file:

sudo cp ./tsocks.conf.complex.example /etc/tsocks.conf

Edit the conf file. Make sure that if you’re not using tor, that you write in the conf file
```
tordns_enable = false
```

Configuring tsocks

The complex configuration file example should have explained all of the features to be set. For my configuration:

Some important settings:

local – this setting, in the format of IP/netmask can be repeated several times, each time to exclude a set of IPs from being diverted to the SOCKS server. For obvious reasons, your SOCKS server will have to exist in one of these excluded IP ranges – otherwise you will never even reach your proxy server.
server and server_port – these should point to the IP address and port of your SOCKS server
server_type – tsocks defaults to SOCKS4 mode. You may wish to set it to 5 for SOCKS5 usage.
tordns_enable – this needs to be set as false if you don’t use Tor.

Using tsocks

Once this is set up, simply prefixing the network command you want to run with tsocks will force a diversion through the proxy connection. For example:

tsocks ssh example.com

The same can be applied to Subversion.

tsocks svn update

will force the svn client to act through the proxy set in tsocks.conf.

SOCKS on localhost

Note that SOCKS services on 127.0.0.1 has a minor gotcha. Sometimes, you are able to SSH into a remote machine, and use that connection as your SOCKS server. This is described in my post about using SSH as a pseudo-VPN, which describes the -D switch. My use case here is that once you do this, all further local SSH connections to other machines should be diverted through the first SSH. For example, I’d like to do:

my-machine$ ssh -D 40000 gateway.example.com # establish a SOCKS server on localhost:40000 to the gateway host

and then:

my-machine$ ssh lan-1.example.com # access the protected lan-1 machine through the SOCKS, which will see me as gateway.example.com

This is very doable in the tsocks setup if you set tsocks.conf:


server = 127.0.0.1/255.255.255.255
server_port = 40000

and then:


my-machine$ ssh -D 40000 gateway.example.com
my-machine$ tsocks ssh lan-1.example.com

This is the gotcha: make sure the netmask is set correctly to 255.255.255.255. Otherwise tsocks will die with a cryptic:

IP (127.0.0.1) & SUBNET (0.0.0.0) != IP on line 22 in configuration file, ignored

It is apparently fairly sensitive about the subnet mask setup to conform to exact standards.

With this tsocks setup, you won’t have to create special VPNs to lock a LAN machine behind a gateway. As long as you can SSH into the gateway machine from your local machine, you can access the resources behind it with any application on your local machine via tsocks. Nifty, huh?

APR and 32-bit/64-bit universal binary compilation

Posted on October 14, 2008January 26, 2009 by yiming

When compiling APR, the Apache Portable Runtime 1.3.3 (as a part of Subversion 1.5.3 as I am doing here, or not) on OS X 10.5 Leopard, you may encounter the following error at compile time.

/bin/sh /tmp/subversion-1.5.3/apr/libtool --silent --mode=compile gcc-4.2 -Os -arch i386 -arch x86_64 -DHAVE_CONFIG_H -DDARWIN -DSIGPROCMASK_SETS_THREAD_MASK -no-cpp-precomp -I./include -I/tmp/subversion-1.5.3/apr/include/arch/unix -I./include/arch/unix -I/tmp/subversion-1.5.3/apr/include/arch/unix -I/tmp/subversion-1.5.3/apr/include -o strings/apr_snprintf.lo -c strings/apr_snprintf.c && touch strings/apr_snprintf.lo strings/apr_snprintf.c: In function ‘conv_os_thread_t’: strings/apr_snprintf.c:500: error: duplicate case value strings/apr_snprintf.c:498: error: previously used here strings/apr_snprintf.c: In function ‘conv_os_thread_t_hex’: strings/apr_snprintf.c:671: error: duplicate case value strings/apr_snprintf.c:669: error: previously used here

This will most likely happen when you are configured to build a dual 32-bit / 64-bit universal binary, whether it be ppc / ppc64, or i386 / x86_64, or any permutation thereof. This ticket over at MacPorts documents a particular instance of this problem, with no apparent solution.

The symptom is easy to explain. Somehow, two case labels in the relevant switch statement in strings/apr_snprintf.c:500:

switch(sizeof(u.tid)) {
    case sizeof(apr_int32_t):
        return conv_10(u.u32, TRUE, &is_negative, buf_end, len);
    case sizeof(apr_int64_t):
        return conv_10_quad(u.u64, TRUE, &is_negative, buf_end, len);
    default:
        /* not implemented; stick 0 in the buffer */
        return conv_10(0, TRUE, &is_negative, buf_end, len);
    }

have evaluated to the same value. In particular, it believes that sizeof(apr_int32_t) and sizeof(apr_int64_t) are the same value. As we all know in C, you cannot have two identical case labels in the same switch statement. However, the root of the problem is a bit more subtle.

In $SRCDIR/include/apr.h, you’re likely to see this fragment of code.

typedef  long       apr_int64_t;
typedef  unsigned long  apr_uint64_t;

Notice that it has typdef’ed apr_int64_t as a long and apr_uint64_t as unsigned long. This is because at configure time, the script detected that long values are 64-bit on this system, so it assigned the apache 64-bit types to longs. However, this only holds true for half of the compilation – because you are building a universal binary for a 32-bit architecture as well. Remember that in 32-bit GCC on OS X, longs are 32-bit rather than 64-bit. Your run-of-the-mill autoconf script, done by a non-OS X programmer, isn’t going to be able to detect this subtlety – if the 64-bit part worked, it’ll keep thinking longs are 64-bit, end of story – and happily generate the incorrect typedef expressions. When you apply sizeof to these types in apr_snprintf.c, both evaluate to 4 bytes under 32-bit compilation, thus blowing up the compile run.

To truly fix the root of the problem requires rewriting the autoconf script to detect Mac OS X and its universal binary building, which can potentially throw quadruple architectures at the same compilation script. However, a quick hack to make this particular problem go away is to change apr.h such that:

typedef  long long       apr_int64_t;
typedef  unsigned long long apr_uint64_t;

Now that we ensure in either 32-bit or 64-bit compilation, apr_int64_t and apr_uint64_t are always typedef’ed to appropriate, guaranteed 64-bit types. The compilation of APR (and Subversion) will proceed normally.

Note that long long is not an standard C type. As a GCC extension, this fix is a kludge. A kludge that works (for me), though.

UPDATE:
There may also be an issue with sizeof definitions that may cause the library to crash. In particular, there may be occurrences of

#define APR_SIZEOF_VOIDP 8

that were generated by configure. To fix this, you will need to remove the define and have the compiler check for 64-bit at compile-time:

#ifndef __LP64__
    #define APR_SIZEOF_VOIDP 4
#else
    #define APR_SIZEOF_VOIDP 8
#endif

In general, any predefined sizeofs need to be changed. I am not sure why the APR developers do hard-coded defines like this, given that the point of having sizeof() calls is to avoid such issues.

Subversion and undefined symbols

Posted on September 29, 2008June 5, 2024 by yiming

This is fast turning out to be a blog about compiling open source software. Maybe I should change the title.

You may get this error while trying to compile Subversion 1.5.2:
ld: Undefined symbols: _svn_fs_txn_root_base_revision referenced from libsvn expected to be defined in libsvn _svn_fs_change_txn_props referenced from libsvn expected to be defined in libsvn _svn_fs_get_mergeinfo referenced from libsvn expected to be defined in libsvn _svn_fs_recover referenced from libsvn expected to be defined in libsvn _svn_fs_upgrade referenced from libsvn expected to be defined in libsvn _svn_fs_node_origin_rev referenced from libsvn expected to be defined in libsvn /usr/bin/libtool: internal link edit command failed make: *** [subversion/libsvn_ra_local/libsvn_ra_local-1.la] Error 1

Turns out there may be a bug in the libtool script shipped with the source tarball, under certain circumstances with OS X 10.4, that causes it to fail to link. Ah, libtool, you make my life so much harder sometimes.

If you copy /usr/bin/glibtool over $SRCDIR/libtool (that is, into the directory for the subversion source code, replacing the one that is placed there by the package itself), the package should compile with no further complaints. make test also shows success on all tests, so this seems a satisfactory solution.

UPDATE 1/31/2009
I’ve been informed in the comments that MacPorts may rely on source compilation from tarball, and thus have this issue. If you’re having this issue with the MacPorts SVN package, please check out the advice in the comment section from farkinga, who has additional notes.