Tag: OS X

transcode compile error – preprocessor macros vs attributes

Posted on by yiming

Had a problem compiling transcode 1.0.4 on my OS X Tiger (PPC).

In file included from /usr/include/math.h:26,
from pvnglobals.h:26,
from pvn.h:15,
from import_pvn.c:39:
/usr/include/architecture/ppc/math.h:179: error: parse error before '__attribute__'
/usr/include/architecture/ppc/math.h:179: error: parse error before 'inline'
/usr/include/architecture/ppc/math.h:180: error: parse error before '__attribute__'
/usr/include/architecture/ppc/math.h:180: error: parse error before 'inline'

… and so forth. The whole math.h business is a red herring; or rather, it’s a symptom, not the cause.

The issue appears to stem from a preprocessor macro problem in src/transcode.h. Around line 85 or so, find the block

#ifndef always_inline
#if defined(__GNUC__) && (__GNUC__ > 3 || __GNUC__ == 3 && __GNUC_MINOR__ > 0)
# define always_inline __attribute__((always_inline)) inline
#else
# define always_inline inline
#endif
#endif

A rather odd and potentially problematic thing to do, because now in math.h:83, we note:

#define __MATH_H_ALWAYS_INLINE__ __attribute__ ((always_inline))

which later gets used in math.h:179, 180, and so forth.

Now, because of the first #define, it seems we’re likely to cause __MATH_H_ALWAYS_INLINE__ to become something like __attribute__((__attribute__ ((always_inline)) inline)). Unlikely to compile cleanly.

One possible solution is to swap around the header orders so that math.h is included before transcode.h clobbers the always_inline definition.

Another workaround: comment out in the inner #if and preserve only the else-branch in transcode.h:

#ifndef always_inline
# define always_inline inline
#endif

It fixes the compilation problem, presumably because it no longer causes the always_inline definition to conflict with the one in math.h. Unfortunately, it also changes the meaning of __attribute__ ((always_inline)) in math.h. The first solution is preferable, even though it might be more tedious to trace the include blowups that result.

The moral of the story is that it’s usually bad to screw with built-in keywords via preprocessor macros.

GUI cues for block-level copy in Disk Utility

Posted on by yiming

In Mac OS X 10.4 Tiger (PPC edition), the only UI difference between a block-level copy and a file-level copy when using Disk Utility’s Restore (a.k.a. disk cloning) mode is that the progress bar label reports "Copying Blocks..." for the former and just plain "Copying..." for the latter. The difference is significant, especially for full-disk cloning operations.

To invoke block-level copy, you:

  • must not be booted from either the source or the destination partition – the Mac OS X DVD is good for this, if you do not have a different partition.
  • must tick the checkbox for Erase Destination when setting for Restore mode in Disk Utility
  • may or may not need to select “Skip Checksum”.

Unfortunately, the progress bar message confirming whether a file-level or block-level copy is displayed after one has already invoked the Restore procedure, and there isn’t a “Cancel” button anywhere in sight. As if potential for hours of time wasted (and quite possibly loss of metadata, since asr in file-copy mode doesn’t bother preserving such trivial things as file creation date) isn’t good enough to warrant a button to express one’s regret.

One wonders why Apple did not simply put in a simple checkbox for “Block-level copy”, and warn you about the requirements for block-level copy (and further, asking you if you wish to proceed with file-level copy anyway if you don’t meet the requirements.)

Probably because they thought (in their infinite Apple wisdom) that it would scare and confuse the “normals” who can’t tell the difference. Never mind that the people who would try to clone their entire drives for backup (as opposed copying files here and there to a USB key) are probably savvy enough to care.

Wonder if this changed in Leopard.

Update
Many seem to arrive at this post wondering what a block-level copy is. Here it is in a nutshell. A typical hard disk is divided into a linear set of n logical blocks, m bytes each. In short, your files are recorded within these blocks. To keep track of things, the filesystem is responsible for maintaining more metadata on top of this. This lets it create such niceties as “folders”, and forms the overall tree of folders nested within folders that you see in Finder.

A file-level copy means that the copying program loads in the directory tree and walks the tree. When it finds a directory, it’ll load metadata to find all the files contained in it. When it finds a file, it’ll go look for the blocks that contain the file’s data, and start copying. This has a lot of overhead, since the program has to load in the nice tree abstraction first, descend into each folder looking for files to copy, and then go find the proper blocks to copy, and then finally copy the data and any metadata associated with it. A block-level copy, on the other hand, recognizes that if you literally want to copy everything from one disk to another, it’s a lot easier and faster to just start copying at block 1 until you get to block n at the end — rather than running up and down that directory tree.

A block-level copy is a literal byte-for-byte (well, one would hope) copy of one disk to another, while file-level copying creates a copy of each file and folder from one disk to another. The distinction here is subtle but important. A file-level copy from disk A to B does not necessarily result in A == B, while a block-level copy (for all intents and purposes) does. As for performance, for a full-disk clone, block-level copying should be dramatically faster than file-by-file copying.

Cisco VPN behind a NAT

Posted on by yiming


Useful if you’re:

  • on OS X
  • using the Cisco VPN Client 4.9.01 or below
  • are behind a router/NAT
  • and having intermittent connectivity issues with the Berkeley Campus Full Tunnel VPN

You might also be able to use this info if you have a similar network setup and having similar problems, but I’m not going to claim that.

Basically, the problem for me was that three connections out of four would get an IP address from the VPN, but the actual network is unreachable. No IP can be ping’ed successfully. The VPN GUI reports “Bytes In: 0, Bytes Out: xxxx”. The VPN log is stuck in a loop of:

Sending DPD request to xx.xx.xx.xx, our seq# = 1234
...
Received DPD ACK from xx.xx.xx.xx, seq# received = 1234, seq# expected =
1234
...

The solution that I’ve found is to switch on Enable Transparent Tunneling -> IPSec over UDP ( NAT / PAT ). This can be done by hitting Modify on the GUI, for the appropriate Connection Entry. Then, use the Transport tab and tick on the appropriate box. For good measure, I also forwarded ports 500 and 4500 on my router’s NAT, to ensure that the conventional Cisco VPN ports are open to the network (and just to do some irrational voodoo). The IPSec over TCP option, btw, does not appear to work, despite what Berkeley IT say in the instructions page. The client refuses to connect with that option active, though in theory it should have worked. Perhaps I’m not forwarding the right ports for it.

In any case, finally, after 1.5 years of this nonsense, the Berkeley VPN doesn’t choke on me anymore (too bad I’ll be leaving here in 6 months. Argh.). Every connection I make gets through on the first time, rather than on the fourth or fifth time. It still doesn’t make sense how I was able to connect to the VPN before, though. Why would it fail intermittently, and not always?

This is why I am not a network engineer. It already gives me a headache.

Entourage sent-mail archival, episode 2

Posted on by yiming


Previously, on The Sarth Repository…

I had this setup going on to automatically redirect most messages I send to a repository for later search and retrieval…A month later, by pure chance, I realized that Entourage wasn’t quite deactivating the CC field on the [redirected] archival email. In essence, all the people I cc’ed on anything got spammed with a duplicate every time I sent a message… 

And now, the continuation…

So Google finally enabled IMAP for my accounts on thallos.org, which allowed me to test a new strategy for archiving sent mail. Again, the goal is to have a copy archived straight from Entourage, whenever I send a new email, to my mail repository. With proper IMAP access, however, this became much easier.

First, configure Entourage for IMAP access to Gmail / Google Apps. This is surpisingly non-trivial, since Entourage is not a supported client as of the time of this post. Rather strange, considering that Entourage must be at least second or third place in terms of install-base for Mac email clients. Follow the generic instructions for IMAP setup, and you should do okay. If you’re on Google Apps, the username is your_name@your_domain.tld, as per this configuration instruction.

You should have an IMAP structure for your Gmail boxes once this is complete. Simply set a rule in Rules -> Outgoing, for all messages, to copy the message to the Gmail/Sent Mail folder. In fact, this is the exact same approach if you were backing up to an IMAP-enabled mail server.

Unfortunately, It broke for me on a couple of messages. Gmail servers reported inconsistent failure messages, such as “Connection to the server failed or was dropped” and “The message could not be copied.” Some message headers also seemed to be mangled in transit, with the sender’s name dropped and so forth. The messages themselves were innocuous, text-only messages with no attachments, HTML, or any other random nonsense, so I find it very curious to be failing on these messages. Will have to look into it a bit more.

UPDATED Nov 22, 2007
See the exciting (yet depressing) episode 3 of my adventures in email archival.

Entourage: thwarting archival strategies since 2004

Posted on by yiming

So I use Microsoft Entourage as my main email client, and had been wanting for some time to get my messages exported out of my local drive. As much I trust my laptop and my backups, one good earthquake later and all of that would be futile.

Getting my message archives preserved (with all metadata intact, like Sent and Received dates, etc) was the easy part. Grabbing all future messages was the hard one. Of course, Microsoft, in its infinite wisdom, didn’t include an auto-bcc for Entourage.

I had this setup going on to automatically redirect most messages I send to a repository for later search and retrieval. I had a process set up where, except for select messages that I mark as confidential, the above rule gets triggered.

A month later, by pure chance, I realized that Entourage wasn’t quite deactivating the CC field on the redirect for archival. There is a bug that resends the message to all CC’ed emails on redirect. For example, if I were sending to [email protected], cc’ed to [email protected], and redirecting to [email protected]:

1. the first copy goes out to a and b.
2. Then, the redirected copy will be sent to archive and b, as b appears on the CC list.
3. End result: a receives 1 copy, b receives 2 copies, and archive receives 1 copy.

In essence, all the people I cc’ed on anything got spammed with a duplicate every time I sent a redirected copy via Entourage’s Outgoing rule. This is stupid, and Microsoft’s website doesn’t warn you about this. Try it for yourself if you don’t believe me.

Had I been more diligent at searching the web or even just testing out this archival strategy, this wouldn’t have happened. Plus, I would have noticed one fellow complaining that all contacts on the CC list, for every email, received a copy of his archived messages. Ouch. I’m glad I didn’t try redirecting all of my sent box (there is another strategy, which I will outline sometime, is far easier – but it can’t do real-time, auto-bcc).

To all the people whom I inadvertently spammed, I’m awfully sorry. This won’t happen again.

Fixing FilePlanet’s stupidity on the Mac

Posted on by yiming


Lately I haven’t been able to download files from fileplanet.com via my Mac. It’s inane, because downloads apparently requires an ActiveX control. I’m appalled at the utter stupidity of excluding all non-Windows platform users from your download service, just to set up a download queue. Can’t you put up a Flash control instead? Just as shiny and unusable, but actually compatible with other operating systems.

It gets better. The good news – the designer had some foresight to set up a fallback mechanism, to use plain old HTML queue. The bad news – it simply presents you with a 403 Forbidden when clicked.

As it turns out, I found a post that contained a possible solution. Actually, that post is a bit unnecessarily complicated. Apparently, they’re blocking all browsers without a Windows user agent. On the fallback solution that was supposed to work for all platforms. Argh.

Until Fileplanet addresses this issue (which could be tomorrow. or never), the simplest solution (that worked for me) was to switch my user agent in Firefox (via the aforementioned and highly recommended User Agent Switcher) to a Windows browser (try the default Opera XP user agent). Then, click on the fallback queuing link, and it should kick you into the download page.

Note that the User Agent option in Safari (at least, v2.0.x) Debug menu will not work straight up. Believe me, as a primarily Safari user, I’ve tried hard to make this work. Because Fileplanet pops up a new window when download is selected, and the Debug menu setting only sets it for the active window. As the download window is a pop-up, you do not get a chance to intervene and change the User Agent code before FilePlanet denies you access to it. So for now, Firefox + User Agent Switcher is the solution. If you have a browser (or Safari, in the future) which allows the fake user agent setting to persist across windows spawned from the initial window, that browser would work too.

UPDATE 3/27/2009:
Feedback in the comments section reports that this is still a problem for many users. Appalling. It’s been 2 years.

Extracting Address Book information

Posted on by yiming

If you’ve left information in the Address Book in OS X, any other program on the machine can use Cocoa APIs to extract information automatically.

#import <Foundation/Foundation.h>
#import <AddressBook/AddressBook.h>
#import <AppKit/NSWorkspace.h>

void sayHello(void)
{
   ABPerson *currUser = [[ABAddressBook sharedAddressBook] me];
   NSString *firstName = [currUser valueForProperty:kABFirstNameProperty];
   NSString *lastName = [currUser valueForProperty:kABLastNameProperty];
   [[NSWorkspace sharedWorkspace] 
                           openURL:[NSURL URLWithString:
                                                     [NSString stringWithFormat:
                                                     @"http://www.google.com/search?hl=en&q=%@+%@", 
                                                          firstName,
                                                          lastName]]];
}

int main (int argc, const char * argv[])
{
   NSAutoreleasePool *pool = [[NSAutoreleasePool alloc] init];
   sayHello();
   [pool release];
   return 0;
} 

This will take the name labelled as the currently logged in user and hit up Google, placing the first and last name in the search box and run a search with it. Other pieces of information are also accessible via the AddressBook API.

As you can well see, this can be used for many purposes: nifty automagical hacks in the OS’s built-in suite of personal information and communcation apps, for example. But I can think of less beneficial uses for this API. You should be able to trust the programs you run.

duplicates and invoking LaunchServices

Posted on by yiming

If you deal with duplicate files a lot, then fdupes is a very quick command-line solution. Among other things, it also does an MD5 comparison of file signatures, can recurse directories and following symlinks, and can be used for scripting as well as directly from standard input.


Bonaparte-Prime:/tmp $ fdupes -r -d ./
[1] ./foo.mp3                 
[2] ./bar/baz.mp3

Set 1 of 4, preserve files [1 - 2, all]:

However, sometimes I forget what the hell foo.mp3 and bar/baz.mp3 really were. Then I have to fire up the Finder (or another Terminal window) and go chase down the file. What I really wanted would be if there were an “open one of these files and let me see what it is, then I’ll make a decision” option.

The beauty of open source is that I can patch fdupes myself and scratch this itch.

Finding the right app to hand off to might be annoying. On OS X, if you want to use the nice automagical way that Finder opens files (meaning that it generally knows what applications are appropriate to open an arbitrary file), you’ll have to link against LaunchServices, a part of the ApplicationServices framework. The code to do that is actually quite trivial. Simply add the ApplicationServices.h header, and then a function such as:


OSStatus open_target_file(const char* filepath)
{
    FSRef ref;
    OSStatus err;
    Boolean isDirectory;
    
    err = FSPathMakeRef((const UInt8 *)filepath, &ref, &isDirectory);
    
    err = LSOpenFSRef(&ref, NULL);
    
    return err;
}

will do the job. Obviously this is simplistic and without error-checking, but you get the idea. LaunchServices will then handle the tedium of finding the right system application to launch the file.

The other half of the patch involves modifying fdupes’s command parser to take an additional variant. I chose a scheme by which if I precede the choice numbers I make with the character ‘o’, it should then hand off the filepath to the aforementioned open_target_file function and return to its previous state. Since the parser already has such a case to handle mistaken input, the patch is also quite simple. Simply check for ‘o’ in the first byte of the preservestr array at the appropriate place, and then should it match, have it execute open_target_file() rather than preserve[number] = 1; If you’re the portability stickler type, wrap these new lines under #ifdef __APPLE__ statements to check for OS X at compile-time.

Source code to simple utilities like these are really quite helpful. They let someone with just a modicum of programming ability tweak the behavior and address some common annoyances, without having re-invent the thing from scratch or to pester the original dev, who probably doesn’t care nearly enough about little features for specific platforms that only specific people might find useful. Write the extension code, create a patch, and keep your patch around in case the utility is ever updated. Scratch your own itch as free software intended.

/Library/Java/Extensions, MATLAB errors, and you

Posted on by yiming

If you installed MATLAB 2006b on OS X and encountered this error when Matlab starts up:

java.lang.IllegalArgumentException: http://java.sun.com/xml/jaxp/properties/schemaSource
at org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.setAttribute(DocumentBuilderFactoryImpl.java:118)
at com.mathworks.xml.XMLValidator.validateFile(XMLValidator.java:54)
at com.mathworks.mlwidgets.util.productinfo.Product.getItems(Product.java:310)
at com.mathworks.mlwidgets.util.productinfo.ProductInfoUtils.pathChanged(ProductInfoUtils.java:160)
at com.mathworks.mlwidgets.util.productinfo.ProductInfoUtils.<init>(ProductInfoUtils.java:67)
at com.mathworks.mlwidgets.util.productinfo.ProductInfoUtils.getAllProductsInfo(ProductInfoUtils.java:910)
at com.mathworks.mlwidgets.util.productinfo.ProductInfoUtils.<clinit>(ProductInfoUtils.java:53)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:164)
at com.mathworks.mde.desk.StartupClassLoader.callClassForName(StartupClassLoader.java:304)
at com.mathworks.mde.desk.StartupClassLoader.access$000(StartupClassLoader.java:27)
at com.mathworks.mde.desk.StartupClassLoader$LoadInfo.<init>(StartupClassLoader.java:80)
at com.mathworks.mde.desk.StartupClassLoader.addLoadInfo(StartupClassLoader.java:219)
at com.mathworks.mde.desk.StartupClassLoader.createLoadInfos(StartupClassLoader.java:195)
at com.mathworks.mde.desk.StartupClassLoader.access$500(StartupClassLoader.java:27)
at com.mathworks.mde.desk.StartupClassLoader$2.run(StartupClassLoader.java:147)
at java.util.TimerThread.mainLoop(Timer.java:512)
at java.util.TimerThread.run(Timer.java:462)
Exception in thread "Timer-2" java.lang.ExceptionInInitializerError
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:164)
at com.mathworks.mde.desk.StartupClassLoader.callClassForName(StartupClassLoader.java:304)
at com.mathworks.mde.desk.StartupClassLoader.access$000(StartupClassLoader.java:27)
at com.mathworks.mde.desk.StartupClassLoader$LoadInfo.<init>(StartupClassLoader.java:80)
at com.mathworks.mde.desk.StartupClassLoader.addLoadInfo(StartupClassLoader.java:219)
at com.mathworks.mde.desk.StartupClassLoader.createLoadInfos(StartupClassLoader.java:195)
at com.mathworks.mde.desk.StartupClassLoader.access$500(StartupClassLoader.java:27)
at com.mathworks.mde.desk.StartupClassLoader$2.run(StartupClassLoader.java:147)
at java.util.TimerThread.mainLoop(Timer.java:512)
at java.util.TimerThread.run(Timer.java:462)
Caused by: java.lang.NullPointerException
at com.mathworks.mlwidgets.util.productinfo.Product.getItems(Product.java:317)
at com.mathworks.mlwidgets.util.productinfo.ProductInfoUtils.pathChanged(ProductInfoUtils.java:160)
at com.mathworks.mlwidgets.util.productinfo.ProductInfoUtils.<init>(ProductInfoUtils.java:67)
at com.mathworks.mlwidgets.util.productinfo.ProductInfoUtils.getAllProductsInfo(ProductInfoUtils.java:910)
at com.mathworks.mlwidgets.util.productinfo.ProductInfoUtils.<clinit>(ProductInfoUtils.java:53)
... 11 more

or something similar, you may have hit one of the mysterious gotchas of Mac OS X 10.4. In /Library/Java/Extensions (or another one of these system Java directories), you have a conflicting copy of xerces.jar, probably from another software package that rudely installed these extensions without informing you of it. Due to the way Java classpaths are being resolved, apparently the copy in Extensions overrides the copy in Matlab’s directory.

The symptoms of this: among other things, Matlab will throw these exceptions at startup; you can’t open certain panels or windows via the menu bar; you also cannot select certain panels in the Preferences window. Whenever you perform one of these operations, you’ll receive a NoClassDefFoundError on one of the Matlab widget classes, even though it already exists in the Matlab directory.

This is a rather classic gotcha, but a lot of people wouldn’t even realize /Library/Java/Extensions actually exists. If you went off chasing the red herring of the NoClassDefFound exception, or went tracing StartupClassLoader.java, you’d never realize the core cause of the issue.

/Library is sometimes an incomprehensible morass of things that cause mysterious failures in other applications (InputManagers, anyone?) and end up exceedingly difficult to trace.