Author: yiming

O(n) person in an O(log n) world

Romance of the Three Kingdoms XI game event stills

Posted on by yiming

In Koei’s Romance of the Three Kingdoms XI for Windows PC (non PUK, English edition), there are a total of 30 event stills ( basically, pictures of important events during a campaign, viewable under Settings -> Reference -> View Event Stills ). Upon collecting all 30 and winning any scenario to unite China, you will unlock top-tier skill sets and all historical portraits when you use Create Officer for your own custom officers. It’s one of the things required for One Hundred Percent Completion of the game. However, there has not been a very good guide as to how to achieve all 30 events, especially with problematic and rare ones like Secret Meeting.

This post is my attempt to catalog all the event stills and how to obtain them in game. The events are listed below in order of their appearance on the View Event Stills page. In general, you only need to view the Event Still once. That means that you can start a game, obtain the Events you want, without actually completing or even saving the game once. Even if you abandon that game later, once you’ve viewed the event, it counts toward your completion of collecting all 30.

Note that I will not suggest any unreasonably contrived means to get events, even if that might be faster, because I consider these tactics basically cheating. For example, I will not suggest that you send your sovereign out to die deliberately to get the Sovereign Death event, even if you might be able to do that. You may also create officers with birth and death dates to your exact specification, to achieve some of these events, but that, too, is basically cheating. Most of the fun in these events is trying to obtain them fairly while playing a game — it adds an additional dimension of strategic planning. If you’re just going to cheat, stop reading this and go download the fully unlocked sysfile that’s floating around the Web.

Page 1

  • Plague – when a city receives a Plague event, causing all sorts of havoc. You will likely get at least one in any scenario.
  • Locusts – when a city receives a Locusts event, decreasing food production in that city. You will likely get at least one in any scenario.
  • Plenty – when a city receives a Plenty event during harvest season, increasing food production in that city. You will likely get at least one in any scenario.
  • Bandits – when bandits appear in your territories, as the name suggests. The easiest way to obtain this still is to start the Yellow Turban Rebellion scenario. During the story intro sequence, the Yellow Turbans will rise against the Han court, and this will unlock the event still.
  • Coronation – when an Emperor is crowned, or when you declare support for the Emperor after capturing the capital (Luoyang, Chang’an, or Xu Chang, depending on the scenario). Easily achievable in most scenarios if you’re winning.
  • Unity: Male – when you unite China under a male sovereign. This event will be unlocked when you win a scenario with a male sovereign. I prefer Liu Bei, but YMMV 🙂
  • Unity: Female – notes not yet available
  • Downfall – when one of the forces in the game is eliminated. At least one ruler will fall in any game, so this is trivial to obtain. You can also get this by losing your city in Tutorial 3, Going into Battle.
  • Independence – when a prefect or viceroy declares independence and forms his or her own force in a game. In Tutorial 6, Capturing Cheng Du, there is a good chance that Jian Ning will declare independence under the prefect Yong Kai, but it is not guaranteed. In the 190 scenario, the free cities in Jiangdong may declare independence under their prefects within the first two years. In any scenario, foment an independence movement in an enemy city via the strategy Collaborate. Alternatively, wait until one of your district viceroys’ or city prefects’ loyalty falls below a certain threshold, at which point an AI force is likely to foment a revolt in that prefecture or district. Finally, when you’re winning a full scenario, at the very end of the game one of your officers will ask you to investigate another officer for treason. If you choose to do so, that officer will be proven innocent, but will revolt anyway due to your distrust of him. This revolt will trigger an Independence event.
  • Omen – This event is displayed at the start of the year, when your strategist warns that one of your officers is about to die (of natural causes, not in battle). If you play through a game with historical age settings, quite a few of your officers will die. Pick a middle scenario, when there are a lot of older officers around.
  • Sovereign Death – when your own sovereign dies of natural causes. The easiest way to get this one, short of deliberate regicide on your part, is to use Tao Qian in 194, Rival Warlords. He will most likely die within the next two years, giving you this event.

Page 2

  • Officer Death : Male – When one of your male officers die. This is relatively trivial to get – one will come within the year, after an Omen, for example. Your officer may also be killed by a cavalry charge or in a duel, if officer deaths are on.
  • Officer Death : Female – When one of your female officers die. This is a little more annoying, given the sparsity of female officers. Still, at least one should die in any game. Your officer may also be killed by a cavalry charge or in a duel.
  • Funeral – When any sovereign (except yours) dies of natural causes — that is, of illness or old age, not in battle. In the 194 scenarios, Tao Qian will die within the next two years. Play the scenario as a ruler other than him, and you will trigger this event still.
  • 2 Sworn Brothers – when you arrange for two male officers to swear brotherhood with each other. Officers with sworn brothers on the same force will not defect, and will obtain bonuses when both are assigned to the same unit. You’ll need two officers of the male gender and at least 500 in Deeds each. You will also need 500 technique points (TPs). At that point, Sovereign -> Mediate will allow you to mediate sworn brotherhood between these two officers and unlock this event.
  • 2 Sworn Sisters – when you arrange for two female officers to swear sisterhood with each other. Officers with sworn sisters on the same force will not defect, and will obtain bonuses when both are assigned to the same unit. You’ll need two officers of the female persuasion and at least 500 in Deeds each. You will also need 500 technique points (TPs). At that point, Sovereign -> Mediate will allow you to mediate sworn sisterhood between these two officers and unlock this event.
  • Male Officer Appears – Triggers at the start of a new year, when a male officer (who is related to one of your current officers) arrives to offer his services to your ruler. For example, as Liu Bei in 211 Liu Bei in Shu scenario, you will likely receive Zhang Bao – Zhang Fei’s son, in 212. Note that your sovereign should be within a city for this event to occur.
  • Marriage – when you marry one of your officers to another – married officers will not defect, and obtain bonuses when assigned to the same unit. You’ll need two officers of opposite gender, at least 500 in Deeds, and unmarried status. You will also need as 500 technique points (TPs). At that point, Sovereign -> Mediate will allow you to conduct a marriage between these officers and unlock this event.
  • Fem. Officer Appears – Triggers at the start of a new year, when a female officer (who is related to one of your current officers) arrives to offer her services to your ruler. This one is fairly tricky, because there aren’t that many female officers. In general, play one of the middle scenarios as a prominent family. For example, if you play as Sun Quan in 211, you will get a female officer from the Sun family in 215.
  • Resignation – when an officer resigns from your force due to low loyalty. This is also a hard one to get if you are a decent player, since you’d never let an officer’s loyalty drop low enough. Note that if your sovereign dies and your officers resign, you will not receive this event still. You must have this officer resign through low loyalty. Officers with loyalty less than 70 will tend to resign, and they usually do so on Jan of a new year. Best way to do this may be to play as Dong Zhuo in 190. Use your overwhelming starting forces to take down Cao Cao, then hire his wife Bian Shi. Use Bian Shi to hire Cao Cao. Dong Zhuo and Cao Cao are disliked officers, so Cao Cao’s loyalty will start in the 70s and drop fast. However, since you have his wife in your service, he will not be hired away by another sovereign. At the start of the next year, Cao Cao will most likely resign from your force.
  • Decree – when an imperial decree arrives. For example, when the Emperor promotes your sovereign to the rank of Duke, he will use this kind of decree. Since to win a game, you’ll likely surpass the rank of Duke, this event is trivial to unlock if you win a game.
  • Orders – when a different type of imperial order arrives. If you start 190 Cao Cao's Ascent, the story intro sequence to the scenario involves Cao Cao’s forged imperial edict arriving at Yuan Shao’s city. This will grant you the event still.

Page 3

  • Fallen Castle – when a city falls to your forces. You are likely to receive this in Tutorial 6, Capturing Cheng Du…when you capture Cheng Du. Otherwise, take any city or base, and the event is yours. Trivial.
  • Conversation – when one of your civil officers head off to search for a friend in the area and obtains his or her services after a debate. This seems to occur randomly, when your sovereign is in the same city as certain civil officers on his force. It is the analogous event to “Search”, which applies only to military (high WAR) officers. The event represents the “conversation” between your officer and his friend. This obviously only triggers when there are hidden officers to find in a city — so start an early scenario with lots of hidden officers for a better shot at the event.
  • Coalition – pick up this event when the coalition against Dong Zhuo forms in 190 Cao Cao's Ascent, at the very beginning during the story intro sequence. Otherwise, play as a strong army (say, Cao Cao in 207), and eventually two of your weaker enemies will merge to form a coalition against you.
  • Search – when one of your military officers head off to search for a famous hero in the area and obtains his or her services after a duel. This seems to occur either randomly or scripted to resemble randomness. This obviously only triggers when there are hidden officers to find in a city — so start an early scenario with lots of hidden officers for a better shot at the event. The 190 scenario with Cao Cao in Xu Chang or Chen Liu should be an easy way to get this event, as you recruit various notables like Dian Wei or Xu Chu.
  • Secret Meeting – when you meet a hidden character for a philosophical discussion. This one is ridiculously obscure and difficult to obtain, to the effect that many people do not know how to obtain it. Forum posts and such suggest that you’d receive this when the AI foments unrest with one of your viceroys or prefects (one of the in-game log messages say “your prefect blah is meeting someone secretly”. However, that is a red herring. One post correctly identifies that the Secret Meeting event is awarded for meeting a hidden character such as Xu Shao during a Search. As Cao Cao in 190 Cao Cao's Ascent, if you keep Searching the area around Chen Liu or Ru Nan, against your strategist’s advice that nothing would be found, you will eventually encounter Xu Shao, who will do a physiogamy reading for you, unlocking this event. One of the early scenarios, such as Rise of Dong Zhuo, is preferred for obtaining this event. Just keep Searching your city, a lot, and fruitlessly most of the time. You’ll get this eventually ( took me under two game years of on-and-off searching in Chen Liu while holding off Dong Zhuo’s army ). A comment from Lance Lidner reports that using Kong Zhou in Rise of Dong Zhuo works best, as Xu Shao appears in Ru Nan at that time. Kong Zhou is also out of immediate reach from most hostile factions in that scenario, so you can afford to take your time.
  • Appoint – when the emperor issues a new title to you. This occurs when you’ve captured enough cities to reach the next rank, and that your sovereign is currently in a city (as opposed to being on campaign). As Liu Bei in the 211 Liu Bei in Shu scenario, taking Xiang Yang (via Jiang Ling – you’ve got Zhuge Liang, Pang Tong, Zhang Fei, Liu Bei, Sun Shang Xiang, Zhao Yun … that’s plenty to take the city from Cao Ren early on) and Yong An will bump you up to Field Marshal, and Emperor Xian will award you the title and this event. As Cao Cao in 190 Cao Cao's Ascent, taking the free cities of Xu Chang and Xin Ye will grant the title of Lt. Governor, which once again easily unlocks this event.
  • 3 Sworn Brothers – when you arrange for three male officers to swear brotherhood, much like the historical Oath of the Peach Garden of Liu Bei, Guan Yu, and Zhang Fei. Officers with sworn brothers on the same force will not defect, and will obtain bonuses when two or more are assigned to the same unit. You’ll need three officers of the male gender and at least 500 in Deeds each. You will also need 500 technique points (TPs). At that point, Sovereign -> Mediate will allow you to mediate sworn brotherhood between these officers and unlock this event.
  • 3 Sworn Sisters – when you arrange for three female officers to swear sisterhood. Officers with sworn sisters on the same force will not defect, and will obtain bonuses when two or more are assigned to the same unit. You’ll need three officers of the female persuasion and at least 500 in Deeds each. You will also need 500 technique points (TPs). At that point, Sovereign -> Mediate will allow you to mediate sworn sisterhood between these officers and unlock this event. The hardest part of this is rounding up 3 female officers to begin with.

redhillonrails_core and broken MySQL empty string defaults

Posted on by yiming

While hacking on a side project in Ruby on Rails, I ran across this weird error when trying to insert new data:

ActiveRecord::StatementInvalid: Mysql::Error: Column 'attr2' cannot be null: INSERT INTO `foo` (`attr1`, `attr2`) ... VALUES ('1', NULL)

where attr2 is a varchar (or t.string, in Rails lingo) and set to not null default '' (or, in other words, :null=>false, :default=>''). Strangely enough, instead of the default value of ”, ActiveRecord was setting the value to nil instead, which translates into a NULL. Since the schema explicitly forbids NULLs on that column, the statement explodes.

After an hour of poking around and hacking up a spike solution, it turns out a plugin was to blame. I’d pulled in the foreign_key_migrations plugin (a highly recommended add-on) to automatically install foreign key constraints (in this day and age, the foremost web framework still can’t automatically handle FOREIGN KEY constraints, the most basic tool for ensuring data integrity in relational databases, for its migrations? Bah!).

This plugin has a dependency on redhillonrails_core, which has a known bug: Incorrectly overwrites mysql empty-string default with nil for string/text/binary types.

The bug is apparently not being worked on as of the time of this writing. The dev doesn’t consider this a bug, as he claims that “[he considers] empty strings to be semantically identical to NULL”.

This position, unfortunately, is not supported by the SQL standard. Wikipedia has a section on common SQL NULL mistakes documenting some of the potential problems involved in making such an assumption. Philip Greenspun has further notes on this. Using NULLs trigger the all the arcane annoyances of three-valued logic, and you must be prepared to consider True, False, and NULL values as comparison outcomes. Someone not very versed in three-valued logic can easily cause a number of subtle mistakes trying to compare values.

To workaround this bug, you will need to comment out the initializer function in

vendor/plugins/redhillonrails_core/lib/red_hill_consulting/core/active_record/connection_adapters/mysql_column.rb

Alternatively, you can delete the file, and remove relevant references to it..

If you agree with dev’s position that NULLs are “semantically identical” to empty strings, remember to pay attention when you formulate your SQL queries (and when Rails formulates those queries) — your results may not be what you expect, if implemented naively. Get your three-valued logic truth tables out 🙂

APR and 32-bit/64-bit universal binary compilation

Posted on by yiming

When compiling APR, the Apache Portable Runtime 1.3.3 (as a part of Subversion 1.5.3 as I am doing here, or not) on OS X 10.5 Leopard, you may encounter the following error at compile time.

/bin/sh /tmp/subversion-1.5.3/apr/libtool --silent --mode=compile gcc-4.2 -Os -arch i386 -arch x86_64 -DHAVE_CONFIG_H -DDARWIN -DSIGPROCMASK_SETS_THREAD_MASK -no-cpp-precomp -I./include -I/tmp/subversion-1.5.3/apr/include/arch/unix -I./include/arch/unix -I/tmp/subversion-1.5.3/apr/include/arch/unix -I/tmp/subversion-1.5.3/apr/include -o strings/apr_snprintf.lo -c strings/apr_snprintf.c && touch strings/apr_snprintf.lo
strings/apr_snprintf.c: In function ‘conv_os_thread_t’:
strings/apr_snprintf.c:500: error: duplicate case value
strings/apr_snprintf.c:498: error: previously used here
strings/apr_snprintf.c: In function ‘conv_os_thread_t_hex’:
strings/apr_snprintf.c:671: error: duplicate case value
strings/apr_snprintf.c:669: error: previously used here

This will most likely happen when you are configured to build a dual 32-bit / 64-bit universal binary, whether it be ppc / ppc64, or i386 / x86_64, or any permutation thereof. This ticket over at MacPorts documents a particular instance of this problem, with no apparent solution.

The symptom is easy to explain. Somehow, two case labels in the relevant switch statement in strings/apr_snprintf.c:500:

switch(sizeof(u.tid)) {
    case sizeof(apr_int32_t):
        return conv_10(u.u32, TRUE, &is_negative, buf_end, len);
    case sizeof(apr_int64_t):
        return conv_10_quad(u.u64, TRUE, &is_negative, buf_end, len);
    default:
        /* not implemented; stick 0 in the buffer */
        return conv_10(0, TRUE, &is_negative, buf_end, len);
    }

have evaluated to the same value. In particular, it believes that sizeof(apr_int32_t) and sizeof(apr_int64_t) are the same value. As we all know in C, you cannot have two identical case labels in the same switch statement. However, the root of the problem is a bit more subtle.

In $SRCDIR/include/apr.h, you’re likely to see this fragment of code.

typedef  long       apr_int64_t;
typedef  unsigned long  apr_uint64_t;

Notice that it has typdef’ed apr_int64_t as a long and apr_uint64_t as unsigned long. This is because at configure time, the script detected that long values are 64-bit on this system, so it assigned the apache 64-bit types to longs. However, this only holds true for half of the compilation – because you are building a universal binary for a 32-bit architecture as well. Remember that in 32-bit GCC on OS X, longs are 32-bit rather than 64-bit. Your run-of-the-mill autoconf script, done by a non-OS X programmer, isn’t going to be able to detect this subtlety – if the 64-bit part worked, it’ll keep thinking longs are 64-bit, end of story – and happily generate the incorrect typedef expressions. When you apply sizeof to these types in apr_snprintf.c, both evaluate to 4 bytes under 32-bit compilation, thus blowing up the compile run.

To truly fix the root of the problem requires rewriting the autoconf script to detect Mac OS X and its universal binary building, which can potentially throw quadruple architectures at the same compilation script. However, a quick hack to make this particular problem go away is to change apr.h such that:

typedef  long long       apr_int64_t;
typedef  unsigned long long apr_uint64_t;

Now that we ensure in either 32-bit or 64-bit compilation, apr_int64_t and apr_uint64_t are always typedef’ed to appropriate, guaranteed 64-bit types. The compilation of APR (and Subversion) will proceed normally.

Note that long long is not an standard C type. As a GCC extension, this fix is a kludge. A kludge that works (for me), though.

UPDATE:
There may also be an issue with sizeof definitions that may cause the library to crash. In particular, there may be occurrences of

#define APR_SIZEOF_VOIDP 8

that were generated by configure. To fix this, you will need to remove the define and have the compiler check for 64-bit at compile-time:

#ifndef __LP64__
    #define APR_SIZEOF_VOIDP 4
#else
    #define APR_SIZEOF_VOIDP 8
#endif

In general, any predefined sizeofs need to be changed. I am not sure why the APR developers do hard-coded defines like this, given that the point of having sizeof() calls is to avoid such issues.

using ffmpeg for cutting media files – and the gotchas involved

Posted on by yiming

So here I was, trying to complete some ordinary transcoding / media file cutting tasks with ffmpeg. Turns out there are some weird gotchas when using some versions of ffmpeg (in my case, SVN-r10571) on the commandline.

Problem 1: transcoding to mp3 insists on using 64kbits/s or 128kbits/s bitrate

So a simple task is to transcode an audio file of some arbitrary format to MP3. In my case, I only wanted a 30-second piece of the original file, converted to MP3. The original was in 256kbits/s, and I decided to use the same bitrate for the output, just for kicks. Reading the man page, -b is for video bitrate, and -ab is for audio bitrate. So I executed:

wrong:  ffmpeg -ss 00:00:30.00 -t 25 -i foo.mp3 -ab 256 foo-new.mp3

The output, of course, was this:

Input #0, mp3, from 'foo.mp3':
...
mdb:109, lastbuf:0 skipping granule 0
size= 393kB time=25.2 bitrate= 128.0kbits/s
video:0kB audio:393kB global headers:0kB muxing overhead 0.007950%

128 kbits/s. That was…not what I wanted.

Turns out, as the ffmpeg man page hints but does not specify clearly, a “k” is required to label the units of the new bitrate. As in:

right:  ffmpeg -ss 00:00:30.00 -t 25 -i foo.mp3 -ab 256k foo-new.mp3

This time, the output was:
Input #0, mp3, from 'foo.mp3':
...
mdb:109, lastbuf:0 skipping granule 0
size= 782kB time=25.0 bitrate= 256.0kbits/s
video:0kB audio:782kB global headers:0kB muxing overhead 0.003996%

Much better. It’s very strange how that if I don’t give it the ending “k” but do give it a higher value than 64, it always bumps up the bitrate to 128kbits/s, from the default of 64k (but not to the number I actually wanted).

 

Problem 2: cutting media file without re-encoding

So ffmpeg can be used to cut a media file, without reencoding the media stream. You simply pass the raw copy codec to the -acodec (or the -vcodec for video) at encode time. For my MP3, I thought it was a pretty trivial problem, so I issued:

wrong: ffmpeg -i bar.mp3 -ss 00:00:30.00 -t 25 -acodec copy bar-new.mp3

This seemed to process correctly, except it created a 555-byte empty file with no content in it. What’s even weirder, if you issued:

wrong: ffmpeg -i bar.mp3 -ss 00:00:10.00 -t 25 -acodec copy bar-new.mp3

That is, -ss 00:00:10.00 to seek to the 10th second, and -t 25 to record 25 seconds worth of audio. Strangely enough, the output file had 15 seconds of audio, the subtraction of 25 by 10. Curiouser and curiouser. Now,

wrong: ffmpeg -i bar.mp3 -ss 00:00:10.00 -t 25 -acodec mp3 bar-new.mp3

does in fact create a 25-second MP3 file. Only, of course, a re-encoded one at 64kbits/s. Something that we are not looking for. But strange.

A post on ffmpeg-devel finally provided enough hints to clue me into the problem. Unlike many command line apps, the order of arguments passed to ffmpeg seem to be silently significant. The correct incantation is:

right: ffmpeg -ss 00:00:30.00 -t 25 -i bar.mp3 -acodec copy bar-new.mp3

Note the switched order for the arguments -i and the args -ss and -t, where the -i must follow the other two. Now, the desired 25-second file, cut from the original, is correctly produced.

When arguments are missing some arbitrary text or in the wrong order, ffmpeg doesn’t sanity-check or warn you of these…it just silently proceeds and does some very strange things. Things that make you scratch your head and wonder, “wtf? did I mistype an argument somewhere?”

Ah, and of course, you make me have to track down all these little idiosyncrasies, wading through blog posts and mailing lists and forums. Agh.

Subversion and undefined symbols

Posted on by yiming

This is fast turning out to be a blog about compiling open source software. Maybe I should change the title.

You may get this error while trying to compile Subversion 1.5.2:
ld: Undefined symbols:
_svn_fs_txn_root_base_revision referenced from libsvn expected to be defined in libsvn
_svn_fs_change_txn_props referenced from libsvn expected to be defined in libsvn
_svn_fs_get_mergeinfo referenced from libsvn expected to be defined in libsvn
_svn_fs_recover referenced from libsvn expected to be defined in libsvn
_svn_fs_upgrade referenced from libsvn expected to be defined in libsvn
_svn_fs_node_origin_rev referenced from libsvn expected to be defined in libsvn
/usr/bin/libtool: internal link edit command failed
make: *** [subversion/libsvn_ra_local/libsvn_ra_local-1.la] Error 1

Turns out there may be a bug in the libtool script shipped with the source tarball, under certain circumstances with OS X 10.4, that causes it to fail to link. Ah, libtool, you make my life so much harder sometimes.

If you copy /usr/bin/glibtool over $SRCDIR/libtool (that is, into the directory for the subversion source code, replacing the one that is placed there by the package itself), the package should compile with no further complaints. make test also shows success on all tests, so this seems a satisfactory solution.

UPDATE 1/31/2009
I’ve been informed in the comments that MacPorts may rely on source compilation from tarball, and thus have this issue. If you’re having this issue with the MacPorts SVN package, please check out the advice in the comment section from farkinga, who has additional notes.

GNUMake 3.80 and failed malloc

Posted on by yiming

Wacky compilation error:

Creating config.mak and config.h...
make(4162) malloc: *** vm_allocate(size=4272971776) failed (error code=3)
make(4162) malloc: *** error: can't allocate region
make(4162) malloc: *** set a breakpoint in szone_error to debug
make[1]: *** virtual memory exhausted. Stop.

This error has nothing to do with the particular code you’re compiling. Turns out GNU Make 3.80 has a bug, in which it sometimes sends a ridiculous requested block size to malloc(), which then fails to allocate the space.

The solution is to upgrade to GNUMake 3.81. Particularly pernicious on OS X 10.4 and Xcode 2.4.1.

AVCodecContext, Perian, and the Xcode dylib fetish

Posted on by yiming

The Context

So recently I was trying to interact with an avi file with a video stream encoded in H.264 and an audio stream encoded in MP3. To my surprise, my trusty Perian 1.1 failed to even play the video and gave me a black screen (though the audio played just fine). With some Google-fu, it was found in a forum post that the SVN version of Perian fixed this error.

So compile my own Perian 1.1.1b1 from SVN, right? No big deal. Heh. And thus begins our story. (Note that this procedure is accurate as of the time of writing. OSS projects change rapidly as bugfixes are checked in. You may not have this problem at all. But it’s still useful as a reference in the future, in case similar problems arise).

Problem 1: AVCodecContext.bits_per_sample

The first problem you hit is probably:
/tmp/perian-orig/ff_private.c:181: error: ‘struct AVCodecContext’ has no member named ‘bits_per_sample’
…
/tmp/perian-orig/FFissionCodec/FFissionDecoder.cpp:289: error: ‘struct AVCodecContext’ has no member named ‘bits_per_sample’

Turns out that the upstream project FFMPEG, who is responsible for libavcodec, has very recently changed the API in one of its latest revisions. bits_per_sample has become
bits_per_coded_sample in revision 15262. Because Perian’s SVN does an external SVN checkout of the HEAD of libavcodec, it pulled these API-breaking changes without considering whether that breaks its own compilation or not.

The fix is simple enough: global search and replace the member variable to bits_per_coded_sample.

Problem 2: Xcode’s unnatural fondness for dynamic libraries

At this point, libavcodec compiles, so you think you’re home free. For most users, that is probably the case. For some of us, though, there’s one more hurdle.

While compiling Perian.component, you might get something like:
ld warning: in /Developer/SDKs/MacOSX10.4u.sdk/usr/local/lib/libavcodec.dylib, file is not of required architecture
Undefined symbols:
"_avcodec_close", referenced from:
…and so on

Closer examination reveals that at some point, you might have compiled FFMPEG or MPlayer or any of the many media projects that depend on FFMPEG libraries (for me, that was transcode, whose own compilation idiosyncracies I documented in an earlier post). When you installed these libraries, they went into your library search path. Note the Perian compilation line immediately preceding the compile failure:
/Developer/usr/bin/g++ ... -read_only_relocs suppress -lavcodec -lavformat -lavutil -lebml -lmatroska ... /tmp/perian/UniversalDetector/build/Release/libuniversaldetector.a -lbz2 -o /tmp/perian/build/Deployment/Perian.component/Contents/MacOS/Perian

As you can see, the -lavcodec linker argument will cause the linker go on the search path and try to find the library. It of course first finds your pre-installed library in /usr/local/lib or some other system lib path, and tries to use it. If you compiled that library as Intel only or PPC only and you tried to build a universal Perian, that would obvious fail and generate an architecture error. If you, like me, compiled only for x86_64 or ppc64, that would also trigger the same architecture problem (as Perian builds as x86 32-bit).

This problem stumped a user “vs”, in this Perian-discuss Google groups thread. Notice how the devs didn’t get the root of the problem, and answered snarkily and defensively to the user bug report.

The devs aren’t wrong, though. Didn’t we just see a statically compiled target, libavcodec.a, in the target list for the Perian Xcode project? Why yes, yes we did! In fact, there are static targets for all of the -lav* libraries in the Xcode project. So why did the linker go off and use -lavcodec rather than, say, $PERIAN_DIR/build/Universal/libavcodec.a, as instructed?

The workaround

In short: Xcode is screwing things up. Xcode 3.1 has a distinct (some say, unnatural) preference for dynamic libraries. When you add a library to the “Link Binary With Library” listing under Targets, it simply adds a -l<libname> to the link line to pass to ld at link time. When this happens, it will look for lib<libname>.dylib (in this case, libavcodec.dylib, libavformat.dylib, libavutil.dylib, etc in your library path. If it finds one, it will use that one rather than the static library you provided in the Xcode project file. Obviously, this ends up linking the wrong library, perhaps with the wrong architecture or the wrong version. Compilation explodes.

The workarounds are not very clean. There seems to be no good way to instruct Xcode to forget its dynamic library fetish and use the static versions you provide. You could remove the dylibs from the system library path entirely — when the linker fails to find the .dylib version, it will fall back to the .a.

Alternatively, you can explicitly instruct it to use static linking as an environment flag and remove the built-in Xcode linker relationships:

  1. Right-click on your Target (in this case, Perian) and Get Info to bring up the Target Info box.
  2. Go to the Build tab
  3. Select your Build configuration – in this case, probably Deployment
  4. Find or locate OTHER_LD_FLAGS, also called “Other Linker Flags”
  5. Manually add the paths to the .a files in this field, one at a time. If you don’t know the path, right-click -> Get Info on the desired libraries under Link Binary With Library, and it should show you a field called Full Path on the General tab. That’s the path to use. Should look something like /tmp/perian/build/Universal/libavcodec.a
  6. Do the same for all offending libraries – this may include libavcodec, libavformat, libavutil, libmatroska, etc.
  7. Remove the libraries you added to Other Linker Flags from your Xcode’s Target -> Target Name -> Link Binary With Library section, to stop Xcode from auto-generating broken linker flags. This is important.

If done correctly, this should no longer find the wrong type of the library. Your compilation of Perian should proceed as normal.

The Moral of the Story

Xcode has some weird idiosyncratic ways of building its projects. It’s useful to poke at things and find out about these behaviors. Its unnatural preference for dynamic libraries over your explicit instructions is disturbing, though.

Don’t attack your users when they pose a problem, and drop the attitude of: ooo, we’re doing this for free, so fsck off, users. In both problems, the exact same attitude surfaced amongst a few of the devs. Arrogance (“we don’t have the responsibility to do anything for you”) and snarky dismissal (‘Well, there’s your problem. We don’t build libavcodec.dylib nor do we place it in /usr/local/lib.’) helps no one. No matter what you thought your build script did, the linker is heading off toward /usr/local/lib, and you better damn well find out why.

It’s obvious that there’s something peculiar going on with the link process, and the user tried to make it clear multiple times (“You missed my point. We build a .a file and we don’t put it in /usr/local/lib.”), but no one seems to pay attention. Following that thread, the user is increasingly frustrated and expresses that, and now the dev’s hammer of arrogance comes down hard ( “You looked like a prick here, to be honest.” — really? not your obtuse-ness or your blindness to a user need?)

When you have a “works-for-me” situation, it’s especially important to communicate effectively with the user. In most cases, you have no idea why it blows up for the user and not for you. Admitting it is fine — could be a very localized issue for that user, or could reveal a subtle systemic problem in your code – either way, it’s useful to have that established. I realize we’re all very busy people, but at the end of the day it’s still your project. Either acknowledge you have no idea what’s going on and solicit help and cooperation with the user who’s having this issue, or man up, put on your detective hat, and fix your own damn problems. The rest of us are busy with our own issues.

The Free Software culture seems to breed this sort of contempt for the user in some devs. Might be a paper in analyzing how corrosive this can be.

UPDATED: I certainly do not mean this last section as a criticism for all Free Software devs. The vast majority of people I’ve talked to in the various communities are really knowledgeable, helpful folks. There are just a few people in some cases, though, who seem to think that if the user can’t make the software work, it’s somehow the user’s fault. It’s a joint problem to be solved together, not “you’re an idiot if you can’t make it work”.

UPDATED 2 (9-29-2008): Perian 1.1.1 has been released. This should fix the original problem with H.264 and .avi files that prompted this post.

transcode 1.0.5 invalid immediate error on Intel OS X

Posted on by yiming

If you’re compiling transcode 1.0.5 from source on OS X Intel (for some odd reason, as I’ve just had to do on 10.5.4), your compile run may blow up in aclib with:


tcmemcpy.c:30:missing or invalid immediate expression `0b111' taken as 0
tcmemcpy.c:30:suffix or operands invalid for `and'
tcmemcpy.c:42:missing or invalid immediate expression `0b1000' taken as 0
tcmemcpy.c:42:suffix or operands invalid for `test'
tcmemcpy.c:52:missing or invalid immediate expression `0b1111' taken as 0

The problem here appears to be that the code in tcmemcpy.c under x86 and x86_64 relies on some inline assembly features not supported by all compiler/assemblers. Namely, the use of 0b (binary immediate) operands in assembly instructions is not apparently supported during compilation under Apple gcc and gcc-4.2.

Setting the CCAS flag did not seem to help. A quick fix, then, is to convert all binary immediates in the asm instructions to hex or decimal.

Thankfully, these operands only existed in tcmemcpy, so there’s not too much work. Look for 0bxxxx immediates and convert them to their hex or decimal equivalent. For example, 0b111 is 0x7, 0b1000 is 0x8, 0b1111 is 0xf. Thus, a line like:
and $0b111, %%eax # ... which is the number of bytes to copyn

becomes
and $0x7, %%eax # ... which is the number of bytes to copyn

Once these operands are modified, the compiler should no longer complain about invalid immediate expressions.

In any case, the latest CVS version of transcode should have resolved these problems already. The workaround should only be necessary if you still must compile the release tarball for 1.0.5.

Updated: There is a second potential problem if you were using ffmpeg-SVN and transcode 1.0.5, in which its hard-coded include lookup for avcodec.h always looks for ffmpeg/avcodec.h. The SVN version of ffmpeg has moved these headers to libavcodec/avcodec.h. A path patch for transcode 1.0.5 and ffmpeg is available, which basically boils down to changing the #include headers and the configure.in file to look for the new path.

Updated: A third problem has now cropped up, with the ffmpeg API change that moved AVCodecContext’s bits_per_sample to bits_per_coded_sample. A global search and replace seems to work for now. See this also in my post on compiling Perian.

libpng12.*.dylib related compile failures on OS X 10.5

Posted on by yiming
broken libpng screenshot

If you’ve been having problems compiling various Unix packages on OS X 10.5.4, and that your compile run fails mysteriously with something like:

i686-apple-darwin9-gcc: /usr/X11/lib/libpng12.0.26.0.dylib: No such file or directory

One strange yet very likely explanation: your libtool archive file /usr/X11/lib/libpng12.la is lying about the location of the shared library for libpng12 — namely, that a file called /usr/X11/lib/libpng12.0.26.0.dylib exists and should be used for linking against libpng12. However, if you actually look in /usr/X11/lib, no such file exists – perhaps you might have libpng.0.24.0.dylib, but not .26. Therefore, packages that make use of this incorrect libtool archive metadata are suitably confused, causing the compiler to bail out when trying to link against this non-existent file.

Since libtool archive (.la) files are text-based, you can open it up in emacs. The quick and dirty fix to this is to simply change the offending library_names and the current and age properties to the correct numbers. In my case, the libpng sitting in /usr/X11/lib was .24, and so I string-replaced all the values in those three properties from 26 to 24. The compilation then proceeded normally.

The long term solution, of course, is to track down what put a wrong .la file there in the first place. I suspect Xcode 3.1 and Mac OS X SDK 10.5, which shipped with the latest iPhone SDK.

UPDATED 9/8/2008:
In the comments section, I’ve been informed that the X11SDK package in Xcode 3.1 in the culprit. Thanks Anonymous!

Processing monitoring and Settlers of Catan save failure

Posted on by yiming

process monitor finding Catan issues
Some versions of Bigfish Games’ Settlers of Catan (a faithful reproduction of the board game) have a strange issue, in which under certain operating contexts, it will not save a game. The error message reported is a generic and not-at-all useful ” an error has occurred while saving “. I suspected this was due to the fact that it failed to create a savegame directory, and indeed, a bit of sleuthing indicates that on Windows XP, the directory at C:\Documents and Settings\All Users\Application Data\Microsoft\MSN Games\Catan is missing (obviously on Vista, this would be somewhere else – probably C:\Users\…). Instead of creating this directory, Catan simply fails to save the game. The program runs fine otherwise.

Of course, it was not obvious where Catan was trying to save its games – finding out that missing directory was the culprit took a bit of investigation. I took a wild stab at the start by creating a “save” directory in its own program files directory. No such luck. Time to bring out the big guns.

A number of ways could have been used, but one is to use the awesome Sysinternals tool Process Monitor, or Procmon.exe. It tracks events and calls from a process, such as filesystem accesses, and has advanced filtering capability to organize and show only the events of interest to a debugging human.

With ProcMon, I simply filtered on the Catan process and tried to save a game as foo. Then, viewing the event log (screenshot 1), it was obvious that CreateFile calls to create foo.sav failed, with the exact target path specified. A quick Windows Explorer excursion confirms that the path does not exist. Creating that directory, of course, solved the savegame problem.

The moral of the story is that ProcMon is a fine tool for tracking mysterious interactions between an application and a system. For something like failing to make a saved game (in this narrow gamig context) or various system-related errors in general (especially when you lack the source code to debug in depth), it sometimes pays to examine the exact sequence of calls and events that led up to the failure. The solution could be very trivial, if you only knew what and where things failed.