As programmers, we face it many times: a build fails on only our system. This is one such tale.

Conflicting declarations in zeromq

We build many system software into a custom FreeBSD platform. The first error message I got was while building zeromq. I had freshly synced a branch, merged some changes from another branch, and started a build.

The compiler complained:

CXX    libzmq_la-gssapi_mechanism_base.lo
In file included from gssapi_mechanism_base.hpp:30,
from gssapi_mechanism_base.cpp:34:
/usr/include/gssapi/gssapi.h:316: error: previous declaration of 'gss_OID_desc_struct* GSS_KRB5_NT_PRINCIPAL_NAME' with 'C++' linkage
/usr/include/gssapi/gssapi_krb5.h:49: error: conflicts with new declaration with 'C' linkage

The error means that we are listing symbols two times, and the two listings conflict with each other. The error happened only on this branch. It built fine on the build server and on other branches I had synced on my system.

I wanted to make progress past this problem quickly. Google showed that the gssapi.h on Mac had an embedded extern "C", but mine did not. I added this to my /usr/include/gssapi/gssapi.h, and restarted the build. The error went away.

Missing MD2 references in net-snmp

The build now stopped in a different package. This time, it was a failure while running configure to build net-snmp. The config.log had this to say:

configure:5735: gcc -I../zeromq-4.1.0/include [...]  -L../zeromq-4.1.0/src/.libs -L/usr/obj/.../secure/lib/libcrypto -lcrypto  -R/usr/obj/.../secure/lib/libcrypto conftest.c -lzmq >&5
/usr/lib/libhx509.so.10: undefined reference to `MD2_Init'
/usr/lib/libhx509.so.10: undefined reference to `MD2_Final'
/usr/lib/libhx509.so.10: undefined reference to `MD2_Update'

The errors mean that we depend on MD2 encryption routines, but we cannot find them in any of the libraries specified. I took the conftest.c source from the same file and tried to compile it with the same options. I got the same error.

Notice that we link with a custom crypto library above. Turns out our custom library does not have MD2 support. What if I tried the standard library? When I used the standard crypto library, I didn’t get those errors! Because it had those routines (‘T’):

/usr/ports/security/openssl$ nm ./work/openssl-1.0.1e/libcrypto.so | grep MD2
000000000006ce20 T MD2
000000000006cc60 T MD2_Final
000000000006cdd0 T MD2_Init
000000000006cce0 T MD2_Update
000000000006cb00 T MD2_options
000000000015eea0 R MD2_version

But, but, it is working for others! Everyone is using the same custom library! What went wrong for me?

Notice that the other library it depends on above is the zeromq library (-lzmq). I replaced its location with my working branch, and that worked too!

(FAIL)

gcc [...] -L../zeromq-4.1.0/src/.libs test.o -lzmq [...]

(OK)

gcc [...] -L../working_branch/zeromq-4.1.0/src/.libs test.o -lzmq [...]

Back to zeromq

So is something fishy in my zeromq library? Maybe the error I saw earlier that I “fixed” had to do something about it?

I got rid of the changes I made earlier in /usr/include/gssapi/gssapi.h and started a build again. It failed with the same declaration conflict error as earlier.

Now, I built zeromq on the other branch where it worked and compared the build output.

At first glance, they looked the same.

To understand the compilation more than the bare CXX, I checked if there was a way. ./configure --help showed that I could do make V=1. I did that, and noticed that there was a -DHAVE_CONFIG_H option. This option means there is a separate config file with “discovered”, i.e. configured features. A search brought up config.hpp, and sure enough, I found this in the broken build:

/* BREAKS BUILD */
/* Define to 1 if you have the `gssapi_krb5' library (-lgssapi_krb5). */
#define HAVE_LIBGSSAPI_KRB5 1

How did this happen? I looked at config.log in both branches:

(FAIL)

configure:17985: checking for gss_init_sec_context in -lgssapi_krb5
configure:18010: gcc -std=gnu99 -o conftest -g -O2 -D__BSD_VISIBLE -D_REENTRANT -D_THREAD_SAFE   conftest.c -lgssapi_krb5  -lrt -lpthread  >&5
configure:18010: $? = 0
configure:18019: result: yes

(OK)

configure:17967: checking for gss_init_sec_context in -lgssapi_krb5
configure:17992: gcc -std=gnu99 -o conftest -g -O2 -D__BSD_VISIBLE -D_REENTRANT -D_THREAD_SAFE   conftest.c -lgssapi_krb5  -lrt -lpthread  >&5
configure:17992: $? = 0
configure:18004: result: no
configure:18014: WARNING: libgssapi_krb5 is needed for GSSAPI security

The same command returns “yes” in one branch, but “no” in another!? What gives?

Never edit generated files

So, I opened up the configure script that was running the command. And this is what I found in the working branch:

# XXX
ac_cv_lib_gssapi_krb5_gss_init_sec_context=no
# XXX

This explains everything. Someone had explicitly “disabled” this library check in the configure script. But this is a generated file! autoconf generates it from configure.ac and configure.in. For some reason, this file was re-generated by my autoconf during the build. When that happened, the customization was dropped!

Why this did not happen to me on the other branches, I still do not know. Maybe the timestamps were wrong only on this particular branch? I tried another clean sync and build, but the error did not happen again. I do not understand all the regeneration criteria yet.

Anyway, the fix was to comment a line in configure.ac instead. It’s been a very interesting evening.