Challenges of shared library environments, Part 2

This is a small article series concerned with static initialization and destruction.

This time we take a look at the atexit call. This is a C standard library function, that registers callbacks with the exit handler:

atexit(3):

The atexit() function registers the given function to be
called at program exit, whether via exit(3) or via
return from the program's main().  Functions so registered
are called in reverse order; no arguments are passed.

Unfortunately on the three platforms (MacOS, Linux, FreeBSD) I tested, none of them was conforming. If you run the test program atexit-breakage yourself, you will invariably notice, that bar_exit, which is registered via atexit is called before main returns or exit is called.

-> dlopen
void bar(void)
-- install bar_atexit
<- dlclose
void bar_exit(void)
-- install main_atexit
exit
void main_atexit(void)

Actually it is called during dlclose, which is the command to unmount a shared library.

Linux is even worse. You can get into a scenario, where dlclose isn’t even involved, yet the order of atexit callbacks is not the reverse order. See: ld-so-breakage

Linux, shared libraries, atexit: a little history

I am picking out Linux here, because I invested a non-neglible amount of time with these bugs, and that was mainly on Linux.

ELF is the shared library file format on Linux. The ELF specification prescribes atexit for use as the shared library destructor mechanism:

Similarly, shared objects may have termination functions, which
are executed with the atexit(BA_OS) mechanism after the base
process begins its termination sequence.

Termination functions are a ELF feature, that can be conveniently accessed with __attribute__((destructor)) using gcc and clang.

Since there is no mention of unloading a shared library in the document, I think it’s safe to assume, that unloading was not a concern at the time of writing. Technically then the use of atexit makes sense and is fine.

Shared library unloading

But at some point in time, shared library unloading was added to Linux (dlopen/dlclose), and that were things went wrong. You can not use atexit anymore for shared library destructors, since these do not happen at “base process termination”, but at any time dlclose gets invoked.

What is supposed to happen is written down in the Linux Core Base documentation in __cxa_finalize. Linux actually wants to implement the Itanium C++ ABI, which explains in much more detail how atexit is supposed to be treated so C and C++ stay compatible.

If you follow the algorithm described in the Itanium C++ ABI, you will see that atexit handlers are treated in a special way: they are saved with a NULL shared library handle. On dlclose only termination functions with a handle != NULL should be called. This would be all well and conforming and atexit would only be called at process end.

But Linux calls all atexit functions of a shared library at the time of dlclose… And it seems most other OS as well.

Circumvention in mulle-atexit

Since I need to have a dependable form of post-process destruction for my tests, I am writing mulle_atexit This will do what atexit should be doing in a cross-platform manner.

Conclusion

No tested OS upholds atexit semantics. Linux can neither guarantee the point in time when atexit functions are run, nor can it guarantee the reverse order property.

So atexit is broken everywhere, do not use it. As inertia is the strongest force in the universe, I would expect that atexit will remain broken for a long time.

Itanium C++ ABI Algorithm Explanation:

The runtime library shall maintain a list of termination functions
with the following information about each:

    A function pointer (a pointer to a function descriptor on Itanium).
    A void* operand to be passed to the function.
    A void* handle for the home DSO of the entry (below).

That translates to C as :

struct termination_function
{
   void  (*function_pointer)( void *);
   void  *operand;
   void  *__dso_handle;
}

Now when a C or C++ coder writes atexit this will happen:

When the user registers exit functions with atexit, they
should be registered with NULL parameters and DSO handles,
i.e. __cxa_atexit ( f, NULL, NULL );

So in the atexit list we have an entry { f1, 0, 0 }. For a C++ constructor or __attribute__((destructor)) we would have:

After constructing a global (or local static) object,
that will require destruction on exit, a termination
function is registered as follows:
  extern "C" int __cxa_atexit ( void (*f)(void *), void *p, void *d );

Where d is the shared library handle (returned by dlopen for instance) So the entry would be { f2, p, d }, where d is not NULL.

So our table now looks like this:

struct termination_function   table[] =
{
   { f1, 0, 0 },
   { f2, p, d }
}

Now comes the interesting part, what happens at ‘dlclose’ time ?

When __cxa_finalize(d) is called, it should walk
the termination function list, calling each in turn
if d matches __dso_handle for the termination function
entry. If d == NULL, it should call all of them.

So dlclose( d) will call __cxa_finalize( d). The handle for a specific shared library is != NULL, so the atexit installed handler will not be called! d with the value NULL is called from the base process at termination time.