I got rid of the brown color, maybe by chance or maybe also because I rewrote my "msize" routine to fight of pointers, that obviously are not in my allocation space. Now I can launch TextEdit.app in pristine white! With lots of debug output spouting out, I was able to make a lot of tweaks to the Mullocator, so that it (should) behave well also in a Desktop application.
Well almost.
The problem is that TextEdit.app is leaking when running with the Mullocator. It know this sounds so wrong :) But I am tracing all the "malloc/realloc/free" calls that are coming in, and I get twice as many "malloc" calls as I get free calls, when TextEdit.app is idling (and the cursor is blinking). So the problem can't be in my code. The problem can't be in TextEdit.app either, because with the normal Apple malloc it doesn't leak. Somehow either some memory is freed behind my back (how ?) or somehow my code induces Carbon/Cocoa to not free as much memory as usually.
This has me stumped at the moment.