Trivial Bugs

if ()

Maxim Kuvyrkov — Sun, 02 May 2010 09:00:51 +0000

A beautiful bug came to my attention several months ago.

An RTOS was being ported from a proprietary compiler to GCC. I fixed a myriad of issues with inline assembly and other quirks, and RTOS was beginning to come to life. But for some reason it worked unimaginably slow — to the point of being indistinguishable from halt.

Investigation revealed that all the CPU did was checking board’s SDRAM and checking it successfully, but over and over again. As the board had 256MB of RAM, every iteration took a while.

The RAM check was a part of RTOS’ main loop and was guarded like so

: extern uchar __VERIFY_ENABLE[]; #define VERIFY_ENABLE ((uint_32)__VERIFY_ENABLE)

: if (VERIFY_ENABLE) { ; }

So where is the bug? Being user-friendly, RTOS developers provided pre-built RTOS binaries that are ready to be linked into final binary image. They also allowed users to disable or enable the check of RAM by setting __VERIFY_ENABLE symbol in the linker script to zero or non-zero value. Neat, isn’t it? Obviously, for a board with lots of RAM, the check should not be enabled, as that would slow the system to a crawl.

And this arrangement worked just fine with a proprietary compiler. GCC, however, being a smart compiler, inferred that as __VERIFY_ENABLE was an address of an array, it could not possibly be NULL. Consequently, if ((uint_32)__VERIFY_ENABLE != NULL) was always true, so GCC removed the condition in the if-statement. Hurray for optimizations!

Fixing this problem required changing just several bytes. While GCC could prove that __VERIFY_ENABLE was not zero, it couldn’t prove that __VERIFY_ENABLE was not one. Substituting comparison of __VERIFY_ENABLE from "!= NULL to "== 1" solved the problem.


#define VERIFY_ENABLE ((uint_32)__VERIFY_ENABLE == 1)
It's that simple. I added these few bytes and the RTOS came finally to life. It turned out this bug was the last one blocking the RTOS from working.



Implementing Compare-And-Swap
Maxim Kuvyrkov — Thu, 10 Dec 2009 09:11:54 +0000
Do you know how to implement a compare-and-swap instruction (CAS) in an instruction set (ISA) which has only a test-and-set (TAS) for an atomic read-modify-write?
Take into account that the implementation should be
(a) fast
and
(b) behave as closely to a real CPU instruction as possible,
since the final product needs to be integrated into GLIBC.
I know a couple of solutions to this problem, but I’m wondering if you know better ways to do CAS which you’d like to share in comments to this post. One approach I tackled is quite complicated and involves kernel magic. The other, having an unfortunate side-effect, is not quite acceptable for general purpose use in GLIBC.
So what are your ideas? I promise to share mine in detail in the next post.

         


Function pointers
Maxim Kuvyrkov — Thu, 26 Nov 2009 19:18:35 +0000
Here is a story about a sneaky bug that got in the way of a hacker, who had to do some pretty important stuff in assembly.
Being a rational hacker, he didn’t start working from scratch, but turned to libraries. Say, he used a function from a dlopen()‘ed library and called it from the assembly code. In C code he wrote:
void *dostuff_ptr;
...
dostuff_ptr = dlsym ("dostuff");
In the assembly he then casually called dostuff_ptr to do the required pretty important stuff.
Being a smart hacker, he decided to re-implement the library function in assembly to show everyone his talent.  And he declared a pointer to his implementation:
extern void *mydostuff;
Being a user-friendly hacker, he allowed users to choose between the library’s implementation of dostuff() and his own by conditionally assigning somewhere in the code dostuff_ptr to the right address:
dostuff_ptr = use_mydostuff_p  ?  mydostuff : dlsym ();
How surprised was the hacker to see that once use_mydostuff_p was true things stopped working.
To fix the problem he launched a debugger and spent some time investigating code.  As you know, the amount of time it takes to debug a problem is very much dependent on how much you trust your toolchain. Well, our talented hacker was working on a new feature which affected the toolchain, and his level of trust was minimal.  So he spent a lot of time debugging, debugging and debugging again.
Anyway, he eventually discovered that dostuff_ptr = mydostuff; puts a bogus value into dostuff_ptr. After taking a close look at the value in dostuff_ptr he noticed that the value in dostuff_ptr is a binary code of the first instructions of mydostuff().  It turned out instead of loading the address of mydostuff() the code loaded contents of mydostuff() into dostuff_ptr.  Well, why did this happen to our good hacker? He simply did not define mydostuff correctly!  Instead of
extern void *mydostuff;
it should’ve been
extern void mydostuff (void);
!
The end.

         


Debugging GLIBC
Maxim Kuvyrkov — Sun, 01 Nov 2009 12:25:47 +0000
When I first started working on GLIBC development, one of the challenges was to understand how to debug GLIBC.  All the usual attempts to compile GLIBC without optimizations ended up with GLIBC grumpily complaining that it cannot be compiled that way.  The development FAQ says
In the early startup of the dynamic loader (_dl_start), before relocation of the PLT, you cannot make function calls. You must inline the functions you will use during early startup, or call compiler

builtins (__builtin_*).
Without optimizations enabled GNU CC will not inline functions. The early startup of the dynamic loader will make function calls via an unrelocated PLT and crash.
OK, fair enough. Can the optimized code be perhaps then debugged, since after all GCC generates adequate debug information even when optimizations are enabled?  Well, not really.  GLIBC is a carefully written piece of software and people who wrote it (a) know quite a lot about compilers and (b) use that knowledge to squeeze every drop of optimizations out of the code.  So though the resulting debug information is usually enough for users to get a sensible backtrace and report bugs to the developers, it is almost useless for single-stepping in a debugger.
The two main problems with single-stepping optimized GLIBC are (1) line numbers and (2) optimized out variables.  The line numbers at -O2 get screwed up beyond repair.   For example, when you try to single step the runtime linker while it is loading dependencies and resolving symbols, the debugger will be thrown between source files and functions at every other line.  So to make the situation more bearable you need to add -fno-schedule-insns -fno-schedule-insns2 to the build flags.
I don’t really know how to properly fix problem (2) with variables being optimized out.  What I usually do is just recompile the only file I’m interested in without optimizations and relink GLIBC.  Normal GLIBC development rarely requires single-stepping the runtime linker before it self-relocates, so try, it might work for you too!
My recipe for debuggable GLIBC is the following:


Build GLIBC, save the log.
Find the command line in the log that compiles the file you’re interested in.  Note: most files are compiled into file.o and file.os; the first one goes into the static library, and the second — into the shared.  If you’re not sure which one you need, it’s probably the .os version.
Go to the source directory in which the file is located.  It is the subdirectory in the source tree which stands right before the file.os in the output destination.  It took me several builds to understand that after you run make in the build tree the first command  is cd  — funny, eh?
Add -E -dD to the end of the compile line and change the output destination to a directory outside both build and source trees.  This way you will get the preprocessed source.  Take a look at it.  Most likely you’d want to remove the original line information from it (sed “/#\ .*/d” < file.i > file.c) and reformat it (indent < file.c > file2.c) to unravel all the hidden sorcery in GLIBC macros.
Remove the -include libc-symbols.h from the command line (it’s the header through which that “glibc cannot be compiled without optimization” error gets in!) and remove -O2 from the optimization flags.
Recompile file.
Rerun make to relink the libraries, the runtime linker and the utilities.  Check the log to make sure only things that were intended for update were updated.
If the make process didn’t notice that file.o was changed, force rebuild by removing the top-level binary, e.g., remove libc-pic.a to force rebuild of libc.so.


Voilà!  Debug at will!

         


Debugging TLS storage
Maxim Kuvyrkov — Wed, 14 Oct 2009 16:14:43 +0000
For background information on TLS check out Wikipedia.
Some time ago I completed implementing TLS/NPTL support for m68k/ColdFire architecture. As a final touch I added support to GDB/gdbserver to handle thread-local variables and used the following simple test  to check that gdb/gdbserver handles TLS properly:
static int __thread foo;
int
main ()
{
  foo = 123;
  return foo;
}
Compiled as:
gcc -g -o test test.c
Then I set a breakpoint in main() and tried printing out the value of foo in a debugger. This didn’t work; as a matter of fact, you can try the above on any architecture with TLS/NPTL support and GDB will just bail out with error that it can’t get to the TLS storage.
After a couple of hours of debugging GDB (which entails running gdbserver under gdbserver and gdb under gdb) I determined that gdbserver doesn’t activate TLS handling because it doesn’t find ‘nptl-version’ symbol.  Following this lead it was easy to find out that:
a) half of the TLS debugging support is in GLIBC’s libthread_db and this library is linked into gdbserver to provide interface between the debugger and GLIBC;
b) the other half is in GLIBC’s libpthread.so, including the ‘nptl-version’ symbol;
c) gdbserver is expecting the ‘nptl-version’ symbol to be defined in the program space, since it reaches the libpthread.so’s part of the TLS/NPTL debugging support only via getting symbols from the program space.
Therefore, the problem with my original testcase turned out to be in not linking it against libpthread.so.  Rebuilding the testcase with
gcc -g -o test -lpthread test.c
solved the problem.  M68k/ColdFire TLS/NPTL support was implemented.

         


id-shared libraries and symbol overriding
Maxim Kuvyrkov — Wed, 30 Sep 2009 16:00:45 +0000
uClinux id-shared libraries and custom ‘new’
uClinux is a flavor of Linux that is designed to run on MMU-less processors,
usually, those are microcontrollers with RAM and flash onboard.  That’s not
the topic of today discussion though; the centerpiece today is how one manages
to use libraries in flat memory model and what implications may jump out of the
corner and bite you in the arse.
uClinux flat memory model means that all the code share the same address space.
Among other things this has an implication that all the applications should be
mapped at different addresses.
Another fact about uClinux is that binaries are usually in the bFLT format;
this format is very minimalistic and doesn’t allow run-time relocations.
This last detail makes sharing of libraries on such a system kind of a trick.
The explanation of how id-shared libraries work is a topic for another discussion.  The fact is that the library’s code stay intact once the library is loaded.  ’Intact’ is this context mainly means the library is not being relocated for
every application that uses it.  This makes symbol preemption within the
library impossible; i.e., one cannot define a-best-of-the-best implementation
of library’s function A and make the library use it.  The example I’ve
recently came about defines C++ operator ‘new’ in an application and expects
libstdc++ library to use it for internal memory allocation.  Well, that didn’t
quite happen.
Let this be a warning to developers who write not-very-big applications that
may be useful to use in a washing machine or a toaster.  A C++ timer
application that allocates a string to show something on a display comes to mind  
First of all, for applications to be able to access libraries they
[the libraries] should be mapped at some constant addresses.  This is where
‘id’ in ‘id-shared’ comes into play.  On such a system each library is compiled
as a id=1, …, id=N library and the kernel then loads the first library at
constant_offset_1, …, and the Nth library in constant_offset_N.  The offsets
are magic numbers that do not change overtime.Some time ago I stumbled upon a bug called ______. The C++ operator was defined as ‘new’ in an application and was expected to be used by libstdc++ library for internal memory allocation. Well, that didn’t happen. 
Some time ago I stumbled upon a bug in a C++ application.  A C++ ‘new’ operator was defined in an application and was expected to override the default definition in the libstdc++ library, even the library’s internal references.  Well, that didn’t happen.
Instead the libstdc++ library was merrily using its default memory allocator, ignoring the override.  For some reason symbol preemption was not working for the shared library.  Interestingly, the erratic behavior appeared only on a uClinux system.
uClinux is a flavor of Linux designed to run on MMU-less processors.  uClinux has a flat memory model that obligates all code to share the same address space.  In uClinux binaries are usually in the bFLT format — a very minimalist format that doesn’t allow run-time relocations.
The last peculiarity of uClinux turned out to be responsible for the problem.  As no run-time relocations are allowed in the bFLT format there is no way for an executable to tailor the library code to its needs.  This is quite a reasonable constraint given that all the executables share the address space and the libraries.
Well, this is one of those bugs that we just have to live with.

         


ACM-ICPC challenge: Factorial Frequencies
Maxim Kuvyrkov — Fri, 18 Sep 2009 07:43:49 +0000
To make my leisure time more challenging I have decided to start solving at least one ACM-ICPC contest problem a week. Last Thursday I embraced factorial frequencies problem dating way back to 1993.
Problem gist:
In an attempt to bolster her sagging palm-reading business, Madam  Phoenix has decided to offer several numerological treats to her  customers.  She has been able to convince them that the frequency  of occurrence of the digits in the decimal representation of  factorials bear witness to their futures.  Unlike palm-reading,  however, she can’t just conjure up these frequencies, so she has  employed you to determine these values.
Recall that the definition of n! (that is, n factorial) is just 1×2×3×…×n.  As she expects to use the day of the week, the day of the month, or the day of the year as the value of n, you must be able to determine the number of occurrences of each decimal digit in numbers as large as 366 factorial (366!), which has 781 digits.
The input data for the program is simply a list of integers for which  the digit counts are desired.  All of these input values will be less  than or equal to 366 and greater than 0, except for the last integer,  which will be zero.  Don’t bother to process this zero value; just  stop your program at that point.  The output format isn’t too  critical.
Well, I wanted the solution to be elegant. And what could be more elegant than prime numbers! So I decided to do a prime factorization of the factorial. The evident first step involved counting the zeroes at the factorial’s end, which was pretty easy. The number of zeroes equals to the exponent of ’5′ in the prime factorization, as you can get a ’10′ only by multiplying a ’5′ by a ’2′.  It is unfortunate though that the number of ’2′s in factorization is certainly greater then the number of ’5′s.
Done with zeroes, I understood that I had no idea how to count the other digits without calculating the whole number of the factorial — which is what I tried elegantly to avoid in the first place. OK then. Perhaps, this number stripped of zeroes can now fit into the 64 bits and the usual CPU arithmetic can be applied?  Umm, but no.  There are 366/2 = 183 ’2′s in the factorization of 366! and only 366/5 = 72 ’5′s.  Therefore, we are left with 111 unaccountable ’2′s, not to mention all the other primes.  This will by no means not fit into 64 bits.
Hmm… So I was left with the easy solution of just computing the factorial in decimal arithmetic. It’s just a 700-digit number after all.  In no time I wrote a program, and it did the job.  The end.
Hey, do you see any better solutions?

         


Signal frame ABI
Maxim Kuvyrkov — Sat, 05 Sep 2009 13:07:41 +0000
 
 
Signal frame ABI is a special intimate relationship between the kernel and the unwinding library. Software developers don’t usually have the time to wonder how the signals are processed and the control is transfered back and forth between the user-space signal handler and the kernel. Myself included… till I had several GLIBC tests fail. Then I started to wonder.
Well, there is no magic in the kernel’s transferring control to user-space — it’s the kernel, the master, the tux, it can do whatever it wants (designed to want)! But what happens after the signal handler returns? Moreover, what will happen if for various reasons we need to show the full backtrace or unwind?
Have you heard about ‘syscall trampoline’? It is a blob the kernel arranges to be placed somewhere and then points the return address of the signal handler to it. The blob is then processed just like any other syscall asking the kernel to do a favor to the user space.
How do we unwind from a signal? We just need to peek into the code where the control would be returned, and if it resembles the trampoline, then the frame must be that of a signal!
Great! Now imagine what happens if the trampoline code gets changed in the kernel. The trampoline that no header will agree to have and that has to be, therefore, hardcoded in the unwinding library. Everything breaks: the unwinding code stops recognizing the signal frames, we loose our precious backtrace and exception cleanup handlers. What we get instead is several hours of exciting debugging.
Having found the bug, I leaped with joy and simply added another trampoline blob to the library.
What a good hunt it was.