Signed integer overflow is undefined behavior in C language. This enables compilers to do all kinds of optimization tricks that can lead to surprising and unexpected behavior between different compilers, compiler versions, and optimization levels.

I encountered something interesting when I tried different versions of GCC and clang with following piece of code:

#include <limits.h>
#include <stdio.h>

int main(void)
{
    int iterations = 0;
    for (int i = INT_MAX - 2; i > 0; i++) {
        iterations++;
    }
    printf("%d\n", iterations);
    return 0;
}

When compiling this with GCC 4.7, GCC 4.9, and clang 3.4 and compilation options for x86_64 architecture, we get following output variations:

$ gcc-4.9 --std=c11 -O0 tst.c && ./a.out
3
$ gcc-4.9 --std=c11 -O3 tst.c && ./a.out
2
$ clang-3.4 --std=c11 -O3 tst.c && ./a.out
4
$ gcc-4.7 --std=c11 -O3 tst.c && ./a.out asdf
<infinite loop...></code>

So let’s analyze these results. The loop for (int i = INT_MAX - 2; i > 0; i++) { should only iterate 3 times until variable i wraps around to negative value due to two’s complement arithmetic. Therefore we would expect iterations variable to be 3. But this only happens when optimizations are turned off. So let’s see why these outputs result in these different values. Is there some clever loop unrolling that completely breaks here or what?

GCC 4.9 -O0

There is nothing really interesting going in the assembly code produced by GCC without optimizations. It looks like what you would expect from this kind of loop for() on the lower level. So the main interest is in optimizations that different compilers do.

GCC 4.9 -O3 and clang 3.4 -O3

GCC 4.9 and clang 3.4 both seem to result in similar machine code. They both optimize the for() loop out and replace the result of it with a (different) constant. GCC 4.9 is just creating code that assigns value 2 to the second parameter for printf() function (see System V Application Binary Interface AMD64 Architecture Processor Supplement section 3.2.3 about parameter passing). Similarly clang 3.4 assigns value 4 to the second parameter for printf().

GCC 4.9 -O3 assembler code for main()

Here we can see how value 2 gets passed as iterations variable:

Dump of assembler code for function main:
5 {
0x00000000004003f0 <+0>: sub $0x8,%rsp

6     int iterations = 0;
7     for (int i = INT_MAX - 2; i > 0; i++) {
8         iterations++;
9     }
10     printf("%d\n", iterations);
0x00000000004003f4 <+4>: mov $0x2,%esi
0x00000000004003f9 <+9>: mov $0x400594,%edi
0x00000000004003fe <+14>: xor %eax,%eax
0x0000000000400400 <+16>: callq 0x4003c0 <printf@plt>

11     return 0;
12 }
0x0000000000400405 <+21>: xor %eax,%eax
0x0000000000400407 <+23>: add $0x8,%rsp
0x000000000040040b <+27>: retq

clang 3.4 -O3 assembler code for main()

Here we can see how value 4 gets passed as iterations variable:

Dump of assembler code for function main:
5 {

6     int iterations = 0;
7     for (int i = INT_MAX - 2; i > 0; i++) {
8         iterations++;
9     }
10     printf("%d\n", iterations);
0x00000000004004e0 <+0>: push %rax
0x00000000004004e1 <+1>: mov $0x400584,%edi
0x00000000004004e6 <+6>: mov $0x4,%esi
0x00000000004004eb <+11>: xor %eax,%eax
0x00000000004004ed <+13>: callq 0x4003b0 <printf@plt>
0x00000000004004f2 <+18>: xor %eax,%eax

11     return 0;
0x00000000004004f4 <+20>: pop %rdx
0x00000000004004f5 <+21>: retq

GCC 4.7 -O3

GCC 4.7 has an interesting result from optimizer that is completely different that could be expected to happen. It basically replaces main() function with just one instruction that does nothing else than jumps at the same address that it resides in and thus creates an infinite loop:

Dump of assembler code for function main:
5 {
0x0000000000400400 <+0>: jmp 0x400400 <main>

Conclusion

This is an excellent example of nasal demons happening when there is undefined behavior ongoing and compilers have free hands to do anything they want. Especially as GCC 4.7 produces really harmful machine code that instead of producing unexpected value as a result, it just makes the program unusable. Other compilers and other architectures could result in different behavior, but let that be something for the future investigation.