C bitwise AND gives different result with -O0 and -O2

I'm working on a PC emulator using both Bochs and DOSBox as references.

When emulating the "NEG Ed" instruction (two's complement negation of a doubleword) I'm getting different results if I compile with -O0 rather than -O2.

This is a test program with just the relevant bits:

#include <stdio.h>
#include <stdint.h>
#include <stdlib.h>
#include <stdbool.h>

int main(int argc, const char **argv)
{
    uint32_t value = strtol(argv[1], NULL, 16);
    uint32_t negation = -(int32_t)(value);
    bool sign = negation & 0x80000000;

    printf("value=%X, negation=%X, sign=%X\n", value, negation, sign);
    
    return 0;
}

The -(int32_t)(value); part is taken from Bochs' NEG_EdM() function; for the equivalent operation DOSBox doesn't cast to signed int.

If you compile this program with GCC 10 using the -O2 option and use the hexadecimal value 0x80000000 as input, you'll get a wrong result for sign:

value=80000000, negation=80000000, sign=0

When compiled with -O0 the result is correct:

value=80000000, negation=80000000, sign=1

Is this undefined behaviour?

As far as I know casting to/from signed and unsigned integers is well defined, as is bitwise & on unsigned values.

Source of Undefined Behavior

The key part of the problem is in the negation of -(int32_t)value.¹

At this point, value is 80000000₁₆ (2³¹). Since that is not representable in int32_t, the conversion is governed by C 2018 6.3.1.3 3, which says the behavior is implementation-defined. GCC 10.2 defines it to wrap modulo 2^N, where the destination width is N bits. Wrapping 80000000₁₆ to int32_t modulo 2³² produces −80000000₁₆.

Then the negation operator, -, is applied. The mathematical negation of −80000000₁₆ is of course 80000000₁₆, but this is not representable in int32_t.² The behavior is governed by C 2018 6.5 5:

If an exceptional condition occurs during the evaluation of an expression (that is, if the result is not mathematically defined or not in the range of representable values for its type), the behavior is undefined.

Thus, the negation has undefined behavior. When -O0 is used, the compiler generates simple direct code. Godbolt shows it generates a negate instruction that wraps, producing the output 80000000₁₆ for the input bits 80000000₁₆ (which would represent −80000000₁₆ as a signed 32-bit integer). When -O2 is used, the compiler performs complicated analysis and transformation of the program, and the lack of defined behavior leaves the compiler free to produce any result. Indeed, Godbolt shows the negate instruction is absent. Effectively, the compiler “knows” that negating an int32_t value can never produce a result with the 2³¹ bit set in a program with defined behavior.

Discussion of Optimization

Consider that the range of values representable in int32_t is −2³¹ to 2³¹−1. The mathematical negations of these are −(2³¹−1) to 2³¹. However, 2³¹ overflows, resulting in an exceptional condition. The range of results that do not overflow is −(2³¹−1) to 2³¹−1. Therefore, in a program with defined behavior, only these results occur, and therefore the compiler may optimize as if only these results occur. In none of these results is the 2³¹ bit set. Therefore, in a program with defined behavior, negation & 0x80000000 is always zero, and the compiler may generate code based on this.

Fix

It appears you want a test for whether the sign bit would be set in an int32_t that is negated using two’s complement, that is, wrapping the result modulo 2³². For this, unsigned arithmetic can be used. If x is either an int32_t value or a uint32_t containing the bits that would represent such a value, then the sign bit of the negated value may be obtained with either of:

bool sign = - (uint32_t) x & 0x80000000u;
bool sign = - (uint32_t) x >> 31;

Footnote

¹ We deduce long is wider than 32 bits. Were it not, strtol("0x80000000", NULL, 16) would return LONG_MAX, per C 2018 7.22.1.4 8. That would be representable in uint32_t and int32_t, so value would be initialized to LONG_MAX, converting to int32_t would keep that value, negation would be −LONG_MAX, and sign would be zero in both the optimized and unoptimized versions of the program.

² If int32_t were narrower than int, the operand would be promoted to int before the negation, and the mathematical result would be representable. That is not the case with the GCC version and options you used, which we can deduce from the observed results.