ARM cross compiler generating invalid branch argets in standard C functions

I'm working on a custom embedded project (using PlatformIO to setup the build environment), and I have found that calls to standard C functions like memset, memcpy are generating bogus code. The disassembly shows that instructions in both those functions (and others I've tried from stdlib) branch unconditionally to locations that contain no code, which of course causes the MCU (a Cortex M4, the Atmel D51) to hard-fault as it tries executing nonsense code. There are no compiler errors, only runtime errors in the form of hard-faults due to invalid instructions.

I believe it's something wrong with my compilation environment, as PlatformIO has some libraries used for an Adafruit board of the same processor, and that correctly links the functions above. Note that I am cross-compiling from Mac. Just below are the disassemblies for the memset function from the Adafruit and Custom projects:

Adafruit:

0x000012de: 02 44               add r2, r0
0x000012e0: 03 46               mov r3, r0
0x000012e2: 93 42               cmp r3, r2
0x000012e4: 00 d1               bne.n   0x12e8 <memset+10>
0x000012e6: 70 47               bx  lr
0x000012e8: 03 f8 01 1b         strb.w  r1, [r3], #1
0x000012ec: f9 e7               b.n 0x12e2 <memset+4>

Custom:

0x000005b4: 00 30               adds    r0, #0
0x000005b6: a0 e1               b.n 0x8fa           <--- branch to address with no code and hard-fault
0x000005b8: 02 20               movs    r0, #2
0x000005ba: 80 e0               b.n 0x6be
0x000005bc: 02 00               movs    r2, r0
0x000005be: 53 e1               b.n 0x868
0x000005c0: 1e ff 2f 01         vrhadd.u16  d0, d14, d31
0x000005c4: 01 10               asrs    r1, r0, #32
0x000005c6: c3 e4               b.n 0xffffff50
0x000005c8: fb ff ff ea                 ; <UNDEFINED> instruction: 0xfffbeaff

Even without the nonsensical branch targets, the custom version has a totally different form from the one above, suggesting to me that something horribly wrong is happening with linking. I assume the issue is at the linking stage, and not during compilation of individual object files. Linking between files that exist solely within my project causes no issue; local branching is correct. This weirdness seems confined to linking prebuilt libraries.

I should mention that the adafruit stuff also includes Arduino code, so part of that compilation process includes C++ whereas mine is purely C. I have based most of the compiler flags and build environment on the Adafruit project as it was the best reference for my own project, but I am not using arduino in any form.

Here's how the linker is called for each of the two projects

Adafruit (g++ can be interchanged with gcc w/ no error):

arm-none-eabi-g++ -o .pio/build/adafruit_grandcentral_m4/firmware.elf -T flash_without_bootloader.ld -mfloat-abi=hard -mfpu=fpv4-sp-d16 -Os -mcpu=cortex-m4 -mthumb -Wl,--gc-sections -Wl,--check-sections -Wl,--unresolved-symbols=report-all -Wl,--warn-common -Wl,--warn-section-align --specs=nosys.specs --specs=nano.specs .pio/build/adafruit_grandcentral_m4/src/main.cpp.o -L.pio/build/adafruit_grandcentral_m4 -L/Users/work-reese/.platformio/packages/framework-arduino-samd-adafruit/variants/grand_central_m4/linker_scripts/gcc -L/Users/work-reese/.platformio/packages/framework-cmsis/CMSIS/Lib/GCC -Wl,--start-group .pio/build/adafruit_grandcentral_m4/libFrameworkArduinoVariant.a .pio/build/adafruit_grandcentral_m4/libFrameworkArduino.a -larm_cortexM4lf_math -lm -Wl,--end-group

Custom:

arm-none-eabi-ar rc .pio/build/commonsense/libFrameworkCommonSense.a .pio/build/commonsense/FrameworkCommonSense/commonsense.o .pio/build/commonsense/FrameworkCommonSense/cortex_handlers.o .pio/build/commonsense/FrameworkCommonSense/led.o .pio/build/commonsense/FrameworkCommonSense/pinConfig.o .pio/build/commonsense/FrameworkCommonSense/startup.o

arm-none-eabi-ranlib .pio/build/commonsense/libFrameworkCommonSense.a

arm-none-eabi-gcc -o .pio/build/commonsense/firmware.elf -T commonsense_linker.ld -mfpu=fpv4-sp-d16 -mthumb -Wl,--gc-sections -Wl,--check-sections -Wl,--unresolved-symbols=report-all -Wl,--warn-common -Wl,--warn-section-align --specs=nosys.specs --specs=nano.specs -mcpu=cortex-m4 .pio/build/commonsense/src/main.o -L.pio/build/commonsense -L/Users/work-reese/.platformio/packages/toolchain-gccarmnoneeabi/arm-none-eabi/lib -L/Users/work-reese/.platformio/packages/framework-cmsis/CMSIS/Lib/GCC -L/Users/work-reese/.platformio/packages/framework-commonsense/linker -Wl,--start-group .pio/build/commonsense/libFrameworkCommonSense.a -larm_cortexM4lf_math -lc_nano -lm -Wl,--end-group

This is using the arm cross compiler, version 7.2.1, and the toolchain contains distributions for libc, libc_nano, libm, etc. All the necessary libraries appear to be present.

Please note I included a few extra lines for the custom version's linking above so you can see what it's building libFrameworkCommonSense.a from. None of those files include any stdlib calls, although cortex_handlers does not have __libc_init_array in the reset handler because that was also causing hard-faults in the same way memset. The linker script is identical between the two; once again, I borrowed heavily from the adafruit project for the interrupt handlers and startup code, but I haven't seen any actual differences between the environments until now.

Adding the --print-multi-lib option shows several options that should work, namely thumb/v7e-m/fpv4-sp/softfp;@mthumb@march=armv7e-m@mfpu=fpv4-sp-d16@mfloat-abi=softfp which should be selected given the compiler flags. Weirdly, it fails to compile when printing the multilib options, citing that the object files to archive (arm-none-eabi-ar) are not present in the build directory. This is probably of no concern.

Here's the compilation for the main file, which includes the calls to memset and memcpy:

arm-none-eabi-gcc -o .pio/build/commonsense/src/main.o -c -std=gnu11 -mfpu=fpv4-sp-d16 -Og -g3 -mlong-calls --specs=nano.specs -specs=nosys.specs -fdata-sections -ffunction-sections -mfloat-abi=softfp -march=armv7e-m -mfpu=fpv4-sp-d16 -marm -mthumb-interwork -ffunction-sections -fdata-sections -Wall -mthumb -nostdlib --param max-inline-insns-single=500 -mcpu=cortex-m4 -DPLATFORMIO=50003 -D__SAMD51P20A__ -D__SAMD51__ -D__FPU_PRESENT -DARM_MATH_CM4 -DENABLE_CACHE -DVARIANT_QSPI_BAUD_DEFAULT=50000000 -DDEBUG -DADAFRUIT_LINKER -DF_CPU=120000000L -Iinclude -Isrc -I/Users/work-reese/.platformio/packages/framework-cmsis/CMSIS/Include -I/Users/work-reese/.platformio/packages/framework-cmsis-atmel/CMSIS/Device/ATMEL -I/Users/work-reese/.platformio/packages/framework-cmsis-atmel/CMSIS/Device/ATMEL/samd51 -I/Users/work-reese/.platformio/packages/framework-commonsense -I/Users/work-reese/.platformio/packages/framework-commonsense/core -I/Users/work-reese/.platformio/packages/framework-commonsense/hal -I/Users/work-reese/.platformio/packages/framework-commonsense/hal/include -I/Users/work-reese/.platformio/packages/framework-commonsense/hal/utils/include -I/Users/work-reese/.platformio/packages/framework-commonsense/hal/utils/src -I/Users/work-reese/.platformio/packages/framework-commonsense/hal/src -I/Users/work-reese/.platformio/packages/framework-commonsense/hpl -I/Users/work-reese/.platformio/packages/framework-commonsense/hri -I/Users/work-reese/.platformio/packages/framework-commonsense/sample src/main.c

Does anyone know why I would be having this behavior with incorrectly linked library functions? I've bashed on it for nearly a week, throwing many combinations of compiler flags at it to no avail. I feel there's something I'm overlooking, but don't know what. I'm glad to provide any additional information.

Side question: What is __libc_init_array(), and how necessary is it to call during program startup? I see this in the reset handler for adafruit and Atmel Studio projects. It's declared locally as a function prototype in their startup files, but reproducing the same thing in my own environment causes a hardfault as soon as the processor tries calling that function. I should think it is a part of libc or similar.

I have found that calls to standard C functions like memset, memcpy are generating bogus code. The disassembly shows that instructions in both those functions (and others I've tried from stdlib) branch unconditionally to locations that contain no code, which of course causes the MCU (a Cortex M4, the Atmel D51) to hard-fault as it tries executing nonsense code.

Actually, that is ARM code rather than thumb code. When you try to disassemble it as thumb, it's nonsense, but disassemble it as ARM it looks plausible.

Of course, your processor can't execute ARM code, but only thumb code, and in any event even a processor which could would have to encounter it in ARM mode. So no mystery on the hard fault.

What is unclear is exactly how you are ending up with ARM code in a thumb project. At first glance it appears your actual invocations of the compiler are specifying thumb, so I'd guess the problem code is actually arriving as a result of linking the wrong library.