STM32F7 - Execute code from RAM and flash

I believe I have a problem with my linker script but I'm not sure if that's the true culprit here.

Background:

I'm using an STM32F730 on a project. The uC has 64K flash and 256K of ram. Of that, 64K is TCM, 192k is regular RAM. The project will exceed the 64k of internal flash, so I'm intending on running extra code from RAM. There is external flash on board, so at boot, the uC will execute a load routine from on board flash that reads off board flash via SPI and stores the data to the correct location in RAM. This part works, but only sometimes. Off board flash is programmed in several ways, but I am confident that the code that is needed resides there. Besides, my error occurs when runnign from the debugger, and loading from off chip flash is bypassed.

All issues will refer to running from debugger. No code is loading from off chip flash at this time.

I have long function calls turned on to allow the jump to RAM, which did solve one issue i was having, but not this one.

Problem:

Sometimes, the code works flawlessly, when loaded over the debugger. Other times, the code doesn't work at all, and hard faults. I'm not an embedded expert (I primarily do PCB design and FPGA work), but I've managed to step through the assembly and find strange things that are causing the issue, but not why the issue is happening. My current project has an IMU driver loaded into RAM, and should send the data back over USB CDC. This works when loaded entirely in flash, but not always in RAM. When I turn on debug optimizations, it works, for example, but no optimizations causes the fault.

I've modified my linker as follows:

/* Specify the memory areas */
MEMORY
{
TCM_RAM (xrw)      : ORIGIN = 0x20000000, LENGTH = 64K
CODE_RAM (xrw)      : ORIGIN = 0x20010000, LENGTH = 64K
DATA_RAM (xrw)      : ORIGIN = 0x20020000, LENGTH = 112K
FLASH (rx)      : ORIGIN = 0x8000000, LENGTH = 64K
}

/* Define output sections */
SECTIONS
{
  /* The startup code goes first into FLASH */
  .isr_vector :
  {
    . = ALIGN(4);
    KEEP(*(.isr_vector)) /* Startup code */
    . = ALIGN(4);
  } >FLASH

    /* NP 2019-03-15 - RAM Executable code should be linked to RAM                              */
    /*      NOTE: With code linked this way it WILL NOT RUN WITHOUT THE DEBUGGER OR BOOTLADED   */
    /*              Any loaded code will be wiped from RAM on a power cycle.                    */
  .coderam : 
  {
    . = ALIGN(4);
    __RAM_CODE_SECTION_START = .;
    *(.text.imu*)
    . = ALIGN(4);
    __RAM_CODE_SECTION_END = .;

  }>FLASH//CODE_RAM 


  /* The program code and other data goes into FLASH */
  .text :
  {
    . = ALIGN(4);
    *(.text)           /* .text sections (code) */
    *(.text*)          /* .text* sections (code) */
    *(.glue_7)         /* glue arm to thumb code */
    *(.glue_7t)        /* glue thumb to arm code */
    *(.eh_frame)

    KEEP (*(.init))
    KEEP (*(.fini))

    . = ALIGN(4);
    _etext = .;        /* define a global symbols at end of code */
  } >FLASH

Every imu driver function starts with imu*, so I"m trying to load all imu functions to RAM with the line *(.text.imu*), I tried to specify imu.o, but couldn't figure out the correct syntax after lots of searching and trial and error. I switch between storing imu.* in flash and RAM by the commented bit. Note that when I use flash, it stores it at address 0x0, why is that? I think this is a very beginner question, but I don't understand why.

Issue: I run into an issue in the initialization function, which is the first function call to a RAM piece of code. This is the C Code snippet:

uint8_t imu_init(void)
{
    uint8_t retry_cnt = 0U;

      imu_trans.device_csn_bank = IMU_CSN_BANK;
      imu_trans.device_csn_pin = IMU_CSN_PIN;
      imu_trans.device = DEVICE_IMU;
      imu_trans.speed = SPI_SPEED_6_75MBIT;
      imu_trans.rxBuf = imuRxBuf;
**Continues, but hard fault occurs earlier**

From looking at the assembly, the following code is running in my init function, when running from RAM:

                                          imu_init:
20010000: 0x000080b5 imu_init+0             push    {r7, lr}
20010002: 0x000082b0 imu_init+2             sub     sp, #8
20010004: 0x000000af imu_init+4             add     r7, sp, #0
 69                                         uint8_t retry_cnt = 0U;
20010006: 0x00000023 imu_init+6             movs    r3, #0
20010008: 0x00000000 imu_init+8             movs    r0, r0
 71                                           imu_trans.device_csn_bank = IMU_CSN_BANK;
2001000a: 0x00001d4b imu_init+10            ldr     r3, [pc, #116]  ; (0x20010080 <imu_init+128>)
2001000c: 0x00000000 imu_init+12            movs    r0, r0
2001000e: 0x00000060 imu_init+14            str     r0, [r0, #0]
 72                                           imu_trans.device_csn_pin = IMU_CSN_PIN;
20010010: 0x00fb7100 imu_init+16                    ; <UNDEFINED> instruction: 0xfb000071
20010014: 0x0000001d imu_init+20            adds    r0, r0, #4
20010016: 0x00004a5a imu_init+22            ldrh    r2, [r1, r1]
 73                                           imu_trans.device = DEVICE_IMU;
20010018: 0x0000001b imu_init+24            subs    r0, r0, r4
2001001a: 0x00004b4f imu_init+26            ldr     r7, [pc, #300]  ; (0x20010148 <imu_process_vals+4>)
2001001c: 0x0000f400 imu_init+28            lsls    r4, r6, #3
 74                                           imu_trans.speed = SPI_SPEED_6_75MBIT;

Obviously, it hardfaults on the line 20010010: 0x00fb7100 imu_init+16 ; <UNDEFINED> instruction: 0xfb000071. Interestingly (and here's what I don't get) that isn't in the hex file!?!

*snip*
:020000042001D9
:1000000080B582B000AF0023FB711D4B1D4A5A60C2
:100010001B4B4FF480721A80194B01221A72184B35
*snip*

The beginning of the third line is the hardfault instruction! but memory browser shows:

0x0000000020010000  B082B580 2300AF00 4B1D0000  .µ.°.¯.#...K
0x000000002001000C  60000000 0071FB00 5A4A1D00  ...`.ûq...JZ

So it does look like there is a bad instruction in RAM, but why!? It also looks like 0x2001000C is corrupted too since it doesn't match the hex file. I initially thought that something in my loading process was doing this, but it happens if i set a breakpoint at the reset handler too, so i don't understand the cause. I was thinking it was a SWD corruption, but it's the same every time, so I'm not sure that's a factor either. I originally thought some of those instructions were wait states (mov r0, r0), but now I'm thinking they're corrupted values?

Unsurprisingly, the code when loaded to flash reads the same as the hex file:

0x000000000800AEC4  B082B580 2300AF00 4B1D71FB 605A4A1D F44F4B1B 801A7280  .µ.°.¯.#ûq.K.JZ`.KOô.r..
0x000000000800AEDC  22014B19 4B18721A 729A2202 4A184B16 4B1560DA 611A4A17  .K.".r.K.".r.K.JÚ`.K.J.a

So, TL;DR: 1. If i load everything in flash, it works fine 2. If I load the imu driver to RAM and everything else in flash, it hardfaults 3. If I turn on debug optimizations -Og, it works again, but might move the error somewhere else.

Any help would be greatly appreciated, I've spent about 30 hours trying to get this seemingly simple thing to work (and it did work! then it stopped), and I'm running out of ideas to try.

Thanks, Nick

EDIT

I think i might have traced this one a bit further... I tried loading the same file from the ST link utility and viewing memory, then trying to load the debugger and viewing memory.

Sure enough, the code loaded by the debugger is different than the code loaded by the ST Link util. Code loaded by the ST link util works as expected - USB enumerates and sends out data.

I did a compare, and everything after the second RAM address (everything after 0x20010008) is corrupt.

So this is potentially a debugger setup issue? I'm using an STlinkV3 via SWD - does the project need special setup to run from RAM? Again, sometimes it works fine from RAM, but other times not.

This ended up being a problem with truestudio. I'm not sure if it was the specific version I'm using, but when using the debug feature in truestudio, the debugger would not consistently program RAM.

The workaround is to use STM32CubeProg, but that means no debugging.

Interestingly, simply configuring the ST link as an external tool worked without issues. https://info.atollic.com/hubfs/AppNotes/st_link_utility_as_ext-tool.pdf

Code now runs / executes from flash / ram and transitions between them with no issues.