Debugging mBed-OS 5 using J-Link OpenSDA Firmware and GDB

I am working with a MK64FN1M0xxx12 development kit that supports OpenSDA. To make things easier to debug, I flashed the J-Link firmware that supports the SEGGER debugger command set. Before doing this I made sure my device was officially supported.

To be more verbose, I am trying trying to debug mbed-os 5, specically the uVisor security patches. The uVisor Github page has a debugging write-up that provides instructions for connecting to the device over a GDB session to receive debug prints. I was able to get the J-Link session properly connected, but I cannot seem to actually debug the binary. Further, I cannot seem to properly flash the device. I am able to push the binary to device RAM, though.

Their instructions read:

(gdb) target remote localhost:2331
(gdb) monitor reset
(gdb) monitor halt
(gdb) monitor semihosting enable
(gdb) monitor loadbin ./build/${target}/source/${your_app}.bin 0
(gdb) monitor flash device = ${device_name}
(gdb) load ./build/${target}/source/${your_app}
(gdb) file ./build/${target}/source/${your_app}

So, I tried to replicate their work. Connect to the device over J-Link

$ JLinkGDBServer -device MK64FN1M0xxx12
SEGGER J-Link GDB Server V6.00g Command Line Version

JLinkARM.dll V6.00g (DLL compiled Aug 17 2016 13:20:32)

-----GDB Server start settings-----
GDBInit file:                  none
GDB Server Listening port:     2331
SWO raw output listening port: 2332
Terminal I/O port:             2333
Accept remote connection:      yes
Generate logfile:              off
Verify download:               off
Init regs on start:            off
Silent mode:                   off
Single run mode:               off
Target connection timeout:     0 ms
------J-Link related settings------
J-Link Host interface:         USB
J-Link script:                 none
J-Link settings file:          none
------Target related settings------
Target device:                 MK64FN1M0xxx12
Target interface:              JTAG
Target interface speed:        1000kHz
Target endian:                 little

Connecting to J-Link...
J-Link is connected.
Firmware: J-Link OpenSDA 2 compiled Sep 15 2016 14:57:31
Hardware: V1.00
S/N: 621000000
Checking target voltage...
Target voltage: 3.30 V
Listening on TCP/IP port 2331
Connecting to target...ERROR: Debugger tries to select target interface JTAG.
This interface is not supported by the connected emulator.
Selection will be ignored by the DLL.

J-Link found 1 JTAG device, Total IRLen = 4
JTAG ID: 0x2BA01477 (Cortex-M4)
Connected to target
Waiting for GDB connection...

Establish GDB client session

(gdb) target remote localhost:2331
Remote debugging using localhost:2331
0x200171fe in ?? ()
(gdb) monitor reset
Resetting target
(gdb) monitor halt
(gdb) monitor semihosting enable
Semi-hosting enabled (Handle on BKPT)
(gdb) monitor flash download 1
Flash download enabled
(gdb) shell cp ../.build/K64F/GCC_ARM/mbed-os-example-uvisor.bin /tmp
(gdb) monitor loadbin /tmp/mbed-os-example-uvisor.bin
Binary file loaded successfully (210240 bytes downloaded)
(gdb) monitor flash device = MK64FN1M0xxx12
Selecting device: MK64FN1M0xxx12
(gdb) load ../.build/K64F/GCC_ARM/source/main.o
Loading section .text._ZN4rtos6ThreadC2E10osPrioritymPh, size 0x54 lma 0x0
Loading section .text._ZN16NetworkInterfaceD2Ev, size 0x28 lma 0x0
Loading section .text._ZN16NetworkInterfaceD0Ev, size 0x1c lma 0x0
Loading section .text._ZN4mbed10DigitalOutC2E7PinName, size 0x24 lma 0x0
Loading section .text._ZN4mbed10DigitalOut5writeEi, size 0x1c lma 0x0
Loading section .text._ZN4mbed10DigitalOutaSEi, size 0x1c lma 0x0
Loading section .rodata._ZL10g_main_acl, size 0xc0 lma 0x0
Loading section .rodata.__uvisor_priv_sys_irq_hooks, size 0xc lma 0x0
Loading section .keep.uvisor.bss.boxes, size 0x80 lma 0x0
Loading section .rodata.__uvisor_mode, size 0x4 lma 0x0
Loading section .keep.uvisor.cfgtbl, size 0x28 lma 0x0
Loading section .keep.uvisor.cfgtbl_ptr_first, size 0x4 lma 0x0
Loading section .text._Z30local_ENET_Transmit_IRQHandlerv, size 0x14 lma 0x0
Loading section .text._Z29local_ENET_Receive_IRQHandlerv, size 0x14 lma 0x0
Loading section .text._ZN16NetworkInterfaceC2Ev, size 0x18 lma 0x0
Loading section .text._ZN12EthInterfaceC2Ev, size 0x24 lma 0x0
Loading section .text._ZN12EthInterfaceD2Ev, size 0x30 lma 0x0
Loading section .text._ZN12EthInterfaceD0Ev, size 0x1c lma 0x0
Loading section .text._ZN17EthernetInterfaceC2Ev, size 0x24 lma 0x0
Loading section .rodata, size 0xd4 lma 0x0
Loading section .text._Z16ethernet_connectv, size 0x54 lma 0x0
Loading section .text.main, size 0x174 lma 0x0
Loading section .text._ZN4mbed8CallbackIFvvEEC2EPS1_, size 0x1c lma 0x0
Loading section .text._ZN4mbed8CallbackIFvvEE6attachEPS1_, size 0x30 lma 0x0
Loading section .text._ZN4mbed8CallbackIFvvEE12_staticthunkEPvS3_, size 0x14 lma 0x0
Loading section .rodata._ZTV12EthInterface, size 0x24 lma 0x0
Loading section .rodata._ZTV16NetworkInterface, size 0x18 lma 0x0
Loading section .text._ZN17EthernetInterfaceD2Ev, size 0x30 lma 0x0
Loading section .text._ZN17EthernetInterfaceD0Ev, size 0x1c lma 0x0
Loading section .text._Z41__static_initialization_and_destruction_0ii, size 0x3c lma 0x0
Loading section .text._GLOBAL__sub_I___uvisor_priv_sys_irq_hooks, size 0x10 lma 0x0
Loading section .init_array, size 0x4 lma 0x0
Start address 0x0, load size 1816
Transfer rate: 1773 KB/sec, 56 bytes/write.
(gdb) file ../.build/K64F/GCC_ARM/source/main.o
A program is being debugged already.
Are you sure you want to change the file? (y or n) y
Reading symbols from ../.build/K64F/GCC_ARM/source/main.o...done.
(gdb) c
Continuing.

Program received signal SIGTRAP, Trace/breakpoint trap.
rtos::Thread::Thread (this=<error reading variable: Cannot access memory at address 0xfffffff4>, priority=<error reading variable: Cannot access memory at address 0xfffffff2>, stack_size=<error reading variable: Cannot access memory at address 0xffffffec>, 
    stack_pointer=<error reading variable: Cannot access memory at address 0xffffffe8>) at ../mbed-os/rtos/rtos/Thread.h:42
42          Thread(osPriority priority=osPriorityNormal,
(gdb) 

Resulting J-Link console output

Waiting for GDB connection...Connected to 127.0.0.1
Reading all registers
Read 4 bytes @ address 0x200171FE (Data = 0xEA126221)
Read 2 bytes @ address 0x200171FE (Data = 0x6221)
Resetting target
Halting target CPU...
...Target halted (PC = 0x200171FE)
Semi-hosting enabled (Handle on BKPT)
Flash download enabled
Loading binary file [/tmp/mbed-os-example-uvisor.bin] ...
Downloading 210240 bytes @ address 0x00000000
Binary file loaded successfully (210240 bytes downloaded)
Selecting device: MK64FN1M0xxx12
Downloading 84 bytes @ address 0x00000000
Downloading 40 bytes @ address 0x00000000
Downloading 28 bytes @ address 0x00000000
Downloading 36 bytes @ address 0x00000000
Downloading 28 bytes @ address 0x00000000
Downloading 28 bytes @ address 0x00000000
Downloading 192 bytes @ address 0x00000000
Downloading 12 bytes @ address 0x00000000
Downloading 128 bytes @ address 0x00000000
Downloading 4 bytes @ address 0x00000000
Downloading 40 bytes @ address 0x00000000
Downloading 4 bytes @ address 0x00000000
Downloading 20 bytes @ address 0x00000000
Downloading 20 bytes @ address 0x00000000
Downloading 24 bytes @ address 0x00000000
Downloading 36 bytes @ address 0x00000000
Downloading 48 bytes @ address 0x00000000
Downloading 28 bytes @ address 0x00000000
Downloading 36 bytes @ address 0x00000000
Downloading 212 bytes @ address 0x00000000
Downloading 84 bytes @ address 0x00000000
Downloading 372 bytes @ address 0x00000000
Downloading 28 bytes @ address 0x00000000
Downloading 48 bytes @ address 0x00000000
Downloading 20 bytes @ address 0x00000000
Downloading 36 bytes @ address 0x00000000
Downloading 24 bytes @ address 0x00000000
Downloading 48 bytes @ address 0x00000000
Downloading 28 bytes @ address 0x00000000
Downloading 60 bytes @ address 0x00000000
Downloading 16 bytes @ address 0x00000000
Downloading 4 bytes @ address 0x00000000
Writing register (PC = 0x00000000)
Read 4 bytes @ address 0x00000000 (Data = 0x00000000)
Read 2 bytes @ address 0x00000000 (Data = 0x0000)
Reading all registers
Read 4 bytes @ address 0x00000000 (Data = 0x00000000)
Starting target CPU...
ERROR: Can not read register 15 (R15) while CPU is running
Reading all registers
ERROR: Can not read register 0 (R0) while CPU is running
ERROR: Can not read register 1 (R1) while CPU is running
ERROR: Can not read register 2 (R2) while CPU is running
ERROR: Can not read register 3 (R3) while CPU is running
ERROR: Can not read register 4 (R4) while CPU is running
ERROR: Can not read register 5 (R5) while CPU is running
ERROR: Can not read register 6 (R6) while CPU is running
ERROR: Can not read register 7 (R7) while CPU is running
ERROR: Can not read register 8 (R8) while CPU is running
ERROR: Can not read register 9 (R9) while CPU is running
ERROR: Can not read register 10 (R10) while CPU is running
ERROR: Can not read register 11 (R11) while CPU is running
ERROR: Can not read register 12 (R12) while CPU is running
ERROR: Can not read register 13 (R13) while CPU is running
ERROR: Can not read register 14 (R14) while CPU is running
ERROR: Can not read register 15 (R15) while CPU is running
ERROR: Can not read register 16 (XPSR) while CPU is running
ERROR: Can not read register 17 (MSP) while CPU is running
ERROR: Can not read register 18 (PSP) while CPU is running
ERROR: Can not read register 24 (PRIMASK) while CPU is running
ERROR: Can not read register 25 (BASEPRI) while CPU is running
ERROR: Can not read register 26 (FAULTMASK) while CPU is running
ERROR: Can not read register 27 (CONTROL) while CPU is running
ERROR: Can not read register 32 (FPSCR) while CPU is running
ERROR: Can not read register 33 (FPS0) while CPU is running
ERROR: Can not read register 34 (FPS1) while CPU is running
ERROR: Can not read register 35 (FPS2) while CPU is running
ERROR: Can not read register 36 (FPS3) while CPU is running
ERROR: Can not read register 37 (FPS4) while CPU is running
ERROR: Can not read register 38 (FPS5) while CPU is running
ERROR: Can not read register 39 (FPS6) while CPU is running
ERROR: Can not read register 40 (FPS7) while CPU is running
ERROR: Can not read register 41 (FPS8) while CPU is running
ERROR: Can not read register 42 (FPS9) while CPU is running
ERROR: Can not read register 43 (FPS10) while CPU is running
ERROR: Can not read register 44 (FPS11) while CPU is running
ERROR: Can not read register 45 (FPS12) while CPU is running
ERROR: Can not read register 46 (FPS13) while CPU is running
ERROR: Can not read register 47 (FPS14) while CPU is running
ERROR: Can not read register 48 (FPS15) while CPU is running
ERROR: Can not read register 49 (FPS16) while CPU is running
ERROR: Can not read register 50 (FPS17) while CPU is running
ERROR: Can not read register 51 (FPS18) while CPU is running
ERROR: Can not read register 52 (FPS19) while CPU is running
ERROR: Can not read register 53 (FPS20) while CPU is running
ERROR: Can not read register 54 (FPS21) while CPU is running
ERROR: Can not read register 55 (FPS22) while CPU is running
ERROR: Can not read register 56 (FPS23) while CPU is running
ERROR: Can not read register 57 (FPS24) while CPU is running
ERROR: Can not read register 58 (FPS25) while CPU is running
ERROR: Can not read register 59 (FPS26) while CPU is running
ERROR: Can not read register 60 (FPS27) while CPU is running
ERROR: Can not read register 61 (FPS28) while CPU is running
ERROR: Can not read register 62 (FPS29) while CPU is running
ERROR: Can not read register 63 (FPS30) while CPU is running
ERROR: Can not read register 64 (FPS31) while CPU is running
Read 4 bytes @ address 0x00000000 (Data = 0x00000000)
Read 4 bytes @ address 0x00000000 (Data = 0x00000000)
Read 4 bytes @ address 0x00000000 (Data = 0x00000000)
WARNING: Failed to read memory @ address 0xFFFFFFF4
WARNING: Failed to read memory @ address 0xFFFFFFF2
WARNING: Failed to read memory @ address 0xFFFFFFEC
WARNING: Failed to read memory @ address 0xFFFFFFE8

It seems things are almost working, but I am not well enough versed in GDB to know where things may be failing. From the J-Link console output it appears as if the second load is overwriting the early contents of RAM. If I omit this instruction (but keep the last file command) then then no errors will be reported, but the connection via nc does not display any debug messages

$ nc localhost 2333
SEGGER J-Link GDB Server V6.00g - Terminal output channel

Despite the uVisor debug write-up (following the GDB command list) indicating that many types of messages should be reporting at run-time. In addition to this, my program that was (seemingly) successfully downloaded to the board does not behave as it should when executing (no blinking LEDs, etc).

As I have found a few times before with uVisor (since it is very new), the documentation may be out of date. Also, I found through my research that others required the use of the monitor flash download = 1 command to enable flash programming, although it did not seem to work for me. Although the output of this command reads Flash download enabled, nothing is ever written to flash as far as I can tell (considering the output and elapsed time).

I was able to properly attach and receive debug statements using the following command set:

file ./build/${target}/source/${your_app}.elf
target remote localhost:2331
monitor reset
monitor semihosting enable
load

It seems it is necessary to load from debug symbols from the elf and not delegate loading the binary via the monitor command proxy. Further, it appears the load command considers the last file to be to used as its default target (the *.elf, in this case). This seems strange, however, considering that mBed-OS does not parse ELF headers or load anything ...

At any rate, the output in the J-Link console looks much more like I expected:

Downloading 1024 bytes @ address 0x00000000
Downloading 16 bytes @ address 0x00000400
Downloading 4096 bytes @ address 0x00000410
Downloading 4096 bytes @ address 0x00001410
Downloading 4096 bytes @ address 0x00002410
Downloading 4096 bytes @ address 0x00003410
...

This is not a complete solution so I will not mark it as such.