LPC4357 jumping to external flash unstable

I have two programs, one is custom bootloader which will be located in BANK A and the other one is the main program which will be located in external spifi flash. Both programs I load and debug from eclipse.

Currently I can jump from the bootloader to the main program, I can start debug with jtag from the reset handler in the main program but sometimes is continues to main and sometimes it breaks after the first step.

From the bootloader I do the following for jumping to main program:

halInit();   // init my hal functions
chSysInit();   // init my OS
chThdSleepMilliseconds(1000);
flashInit(); // init the spifi driver

uint32_t NewAddressOfNewVectorTable = 0x14000000;

__disable_irq();

/**  Load new Vector Table  **/
SCB->VTOR = AddressOfNewVectorTable;
__set_MSP(*(uint32_t*)(AddressOfNewVectorTable));
LPC_CREG->M4MEMMAP = AddressOfNewVectorTable;

/**
* Jump to Reset Vector of new Vector Table
* -> thats the second word (hence fourth byte).
*/
((user_code_pointer_t)(*((uint32_t*)(AddressOfNewVectorTable + 4))))();

This seems to work alright and it jumps to the 0x14000000 address.

Then when I start to debug with JTAG the app, it will come first in the reset handler.

I firstly had to disable __early_init() for the main app which configures the clocks, this would make all crash. Is this correct?

Then sometimes I can step through all into the main() function.

But also regularly it crashes after doing the first step with JTAG with the following output:

SEGGER J-Link GDB Server V5.02l Command Line Version

JLinkARM.dll V5.02l (DLL compiled Nov 24 2015 09:36:30)

-----GDB Server start settings-----
GDBInit file:                  none
GDB Server Listening port:     2331
SWO raw output listening port: 2332
Terminal I/O port:             2333
Accept remote connection:      localhost only
Generate logfile:              off
Verify download:               on
Init regs on start:            on
Silent mode:                   off
Single run mode:               on
Target connection timeout:     0 ms
------J-Link related settings------
J-Link Host interface:         USB
J-Link script:                 none
J-Link settings file:          none
------Target related settings------
Target device:                 LPC4357_M4
Target interface:              SWD
Target interface speed:        1000kHz
Target endian:                 little

Connecting to J-Link...
J-Link is connected.
Firmware: J-Link V9 compiled Oct  9 2015 20:34:47
Hardware: V9.30
S/N: 269301849
OEM: SEGGER-EDU
Feature(s): FlashBP, GDB
Checking target voltage...
Target voltage: 3.31 V
Listening on TCP/IP port 2331
Connecting to target...Connected to target
Waiting for GDB connection...Connected to 127.0.0.1
Reading all registers
WARNING: Failed to read memory @ address 0x00000000
Target interface speed set to 1000 kHz
Resetting target
Halting target CPU...
...Target halted (PC = 0x1A000140)
R0 = 1A000000, R1 = 1A000000, R2 = 400F1FC0, R3 = 12345678
R4 = 40045000, R5 = 1008000C, R6 = 00000000, R7 = 00000000
R8 = 00000000, R9 = 00000000, R10= 00000000, R11= 00000000
R12= 00000000, R13= 10000200, MSP= 10000200, PSP= 00000000
R14(LR) = 10405B25, R15(PC) = 1A000140
XPSR 41000000, APSR 40000000, EPSR 01000000, IPSR 00000000
CFBP 00000000, CONTROL 00, FAULTMASK 00, BASEPRI 00, PRIMASK 00
Reading all registers
Select auto target interface speed (2000 kHz)
Flash breakpoints enabled
Semi-hosting enabled (Handle on BKPT)
Semihosting I/O set to TELNET Client
SWO disabled succesfully.
SWO enabled succesfully.
Downloading 276 bytes @ address 0x14000000 - Verified OK
Downloading 16192 bytes @ address 0x14000120 - Verified OK
Downloading 16080 bytes @ address 0x14004060 - Verified OK
Downloading 16176 bytes @ address 0x14007F30 - Verified OK
Downloading 16064 bytes @ address 0x1400BE60 - Verified OK
Downloading 16096 bytes @ address 0x1400FD20 - Verified OK
Downloading 16096 bytes @ address 0x14013C00 - Verified OK
Downloading 16112 bytes @ address 0x14017AE0 - Verified OK
Downloading 8492 bytes @ address 0x1401B9D0 - Verified OK
Downloading 8 bytes @ address 0x1401DAFC - Verified OK
Downloading 2308 bytes @ address 0x1401DB08 - Verified OK
Comparing flash   [....................] Done.
Verifying flash   [....................] Done.
Writing register (PC = 0x14000120)
Read 4 bytes @ address 0x14000120 (Data = 0x4100F081)
Read 2 bytes @ address 0x14015764 (Data = 0x2300)
Read 2 bytes @ address 0x140157C8 (Data = 0x2300)
Read 2 bytes @ address 0x140166D8 (Data = 0x2300)
Read 2 bytes @ address 0x140166DC (Data = 0x4818)
Read 2 bytes @ address 0x14001370 (Data = 0xB672)
Read 2 bytes @ address 0x140013E8 (Data = 0xF015)
Read 2 bytes @ address 0x140166D8 (Data = 0x2300)
Read 2 bytes @ address 0x1401670E (Data = 0xAB03)
Read 2 bytes @ address 0x14001370 (Data = 0xB672)
Read 2 bytes @ address 0x14001370 (Data = 0xB672)
Read 2 bytes @ address 0x14016764 (Data = 0xF7EC)
Read 2 bytes @ address 0x14016764 (Data = 0xF7EC)
Read 2 bytes @ address 0x14001370 (Data = 0xB672)
Read 2 bytes @ address 0x140013D4 (Data = 0xF7FF)
Read 2 bytes @ address 0x14015764 (Data = 0x2300)
Read 2 bytes @ address 0x140157BA (Data = 0x4850)
Read 2 bytes @ address 0x1400ACB8 (Data = 0x4806)
Read 2 bytes @ address 0x1400ACBA (Data = 0x4907)
Read 2 bytes @ address 0x1400ACC4 (Data = 0x4B05)
Resetting target
Halting target CPU...
...Target halted (PC = 0x1A000140)
Read 2 bytes @ address 0x14016764 (Data = 0xF7EC)
Read 2 bytes @ address 0x14016764 (Data = 0xF7EC)
Read 2 bytes @ address 0x14016764 (Data = 0xF7EC)
R0 = 1A000000, R1 = 1A000000, R2 = 400F1FC0, R3 = 12345678
R4 = 40045000, R5 = 1008000C, R6 = 00000000, R7 = 00000000
R8 = 00000000, R9 = 00000000, R10= 00000000, R11= 00000000
R12= 00000000, R13= 10000200, MSP= 10000200, PSP= 00000000
R14(LR) = 10405B25, R15(PC) = 1A000140
XPSR 41000000, APSR 40000000, EPSR 01000000, IPSR 00000000
CFBP 00000000, CONTROL 00, FAULTMASK 00, BASEPRI 00, PRIMASK 00
Reading all registers
Read 4 bytes @ address 0x1A000140 (Data = 0x4C24B672)
Setting breakpoint @ address 0x14001370, Size = 2, BPHandle = 0x0001
Setting breakpoint @ address 0x140013D4, Size = 2, BPHandle = 0x0002
Setting breakpoint @ address 0x140013E8, Size = 2, BPHandle = 0x0003
Setting breakpoint @ address 0x1400ACC4, Size = 2, BPHandle = 0x0004
Setting breakpoint @ address 0x140166DC, Size = 2, BPHandle = 0x0005
Setting breakpoint @ address 0x1401670E, Size = 2, BPHandle = 0x0006
Setting breakpoint @ address 0x14016764, Size = 2, BPHandle = 0x0007
Starting target CPU...
Erasing flash     [....................] Done.
Programming flash [....................] Done.
Verifying flash   [....................] Done.
...Breakpoint reached @ address 0x14001370
Reading all registers
Removing breakpoint @ address 0x14001370, Size = 2
Removing breakpoint @ address 0x140013D4, Size = 2
Removing breakpoint @ address 0x140013E8, Size = 2
Removing breakpoint @ address 0x1400ACC4, Size = 2
Removing breakpoint @ address 0x140166DC, Size = 2
Removing breakpoint @ address 0x1401670E, Size = 2
Removing breakpoint @ address 0x14016764, Size = 2
Read 4 bytes @ address 0x14001370 (Data = 0x4C23B672)
Read 4 bytes @ address 0x1A0025B4 (Data = 0xF8D39B03)
Read 4 bytes @ address 0x1A0025B4 (Data = 0xF8D39B03)
Performing single step...
...Target halted (PC = 0xFFFFFFFE)
Reading all registers
WARNING: Failed to read memory @ address 0xFFFFFFFE
Read 4 bytes @ address 0x1000041C (Data = 0x00000000)
Reading 64 bytes @ address 0x10000400
Read 4 bytes @ address 0x00000000 (Data = 0x10000400)
Read 4 bytes @ address 0x00000000 (Data = 0x10000400)

Am I missing something that is needed after the jump to external flash? Or can anyone give me a pointer why this is not functioning properly all the time?