IMX8QXPC0 reset issue during boot

Our team have developed a custom board based on IMX8QXP Mek, but with C0 silicon (MIMX8QX6AVLFZAC). We are using the same PMIC and SD card, eMMC memory connections etc, but with increased amount of RAM (4GB: Micron MT53D1024M32D4DT-053).

Upon receiving the board everything looked fine, voltages are ok etc. For a quick test I've tried to run our Yocto based image for MEK (built for IMX8QXPC0 mek). Unfortunately shortly after "Starting kernel" the boot process reboots.

We've decided then to port SCFW (porting kit 1.7.0) to support 4GB of RAM . In addition we've added imx-seco version 3.7.5 to support new silicon (SCU debug monitor complained about using old version 3.7.1). Upon doing so we've received exactly the same behaviour at exactly the same time. Upon this We've made a clean u-boot and kernel device-tree to basically only support uart0 for console purposes, yet again - same result at the same time. Here is our boot log:

U-Boot 2020.04-5.4.47-2.2.0+ga02bdde0 (Apr 09 2021 - 13:23:18 +0000)

CPU: NXP i.MX8QXP RevC A35 at 1200 MHz at 35C

Model: NXP i.MX8QXP TEST
Board: iMX8QXP TEST

BuildInfo:
- SCFW 0d54291f, SECO-FW 640b5efb, IMX-MKIMAGE 8da5cd23, ATF c949a88
- U-Boot 2020.04-5.4.47-2.2.0+ga02bdde0

Boot: SD1
DRAM: 4 GiB
MMC: FSL_SDHC: 0, FSL_SDHC: 1
Loading Environment from MMC... Run CMD11 1.8V switch
*** Warning - bad CRC, using default environment

Fail to setup video link
In: serial
Out: serial
Err: serial
Run CMD11 1.8V switch
switch to partitions #0, OK
mmc1 is current device
flash target is MMC:1
Run CMD11 1.8V switch
Net: No ethernet found.
Fastboot: Normal
Normal Boot
Hit any key to stop autoboot: 0
Run CMD11 1.8V switch
switch to partitions #0, OK
mmc1 is current device
Run CMD11 1.8V switch
27941376 bytes read in 353 ms (75.5 MiB/s)
Booting from mmc ...
84591 bytes read in 14 ms (5.8 MiB/s)
## Flattened Device Tree blob at 83000000
Booting using the fdt blob at 0x83000000
Using Device Tree in place at 0000000083000000, end 0000000083017a6e
Disable clock-controller@5a4d0000 rsrc 62 not owned
Disable clock-controller@5ac90000 rsrc 102 not owned
Disable clock-controller@5b260000 rsrc 253 not owned

Starting kernel ...

[ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd042]
[ 0.000000] Linux version 5.4.47-2.2.0+g276151221 (oe-user@oe-host) (gcc ver1
[ 0.000000] Machine model: i.MX8QXP TEST
[ 0.000000] efi: Getting EFI parameters from FDT:
[ 0.000000] efi: UEFI not found.
[ 0.000000] Reserved memory: created CMA memory pool at 0x0000000096000000, B
[ 0.000000] OF: reserved mem: initialized node linux,cma, compatible id sharl
[ 0.000000] Reserved memory: created DMA memory pool at 0x0000000090000000, B
[ 0.000000] OF: reserved mem: initialized node vdev0vring0@90000000, compatil
[ 0.000000] Reserved memory: created DMA memory pool at 0x0000000090008000, B
[ 0.000000] OF: reserved mem: initialized node vdev0vring1@90008000, compatil
[ 0.000000] Reserved memory: created DMA memory pool at 0x0000000090010000, B
[ 0.000000] OF: reserved mem: initialized node vdev1vring0@90010000, compatil
[ 0.000000] Reserved memory: created DMA memory pool at 0x0000000090018000, B
[ 0.000000] OF: reserved mem: initialized node vdev1vring1@90018000, compatil
[ 0.000000] Reserved memory: created DMA memory pool at 0x0000000090400000, B
[ 0.000000] OF: reserved mem: initialized node vdevbuffer, compatible id shal
[ 0.000000] earlycon: lpuart32 at MMIO 0x000000005a060000 (options '')
[ 0.000000] printk: bootconsole [lpuart32] enabled
[ 0.000000] NUMA: No NUMA configuration found
[ 0.000000] NUMA: Faking a node at [mem 0x0000000080200000-0x00000008fffffff]
[ 0.000000] NUMA: NODE_DATA [mem 0x8ff807500-0x8ff808fff]
[ 0.000000] Zone ranges:
[ 0.000000] DMA32 [mem 0x0000000080200000-0x00000000ffffffff]
[ 0.000000] Normal [mem 0x0000000100000000-0x00000008ffffffff]
[ 0.000000] Movable zone start for each node
[ 0.000000] Early memory node ranges
[ 0.000000] node 0: [mem 0x0000000080200000-0x0000000083ffffff]
[ 0.000000] node 0: [mem 0x0000000086200000-0x0000000087ffffff]
[ 0.000000] node 0: [mem 0x0000000090020000-0x00000000901fffff]
[ 0.000000] node 0: [mem 0x0000000090500000-0x0000000091ffffff]
[ 0.000000] node 0: [mem 0x0000000094c00000-0x00000000ffffffff]
[ 0.000000] node 0: [mem 0x0000000880000000-0x00000008ffffffff]
[ 0.000000] Initmem setup node 0 [mem 0x0000000080200000-0x00000008ffffffff]
[ 0.000000] psci: probing for conduit method from DT.
[ 0.000000] psci: PSCIv1.1 detected in firmware.
[ 0.000000] psci: Using standard PSCI v0.2 function IDs
[ 0.000000] psci: MIGRATE_INFO_TYPE not supported.
[ 0.000000] psci: SMC Calling Convention v1.1
[ 0.000000] percpu: Embedded 24 pages/cpu s58904 r8192 d31208 u98304
[ 0.000000] Detected VIPT I-cache on CPU0
[ 0.000000] CPU features: detected: GIC system register CPU interface
[ 0.000000] Speculative Store Bypass Disable mitigation not required
[ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 978988
[ 0.000000] Policy zone: Normal
[ 0.000000] Kernel command line: console=ttyLP0,115200 earlycon root=/dev/mmw
[ 0.000000] Dentry cache hash table entries: 524288 (order: 10, 4194304 byte)
[ 0.000000] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes,)
[ 0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[ 0.000000] software IO TLB: mapped [mem 0xfbfff000-0xfffff000] (64MB)
[ 0.000000] Memory: 2820512K/3978112K available (16508K kernel code, 1238K r)
[ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
[ 0.000000] rcu: Preemptible hierarchical RCU implementation.
[ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=256 to nr_cpu_ids=4.
[ 0.000000] Tasks RCU enabled.
[ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 ji.
[ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4
[ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
[ 0.000000] GICv3: GIC: Using split EOI/Deactivate mode
[ 0.000000] GICv3: 512 SPIs implemented
[ 0.000000] GICv3: 0 Extended SPIs implemented
[ 0.000000] GICv3: Distributor has no Range Selector support
[ 0.000000] GICv3: 16 PPIs implemented
[ 0.000000] GICv3: no VLPI support, no direct LPI support
[ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x0000000051b00000
[ 0.000000] random: get_random_bytes called from start_kernel+0x2b8/0x44c wi0
[ 0.000000] arch_timer: cp15 timer(s) running at 8.00MHz (phys).
[ 0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycless
[ 0.000004] sched_clock: 56 bits at 8MHz, resolution 125ns, wraps every 2199s
[ 0.008779] Console: colour dummy device 80x25
[ 0.012968] Calibrating delay loop (skipped), value calculated using timer f)
[ 0.023134] pid_max: default: 32768 minimum: 301
[ 0.027827] LSM: Security Framework initializing
[ 0.032408] Mount-cache hash table entries: 8192 (order: 4, 65536 bytes, lin)
[ 0.039738] Mountpoint-cache hash table entries: 8192 (order: 4, 65536 bytes)
[ 0.071642] ASID allocator initialised with 32768 entries
[ 0.084738] rcu: Hierarchical SRCU implementation.
[ 0.099245] EFI services will not be available.
[ 0.111476] smp: Bringing up secondary CPUs ...
[ 0.148074] Detected VIPT I-cache on CPU1
[ 0.148105] GICv3: CPU1: found redistributor 1 region 0:0x0000000051b20000
[ 0.148143] CPU1: Booted secondary processor 0x0000000001 [0x410fd042]
[ 0.180118] Detected VIPT I-cache on CPU2
[ 0.180139] GICv3: CPU2: found redistributor 2 region 0:0x0000000051b40000
[ 0.180162] CPU2: Booted secondary processor 0x0000000002 [0x410fd042]
[ 0.212186] Detected VIPT I-cache on CPU3
[ 0.212205] GICv3: CPU3: found redistributor 3 region 0:0x0000000051b60000
[ 0.212225] CPU3: Booted secondary processor 0x0000000003 [0x410fd042]
[ 0.212294] smp: Brought up 1 node, 4 CPUs
[ 0.268189] SMP: Total of 4 processors activated.
[ 0.272877] CPU features: detected: 32-bit EL0 Support
[ 0.278011] CPU features: detected: CRC32 instructions
[ 0.290138] CPU: All CPU(s) started at EL2
[ 0.293933] alternatives: patching kernel code
[ 0.299588] devtmpfs: initialized
[ 0.316780] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, ms
[ 0.326201] futex hash table entries: 1024 (order: 4, 65536 bytes, linear)
[ 0.364934] pinctrl core: initialized pinctrl subsystem
[ 0.370871] DMI not present or invalid.
[ 0.374691] NET: Registered protocol family 16
[ 0.387399] DMA: preallocated 256 KiB pool for atomic allocations
[ 0.393190] audit: initializing netlink subsys (disabled)
[ 0.398822] audit: type=2000 audit(0.340:1): state=initialized audit_enabled1
[ 0.406594] cpuidle: using governor menu
[ 0.411115] hw-breakpoint: found 6 breakpoint and 4 watchpoint registers.
[ 0.419322] Serial: AMBA PL011 UART driver
[ 0.423158] imx mu driver is registered.
[ 0.427000] imx rpmsg driver is registered.
[ 0.480255] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages
[ 0.486633] HugeTLB registered 32.0 MiB page size, pre-allocated 0 pages
[ 0.493309] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
[ 0.499998] HugeTLB registered 64.0 KiB page size, pre-allocated 0 pages
[ 0.507734] cryptd: max_cpu_qlen set to 1000
[ 0.515899] ACPI: Interpreter disabled.
[ 0.520828] iommu: Default domain type: Translated
[ 0.525570] vgaarb: loaded
[ 0.528462] SCSI subsystem initialized
[ 0.532293] usbcore: registered new interface driver usbfs
[ 0.537499] usbcore: registered new interface driver hub
[ 0.542804] usbcore: registered new device driver usb
[ 0.549420] mc: Linux media interface: v0.10
[ 0.553565] videodev: Linux video capture interface: v2.00
[ 0.558907] pps_core: LinuxPPS API ver. 1 registered
[ 0.563773] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giom>
[ 0.572906] PTP clock support registered
[ 0.577106] EDAC MC: Ver: 3.0.0
[ 0.581345] No BMan portals available!
[ 0.585128] QMan: Allocated lookup table at (____ptrval____), entry count 657
[ 0.592729] No QMan portals available!
[ 0.597067] No USDPAA memory, no 'fsl,usdpaa-mem' in device-tree
[ 0.603541] FPGA manager framework
[ 0.606701] Advanced Linux Sound Architecture Driver Initialized.
[ 0.613215] Bluetooth: Core ver 2.22
[ 0.616474] NET: Registered protocol family 31
[ 0.620890] Bluetooth: HCI device and connection manager initialized
[ 0.627230] Bluetooth: HCI socket layer initialized
[ 0.632094] Bluetooth: L2CAP socket layer initialized
[ 0.637141] Bluetooth: SCO socket layer initialized
[ 0.643070] imx-scu scu: NXP i.MX SCU Initialized
[ 0.653174] random: fast init done

No matter what system We put to the board (using SD card) the boot process goes up to the "random: fast init done" line and then a reset is performed.

We are using imx layers for Yocto project with branch Zeus (cannot use the latest due to some external layers dependencies which are not yet supported by Gatesgarth layer nor Dunfell).

Have anyone encountered a similar issue and could help us out ? How can We debug the issue, could it be perhaps some kernel config that is causing it ? We are using not modified imx_v8_defconfig. In addition we have removd optee from our build environment.

Thanks in advance. BR

EDIT

We have figured out that processor is asserting SCU_WDOG_OUT pin. What could be the possible cause of this ? Is this strictly hardware issue of should we investigate firmware?

EDIT2

We have managed to run DDR test with success upon some board modifications. In addition we have downlaoded image for imx8qxpc0mek from NXP website. This image was old and based on kernel 4.14. With this image we've received more logs... and kernel panic:

[    3.041980] Unable to handle kernel paging request at virtual address fffffffffffffff8
[    3.049908] Mem abort info:
[    3.052706]   Exception class = DABT (current EL), IL = 32 bits
[    3.058629]   SET = 0, FnV = 0
[    3.061678]   EA = 0, S1PTW = 0
[    3.064813] Data abort info:
[    3.067690]   ISV = 0, ISS = 0x00000004
[    3.071521]   CM = 0, WnR = 0
[    3.074487] swapper pgtable: 4k pages, 48-bit VAs, pgd = ffff00000973c000
[    3.081278] [fffffffffffffff8] *pgd=0000000000000000
[    3.086249] Internal error: Oops: 96000004 [#1] PREEMPT SMP
[    3.091824] Modules linked in:
[    3.094888] CPU: 3 PID: 1560 Comm: kworker/3:2 Not tainted 4.14.98-2.3.0+g0f549d8c4d5e #1
[    3.103066] Hardware name: Freescale i.MX8QXP MEK (DT)
[    3.108222] Workqueue: events deferred_probe_work_func
[    3.113360] task: ffff80083af86c00 task.stack: ffff00000df48000
[    3.119282] PC is at plist_add+0x60/0xd8
[    3.123209] LR is at pm_qos_update_target+0x164/0x220
[    3.128261] pc : [<ffff000008db61a8>] lr : [<ffff000008114b84>] pstate: 200000c5
[    3.135660] sp : ffff00000df4bb10
[    3.138969] x29: ffff00000df4bb10 x28: 0000000000000000 
[    3.144286] x27: 0000000000000000 x26: 0000000fffffffe0 
[    3.149602] x25: 0000000000000040 x24: 0000000000000000 
[    3.154919] x23: 0000000000000000 x22: ffff0000096d8000 
[    3.160236] x21: 0000000000000000 x20: ffff80083ba04e88 
[    3.165553] x19: ffff000009542c80 x18: 0000000000000000 
[    3.170870] x17: ffff80083b9d9c18 x16: ffff80083b9d9c18 
[    3.176186] x15: ffffffffffffffff x14: ffff80083b96791c 
[    3.181503] x13: 0000000000000000 x12: 0000000000000030 
[    3.186820] x11: 0101010101010101 x10: 7f7f7f7f7f7f7f7f 
[    3.192137] x9 : 716e6573606b6f2e x8 : ffff80083ba04e90 
[    3.197453] x7 : ffff80083ba04ea0 x6 : ffff000009542c80 
[    3.202770] x5 : ffff80083ade4e88 x4 : 0000000000000000 
[    3.208087] x3 : ffff80083ade4e88 x2 : fffffffffffffff8 
[    3.213404] x1 : fffffffffffffff8 x0 : ffff80083ba04e88 
[    3.218724] Process kworker/3:2 (pid: 1560, stack limit = 0xffff00000df48000)
[    3.225864] Call trace:
[    3.228316] Exception stack(0xffff00000df4b9d0 to 0xffff00000df4bb10)
[    3.234762] b9c0:                                   ffff80083ba04e88 fffffffffffffff8
[    3.242598] b9e0: fffffffffffffff8 ffff80083ade4e88 0000000000000000 ffff80083ade4e88
[    3.250434] ba00: ffff000009542c80 ffff80083ba04ea0 ffff80083ba04e90 716e6573606b6f2e
[    3.258270] ba20: 7f7f7f7f7f7f7f7f 0101010101010101 0000000000000030 0000000000000000
[    3.266108] ba40: ffff80083b96791c ffffffffffffffff ffff80083b9d9c18 ffff80083b9d9c18
[    3.273942] ba60: 0000000000000000 ffff000009542c80 ffff80083ba04e88 0000000000000000
[    3.281780] ba80: ffff0000096d8000 0000000000000000 0000000000000000 0000000000000040
[    3.289614] baa0: 0000000fffffffe0 0000000000000000 0000000000000000 ffff00000df4bb10
[    3.297451] bac0: ffff000008114b84 ffff00000df4bb10 ffff000008db61a8 00000000200000c5
[    3.305286] bae0: 0000000000000000 0000000000000000 ffffffffffffffff ffff8008383f7a80
[    3.313121] bb00: ffff00000df4bb10 ffff000008db61a8
[    3.318006] [<ffff000008db61a8>] plist_add+0x60/0xd8
[    3.322974] [<ffff000008114cd0>] pm_qos_add_request+0x90/0xb8
[    3.328729] [<ffff000008b082a4>] tcpm_register_port+0x3f4/0x4e0
[    3.334651] [<ffff000008b092a0>] tcpci_probe+0x208/0x608
[    3.339970] [<ffff0000089894ec>] i2c_device_probe+0x304/0x328
[    3.345717] [<ffff0000086df4b8>] driver_probe_device+0x210/0x2d0
[    3.351729] [<ffff0000086df6d4>] __device_attach_driver+0x9c/0xf8
[    3.357829] [<ffff0000086dd6d4>] bus_for_each_drv+0x4c/0x98
[    3.363405] [<ffff0000086df178>] __device_attach+0xc0/0x138
[    3.368982] [<ffff0000086df790>] device_initial_probe+0x10/0x18
[    3.374909] [<ffff0000086de634>] bus_probe_device+0x94/0xa0
[    3.380484] [<ffff0000086deae8>] deferred_probe_work_func+0xa0/0x148
[    3.386848] [<ffff0000080e7564>] process_one_work+0x1d4/0x348
[    3.392597] [<ffff0000080e791c>] worker_thread+0x244/0x470
[    3.398089] [<ffff0000080eda24>] kthread+0x12c/0x130
[    3.403058] [<ffff000008084ed8>] ret_from_fork+0x10/0x18
[    3.408377] Code: d1002021 eb03003f 54000100 aa0103e2 (b9400041) 
[    3.414472] ---[ end trace cc1efc4670b6fd92 ]---
[    3.419121] note: kworker/3:2[1560] exited with preempt_count 1

Watchdog wasn't triggered at all, and after few minutes this shows up:

[  289.222430] random: crng init done

Also without watchdog trigger. Does anyone know where should we look at ?

EDIT 3

Interesting fact - we have managed to boot linux using IMX sumo BSP, with exaclty the same changes we have been making in newer releases... This means that somehow kernel 5.4 and 5.10 is not compatible with imx8qxpc0.