Kernel panic on linux-omap 4.3.0-rc5 on IGEPv2 board

I'm trying to run the latest linux-omap kernel Linux-4.3.0-rc5-12674-g80923cf8e that I got from the official linux-omap GIT.

I compiled it with the multi_v7_defconfig, disabling unneeded boards and adding some debug options like earlyprintk.

I am now stuck with a kernel panic and I can't find what the problem is.

The board is IGEPv2 and I am using the omap3-igep0020.dts device tree file.

Kernel is loaded using the latest u-boot, if it helps.

Here's the bootlog. Any idea ?

## Booting kernel from Legacy Image at 80008000 ...
   Image Name:   Linux-4.3.0-rc5-12674-g80923cf8e
   Image Type:   ARM Linux Kernel Image (uncompressed)
   Data Size:    4945528 Bytes = 4.7 MiB
   Load Address: 80008000
   Entry Point:  80008000
   Verifying Checksum ... OK
## Flattened Device Tree blob at 81f00000
   Booting using the fdt blob at 0x81f00000
   Loading Kernel Image ... OK
   Loading Device Tree to 8ffec000, end 8ffff0a2 ... OK

Starting kernel ...

[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Linux version 4.3.0-rc5-12674-g80923cf8eb3b (b@b43) (gcc version 7.2.0 (Ubuntu/Linaro 7.2.0-6ubuntu1) ) #2 SMP Wed Feb 28 00:49:54 EST 2018
[    0.000000] CPU: ARMv7 Processor [411fc083] revision 3 (ARMv7), cr=10c5387d
[    0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT nonaliasing instruction cache
[    0.000000] Machine model: IGEPv2 Rev. C (TI OMAP AM/DM37x)
[    0.000000] bootconsole [earlycon0] enabled
[    0.000000] cma: Reserved 64 MiB at 0x9b800000
[    0.000000] Memory policy: Data cache writeback
[    0.000000] On node 0 totalpages: 130816
[    0.000000] free_area_init_node: node 0, pgdat c09aee00, node_mem_map dfad9000
[    0.000000]   Normal zone: 1024 pages used for memmap
[    0.000000]   Normal zone: 0 pages reserved
[    0.000000]   Normal zone: 130816 pages, LIFO batch:31
[    0.000000] CPU: All CPU(s) started in SVC mode.
[    0.000000] OMAP3430/3530 ES3.1 (l2cache iva sgx neon isp )
[    0.000000] PERCPU: Embedded 12 pages/cpu @dfa86000 s18572 r8192 d22388 u49152
[    0.000000] pcpu-alloc: s18572 r8192 d22388 u49152 alloc=12*4096
[    0.000000] pcpu-alloc: [0] 0 
[    0.000000] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 129792
[    0.000000] Kernel command line: mem=512M console=ttyO2,115200n8 debug earlyprintk omapfb.mode=dvi:1024x768MR-16@60 root=/dev/mmcblk0p2 rw rootwait
[    0.000000] PID hash table entries: 2048 (order: 1, 8192 bytes)
[    0.000000] Dentry cache hash table entries: 65536 (order: 6, 262144 bytes)
[    0.000000] Inode-cache hash table entries: 32768 (order: 5, 131072 bytes)
[    0.000000] Memory: 442476K/523264K available (6856K kernel code, 370K rwdata, 2264K rodata, 396K init, 295K bss, 15252K reserved, 65536K cma-reserved, 0K highmem)
[    0.000000] Virtual kernel memory layout:
[    0.000000]     vector  : 0xffff0000 - 0xffff1000   (   4 kB)
[    0.000000]     fixmap  : 0xffc00000 - 0xfff00000   (3072 kB)
[    0.000000]     vmalloc : 0xe0800000 - 0xff000000   ( 488 MB)
[    0.000000]     lowmem  : 0xc0000000 - 0xe0000000   ( 512 MB)
[    0.000000]     pkmap   : 0xbfe00000 - 0xc0000000   (   2 MB)
[    0.000000]     modules : 0xbf000000 - 0xbfe00000   (  14 MB)
[    0.000000]       .text : 0xc0008000 - 0xc08f0590   (9122 kB)
[    0.000000]       .init : 0xc08f1000 - 0xc0954000   ( 396 kB)
[    0.000000]       .data : 0xc0954000 - 0xc09b0980   ( 371 kB)
[    0.000000]        .bss : 0xc09b3000 - 0xc09fcc0c   ( 296 kB)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[    0.000000] Hierarchical RCU implementation.
[    0.000000] ⇥    Build-time adjustment of leaf fanout to 32.
[    0.000000] ⇥    RCU restricting CPUs from NR_CPUS=16 to nr_cpu_ids=1.
[    0.000000] RCU: Adjusting geometry for rcu_fanout_leaf=32, nr_cpu_ids=1
[    0.000000] NR_IRQS:16 nr_irqs:16 16
[    0.000000] IRQ: Found an INTC at 0xfa200000 (revision 4.0) with 96 interrupts
[    0.000000] Clocking rate (Crystal/Core/MPU): 26.0/332/600 MHz
[    0.000000] OMAP clockevent source: timer1 at 32768 Hz
[    0.000000] clocksource: 32k_counter: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 58327039986419 ns
[    0.000000] sched_clock: 32 bits at 32kHz, resolution 30517ns, wraps every 65535999984741ns
[    0.008697] OMAP clocksource: 32k_counter at 32768 Hz
[    0.014770] Console: colour dummy device 80x30
[    0.019439] Calibrating delay loop... 597.60 BogoMIPS (lpj=2988032)
[    0.084869] pid_max: default: 32768 minimum: 301
[    0.089874] Mount-cache hash table entries: 1024 (order: 0, 4096 bytes)
[    0.096801] Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes)
[    0.105255] CPU: Testing write buffer coherency: ok
[    0.110809] CPU0: thread -1, cpu 0, socket -1, mpidr 0
[    0.116394] Setting up static identity map for 0x80008340 - 0x80008398
[    0.124786] Brought up 1 CPUs
[    0.127929] SMP: Total of 1 processors activated (597.60 BogoMIPS).
[    0.134521] CPU: All CPU(s) started in SVC mode.
[    0.141571] devtmpfs: initialized
[    0.190948] VFP support v0.3: implementor 41 architecture 3 part 30 variant c rev 1
[    0.221343] omap_hwmod: mcbsp2_sidetone using broken dt data from mcbsp
[    0.229003] omap_hwmod: mcbsp3_sidetone using broken dt data from mcbsp
[    0.296447] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[    0.311370] pinctrl core: initialized pinctrl subsystem
[    0.319793] NET: Registered protocol family 16
[    0.328643] DMA: preallocated 256 KiB pool for atomic coherent allocations
[    0.338317] cpuidle: using governor ladder
[    0.342712] cpuidle: using governor menu
[    0.347473] Reprogramming SDRC clock to 332000000 Hz
[    0.360107] OMAP GPIO hardware version 2.5
[    0.385833] omap-gpmc 6e000000.gpmc: GPMC revision 5.0
[    0.391235] gpmc_mem_init: disabling cs 0 mapped at 0x0-0x1000000
[    0.400299] platform 49042000.serial: Cannot lookup hwmod 'uart4'
[    0.412506] No ATAGs?
[    0.414703] hw-breakpoint: debug architecture 0x4 unsupported.
[    0.422790] OMAP DMA hardware revision 4.0
[    0.427307] Serial: AMBA PL011 UART driver
[    0.458862] omap-dma-engine 48056000.dma-controller: OMAP DMA engine driver
[    0.474639] vgaarb: loaded
[    0.478942] SCSI subsystem initialized
[    0.483337] libata version 3.00 loaded.
[    0.488037] usbcore: registered new interface driver usbfs
[    0.493957] usbcore: registered new interface driver hub
[    0.499603] usbcore: registered new device driver usb
[    0.506896] omap_i2c 48070000.i2c: could not find pctldev for node /ocp/l4@48000000/scm@2000/pinmux@30/pinmux_i2c1_pins, deferring probe
[    0.520629] omap_i2c 48072000.i2c: bus 1 rev3.3 at 100 kHz
[    0.526519] omap_i2c 48060000.i2c: could not find pctldev for node /ocp/l4@48000000/scm@2000/pinmux@30/pinmux_i2c3_pins, deferring probe
[    0.539611] pps_core: LinuxPPS API ver. 1 registered
[    0.544860] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[    0.554412] PTP clock support registered
[    0.558990] EDAC MC: Ver: 3.0.0
[    0.565734] clocksource: Switched to clocksource 32k_counter
[    0.593200] NET: Registered protocol family 2
[    0.598846] TCP established hash table entries: 4096 (order: 2, 16384 bytes)
[    0.606323] TCP bind hash table entries: 4096 (order: 3, 32768 bytes)
[    0.613128] TCP: Hash tables configured (established 4096 bind 4096)
[    0.619903] UDP hash table entries: 256 (order: 1, 8192 bytes)
[    0.626068] UDP-Lite hash table entries: 256 (order: 1, 8192 bytes)
[    0.632873] NET: Registered protocol family 1
[    0.638061] RPC: Registered named UNIX socket transport module.
[    0.644226] RPC: Registered udp transport module.
[    0.649230] RPC: Registered tcp transport module.
[    0.654144] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    0.660919] PCI: CLS 0 bytes, default 64
[    0.668243] hw perfevents: enabled with armv7_cortex_a8 PMU driver, 5 counters available
[    0.680969] futex hash table entries: 256 (order: 2, 16384 bytes)
[    0.707794] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    0.715759] NFS: Registering the id_resolver key type
[    0.721191] Key type id_resolver registered
[    0.725585] Key type id_legacy registered
[    0.729949] ntfs: driver 2.1.32 [Flags: R/O].
[    0.735290] jffs2: version 2.2. (NAND) <0xc2><0xa9> 2001-2006 Red Hat, Inc.
[    0.743438] fuse init (API version 7.23)
[    0.750671] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 249)
[    0.758483] io scheduler noop registered
[    0.762634] io scheduler deadline registered
[    0.767242] io scheduler cfq registered (default)
[    0.776153] pinctrl-single 48002030.pinmux: 284 pins at pa fa002030 size 568
[    0.783874] pinctrl-single 48002a00.pinmux: 46 pins at pa fa002a00 size 92
[    0.791748] Unhandled fault: external abort on non-linefetch (0x1028) at 0xfa0025a0
[    0.799713] pgd = c0004000
[    0.802581] [fa0025a0] *pgd=48011452(bad)
[    0.806823] Internal error: : 1028 [#1] SMP ARM
[    0.811553] Modules linked in:
[    0.814819] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.3.0-rc5-12674-g80923cf8eb3b #2
[    0.823028] Hardware name: Generic OMAP36xx (Flattened Device Tree)
[    0.829589] task: db074000 ti: db078000 task.ti: db078000
[    0.835235] PC is at pcs_readw+0x0/0x10
[    0.839294] LR is at pcs_probe+0x388/0x804
[    0.843597] pc : [<c02be958>]    lr : [<c02c0700>]    psr: 20000153
[    0.843597] sp : db079de0  ip : 00000000  fp : 0000002e
[    0.855560] r10: 00000000  r9 : 00000002  r8 : 00000000
[    0.861053] r7 : 00000001  r6 : 00000000  r5 : 00000002  r4 : db2d4e10
[    0.867828] r3 : c02be958  r2 : db21fc00  r1 : 00000002  r0 : fa0025a0
[    0.874664] Flags: nzCv  IRQs on  FIQs off  Mode SVC_32  ISA ARM  Segment none
[    0.882171] Control: 10c5387d  Table: 80004019  DAC: 00000051
[    0.888183] Process swapper/0 (pid: 1, stack limit = 0xdb078218)
[    0.894439] Stack: (0xdb079de0 to 0xdb07a000)
[    0.899017] 9de0: c0824d64 dfac90a0 c0974cd8 00000000 00000002 dfac90a0 0000000c c0824fcc
[    0.907531] 9e00: db2d4ea8 00000000 db161c00 00000000 db2d4e40 db2d4e28 db2da9b0 db2da9b0
[    0.916046] 9e20: c09b082f c0156d78 db2da9b0 00000000 00000000 c08302ec 00000001 c0156f34
[    0.924560] 9e40: db163cd0 c08302ec 00000001 000000ec db163cd0 db2da7d0 db2da9b0 db163cd0
[    0.933074] 9e60: c08302ec db161c10 ffffffed c0974cd8 fffffdfb 00000000 000000ec c094d048
[    0.941589] 9e80: 00000000 c038b544 c09ec6ec db161c10 c0974cd8 00000000 00000000 c0389e2c
[    0.950103] 9ea0: db161c10 c0974cd8 db161c44 00000000 00000000 c0389f9c 00000000 c0974cd8
[    0.958587] 9ec0: c0389f18 c03882d0 db00cb5c db162434 c0974cd8 db2d9780 c0987578 c038940c
[    0.967102] 9ee0: c0824d64 c09b3000 c0974cd8 c0974cd8 c0959bb0 db2db240 c0912288 c038a68c
[    0.975616] 9f00: c0959bb0 c0959bb0 db2db240 c000985c 00000000 c014e320 c014ddd8 db11ab80
[    0.984130] 9f20: 00000000 c096a858 00000000 c06aad30 dfeff132 c00504d0 cccccccd c08bebb8
[    0.992645] 9f40: c085f958 00000006 00000006 c07f9410 c096a840 c09b3000 c09b3000 c0939830
[    1.001159] 9f60: c09b3000 c09b3000 c0939834 c093983c 00000007 c08f1dc0 00000006 00000006
[    1.009674] 9f80: 00000000 c08f159c 00000000 c0683b30 00000000 00000000 00000000 00000000
[    1.018188] 9fa0: 00000000 c0683b38 00000000 c000f638 00000000 00000000 00000000 00000000
[    1.026702] 9fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[    1.035186] 9fe0: 00000000 00000000 00000000 00000000 00000013 00000000 fdfeffff f7ff777f
[    1.043731] [<c02be958>] (pcs_readw) from [<c02c0700>] (pcs_probe+0x388/0x804)
[    1.051269] [<c02c0700>] (pcs_probe) from [<c038b544>] (platform_drv_probe+0x48/0xa0)
[    1.059417] [<c038b544>] (platform_drv_probe) from [<c0389e2c>] (driver_probe_device+0x1ec/0x2d8)
[    1.068664] [<c0389e2c>] (driver_probe_device) from [<c0389f9c>] (__driver_attach+0x84/0x88)
[    1.077423] [<c0389f9c>] (__driver_attach) from [<c03882d0>] (bus_for_each_dev+0x4c/0x9c)
[    1.085937] [<c03882d0>] (bus_for_each_dev) from [<c038940c>] (bus_add_driver+0x18c/0x210)
[    1.094543] [<c038940c>] (bus_add_driver) from [<c038a68c>] (driver_register+0x78/0xf4)
[    1.102874] [<c038a68c>] (driver_register) from [<c000985c>] (do_one_initcall+0x8c/0x1d8)
[    1.111389] [<c000985c>] (do_one_initcall) from [<c08f1dc0>] (kernel_init_freeable+0x158/0x1e8)
[    1.120452] [<c08f1dc0>] (kernel_init_freeable) from [<c0683b38>] (kernel_init+0x8/0xe8)
[    1.128875] [<c0683b38>] (kernel_init) from [<c000f638>] (ret_from_fork+0x14/0x3c)
[    1.136779] Code: e5d00000 e6ef0070 f57ff04f e12fff1e (e1d000b0) 
[    1.143157] ---[ end trace 51c6d6f40dd2a6d2 ]---
[    1.148040] In-band Error seen by MPU  at address 0
[    1.153167] ------------[ cut here ]------------
[    1.158020] WARNING: CPU: 0 PID: 1 at drivers/bus/omap_l3_smx.c:166 omap3_l3_app_irq+0xc4/0x118()
[    1.167236] Modules linked in:
[    1.170471] CPU: 0 PID: 1 Comm: swapper/0 Tainted: G      D         4.3.0-rc5-12674-g80923cf8eb3b #2
[    1.179962] Hardware name: Generic OMAP36xx (Flattened Device Tree)
[    1.186523] [<c00174f0>] (unwind_backtrace) from [<c0012ff4>] (show_stack+0x10/0x14)
[    1.194580] [<c0012ff4>] (show_stack) from [<c028eea8>] (dump_stack+0x80/0x90)
[    1.202117] [<c028eea8>] (dump_stack) from [<c00365f0>] (warn_slowpath_common+0x80/0xb0)
[    1.210540] [<c00365f0>] (warn_slowpath_common) from [<c00366bc>] (warn_slowpath_null+0x1c/0x24)
[    1.219665] [<c00366bc>] (warn_slowpath_null) from [<c02b5ec0>] (omap3_l3_app_irq+0xc4/0x118)
[    1.228546] [<c02b5ec0>] (omap3_l3_app_irq) from [<c00787e0>] (handle_irq_event_percpu+0x48/0x138)
[    1.237884] [<c00787e0>] (handle_irq_event_percpu) from [<c0078918>] (handle_irq_event+0x48/0x6c)
[    1.247100] [<c0078918>] (handle_irq_event) from [<c007b584>] (handle_level_irq+0xac/0x144)
[    1.255798] [<c007b584>] (handle_level_irq) from [<c0077f24>] (generic_handle_irq+0x24/0x34)
[    1.264587] [<c0077f24>] (generic_handle_irq) from [<c00781b0>] (__handle_domain_irq+0x5c/0xb4)
[    1.273620] [<c00781b0>] (__handle_domain_irq) from [<c0013bd4>] (__irq_svc+0x54/0x70)
[    1.281860] [<c0013bd4>] (__irq_svc) from [<c0038028>] (do_exit+0x3e0/0x988)
[    1.289215] [<c0038028>] (do_exit) from [<c0013324>] (die+0x32c/0x330)
[    1.296020] [<c0013324>] (die) from [<c0009380>] (do_DataAbort+0xb4/0xb8)
[    1.303100] [<c0009380>] (do_DataAbort) from [<c0013b60>] (__dabt_svc+0x40/0x60)
[    1.310821] Exception stack(0xdb079d90 to 0xdb079dd8)
[    1.316101] 9d80:                                     fa0025a0 00000002 db21fc00 c02be958
[    1.324615] 9da0: db2d4e10 00000002 00000000 00000001 00000000 00000002 00000000 0000002e
[    1.333129] 9dc0: 00000000 db079de0 c02c0700 c02be958 20000153 ffffffff
[    1.340026] [<c0013b60>] (__dabt_svc) from [<c02be958>] (pcs_readw+0x0/0x10)
[    1.347381] [<c02be958>] (pcs_readw) from [<c02c0700>] (pcs_probe+0x388/0x804)
[    1.354888] [<c02c0700>] (pcs_probe) from [<c038b544>] (platform_drv_probe+0x48/0xa0)
[    1.363067] [<c038b544>] (platform_drv_probe) from [<c0389e2c>] (driver_probe_device+0x1ec/0x2d8)
[    1.372283] [<c0389e2c>] (driver_probe_device) from [<c0389f9c>] (__driver_attach+0x84/0x88)
[    1.381072] [<c0389f9c>] (__driver_attach) from [<c03882d0>] (bus_for_each_dev+0x4c/0x9c)
[    1.389587] [<c03882d0>] (bus_for_each_dev) from [<c038940c>] (bus_add_driver+0x18c/0x210)
[    1.398193] [<c038940c>] (bus_add_driver) from [<c038a68c>] (driver_register+0x78/0xf4)
[    1.406524] [<c038a68c>] (driver_register) from [<c000985c>] (do_one_initcall+0x8c/0x1d8)
[    1.415039] [<c000985c>] (do_one_initcall) from [<c08f1dc0>] (kernel_init_freeable+0x158/0x1e8)
[    1.424072] [<c08f1dc0>] (kernel_init_freeable) from [<c0683b38>] (kernel_init+0x8/0xe8)
[    1.432495] [<c0683b38>] (kernel_init) from [<c000f638>] (ret_from_fork+0x14/0x3c)
[    1.440368] ---[ end trace 51c6d6f40dd2a6d3 ]---
[    1.445343] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
[    1.445343] 
[    1.454925] Rebooting in 300

I'm confused because there seem to be several stacktraces. Also I wasn't able to get a more detailed stacktrace despite compiling the debug info in the kernel (but I suspect the debug info is not actually there as the file is surprisingly small)

The last supported kernel from the manufacturer ISEE for this board is 2.6.37 which is horribly old. There does not seem to be any problem with running more recent kernels but it's possible the device tree files have not been updated for long... I'm just looking for pointers of where this error might come from and I'll try to fix it.

Thanks !

I don't know what pcs is, but looks like it reads from wrong memory region.

[    1.340026] [<c0013b60>] (__dabt_svc) from [<c02be958>] (pcs_readw+0x0/0x10)
[    1.347381] [<c02be958>] (pcs_readw) from [<c02c0700>] (pcs_probe+0x388/0x804)
[    1.354888] [<c02c0700>] (pcs_probe) from [<c038b544>] (platform_drv_probe+0x48/0xa0)

The reason why you getting page fault is an data abort in SVC (kernel) mode. That might happens because:

1. There is no any physical memory or IO space under physical addresses, which was specified in device tree for PCS driver. (possibly the used dtb is not compatible with the soc).
2. The memory which was accessed by pcs_readw is not mapped at all (or mapped with wrong attributes).

To figure out you need to check where virtual addresses, which are accessed by pcs_readw was mapped. What physical addresses are used. Check soc manual to be sure that this physical address fits in soc memory map.