Ubuntu: updated mainline kernel, now drops to initramfs

I attempted to upgrade our system's mainline kernel and am now dropping into initramfs.

< update>: The commands I used were:

apt-get update
apt-get upgrade
apt-cache search linux-image
//grabbed last entry
apt-get install linux-image-4.4.0-67-generic

< /update>

I have researched the topic and found that I should be able to access the original kernel through the grub menu, however, attempts at holding shift or ESC do not result in bringing up the menu. This could be due to coming in over a serial port.

Some details on the system, as the server is not a standard tower. Working on a ARM cluster, 16 nodes in the cluster, only access to the nodes is via serial port through chassis, a virtual serial port through chassis manager, and over ssh (when nodes are actually able to boot). The chassis does not have a standard USB port to load a live disc (although it doesn't have a miniUSB port).

I tried updating a single node, so 15 nodes still exist in working state and I can see debugging information from them if needed. Connecting to the failed node via a virtual serial port, I see the following output at startup:

U-Boot 2013.04 (Nov 03 2014 - 12:38:51)

ProLiant m400 Server Cartridge - U02 (10/02/2014)
Copyright 2013 - 2014 Hewlett-Packard Development Company, L.P.
Copyright 2000 - 2012 Wolfgang Denk, DENX Software Engineering, wd@denx.de


CPU0: APM ARM 64-bit Potenza Rev B0 2400MHz PCP 2400MHz
     32 KB ICACHE, 32 KB DCACHE
     SOC 2000MHz IOBAXI 400MHz AXI 250MHz AHB 200MHz GFC 125MHz
Boot from SPI-NOR
Slimpro FW: Ver: 2.2 (build 2014/09/23)
I2C:   ready
DRAM:  ECC 64 GiB @ 1333MHz
relocation Address is: 0x4ffff27000
Using default environment

API sig @ 0x0000004ffdf17170
In:    serial
Out:   serial
Err:   serial
CPUs:  11111111
CPLD: 0A
PCIE3: (RC) X8 GEN-2 link up
  00:00.0     - 19aa:e008 - Bridge device
   01:00.0    - 15b3:1007 - Network controller
SF: Detected MX25L12805D with page size 64 KiB, total 16 MiB
SF: 16384 KiB MX25L12805D at 0:0 is now current device

SF: flash read success (18319 bytes @ 0xe0000)
.
SF: flash read success (65568 bytes @ 0xc0000)
Node Boot Start Time: 2017-03-15T04:22:06
Node Serial Number: CN7505VJ3S
Cartridge Chassis Slot ID: 1
Cartridge Serial Number: CN7505VJ3S
Chassis Serial Number: USE516NS6D
Chassis Asset Tag:
Node UUID: FE95EDB2-4175-5E16-BC19-E52E0B7FC04A
Product ID: 721717-B21
Timezone Name: America/Los_Angeles
SCSI:  Target spinup took 0 ms.
AHCI2 0001.0300 32 slots 2 ports 6 Gbps 0x3 impl SATA mode
flags: 64bit ncq pm only pmp fbss pio slum part ccc
scanning bus for devices...
  Device 0: (4:0) Vendor: ATA      Prod.: XR0120GEBLT Rev: HPS4
            Type: Hard Disk
            Capacity: 114473.4 MB = 111.7 GB (234441648 x 512)
Found 1 device(s).
Searching for SCSI boot device ...
Trying scsi 0 /boot.scr...
252 bytes read in 31 ms (7.8 KiB/s)
Verifying...done
Boot: M.2, PXE
Mellanox ConnectX3 U-Boot driver version 1.1
Mellanox ConnectX3 Firmware Version 2.32.1092
Net:   NIC1 [PRIME], NIC2

Booting M.2
252 bytes read in 32 ms (6.8 KiB/s)
## Executing script at 4004000000
14393920 bytes read in 386 ms (35.6 MiB/s)
30092224 bytes read in 775 ms (37 MiB/s)
## Booting kernel from Legacy Image at 4002000000 ...
   Image Name:   kernel 4.4.0-66-generic
   Created:      2017-03-13  20:59:56 UTC
   Image Type:   ARM Linux Kernel Image (uncompressed)
   Data Size:    14393856 Bytes = 13.7 MiB
   Load Address: 00080000
   Entry Point:  00080000
   Verifying Checksum ... OK
## Loading init Ramdisk from Legacy Image at 4005000000 ...
   Image Name:   ramdisk 4.4.0-66-generic
   Created:      2017-03-13  20:59:57 UTC
   Image Type:   ARM Linux RAMDisk Image (gzip compressed)
   Data Size:    30092160 Bytes = 28.7 MiB
   Load Address: 00000000
   Entry Point:  00000000
   Verifying Checksum ... OK
## Flattened Device Tree blob at 4003000000
   Booting using the fdt blob at 0x0000004003000000
   Loading Kernel Image ... OK
OK
   Loading Ramdisk to 4fee34d000, end 4feffffb80 ... OK
   Loading Device Tree to 0000004000ff8000, end 0000004000fff78e ... OK

Starting kernel ...

L3C: 8MB
[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Initializing cgroup subsys cpuacct
[    0.000000] Linux version 4.4.0-66-generic (buildd@bos01-arm64-009) (gcc version 4.8.4 (Ubuntu/Linaro 4.8.4-2ubuntu1~14.04.3) ) #87~14.04.1-Ubuntu SMP Sat Mar 4 06:58:48 UTC 2017 (Ubuntu 4.4.0-66.87~14.04.1-generic 4.4.44)
[    0.000000] Boot CPU: AArch64 Processor [500f0001]
[    0.000000] efi: Getting EFI parameters from FDT:
[    0.000000] efi: UEFI not found.
[    0.000000] No NUMA configuration found
[    0.000000] NUMA: Faking a node at [mem 0x0000000000000000-0x0000004fffffffff]
[    0.000000] NUMA: Adding memblock [0x4000000000 - 0x4fffffffff] on node 0
[    0.000000] NUMA: Initmem setup node 0 [mem 0x4000000000-0x4fffffffff]
[    0.000000] NUMA: NODE_DATA [mem 0x4ffffe8280-0x4ffffe9fff]
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000004000000000-0x00000040ffffffff]
[    0.000000]   Normal   [mem 0x0000004100000000-0x0000004fffffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000004000000000-0x0000004fffffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000004000000000-0x0000004fffffffff]
[    0.000000] WARNING: x1-x3 nonzero in violation of boot protocol:
[    0.000000]  x1: 0000000000000000
[    0.000000]  x2: 0000000002faf080
[    0.000000]  x3: 0000000000000001
[    0.000000] This indicates a broken bootloader or old kernel
[    0.000000] PERCPU: Embedded 17 pages/cpu @ffff800ffff2d000 s31360 r8192 d30080 u69632
[    0.000000] Detected PIPT I-cache on CPU0
[    0.000000] Built 1 zonelists in Node order, mobility grouping on.  Total pages: 16515072
[    0.000000] Policy zone: Normal
[    0.000000] Kernel command line: console=ttyS0,9600n8r ro
[    0.000000] log_buf_len individual max cpu contribution: 4096 bytes
[    0.000000] log_buf_len total cpu_extra contributions: 28672 bytes
[    0.000000] log_buf_len min size: 16384 bytes
[    0.000000] log_buf_len: 65536 bytes
[    0.000000] early log buf free: 13860(84%)
[    0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
[    0.000000] software IO TLB [mem 0x40fbfff000-0x40fffff000] (64MB) mapped at [ffff8000fbfff000-ffff8000ffffefff]
[    0.000000] Memory: 65949020K/67108864K available (8396K kernel code, 984K rwdata, 3832K rodata, 768K init, 786K bss, 1159844K reserved, 0K cma-reserved)
[    0.000000] Virtual kernel memory layout:
[    0.000000]     vmalloc : 0xffff000000000000 - 0xffff7bffbfff0000   (126974 GB)
[    0.000000]     vmemmap : 0xffff7bffc0000000 - 0xffff7fffc0000000   (  4096 GB maximum)
[    0.000000]               0xffff7bffc0000000 - 0xffff7c0000000000   (  1024 MB actual)
[    0.000000]     fixed   : 0xffff7ffffa7fd000 - 0xffff7ffffac00000   (  4108 KB)
[    0.000000]     PCI I/O : 0xffff7ffffae00000 - 0xffff7ffffbe00000   (    16 MB)
[    0.000000]     modules : 0xffff7ffffc000000 - 0xffff800000000000   (    64 MB)
[    0.000000]     memory  : 0xffff800000000000 - 0xffff801000000000   ( 65536 MB)
[    0.000000]       .init : 0xffff800000c73000 - 0xffff800000d33000   (   768 KB)
[    0.000000]       .text : 0xffff800000080000 - 0xffff800000c73000   ( 12236 KB)
[    0.000000]       .data : 0xffff800000d44000 - 0xffff800000e3a200   (   985 KB)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=8, Nodes=1
[    0.000000] Hierarchical RCU implementation.
[    0.000000]  Build-time adjustment of leaf fanout to 64.
[    0.000000]  RCU restricting CPUs from NR_CPUS=128 to nr_cpu_ids=8.
[    0.000000] RCU: Adjusting geometry for rcu_fanout_leaf=64, nr_cpu_ids=8
[    0.000000] NR_IRQS:64 nr_irqs:64 0
[    0.000000] Architected cp15 timer(s) running at 50.00MHz (phys).
[    0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0xb8812736b, max_idle_ns: 440795202655 ns
[    0.000002] sched_clock: 56 bits at 50MHz, resolution 20ns, wraps every 4398046511100ns
[    0.000080] Console: colour dummy device 80x25
[    0.000108] Calibrating delay loop (skipped), value calculated using timer frequency.. 100.00 BogoMIPS (lpj=200000)
[    0.000113] pid_max: default: 32768 minimum: 301
[    0.000142] Security Framework initialized
[    0.000146] Yama: becoming mindful.
[    0.000179] AppArmor: AppArmor initialized
[    0.008069] Dentry cache hash table entries: 8388608 (order: 14, 67108864 bytes)
[    0.022824] Inode-cache hash table entries: 4194304 (order: 13, 33554432 bytes)
[    0.029218] Mount-cache hash table entries: 131072 (order: 8, 1048576 bytes)
[    0.029347] Mountpoint-cache hash table entries: 131072 (order: 8, 1048576 bytes)
[    0.029955] Initializing cgroup subsys io
[    0.029965] Initializing cgroup subsys memory
[    0.029979] Initializing cgroup subsys devices
[    0.029984] Initializing cgroup subsys freezer
[    0.029988] Initializing cgroup subsys net_cls
[    0.029995] Initializing cgroup subsys perf_event
[    0.029999] Initializing cgroup subsys net_prio
[    0.030003] Initializing cgroup subsys hugetlb
[    0.030006] Initializing cgroup subsys pids
[    0.030024] ftrace: allocating 30538 entries in 120 pages
[    0.054710] EFI services will not be available.
[    0.054720] ASID allocator initialised with 65536 entries
[    0.055955] Detected PIPT I-cache on CPU1
[    0.055981] CPU1: Booted secondary processor [500f0001]
[    0.056147] Detected PIPT I-cache on CPU2
[    0.056167] CPU2: Booted secondary processor [500f0001]
[    0.056322] Detected PIPT I-cache on CPU3
[    0.056334] CPU3: Booted secondary processor [500f0001]
[    0.056489] Detected PIPT I-cache on CPU4
[    0.056509] CPU4: Booted secondary processor [500f0001]
[    0.056653] Detected PIPT I-cache on CPU5
[    0.056665] CPU5: Booted secondary processor [500f0001]
[    0.056816] Detected PIPT I-cache on CPU6
[    0.056836] CPU6: Booted secondary processor [500f0001]
[    0.056985] Detected PIPT I-cache on CPU7
[    0.056997] CPU7: Booted secondary processor [500f0001]
[    0.057024] Brought up 8 CPUs
[    0.057039] SMP: Total of 8 processors activated.
[    0.057043] CPU: All CPU(s) started at EL2
[    0.057395] devtmpfs: initialized
[    0.058640] evm: security.selinux
[    0.058644] evm: security.SMACK64
[    0.058646] evm: security.SMACK64EXEC
[    0.058648] evm: security.SMACK64TRANSMUTE
[    0.058649] evm: security.SMACK64MMAP
[    0.058651] evm: security.ima
[    0.058653] evm: security.capability
[    0.058736] DMI not present or invalid.
[    0.058843] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    0.059009] pinctrl core: initialized pinctrl subsystem
[    0.059401] NET: Registered protocol family 16
[    0.070568] cpuidle: using governor ladder
[    0.082573] cpuidle: using governor menu
[    0.082595] vdso: 2 pages (1 code @ ffff800000d49000, 1 data @ ffff800000d48000)
[    0.082608] hw-breakpoint: found 4 breakpoint and 4 watchpoint registers.
[    0.082654] DMA: preallocated 256 KiB pool for atomic allocations
[    0.082718] Serial: AMBA PL011 UART driver
[    0.099240] ACPI: Interpreter disabled.
[    0.099448] vgaarb: loaded
[    0.099917] SCSI subsystem initialized
[    0.100080] usbcore: registered new interface driver usbfs
[    0.100099] usbcore: registered new interface driver hub
[    0.100134] usbcore: registered new device driver usb
[    0.100339] dmi: Firmware registration failed.
[    0.100650] NetLabel: Initializing
[    0.100653] NetLabel:  domain hash size = 128
[    0.100655] NetLabel:  protocols = UNLABELED CIPSOv4
[    0.100676] NetLabel:  unlabeled traffic allowed by default
[    0.100786] clocksource: Switched to clocksource arch_sys_counter
[    0.114898] AppArmor: AppArmor Filesystem Enabled
[    0.115028] pnp: PnP ACPI: disabled
[    0.118228] NET: Registered protocol family 2
[    0.118823] TCP established hash table entries: 524288 (order: 10, 4194304 bytes)
[    0.120243] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
[    0.120614] TCP: Hash tables configured (established 524288 bind 65536)
[    0.120742] UDP hash table entries: 32768 (order: 8, 1048576 bytes)
[    0.121384] UDP-Lite hash table entries: 32768 (order: 8, 1048576 bytes)
[    0.122012] NET: Registered protocol family 1
[    0.122138] Trying to unpack rootfs image as initramfs...
[    0.746005] Freeing initrd memory: 29384K (ffff800fee34d000 - ffff800feffff000)
[    0.746162] hw perfevents: enabled with armv8_pmuv3 PMU driver, 5 counters available
[    0.746314] kvm [1]: interrupt-controller@780c0000 IRQ5
[    0.746440] kvm [1]: timer IRQ3
[    0.746451] kvm [1]: Hyp mode initialized successfully
[    0.747101] futex hash table entries: 2048 (order: 6, 262144 bytes)
[    0.747154] audit: initializing netlink subsys (disabled)
[    0.747191] audit: type=2000 audit(0.728:1): initialized
[    0.747360] Initialise system trusted keyring
[    0.747560] HugeTLB registered 2 MB page size, pre-allocated 0 pages
[    0.750244] zbud: loaded
[    0.750633] VFS: Disk quotas dquot_6.6.0
[    0.750700] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    0.751236] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    0.751761] fuse init (API version 7.23)
[    0.752032] Key type big_key registered
[    0.752072] Allocating IMA MOK and blacklist keyrings.
[    0.752638] Key type asymmetric registered
[    0.752644] Asymmetric key parser 'x509' registered
[    0.752723] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 249)
[    0.752792] io scheduler noop registered
[    0.752798] io scheduler deadline registered (default)
[    0.752865] io scheduler cfq registered
[    0.753288] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
[    0.753303] pciehp: PCI Express Hot Plug Controller Driver version: 0.4
[    0.753382] PCI host bridge /soc/pcie@1f500000 ranges:
[    0.753387]   No bus range found for /soc/pcie@1f500000, using [bus 00-ff]
[    0.753396]   MEM 0xa130000000..0xa1afffffff -> 0x30000000
[    0.753427] xgene-pcie 1f500000.pcie: (rc) x8 gen-2 link up
[    0.753489] xgene-pcie 1f500000.pcie: PCI host bridge to bus 0000:00
[    0.753494] pci_bus 0000:00: root bus resource [bus 00-ff]
[    0.753498] pci_bus 0000:00: root bus resource [mem 0xa130000000-0xa1afffffff] (bus address [0x30000000-0xafffffff])
[    0.756796] pci 0000:01:00.0: VF(n) BAR2 space: [mem 0x00000000-0x03ffffff 64bit pref] (contains BAR2 for 8 VFs)
[    0.764916] pci 0000:00:00.0: BAR 15: assigned [mem 0xa130000000-0xa1347fffff 64bit pref]
[    0.764919] pci 0000:00:00.0: BAR 14: assigned [mem 0xa134800000-0xa1348fffff]
[    0.764928] pci 0000:01:00.0: BAR 2: assigned [mem 0xa130000000-0xa1307fffff 64bit pref]
[    0.765107] pci 0000:01:00.0: BAR 9: assigned [mem 0xa130800000-0xa1347fffff 64bit pref]
[    0.765174] pci 0000:01:00.0: BAR 0: assigned [mem 0xa134800000-0xa1348fffff 64bit]
[    0.765356] pci 0000:00:00.0: PCI bridge to [bus 01]
[    0.765363] pci 0000:00:00.0:   bridge window [mem 0xa134800000-0xa1348fffff]
[    0.765368] pci 0000:00:00.0:   bridge window [mem 0xa130000000-0xa1347fffff 64bit pref]
[    0.765467] pcieport 0000:00:00.0: Signaling PME through PCIe PME interrupt
[    0.765470] pci 0000:01:00.0: Signaling PME through PCIe PME interrupt
[    0.766009] Serial: 8250/16550 driver, 32 ports, IRQ sharing enabled
[    0.768395] console [ttyS0] disabled
[    0.788556] 1c021000.serial: ttyS0 at MMIO 0x1c021000 (irq = 27, base_baud = 3125000) is a U6_16550A
[   13.140347] console [ttyS0] enabled
[   13.182405] msm_serial: driver initialized
[   13.231876] Unable to detect cache hierarchy from DT for CPU 0
[   13.306141] brd: module loaded
[   13.345130] loop: module loaded
[   13.383175] libphy: Fixed MDIO Bus: probed
[   13.432321] tun: Universal TUN/TAP device driver, 1.6
[   13.492935] tun: (C) 1999-2004 Max Krasnyansky 
[   13.567257] PPP generic driver version 2.4.2
[   13.618610] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[   13.696969] ehci-pci: EHCI PCI platform driver
[   13.750296] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[   13.824491] ohci-pci: OHCI PCI platform driver
[   13.877821] uhci_hcd: USB Universal Host Controller Interface driver
[   13.954246] mousedev: PS/2 mouse device common for all mice
[   14.021414] xgene-rtc 10510000.rtc: rtc core: registered 10510000.rtc as rtc0
[   14.107091] i2c /dev entries driver
[   14.149250] device-mapper: uevent: version 1.0.3
[   14.204794] device-mapper: ioctl: 4.34.0-ioctl (2015-10-28) initialised: dm-devel@redhat.com
[   14.306197] ledtrig-cpu: registered to indicate activity on CPUs
[   14.378683] NET: Registered protocol family 10
[   14.432335] NET: Registered protocol family 17
[   14.485666] Key type dns_resolver registered
[   14.536920] Registered cp15_barrier emulation handler
[   14.597536] Registered setend emulation handler
[   14.652145] registered taskstats version 1
[   14.701309] Loading compiled-in X.509 certificates
[   14.760145] Loaded X.509 cert 'Build time autogenerated kernel key: 406f2834fe2e92818297d6e65094bdfdf2455038'
[   14.954278] zswap: loaded using pool lzo/zbud
[   15.009671] Key type trusted registered
[   15.066578] Key type encrypted registered
[   15.114686] AppArmor: AppArmor sha1 policy hashing enabled
[   15.180515] ima: No TPM chip found, activating TPM-bypass!
[   15.246390] evm: HMAC attrs: 0x1
[   15.285359] xgene-rtc 10510000.rtc: setting system clock to 2017-03-15 04:22:20 UTC (1489551740)
[   15.391396] Freeing unused kernel memory: 768K (ffff800000c73000 - ffff800000d33000)
[   15.484364] Freeing alternatives memory: 48K (ffff800000d33000 - ffff800000d3f000)
Loading, please wait...
[   15.596007] systemd-udevd[164]: starting version 204
[   15.681757] xgene-ahci 1a800000.sata: can't request region for resource [mem 0x1f23a000-0x1f23a0ff]
[   15.685895] mlx4_core: Mellanox ConnectX core driver v2.2-1 (Feb, 2014)
[   15.685902] mlx4_core: Initializing 0000:01:00.0
[   15.925188] xgene-ahci: probe of 1a800000.sata failed with error -16
[   22.670250] mlx4_core 0000:01:00.0: PCIe BW is different than device's capability
[   22.760079] mlx4_core 0000:01:00.0: PCIe link speed is 5.0GT/s, device supports 8.0GT/s
[   22.856163] mlx4_core 0000:01:00.0: PCIe link width is x8, device supports x8
[   22.951830] pps_core: LinuxPPS API ver. 1 registered
[   23.011406] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti 
[   23.123078] PTP clock support registered
[   23.180540] mlx4_en: Mellanox ConnectX HCA Ethernet driver v2.2-1 (Feb 2014)
[   23.265360] mlx4_en 0000:01:00.0: Activating port:1
[   23.329356] mlx4_en: 0000:01:00.0: Port 1: Using 64 TX rings
[   23.397276] mlx4_en: 0000:01:00.0: Port 1: Using 4 RX rings
[   23.464152] mlx4_en: 0000:01:00.0: Port 1:   frag:0 - size:1522 prefix:0 stride:1536
[   23.557425] mlx4_en: 0000:01:00.0: Port 1: Initializing port
[   23.625693] mlx4_en 0000:01:00.0: registered PHC clock
[   23.688678] mlx4_en 0000:01:00.0: Activating port:2
[   23.754351] mlx4_en: 0000:01:00.0: Port 2: Using 64 TX rings
[   23.822285] mlx4_en: 0000:01:00.0: Port 2: Using 4 RX rings
[   23.889180] mlx4_en: 0000:01:00.0: Port 2:   frag:0 - size:1522 prefix:0 stride:1536
[   23.982542] mlx4_en: 0000:01:00.0: Port 2: Initializing port
[   24.622503] mlx4_en: eth0: Link Up
Begin: Loading essential drivers ... done.
Begin: Running /scripts/init-premount ... done.
Begin: Mounting root file system ... Begin: Running /scripts/local-top ... done.
Begin: Loading essential drivers ... done.
Begin: Running /scr[   55.175695] hidraw: raw HID events driver (C) Jiri Kosina
ipts/init-premount ... done.
Begin: Mounting root file system [   55.300769] usbcore: registered new interface driver usbhid
... Begin: Running /scripts/local-top ... done.
Gave up waitin[   55.431163] usbhid: USB HID core driver
g for root device.  Common problems:
 - Boot args (cat /proc/cmdline)
   - Check rootdelay= (did the system wait long enough?)
   - Check root= (did the system wait for the right device?)
 - Missing modules (cat /proc/modules; ls /dev)
ALERT!  /dev/disk/by-label/cloudimg-rootfs does not exist.  Dropping to a shell!


BusyBox v1.21.1 (Ubuntu 1:1.21.0-1ubuntu1) built-in shell (ash)
Enter 'help' for a list of built-in commands.

(initramfs)

I have noticed that while holding left shift during boot sequence, nothing changes from above output. However, holding ESC does result in the system trying to do a PXE boot (which fails as I've never set this up).

Update: Adding additional debugging output from trying to mount a drive within initramfs (no /etc/fstab or /dev/sdaX).

(initramfs) cat /proc/cmdline
console=ttyS0,9600n8r ro

(initramfs) cat /proc/modules
usbhid 61440 0 - Live 0xffff7ffffc1b7000
hid 131072 1 usbhid, Live 0xffff7ffffc18b000
mlx4_en 126976 0 - Live 0xffff7ffffc131000
vxlan 53248 1 mlx4_en, Live 0xffff7ffffc107000
ip6_udp_tunnel 16384 1 vxlan, Live 0xffff7ffffc0f9000
udp_tunnel 16384 1 vxlan, Live 0xffff7ffffc0e9000
ptp 24576 1 mlx4_en, Live 0xffff7ffffc0d1000
pps_core 24576 1 ptp, Live 0xffff7ffffc0b9000
mlx4_core 335872 1 mlx4_en, Live 0xffff7ffffc046000
ahci_xgene 16384 0 - Live 0xffff7ffffc03d000
libahci_platform 24576 1 ahci_xgene, Live 0xffff7ffffc032000
libahci 45056 2 ahci_xgene,libahci_platform, Live 0xffff7ffffc01f000
aes_neon_blk 20480 0 - Live 0xffff7ffffc015000
ablk_helper 16384 1 aes_neon_blk, Live 0xffff7ffffc00d000
cryptd 24576 1 ablk_helper, Live 0xffff7ffffc000000

(initramfs) ls -l /
drwxr-xr-x    9      3620 dev
drwx------    2        40 root
drwxr-xr-x    2       620 bin
drwxr-xr-x    2       340 sbin
drwxr-xr-x    5       100 run
drwxr-xr-x    9       240 lib
drwxr-xr-x    7       200 etc
drwxr-xr-x    8       220 scripts
drwxr-xr-x    3       100 conf
-rwxr-xr-x    1      6664 init
dr-xr-xr-x   13         0 sys
dr-xr-xr-x  118         0 proc
drwxr-xr-x    2        40 tmp
drwxr-xr-x    3        60 var

(initramfs) ls -l dev/
crw-------    1   10,  63 vga_arbiter
crw-r--r--    1   10,  62 rfkill
crw-r-----    1    1,   1 mem
crw-rw-rw-    1    1,   3 null
crw-r-----    1    1,   4 port
crw-rw-rw-    1    1,   5 zero
crw-rw-rw-    1    1,   7 full
crw-rw-rw-    1    1,   8 random
crw-rw-rw-    1    1,   9 urandom
crw-r--r--    1    1,  11 kmsg
crw-rw-rw-    1    5,   0 tty
crw-------    1    5,   1 console
crw--w----    1    4,   0 tty0
crw-------    1    7,   0 vcs
crw-------    1    7, 128 vcsa
crw-------    1    7,   1 vcs1
crw-------    1    7, 129 vcsa1
crw--w----    1    4,   1 tty1
crw--w----    1    4,   2 tty2
crw--w----    1    4,   3 tty3
crw--w----    1    4,   4 tty4
crw--w----    1    4,   5 tty5
crw--w----    1    4,   6 tty6
crw--w----    1    4,   7 tty7
crw--w----    1    4,   8 tty8
crw--w----    1    4,   9 tty9
crw--w----    1    4,  10 tty10
crw--w----    1    4,  11 tty11
crw--w----    1    4,  12 tty12
crw--w----    1    4,  13 tty13
crw--w----    1    4,  14 tty14
crw--w----    1    4,  15 tty15
crw--w----    1    4,  16 tty16
crw--w----    1    4,  17 tty17
crw--w----    1    4,  18 tty18
crw--w----    1    4,  19 tty19
crw--w----    1    4,  20 tty20
crw--w----    1    4,  21 tty21
crw--w----    1    4,  22 tty22
crw--w----    1    4,  23 tty23
crw--w----    1    4,  24 tty24
crw--w----    1    4,  25 tty25
crw--w----    1    4,  26 tty26
crw--w----    1    4,  27 tty27
crw--w----    1    4,  28 tty28
crw--w----    1    4,  29 tty29
crw--w----    1    4,  30 tty30
crw--w----    1    4,  31 tty31
crw--w----    1    4,  32 tty32
crw--w----    1    4,  33 tty33
crw--w----    1    4,  34 tty34
crw--w----    1    4,  35 tty35
crw--w----    1    4,  36 tty36
crw--w----    1    4,  37 tty37
crw--w----    1    4,  38 tty38
crw--w----    1    4,  39 tty39
crw--w----    1    4,  40 tty40
crw--w----    1    4,  41 tty41
crw--w----    1    4,  42 tty42
crw--w----    1    4,  43 tty43
crw--w----    1    4,  44 tty44
crw--w----    1    4,  45 tty45
crw--w----    1    4,  46 tty46
crw--w----    1    4,  47 tty47
crw--w----    1    4,  48 tty48
crw--w----    1    4,  49 tty49
crw--w----    1    4,  50 tty50
crw--w----    1    4,  51 tty51
crw--w----    1    4,  52 tty52
crw--w----    1    4,  53 tty53
crw--w----    1    4,  54 tty54
crw--w----    1    4,  55 tty55
crw--w----    1    4,  56 tty56
crw--w----    1    4,  57 tty57
crw--w----    1    4,  58 tty58
crw--w----    1    4,  59 tty59
crw--w----    1    4,  60 tty60
crw--w----    1    4,  61 tty61
crw--w----    1    4,  62 tty62
crw--w----    1    4,  63 tty63
crw-------    1   10, 232 kvm
crw-------    1   10,  61 ecryptfs
crw-rw-rw-    1   10, 229 fuse
crw-rw-rw-    1    5,   2 ptmx
crw-------    1    4,  65 ttyS1
crw-------    1    4,  66 ttyS2
crw-------    1    4,  67 ttyS3
crw-------    1    4,  68 ttyS4
crw-------    1    4,  69 ttyS5
crw-------    1    4,  70 ttyS6
crw-------    1    4,  71 ttyS7
crw-------    1    4,  72 ttyS8
crw-------    1    4,  73 ttyS9
crw-------    1    4,  74 ttyS10
crw-------    1    4,  75 ttyS11
crw-------    1    4,  76 ttyS12
crw-------    1    4,  77 ttyS13
crw-------    1    4,  78 ttyS14
crw-------    1    4,  79 ttyS15
crw-------    1    4,  80 ttyS16
crw-------    1    4,  81 ttyS17
crw-------    1    4,  82 ttyS18
crw-------    1    4,  83 ttyS19
crw-------    1    4,  84 ttyS20
crw-------    1    4,  85 ttyS21
crw-------    1    4,  86 ttyS22
crw-------    1    4,  87 ttyS23
crw-------    1    4,  88 ttyS24
crw-------    1    4,  89 ttyS25
crw-------    1    4,  90 ttyS26
crw-------    1    4,  91 ttyS27
crw-------    1    4,  92 ttyS28
crw-------    1    4,  93 ttyS29
crw-------    1    4,  94 ttyS30
crw-------    1    4,  95 ttyS31
crw-------    1    4,  64 ttyS0
crw-------    1    5,   3 ttyprintk
crw-------    1   10, 183 hwrng
drwxr-xr-x    2        60 lightnvm
brw-------    1    1,   0 ram0
brw-------    1    1,   1 ram1
brw-------    1    1,   2 ram2
brw-------    1    1,   3 ram3
brw-------    1    1,   4 ram4
brw-------    1    1,   5 ram5
brw-------    1    1,   6 ram6
brw-------    1    1,   7 ram7
brw-------    1    1,   8 ram8
brw-------    1    1,   9 ram9
brw-------    1    1,  10 ram10
brw-------    1    1,  11 ram11
brw-------    1    1,  12 ram12
brw-------    1    1,  13 ram13
brw-------    1    1,  14 ram14
brw-------    1    1,  15 ram15
crw-------    1   10, 237 loop-control
brw-------    1    7,   0 loop0
brw-------    1    7,   1 loop1
brw-------    1    7,   2 loop2
brw-------    1    7,   3 loop3
brw-------    1    7,   4 loop4
brw-------    1    7,   5 loop5
brw-------    1    7,   6 loop6
brw-------    1    7,   7 loop7
drwxr-xr-x    2        60 net
crw-------    1  108,   0 ppp
drwxr-xr-x    2        60 input
crw-------    1   10,   1 psaux
crw-------    1   10, 223 uinput
crw-------    1  251,   0 rtc0
drwxr-xr-x    2        60 mapper
crw-------    1   10,  59 cpu_dma_latency
crw-------    1   10,  58 network_latency
crw-------    1   10,  57 network_throughput
crw-------    1   10,  56 memory_bandwidth
drwxr-xr-x    2         0 pts
lrwxrwxrwx    1        11 core -> /proc/kcore
lrwxrwxrwx    1        13 fd -> /proc/self/fd
lrwxrwxrwx    1        15 stdin -> /proc/self/fd/0
lrwxrwxrwx    1        15 stdout -> /proc/self/fd/1
lrwxrwxrwx    1        15 stderr -> /proc/self/fd/2
crw-------    1   10, 234 btrfs-control
drwxr-xr-x    2      2920 char
lrwxrwxrwx    1         4 rtc -> rtc0
drwxr-xr-x    2       520 block
crw-------    1  246,   0 ptp0
crw-------    1    7,   2 vcs2
crw-------    1    7, 130 vcsa2
crw-------    1    7,   3 vcs3
crw-------    1    7, 131 vcsa3
crw-------    1    7,   4 vcs4
crw-------    1    7, 132 vcsa4
crw-------    1    7,   5 vcs5
crw-------    1    7, 133 vcsa5
crw-------    1    7,   6 vcs6
crw-------    1    7, 134 vcsa6
crw-------    1    7,   7 vcs7
crw-------    1    7, 135 vcsa7

I got this to work. A very specialized case, but just in case it helps someone in the future...

The 16 nodes are all cartridges. I removed the faulty cartridge trying to directly access the SSD. It looked like a DIMM (M.2 SSD). I made sure it was safe to remove, then found an external enclosure at Frys. Plugged it into a Linux box (important, Windows didn't recognize the file system) and had access to all the data. I backed up everything, then copied everything from /boot on a working cartridge over to the non-booting one. Put the SSD back into the cartridge, powered on, works!

So. If you've exhausted all other options, try getting access to the harddrive.