GRUB2 fails to boot multiboot compliant kernel that worked with GRUB Legacy

I have been trying to port a non-Linux mostly multiboot 0.6.96 compliant (the multiboot header is past 8 KiB) 32-bit elf kernel from GRUB Legacy to GRUB2. Currently, the kernel and GRUB are placed on a USB which is used to boot a different computer via the BIOS, like a live USB. The computer I have been testing on is an Intel i5 Skylake. Previously, GRUB Legacy was able to boot the kernel with this configuration:

# grub.cfg
timeout 5
title "USB Boot"
kernel /boot/kernel
boot

For GRUB2 I changed it to:

# grub.cfg
set timeout=5
debug=all
serial --speed=115200
terminal_output --append serial

menuentry "USB Boot" {
    set root="(hd0,msdos1)" # USB drive
    multiboot /boot/kernel 
    boot
}

But this results in GRUB2 having a forever blinking cursor after attempting to jump to the kernel.

To get this far I built GRUB2 from scratch with:

wget ftp://ftp.gnu.org/gnu/grub/grub-2.02.tar.gz
tar -xzvf grub-2.02.tar.gz
cd grub-2.02
# Change MULTIBOOT_SEARCH from 8192 to 32708 in include/multiboot.h
./autogen.sh
./configure

The final output from ./configure was:

*******************************************************
GRUB2 will be compiled with following components:
Platform: i386-pc
With devmapper support: No (need libdevmapper header)
With memory debugging: No
With disk cache statistics: No
With boot time statistics: No
efiemu runtime: Yes
grub-mkfont: No (need freetype2 library)
grub-mount: No (need FUSE library)
starfield theme: No (No build-time grub-mkfont)
With libzfs support: No (need zfs library)
Build-time grub-mkfont: No (need freetype2 library)
Without unifont (no build-time grub-mkfont)
With liblzma from -llzma (support for XZ-compressed mips images)
*******************************************************

I made GRUB2 and populated the USB with:

make -j12
# Some fdisk formating to the USB removed for brevity
sudo mount /dev/sdc1 /mnt # /dev/sdc is the USB GRUB will attempt to boot off of
sudo ./grub-install --no-floppy --boot-directory=/mnt/boot --target=i386-pc --directory=grub-core /dev/sdc
sudo cp grub.cfg /mnt/boot/grub/grub.cfg # grub.cfg is the GRUB2 config file from above
sudo cp kernel /mnt/boot/
sync
sudo umount /mnt

I moved the USB from my computer to the test machine and attempted to boot the kernel. The debug output from the attempt was:

script/script.c:65: free 0xd9d6f8e0
  Booting `USB Boot'free 0xd9d6f900
script/script.c:65: free 0xd9d6f920
script/lexer.c:321: token 288 text [setparams]
script/script.c:50: malloc 0xd9d70830
script/script.c:50: malloc 0xd9d70800
script/script.c:163: arglist9d6cee0
script/script.c:50: malloc 0xd9d707d0
script/lexer.c:321: token 289 text []
script/script.c:50: malloc 0xd9d706a0
script/script.c:50: malloc 0xcba8c4a0
script/lexer.c:321: token 289 text [USB Boot]
script/script.c:50: malloc 0xd9d70670
script/script.c:50: malloc 0xd9d70650
script/lexer.c:321: token 289 text []
script/script.c:50: malloc 0xd9d70620
script/script.c:50: malloc 0xcba900d0
script/script.c:163: arglist
script/script.c:50: malloc 0xd9d705f0
script/lexer.c:321: token 259 text [
]
script/script.c:50: malloc 0xd9d705c0
script/script.c:50: malloc 0xd9d705a0
script/script.c:198: cmdline
script/script.c:50: malloc 0xd9d70570
script/lexer.c:321: token 0 text []
script/script.c:50: malloc 0xd9d708d0
script/script.c:50: malloc 0xd9d708b0
script/script.c:294: append command
script/script.c:50: malloc 0xd9d70550
script/script.c:65: free 0xd9d70550
script/script.c:65: free 0xd9d708b0
script/script.c:65: free 0xd9d708d0
script/script.c:65: free 0xd9d70570
script/script.c:65: free 0xd9d705a0
script/script.c:65: free 0xd9d705c0
script/script.c:65: free 0xd9d705f0
script/script.c:65: free 0xcba900d0
script/script.c:65: free 0xd9d70620
script/script.c:65: free 0xd9d70650
script/script.c:65: free 0xd9d70670
script/script.c:65: free 0xcba8c4a0
script/script.c:65: free 0xd9d706a0
script/script.c:65: free 0xd9d707d0
script/script.c:65: free 0xd9d70800
script/script.c:65: free 0xd9d70830
script/lexer.c:321: token 259 text [
]
script/script.c:50: malloc 0xd9d707b0
script/script.c:50: malloc 0xcba900d0
script/lexer.c:321: token 0 text []
script/script.c:50: malloc 0xd9d70780
script/script.c:50: malloc 0xcba8c4a0
script/script.c:65: free 0xcba8c4a0
script/script.c:65: free 0xd9d70780
script/script.c:65: free 0xcba900d0
script/script.c:65: free 0xd9d707b0
script/lexer.c:321: token 288 text [set]
script/script.c:50: malloc 0xd9d70770
script/script.c:50: malloc 0xcba8c4a0
script/script.c:163: arglist
script/script.c:50: malloc 0xd9d70740
script/lexer.c:321: token 289 text [root=]
script/script.c:50: malloc 0xd9d705e0
script/script.c:50: malloc 0xcba900d0
script/lexer.c:321: token 289 text [(hd0,msdos1)]
script/script.c:50: malloc 0xd9d705b0
script/script.c:50: malloc 0xd9d70580
script/lexer.c:321: token 289 text []
script/script.c:50: malloc 0xd9d706d0
script/script.c:50: malloc 0xd9d70560
script/script.c:163: arglist
script/script.c:50: malloc 0xd9d70530
script/lexer.c:321: token 259 text [
]
script/script.c:50: malloc 0xd9d70500
script/script.c:50: malloc 0xd9d704e0
script/script.c:198: cmdline
script/script.c:50: malloc 0xd9d704b0
script/lexer.c:321: token 0 text []
script/script.c:50: malloc 0xd9d70810
script/script.c:50: malloc 0xd9d707f0
script/script.c:294: append command
script/script.c:50: malloc 0xd9d70490
script/script.c:65: free 0xd9d70490
script/script.c:65: free 0xd9d707f0
script/script.c:65: free 0xd9d70810
script/script.c:65: free 0xd9d704b0
script/script.c:65: free 0xd9d704e0
script/script.c:65: free 0xd9d70500
script/script.c:65: free 0xd9d70530
script/script.c:65: free 0xd9d70560
script/script.c:65: free 0xd9d706d0
script/script.c:65: free 0xd9d70580
script/script.c:65: free 0xd9d705b0
script/script.c:65: free 0xcba900d0
script/script.c:65: free 0xd9d705e0
script/script.c:65: free 0xd9d70740
script/script.c:65: free 0xcba8c4a0
script/script.c:65: free 0xd9d70770
script/lexer.c:321: token 288 text [multiboot]
script/script.c:50: malloc 0xd9d70750
script/script.c:50: malloc 0xd9d70720
script/script.c:163: arglist
script/script.c:50: malloc 0xd9d706f0
script/lexer.c:321: token 289 text [/boot/kernel]
script/script.c:50: malloc 0xd9d705c0
script/script.c:50: malloc 0xd9d70590
script/script.c:163: arglist
script/script.c:50: malloc 0xd9d70560
script/lexer.c:321: token 259 text [
]
script/script.c:50: malloc 0xd9d70530
script/script.c:50: malloc 0xcba900d0
script/script.c:198: cmdline
script/script.c:50: malloc 0xd9d70500
script/lexer.c:321: token 0 text []
script/script.c:50: malloc 0xd9d707f0
script/script.c:50: malloc 0xcba8c4a0
script/script.c:294: append command
script/script.c:50: malloc 0xd9d707d0
kern/disk.c:196: Opening `hd0,msdos1'...
partmap/msdos.c:188: partition 0: flag 0x80, type 0xef, start 0x800, len
0x1cd125a
kern/fs.c:56: Detecting fat...
fs/fat.c:553: fat_size=32, next_cluster=143
kern/disk.c:295: Closing `hd0'.
kern/dl.c:602: module at 0xcba853b0, size 0x36d8
kern/dl.c:626: relocating to 0xcba8ba10
kern/disk.c:196: Opening `hd0,msdos1'...
partmap/msdos.c:188: partition 0: flag 0x80, type 0xef, start 0x800, len
0x1cd125a
kern/fs.c:56: Detecting fat...
kern/disk.c:295: Closing `hd0'.
kern/dl.c:602: module at 0xcba89f20, size 0x17a4
kern/dl.c:626: relocating to 0xcba8b880
kern/dl.c:556: flushing 0x15c4 bytes at 0xcba83dd0
kern/dl.c:649: module name: video
kern/dl.c:650: init function: 0x0
kern/disk.c:196: Opening `hd0,msdos1'...
partmap/msdos.c:188: partition 0: flag 0x80, type 0xef, start 0x800, len
0x1cd125a
kern/fs.c:56: Detecting fat...
fs/fat.c:553: fat_size=32, next_cluster=160
fs/fat.c:553: fat_size=32, next_cluster=327
fs/fat.c:553: fat_size=32, next_cluster=328
fs/fat.c:553: fat_size=32, next_cluster=329
fs/fat.c:553: fat_size=32, next_cluster=330
fs/fat.c:553: fat_size=32, next_cluster=331
kern/disk.c:295: Closing `hd0'.
kern/dl.c:602: module at 0xd9d31930, size 0xb810
kern/dl.c:626: relocating to 0xcba8b710
kern/disk.c:196: Opening `hd0,msdos1'...
partmap/msdos.c:188: partition 0: flag 0x80, type 0xef, start 0x800, len
0x1cd125a
kern/fs.c:56: Detecting fat...
fs/fat.c:553: fat_size=32, next_cluster=160
kern/disk.c:295: Closing `hd0'.
kern/dl.c:602: module at 0xcba8ad80, size 0x620
kern/dl.c:626: relocating to 0xcba8b5a0
kern/dl.c:556: flushing 0x456 bytes at 0xcba8a910
kern/dl.c:649: module name: priority_queue
kern/dl.c:650: init function: 0x0
kern/disk.c:196: Opening `hd0,msdos1'...
partmap/msdos.c:188: partition 0: flag 0x80, type 0xef, start 0x800, len
0x1cd125a
kern/fs.c:56: Detecting fat...
fs/fat.c:553: fat_size=32, next_cluster=160
fs/fat.c:553: fat_size=32, next_cluster=334
kern/disk.c:295: Closing `hd0'.
kern/dl.c:602: module at 0xcba8a1b0, size 0x4e4
kern/dl.c:626: relocating to 0xcba8b430
kern/dl.c:556: flushing 0x346 bytes at 0xcba8b0b0
kern/dl.c:649: module name: datetime
kern/dl.c:650: init function: 0x0
kern/dl.c:556: flushing 0x105d5 bytes at 0xd9cdb9c0
kern/dl.c:649: module name: net
kern/dl.c:650: init function: 0xd9cdd964
kern/disk.c:196: Opening `hd0,msdos1'...
partmap/msdos.c:188: partition 0: flag 0x80, type 0xef, start 0x800, len
0x1cd125a
kern/fs.c:56: Detecting fat...
fs/fat.c:553: fat_size=32, next_cluster=160
fs/fat.c:553: fat_size=32, next_cluster=288
kern/disk.c:295: Closing `hd0'.
kern/dl.c:602: module at 0xcba73d10, size 0x3a38
kern/dl.c:626: relocating to 0xcba7aea0
kern/disk.c:196: Opening `hd0,msdos1'...
partmap/msdos.c:188: partition 0: flag 0x80, type 0xef, start 0x800, len
0x1cd125a
kern/fs.c:56: Detecting fat...
fs/fat.c:553: fat_size=32, next_cluster=160
fs/fat.c:553: fat_size=32, next_cluster=291
kern/disk.c:295: Closing `hd0'.
kern/dl.c:602: module at 0xcba789e0, size 0x2154
kern/dl.c:626: relocating to 0xcba7ad30
kern/dl.c:556: flushing 0x1f60 bytes at 0xcba599d0
kern/dl.c:649: module name: mmap
kern/dl.c:650: init function: 0xcba5aa24
kern/dl.c:556: flushing 0x37ce bytes at 0xcba558d0
kern/dl.c:649: module name: relocator
kern/dl.c:650: init function: 0x0
kern/disk.c:196: Opening `hd0,msdos1'...
partmap/msdos.c:188: partition 0: flag 0x80, type 0xef, start 0x800, len
0x1cd125a
kern/fs.c:56: Detecting fat...
fs/fat.c:553: fat_size=32, next_cluster=160
fs/fat.c:553: fat_size=32, next_cluster=334
kern/disk.c:295: Closing `hd0'.
kern/dl.c:602: module at 0xcba7a0c0, size 0x8e8
kern/dl.c:626: relocating to 0xcba7aba0
kern/dl.c:556: flushing 0x6fe bytes at 0xcba799b0
kern/dl.c:649: module name: lsapm
kern/dl.c:650: init function: 0xcba79b84
kern/disk.c:196: Opening `hd0,msdos1'...
partmap/msdos.c:188: partition 0: flag 0x80, type 0xef, start 0x800, len
0x1cd125a
kern/fs.c:56: Detecting fat...
fs/fat.c:553: fat_size=32, next_cluster=160
fs/fat.c:553: fat_size=32, next_cluster=334
fs/fat.c:553: fat_size=32, next_cluster=375
kern/disk.c:295: Closing `hd0'.
kern/dl.c:602: module at 0xcba771e0, size 0x2674
kern/dl.c:626: relocating to 0xcba7aa30
kern/disk.c:196: Opening `hd0,msdos1'...
partmap/msdos.c:188: partition 0: flag 0x80, type 0xef, start 0x800, len
0x1cd125a
kern/fs.c:56: Detecting fat...
fs/fat.c:553: fat_size=32, next_cluster=155
fs/fat.c:553: fat_size=32, next_cluster=156
kern/disk.c:295: Closing `hd0'.
kern/dl.c:602: module at 0xd9d38060, size 0x50e0
kern/dl.c:626: relocating to 0xcba7a8a0
kern/dl.c:556: flushing 0x4eec bytes at 0xd9ca7060
kern/dl.c:649: module name: video_fb
kern/dl.c:650: init function: 0x0
kern/dl.c:556: flushing 0x275e bytes at 0xcba744c0
kern/dl.c:649: module name: vbe
kern/dl.c:650: init function: 0xcba75603
kern/dl.c:556: flushing 0x3514 bytes at 0xd9d39c20
kern/dl.c:649: module name: multiboot
kern/dl.c:650: init function: 0xd9d3b50e
kern/disk.c:196: Opening `hd0,msdos1'...
partmap/msdos.c:188: partition 0: flag 0x80, type 0xef, start 0x800, len
0x1cd125a
kern/fs.c:56: Detecting fat...
lib/relocator.c:115: relocators_size=7
fs/fat.c:553: fat_size=32, next_cluster=400
fs/fat.c:553: fat_size=32, next_cluster=401
fs/fat.c:553: fat_size=32, next_cluster=402
lib/relocator.c:1251: min_addr = 0x0, max_addr = 0xffffffff, target = 0x400000
lib/relocator.c:434: trying to allocate in 0x400000-0xffffffff aligned 0x1 size
0x40aae000
lib/relocator.c:1198: allocated: 0x400000+0x40aae000
lib/relocator.c:1287: allocated 0x400000/0x400000
lib/relocator.c:1305: relocators_size=7
lib/relocator.c:1313: relocators_size=7
lib/relocator.c:1320: cur = 0xcba88860, next = 0x0
loader/multiboot.c:127: link_base_addr=0x400000, load_base_addr=0x400000,
load_size=0x40aae000, relocatable=0
loader/multiboot.c:140: segment 0: paddr=0x400000, memsz=0x3d1e5,
vaddr=0x400000
fs/fat.c:553: fat_size=32, next_cluster=400
fs/fat.c:553: fat_size=32, next_cluster=401
fs/fat.c:553: fat_size=32, next_cluster=402
fs/fat.c:553: fat_size=32, next_cluster=403
fs/fat.c:553: fat_size=32, next_cluster=404
fs/fat.c:553: fat_size=32, next_cluster=405
fs/fat.c:553: fat_size=32, next_cluster=406
fs/fat.c:553: fat_size=32, next_cluster=407
fs/fat.c:553: fat_size=32, next_cluster=408
fs/fat.c:553: fat_size=32, next_cluster=409
fs/fat.c:553: fat_size=32, next_cluster=410
fs/fat.c:553: fat_size=32, next_cluster=411
fs/fat.c:553: fat_size=32, next_cluster=412
fs/fat.c:553: fat_size=32, next_cluster=413
fs/fat.c:553: fat_size=32, next_cluster=414
fs/fat.c:553: fat_size=32, next_cluster=415
fs/fat.c:553: fat_size=32, next_cluster=416
fs/fat.c:553: fat_size=32, next_cluster=417
fs/fat.c:553: fat_size=32, next_cluster=418
fs/fat.c:553: fat_size=32, next_cluster=419
fs/fat.c:553: fat_size=32, next_cluster=420
fs/fat.c:553: fat_size=32, next_cluster=421
fs/fat.c:553: fat_size=32, next_cluster=422
fs/fat.c:553: fat_size=32, next_cluster=423
fs/fat.c:553: fat_size=32, next_cluster=424
fs/fat.c:553: fat_size=32, next_cluster=425
fs/fat.c:553: fat_size=32, next_cluster=426
fs/fat.c:553: fat_size=32, next_cluster=427
fs/fat.c:553: fat_size=32, next_cluster=428
fs/fat.c:553: fat_size=32, next_cluster=429
fs/fat.c:553: fat_size=32, next_cluster=430
fs/fat.c:553: fat_size=32, next_cluster=431
fs/fat.c:553: fat_size=32, next_cluster=432
loader/multiboot.c:140: segment 1: paddr=0x43d1f0, memsz=0x3, vaddr=0x43d1f0
loader/multiboot.c:140: segment 2: paddr=0x600000, memsz=0x2bd4,
vaddr=0x600000
# fs/fat.c:553: fat_size=32, next_cluster=0 ... 1024 repeats for about 5000 more lines
kern/disk.c:196: Opening `hd0,msdos1'...
partmap/msdos.c:188: partition 0: flag 0x80, type 0xef, start 0x800, len
0x1cd125a
kern/disk.c:295: Closing `hd0'.
kern/disk.c:196: Opening `hd0,msdos1'...
partmap/msdos.c:188: partition 0: flag 0x80, type 0xef, start 0x800, len
0x1cd125a
kern/disk.c:295: Closing `hd0'.
kern/disk.c:295: Closing `hd0'.
script/script.c:65: free 0xd9d707d0
script/script.c:65: free 0xcba8c4a0
script/script.c:65: free 0xd9d707f0
script/script.c:65: free 0xd9d70500
script/script.c:65: free 0xcba900d0
script/script.c:65: free 0xd9d70530
script/script.c:65: free 0xd9d70560
script/script.c:65: free 0xd9d70590
script/script.c:65: free 0xd9d705c0
script/script.c:65: free 0xd9d706f0
script/script.c:65: free 0xd9d70720
script/script.c:65: free 0xd9d70750
script/lexer.c:321: token 288 text [boot]
script/script.c:50: malloc 0xcba81850
script/script.c:50: malloc 0xcba81830
script/script.c:163: arglist
script/script.c:50: malloc 0xcba81800
script/lexer.c:321: token 259 text [
]
script/script.c:50: malloc 0xcba817d0
script/script.c:50: malloc 0xcba817b0
script/script.c:198: cmdline
script/script.c:50: malloc 0xcba81780
script/lexer.c:321: token 0 text []
script/script.c:50: malloc 0xcba818f0
script/script.c:50: malloc 0xcba81760
script/script.c:294: append command
script/script.c:50: malloc 0xcba81740
lib/relocator.c:1397: chunks = 0xcba79190
lib/relocator.c:434: trying to allocate in 0x10000-0x9b5a4 aligned 0x4 size
0x4a5c
lib/relocator.c:1423: Adjusted limits from 10000-9b5a4 to 0-100000
lib/relocator.c:434: trying to allocate in 0x0-0x100000 aligned 0x4 size
0x4a5c
lib/relocator.c:434: trying to allocate in 0x40eae000-0xffffffff aligned 0x1
size 0x4a5c
lib/relocator.c:1198: allocated: 0xd9d647f4+0x4a5c
lib/relocator.c:1478: relocators_size=7
lib/relocator.c:1486: relocators_size=29
lib/relocator.c:1492: cur = 0xcba88800, next = 0xcba79190
lib/relocator.c:1397: chunks = 0xcba88800
lib/relocator.c:434: trying to allocate in 0x1000-0x99f30 aligned 0x10 size
0xd0
lib/relocator.c:1423: Adjusted limits from 1000-99f30 to 0-100000
lib/relocator.c:434: trying to allocate in 0x0-0x100000 aligned 0x10 size 0xd0
lib/relocator.c:434: trying to allocate in 0x40eae000-0xffffffff aligned 0x1
size 0xd0
lib/relocator.c:1198: allocated: 0xd9d73800+0xd0
lib/relocator.c:1478: relocators_size=29
lib/relocator.c:1486: relocators_size=51
lib/relocator.c:1492: cur = 0xcba81710, next = 0xcba88800
lib/relocator.c:1533: Preparing relocs (size=51)
lib/relocator.c:434: trying to allocate in 0x0-0xffffffcd aligned 0x1 size
0x33
lib/relocator.c:1198: allocated: 0x161056+0x33
lib/relocator.c:1545: Relocs allocated at 0x161056 # This is location it jumps to
lib/relocator.c:1560: chunk 0xd9d73800->0x1000, 0xd0
lib/relocator.c:1560: chunk 0xd9d647f4->0x10000, 0x4a5c
lib/relocator.c:1560: chunk 0x15e57f->0x15e57f, 0x2ad7
lib/relocator.c:1560: chunk 0x15e3dc->0x15e3dc, 0x1a3
lib/relocator.c:1560: chunk 0x10005c->0x10005c, 0x5e380
lib/relocator.c:1560: chunk 0x100028->0x100028, 0x33
lib/relocator.c:1560: chunk 0x100000->0x100000, 0x28
lib/relocator.c:1560: chunk 0x400000->0x400000, 0x40aae000
lib/relocator.c:1604: sorted chunk 0x100000->0x100000, 0x28
lib/relocator.c:1604: sorted chunk 0x100028->0x100028, 0x33
lib/relocator.c:1604: sorted chunk 0x10005c->0x10005c, 0x5e380
lib/relocator.c:1604: sorted chunk 0x15e3dc->0x15e3dc, 0x1a3
lib/relocator.c:1604: sorted chunk 0x15e57f->0x15e57f, 0x2ad7
lib/relocator.c:1604: sorted chunk 0x400000->0x400000, 0x40aae000
lib/relocator.c:1604: sorted chunk 0xd9d647f4->0x10000, 0x4a5c
lib/relocator.c:1604: sorted chunk 0xd9d73800->0x1000, 0xd0

The last couple of lines GRUB2 makes it to is in the file grub/grub-core/lib/i386/relocator.c and the function grub_relocator32_boot:

// Line 104 - 113
grub_memmove (get_virtual_current_address (ch), &grub_relocator32_start,
    RELOCATOR_SIZEOF (32));

err = grub_relocator_prepare_relocs (rel, get_physical_target_address (ch),
    &relst, NULL);

if (err)
    return err;

asm volatile ("cli");
((void (*) (void)) relst) (); // This last line run, its job is to jump into the kernel
// The value of relst is 0x161056

After this the kernel never starts running and the GRUB2's cursor blinks forever and is unresponsive (typing doesn't work). The kernel's multiboot header only specifies flag bit 1, which means it expects some memory information when booting. Does anyone have a clue as to why this is happening? I checked the memory locations against Intel's known bad memory regions but that did not result in anything. I also tried the --quirk options for the multiboot command but neither helped.