A short journey to x86 long mode in coreboot on recent Intel platforms

While it was difficult to add initial x86_64 support in coreboot, as described in my last blog article how-to-not-add-x86_64-support-to-coreboot it was way easier on real hardware. During the OSFC we did a small hackathon at 9elements and got x86_64 working in coreboot on recent Intel platforms.

If you want to test new code that deals with low level stuff like enabling x86_64 mode in assembly, it's always good to test it on qemu using KVM. It runs the code in ring 0 instead of emulating every single instruction and thus is very close to bare metal machines.

$ qemu-system-x86_64 -M q35 -accel kvm -bios build/coreboot.rom

But, running coreboot's x86_64 code on KVM gave more magic errors than you could find in books about some famous magician with a scarf on his forehead.

To summarize:

• On recent AMD platforms it stops after entering x86_64 long mode.
• On older Intel platforms everything works.
• On recent Intel platforms after entering long mode every instruction causes a fault, and thus the instruction is emulated by the kernel, which doesn't handle FPU instruction that well...
• On recent Intel platforms the MMU aborts walking page tables and returns the data within the page table itself when looking up some virtual addresses...

Tests on real hardware, in this case the X11SSH-TF showed none of the problems above. After fixing integer-void pointer conversion (CB:48166) , (CB:48167) , (CB:48177) and adding assembly code to enable long mode in Cache-As-RAM (CB:48170) it booted straight to romstage.

coreboot-4.13-241-g52ab788549-dirty Tue Dec 1 18:23:08 UTC 2020 bootblock starting (log level: 7)...
CPU: Intel(R) Xeon(R) CPU E3-1240 v6 @ 3.70GHz
CPU: ID 906e9, Kabylake H B0, ucode: 000000d5
CPU: AES supported, TXT supported, VT supported
MCH: device id 5918 (rev 05) is Kabylake DT 2
PCH: device id a149 (rev 31) is Skylake PCH-H C236
IGD: device id ffff (rev ff) is Unknown
FMAP: Found "FLASH" version 1.1 at 0xb10000.
FMAP: base = 0xff000000 size = 0x1000000 #areas = 4
FMAP: area COREBOOT found @ b10200 (5176832 bytes)
CBFS: Found 'fallback/romstage' @0x80 size 0xe334
BS: bootblock times (exec / console): total (unknown) / 53 ms

coreboot-4.13-241-g52ab788549-dirty Tue Dec 1 18:23:08 UTC 2020 romstage starting (log level: 7)...
pm1_sts: 0900 pm1_en: 4000 pm1_cnt: 00000000
gpe0_sts[0]: 00000000 gpe0_en[0]: 00000000
gpe0_sts[1]: 00000000 gpe0_en[1]: 00000000
gpe0_sts[2]: 00000000 gpe0_en[2]: 00000000
gpe0_sts[3]: 00000000 gpe0_en[3]: 00000000
TCO_STS: 0000 0000
GEN_PMCON: e0810200 000018c8
GBLRST_CAUSE: 00000002 00000000
prev_sleep_state 0
FMAP: area COREBOOT found @ b10200 (5176832 bytes)
CBFS: Found 'fspm.bin' @0x5fdc0 size 0x63000
POST: 0x34
FMAP: area RW_MRC_CACHE found @ b00000 (65536 bytes)
MRC: no data in 'RW_MRC_CACHE'
No memory dimm at address A2
No memory dimm at address A4
POST: 0x36
POST: 0x92

ghost

It hung at entering FSP-M, which as it's a binary blob, wasn't automatically recompiled to x86_64. A wrapper (CB:48175) , written in assembly, fixed the problem by falling back to x86_32 when calling into FSP.
The wrapper will automatically switch back into x86_64 mode when the function returns. This is slow, but as we don't have proper blobs there's no other way around it.

coreboot-4.13-242-g04129be978-dirty Tue Dec 1 18:42:20 UTC 2020 romstage starting (log level: 7)...
pm1_sts: 0900 pm1_en: 0000 pm1_cnt: 00000000
gpe0_sts[0]: 00000000 gpe0_en[0]: 00000000
gpe0_sts[1]: 00000000 gpe0_en[1]: 00000000
gpe0_sts[2]: 00000000 gpe0_en[2]: 00000000
gpe0_sts[3]: 00000000 gpe0_en[3]: 00000000
TCO_STS: 0000 0000
GEN_PMCON: e0810200 000018c8
GBLRST_CAUSE: 00000002 00000000
prev_sleep_state 0
FMAP: area COREBOOT found @ b10200 (5176832 bytes)
CBFS: Found 'fspm.bin' @0x5fdc0 size 0x63000
POST: 0x34
FMAP: area RW_MRC_CACHE found @ b00000 (65536 bytes)
MRC: no data in 'RW_MRC_CACHE'
No memory dimm at address A2
No memory dimm at address A4
POST: 0x36
POST: 0x92
POST: 0x98
FspMemoryInit returned 0x80000002
POST: 0xe3
FspMemoryInit returned an error!

The FSP was now able to run, but it returned an error Invalid parameter, which was due to the fact that FSP's config structures contained void pointers, which on x86_64 have a different size and doesn't match what FSP expects.

Fixing those headers is an ongoing tasks, but was hacked around.

IOAPIC: Initializing IOAPIC at 0xfec00000
IOAPIC: Bootstrap Processor Local APIC = 0x00
IOAPIC: ID = 0x02
PCI: 00:1f.0 init finished in 9 msecs
POST: 0x75
POST: 0x75
PCI: 00:1f.2 init
RTC Init
Set power on after power failure.
Disabling ACPI via APMC.

coreboot-4.13-241-g52ab788549-dirty Tue Dec 1 18:23:08 UTC 2020 smm starting (log level: 7)...
SMI_STS: PM1 APM
SMI#: ACPI disabled.
canary 0xcdcdcdcd7f9ff800 != 0x7f9ff800
SMM Handler caused a stack overflow

Finally it booted into SMM, but crashed due to stack trashing. That turned out to be a false positive, as the stack canary is the size of a void pointer and is written in x86_32 assembly, but checked in x86_64 C code and thus failed. Writing 4 additional bytes in assembly code fixed the stack canary check and it finally booted.(CB:48215)

/* Write canary to the bottom of the stack */
   movl   stack_size, %eax
   subl   %eax, %ebx /* %ebx(stack_top) - size = %ebx(stack_bottom) */
   movl   %ebx, (%ebx)
+ #if ENV_X86_64
+   movl   $0, 4(%ebx)
+ #endif

Summarizing it took about a day to add x86_64 support and half of the code needed to be written in assembly code. With those patches in place it should be easier to port additional platforms to x86_64, reducing the time to a few hours.
I invite everyone to play with the changes, hack the code and improve it to make this open source project even more awesome.