This is the second article about adding x86_64 support to coreboot's recent Intel x86 platforms. You can find the first article here.

A new approach

As described in the previous article, mixing x86_32 and x86_64 doesn't work well, and having page tables required for x86_64 in heap is problematic as well.

Tests in QEMU showed that it is possible to place page tables in ROM. As the ROM is memory-mapped on x86 platforms, it is available right after power-on reset at no further cost. The MMU is happy to use it as long as no bits, like the "dirty" or "accessed" bit, needs to be written.

The page tables are generated by a small utility that's called pgtblgen.
After adding some assembly code to set up long mode using the already present page tables and coreboot can run x86_64 in all stages!

Fixing the odds of x86_64

Now the CPU is running in x86_64 mode right from the beginning of the bootblock, the remaining bugs need to be fixed. It turns out that there are quite a few bummers.

Fixing the PCI and CPU drivers:
coreboot uses a linker script to add arbitrary PCI and CPU drivers to a NULL-terminated array of structs at compile-time. The array is then consumed by the ramstage core driver to execute the drivers one after another at runtime. On x86_64 that was suddenly not working anymore.

It turns out the linker tries to be smart and aligns structs inside the array on a 16 byte boundary instead of the expected byte boundary. This can be disabled using the compiler argument -malign-data=abi as done in CB:30116, which makes the compiler behave as in the corresponding SysV ABI specification.

Fixing GDT loading:
coreboot loads the GDT multiple times, to support a basic mode in early stages and complex memory setups in ramstage. The GDT needs to be reloaded from withing x86_64, but that turns out the be error-prone.

The assembler instruction lgdt gdtptr emits the opcode 0f 01 14 xx xx xx xx, which is "lgdt 32bit displacement only addressing mode", where the displacement is a signed integer from current PC.

This works fine in x86_32 as the PC always overflows to the correct address, but it doesn't in x86_64 with 64bit registers. QEMU doesn't care and thus it wasn't noticed when implementing x86_64 support

The fix is simple: Force the assembler to emit 64bit addressing mode when loading a gdt by using the movabs instruction as done in CB:43136.

The instruction movabs gdtptr,%rax lgdt (%rax) now generates the correct opcode that uses 64bit addressing mode.

Unexplainable CPU behavior:
Tests on real hardware showed that 50% of the devices boot, the others showed unexplainable behavior. For some reason entering x86_64 mode causes a bug in the CPU silicon/microcode and the following issues can be observed:

  • After entering long-mode, the floating-point unit doesn't work anymore, including accessing MMX registers. It works fine before entering long mode.
  • Reading from virtual memory where the lower twelve address bits are zero returns a fixed constant. Writing to that memory location does not affect. Depending on the variables stored in heap and stack and where the code resides, undefined behavior happens whenever the lower twelve bits are zero.
  • Disabling paging in compatibility mode crashes the CPU.
  • Returning from long mode to compatibility mode crashes the CPU.

This issue is still open and blocks further commercial use of x86_64 in coreboot.

Conclusion

In BIOS development you have to deal with all kind of strange hardware bugs, as most parts of the computer haven't been initialized.

We will keep working on that topic and keep you updated.
Let us know if you liked the article.