CoreOS: Boot on Bare Metal with PXE
In this technical retrospective (2013), @philips shares practical architectural lessons learned from running, scaling, or migrating production software under lean constraints.
Core Principles for Independent Builders:
Defaulting to simpler architectures (such as monolithic frameworks, SQLite, or managed VPS) prevents early engineering bankruptcy.
Evaluating true compute and egress costs reveals that high cloud bills can often be slashed by 80%+ through judicious re-platforming.
Code you write at 2 AM as a solo engineer must be easy to debug six months later without complex microservice dependencies.
Community Debate Themes:
HN engineers debated the tradeoffs between cloud convenience and raw server economics, database indexing strategies, and resisting premature architectural complexity.
CATEGORY Frugal Engineering →
HISTORICAL ERA Web 2.0 Genesis (2007–2013)
COMMUNITY SCORE ★ 145 points
TOTAL DISCUSSIONS 55 comments
For anyone who does not know what PXE (pixie) boot is, it is an environment to boot computers using a network interface independently of data storage devices (like hard disks) or installed operating systems. [1] In the BIOS, rather than booting from CD, or Hard disk, you would select Network.
A very simplified explanation is that the PXE enabled network card (almost all modern network cards support PXE, desktop and servers), can make a DHCP request (outside of an operating system), download via TFTP, then boot a kernel/initial ram disk, say for example an OS installer, a LIVE CD, or CoreOS.
``` +-----> #1 PXE makes DHCP request, | redirects to TFTP server, | loads kernel/initial ram disk via TFTP | | +--+ #2 PXE boots kernel/initial ram disk | | + v +------------------+ | PXE Network Card | |------------------| | server | | hardware/os | +------------------+
``` So, with a little infrastructure (DHCP, TFTP, and elbow grease), you can boot many things over the network without even having a hard drive or cd-rom in a machine. I use this often for installing new desktops and servers. Just boot into the PXE menu and then select the installer that I want. Then you use something like puppet to configure the machine as needed. A typical RHEL install can take ~5 minutes.
In summary, it looks like CoreOS provides you with their kernel and initial ram disk [2], then you can just boot the machine over the network without actually installing anything. Basically running everything out of RAM, just like a LIVE CD, most likely with the option to install to disk for persistent storage of your images, etc.
ps. It is common to use NFS to mount persistent storage in these environments too, or if you are using a HPC environment, then Luster or something like that.
[1] http://en.wikipedia.org/wiki/Preboot_Execution_Environment
[2] http://coreos.com/docs/pxe/
Good luck finding a network infrastructure and PXE server able to boot a few hundred machines simultaneously during a power event. Yes in theory PXE boot sounds great. In reality it's a pointless SPOF.
Also most likely makes it harder to reuse most of the trusted boot infrastructure that already exists for Linux. So we can assume at least in the initial release Mallory can race with the real PXE server assuming a network that hasn't been partitioned with a crazy complex config (i.e. basically all of them).
Since I asked for this in the original CoreOS thread, let me be the first to say thanks. I think stateless immutable servers that boot from the network and run from RAM are going to be a great base layer to build on.
BTW, if people haven't tried PXE booting before, it's pretty easy with dnsmasq. You can basically read the sample config file and uncomment a few lines. I recommend experimenting with PXE in Vagrant or on a separate physical network to avoid breaking your production DHCP.
PXE support on some consumer boards is a mess - often I have to use iPXE [1] just to get them loading from TFTP reliably. Now I've played with UEFI PXE boot, and it seems to be even worse - instead of requesting an "x64 bootloader", the NIC seems to request a "UEFI bytecode bootloader" which I haven't been able to supply.
[1]: http://ipxe.org/
Awesome! My favorite part is this: http://coreos.com/docs/pxe/#state-only-installation
which would enable you to run CoreOS in memory only (loaded from PXE), but still store all of your containers on filesystem. It liberates me from having to install OS on a cluster, but still lets me use persistent storage.