Provisioning a Bare Metal Server in 47 Seconds: Our Boot Pipeline

The 47-second pipeline

When a customer requests a bare metal server through our API, 47 seconds later they receive SSH access to a fully configured machine running their chosen operating system. That number isn't a theoretical best case — it's our fleet-wide p50 for the last 30 days. The p99 is 78 seconds, with the tail driven primarily by BIOS POST time variation across hardware generations.

Phase 1: Inventory selection (0–2 seconds)

Our inventory system maintains a real-time pool of pre-powered, pre-health-checked servers in each region. When a provisioning request arrives, the system selects a server matching the requested configuration (CPU, RAM, disk, NIC) using a bin-packing algorithm that optimizes for minimal rack-level concentration — we avoid putting too many new customers in the same rack to limit blast radius from rack-level failures.

The selected server's BMC (Baseboard Management Controller) is already connected to our out-of-band management network. We issue an IPMI command to set the boot order to PXE and trigger a reboot. This takes approximately 0.3 seconds from API call to reboot command.

Phase 2: PXE boot and imaging (2–35 seconds)

The server's BIOS POST takes 8-20 seconds depending on hardware generation (newer AMD EPYC systems POST faster than older Intel Xeon systems). Once POST completes, the NIC's PXE ROM contacts our DHCP server, receives an IP address and the location of our iPXE chainloader.

#!ipxe
# Sigilhosting bare metal provisioning chain
set server-id ${mac}
set api-url https://provision.internal/v1

# Fetch provisioning config
chain ${api-url}/boot/${server-id}?token=${secret}

The iPXE chainloader pulls the provisioning configuration from our API, which includes the target operating system, disk layout, network configuration, and customer SSH keys. It then loads a minimal Linux kernel and initramfs (approximately 45MB combined) into memory via HTTP — this transfer takes 0.8 seconds over our 25Gbps internal provisioning network.

The provisioning kernel writes the OS image to disk using a streaming approach — we don't download the entire image first, then write. The image is streamed from our regional image cache and written directly to the NVMe drives at approximately 3.2 GB/s. A typical Ubuntu 22.04 base image is 2.8GB, so the write completes in under a second.

Phase 3: Configuration injection (35–42 seconds)

After the OS image is written, the provisioning kernel mounts the root filesystem and injects customer-specific configuration: SSH authorized keys, network configuration (IP addresses, default gateway, DNS resolvers), hostname, cloud-init userdata if provided, and our monitoring agent.

[Match]
Name=eth0

[Network]
Address={{ .PublicIPv4 }}/{{ .Netmask }}
Gateway={{ .Gateway }}
DNS=198.51.100.53
DNS=198.51.100.54

[Address]
Address={{ .PublicIPv6 }}/64

We use systemd-networkd with static IP assignments — no DHCP for customer-facing interfaces, so network configuration survives reboots without external dependencies.

Phase 4: First boot (42–47 seconds)

The provisioning kernel triggers a kexec into the installed operating system — this avoids a full BIOS reboot cycle, saving 15-20 seconds. The OS boots, systemd starts services, and our monitoring agent phones home to confirm successful provisioning. The API marks the server as "active" and returns connection details.

Pre-warming: the hidden optimization

The 47-second number is only possible because of background work. Our inventory system maintains a pool of pre-powered servers using demand forecasting — an exponential smoothing model trained on the last 90 days of provisioning requests per region and configuration type. When the pool drops below target (typically 15-20% of capacity), new servers are automatically powered on, health-checked, and added. This pre-warming takes 8-12 minutes per server but is invisible to customers.

Failure handling

Approximately 2.3% of attempts fail due to hardware issues discovered during imaging. The failed server is automatically quarantined, and the request retries on a different server from the ready pool. Customers see slightly longer provisioning (90-120 seconds) but don't need to take action.