Our network carries 4.2 Tbps of customer traffic through four transit providers and seven IXPs. Here's how we designed it to survive any single failure — and most double failures.
Our network design follows three principles that we refuse to compromise on. First, no single point of failure at any layer — not in hardware, not in software, not in vendor relationships. Second, every failure mode must be automatically recoverable without human intervention within 60 seconds. Third, the control plane must survive independently of the data plane, so we can always reach and manage our infrastructure even during catastrophic events.
These principles sound obvious when stated plainly, but they have expensive consequences. They mean we maintain at least two transit providers in every region, even regions where traffic volume doesn't justify the cost. They mean we run our own route servers rather than depending on IXP route servers alone. They mean we have out-of-band management networks on entirely separate physical infrastructure from customer traffic.
Sigilhosting operates AS398330. We announce approximately 2,400 IPv4 prefixes and 800 IPv6 prefixes across our 28 regions. Each region has a minimum of two border routers, each connected to at least two different transit providers.
Our transit providers are NTT (AS2914), Cogent (AS174), Lumen (AS3356), and Telia (AS1299). We chose these four specifically because their backbone networks have minimal physical path overlap — a fiber cut affecting NTT's path between two cities is unlikely to simultaneously affect Cogent's path between the same cities.
router bgp 398330
bgp router-id 10.255.0.1
bgp bestpath as-path multipath-relax
maximum-paths 16
neighbor TRANSIT_NTT peer-group
neighbor TRANSIT_NTT remote-as 2914
neighbor TRANSIT_NTT route-map TRANSIT-IN in
neighbor TRANSIT_NTT route-map TRANSIT-OUT out
neighbor TRANSIT_NTT maximum-prefix 900000
neighbor PEERING_IXP peer-group
neighbor PEERING_IXP route-map PEER-IN in
neighbor PEERING_IXP route-map PEER-OUT outWe use BGP communities extensively to control traffic engineering. Every prefix is tagged with communities indicating region of origin, traffic type (anycast DNS vs unicast customer IP), and intended traffic policy. This allows us to drain all traffic from a specific region for maintenance by simply modifying the communities we announce from that region's border routers.
We maintain peering sessions at seven Internet Exchange Points: AMS-IX (Amsterdam), DE-CIX (Frankfurt), LINX (London), Equinix IX (Ashburn), NYIIX (New York), Equinix IX (Singapore), and JPNAP (Tokyo). Peering traffic currently accounts for approximately 45% of our total traffic volume.
| IXP | Location | Port Speed | Peak Traffic | Peers |
|---|---|---|---|---|
| AMS-IX | Amsterdam | 100 GbE | 42 Gbps | 847 |
| DE-CIX | Frankfurt | 100 GbE | 38 Gbps | 1,241 |
| LINX | London | 100 GbE | 31 Gbps | 592 |
| Equinix IX | Ashburn | 100 GbE | 56 Gbps | 438 |
| NYIIX | New York | 10 GbE | 8 Gbps | 214 |
| Equinix IX | Singapore | 100 GbE | 22 Gbps | 316 |
| JPNAP | Tokyo | 100 GbE | 19 Gbps | 271 |
Peering is important because it reduces both latency and cost. Traffic exchanged at an IXP travels a shorter path (fewer network hops) and doesn't incur transit fees. Our blended transit cost is approximately $0.40/Mbps committed, while peering is a flat port fee regardless of volume.
Every quarter, we run "chaos days" — deliberately failing transit links, border routers, and IXP connections. In our most recent test, we simultaneously failed both transit providers in Frankfurt plus our DE-CIX session. Traffic automatically rerouted: 60% through Amsterdam, 40% through London. Total reconvergence: 8 seconds for transit, 3 seconds for anycast DNS.
The best time to discover a failover bug is during a planned test, not during an actual outage at 3 AM.
We monitor BGP convergence time continuously. Our internal tooling measures the time between a route withdrawal and the last border router updating its forwarding table. Current 30-day average: 4.2 seconds, well within our 60-second target.
We're evaluating Segment Routing v6 (SRv6) to replace MPLS-based traffic engineering within our backbone. SRv6 would give us more granular traffic steering and eliminate the operational complexity of maintaining a separate label distribution protocol. Phased rollout expected Q3 2026, starting with Americas regions.
We're also expanding peering to IX.br (São Paulo) and HKIX (Hong Kong), both expected live by end of Q2 2026.