S2 moved away from AWS Route53 after discovering that the provider's API rate limits—approximately 5 UPSERTs per second—prevented the company from offering unlimited 'basins' to its clients. This bottleneck became critical when a malicious actor rapidly created tens of thousands of basins, saturating the quota and creating a multi-hour backlog that stalled all DNS updates.
The Scaling Bottleneck
S2's 'basins' are core namespace constructs that act as top-level containers for data streams. Each basin is represented as a unique subdomain, such as my-basin.b.s2.dev, which requires a CNAME record to route clients to the correct architecture cell.
While managed services like Route53 are generally robust, their control plane APIs are not designed for high-frequency, high-volume record creation. S2 found that even with batching, these limits made it impossible to guarantee that new basins would be instantly resolvable, a requirement for strong consistency.
“It became obvious when a nefarious actor rapidly created 10s of thousands of basins, saturating our entire quota of Route53 records.”
— It’s always DNS, so why are we running our own?
Custom DNS Architecture
The new system functions by treating the DNS server as a direct caller of S2's existing account database. Instead of syncing records to a separate system, the server performs a live SELECT query against Google Cloud Spanner to resolve the mapping for every incoming request.
This diagram illustrates the core architecture of the new DNS server which replaces the standard Route53 sync method.
The server was built in Go using the miekg/dns library to handle authoritative queries for A, AAAA, NS, CNAME, and SOA records. The team prioritized a library-based approach over a CoreDNS module to maintain maximum flexibility for their specific use case.
Security and Infrastructure Trade-offs
A significant risk of querying a database for every DNS request is the 'NXDOMAIN' attack, where a flood of queries for non-existent records could overwhelm the database. To defend against this, S2 deployed a Cloudflare DNS Firewall, configuring the origin servers to only accept traffic from Cloudflare's known IP addresses.
The production environment runs on two separate GKE clusters in different regions. The final setup involves a complex delegation chain: the main domain is managed by Cloudflare, which delegates the basin subdomain to a DNS Firewall service, which then connects to the GKE-hosted DNS server.
This visual explains the security layer added to protect the database from NXDOMAIN attacks.
Remaining Uncertainties
While the system is currently stable and supports thousands of basins, several optimizations remain on the internal roadmap. These include tuning DNS caching, optimizing the communication path between the DNS Firewall and the application, and exploring future DNSSEC extensions.

This chart demonstrates the scale of traffic the new system handles successfully.
What Hacker News readers are saying
- Open question: One commenter questioned why the company chose explicit DNS records over using wildcards with subdomains, suggesting that a wildcard CNAME might have avoided the risks associated with hitting a database on every query. comment ↗
- Alternative: Another user suggested that using PowerDNS with a CloudSQL backend might have been a viable alternative to protect the Spanner database from DNS spikes. comment ↗
- Skeptical: A commenter raised concerns about the difficulty of achieving strong consistency in DNS, noting that because DNS is designed to be eventually consistent, ensuring a new record is instantly resolvable can be complex. comment ↗
Summarized opinions from the Hacker News thread, not verified facts. Read the full discussion ↗