(new Soapbox())->shout(array_map('strtoupper', $opinions)); //Shaun's blog

Me, elsewhere

Miscellaneous public code

A PHP API client for Reddit

I don't tweet much

XMPP chat
(Pidgin, Miranda, Swift, etc.)

Perfect is the enemy of good enough. Fast, but not so accurate (yet)

Posted April 05, 2018 by shaun

The launch of Cloudflare's new DNS service has brought much fanfare and a heap of praise. More choice is always a good thing, and support for both DNS-over-TLS and DNS-over-HTTPS make an interesting option. In this post I'm going to explain why I won't use it yet, and why you might not want to switch either without doing some testing first.

Fastest isn't always best

Cloudflare touts as "the Internet's fastest DNS directory." In terms of pure response time, they might be right. But when dealing with DNS, response time is only part of the equation. Just as important are the replies you get from a DNS server, the IP addresses it hands out in response to domain queries. Responses from centralized DNS services are sometimes unsatisfactory and can lead to service degradation. The answers come fast, but are they the right ones?

I like to use a telephone analogy here. Imagine you dial 9-1-1 and your call is answered so quickly you never even hear it ring. But it turns out you've been connected to a dispatcher in a city hundreds of miles away, and while she can send the authorities, it's going to take them awhile to reach you. The response was fast, yes, but it wasn't optimal. You would probably have preferred to speak to your local 9-1-1 dispatcher instead, even if you had to wait 3 or 4 rings for them to pick up the phone.

So, too, is the case with DNS. Public name servers may answer faster than your ISP's DNS, but that speed is negated if the servers they point you to are very distant.

Testing public resolvers for speed

Cloudflare makes bold speed claims, so I've done some basic testing to gauge how stacks up against other options. Measurements were taken of the following DNS services:

  • - Cloudflare
  • - Level3
  • - Google
  • - Comcast (customer-only)

Because is promoted as a consumer DNS service, all observations below are from my residential Comcast cable connection in Memphis, TN.


Cloudflare's service appears to have robust connectivity and a diverse geographic presence. is physically closer to me than other common public DNS options. The server I'm routed to,, is 7 network hops out, with an average ping of only 13.1 ms. It responds to an RFC4892 query with mem01, so is presumably located here in Memphis. That's about even with Comcast's, which also has an anycast presence in town, and it beats the other public resolvers. See the traceroutes for details.


For the first round of DNS testing, I queried each server for 100 invalid domains in the .com zone to gauge NXDOMAIN response time. ( hijacks NXDOMAIN and returns two advertising servers instead.)

  [parse@word ~]$ for i in {1..100}; do dig A $(head /dev/urandom | sha1).com \ 
      @; sleep 5; done > &

Cold queries

Next, I queried 100 randomly generated hostnames on a domain configured for wildcard DNS. This test was intended to measure "cold" or uncached query response time.

  [parse@word ~]$ for i in {1..100}; do dig A $(head /dev/urandom | sha1).shaun.ga \ 
      @; sleep 5; done > &

Likely cached queries

Finally, I queried the top 100 domains on the Cisco Umbrella list. DNS servers are likely to have answers for these popular domains cached, improving response time.

  [parse@word ~]$ head -100 top-1m.csv | cut -f2 -d"," | dos2unix > top-100.txt
  [parse@word ~]$ while read host; do dig A $host @; sleep 5; \
      done < top-100.txt > &

The same series of tests was performed against all 5 DNS services. The output from each test was processed to calculate the mean average query times in milliseconds, like this:

  [parse@word ~]$ grep 'Query time' | cut -f4 -d" " | \
      awk '{ sum += $1 } END { if (NR > 0) print sum / NR }'


DNS server  | Avg NXDOMAIN | Avg cold | Avg top
------------|--------------|----------|---------     | 47.03 ms     | 41.99 ms | 28.34 ms     | 35.57 ms     | 34.03 ms | 31.08 ms     | 48.68 ms     | 91.66 ms | 36.64 ms     | 68.99 ms     | 57.72 ms | 50.14 ms | 28.78 ms     | 49.50 ms | 23.48 ms performed reasonably well speed-wise, coming in second or third in these tests.

The mean average trends deceptively high for the "Avg top" (Cisco Umbrella top 100) column. 65 of the 100 responses from were faster than 20 ms, but the 5 outlier responses over 100 ms pushed the mean up. Time permitting, I might revisit these results and create scatter plots that better represent these figures.

All of the raw dig output is available.

Testing public resolvers for response quality

As mentioned previously, the quality of DNS responses is just as important as the speed at which you receive them. Returning to the 9-1-1 analogy, when you make that phone call, you want to be connected to your local emergency dispatcher. Likewise, when you visit a website that uses a geographically sensitive content distribution network, the ideal scenario gets you pointed to a server relatively close by.

In this test, I asked each DNS server to resolve several popular distributed web services. These hostnames return a different IP address depending on who you're asking and where you're asking from. I avoided records configured to return multiple IPs in a "round-robin" fashion, like reddit.com (view) and twitter.com (view), which always resolve to the same RRset regardless of where in North America a query originates.

Each DNS server was queried only one time, mimicking a user typing a domain into their web browser, and the quality of that response was judged.

Amazon provided a mediocre answer. It resolved www.amazon.com to a server in northern Virginia. had the worst response, pointing me to Montreal, while the other services gave IPs in Dallas, which is closer to me. View test data.

DNS server   | Answer RR        | City             | Avg Latency
-------------|------------------|------------------|------------      |     | Dulles, VA       | 46.8 ms      |   | Dallas, TX       | 21.5 ms      |     | Dallas, TX       | 21.5 ms      |     | Montreal, QC     | 60.8 ms  |    | Dallas, TX       | 22.7 ms

Facebook tied for best answer here. It resolved www.facebook.com to a server in Dallas; the same IP was given out by every server except Quad9. View test data.

DNS server   | Answer RR        | City             | Avg Latency
-------------|------------------|------------------|------------      |    | Dallas, TX       | 21.8 ms      |    | Dallas, TX       | 21.8 ms      |    | Dallas, TX       | 21.8 ms      |      | Atlanta, GA      | 33.1 ms  |    | Dallas, TX       | 21.8 ms

Google produced the least optimal answer. It resolved google.com to a server in New York, the most distant IP given by any of the DNS services tested. View test data.

DNS server   | Answer RR        | City         | Avg Latency
-------------|------------------|--------------|------------      |   | New York, NY | 50.8 ms      |   | Dallas, TX   | 21.5 ms      |    | Dallas, TX   | 21.6 ms      |   | Atlanta, GA  | 31.8 ms  |    | Dallas, TX   | 21.5 ms

Slack provided a mediocre answer. It resolved slack.com to a server in Minneapolis, which is fairly distant. Quad9 pointed me to Montreal again, and Comcast had the best response. View test data.

DNS server   | Answer RR        | City             | Avg Latency
-------------|------------------|------------------|------------      |    | Minneapolis, MN  | 50.0 ms      |     | St. Louis, MO    | 36.0 ms      |    | Dulles, VA       | 45.2 ms      |     | Montreal, QC     | 58.2 ms  |     | Dallas, TX       | 22.2 ms

YouTube produced the least optimal answer. Surprisingly, it resolved youtube.com to a server in Bogota, Colombia, South America. The other DNS services pointed me to servers in Dallas and Atlanta. View test data.

DNS server   | Answer RR        | City             | Avg Latency
-------------|------------------|------------------|------------      |   | Bogota, Colombia | 86.6 ms      |     | Dallas, TX       | 21.4 ms      |    | Dallas, TX       | 21.5 ms      |    | Atlanta, GA      | 32.4 ms  |     | Dallas, TX       | 20.9 ms

Room for improvement

Cloudflare's DNS service is really fast, but from my perspective, it can provide low-quality responses (so does Quad9). Answers from were geographically incongruous, directing me to distant servers when much closer points of presence exist.

That said, I'm confident that will improve over time. DNS operators have a variety of tools like EDNS ECS and RPZ to enhance DNS resolution based on query origin. Upstream operators will gradually optimize for's traffic, too, aligning their own responses as they learn about the different nodes in Cloudflare's service. All of this takes time and study. Of the public DNS services I tested, the ones which have been operating longer gave better answers.

Potential privacy implications

Many users avoid their ISP's DNS service for a variety of reasons. Some ISPs hijack NXDOMAIN responses to insert advertising, which has a frustrating tendency to break non-web protocols entirely. Some ISPs collect your queries for marketing purposes or otherwise share them with third parties. Some smaller ISPs don't even operate their own name servers. These are all good cases for looking to an external service. Be aware that your ISP can still see your DNS queries to any server unless you're using DNS-over-TLS or DNS-over-HTTPS (both technologies are supported by but I haven't evaluated them).

For its part, Cloudflare explicitly states that they don't log your IP address when you use But the service is offered in cooperation with APNIC, whose wording isn't so clear. They promise to destroy "all ‘raw’ DNS data as soon as we have performed statistical analysis on the data flow," but don't seem to lay out a timeline for that process. In the meantime, raw data could be subject to legal demands by entities with jurisdiction over APNIC and its constituent countries.

If you're a Comcast customer, I suggest sticking with Comcast's own DNS servers ( and unless you have a privacy concern. Comcast has built out local anycast nodes in all of their markets, and their service consistently provides high quality answers. They're able to do this because they can see more pieces of the puzzle: they know your location, while a public DNS service may be guessing; and they know how and where they prefer to route traffic, while a public DNS service has no idea.

Who you use for DNS is your choice and can have a significant impact on your internet experience. It's good to have more options, and I look forward to watching evolve.

Recent articles

📰 Caveat with Vantec SATA/IDE to USB 2.0 Adapter and Macrium software

📰 Jay Niffley, Man of Mystery

📰 Compiling Doxygen on FreeBSD without LaTeX and Ghostscript

📰 Introducing Snuze, a PHP client for the Reddit API

📰 jisusaiche: Java's installer telemetry

📰 BIND client log error "query_find: query_getdb failed"

📰 Resolving "The lang/perl5.24 port has been deleted: Has expired" portmaster error

📰 Armagaddon2 interim fix for Firefox 56 and other old versions

📰 Strange DNS queries: qname "miep", qtype ANY

📰 Undeliverable as addressed: A massive broken spam campaign?

📰 Using WITH_META_MODE and ccache for FreeBSD build boosts

📰 Resolving subversion error E000013: Unable to create pristine install stream

📰 Enhancements to SmokePing's AnotherDNS probe

📰 Generating vanity DNSSEC key tags

📰 DDoS involving forged packets from

▲ Back to top | Permalink to this page