(new Soapbox())->shout(array_map('strtoupper', $opinions)); //Shaun's blog


Me, elsewhere

GitHub
parseword
Miscellaneous public code

Twitter
@parseword
I don't tweet much

XMPP chat
xmpp@shaunc.com
(Pidgin, Miranda, Swift, etc.)

1.1.1.1: Fast, but not so accurate (yet)

Posted April 05, 2018 by shaun

The launch of Cloudflare's new 1.1.1.1 DNS service has brought much fanfare and a heap of praise. More choice is always a good thing, and support for both DNS-over-TLS and DNS-over-HTTPS make 1.1.1.1 an interesting option. In this post I'm going to explain why I won't use it yet, and why you might not want to switch either without doing some testing first.

Fastest isn't always best

Cloudflare touts 1.1.1.1 as "the Internet's fastest DNS directory." In terms of pure response time, they might be right. But when dealing with DNS, response time is only part of the equation. Just as important are the replies you get from a DNS server, the IP addresses it hands out in response to domain queries. Responses from centralized DNS services are sometimes unsatisfactory and can lead to service degradation. The answers come fast, but are they the right ones?

I like to use a telephone analogy here. Imagine you dial 9-1-1 and your call is answered so quickly you never even hear it ring. But it turns out you've been connected to a dispatcher in a city hundreds of miles away, and while she can send the authorities, it's going to take them awhile to reach you. The response was fast, yes, but it wasn't optimal. You would probably have preferred to speak to your local 9-1-1 dispatcher instead, even if you had to wait 3 or 4 rings for them to pick up the phone.

So, too, is the case with DNS. Public name servers may answer faster than your ISP's DNS, but that speed is negated if the servers they point you to are very distant.

Testing public resolvers for speed

Cloudflare makes bold speed claims, so I've done some basic testing to gauge how 1.1.1.1 stacks up against other options. Measurements were taken of the following DNS services:

  • 1.1.1.1 - Cloudflare
  • 4.2.2.4 - Level3
  • 8.8.8.8 - Google
  • 9.9.9.9 - GCA/PCH/IBM
  • 75.75.75.75 - Comcast (customer-only)

Because 1.1.1.1 is promoted as a consumer DNS service, all observations below are from my residential Comcast cable connection in Memphis, TN.

Latency

Cloudflare's service appears to have robust connectivity and a diverse geographic presence. 1.1.1.1 is physically closer to me than other common public DNS options. The server I'm routed to, 172.69.197.23, is 7 network hops out, with an average ping of only 13.1 ms. It responds to an RFC4892 query with mem01, so is presumably located here in Memphis. That's about even with Comcast's 75.75.75.75, which also has an anycast presence in town, and it beats the other public resolvers. See the traceroutes for details.

NXDOMAIN

For the first round of DNS testing, I queried each server for 100 invalid domains in the .com zone to gauge NXDOMAIN response time. (4.2.2.4 hijacks NXDOMAIN and returns two advertising servers instead.)

  [parse@word ~]$ for i in {1..100}; do dig A $(head /dev/urandom | sha1).com \ 
      @1.1.1.1; sleep 5; done > 1.1.1.1.nxdomain.txt &

Cold queries

Next, I queried 100 randomly generated hostnames on a domain configured for wildcard DNS. This test was intended to measure "cold" or uncached query response time.

  [parse@word ~]$ for i in {1..100}; do dig A $(head /dev/urandom | sha1).shaun.ga \ 
      @1.1.1.1; sleep 5; done > 1.1.1.1.cold.txt &

Likely cached queries

Finally, I queried the top 100 domains on the Cisco Umbrella list. DNS servers are likely to have answers for these popular domains cached, improving response time.

  [parse@word ~]$ head -100 top-1m.csv | cut -f2 -d"," | dos2unix > top-100.txt
  [parse@word ~]$ while read host; do dig A $host @1.1.1.1; sleep 5; \
      done < top-100.txt > 1.1.1.1.top.txt &

The same series of tests was performed against all 5 DNS services. The output from each test was processed to calculate the mean average query times in milliseconds, like this:

  [parse@word ~]$ grep 'Query time' 1.1.1.1.nxdomain.txt | cut -f4 -d" " | \
      awk '{ sum += $1 } END { if (NR > 0) print sum / NR }'

Results

DNS server  | Avg NXDOMAIN | Avg cold | Avg top
------------|--------------|----------|---------
1.1.1.1     | 47.03 ms     | 41.99 ms | 28.34 ms
4.2.2.4     | 35.57 ms     | 34.03 ms | 31.08 ms
8.8.8.8     | 48.68 ms     | 91.66 ms | 36.64 ms
9.9.9.9     | 68.99 ms     | 57.72 ms | 50.14 ms
75.75.75.75 | 28.78 ms     | 49.50 ms | 23.48 ms

1.1.1.1 performed reasonably well speed-wise, coming in second or third in these tests.

The mean average trends deceptively high for the "Avg top" (Cisco Umbrella top 100) column. 65 of the 100 responses from 1.1.1.1 were faster than 20 ms, but the 5 outlier responses over 100 ms pushed the mean up. Time permitting, I might revisit these results and create scatter plots that better represent these figures.

All of the raw dig output is available.

Testing public resolvers for response quality

As mentioned previously, the quality of DNS responses is just as important as the speed at which you receive them. Returning to the 9-1-1 analogy, when you make that phone call, you want to be connected to your local emergency dispatcher. Likewise, when you visit a website that uses a geographically sensitive content distribution network, the ideal scenario gets you pointed to a server relatively close by.

In this test, I asked each DNS server to resolve several popular distributed web services. These hostnames return a different IP address depending on who you're asking and where you're asking from. I avoided records configured to return multiple IPs in a "round-robin" fashion, like reddit.com (view) and twitter.com (view), which always resolve to the same RRset regardless of where in North America a query originates.

Each DNS server was queried only one time, mimicking a user typing a domain into their web browser, and the quality of that response was judged.

Amazon

1.1.1.1 provided a mediocre answer. It resolved www.amazon.com to a server in northern Virginia. 9.9.9.9 had the worst response, pointing me to Montreal, while the other services gave IPs in Dallas, which is closer to me. View test data.

DNS server   | Answer RR        | City             | Avg Latency
-------------|------------------|------------------|------------
1.1.1.1      | 52.85.142.82     | Dulles, VA       | 46.8 ms
4.2.2.4      | 184.29.151.228   | Dallas, TX       | 21.5 ms
8.8.8.8      | 54.192.5.179     | Dallas, TX       | 21.5 ms
9.9.9.9      | 52.84.87.216     | Montreal, QC     | 60.8 ms
75.75.75.75  | 13.32.171.225    | Dallas, TX       | 22.7 ms

Facebook

1.1.1.1 tied for best answer here. It resolved www.facebook.com to a server in Dallas; the same IP was given out by every server except Quad9. View test data.

DNS server   | Answer RR        | City             | Avg Latency
-------------|------------------|------------------|------------
1.1.1.1      | 157.240.19.35    | Dallas, TX       | 21.8 ms
4.2.2.4      | 157.240.19.35    | Dallas, TX       | 21.8 ms
8.8.8.8      | 157.240.19.35    | Dallas, TX       | 21.8 ms
9.9.9.9      | 31.13.65.36      | Atlanta, GA      | 33.1 ms
75.75.75.75  | 157.240.19.35    | Dallas, TX       | 21.8 ms

Google

1.1.1.1 produced the least optimal answer. It resolved google.com to a server in New York, the most distant IP given by any of the DNS services tested. View test data.

DNS server   | Answer RR        | City         | Avg Latency
-------------|------------------|--------------|------------
1.1.1.1      | 172.217.12.174   | New York, NY | 50.8 ms
4.2.2.4      | 216.58.194.110   | Dallas, TX   | 21.5 ms
8.8.8.8      | 172.217.6.174    | Dallas, TX   | 21.6 ms
9.9.9.9      | 172.217.164.46   | Atlanta, GA  | 31.8 ms
75.75.75.75  | 172.217.9.142    | Dallas, TX   | 21.5 ms

Slack

1.1.1.1 provided a mediocre answer. It resolved slack.com to a server in Minneapolis, which is fairly distant. Quad9 pointed me to Montreal again, and Comcast had the best response. View test data.

DNS server   | Answer RR        | City             | Avg Latency
-------------|------------------|------------------|------------
1.1.1.1      | 52.222.216.88    | Minneapolis, MN  | 50.0 ms
4.2.2.4      | 52.84.216.34     | St. Louis, MO    | 36.0 ms
8.8.8.8      | 52.84.124.154    | Dulles, VA       | 45.2 ms
9.9.9.9      | 52.84.91.113     | Montreal, QC     | 58.2 ms
75.75.75.75  | 13.32.171.57     | Dallas, TX       | 22.2 ms

YouTube

1.1.1.1 produced the least optimal answer. Surprisingly, it resolved youtube.com to a server in Bogota, Colombia, South America. The other DNS services pointed me to servers in Dallas and Atlanta. View test data.

DNS server   | Answer RR        | City             | Avg Latency
-------------|------------------|------------------|------------
1.1.1.1      | 216.58.222.238   | Bogota, Colombia | 86.6 ms
4.2.2.4      | 172.217.9.14     | Dallas, TX       | 21.4 ms
8.8.8.8      | 172.217.6.174    | Dallas, TX       | 21.5 ms
9.9.9.9      | 172.217.0.142    | Atlanta, GA      | 32.4 ms
75.75.75.75  | 172.217.9.14     | Dallas, TX       | 20.9 ms

Room for improvement

Cloudflare's DNS service is really fast, but from my perspective, it can provide low-quality responses (so does Quad9). Answers from 1.1.1.1 were geographically incongruous, directing me to distant servers when much closer points of presence exist.

That said, I'm confident that 1.1.1.1 will improve over time. DNS operators have a variety of tools like EDNS ECS and RPZ to enhance DNS resolution based on query origin. Upstream operators will gradually optimize for 1.1.1.1's traffic, too, aligning their own responses as they learn about the different nodes in Cloudflare's service. All of this takes time and study. Of the public DNS services I tested, the ones which have been operating longer gave better answers.

Potential privacy implications

Many users avoid their ISP's DNS service for a variety of reasons. Some ISPs hijack NXDOMAIN responses to insert advertising, which has a frustrating tendency to break non-web protocols entirely. Some ISPs collect your queries for marketing purposes or otherwise share them with third parties. Some smaller ISPs don't even operate their own name servers. These are all good cases for looking to an external service. Be aware that your ISP can still see your DNS queries to any server unless you're using DNS-over-TLS or DNS-over-HTTPS (both technologies are supported by 1.1.1.1 but I haven't evaluated them).

For its part, Cloudflare explicitly states that they don't log your IP address when you use 1.1.1.1. But the service is offered in cooperation with APNIC, whose wording isn't so clear. They promise to destroy "all β€˜raw’ DNS data as soon as we have performed statistical analysis on the data flow," but don't seem to lay out a timeline for that process. In the meantime, raw data could be subject to legal demands by entities with jurisdiction over APNIC and its constituent countries.

If you're a Comcast customer, I suggest sticking with Comcast's own DNS servers (75.75.75.75 and 75.75.76.76) unless you have a privacy concern. Comcast has built out local anycast nodes in all of their markets, and their service consistently provides high quality answers. They're able to do this because they can see more pieces of the puzzle: they know your location, while a public DNS service may be guessing; and they know how and where they prefer to route traffic, while a public DNS service has no idea.

Who you use for DNS is your choice and can have a significant impact on your internet experience. It's good to have more options, and I look forward to watching 1.1.1.1 evolve.



Recent articles

📰 Generating vanity DNSSEC key tags

📰 DDoS involving forged packets from 23.225.141.70

📰 Website integrity monitoring through version control

📰 SpamAssassin 3.4.2 fixes security problems, adds HashBL and phishing plugins

📰 Bug or turf war? ICQ via Pidgin now fails with "startOSCARSession: Request Timeout"

📰 πŸŽ‚

📰 SFSQuery, a PHP class to query the StopForumSpam API and DNSBL

📰 Resolving portmaster error "pkg-static: automake-1.16.1 conflicts with automake-wrapper-20131203"

📰 Resolving LibreNMS error "RuntimeException: The only supported ciphers are AES-128-CBC and AES-256-CBC with the correct key lengths"

📰 1.1.1.1: Fast, but not so accurate (yet)

📰 autodiscover.xml as an Indicator of Attack

📰 Blocking Facebook's Tracking and Surveillance: A Comprehensive Approach

📰 Let's Encrypt Readies for Certificate Transparency with Embedded SCTs

📰 Evaluating DNSBL Effectiveness with Postfix Logs

📰 Resolving subversion error E145001: Node has unexpectedly changed kind

▲ Back to top | Permalink to this page