[ root | ruminations | remote ]

Identifying the DNS server used to fulfill an HTTP request


tl;dr: I hacked up an endpoint that will identify the DNS server that an end-user is using; this helps us diagnose DNS issues that might show up in production.

http://webutils.flourishworks.com/dns

As developers we have a handful of tools at our disposal for remotely debugging HTTP issues. You can pass along a debug URL to your end users and have them send back the results1. From JavaScript we have access to the basics that come along with the user-agent: browser, platform, and versions. It's also easy to include things like their IP and request headers by setting up an endpoint that echos back the request. Put the right headers on that endpoint and you can then include that additional information by way of XHR request. This on its own will get you a good chunk of information: you can infer things like what carrier they're on by the IP, whether the request is being changed in unexpected ways, if there's a proxy in the mix, and so on.

Our old tool chest is insufficient when it comes to DNS

On the Firebase diagnostics page, we gather as much information as we possibly can about a user to try to diagnose connectivity problems. As we've scaled, we've started to notice that DNS is not always the perfect system we might like it to be, and sometimes our end users have problems with inaccurate or mangled DNS requests. To help us get to the bottom of these problems I created a hack to let me identify the end user's DNS server.

The hack

There are plenty of tools out there (ifconfig.me, icanhasip.com, ...) that do the basics but I couldn't find2 any that gave information about the intermediate DNS server. More often than not, it's not enough to get the user to share their network setup, resolv.conf, or the results of dig/nslookup since it'll have something such as ;; SERVER: 10.0.0.1#53(10.0.0.1) — an internal/private network IP. We can, however, get the user's browser to kick off a specially crafted DNS request which will eventually get delegated to a server we control. I put together the following hack to gather the information I'm after.

Things you'll need:

  1. A host you can bind a custom DNS server on the standard port 53.
  2. A domain for which you can delegate off a subdomain with an IN NS record.
  3. An HTTP server/virtual host configured to answer for wildcard subdomains.

Then, the setup goes like this:

  1. Spin up a custom DNS server using your favorite library: Java, Ruby, Python, Node, etc. There are plenty of them out there. You don't need anything complicated or feature complete because we'll be able to get away with serving up simple responses. Here is a gist of the implementation I'm using.
  2. Delegate a subdomain.domain.com to point to your custom DNS server by adding an IN NS entry to your zone.
  3. Modify the DNS server code to respond to every request with an answer that points to a virtual host that will reply with the information we want to return. Prior to actually sending the DNS query answer, save off the query information — this is where we'll capture the source IP of the DNS server and request headers.
  4. Have your virtual host 301 all requests for subdomain.domain.com to another random and unique subdomain; this is so we can properly identify the request to the DNS query. I'm tacking on a timestamp and a random number to the subdomain during the redirect.
  5. Have your virtual host reply with the information that was saved off in step 3.
  6. Done! Easy as cake! 🍰

After that's all wired up, here's what the results look like. Note curl kicking off requests for A (type 1) & AAAA (type 28) queries. Most browsers will just ask for the A record.

$ curl -Ls http://webutils.flourishworks.com/dns | python -mjson.tool
{
    "-IklVnhBlhNbrm1YrUdl": {
        "address": {
            "address": "76.96.98.5", 
            "family": "IPv4", 
            "port": 36129, 
            "size": 91
        }, 
        "header": {
            "aa": 0, 
            "id": 37288, 
            "opcode": 0, 
            "qr": 0, 
            "ra": 0, 
            "rcode": 0, 
            "rd": 0, 
            "res1": 0, 
            "res2": 0, 
            "res3": 0, 
            "tc": 0
        }, 
        "poweredBy": "Firebase", 
        "question": [
            {
                "class": 1, 
                "name": "t1358040349355.r-780970521833157226.webutils.flourishworks.com", 
                "type": 1
            }
        ], 
        "timestamp": 1358040517054
    }, 
    "-IklVnhO6WJ8aUn0MGmM": {
        "address": {
            "address": "76.96.98.4", 
            "family": "IPv4", 
            "port": 34293, 
            "size": 91
        }, 
        "header": {
            "aa": 0, 
            "id": 59193, 
            "opcode": 0, 
            "qr": 0, 
            "ra": 0, 
            "rcode": 0, 
            "rd": 0, 
            "res1": 0, 
            "res2": 0, 
            "res3": 0, 
            "tc": 0
        }, 
        "poweredBy": "Firebase", 
        "question": [
            {
                "class": 1, 
                "name": "t1358040349355.r-780970521833157226.webutils.flourishworks.com", 
                "type": 28
            }
        ], 
        "timestamp": 1358040517052
    }
}

In $id.address.address you'll find the IP of the DNS server; in my case, that turned out to be 76.96.98.4 (utah-dnssec01.saltlakecity.ut.utah.comcast.net). Now we have a tool we can send to users to help us further diagnose issues with their connections. With a few tweaks to the HTML kicking off the queries, we can even instrument the tests to see if the intermediate DNS servers are honoring TTLs on our records.

Down the rabbit hole we go.


  1. On a healthy connection we'd typically do something that would automatically POST back the results to us; however, in many cases this is precisely what we are troubleshooting. The diagnostic results are generally sent back to us over email after the user has run the browser tests.
  2. Searching for a tool like this proved to be quite a challenge; any mix of DNS, tool, and HTTP brings back hits for tools that are web interfaces for nslookup, dig, or whois. What search terms would one use to find the tool described in this blog post?

If you enjoyed reading this, catch me on Twitter