Mac OS X Lion, /etc/hosts Bugs, and DNS Resolution

Final Update: I’m working on a follow-up post, but thanks to all the comments, it seems that somehow during the Snow Leopard to Lion upgrade, the /etc/hosts file was encoded incorrectly (likely the type of newlines used), and that Lion would fail to read it properly. Please try to copy the contents of the /etc/hosts file, recreate the file, and paste to fix your problem.

Update: I was still having problems with using .local for my development sites. After some extensive debugging and research I’ve found out some interesting things. It appears that Multicast DNS (mDNS), which is used in Apple’s Bonjuor for network discovery without a tradition DNS Server, has the Top Level Domain .local reserved for special mDNS functionality. Mac OS X will treat domains ending with .local differently! Use another TLD, like .dev, and it will work correctly.

This article will still help you fix problems with /etc/hosts abnormalities with it not having the correct priority.

Update 2: Thanks for all the comments! A few updates: First off, per the comments, if you declare both a IPv4 and IPv6 address for a .local domain like so:

27.0.0.1 domain.local
::1 domain.local

Then it will work correctly. Second, I ended up doing a fresh install of Lion and this problem has gone away for me. So it appears that this issue would affect some people who upgraded from Snow Leopard. So as Chris suggested below, I don’t think it has to do with the resolve order. I haven’t been able to figure out the difference between Lion installs that work correctly, and those that don’t. So if you are have the following problems, the suggestions below will work to fix it.

This morning I was up a few hours earlier than normal today, and was looking forward to getting in some solid programming hours earlier so I wouldn’t work late tonight. Instead, I spent about four hours fighting a new change (I’d call it a bug) in Mac OS X Lion.

The /etc/hosts File

Now, instead of running a full blown DNS server on my machine, which is overkill and more work to edit and maintain, I would make manual adjustments to my /etc/hosts file. For those not familiar with the /etc/hosts file, it allows you to add entries that traditionally your computer will look up first before trying your server DNS. This is where your computer would set “localhost” is the IP “127.0.0.1”.

I use this a lot with local development. For example, when I work on joind.in, I use MAMP and create a VirtualHost for the code under the server name local.joind.in (i.e. http://local.joind.in/). I don’t have access to the joind.in DNS, so I create an entry in my /etc/hosts file. You have to have admin permissions to edit the file, so I normally run the command “sudo nano /etc/hosts” and make my changes and then save.

The /etc/hosts file looks something like this:

##
# Host Database
#
# localhost is used to configure the loopback interface
# when the system is booting.  Do not change this entry.
##
127.0.0.1   localhost
255.255.255.255 broadcasthost
::1             localhost
fe80::1%lo0    localhost

127.0.0.1       local.joind.in
209.114.40.229  www.example.com

Now, my actual /etc/hosts file contains 83 entries. Some are just short cuts to machines I often connect to. Sometimes during a migration when I’m making DNS entries, I can make the change in my /etc/hosts file as well, so I don’t have to wait for the DNS changes to propagate to test. All in all, it is a very handle little file that I use a great deal. Even windows has their own hosts file that works well.

Problem #1: Bugs with Multiple Domains per Line

With the new Lion Update comes better support for IPv6, however this seems to have broken the /etc/hosts file from allowing multiple domains per line. Before you could do something like this, which is works in several other operating systems:

127.0.0.1 local.joind.in local.example.com

However, this now seems to break and not work. You must keep each line to one ip and one domain. So your configuration should look like this:

127.0.0.1 local.joind.in
127.0.0.1 local.example.com

There was also reports about the order of IPv4 and IPv6 entries (blog post about it). I personally did not have this problem, but others seem to have had it. The problem was that OS X Lion seemed to ignore IPv4 entries after a IPv6 entry was made.

Problem #2: DNS Resolution Orders has DNS Servers Before the Hosts File

This is the problem that really, really ticked me off and caused me to waste hours and hours trying to debug and find a work-around. I was working on my local copy of Clipish, and because we have different domains for it that work together, I replace .com with .local. So I would have something like this in my /etc/hosts file:

127.0.0.1 staff.example.local
127.0.0.1 images.example.local
127.0.0.1 www.example.local

However, anytime I made a request in Chrome to any of those domains, it would take 5.01 seconds. At first I thought it might be a bug with the new version of MAMP for OS X Lion. But I found using Chrome’s & FireFox’s developer tools that it was taking 5 seconds for the DNS lookup, and 0.01 seconds for the rest. So I would go into my terminal and ping the domain. It worked just fine without the delay.

Apparently, the terminal and some of the BSD Unix tools correctly use /etc/resolv.conf and the correct order of /etc/hosts first and then DNS servers. However, everything else on OS X Lion, including all of your Applications, do it backwards! This wasn’t the case in Tiger, Leopard, or Snow Leopard.

You can test this by running the command “scutil –dns”:


Justin-Carmonys-MacBook-Pro-2:etc justin$ scutil --dns
DNS configuration

resolver #1
  nameserver[0] : 8.8.8.8
  nameserver[1] : 192.168.1.1

resolver #2
  domain   : local
  options  : mdns
  timeout  : 5
  order    : 300000

resolver #3
  domain   : 254.169.in-addr.arpa
  options  : mdns
  timeout  : 5
  order    : 300200

resolver #4
  domain   : 8.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  order    : 300400

resolver #5
  domain   : 9.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  order    : 300600

resolver #6
  domain   : a.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  order    : 300800

resolver #7
  domain   : b.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  order    : 301000

DNS configuration (for scoped queries)

resolver #1
  nameserver[0] : 127.0.0.1
  if_index : 5 (en1)
  flags    : Scoped

I spent hours and hours googling, finding a bunch of people with this same problem, but no solution on how to fix the ordering. I have my suspicions why Apple made this change, which I’ll address later on. But after I couldn’t figure out a way to change the resolve order myself, I had to find another alternative. (Of course, OS X removed /etc/nsswitch.conf so I can’t change it there, grrr!)

Solution: DNSMasq

The only solution we have, that I could find, is to use your own DNS Server locally on your own machine. I’ve heard of others setting up BIND on OS X. Now, the problem with this solution is that it isn’t nearly as quick and easy to manage DNS entries.

Then, I found out about DNSMasq. It is a lightweight DNS server that is easy to configure. However, it had one key feature we needed: it will read the /etc/hosts file and serve it’s entries. This means we can have our OS check the DNSMasq instance first, which will check the /etc/hosts file, restoring the proper order.

Installing DNSMasq

After some Googling around, I think installing dnsmasq using MacPorts is the easiest. First, you’ll need to install Xcode so your machine can compile with the tools. It is free on the Mac App Store (warning, it is a kind of big download). Then, you’ll need to download and install the MacPorts for Lion. If you already have MacPorts installed, make sure it is the current version of Lion.

Once it is installed, open up the terminal and then execute these commands. First, run selfupdate to make sure your MacPorts repositories are up to date:

sudo port selfupdate

Then, install dnsmasq:

sudo port install dnsmasq

That should execute with no problems. Now, it is installed, but we need to have OS X run it:

sudo port load dnsmasq

Now, if you want to make some changes to your dnsmasq configurations, you can do that in this file:

/opt/local/etc/dnsmasq.conf

One thing I like to do is control which DNS Server’s dnsmasq will use. By default it will use /etc/resolv.conf, but OS X will overwrite that file on reboot and anytime you change internet connections (like switching wifi hotspots). So I would copy /etc/resolv.conf to /etc/resolv.dnsmasq.conf. When dealing with files in /etc/ and /opt/ you’ll need to run these commands using sudo.

sudo cp /etc/resolv.conf /etc/resolv.dnsmasq.conf
sudo nano /etc/resolv.dnsmasq.conf

Then I set dnsmasq to read my new resolve.dnsmasq.conf by setting “resolv-file=/etc/resolv.dnsmasq.conf” in /opt/local/etc/dnsmasq.conf.

Now, we need to configure OS X to add 127.0.0.1 to our DNS Servers. Go to System Preferences > Network > (Select Connection, i.e. Wi-Fi) > Advanced > DNS.

You can have other DNS entries, like Google’s DNS Entries (8.8.8.8, 8.8.4.4). Just make sure 127.0.0.1 is on top.

One thing I haven’t been able to figure out is how to reboot dnsmasq, but if you kill the process in the Activity Monitor OS X will restart it:

This is a lot of work to setup (especially if you don’t have Xcode install). However, it will work, and give you a lot more control over your DNS. So in a way it is an improvement! Just wish it wasn’t necessary.

If you would like to do wildcard dns entries, something you couldn’t do in /etc/hosts, then you can add this to your dnsmasq config:

address=/.local/127.0.0.1

Now any domains ending in .local will route to 127.0.0.1

Why is this necessary Apple?

What I don’t get, is why this change Apple? No other operating system that I know if, including Windows for crying out loud, does it this way. This had to be a deliberate change by Apple. I can only see one of the following reasons:

  • Security – Maybe, just maybe, it was for security. Technically, a virus can overwrite the /etc/hosts file if it the virus can prompt the user for their password to run a command as root. So you could maybe have a user visit http://www.google.com when in reality it isn’t a google server. But even then, HTTPS wouldn’t work without ugly errors. So this might be a reason, but out of all the things that can go wrong with a virus, it is a poor excuse. The virus could just hack mDNSresponder to use /etc/hosts.
  • Prevent User Hacks – Most likely, I think this is why Apple made this change. I know with jailbreaking iOS devices and getting hacked apps, iTunes will try to validate an App by “phoning home” to Apple. However, if you change the entries in your /etc/hosts file so iTunes can’t reach the correct Apple Servers. I know people who have circumvented other software activation systems do the same technique. I think this is why Apple made the change, though at the expensive of all the professionals who actually use /etc/hosts for work.
  • An Honest Bug – It is possible this was a bug, but seeing as how you would have to deliberately change the resolve orders for the operating system, I doubt this is the case.

If this was an intended change by Apple, it is a frustrating one. One of my favorite things is that underneath the OS X’s nice UI and Applications is a BSD Unix system. I love how I can compile and install so many things just like I would on Ubuntu. But if they continue to change that Unix underbelly and make it less and less like it is everywhere else, then I think they’ll be frustrating many, many professionals.

Hopefully in the future I’ll find a way to change the resolve order in Mac OS X, but until this, this is how I worked around the problem.

97 thoughts on “Mac OS X Lion, /etc/hosts Bugs, and DNS Resolution

  1. Still no luck, I followed line-by-line your tutorial and other guides found on web but my virtualhosts still resolve to “main” localhost…

    I do not understand how (or if) /etc/resolv.dnsmasq.conf has to be edited, mine is just one line:
    nameserver 192.168.0.1

    My OS is Lion, 10.7.3

    T.

    Like

  2. Thank you VERY much, this 5s timeout was driving me mad! For me, just changing from .local to .dev solved it.

    Like

  3. Thanks so much, I just googled for “long delay when calling local host mamp” and your post popped right up. Changing form .local to .dev just saved me hours of debugging 🙂 It solved my problem.

    Like

  4. Thank you so much for posting this – helped solve that 5 second problem I was experiencing.

    Like

  5. Thanks for the great information.

    Unfortunately, adding a local DNS server in System Preferences throws away any DNS servers that are assigned by a DHCP server. Is there any way to have a local DNS server (127.0.0.1) in addition to the DNS servers assigned by DHCP?

    Like

  6. Frank Drouillard April 14, 2012 — 5:47 pm

    Solved my problem with localhost not resolving with Apache 2.4.1 under OS X Lion.

    I specified “Listen my.numerical.IP.addy:80” at or near line 50 in my Apache configuration file. When I changed it to “Listen 80” only, everything began working as expected.

    Like

  7. Andy Charalambous April 22, 2012 — 1:40 pm

    Thanks – you saved my sanity. I spent hours late last night trying to figure out if it was nginx, passenger or my slightly not-quite-supported build of ruby on OS X Lion via RVM that was at fault – should have checked the browser network profiling first I guess…forest for the trees.

    Like

  8. To restart dnsmasq, send it a HUP signal.
    sudo kill -HUP `cat /opt/local/var/run/dnsmasq.pid`

    Like

  9. Thanks a lot for this! Solution from the final update works for me.

    Like

  10. This is what solved it for me: The line ending of my hosts file were Mac Classic (CR), which is wrong. Open it using TextWrangler ( http://www.barebones.com/products/TextWrangler/ ) and on the bottom of the window, in the middle, you must change is to Unix (LF) and save it. Boom, worked.

    Like

  11. I didn’t actually find this site via an issue with .local, but discovered DNSMasq, saw that it would be a perfect solution to a problem I was having (which included skipping the hosts file for each new entry I added), and went in search for a good post explaining how to set it up, and that post was this one. Thanks!

    Like

  12. Thanks a lot for the investigation.

    I’ve found yet another workaround for the problem. If you can choose the suffix for your local domains, try “.localhost” instead of “.local” or something else (that is, use domains “foo.localhost” instead of “foo.local”).

    It seems that Mac OS X has a shortcut for “localhost”-type domains.

    Like

  13. Thank you so much! This solves my problem. I used homebrew to install dnsmasq.

    Like

  14. Thank you for this. We us a special internal domain name at work that is a unique tld. For people inside the company it seems to be fine. But for people that VPN into work and need to use one or these special tld domains, then it fails for people on OSX. People using Windows or Linux have no problem resolving these special tld’s. Once dnsmasq was installed and configured all was well.

    I aslo used homebrew to install dnsmasq.

    Like

  15. I just wanted to thank you for your post. I spent more of my night than I’d care to admit tracking down mDNS issues and tweaking my /etc/hosts file only to realize I was being impacted by the multiple hosts/line bug (FWIW in Mountain Lion I can get 10 hosts to a line before it breaks).

    Like

  16. Jan Mark Salarda February 4, 2013 — 6:38 am

    Thanks.I was able to intall dnsmasq succcessfully with my os lion. I have this problem where in I have to refresh browser to load the page for my development site. Because it will load initially the working site. Thanks again.

    Like

  17. I still have no idea why my /etc/hosts file wasn’t (isn’t) working, but this saved me from further grief. Many thanks.

    Like

  18. Have you perhaps checked that your file didn’t accidentally get the line endings changed from n to r?

    That’s what happened to mine. This fixed it for me:
    sudo tr ‘r’ ‘n’ /etc/hosts

    Like

  19. Ah, and of course the forum software strips the gt/lt characters instead of converting them to html entities.

    Hopefully this will work:
    tr ‘r’ ‘n’ < /etc/hosts > ~/Desktop/hosts

    Like

  20. And then I paste the wrong thing. Go me.

    THIS is what you need to run:
    sudo tr ‘r’ ‘n’ < /etc/hosts > /etc/hosts

    Like

  21. Which step makes the hosts file load before the DNS servers? You didn’t mention what goes into the resolv.dnsmasq.config file, do we need to edit anything in there? Thanks.

    Like

  22. I also came across similar DNS issues in a Mountain Lion where my host file seems to be ignored. Turns out all I needed todo was add the lookup order in resolv.conf as follows:
    1 – Launch Terminal
    2 – sudo vi /etc/resolv.conf #You’ll be prompted for you password
    3 – #At top of the file press ‘I’ to goto insert mode
    4 – add “order hosts, bind” #without the quotation marks
    5 – press Esc (out of insert mode) and then type :wq and enter

    Like

  23. Good web site you’ve got here.. It’s hard to find quality writing like yours nowadays. I seriously appreciate people like you! Take care!!

    Like

  24. Changing the line endings from CR to LF solved it for me.

    Like

  25. To products a fuller perception of area, a much further thorough advanced education experience and too to create the entire normal commencing the university. Clothed in his peculiar lexis, Robert Zimmer, Leader of the University allied with Chicago.

    Like

  26. Hiya. Vastly nice website!! Guy .. Magnificent .. Superb .. I’ll bookmark your website and assume the feeds also…I’m satisfied to locate numerous helpful information here within the article. Be grateful you for sharing..

    Like

  27. The the next time I read a blog, Im hoping whos doesnt disappoint me about that one. I mean, I know it was my replacement for read, but I really thought youd have some thing fascinating to talk about. All I hear is really a number of whining about something you could fix when you werent too busy searching for attention.

    Like

  28. Thanks a lot, it works for me!!

    Like

  29. The command you are looking for for stoping and starting dnsmasq are

    sudo launchctl stop homebrew.mxcl.dnsmasq

    sudo launchctl start homebrew.mxcl.dnsmasq

    Of course I installed with homebrew as follows:

    brew up
    brew install dnsmasq

    Like

  30. Some truly nice and utilitarian info on this site, likewise I think the layout holds superb features.

    Like

  31. For me this actually turned out to be a permissions issue. Make sure the hosts file has the following permissions:

    -rw-r–r– 1 root wheel hosts

    Like

  32. “.local is reserved for special mDNS functionality” – priceless… Computer gods are agains me.

    Like

  33. Anonymize your trusty old true IP address, and additionally fly under the radar.

    Like

  34. Great!
    ” .local reserved for special mDNS functionality. ”
    Thanks for telling me this!

    Like

  35. Maybe a minor off-topic, however, you have a very nice theme website, very clear, so that the fantastic reads your blog 🙂

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this:
search previous next tag category expand menu location phone mail time cart zoom edit close