Final Update: I’m working on a follow-up post, but thanks to all the comments, it seems that somehow during the Snow Leopard to Lion upgrade, the /etc/hosts file was encoded incorrectly (likely the type of newlines used), and that Lion would fail to read it properly. Please try to copy the contents of the /etc/hosts file, recreate the file, and paste to fix your problem.
Update: I was still having problems with using .local for my development sites. After some extensive debugging and research I’ve found out some interesting things. It appears that Multicast DNS (mDNS), which is used in Apple’s Bonjuor for network discovery without a tradition DNS Server, has the Top Level Domain .local reserved for special mDNS functionality. Mac OS X will treat domains ending with .local differently! Use another TLD, like .dev, and it will work correctly.
This article will still help you fix problems with /etc/hosts abnormalities with it not having the correct priority.
Update 2: Thanks for all the comments! A few updates: First off, per the comments, if you declare both a IPv4 and IPv6 address for a .local domain like so:
22.214.171.124 domain.local ::1 domain.local
Then it will work correctly. Second, I ended up doing a fresh install of Lion and this problem has gone away for me. So it appears that this issue would affect some people who upgraded from Snow Leopard. So as Chris suggested below, I don’t think it has to do with the resolve order. I haven’t been able to figure out the difference between Lion installs that work correctly, and those that don’t. So if you are have the following problems, the suggestions below will work to fix it.
This morning I was up a few hours earlier than normal today, and was looking forward to getting in some solid programming hours earlier so I wouldn’t work late tonight. Instead, I spent about four hours fighting a new change (I’d call it a bug) in Mac OS X Lion.
The /etc/hosts File
Now, instead of running a full blown DNS server on my machine, which is overkill and more work to edit and maintain, I would make manual adjustments to my /etc/hosts file. For those not familiar with the /etc/hosts file, it allows you to add entries that traditionally your computer will look up first before trying your server DNS. This is where your computer would set “localhost” is the IP “127.0.0.1”.
I use this a lot with local development. For example, when I work on joind.in, I use MAMP and create a VirtualHost for the code under the server name local.joind.in (i.e. http://local.joind.in/). I don’t have access to the joind.in DNS, so I create an entry in my /etc/hosts file. You have to have admin permissions to edit the file, so I normally run the command “sudo nano /etc/hosts” and make my changes and then save.
The /etc/hosts file looks something like this:
## # Host Database # # localhost is used to configure the loopback interface # when the system is booting. Do not change this entry. ## 127.0.0.1 localhost 255.255.255.255 broadcasthost ::1 localhost fe80::1%lo0 localhost 127.0.0.1 local.joind.in 126.96.36.199 www.example.com
Now, my actual /etc/hosts file contains 83 entries. Some are just short cuts to machines I often connect to. Sometimes during a migration when I’m making DNS entries, I can make the change in my /etc/hosts file as well, so I don’t have to wait for the DNS changes to propagate to test. All in all, it is a very handle little file that I use a great deal. Even windows has their own hosts file that works well.
Problem #1: Bugs with Multiple Domains per Line
With the new Lion Update comes better support for IPv6, however this seems to have broken the /etc/hosts file from allowing multiple domains per line. Before you could do something like this, which is works in several other operating systems:
127.0.0.1 local.joind.in local.example.com
However, this now seems to break and not work. You must keep each line to one ip and one domain. So your configuration should look like this:
127.0.0.1 local.joind.in 127.0.0.1 local.example.com
There was also reports about the order of IPv4 and IPv6 entries (blog post about it). I personally did not have this problem, but others seem to have had it. The problem was that OS X Lion seemed to ignore IPv4 entries after a IPv6 entry was made.
Problem #2: DNS Resolution Orders has DNS Servers Before the Hosts File
This is the problem that really, really ticked me off and caused me to waste hours and hours trying to debug and find a work-around. I was working on my local copy of Clipish, and because we have different domains for it that work together, I replace .com with .local. So I would have something like this in my /etc/hosts file:
127.0.0.1 staff.example.local 127.0.0.1 images.example.local 127.0.0.1 www.example.local
However, anytime I made a request in Chrome to any of those domains, it would take 5.01 seconds. At first I thought it might be a bug with the new version of MAMP for OS X Lion. But I found using Chrome’s & FireFox’s developer tools that it was taking 5 seconds for the DNS lookup, and 0.01 seconds for the rest. So I would go into my terminal and ping the domain. It worked just fine without the delay.
Apparently, the terminal and some of the BSD Unix tools correctly use /etc/resolv.conf and the correct order of /etc/hosts first and then DNS servers. However, everything else on OS X Lion, including all of your Applications, do it backwards! This wasn’t the case in Tiger, Leopard, or Snow Leopard.
You can test this by running the command “scutil –dns”:
Justin-Carmonys-MacBook-Pro-2:etc justin$ scutil --dns DNS configuration resolver #1 nameserver : 188.8.131.52 nameserver : 192.168.1.1 resolver #2 domain : local options : mdns timeout : 5 order : 300000 resolver #3 domain : 254.169.in-addr.arpa options : mdns timeout : 5 order : 300200 resolver #4 domain : 8.e.f.ip6.arpa options : mdns timeout : 5 order : 300400 resolver #5 domain : 9.e.f.ip6.arpa options : mdns timeout : 5 order : 300600 resolver #6 domain : a.e.f.ip6.arpa options : mdns timeout : 5 order : 300800 resolver #7 domain : b.e.f.ip6.arpa options : mdns timeout : 5 order : 301000 DNS configuration (for scoped queries) resolver #1 nameserver : 127.0.0.1 if_index : 5 (en1) flags : Scoped
I spent hours and hours googling, finding a bunch of people with this same problem, but no solution on how to fix the ordering. I have my suspicions why Apple made this change, which I’ll address later on. But after I couldn’t figure out a way to change the resolve order myself, I had to find another alternative. (Of course, OS X removed /etc/nsswitch.conf so I can’t change it there, grrr!)
The only solution we have, that I could find, is to use your own DNS Server locally on your own machine. I’ve heard of others setting up BIND on OS X. Now, the problem with this solution is that it isn’t nearly as quick and easy to manage DNS entries.
Then, I found out about DNSMasq. It is a lightweight DNS server that is easy to configure. However, it had one key feature we needed: it will read the /etc/hosts file and serve it’s entries. This means we can have our OS check the DNSMasq instance first, which will check the /etc/hosts file, restoring the proper order.
After some Googling around, I think installing dnsmasq using MacPorts is the easiest. First, you’ll need to install Xcode so your machine can compile with the tools. It is free on the Mac App Store (warning, it is a kind of big download). Then, you’ll need to download and install the MacPorts for Lion. If you already have MacPorts installed, make sure it is the current version of Lion.
Once it is installed, open up the terminal and then execute these commands. First, run selfupdate to make sure your MacPorts repositories are up to date:
sudo port selfupdate
Then, install dnsmasq:
sudo port install dnsmasq
That should execute with no problems. Now, it is installed, but we need to have OS X run it:
sudo port load dnsmasq
Now, if you want to make some changes to your dnsmasq configurations, you can do that in this file:
One thing I like to do is control which DNS Server’s dnsmasq will use. By default it will use /etc/resolv.conf, but OS X will overwrite that file on reboot and anytime you change internet connections (like switching wifi hotspots). So I would copy /etc/resolv.conf to /etc/resolv.dnsmasq.conf. When dealing with files in /etc/ and /opt/ you’ll need to run these commands using sudo.
sudo cp /etc/resolv.conf /etc/resolv.dnsmasq.conf sudo nano /etc/resolv.dnsmasq.conf
Then I set dnsmasq to read my new resolve.dnsmasq.conf by setting “resolv-file=/etc/resolv.dnsmasq.conf” in /opt/local/etc/dnsmasq.conf.
Now, we need to configure OS X to add 127.0.0.1 to our DNS Servers. Go to System Preferences > Network > (Select Connection, i.e. Wi-Fi) > Advanced > DNS.
You can have other DNS entries, like Google’s DNS Entries (184.108.40.206, 220.127.116.11). Just make sure 127.0.0.1 is on top.
One thing I haven’t been able to figure out is how to reboot dnsmasq, but if you kill the process in the Activity Monitor OS X will restart it:
This is a lot of work to setup (especially if you don’t have Xcode install). However, it will work, and give you a lot more control over your DNS. So in a way it is an improvement! Just wish it wasn’t necessary.
If you would like to do wildcard dns entries, something you couldn’t do in /etc/hosts, then you can add this to your dnsmasq config:
Now any domains ending in .local will route to 127.0.0.1
Why is this necessary Apple?
What I don’t get, is why this change Apple? No other operating system that I know if, including Windows for crying out loud, does it this way. This had to be a deliberate change by Apple. I can only see one of the following reasons:
- Security – Maybe, just maybe, it was for security. Technically, a virus can overwrite the /etc/hosts file if it the virus can prompt the user for their password to run a command as root. So you could maybe have a user visit http://www.google.com when in reality it isn’t a google server. But even then, HTTPS wouldn’t work without ugly errors. So this might be a reason, but out of all the things that can go wrong with a virus, it is a poor excuse. The virus could just hack mDNSresponder to use /etc/hosts.
- Prevent User Hacks – Most likely, I think this is why Apple made this change. I know with jailbreaking iOS devices and getting hacked apps, iTunes will try to validate an App by “phoning home” to Apple. However, if you change the entries in your /etc/hosts file so iTunes can’t reach the correct Apple Servers. I know people who have circumvented other software activation systems do the same technique. I think this is why Apple made the change, though at the expensive of all the professionals who actually use /etc/hosts for work.
- An Honest Bug – It is possible this was a bug, but seeing as how you would have to deliberately change the resolve orders for the operating system, I doubt this is the case.
If this was an intended change by Apple, it is a frustrating one. One of my favorite things is that underneath the OS X’s nice UI and Applications is a BSD Unix system. I love how I can compile and install so many things just like I would on Ubuntu. But if they continue to change that Unix underbelly and make it less and less like it is everywhere else, then I think they’ll be frustrating many, many professionals.
Hopefully in the future I’ll find a way to change the resolve order in Mac OS X, but until this, this is how I worked around the problem.
97 thoughts on “Mac OS X Lion, /etc/hosts Bugs, and DNS Resolution”
You are still running
dscacheutil -flushcacheafter you edit your /etc/hosts file right?
Ah! I forgot, yes, I did do that, though I forgot to add that in the post.
your conclusion about “terminal and some of the BSD Unix tools” is wrong. you only used ping. now ping is an ipv4-only tool, use ping6 to ping v6 hosts.
what you can do for testing is using telnet. use telnet -4 or -6 as option if you want to enforce ip 4 or 6 usage.
i hate not beeing able to tell how, but all of my host file issues disappeared last night. i’ using v4, v6, single and multiple domain per line, and it all just works. i revert all changes in hosts except that i renamed all “domains” ending in .local to .dev
Why do you need multiple names per line? I usually make a separate line for each name.
I haven’t experienced any of these issues since upgrading to Lion. My scutil output is similar to yours and I’ve had none of the other issues.
In my /etc/hosts file, I have several lines of 127.0.0.1, each with multiple entries and all my local hosts work fine.
I’m using Macports for php and apache, though I doubt that would make a difference.
I know you said you don’t like using BIND because it is overkill and setup is complex, but it is bundled with OSX and once you are done configuring it isn’t so bad anymore. Check out my post here: http://clintberry.com/2011/05/wildcard-sub-domains-on-osx-web-development-on-localhost/
Thanks for the comments & corrections everyone.
The biggest problems that I could replicate consistently were first: the non-standard TLDs (.local, .dev). I think that was because with the real TLDs, the DNS servers would come back quickly saying the domain did not exist, and then it would hit the hosts file. But with the non-standard TLDs, it would have to timeout before hitting the hosts file.
Second: not being able to use my /etc/hosts to overwrite normal DNS lookups. For example, if I want to skip my load balancer and test against a specific server behind the load balancer, I would edit my /etc/hosts file. But with Lion using the DNS Servers before the /etc/hosts, that trick doesn’t work, and I use it often. Using dnsmasq, it works like before.
Justin, great post on the whole issue. I had just found some other post on dnsmasq right before seeing this. Your post handles it much more comprehensively.
For some reason MacPorts wouldn’t install dnsmasq for me. I get “Error: Target org.macports.build returned: shell command failed (see log for details)”. I had to use homebrew to install dnsmasq so it was a little non-standard. (I typically use MacPorts for everything.)
Thanks for discovering that .local is reserved. I was using it, but with mixed results in Lion. Sounds like switching away from it will be helpful.
Thank you for taking the time to write all of this up. I just added a new domain to my hosts file and was wondering why it wasn’t resolving to the IP that I specified. Your analysis makes sense and I’m going to try your suggestion about DNSmasq.
Hey Apple – please restore the original behavior of /etc/hosts!
Just a follow-up. Lion was indeed supporting multiple hostnames per line in /etc/hosts, for me at least.
Nevertheless, I have DNSmasq installed and all seems to be working well.
I can confirm the same issue happens on SSH, each time when I ssh to a host with a shortcut in /etc/hosts, it takes 5 seconds or so to connect, while using IP only takes like 1 second.
Just a massive thank you for this.
I’ve been having some serious issues with dns with Lion. As a developer i also use .local for around 30 sites i host locally…
I’ve been pulling apart apache, mysql… nightmare!
Changing to .dev has fixed thing for me, big thanks to you!
Instead of using /etc/hosts, you can use the dscl command to manage your hosts on Mac OS X (example, http://tomafro.net/2009/07/dscl-the-easy-way-to-add-hosts-on-osx).
More simple, just install the ruby gem named “ghost” (https://github.com/bjeanes/ghost).
I’m able to use local address and override dns via host file in lion. i have my local addresses above fe80::1%lo0 and my overrides after it. seems to be working fine for me.
For what it’s worth, I just tested this by pointing amazon.com to one of my own servers using /etc/hosts, and it worked just fine. Visiting amazon.com in Chrome, Safari, and Firefox worked as expected. Things like ping and telnet also worked as expected.
Because of this, I suspect the issue has nothing to do with the order of DNS resolvers (by the way, you’re missing a character; it’s scutil –dns).
Now I know why you’re missing a character. Your blog thinks two consecutive hyphens is shorthand for an em dash or something. (Preview would be nice.)
There is a little confusion here. In Lion, DNS resolution waits a little longer if there is no response to AAAA (IPv4 and IPv6 are looked up at the same time). /etc/hosts is always consulted first, but if there is no IPv6 entry, it asks the external DNS server and waits for 5 seconds to timeout for .local (for non-dotlocal, it might take longer but the external DNS server might answer with “No response”.)
So, try this: Add a host that does not end in local, it should respond immediately (assuming the external DNS server responds to AAAA query immediately which most of them do)
Thanks a lot for writing about this.
I’ve tried Anonymous’s solution and it works just fine.
In /etc/hosts, add 2 lines for each domain :
Just stumbled upon the same problem here. Turns out it is because I was using a symlink to my hosts file that I keep in a Git repo.
I can confirm both Anonymous and Nedios suggestions.
I’m testing Active Directory integration, and had trouble accessing network accounts in a .local domain after binding and a subsequent reboot.
The .local server worked ok after I put in an entry for it in /etc/hosts with an additivitet line for the IPv6 loopback for the same server. After a reboot, that is.
When I tried commenting out the IPv6 loopback, the server wasn’t working anymore, and the connection was OK with the loopback entry activated again. (with reboots in between).
Hope this is fixed in 10.7.2, I’ll be testing with the beta.
Ah that’s great, thanks Anonymous and nekio – I can also confirm that this works:
Maybe worth an update to the article?
I’ve updated the post, thanks for all the suggestions guys!
Having the IPv4 entries BEFORE the IPv6 entries made it work for me. no need to install anything, just switching some lines to the beginning of the file.
Thank you thank you thank you!
The multiple-entries-per-line in /etc/hosts was my issue. Although I was unable to ping localhost, browsers would resolve it in about 5.01 seconds. I never thought to change the format of my hosts file.
I too was banging my head to get this resolved.
You solution looked too complicated, …
AND I found a MUCH better solution.
Use Chales Web Debugging Proxy
Under Tools Select DNS Spoofing
Select Enabled DNS Spoofing
And ADD a new entry
Host Name and IP address.
I recommend when others google this problem that you add a note about charles in there.
Thanks for this great post but I still have some serious problems with the hosts file. My hosts file is like following;
I can’t manage http://first.localhost/ to work. http://second.localhost/ works but http://first.localhost/ doesn’t work. If I delete the last two lines (which corresponds to SECOND) than http://first.localhost/ works.
How can I add more than one local site in my hosts file? Btw I’m using Lion 10.7.2 which I upgraded today and all those problems arose after that upgrade.
Ghost (or using dscl) does not help when in this situation. All of my DNS entries were made using ghost and I am still having the same issue as the author.
Well, I used the solution provided by FabD (dscl command) and my life was happy and fabulous for some weeks. The resolution of local domains was absolutely immediate. Until today, that I updated to 10.7.2. All of the sudden, all local domains are unreachable, even ping fails miserably (something like “cannot resolve testdomain.dev: Unknown host”). I tried deleting and re-adding the domains, even installed the ghost gem, with no luck. Any help or advice?
Thank you, Justin. I’ve spent a week trying to restore my local environment after 10.7.2 update. And finally I’ve found a solution. You and DNSMasq saved my mind. Thanks again.
Thanks for a very thorough examination and a lot of detective work. This was linked on the Apache Friends Forum and hopefully other developers will find it.
What WAS on Apple’s mind anyway???
In my case, changing line ending characters in /etc/hosts helped. All i had to do was [perl 1=”-e” 2=”'s/
/g'” 3=”/etc/hosts” language=”-pi~”][/perl] and everything started to work again.
Site stripped my script… For all lazy developers out there: perl -pi~ -e ‘s/rn?/n/g’ “/etc/hosts”
Thanks a lot for this detailed tutorial. It really did put me on the track.
I’d like to add a few notes for users who might have experienced further problems, even with dnsmasq properly setup (as per your instructions)
I did upgrade to Lion from SL, not doing a “clean install”. And as some other users have experienced weird behaviors ensued. Such as ‘dig’, ‘nslookup’, ‘host’ commands giving proper DNSmasq redirections, as specified in your /etc/hosts and/or /usr/local/etc/dnsmask.conf (place where dnsmasq puts its config if you install it with homebrew instead of macports). On the other hand, ‘ssh’, ‘ping’, ‘curl’ do not. They just plainly ignored the DNS server on 127.0.0.1
Reason is: ‘dig’, ‘nslookup’ and other BSD tools will read from /etc/resolv.conf whereas some others like ‘ping’ or ‘ssh’ are apple-i-fied and will always query MDNSResponder first. And MDNSResponder will do what it now does, not give precedence to your /etc/hosts settings
The workaround (which worked for me, after hours of googling) is to add a key/value pair to ‘com.apple.mDNSResponder.plist’ (StrictUnicastOrdering bool true) to MAKE the bloody MDNSResponder actually check 127.0.0.1 first.
Short solution : (backup your ‘com.apple.mDNSResponder.plist’ before!!)
$ sudo /usr/libexec/PlistBuddy -c “Add :StrictUnicastOrdering bool true” /System/Library/LaunchDaemons/com.apple.mDNSResponder.plist
and reverse it by running:
$ sudo /usr/libexec/PlistBuddy -c “Delete :StrictUnicastOrdering” com.apple.mDNSResponder.plist
Long answer here :http://superuser.com/questions/84144/how-is-dns-used-by-individual-processes#answer-84178
Even longer answer : http://discussions.apple.com/message.jspa?messageID=10145359
I’m not a DNS pro, so please feel free to correct any approximations I could have made in this answer !
I was experiencing the same issues as discussed. I managed to work with AppleCare by some stroke of luck, and the Apple engineers determined that my hosts file was corrupted–although the file did not appear corrupted as I was able to cat, less, and otherwise edit the file. Upon deleting and putting a new hosts file in place, everything started working as expected once again. I suspect the upgrade from Snow Leopard to Lion caused the file corruption, but I have no concrete evidence of same.
I can confirm Roberts response, I was experiencing the same issue as the author, am using an SL to Lion upgraded iMac, and showed the same order of services in scutil.
The only thing I needed to do to fix the issue was to cat /etc/hosts, copy it into my paste buffer, delete and recreate the file (from the paste buffer), save it, and run dscacheutil -flushcache.
Lion now respects my hosts file entries. I DID NOT have to change my plist.
I had the same problem with lines ending by CR/LF (Windows style). I saved the file with a LF line ending (Unix / OS X style) and that solved the problem.
To stop dnsmasq, just unload the Launch Daemon:
this will not survive a reboot:
sudo launchctl unload /Library/LaunchDaemons/org.macports.dnsmasq.plist
this will keep dnsmasq shutdown through reboots
sudo launchctl unload -w /Library/LaunchDaemons/org.macports.dnsmasq.plist
this will reload dnsmasq (not sure if this survives a reboot)
sudo launchctl load /Library/LaunchDaemons/org.macports.dnsmasq.plist
this will reload dnsmasq permanently
sudo launchctl load -w /Library/LaunchDaemons/org.macports.dnsmasq.plist
Hm, I find (Lion 10.7.2, came installed on a mid-2011 Mini, though I’ve had to reinstall it four times already)
# Host Database
# localhost is used to configure the loopback interface
# when the system is booting. Do not change this entry.
so this is a holdover from Snow Leopard if I understand the above; not sure i do.
hm: from http://code.google.com/p/gmask/issues/detail?id=39
Comment 11 by robbyd…@gmail.com, Nov 23, 2011
The problem for me was actually the encoding of the host files that Gas Mask was saving. I was having the issue where no matter what I put in the file via the Gas Mask editor, nothing changed. It turned out that the hosts file that was being saved was Unicode (UTF 8). When I re-encoded in TextWrangler to Western (Mac OS Roman), it worked just fine. Perhaps Gas Mask could be changed to write files in Western instead of Unicode.
I’m essentially having the reverse problem. I run DNS servers that provide lookups for .local addresses. the idea is to have the response be the “closest” address. so at home, lookups for $homeserver are returned on $home-lan (192.168.1.0/24) but lookups for my shell server are returned on $vpn-lan. the problem is my laptop can be both. if I use /etc/hosts, all traffic for $homeserver is routed over the VPN all the time, even when I’m plugged into the same switch. this results in a huge performance drop! my goal is to have DNSmasq (or resolv.conf) queried first and then fall back to /etc/hosts, but this doesnt seem to be the case.
Thanks for the post.
Made leap from linux to mac dev environment and never would have thought using .local would bring my dev work to a painful snail’s pace 😦 Actually started dreading hitting refresh on my browser to check my code changes.
Anyone have this problem?
%> host cm-11
cm-11.foo.com has address 192.168.199.71
%> ssh cm-11
ssh: Could not resolve hostname cm-11: nodename nor servname provided, or not known
host, dig, and nslookup all work, but ssh, ping, ftp etc fail name lookups.
I’m using dnsmasq, network is set to use 127.0.0.1:
%> scutil –dns
search domain : foo.com
nameserver : 127.0.0.1
Your final update about the line endings on the hosts file helped me. I wonder what could have changed it. I have been using Lion for a while then suddenly it changed sometime after I connected to my school’s WPA2 Enterprise network. Breaking the hosts file broke my localhost hostname which broke my ability to debug Java in Eclipse with messages like “Invalid memory access of location 0x14 rip=” + some pointer. Thanks for the help.
Small note to anyone who’s having issues with MAMP and multiple local domains. I managed to get one local domain to work, but all domains listed after it in /etc/hosts would resolve to the first one. E.g.:
example2.dev and example3.dev would resolve to example1.dev. Turns out I hadn’t enable name-based virtual hosting for MAMP’s port 8888 (I only had the default port 80). The fix was adding this to my httpd-vhosts.conf file:
I found that it was necessary to force quit ‘named’. (perhaps i had BIND running on my machine??)
Everything seems to be working for me with one exception –
I want DNSMasq to send requests up stream based on my resolve.conf file.
This is because I work from several different locations and need to use different dns servers depending upon which location i’m at.
my solution for now –
my machine looks at the hosts file but devices on sharing are hitting Dnsmasq.