Saturday, January 29, 2011

Inconsistent DNS resolution in IOS

In a comment to one of Greg Ferro's posts, Jon Still pointed out that IOS only resolves hostnames used in it's configuration file once:  At the instant the configuration is applied.

It turns out that it's not so straightforward.  The behavior depends on the IOS version.

Jon and I stumbled across the same behavior, and for the same reason:  We both tried to use pool.ntp.org in an IOS configuration.  Jon and I both went on to illustrate this behavior to someone else, and this is where our experiences diverged:  Jon got the behavior he was expecting in his later demonstration (on Greg's blog), and I had the IOS contradict me (on a different router).

Jon's experience looks like this:

router(config)#ntp server pool.ntp.org
Translating "pool.ntp.org"...domain server (8.8.8.8) [OK]


router(config)#do sho run | inclu ntp server
ntp server 72.18.205.157
router(config)#

Here we can see that when the 'ntp server' configuration was added to the router, it immediately did a lookup against google's DNS server at 8.8.8.8, and put the resulting IP address into the running configuration.

This is kind of a bummer, because you can't rely on any particular pool.ntp.org server to be available long term, so you might like the router to have the option to go back to DNS when required.  When exactly it goes back is a whole other story - NTP is complicated.

Here's what happened when I was trying to illustrate the issue:
router(config)#ntp server pool.ntp.org
Translating "poo.ntp.org"...domain server (8.8.8.8)

router(config)#do sho run | inclu ntp server
ntp server pool.ntp.org
router(config)#

Huh!  So, this time, the NTP hostname remained in the configuration.  The difference?  The first router is running 12.4(25d), while the second is running 12.4(15r).

Now, fortunately for the sane operation of NTP, the NTP process caches the IP address of the server to which it's associated, even though the running configuration has not.



But things get even weirder when we decide later to remove the configuration:
router(config)#no ntp server pool.ntp.org
Translating "pool.ntp.org"...domain server (8.8.8.8) [OK]

%NTP: unrecognized peer
router(config)#do sho run | inclu ntp server
ntp server pool.ntp.org
router(config)#

Huh.  Why can't I remove the configuration?

router(config)#do ping pool.ntp.org

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 169.229.70.64, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)
router(config)#do sho ntp ass

  address         ref clock       st   when   poll reach  delay  offset   disp
*~216.93.242.12   137.146.28.85    2     29    128   177 18.484  -5.244 65.191
 * sys.peer, # selected, + candidate, - outlyer, x falseticker, ~ configured
router(config)#


Apparently the no ntp server pool.ntp.org command initiated a new lookup, and the IP address associated with that name record has changed.  The record currently resolves to 169.229.70.64, but the NTP process is associated with a previously resolved IP of 216.93.242.12.

router(config)#no ntp server 216.93.242.12
router(config)#do sho run | inclu ntp server
router(config)#


Removing the NTP server by IP address removed both the association from the NTP process and the configuration line used to add the association (by name), even though the two are no longer linked anywhere except in the memory of the IOS NTP process.

8 comments:

  1. Weird. Not sure why simple name resolution is such a problem for IOS.

    Good article!

    ReplyDelete
  2. Thanks for the compliment, Stretch.

    The inability to remove an existing line from the configuration really surprised me. Other than that, I'd say that the newer IOS is doing everything reasonably.

    ReplyDelete
  3. Great post Chris - thanks for following up on my original comments as I'd somewhat naively assumed that what I was seeing was the same everywhere!

    It's interesting that the earlier version has slightly more useful (but inconsistent) behaviour, while in the later version they've seemingly given up on putting names in the config.

    ReplyDelete
  4. From my (unreliable) memory, the default action for DNS names is that IOS will perform a lookup every four hours so as to update the entry.

    Note that this is suitable for things like NTP, and no so great for ACLs where a server failure using DNS to change IP address means you would have to wait up to four hours for failover to occur.

    ReplyDelete
  5. Great article man! Got same problem and you resolved it! Hapy new year 2012!

    ReplyDelete
  6. Hi,

    stumbled across the same issue and resolved it within one minute, thanks to your great post!

    Thanks a lot!

    ReplyDelete
  7. Thanks, it helps me !

    ReplyDelete
  8. Very good article!!!

    ReplyDelete