Tuesday, July 19, 2011

OSPF: ABR as a Black Hole

Area Border Routers in an OSPF topology sometimes make promises they can't keep.  These promises come in the form of summary route advertisements (an advertised route is a promise to deliver packets, after all), and they're usually not a problem.

But sometimes they are a problem, and in some topologies they can result in packet loss that's sustained for several seconds.  Most discussions of packet loss and reconvergence in modern networks concern themselves with detection and correction of node or link failures.  Surprisingly, node recovery can be worse than the failure!  Here's the scenario:

R2 and R3 announce a summary default route into Area 1, rather than announcing R1's external prefixes.  They summarize because area 1 is a stub area, and that's just how it works.  ABRs for a stub area unconditionally inject a default route into the area.

If one of the ABRs fails, that's no problem.  R4 will detect the failure quickly (for some values of quickly), and redirect all traffic to R3.

But what happens when the ABR comes back online?  It will introduce a default route into area 1 unconditionally.  Even if it hasn't fully converged on the area 0 side.  Not good.

I labbed this up to see what would happen.  R1 is introducing 500 external prefixes (500 loopback interfaces with 'redistribute connected'), and R4 is pinging one of those prefixes.  I chose a prefix that CEF will hash to the path via R2 when both paths are available.

R4#debug ip routing
IP routing debugging is on
Protocol [ip]:
Target IP address:
Repeat count [5]: 100000
Datagram size [100]:
Timeout in seconds [2]:
Extended commands [n]:
Sweep range of sizes [n]:
Type escape sequence to abort.
Sending 100000, 100-byte ICMP Echos to, timeout is 2 seconds:

R2 was reloaded and a single ping was lost as R4 reconverged.  Not bad.  The outage duration was somewhere less than the 2 second ping timeout.  Debug output on the following lines shows the interface failure, OSPF neighbor loss and removal of the bad route.

*Mar  1 08:20:15.403: RT: del via, ospf metric [110/2]
*Mar  1 08:20:15.403: RT: NET-RED
*Mar  1 08:20:15.539: %OSPF-5-ADJCHG: Process 1, Nbr on Serial3/3 from FULL to DOWN, Neighbor Down: Dead timer expired
*Mar  1 08:20:15.571: %LINK-3-UPDOWN: Interface Serial3/3, changed state to down
*Mar  1 08:20:15.571: RT: is_up: Serial3/3 0 state: 0 sub state: 1 line: 0 has_route: True
*Mar  1 08:20:15.571: RT: interface Serial3/3 removed from routing table
*Mar  1 08:20:15.571: RT: del via, connected metric [0/0]
*Mar  1 08:20:15.571: RT: delete subnet route to
*Mar  1 08:20:15.575: RT: NET-RED
*Mar  1 08:20:16.571: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial3/3, changed state to down
*Mar  1 08:20:16.571: RT: is_up: Serial3/3 0 state: 0 sub state: 1 line: 0 has_route: False

Now R2 is coming back up.  R4's debugs show the interface come online and the addition of the connected route:
*Mar  1 08:22:43.463: %LINK-3-UPDOWN: Interface Serial3/3, changed state to up
*Mar  1 08:22:43.463: RT: is_up: Serial3/3 1 state: 4 sub state: 1 line: 0 has_route: False
*Mar  1 08:22:43.463: RT: SET_LAST_RDB for NEW rdb: is directly connected
*Mar  1 08:22:43.467: RT: add via, connected metric [0/0]
*Mar  1 08:22:43.467: RT: NET-RED
*Mar  1 08:22:43.467: RT: interface Serial3/3 added to routing table
*Mar  1 08:22:43.467: RT: is_up: Serial3/3 1 state: 4 sub state: 1 line: 0 has_route: True
*Mar  1 08:22:44.463: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial3/3, changed state to up
*Mar  1 08:22:44.463: RT: is_up: Serial3/3 1 state: 4 sub state: 1 line: 0 has_route: True

And the following debugs show R4's OSPF neighbor come back up, and the installation of the default route via R2.
*Mar  1 08:22:53.395: %OSPF-5-ADJCHG: Process 1, Nbr on Serial3/3 from LOADING to FULL, Loading Done
*Mar  1 08:22:56.823: RT: NET-RED
*Mar  1 08:22:58.967: RT: add via, ospf metric [110/2]
*Mar  1 08:22:58.967: RT: NET-RED
*Mar  1 08:22:58.971: RT: NET-RED
Immediately after the addition of the default route at 08:22:58.967, pings began failing.  The U indicates that R2 dropped our ping, rather than delivering it to R1 and then sent us an ICMP unreachable message to let us know what happened.  R2 has promised to deliver packets to anywhere, but has not yet learned the route to (a loopback on R1).  10 pings time out over about 20 seconds before connectivity is restored:
Success rate is 99 percent (3204/3216), round-trip min/avg/max = 52/58/108 ms

The final lost ping happened because I killed off the ping process mid-ping.

Nifty, huh?  Reconvergence around a failed router happens in under two seconds, but reconvergence through a recovered router takes 20 seconds.

I'm not aware of a way to configure around this behavior in IOS.  Conditional default advertisement, or a stub gateway delay option would be nice, but I don't know how to configure such a thing in IOS.  Do you know how to prevent this problem?  Please let me know!