Thursday, August 12, 2010

Reverse Telnet by IP address

There's lots of pages out there detailing how to use your 2511 router or NM-16A/NM-32A for reverse telnet (ugh, I don't care for that term) duty.

Generally they end with each serial interface being associated with a TCP port on the router.   The TCP port for each interface is the interface's "line number" + 2000.

So, if your NM-16A represents lines 33-48, you'd be able to access the console attached to the 3rd serial interface by with "telnet <router> 2035"

While this is adequate for most labs where things get moved around a lot, I find it's difficult to keep track of these port numbers in a long-term deployment.  Especially when there are multiple routers doing this job:  The overlap created by multiple devices means you can't just map the device names in your services (files/NIS/LDAP) database.

My solution is to map each interface to a separate IP address, then create DNS records for every one of them.  Actually, two DNS records for every interface:  A permanent "A" record that references the device and port (, and a "CNAME" record that references the managed device's console port (switch3-console).  The CNAME points at the A record.

Accessing each serial line by a unique IP address makes this possible, but it requires some address translation.  Here's the relevant configuration, assuming you've already got reverse telnet working:

Configure a loopback interface with a netmask roomy enough for all of your serial interface IPs:

interface Loopback0
 ip address
 ip nat inside
 ip ospf network point-to-point

This interface is "inside" from a NAT perspective.  The point-to-point line is required because without it, OSPF always advertises loopback interfaces with a 32-bit mask.  Skin that cat however you see fit.

Next, set your Ethernet interface for NAT outside:

interface Ethernet0
 ip nat outside

Now, setup the NAT rules to forward incoming connections.  The connections arrive on unique IPs, all port 23 (no ssh support on my router), and are forwarded to the loopback interface on the unique per-line TCP port:

ip nat inside source static tcp 2033 23
ip nat inside source static tcp 2034 23
ip nat inside source static tcp 2035 23
ip nat inside source static tcp 2036 23
ip nat inside source static tcp 2037 23
ip nat inside source static tcp 2038 23
ip nat inside source static tcp 2039 23
ip nat inside source static tcp 2040 23
ip nat inside source static tcp 2041 23
ip nat inside source static tcp 2042 23
ip nat inside source static tcp 2043 23
ip nat inside source static tcp 2044 23
ip nat inside source static tcp 2045 23
ip nat inside source static tcp 2046 23
ip nat inside source static tcp 2047 23
ip nat inside source static tcp 2048 23

To connect to the 5th serial interface, I'd telnet to

Sunday, August 1, 2010

TFTP Oddities

Like RS-232, TFTP is less and less used these days, but remains an absolutely critical protocol for network engineers.

Here I'll present a couple of TFTP implementation oddities that sometimes break TFTP.

TFTP Uses Ephemeral Ports
The bulk of the data flow between a TFTP client and server uses ephemeral ports on both ends.  This is strange behavior.  Most applications use ephemeral ports on the client side, but the server responds from its well known port number.  Here's an example of a normal application (a DNS query):
13:12:13.497988 IP >  54600+ A? (32)
13:12:13.583574 IP >  54600 5/4/4 CNAME,[|domain]
The client originated the query from an ephemeral port (56119), and sent it to the server's well known port (53).  The server's response was symmetric: 53 -> 56119.

TFTP servers don't generally behave this way.  Consider this TFTP transaction:

12:55:33.188470 IP
34617 >  22 RRQ "myfile.dat" netascii
12:55:33.189962 IP
42829 > UDP, length 516
12:55:33.190060 IP
34617 > UDP, length 4
12:55:33.190088 IP
42829 > UDP, length 92
12:55:33.190190 IP
34617 > UDP, length 4
The client originated the transaction from an ephemeral port (34617), and sent it to the server's well known port (69).  But the server's response was NOT symmetric.  Rather than replying from port 69, the server spawned a new ephemeral port (42829) when replying to the client.  After the initial request packet, the four packets responsible for the file transfer (data, ack, data, ack) were exchanged between two ephemeral ports.

This is strange behavior, and most flow-mangling smart devices (firewalls, NAT engines, reflexive ACLs, etc...) won't recognize the second or later packets as being related to the first.  Unlike active mode FTP, TFTP doesn't include an application header describing what's going to happen next.  Accordingly, there's nothing for a protocol-aware firewall module to monitor in order to open a pinhole for return traffic.  TFTP is a perfect candidate for a man-in-the-middle attack.

I'm not aware of any requirement for TFTP servers to act this way, but most do.  My guess about the reason for the behavior is this:  As shown here, the TFTP server merely needs to fork() itself for each client request.  The new thread opens its own socket to handle the client request, then exits when done.  Without the separate thread and socket, a single server thread would have to keep track of all clients simultaneously, and would have to demultiplex incoming client packets for each flow (because they'd all flow in the single connectionless UDP socket).

TFTP Supports Broadcast Servers
Sort of.
I expect lots of people to run into this issue as enterprises upgrade their Cisco wireless environments to 802.11N, the old gear floods onto eBay, and people attempt to load the autonomous AP code.  There's a peculiarity with the TFTP implementation on the Cisco Aironet autonomous-capable access points.  Consider my AIR-AP1231G-A-K9 trying to download its IOS software from the TFTP server.  The AP's console reports:

process_config_recovery: set IP address and config to default
process_config_recovery: image recovery
image_recovery: Download default IOS tar image tftp://
examining image...
extracting info (274 bytes)
Premature end of tar file
ERROR: Image is not a valid IOS image archive.Loading "flash:/c1200-blah-blah-blah

What's going on here?  I'm not the first to notice this problem, but I haven't seen an explanation anywhere yet.  First, notice that the client is hitting its TFTP server at  Cisco's documentation of this process directs that the TFTP server be configured with an address between and  That's because the AP will be running with, and it doesn't know exactly where server will be, so it hits the all-ones address.  Why we're not supposed to use, I don't know.  On the wire we get this:

12:50:54.498194 IP >  31 RRQ "c1200-k9w7-tar.default" octet
Next, the server unicasts the first chunk of the IOS image to the AP (ephemeral port to ephemeral port again):
12:50:54.547794 IP > UDP, length 516
So far, so good, but we're getting to the point where things come off the rails.  The AP, having just received a packet from, should know the server's address.  But it sends the TFTP acknowledgement for the first chunk of the file to the broadcast address, not directly to the server:
12:50:54.606812 IP > UDP, length 4
The TFTP server doesn't hear these ACKs, and both ends endlessly retransmit their last packet.  I'm not clear on whether this should be reasonably expected to work, but...
  • It flies in the face of the thread-per-client model which I believe is behind the port symmetry issue
  • My antique Xyplex terminal servers use broadcast mode TFTP, and get the replies right
  • It doesn't work
Here's the whole exchange, with retransmissions tagged in red:

12:50:54.498194 IP >  31 RRQ "c1200-k9w7-tar.default" octet
12:50:54.547794 IP > UDP, length 516
12:50:54.606812 IP > UDP, length 4
12:50:55.543296 IP > UDP, length 516
12:50:57.543421 IP > UDP, length 516
12:51:01.543679 IP > UDP, length 516
12:51:01.544614 IP > UDP, length 4
12:51:06.626855 IP > UDP, length 4
12:51:09.544176 IP > UDP, length 516
12:51:14.627345 IP > UDP, length 4
In 20 seconds, the first block of the file got sent 5 times, and was (poorly) acknowledged 4 times, before both ends gave up.

That's what happends with tftp-hpa 0.49, anyway.  Philippe Jounin's TFTPD32 v3.33 doesn't seem to mind this sort of thing:

13:41:02.083839 IP >  31 RRQ "c1200-k9w7-tar.default" octet
13:41:02.139137 IP > UDP, length 516
13:41:02.197885 IP > UDP, length 4
13:41:02.198033 IP > UDP, length 516
13:41:02.211902 IP > UDP, length 4
13:41:02.211987 IP > UDP, length 516
13:41:02.213473 IP > UDP, length 4
So, when installing new code on your Aironet APs, be careful which TFTP daemon you use.