Wednesday, September 21, 2016

Cisco Debug Persists Through Reboot

Normal boot time messages from a C881 router look something like this:
 System Bootstrap, Version 15.4(1r)T, RELEASE SOFTWARE (fc1)  
 Technical Support: http://www.cisco.com/techsupport  
 Copyright (c) 2013 by cisco Systems, Inc.  
   
 Total memory size = 1024 MB  
 C881-K9      platform with 1048576 Kbytes of main memory  
 Main memory is configured to 32 bit mode   
   
 Readonly ROMMON initialized  
   
   
 IOS Image Load Test   
 ___________________   
 Digitally Signed Production Software   
 Self decompressing the image : ###<snip>### [OK]  
   

But there's one router in the fleet which does this instead:
 System Bootstrap, Version 15.4(1r)T, RELEASE SOFTWARE (fc1)  
 Technical Support: http://www.cisco.com/techsupport  
 Copyright (c) 2013 by cisco Systems, Inc.  
   
 Total memory size = 1024 MB  
 C881-K9      platform with 1048576 Kbytes of main memory  
 Main memory is configured to 32 bit mode  
   
 Readonly ROMMON initialized  
 Using monlib version 2  
 Using version info 2  
   
  dfs_openfile: Opening file.....  
  dfs_openfile: Opened file / with fib = 4019e5c  
 Reading cluster = 126, offset = 0, nsecs = 8  
 Reading cluster = 133, offset = 0, nsecs = 8  
 Reading cluster = 17013, offset = 0, nsecs = 8  
 Reading cluster = 17458, offset = 0, nsecs = 8  
 Reading cluster = 18056, offset = 0, nsecs = 8  
 Reading cluster = 17053, offset = 0, nsecs = 8  
  dfs_closefile: Closing file.... 4019e5c  
  dfs_closehandle: Closed file.... 4019e5cUsing monlib version 2  
 Using version info 2  
   
  dfs_openfile: Opening file....c800-universalk9-mz.SPA.154-3.M4.bin  
  dfs_closefile: Closing file.... 4019e5c  
  dfs_closehandle: Closed file.... 4019e5c  
  dfs_openfile: Opened file /c800-universalk9-mz.SPA.154-3.M4.bin with fib = 4019e5c  
 Reading cluster = 17053, offset = 0, nsecs = 8  
   
 Reading cluster = 17053, offset = 0, nsecs = 128  
 Reading cluster = 17069, offset = 0, nsecs = 128  
 Reading cluster = 17085, offset = 0, nsecs = 128  
 <--- ~1300 lines removed --->  
 Reading cluster = 58703, offset = 0, nsecs = 128  
 Reading cluster = 58719, offset = 0, nsecs = 80  
 IOS Image Load Test  
 ___________________  
 Digitally Signed Production Software  
 Self decompressing the image : ###<snip>### [OK]  

Um. Okay. What's going on here?

It turns out that there are a couple of funny things about debugging filesystem operations:
  1. You can't see that it's turned on from within IOS:
  2. C881#debug filesystem flash:
    C881#show debugging
    
    
    
    
    C881#
    

  3. The setting persists across both reboots and power interruptions. Apparently this directive sets a hardware flag somewhere. no debug filesystem flash: made those messages go away.
The setting probably applies to any storage device available from within the ROMMON:
 rommon 1 > dev  
 Devices in device table:  
     id name  
   flash: compact flash         
 bootflash: boot flash           
 usbflash0: usbflash0           

This was new to me.