What did West Virginia officials buy exactly?

The state of West Virginia has landed in the headlines a couple of times recently due to their purchase of wildly inappropriate Cisco routers for over 1000 locations around the state. There have been no shortage of commenters laying the blame at Cisco's feet, but the auditor's report disagrees. It's back in the headlines now because of the recent release of the report, and because of Cisco's pledge to buy the gear back and help the state buy more appropriate equipment.

The auditor's report is an interesting read. Here are some of the highlights from my reading of it:

3945 routers were selected because of two key requirements communicated by the state:

• Dual internal power supplies, which narrows the selection to 3925 and 3945 models
• Use of two service module slots on day one, and a desire to have the option to grow beyond these two at some point in the future.

Unfortunately, these two requirements seem to have been communicated in a meeting, because no emails were available to corroborate them, but members of the team who purchased the gear acknowledge that the requirement did originate with the state. So, that's how they wound up in a 3945 chassis.

No analysis was done to understand the user base or  network requirements of any of the sites. In one interesting example, 77 of these routers were delivered to the State Police, who have only made use of 2 of them. For the most part, the State Police, which is one of the bigger network users in the program, is getting by on smaller routers from a previous generation of Cisco Integrated Service Routers. The two routers that have been deployed by the WVSP were upgraded to meet the WVSP network's requirements. The rest languish in storage.

There's at least one case where one of the routers seems to have cost more than the buiding into which it's been deployed:

There's a $20,000 router in here

The State's Broadband Technology Opportunity Program (BTOP) Grant Implementation Team

 (GIT) decided that rather than doing any analysis, they'd deploy the same device everywhere. So, instead of sending a $300-$400 router into that one room library, they sent the following to every site:

• An $8000 router bundle (3945). About 20% of this amount is for unused voice features.
• $750 in power supply upgrades. Also voice related.
• $7200 for five years of 8x5xNBD support
• A $500 "Data" routing license. Not completely out of line, but probably not required and would have cost less on a smaller router.
• A $2600 internal 16-port layer 3 switch. Why L3 feature set on the switch? Crazy.
• $1000 dual T1 card
• $1600 Network Analysis Module (sniffer type thing - unused, I'm sure)

According to the auditor's report, the state's Chief Technology Officer said that her team "decided to have all routers identically equipped." Frankly, I think this is the most damning phrase in the whole report.

How some have come to the conclusion that this debacle is Cisco's fault, I can't imagine. Anyone who has ever participated in a bid RFQ/RFP process knows that bidders don't have the opportunity to shape the requirements much. I've many times wanted clarification on some seemingly contradictory requirements, or to point out that a little design change could reduce the cost dramatically, but the opportunity to have these discussions just doesn't exist in these formal processes, especially when dealing with government agencies.

The State's Auditor clearly lays the blame at the feet of the folks who made the purchase, not the vendor:

the ultimate cause of the state purchasing inappropriately sized routers is that neither a capacity study nor a user need study was conducted.

Having said that, the auditor also said:

The Legislative Auditor believes that the Cisco sales representatives and engineers had a moral responsibility to propose a plan which reasonably complied with Cisco’s own engineering standards. It is the opinion of the Legislative Auditor that the Cisco representatives showed a wanton indifference to the interests of the public in recommending using $24 million of public funds to purchase 1,164 Cisco model 3945 branch routers. 

Moral responsibility? Wanton indeference? Maybe, but these sort of qualitative judgements aren't too useful when it comes to matters of business in my opinion. Furthermore, Cisco wasn't in a position to recommend an alternative platform because no analysis of the actual requirements had been completed. The fact remains that the requirements which led to the selection of both the platform and the options installed were chosen by members of the GIT, and Cisco wasn't in a position to challenge them.

If you've read more than a few of my blog posts, you'll know that I'm passionate about understanding the Cisco product catalog and the ways to get the most from it. Unfortunately, I've over-spent my customers money (at their insistence!) enough times that I would probably have protested a little and then wound up making that same sale, though the fact that we're talking about public funds might have elicited more protest from me than usual. Unfortunately, I've learned that you can't talk fiscal responsibility into many people if (a) they're spending their employer's money and (b) responsibility would require them to do work.

So, what did they buy exactly? Working backward from the purchase order, I've reconstructed what the configset probably looked like.