Friday, October 15, 2010

You CAN connect a switch to a Nexus Fabric Extender

I didn't mention it in the data center design writeup, but it is possible to connect a switch to a Nexus fabric extender.

Search queries that have led readers here, conversations with customers, and various blog posts, have made it clear that people are running full-speed into an interesting feature of the Cisco Nexus Fabric Extenders:  FEX ports always run bpduguard.  You can't turn it off.

For the uninitiated, Bridge Protocol Data Units (BPDUs) are the packets used by bridges (switches) to build a loop-free topology.  BPDU Guard is a Cisco interface feature that immediately disables an interface if a BPDU arrives there.  It's appropriate on interfaces where you never expect to plug in a switch.  If you've ever plugged a switch into your cubicle jack at work, you may have experienced this feature firsthand.

BPDU Guard tends to go hand-in-hand with portfast, a feature that makes switch interfaces ready for use as soon as they link up, instead of forcing these fresh links to jump through the Spanning Tree Protocol (STP) loop prevention hoops.

If you're going to use portfast, you must use bpduguard.

Nexus Fabric Extenders (2148, 2248, 2232) run bpduguard on their interfaces all the time.  It can't be disabled.

Who cares?
Bpdugaurd means that you can't hang a switch from a FEX.  If you've adopted Cisco's vision of the modern data center, this can become a problem because the 2148 fabric extenders can only do gigabit.  No 10/100 capability here.  It turns out there's a lot of stuff in a modern data center that still can't do gigabit, but lives out in the general population that's intended to be served by the Fabric Extenders:
  • HP server iLO interfaces
  • Terminal server appliances
  • Power strips
  • Environmental monitors
  • KVM equipment
The best option for these small, far-flung clients might be the installation of a small, 10/100 capable switch nearby.  I selected the WS-C2960-24TC-S for this purpose in a recent build because it's super cheap (list price is $725) and because it has dual-purpose uplink ports.  The natural inclination is to try to uplink directly into the nearby 2148T fabric extender.  But, as soon as you do, the 2960 sends a BPDU, and the 2148 shuts down the interface.

Now what?
You could disable spanning-tree on the 2960, but that's asking for trouble, and makes redundancy impossible.  You could disable spanning tree on just the uplink interface because it's safer, but still doesn't accomplish redundancy.  You could link the small switch directly to the distribution layer, but that's a lot of buck for relatively little bang.

Another possible answer is flexlinks, a long-forgotten uplink redundancy mechanism that probably predates stable STP operation.  I had assumed so, but it doesn't look like flex links is quite that old.  I don't know why this feature was introduced, but it's useful here.

Enabling flexlinks is a one-liner:  In 'interface GigabitEthernet 0/1 configuration context' do:
switchport backup interface GigabitEthernet 0/2
Now, Gig0/2 backs up Gig0/1.  Spanning tree protocol is disabled on these two ports, so BPDU Guard won't cause a problem, and there's no risk of these ports creating a topology loop because only one one of them will be in forwarding mode.

Plug both interfaces into a fabric extender and you're in business.

The interfaces can be trunks or they can be access ports.  If they're trunks, you can even balance the vlans across the uplinks if you're so inclined (personally, I don't care for it, but this is a very common strategy).

A topology change will flood bogus packets which appear to be sourced from your client systems so that upstream switches update their forwarding tables.  ...A process that can be made even quicker with the 'swichport backup mmu' mechanism - but I doubt the Nexus supports that anyway.

Going forward:
100Mb/s is really the only real requirement.  I can only think of two devices that are limited to only 10Mb/s in any of the networks I work on.  And both of those are in my basement, which has not yet been migrated to the Nexus platform.  Fortunately, the Nexus 2248 Fabric Extenders can do 100/1000, so you'll be much less inclined to try to hang a switch off of them.  Plus they're cheaper, and have a few other benefits over the 2148T.  As far as I'm aware, there's not a single technical reason to prefer a 2148 over the 2248, so stop buying 2148s.


  1. The real problem is when we try to hang a packetshaper, HP blade chassis (with non-cisco switches installed), some phone systems, and a variety of other devices that send BPDUs, which CANT be disabled. After buying and installing 10 of the 2248s, we ran into a brick wall with this "feature". Most likely, we'll try to trade the 2248s on the used market (even tho they've only been used 5 hours), and replace them all with 2960S w/10g uplinks, keeping the nexus 5548.
    Its silly things like this that keeps me asking why I bother running Cisco equipment.

    1. Just got this from Cisco:
      Brian, yes you can connect a switch to a 2248 FEX. What you have to be careful is in that case to not introduce a loop in the network as there is no spanning-tree running on the FEX. So if you're purpose is to add a switch to connect hosts that should be okay, however if you introduce a loop, it will take your network down. To get the port up on the FEX please use spanning-tree bpdufiler command. It is not best practice and we don't recommend it, however you can achieve this.

    2. Hey Brian,

      Flexlinks will be safer because it makes it less likely that you'll form a loop, and still allows you to connect the downstream switch in a redundant fashion.

  2. If you configure the nexus switch port with "spanning-tree port type edge" it'll ignore the BPDUs being sent from the connected switch or other device.

    Doesn't allow you to have redundancy, but at least it'll get ya working!

  3. "port type edge" won't solve this problem. It just forces the port to transition into forwarding state immediately.

    BPDUguard can't be disabled (unless something has changed recently)

  4. i too was ready to sell my 2248's when connecting to other equipment, but i can confirm that this works. Here's how:

    inter e101/1/1
    spanning-tree bpdufilter enable

  5. we have HP enclosures with switches behind them,
    we want to terminate each enclosure to 2 different 2248.

    if we disable bpdu filter on fex ports, then are we creating loops?
    whats will be the best design in achieving this.

  6. Lets start with this: What you want to do is dangerous and not supported. 2248 is intended to be the edge device, not aggregate connections from other edge devices (the switches in your HP enclosures).

    BPDUfilter is disabled by default. Some folks *enable* bpdufilter on FEX ports to allow connections of STP speakers to these ports. This is dangerous. Please don't do it. With two links like you're suggesting, you *will* be creating loops.

    What sort of switches are in the HP enclosure? If they're Catalysts, you can use flexlinks.

  7. well they are not Catalysts switches, i dont even hv cli to them...i hv to manage them via HP management gui, which only allows limited configuration such as vlans,ports trunk, access, port-channel.

    & they are copper ports so cant even terminate on 5548up as we dont have GLC-T sfps.

    we had planned to aggregate these enclosures within the rack with TOR 2248 switches.

  8. Aggregating switches to FEXen is not a good idea. Your best bet is to spring for the copper transceivers for your 5548s.

    If you decide to home these things to FEXen anyway, BPDUfilter on the FEX ports will allow it, but you need to be sure you don't form any loops. Be wary of the intra-enclosure links which may be connecting between bays 1 and 2, 3 and 4, etc...

  9. Can someone help me please? I am running a Nexus 5020 extened by a 2148. To the 2148 I want to connect a cathalyst 3550 in order to connect ILOS. If I connect the cathalist to the fabricextender the Light stays off. I already tried on the extender:
    #configure terminal
    #interface eth100/1/48
    #spanning-tree bdufilter enable
    #spanning-tree port edge

    the lights are still offline!
    Please help. THX!!!!

  10. What port are you using on your 3550?

  11. @Chris
    I have just ethernet ports available. So I tried port 1,2 and 48.
    If anyone could give me a minihowto (command on the nexus or if also needed on the cathalyst) I would be very happy.


    1. The only ports on a 3550 which will work for your application are the gigabit ports. This is because your 2148T FEX is gig-only. no 10/100 mode.

      Your 3550 has 48 10/100 ports and two gig ports.

      You're trying to connect a 10/100 port to a gig-only port.

      Put a Cisco WS-G5483 into one of the GBIC slots and you'll be able to connect to the 2148T just fine (save the caveats about BPDU ingress).

  12. Does it mean if I connect multiple agregated ports of the host server like IBM 700 (Power VM) or ESXi with internal virtual switch, I need to use flexlinks.?

  13. If your device produces BPDUs, you will have the problems described here.

    ESXi vSwitch does *not* produce BPDUs as far as I know.

    Aggregated links (802.3ax stuff) are another matter altogether. You'll need to either run LACP (if your device supports it), or configure things manually.

    Make sure you're doing the same things on both ends of these links. Manual is risky.

    If by "aggregated" you mean "vPort pinning" (or whatever vmware calls it), then you just configure the upstream switch as normal access ports, don't need to do anything special.

  14. Thanks Chris, you have saved our life..i have enabled the flex links on the cat switch and it works like a charm!

  15. Ran into this tonight. Its 4 ports each on a different single vlan so no risk of a loop but it still sends BPDU. Old Cisco Local Director, we just need to move it to the new switch temporarily.