While I was working on a Nexus lab, I ran into an interesting issue that had me scratching my head for a while before I managed to figure out what was going on. You gotta love those unintentional troubleshooting exercises! This was the situation:

I had two Nexus 5000 switches with an identical configuration and identical software version, but an interesting difference in behavior. On both switches I had configured a VLAN interface, primarily as a testing point that I could ping to from some other switches. When I actually started pinging to see if the IP addresses on these SVIs were reachable, I noticed that I could ping one of the Nexus 5000 switches, but not the other. While performing the usual Layer 1 and Layer 2 connectivity checks I noticed that the SVI on the Nexus 5000 that I could not ping was down! The relevant configuration on this switch was as follows:

feature interface-vlan
vlan 1,75
interface Vlan75
  no shutdown
  ip address 10.37.37.77/24

However, the interface itself showed the following:

N5K-1# show int vlan 75
Vlan75 is down, line protocol is down
  Hardware is EtherSVI, address is  0005.73c1.9981
  Internet Address is 10.37.37.77/24
  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec

The most common causes for this would be either a missing VLAN, or not having any spanning-tree forwarding ports for the VLAN. However, I had already performed my spanning tree checks and I knew that I had some ports in forwarding state for VLAN 75.

I compared the configuration of the misbehaving switch to the configuration on the other Nexus 5000, but it was identical except for the IP addresses. However, on the other switch the SVI had come up without a problem. A key hint about what was going on was shown in the output of the show interface brief command:

N5K-1# show int vlan 75 brief

-------------------------------------------------------------------------------
Interface Secondary VLAN(Type)                    Status Reason
-------------------------------------------------------------------------------
Vlan75    --                                      down   Non-routable VDC mode

Interestingly, this command gave me a reason why the interface was down, which was not shown in the regular show interface command. Apparently, the SVI was down because the switch was in a “non-routable VDC mode”. This still didn’t really get me anywhere, but at least I had a hint now. A bit of googling turned up the following link on the Cisco support forums:

https://supportforums.cisco.com/thread/2131130

Unfortunately, this post was about the Nexus 7000, not the Nexus 5000, and had to do with the lack of M1 linecards to provide the routing function for a VDC. However, it did yield another piece of the puzzle. By adding the management command to the SVI, I managed to get the interface up:

N5K-1(config)# int vlan 75
N5K-1(config-if)# management
N5K-1(config-if)# show int vlan 75 brief

-------------------------------------------------------------------------------
Interface Secondary VLAN(Type)                    Status Reason
-------------------------------------------------------------------------------
Vlan75    --                                      up     --

So I had found the workaround that would allow me to continue with my lab scenario, but I didn’t find this very satisfying. I could still not explain why one of these two Nexus 5000 switches required the obscure management command, which I had never needed before for this type of configuration, while the other worked as expected. So I started digging for differences between the two switches. As mentioned, the NX-OS version was exactly the same and also the hardware was identical. Both switches were 5548Ps without a Layer 3 module or expansion module installed.

Finally I found the difference. Although neither switch had a Layer 3 module, the misbehaving switch actually had a Layer 3 Base Services license (LAN_BASE_SERVICES_PKG) installed:

N5K-1# show license usage
Feature                      Ins  Lic   Status Expiry Date Comments
                                 Count
--------------------------------------------------------------------------------
FCOE_NPV_PKG                  Yes   -   Unused Never       -
FM_SERVER_PKG                 No    -   Unused             -
ENTERPRISE_PKG                Yes   -   Unused Never       -
FC_FEATURES_PKG               Yes   -   Unused Never       -
VMFEX_FEATURE_PKG             No    -   Unused             -
ENHANCED_LAYER2_PKG           Yes   -   Unused Never       -
LAN_BASE_SERVICES_PKG         Yes   -   Unused Never       -
LAN_ENTERPRISE_SERVICES_PKG   Yes   -   Unused Never       -
--------------------------------------------------------------------------------

The other switch did not have the Layer 3 Base Services package installed:

N5K-2(config-vlan)# show license usage
Feature                      Ins  Lic   Status Expiry Date Comments
                                 Count
--------------------------------------------------------------------------------
FCOE_NPV_PKG                  No    -   Unused             -
FM_SERVER_PKG                 No    -   Unused             -
ENTERPRISE_PKG                No    -   Unused             -
FC_FEATURES_PKG               No    -   Unused             Grace 119D 22H
VMFEX_FEATURE_PKG             No    -   Unused             -
ENHANCED_LAYER2_PKG           Yes   -   Unused Never       -
LAN_BASE_SERVICES_PKG         No    -   Unused             -
LAN_ENTERPRISE_SERVICES_PKG   Yes   -   Unused Never       -
--------------------------------------------------------------------------------

So apparently, my misbehaving switch must have had a Layer 3 daughter card  at some point and somebody had installed the Layer 3 Base license for it, but currently the switch did not have the Layer engine installed anymore. So I had a hunch that this might be the cause of my issue. To confirm this hypothesis I uninstalled the license:

N5K-1# clear license MDS20111121084524878.lic
Clearing license MDS20111121084524878.lic:
SERVER this_host ANY
VENDOR cisco
INCREMENT LAN_BASE_SERVICES_PKG cisco 1.0 permanent uncounted \
        VENDOR_STRING=MDS_SWIFTN55-BAS1K9= \
        HOSTID=VDH=SSI14410JQG \
        NOTICE="201111210845248781 \
        " SIGN=70188100AC20

Do you want to continue? (y/n) y
Clearing license ......2012 Oct  2 17:13:20 N5K-p5-1 %LICMGR-1-LOG_LIC_LICENSE_EXPIRED: Evaluation license expired for feature LAN_BASE_SERVICES_PKG.
done
N5K-1# 2012 Oct  2 17:13:20 N5K-1 %PFMA-2-MOD_REMOVE: Module 3 removed (Serial number JAF1440AMME)

Then I removed the management command from the SVI to see what this would do:

N5K-1(config)# int vlan 75
N5K-1(config-if)# no management
N5K-1(config-if)# show int vlan 75 brief

-------------------------------------------------------------------------------
Interface Secondary VLAN(Type)                    Status Reason
-------------------------------------------------------------------------------
Vlan75    --                                      up     --

And as expected, the interface stayed up! Just to be absolutely sure, I reinstalled the license:

N5K-1# install license MDS20111121084524878.lic
Installing license ......Enable Layer 3.
Error: while enabling/disabling L3: btcm, err: enabling/disabling feature is in progress (0x40aa0004)

Please reload the switch if the switch has previously gone through a non-disruptive NX-OS upgrade.
done
N5K-1# show int vlan 75 brief

-------------------------------------------------------------------------------
Interface Secondary VLAN(Type)                    Status Reason
-------------------------------------------------------------------------------
Vlan75    --                                      down   Non-routable VDC mode

And our “Non-routable VDC mode” error has reappeared.

In hindsight, this behavior has some obvious parallels with the Nexus 7000 case that I found on the Cisco Support Community. We have a VDC (in case of the Nexus 5000 this is the only VDC), which is functioning in Layer 3 mode, but which misses the actual forwarding hardware that would allow it to route packets. The lack of forwarding capability then forces the switch to disable its SVIs. I presume that this is done to prevent the switch from becoming a routing black hole if the forwarding hardware would die or get removed (for example when performing OIR on a Nexus 5596UP). To force the interface up, even if there is no routing capability, you have to mark it as a management interface. (I haven’t had time to test this yet, but I would guess that this also exempts the interface from the Layer 3 switching function and causes it to act as a stub host on the segment, rather than a router interface.)

So what can we conclude from this behavior?

Simply installing a Layer 3 Base Services license changes the primary capability of the switch. Instead of behaving like a pure Layer 2 switch, the switch now considers itself to be a Layer 3 switch, even if there is no Layer 3 forwarding engine installed. However, by lack of a forwarding engine it will disable all SVIs to prevent possible traffic black-holing. To force an SVI up in this mode, you can mark the SVI as a management interface through use of the management command.

Although not intended, I got a very interesting troubleshooting exercise thrown into my lab and I had a lot of fun figuring out what caused the SVI on my Nexus 5000 to stay down. It is really a corner case (why would you install a LAN Base license on a switch that doesn’t have a Layer 3 module?), but I still thought it would be worth documenting, just in case somebody bumps into this same error message. And, if I am ever in need of a really devious troubleshooting exercise for one of my Nexus classes, I now have a very interesting issue to throw at my students!