When a FabricPath edge switch needs to send a frame to a remote MAC address, it performs a MAC address table lookup and finds an entry of the form SWID.SSID.LID. The SWID represents the switch-ID of the remote FabricPath edge switch, the SSID represents the sub-switch ID (which is only used in vPC+) and the LID represents the outbound port on the remote edge switch. However, the method by which these LIDs are derived doesn’t seem to be very well documented and this had been bugging me for a while. So I decided to dig in and see if I could find out a bit more about the way LIDs are used on the Nexus switches.

I found some hints in the excellent Cisco Live Presentation BRKDCT-3313 – Fabric Path operation and troubleshooting (2013 London). Inside this presentation I found a somewhat cryptic sentence that states “for N7K the LID is the port index of the ingress interface, for N5K LID most of the time will be 0”. Let’s see what we can make of that.

The acronym LID stands for “Local ID” and, as the name implies, it has local significance to the switch that a particular MAC address resides on. As such, it is up to the implementation to determine how to derive a unique LID to represent its ports. Apparently, the Nexus 5000 and Nexus 7000 engineering teams did not talk to each other to agree on some consistent method of assigning the LIDs, but each created their own platform-specific implementation.

For the N5K I couldn’t really find more than that the LID will usually be 0, but there may be some exceptions. For the N7K, the Cisco Live presentation states that the LID maps to the “port index” of the ingress interface. (The interface represented by the LID is an ingress interface from the perspective of the edge switch that inserts the LID into the outer source address. For the switch sending to the MAC address it represents the egress port at the destination edge switch).

So I decided to get into the lab and see if I could find some commands that would help me establish the relation between the LID and the outbound interface on the edge switch. I created a very simple FabricPath network and performed a couple of pings to generate some MAC address table entries.

Let’s have a look at a specific entry in the MAC address table of a Nexus 7000:

N7K-1-pod5# show mac address-table dynamic vlan 100
Legend:
        * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
        age - seconds since last seen,+ - primary entry using vPC Peer-Link,
        (T) - True, (F) - False
   VLAN     MAC Address      Type      age     Secure NTFY Ports/SWID.SSID.LID
---------+-----------------+--------+---------+------+----+------------------
* 100      0005.73e9.8c81    dynamic   960        F    F  Eth3/15
  100      0005.73e9.fcfc    dynamic   960        F    F  16.0.14
  100      00c0.dd18.6ce0    dynamic   420        F    F  16.0.14
  100      00c0.dd18.6ce1    dynamic   0          F    F  16.0.14
* 100      00c0.dd18.6e08    dynamic   0          F    F  Eth3/15
* 100      00c0.dd18.6e09    dynamic   0          F    F  Eth3/15

So for example, let’s zoom in on the MAC address 0005.73e9.fcfc. According the table, frames for this destination should be sent to SWID.SSID.LID “16.0.14”. From the SWID part, we can see that the MAC address resides on the switch with ID “16”. To find the corresponding switch hostname we can use the following command:

System-ID        Primary  Secondary  Reachable  Bcast-Priority Ftag-Root Capable Hostname
MT-0

b414.89dc.7a44   16  [C]       0[C]  Yes                64          [S]  Y     N7K-2-pod6

f025.72a8.bf44*  15  [C]       0[C]  Yes                64          [S]  Y     N7K-1-pod5

So we jump to switch N7K-2-pod6 and perform another MAC address table lookup:

N7K-2-pod6# show mac address-table address 0005.73e9.fcfc
Legend:
        * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
        age - seconds since last seen,+ - primary entry using vPC Peer-Link,
        (T) - True, (F) - False
   VLAN     MAC Address      Type      age     Secure NTFY Ports/SWID.SSID.LID
---------+-----------------+--------+---------+------+----+------------------
* 100      0005.73e9.fcfc    dynamic   450        F    F  Eth3/15

Now we know that the outbound interface for the MAC address on the destination edge switch is Ethernet 3/15. So how can we map the LID “14” to this interface?

The Cisco Live presentation states that the LID corresponds to the “port index” for the interface. So how can we find the port index? The port index is an internal identifier for the interface, also referred to as the LTL and there are some show commands to determine these LTLs. For example, if we wanted to know the LTL for interface E3/15, we could issue the following command:

N7K-2-pod6# show system internal pixm info interface e 3/15
LTL TYPE         LTL
========================
PHY_PORT        0xe
FLOOD_W_FPOE    0x8031
FLOOD_W_FPOE    0x8035

Here we find that the LTL for the interface is 0xe, which equals 14 in decimal. This shows that the LID is actually the decimal representation of the LTL. (FabricPath switch-IDs, subswitch-IDs and Local IDs are represented in decimal by default).

This lookup can also be performed in reverse. If we take the LID and convert it to its hexadecimal representation of 0xe, we can find the corresponding interface as follows:

N7K-2-pod6# show system internal pixm info ltl 0xe

Member info
------------------
Type            LTL
---------------------------------
PHY_PORT       Eth3/15
FLOOD_W_FPOE   0x8035
FLOOD_W_FPOE   0x8031

So through use of these two commands, we can map a FabricPath LID to an interface and vice versa on a Nexus 7000.