I had a (perhaps common) misunderstanding about how MPLS applied labels. I thought that MPLS would stack up labels for the entire path (so like 6 hops, 5 labels). This of course would affect MTU of MPLS packets, especially on the originating edge.
While reading the excellent presentation Demystifying IPv6 Over MPLS by Kam Agahian, I discovered that my understanding of how MPLS worked was actually quite wrong, and that labels weren’t stacked, they were swapped. This made sense as I’d heard label swapping before, but somehow had missed connecting it to what was actually happening.
Seeing it in action made it click for me. Swapping, not stacking.
This traceroute shows the labels from the MPLS forwarding tables on each router. That out-label is matched to in-label on the next router. In networks with penultimate hop popping this repeats until the pentultimate router where no label added and the packet is routed directly using the routing table.
> /tool traceroute src-address=10.xxx.0.14 10.xxx.0.2 # ADDRESS LOSS SENT LAST AVG BEST WORST STD-DEV STATUS 1 yyy.zz.1.53 0% 4 11.1ms 12.6 10.3 14.7 1.9 <MPLS:L=635,E=6> 2 yyy.zz.1.49 0% 4 8.4ms 8 6.9 9.4 1 <MPLS:L=88,E=6> 3 yyy.zz.1.17 0% 4 10.1ms 10 9.5 10.6 0.4 <MPLS:L=18,E=6> 4 yyy.zz.1.5 0% 4 12.3ms 10.6 9.2 12.3 1.1 <MPLS:L=25,E=6> 5 10.xxx.0.2 0% 4 8.1ms 9 7.3 11 1.4
I used the following loop to get the MPLS forwarding table from all the routers in the line:
for i in router1 router2 router3 router4 penultimate ; do ssh $i "/mpls forwarding-table print where destination=10.xxx.0.2/32" done
Flags: H - hw-offload, L - ldp, V - vpls, T - traffic-eng # IN-LABEL OUT-LABELS DESTINATION INT... 0 L 220 635 10.xxx.0.2/32 upl...
Flags: H - hw-offload, L - ldp, V - vpls, T - traffic-eng # IN-LABEL OUT-LABELS DESTINATION INT... 0 L 635 88 10.xxx.0.2/32 upl...
Flags: H - hw-offload, L - ldp, V - vpls, T - traffic-eng # IN-LABEL OUT-LABELS DESTINATION INT... 0 L 88 18 10.xxx.0.2/32 upl...
Flags: H - hw-offload, L - ldp, V - vpls, T - traffic-eng # IN-LABEL OUT-LABELS DESTINATION INT... 0 L 18 25 10.xxx.0.2/32 upl...
Flags: H - hw-offload, L - ldp, V - vpls, T - traffic-eng # IN-LABEL OUT-LABELS DESTINATION INT... 0 L 25 10.xxx.0.2/32 upl...
So whats actually happening here?
Traceroute shows the path that packets take. In this case it also shows the MPLS labels that apply to each hop. This combined with the MPLS forwarding table from each router gives a clearer picture of whats happening here.
ICMP packet from 10.xxx.0.14 to 10.xxx.0.2
Originating router applies the MPLS label 635.
The first hop router sees the label 635 coming in, and knows to put the label 88 outbound.
The second hop router sees label 88 inbound, and knows to put label 18 outbound.
The third hop router sees label 18 inbound, and knows to put label 25 outbound.
The penultimate router sees 25 inbound, and routes the last hop via its routing table as there is no outgoing label towards that destination.
Each router has its own list of labels, and exchanges labels with the router beside it over Label Distribution Protocol (LDP).
That is how MPLS knows how to swap the labels at each hop without needing to do a route lookup, and this is where the efficiency of MPLS comes in.
Normally MPLS is abstracted out on network diagrams as one or more clouds with MPLS written on them. In those networks, it is someone else’s job to run that. This blog series is about operating an MPLS Service Provider cloud from the inside, instead of the usual perspective of using someone else’s MPLS.