EVPN Troubleshooting Across Layers
A repeatable 10-layer ladder for overlay incidents: host attachment → local bridge → VNI mapping → VTEP reachability → EVPN control plane → remote MAC/IP state → tunnel object → ASIC programming → packet capture → counters. Each rung names the command that confirms or clears it, so you always know where to look next.
Prerequisites: Lessons 4.1–4.5
Learning objectives
- →Apply a fixed 10-rung ladder to isolate any EVPN/VXLAN failure to a single layer.
- →For each rung, name the confirming command and the 'good' evidence.
- →Recognize the classic split states (RIB-present/ASIC-absent, tunnel-up/forwarding-fails) and where they sit on the ladder.
Overlay problems span FRR, several databases, the tunnel, and the ASIC. Without a fixed ladder you thrash. With one, every incident becomes 'which is the lowest rung that is still green?'
This ladder is the spine of Lab 4.6 (failure injection) and of the module's nextLayer question — it turns panic into procedure.
| In the analogy | In SONiC |
|---|---|
| Is the parcel even at the origin dock, correctly labeled? | Rungs 1–2 (host, local bridge) |
| Right tenant label, and can we reach the destination terminal? | Rungs 3–4 (VNI map, VTEP reachability) |
| Directory correct on both ends? | Rungs 5–6 (EVPN, remote MAC/IP) |
| Container actually loaded onto a truck? | Rungs 7–8 (tunnel, ASIC) |
| Watch the truck on the road; read the gate tallies | Rungs 9–10 (capture, counters) |
Where the analogy stops working
A physical inspection is sequential and safe; some SONiC checks (a disruptive clear, a config change) can perturb the very state you are measuring. Mark and treat those as disruptive, and prefer read-only rungs first — the linear-inspection image understates that risk.
The 10-layer EVPN/VXLAN ladder
| # | Layer | Confirm with | Green means |
|---|---|---|---|
| 1 | Host attachment | host ip link, show interfaces status, show mac | Host up, MAC learned on the access port/VLAN |
| 2 | VLAN / local bridge | show vlan brief | Access port is a member of the intended VLAN |
| 3 | VNI mapping | show vxlan vlanvnimap, show vxlan remotevni | VLAN mapped to the intended VNI on both leaves |
| 4 | VTEP reachability (underlay) | ping -I <Lo0> <remote Lo0>, show ip route | Remote loopback reachable, ideally via ECMP |
| 5 | EVPN control plane | show bgp l2vpn evpn summary | EVPN session Established; prefixes exchanged |
| 6 | Remote MAC/IP state | show bgp l2vpn evpn, show evpn mac vni | Expected type-2/type-5 present and imported (RT match) |
| 7 | Tunnel object | show vxlan tunnel, show vxlan remotemac | Tunnel to remote VTEP present and up |
| 8 | ASIC programming | redis-cli -n 1 KEYS "*FDB*" / VXLAN keys | Remote FDB/tunnel objects exist in ASIC_DB |
| 9 | Packet capture | tcpdump on underlay iface (UDP 4789) | Encapsulated packets seen leaving/arriving |
| 10 | Counters | show interfaces counters, drop counters | Traffic increments; no matching drops |
EVPN session up but route absent → rung 6: suspect route-target/import mismatch or the origin not advertising. Remote MAC present in RIB but not programmed → rung 8: suspect SAI error, resource limit, or a tunnel that is present but not up (rung 7). Tunnel present but forwarding fails → rung 9/10: suspect MTU, ECMP hashing to a dead path, or a decap mismatch. Naming the state tells you which rung to jump to.
Read-only sweep (safe first pass)
show mac # rungs 1-2
show vlan brief # rung 2
show vxlan vlanvnimap # rung 3
show vxlan remotevni # rung 3
show ip route <remote-lo0> # rung 4
vtysh -c 'show bgp l2vpn evpn summary' # rung 5
vtysh -c 'show bgp l2vpn evpn' # rung 6
show vxlan tunnel # rung 7
show vxlan remotemac # rung 7
redis-cli -n 1 KEYS "*VXLAN*" # rung 8 (ASIC_DB)A non-disruptive top-to-bottom sweep you can run before touching anything. Nothing here changes state.
Disruptive checks — use with intent
# Clearing/reset a session forces re-advertisement but drops it briefly:
vtysh -c 'clear bgp l2vpn evpn *' # DISRUPTIVE: flaps EVPN sessions
# Bouncing an interface or the vxlan config re-triggers programming:
sudo config interface shutdown Ethernet0 # DISRUPTIVE
sudo config interface startup Ethernet0These change or perturb state. Only after the read-only sweep localizes the rung, and with a snapshot to revert to.
On VS, rung 8 (ASIC_DB) reflects programming intent but VS does not fully model real ASIC forwarding, and rungs 9–10 (capture/counters) will not match hardware behavior. Use VS to drill the ladder and the commands; confirm forwarding-level rungs on real hardware.
- Always identify the lowest non-green rung before acting; fix that, then re-climb.
- Prefer read-only rungs; reach for disruptive commands only after localization and with a rollback point.
- Record image/platform — several rungs behave differently on VS vs hardware.
Key takeaways
- ✓Ten rungs, host to counters; the fault is the lowest rung not green.
- ✓Three named split states each map to a specific rung to jump to.
- ✓Read-only sweep first; disruptive checks only after localization.
- ✓This ladder is the module's throughline — it powers Lab 4.6 and the nextLayer question.
Tested against: SONiC 202305 / 202311 lineage · Upstream (community) unless noted · SONiC-VS (virtual) baseline · last reviewed 2026-07. Exact commands and behavior can vary by SONiC release, image, hardware platform, and enterprise distribution.
Knowledge check
Rungs 1–4 (host, local bridge, VNI mapping, VTEP reachability) are already confirmed green, and rung 5 (EVPN control plane / session) is Established.
An overlay is failing between two leaves. You confirm the EVPN BGP session is Established, but the expected type-2 route for the remote host is absent from this leaf's EVPN RIB. Using the 10-layer ladder, which layer do you inspect NEXT?