VXLAN Forwarding Behavior
Follow one tenant packet end to end: local source host → ingress leaf FDB/routing decision → remote VTEP resolution → VXLAN encapsulation → underlay forwarding → remote decapsulation → destination delivery. Each step names the SONiC object that must be correct for the packet to advance.
Prerequisites: Lessons 4.1 and 4.3
Learning objectives
- →Trace the full ingress-to-egress path of a VXLAN-encapsulated tenant frame.
- →Identify, at each hop, the SONiC object (FDB, tunnel, route, ASIC entry) that gates forwarding.
- →Distinguish the bridged (L2VNI) path from the routed (L3VNI/IRB) path at the decision point.
When forwarding fails, you need a mental packet-walk so you can ask 'how far did it get?' rather than guessing. Each step in this walk is a place a real packet dies.
The walk is the backbone of Lab 4.4 (correlation) and Lab 4.5 (capture); knowing it cold makes those labs interpretation instead of mystery.
| In the analogy | In SONiC |
|---|---|
| The dock clerk deciding which terminal the parcel is bound for | Ingress leaf FDB/route lookup |
| Looking up the destination terminal's street address | Remote VTEP resolution |
| Loading the parcel into a labeled container | VXLAN encapsulation |
| Driving the container over the road network | Underlay forwarding |
| The destination terminal unloading the container | Remote decapsulation |
Where the analogy stops working
A parcel is one object handled once; a switch makes the bridge-vs-route decision per packet in hardware, and for BUM traffic it *replicates* the container to many terminals (head-end replication) rather than sending one. The single-parcel image hides that fan-out.
The forwarding walk (known unicast, L2VNI)
- Local source host → ingress leaf. Host1 sends a frame; Leaf1 receives it on the access port and learns/refreshes the source MAC in the local bridge FDB for the VLAN.
- FDB / routing decision. Leaf1 looks up the destination. Same subnet (dest MAC known) → bridge via L2VNI. Different subnet → route (L3VNI/IRB path, see Lesson 4.5). Assume same-subnet bridging here.
- Remote VTEP resolution. The destination MAC's FDB entry (installed earlier from a type-2 route) points at a remote VTEP IP. If the entry is missing, the frame is treated as unknown-unicast/BUM and flooded per the type-3 list instead.
- VXLAN encapsulation. Leaf1 builds outer Ethernet / outer IP (src = its Lo0, dst = remote VTEP Lo0) / UDP(4789) / VXLAN(VNI) around the original frame. This is an ASIC operation once programming is correct.
- Underlay forwarding. The encapsulated packet is routed by the underlay from Leaf1's loopback to Leaf2's loopback over one of the ECMP paths — the fabric sees only outer IP.
- Remote decapsulation. Leaf2 receives the packet on its VTEP, matches the outer dst = its Lo0 + UDP 4789, strips the outer headers, and reads the VNI to select the tenant bridge domain.
- Destination forwarding. Leaf2 looks up the inner destination MAC in that VNI's FDB and delivers the original frame out the local access port to Host2.
Where each step can die (and what to read)
| Step | Gating object | If broken, you see |
|---|---|---|
| Local learn | Bridge FDB (VLAN) | Source never learned; no type-2 originated |
| Decision | VLAN/VRF binding | Wrong path chosen (bridge vs route) |
| VTEP resolution | Remote FDB entry (ASIC_DB) | Flooding instead of unicast, or drops |
| Encapsulation | VXLAN tunnel object (STATE_DB/ASIC_DB) | Frame not encapsulated; local-only reachability |
| Underlay | Loopback route + MTU | Small frames pass, large frames drop (MTU) |
| Decap | Remote tunnel + VNI map | Packet arrives but is dropped/misassigned |
| Deliver | Egress FDB in VNI | Decapped but not delivered to host |
Broadcast/unknown-unicast/multicast (e.g., the ARP that *precedes* unicast) is head-end replicated: the ingress VTEP sends a copy to every peer VTEP in the VNI's type-3 list. If ARP resolution fails but you 'know' the remote host exists, suspect the type-3/flood path before the type-2 unicast path.
Objects to read along the walk
show mac # local + remote FDB (VLAN, port/VTEP)
show vxlan tunnel # tunnel operational state
show vxlan remotevni # remote VNIs learned per VTEP
show vxlan remotemac # remote MACs mapped to VTEPs
# ASIC truth (SONiC-VS caveat: VS does not fully model real ASIC forwarding):
redis-cli -n 1 KEYS "*VXLAN*" # ASIC_DB VXLAN/tunnel objects
redis-cli -n 1 KEYS "*FDB*" # ASIC_DB FDB entriesThe commands that expose each gating object; use them in the same order as the steps.
On the virtual switch, ASIC_DB reflects programming but VS does not reproduce true hardware forwarding/ASIC behavior. Use VS to learn the object flow and the commands; validate real forwarding on hardware. Note your image and platform in every lab log.
- Localize the failure to a step before choosing a tool; the step tells you which object and command to use.
- Separate the BUM path from the unicast path in your reasoning and your tests.
- On VS, trust the object flow but not the forwarding fidelity.
Key takeaways
- ✓Memorize the seven-step walk; each step names a gating SONiC object.
- ✓Same-subnet bridges via L2VNI; cross-subnet routes via L3VNI/IRB.
- ✓BUM is head-end replicated on the type-3 path, distinct from type-2 unicast.
- ✓VS shows the object flow faithfully but not real ASIC forwarding.
Tested against: SONiC 202305 / 202311 lineage · Upstream (community) unless noted · SONiC-VS (virtual) baseline · last reviewed 2026-07. Exact commands and behavior can vary by SONiC release, image, hardware platform, and enterprise distribution.
Knowledge check
Order the steps a known-unicast tenant frame takes from Host1 to Host2 across the overlay.
- 1Leaf1 builds VXLAN encapsulation (outer Eth/IP/UDP/VXLAN-VNI)
- 2Leaf1 learns the source MAC and looks up the destination (bridge decision)
- 3The underlay routes the packet from Leaf1's loopback to Leaf2's loopback
- 4Leaf2 decapsulates and reads the VNI to select the tenant bridge domain
- 5Leaf2 looks up the inner destination MAC and delivers the frame to Host2
- 6Leaf1 resolves the destination MAC's remote VTEP from its FDB
On Leaf2 you confirm Host1's MAC is present in the EVPN RIB (type-2 route imported) but it is NOT in show mac/the FDB, and traffic to Host1 fails. How is this best characterized?