Underlay Design
The overlay is only as good as the road beneath it. This lesson covers eBGP vs iBGP underlays, loopback and point-to-point addressing, ECMP, the reachability the VTEPs actually require, the MTU math VXLAN forces on you, and how underlay design draws your failure domains.
Prerequisites: Lesson 4.1 · BGP configuration in FRR (Module 3)
Learning objectives
- →Choose between an eBGP (per-leaf ASN) and iBGP (route-reflector) underlay and justify it.
- →State the *minimum* reachability the overlay needs: VTEP loopback to VTEP loopback.
- →Compute the MTU budget VXLAN overhead imposes and set underlay MTU accordingly.
- →Explain how ECMP is achieved and verified, and how underlay topology defines failure domains.
A large fraction of 'EVPN is broken' tickets are underlay tickets in disguise: a missing loopback route, an MTU too small for the encapsulated frame, or ECMP that silently isn't.
If you validate the underlay rigorously first, every later overlay failure is genuinely an overlay failure — which halves your search space.
What your CCNP already gives you
- • eBGP vs iBGP mechanics, route reflection, next-hop behavior.
- • ECMP concepts and BGP multipath.
- • Path-MTU and fragmentation fundamentals.
What is new in SONiC
- • FRR is the BGP implementation; underlay config is standard FRR you already read in Module 3.
- • SONiC interface MTU is set per-interface in CONFIG_DB and must account for VXLAN overhead explicitly.
- • Loopback advertisement and multipath are verified through both
show ip routeand vtysh.
| In the analogy | In SONiC |
|---|---|
| The public road network | Underlay routed fabric |
| The street addresses of the terminals | VTEP loopbacks |
| Multiple parallel highways between two cities | ECMP paths |
| The tunnel/bridge height limit trucks must clear | MTU headroom |
Where the analogy stops working
Roads are shared public infrastructure; a data-center underlay is private and you control every hop, so you can (and should) engineer MTU and ECMP precisely rather than hope. Also, unlike a truck that visibly can't fit, an MTU problem often manifests as *intermittent* failure (pings pass, bulk transfers hang) — a subtlety the analogy understates.
eBGP vs iBGP underlay
| Aspect | eBGP underlay (per-device ASN) | iBGP underlay (single ASN + route reflectors) |
|---|---|---|
| AS design | Unique ASN per leaf/spine (often private 4-byte) | One ASN fabric-wide; spines are route reflectors |
| Next-hop / path visibility | AS-path gives loop protection and easy multipath | Needs RR + next-hop-unchanged discipline for overlay |
| Config volume | More neighbor/ASN config, very explicit | Fewer ASNs, but RR design must be correct |
| ECMP | Multipath across spines is natural with distinct AS-paths | Requires bgp bestpath / add-path tuning |
| Common DC choice | Popular for CLOS fabrics; simple to reason about | Common where an existing iBGP core is reused |
The underlay carries loopback reachability; the EVPN (overlay) BGP sessions are typically established between loopbacks (often multihop). Do not conflate the two sessions — in troubleshooting they fail independently. A healthy underlay session does not imply a healthy EVPN session, and vice versa.
The only reachability the overlay demands
Reduce the requirement to its core: every VTEP loopback must be routable from every other VTEP loopback, and the EVPN sessions between those loopbacks must come up. That is it. Tenant subnets, VLANs, and VNIs are never carried by the underlay.
This is why underlay validation in Lab 4.1 is loopback-to-loopback ping plus route inspection — nothing tenant-related appears until the overlay exists.
MTU: the overhead you cannot ignore
VXLAN adds roughly 50 bytes of outer headers to every frame: outer Ethernet (14) + outer IPv4 (20) + UDP (8) + VXLAN (8), and more with IPv6 or additional tags. If tenant hosts send standard 1500-byte frames, the underlay must carry ~1550-byte packets without fragmentation.
The clean fix is jumbo frames on the underlay (e.g., interface MTU 9100/9216) so overhead is invisible to tenants. If you cannot raise underlay MTU, you must lower the tenant MTU — but that pushes complexity onto hosts. Either way, decide it deliberately; the default 1500 everywhere is the classic silent-VXLAN-failure setup.
MTU budget (IPv4 underlay, illustrative)
| Layer | Bytes | Running total |
|---|---|---|
| Tenant frame (payload as seen by host) | 1500 | 1500 |
| Inner Ethernet header (already in the 1500? no — added) | 14 | 1514 |
| VXLAN header | 8 | 1522 |
| UDP header | 8 | 1530 |
| Outer IPv4 header | 20 | 1550 |
| Outer Ethernet header | 14 | 1564 |
ECMP and failure domains
In a CLOS fabric, ECMP across spines gives you both bandwidth and resilience: losing one spine costs 1/N of capacity, not connectivity. In FRR this means BGP multipath must be enabled and the loopback must be learned via multiple equal-cost next hops — verify it, don't assume it.
Your underlay topology *is* your failure-domain map. A single shared spine, a single underlay link, or a single-homed leaf each defines a blast radius. Design so that no single underlay failure isolates a VTEP that hosts live tenants.
- Before blaming EVPN, ping remote loopback from local loopback (correct source) — that is the overlay's true dependency.
- Test large-frame reachability (DF-bit, size ~1500+) to expose MTU problems the small pings hide.
- Confirm multipath in the RIB; a missing second next hop is a latent outage.
Key takeaways
- ✓The underlay's sole job: loopback-to-loopback reachability, ideally over ECMP.
- ✓eBGP-per-leaf and iBGP-with-RR are both valid; choose deliberately and keep overlay peering separate.
- ✓Budget ~50 bytes of VXLAN overhead — raise underlay MTU or you will hit silent large-frame failures.
- ✓Underlay topology defines your failure domains; validate the underlay fully before touching the overlay.
Tested against: SONiC 202305 / 202311 lineage · Upstream (community) unless noted · SONiC-VS (virtual) baseline · last reviewed 2026-07. Exact commands and behavior can vary by SONiC release, image, hardware platform, and enterprise distribution.
Knowledge check
Two-leaf VXLAN fabric; underlay interfaces were left at their default MTU.
Small pings between two tenant hosts succeed, but bulk file transfers hang and never complete across the overlay. The EVPN session is Established and the remote MAC is programmed. What is the most likely root cause?
Related lessons