Build the Underlay

60 minLabConfigurationVirtualOpen-source

Stand up the routed fabric the overlay will depend on: point-to-point interfaces, loopbacks, BGP between the leaves, verified loopback-to-loopback reachability (ideally over ECMP), and a validated MTU that leaves room for VXLAN overhead. No overlay yet — this lab exists so later overlay failures are truly overlay failures.

Safety: Caution

May change state — take a snapshot first.

Changing interface addressing/MTU can briefly disrupt the interface — hence 'caution'. Work from the baseline snapshot.

Two-leaf VXLAN lab (Host1—Leaf1—underlay—Leaf2—Host2)

  Host1 (Linux)                                   Host2 (Linux)
  10.10.10.11/24                                  10.10.10.12/24
      │ eth1                                           eth1 │
      │                                                     │
  ┌───┴────────┐        virtual underlay          ┌────────┴───┐
  │  Leaf1     │  Ethernet0  ── p2p ──  Ethernet0  │  Leaf2     │
  │  SONiC-VS  │◀───────── routed IP fabric ──────▶│  SONiC-VS  │
  │  Lo0 =     │        (BGP, ECMP-capable)        │  Lo0 =     │
  │  10.0.0.1  │                                   │  10.0.0.2  │
  └────────────┘                                   └────────────┘
   access: EthernetX (VLAN 10 ↔ VNI 10010)   access: EthernetX (VLAN 10 ↔ VNI 10010)

Objective

• Configure p2p routed interfaces and Loopback0 on both leaves.
• Bring up underlay BGP and advertise the loopbacks.
• Verify loopback-to-loopback reachability and, where the topology allows, ECMP.
• Validate underlay MTU has headroom for ~50 bytes of VXLAN overhead.

Starting state

• Both leaves powered on, management reachable, at default config plus baseline snapshot.
• Underlay interface(s) cabled in the virtual topology but unconfigured.

Prerequisites: Lessons 4.1 and 4.2 complete. · Two SONiC-VS leaves (Leaf1, Leaf2) with at least one interconnect (a virtual underlay/spine is fine). · A 'baseline' VM snapshot on each leaf to revert to.

Tasks

1
Configure loopbacks and p2p interfaces
Loopback0 is the VTEP source and BGP router-id anchor. The p2p interface carries underlay BGP.
SONiC CLILeaf1 (use 10.0.0.2/32 on Leaf2)
```
sudo config interface ip add Loopback0 10.0.0.1/32
```
SONiC CLILeaf1 p2p (10.1.12.0/31 on Leaf2)
```
sudo config interface ip add Ethernet0 10.1.12.1/31
```
SONiC CLI
```
sudo config interface startup Ethernet0
```
Expected:
- Loopback0 and Ethernet0 show the configured IPs in show ip interfaces.
What happened internally: These write to CONFIG_DB (LOOPBACK_INTERFACE, INTERFACE); managers push to APPL_DB and the kernel. Nothing overlay-related exists yet.
2
Set underlay MTU with VXLAN headroom
Raise the underlay interface MTU so a full tenant frame plus ~50 bytes of VXLAN overhead passes without fragmentation. Jumbo (9100) is the clean choice.
SONiC CLI
```
sudo config interface mtu Ethernet0 9100
```
SONiC CLIconfirm MTU applied
```
show interfaces status
```
Expected:
- Ethernet0 MTU reads 9100 (or your fabric's jumbo value) on both leaves.
What happened internally: MTU is per-interface in CONFIG_DB. Default 1500 is the classic silent-VXLAN failure; jumbo removes overhead from the tenant's concern.
3
Bring up underlay BGP and advertise loopbacks
Use the eBGP-per-leaf or iBGP model your design chose (Lesson 4.2). Advertise Loopback0 so the far VTEP is routable.
FRR / vtysh
```
vtysh
```
FRR / vtysh
```
configure terminal
```
FRR / vtyshLeaf1 ASN (65002 on Leaf2 for eBGP)
```
router bgp 65001
```
FRR / vtysh
```
 neighbor 10.1.12.0 remote-as 65002
```
FRR / vtysh
```
 address-family ipv4 unicast
```
FRR / vtysh
```
  network 10.0.0.1/32
```
FRR / vtyshenable ECMP
```
  maximum-paths 4
```
FRR / vtysh
```
end
```
Expected:
- BGP session to the neighbor reaches Established; loopbacks appear in each other's RIB.
What happened internally: This is standard FRR from Module 3. The overlay EVPN session (loopback-to-loopback) does not exist yet — that is Lab 4.3.
4
Verify reachability and ECMP
SONiC CLIfrom Leaf1
```
show ip route 10.0.0.2/32
```
Linuxsource from the local loopback
```
ping -I 10.0.0.1 10.0.0.2
```
FRR / vtysh
```
vtysh -c 'show ip bgp 10.0.0.2/32'
```
Expected:
- Remote loopback reachable; if the topology has parallel paths, multiple next hops appear (ECMP).
What happened internally: Loopback-to-loopback reachability with the correct source is the exact dependency the overlay has. Multipath in the RIB confirms ECMP is real, not assumed.
5
Validate MTU end to end
Prove large frames survive the underlay by sending a DF-bit ping sized above 1500 between loopbacks.
LinuxDF-bit, 8000-byte payload
```
ping -I 10.0.0.1 -M do -s 8000 10.0.0.2
```
Expected:
- Large DF pings succeed. Failure here means MTU is too small somewhere on the path — fix before any overlay.
What happened internally: Small pings hide MTU problems; the DF large ping is the honest test. This is why MTU validation belongs in the underlay lab, not after VXLAN 'mysteriously' fails on big transfers.

Troubleshooting branches

If: BGP stuck in Active/Connect

Then: Verify p2p /31 addressing matches, interface is startup, and no ACL blocks TCP 179.

If: Loopback not in remote RIB

Then: Confirm network 10.0.0.x/32 (or redistribute) under the correct address-family.

If: Small pings pass, big DF pings fail

Then: MTU too small on some hop; raise interface MTU to the jumbo value on both ends.

If: Only one ECMP next hop

Then: maximum-paths set? Are the parallel paths truly equal-cost with distinct next hops?

Cleanup / rollback

Keep the underlay — Labs 4.2–4.6 build directly on it.
Take a new snapshot labeled 'underlay-good' on each leaf.

Lab evidence checklist

☐ Loopback0 and p2p IPs configured on both leaves.
☐ Underlay BGP Established; loopbacks mutually reachable.
☐ ECMP confirmed in the RIB where the topology allows.
☐ Large DF ping between loopbacks succeeds (MTU validated).
☐ 'underlay-good' snapshot exists on each leaf.

Reflection

Why must loopback-to-loopback reachability and MTU be correct before touching VXLAN?
If you skipped the DF large-ping test, which later overlay symptom would you have misdiagnosed?

Lab notes & observations