Failure Scenarios

75 minLabFailure injectionTroubleshootingVirtualOpen-source

Failure-injection lab: deliberately break the working fabric one way at a time — underlay route missing, VNI mismatch, MTU problem, EVPN session up but route absent, MAC learned locally but not advertised, remote route present but not programmed, tunnel present but forwarding fails — and drive each to root cause with the 10-layer ladder. Revert to a known-good snapshot between scenarios.

Safety: Disruptive — lab only

Disrupts forwarding or services. Run only in a dedicated virtual lab you can roll back. Never on production or shared physical switches.

This lab deliberately breaks a working fabric — it is disruptive. Only run it against the lab topology, one fault at a time, reverting to a snapshot between scenarios.

On SONiC-VS, forwarding-level rungs (9–10) do not fully model hardware; validate those on real hardware where possible.

Two-leaf VXLAN lab (Host1—Leaf1—underlay—Leaf2—Host2)

  Host1 (Linux)                                   Host2 (Linux)
  10.10.10.11/24                                  10.10.10.12/24
      │ eth1                                           eth1 │
      │                                                     │
  ┌───┴────────┐        virtual underlay          ┌────────┴───┐
  │  Leaf1     │  Ethernet0  ── p2p ──  Ethernet0  │  Leaf2     │
  │  SONiC-VS  │◀───────── routed IP fabric ──────▶│  SONiC-VS  │
  │  Lo0 =     │        (BGP, ECMP-capable)        │  Lo0 =     │
  │  10.0.0.1  │                                   │  10.0.0.2  │
  └────────────┘                                   └────────────┘
   access: EthernetX (VLAN 10 ↔ VNI 10010)   access: EthernetX (VLAN 10 ↔ VNI 10010)

Objective

• Inject each of seven classic overlay faults, one at a time.
• Localize every fault to a single ladder rung using the read-only sweep first.
• Confirm the fix restores forwarding, then revert before the next scenario.

Starting state

• Fully working fabric: Host1 ↔ Host2 forwarding over the overlay. Snapshots exist to revert to.

Prerequisites: Labs 4.1–4.4 complete; a known-good 'evpn-established' snapshot on each leaf. · Lesson 4.6 (the 10-layer ladder) studied.

Tasks

1
Scenario A — Underlay route missing
Remove the loopback advertisement so the remote VTEP becomes unreachable.
Predict the rung before you look.
FRR / vtysh Disruptive
```
vtysh -c 'conf t' -c 'router bgp 65001' -c 'address-family ipv4 unicast' -c 'no network 10.0.0.1/32'
```
Expected:
- Forwarding fails; ladder localizes to rung 4 (VTEP reachability) — remote loopback no longer in RIB.
What happened internally: The overlay session over loopbacks may also drop, illustrating the underlay↔overlay dependency.
2
Scenario B — VNI mismatch
Change the VLAN-VNI map on one leaf only.
SONiC CLI Disruptive
```
sudo config vxlan map del vtep 10 10010
```
SONiC CLI Disruptivenow asymmetric
```
sudo config vxlan map add vtep 10 10011
```
Expected:
- Forwarding fails; rung 3 (VNI mapping). Capture (rung 9) shows VNI on wire not matching the peer's map.
What happened internally: The tunnel and EVPN session stay up — proving a green upper rung does not clear a broken lower rung.
3
Scenario C — MTU problem
Lower underlay MTU below the encapsulated frame size.
SONiC CLI Disruptive
```
sudo config interface mtu Ethernet0 1500
```
Expected:
- Small pings pass, large transfers hang; rung 4/10. DF large-ping reproduces it; counters show drops.
What happened internally: The intermittent signature (small ok, big fails) is the MTU fingerprint from Lesson 4.2.
4
Scenario D — EVPN session up but route absent
Break route-target import (or remove advertise-all-vni) so the session stays up but routes don't cross.
FRR / vtysh Disruptive
```
vtysh -c 'conf t' -c 'router bgp 65001' -c 'address-family l2vpn evpn' -c 'no advertise-all-vni'
```
Expected:
- Session Established but type-2/type-3 absent on the peer; rung 5 green, rung 6 red.
What happened internally: The canonical 'session up, route absent' split state — the module's nextLayer question lives here.
5
Scenario E — MAC learned locally but not advertised
Simulate a local-learn-but-no-origination condition (e.g., VNI not EVPN-enabled for that VLAN while local bridging still works).
SONiC CLIlearned locally on Leaf1
```
show mac | grep -i <host1-mac>
```
FRR / vtyshbut NOT originated
```
vtysh -c 'show bgp l2vpn evpn route type macip'
```
Expected:
- MAC present locally (rung 1–2 green) but no type-2 route originated; fault at rung 5/6 origination.
What happened internally: Distinguishes 'not learned' from 'learned but not advertised' — different fixes entirely.
6
Scenario F — Remote route present but not programmed
Induce a programming gap: route imported into the RIB but not installed in the FDB/ASIC (e.g., tunnel not up, or inspect an orchagent/SAI error condition).
FRR / vtyshroute present on Leaf2
```
vtysh -c 'show bgp l2vpn evpn route type macip'
```
SONiC CLIbut absent from FDB
```
show mac | grep -i <host1-mac>
```
SONiC DBASIC_DB check (rung 8)
```
redis-cli -n 1 KEYS "*FDB*"
```
Expected:
- Route in RIB (rung 6 green) but no FDB/ASIC entry (rung 7/8 red) — the 'present but not programmed' state.
What happened internally: Send you to rung 7 (tunnel up?) then rung 8 (SAI/resource/orchagent logs).
7
Scenario G — Tunnel present but forwarding fails
Tunnel object shows up and remote MAC is programmed, yet packets don't arrive (e.g., ECMP hashing onto a broken path, or a decap-side mismatch).
SONiC CLItunnel shows present/up
```
show vxlan tunnel
```
Linuxrung 9: are packets on the wire?
```
sudo tcpdump -ni Ethernet0 udp port 4789 -c 10
```
SONiC CLIrung 10: drops?
```
show interfaces counters
```
Expected:
- Upper rungs green but no delivery; capture/counters localize to the data path (rung 9/10).
What happened internally: The last split state: object model looks perfect, so you must drop to capture and counters to find the truth.

Troubleshooting branches

If: Session up but route absent

Then: Rung 6: advertise-all-vni present? RT import matches? Origin actually advertising?

If: MAC local but not advertised

Then: Rung 5/6: is the VLAN/VNI EVPN-enabled for origination? Local bridging alone won't originate type-2.

If: Route present but not programmed

Then: Rung 7 then 8: tunnel up? ASIC_DB FDB present? Check orchagent/syncd logs for SAI errors or resource limits.

If: Tunnel present but forwarding fails

Then: Rung 9/10: capture UDP 4789, check MTU, ECMP path health, and decap-side VNI/VLAN map.

If: Small pings pass, big transfers hang

Then: Rung 4: MTU too small; validate with a DF large ping and raise underlay MTU.

Cleanup / rollback

After EACH scenario, revert to the 'evpn-established' snapshot before injecting the next fault.
Confirm baseline forwarding is restored (Host1 ↔ Host2) before ending the lab.

Lab evidence checklist

☐ Each of the seven scenarios injected and localized to a single ladder rung.
☐ For each: the read-only sweep evidence that pointed to the rung, and the confirming fix.
☐ Snapshot reverted between scenarios; final baseline forwarding restored.

Reflection

For scenario D, having confirmed the session is Established, which rung do you inspect next and what exactly do you look for?
Which two scenarios produced nearly identical top-level symptoms but different root-cause rungs, and how did the sweep separate them?

Lab notes & observations