SONiC

Final Capstone

Operate and troubleshoot a small SONiC fabric: two leaves connect two Linux hosts over an EVPN/VXLAN overlay. Build it, prove it, break it, diagnose it, and recommend a distribution.

Capstone progress0/11 stages
Lab discipline

Failure injection happens only in the virtual lab you can roll back. Snapshot before each injected failure, and never run these steps on production or shared physical switches.

Stages

  1. 1. Validate the virtual lab

    Revert both leaves and both hosts to a known-good snapshot so you start from a clean, reproducible baseline. Everything you claim later must be traceable to this starting point.

    Confirm each leaf boots cleanly: no failed systemd units, the core containers are running (database, swss, syncd, bgp), and the management plane is reachable. Confirm both hosts are up and can reach their access ports.

    Record the image version and platform on each leaf. On SONiC-VS the platform is virtual — note this now, because hardware-dependent results (scale, counters, QoS) will not be authoritative later.

    Acceptance: Both leaves reachable; no failed systemd units; core containers running.; Both hosts up and connected to their access ports.; Image version and platform recorded for each leaf; snapshot labeled 'capstone-baseline'.

  2. 2. Build the routed underlay

    Configure the point-to-point routed link(s) between leaf1 and leaf2 and assign each leaf a loopback that will source its VTEP (e.g. leaf1 Lo 10.0.0.1/32, leaf2 Lo 10.0.0.2/32).

    Set the underlay MTU with headroom for VXLAN encapsulation (~50 bytes). Standardize on a jumbo underlay MTU end to end; MTU mistakes here surface much later as 'small pings pass, real traffic fails'.

    The underlay's only job is to carry loopback-to-loopback reachability between the VTEPs. Prove that before adding any overlay.

    Acceptance: Routed P2P link up with correct addressing on both ends.; Each leaf has a /32 loopback for its VTEP source.; Underlay MTU set with VXLAN headroom (jumbo end to end).

  3. 3. Establish BGP

    Bring up the underlay BGP session between the leaves (eBGP over the P2P links, or iBGP with a route-reflector if you use a spine). Advertise the loopbacks so each VTEP can reach the other's source address.

    Confirm the session reaches Established and that each leaf installs a route to the remote loopback. This is the transport the overlay will ride on.

    Acceptance: Underlay BGP session Established between the leaves.; Each leaf has a route to the remote VTEP loopback.; Loopback-to-loopback ping succeeds.

  4. 4. Configure VTEPs and VNIs

    Create the VXLAN tunnel endpoint on each leaf sourced from its loopback, then map the access VLAN (Vlan10) to the L2VNI (10010) on both leaves.

    The VLAN-to-VNI mapping must exist and match on both sides. A missing mapping on one leaf is a classic 'local host learns, remote never installs' failure you will inject later.

    Acceptance: VTEP created on both leaves sourced from the loopback.; Vlan10 <-> VNI 10010 mapping present and identical on both leaves.; `show vxlan tunnel` shows the tunnel source correctly.

  5. 5. Establish EVPN

    Activate the l2vpn evpn address family on the BGP session so the leaves exchange EVPN routes, and advertise the L2VNI. Confirm the EVPN session is Established and the VNI is up with the expected RD and route-targets.

    Once a host sends a frame, the originating leaf should generate a type-2 (MAC/IP) route and a type-3 (IMET) route for BUM handling; the remote leaf should import them (route-targets must match).

    Acceptance: L2VPN EVPN session Established between the leaves.; VNI 10010 up on both leaves with matching route-targets.; Type-2 and type-3 routes exchanged once hosts are active.

  6. 6. Prove host-to-host forwarding

    Put both hosts in the same subnet (e.g. 192.168.10.11/24 and 192.168.10.12/24) on Vlan10 and prove end-to-end connectivity across the VXLAN overlay.

    Prove it at the data plane, not just the control plane: ping succeeds, and a large-frame test survives (validating MTU headroom for the VXLAN header). Capture on the underlay to see the encapsulated frame if you want direct proof.

    Acceptance: Host-a and host-b ping each other across the overlay.; Large-frame DF test passes (MTU headroom confirmed).; VXLAN (UDP/4789) frames observed on the underlay.

  7. 7. Trace one MAC end to end

    Pick host-a's MAC and follow it through every stage: (a) local learning on leaf1 (FDB in APPL_DB), (b) EVPN advertisement as a type-2 route by leaf1's bgpd, (c) reception and import on leaf2's BGP EVPN table, (d) installation as a remote MAC against the remote VTEP, and (e) the ASIC-facing FDB/tunnel object on leaf2.

    At the ASIC-facing stage, remember ASIC_DB is the sync path's representation of what syncd pushed — confirm real forwarding with counters or capture, not the DB entry alone.

    Acceptance: MAC observed at each stage: local FDB -> type-2 advertised -> type-2 received/imported -> remote-MAC installed -> ASIC-facing FDB object.; Forwarding confirmed with counters/capture, not the ASIC_DB entry alone.

  8. 8. Trace one route end to end

    Pick one prefix (a loopback or host route) and follow it top to bottom: FRR selects it (zebra/bgpd) -> it is mirrored into APPL_DB ROUTE_TABLE -> orchagent (swss) turns it into SAI calls -> syncd programs it and it appears in the ASIC-facing state -> traffic to it is forwarded.

    This is the routing counterpart to the MAC trace. It shows that 'present in FRR' and 'programmed in hardware' are different claims connected by several services, each a place state can stall.

    Acceptance: Route traced FRR -> APPL_DB -> (swss/syncd) -> ASIC-facing state -> forwarding.; Egress counters confirm traffic is actually forwarded to the prefix.

  9. 9. Diagnose at least three injected failures

    Have a partner (or the lab harness) inject failures from the failure catalog without telling you which. For each, work the diagnostic ladder from intent downward and record: the symptom, your isolation steps, the layer where you localized it, the root cause, and the fix.

    Diagnose at least three (the catalog provides six or more). Resist jumping to the overlay — many overlay symptoms are underlay or dependency problems in disguise.

    Acceptance: At least three failures diagnosed, each with symptom, isolation steps, localized layer, root cause, and corrective action.; Evidence captured (command output) supporting each root-cause claim.

  10. 10. Compare virtual vs physical enterprise SONiC (when available)

    If a physical SONiC switch (enterprise distribution) is available, reproduce key checks there and compare. Expect control-plane and workflow behavior to match, but expect hardware-dependent behavior to differ: CRM table limits, hardware counters, QoS/buffer behavior, and some SAI features that SONiC-VS only approximates.

    Document explicitly which of your capstone claims were validated on VS versus on real silicon. If no physical switch is available, state that and mark the hardware-dependent findings as VS-only / unverified on hardware.

    Acceptance: A comparison table of matching vs diverging behavior, or a documented note that no physical switch was available.; Each capstone claim tagged as verified-on-VS or verified-on-hardware.

  11. 11. Open-source vs enterprise recommendation

    Write a recommendation for a fictional organization — 'Meridian Freight', a mid-size logistics company running two data centers with a small (three-person) network team and no in-house NOS developers — on whether to adopt community SONiC or a commercially supported enterprise SONiC distribution.

    Justify the recommendation across concrete axes: operational risk, team staffing and skills, vendor support and escalation, feature/HWSKU certification, lifecycle/upgrade cadence, and total cost. Do not present one option as universally correct; tie the recommendation to this organization's constraints.

    Acceptance: A clear recommendation for the fictional org, tied to its constraints.; Justification covering risk, staffing/skills, support, feature/HWSKU certification, lifecycle, and cost.; Explicit acknowledgement of the trade-offs of the option not chosen.

Injected failures

Diagnose at least three. For each, record isolation steps, root cause, and fix.

⚠ Incorrect underlay address

Injected: A point-to-point underlay interface is given the wrong IP (e.g. wrong /31 pairing) so the two leaves are not on the same subnet.

Symptom: Underlay BGP never establishes; loopbacks unreachable; EVPN and the whole overlay are down.

Isolation:

  • Check underlay reachability and BGP before touching the overlay: `ping` the directly-connected neighbor address.
  • Compare the P2P addresses on both ends — confirm they are in the same subnet and correctly paired.
  • Confirm loopback-to-loopback reachability only after the P2P link is correct.

Root cause: Underlay reachability failure from a mis-addressed P2P link; the overlay cannot form over an unreachable VTEP.

Fix: Correct the interface address so both ends share the P2P subnet; confirm the underlay BGP session comes up and loopback-to-loopback ping succeeds.

⚠ BGP neighbor ASN mismatch

Injected: One leaf's neighbor statement configures the wrong remote-AS for the peer.

Symptom: BGP session will not reach Established; it flaps around OpenSent/OpenConfirm with an AS-mismatch notification.

Isolation:

  • Read `show ip bgp summary` — the session is stuck below Established.
  • Inspect `show bgp neighbors <peer>` for the last notification / reason (expected vs received AS).
  • Compare configured remote-AS on this leaf against the peer's actual local-AS.

Root cause: Configured remote-AS does not match the peer's local-AS, so the OPEN is rejected.

Fix: Set the correct remote-AS in the neighbor configuration; the session establishes and address families negotiate.

⚠ Missing VNI mapping

Injected: The Vlan10 -> VNI 10010 mapping is removed from one leaf only.

Symptom: The local host learns fine, but the remote leaf never installs its MAC and host-to-host traffic fails in one direction (or both).

Isolation:

  • Confirm the underlay and EVPN session are up (rule those out first).
  • Compare `show vxlan vlanvnimap` on both leaves — the mapping is present on one, absent on the other.
  • Correlate: the leaf missing the mapping neither advertises nor imports the L2VNI's type-2 routes.

Root cause: VLAN-to-VNI mapping absent on one side, so that leaf does not associate the VLAN with the VNI for advertisement/import.

Fix: Add the Vlan10 -> VNI 10010 mapping on the affected leaf; confirm remote-MAC install and bidirectional forwarding.

⚠ MTU mismatch

Injected: The underlay MTU is lowered below what the VXLAN overhead requires (no jumbo headroom).

Symptom: Small pings between hosts pass, but large frames / real application traffic fail after encapsulation.

Isolation:

  • Notice the tell-tale pattern: small ICMP works, DF large-frame test fails.
  • Run `ping -M do -s <large>` between hosts to bracket the failing size.
  • Compare underlay interface MTU against the ~50-byte VXLAN overhead requirement end to end.

Root cause: Underlay MTU too small to carry the original frame plus the VXLAN header, so encapsulated large frames are dropped.

Fix: Raise the underlay MTU (jumbo) end to end so encapsulated frames fit; re-run the large-frame DF test to confirm.

⚠ Remote MAC route absent

Injected: A route-target import/export mismatch is introduced on one leaf so type-2 routes are advertised but not imported.

Symptom: The remote host's MAC never appears as a remote MAC on the local leaf even though the EVPN session is up.

Isolation:

  • Confirm the EVPN session is Established (it is) so this is not a session problem.
  • On the originator, confirm the type-2 route is advertised: `show bgp l2vpn evpn route type macip`.
  • On the receiver, confirm the route arrives but is not imported — compare export RT on one side with import RT on the other.

Root cause: Route-target import/export mismatch prevents the receiving leaf from importing the advertised type-2 route into the VNI.

Fix: Align the route-targets (or fix the auto-RT/AS assumption) so import succeeds; the remote MAC installs and forwarding resumes.

⚠ Next-hop unresolved

Injected: The route to the remote VTEP loopback is removed/blackholed (e.g. loopback advertisement withdrawn), so the tunnel next-hop cannot resolve.

Symptom: EVPN routes are present and the remote MAC may even show, but encapsulated traffic to the remote VTEP fails because the tunnel endpoint is unreachable.

Isolation:

  • Check reachability to the remote VTEP loopback: `show ip route <remote-loopback>` and ping it.
  • Confirm the tunnel next-hop is unresolved while the overlay control plane looks otherwise healthy.
  • Correlate the loss with the missing/withdrawn underlay route to the loopback.

Root cause: The VXLAN tunnel's next-hop (remote VTEP loopback) is unresolved because the underlay route to it is gone; the overlay cannot forward over an unreachable endpoint.

Fix: Restore the underlay route/advertisement to the remote VTEP loopback so the tunnel next-hop resolves; confirm encapsulated traffic passes.

⚠ Service stopped

Injected: A core container (e.g. swss) is stopped on one leaf.

Symptom: Configuration changes and control-plane updates on that leaf stop taking effect in hardware; forwarding may degrade depending on what was already programmed.

Isolation:

  • Check service/container health first: `docker ps` and `systemctl status <service>` show the container down.
  • Observe that new objects reach APPL_DB (or FRR) but are not reflected in the ASIC-facing state.
  • Read the container logs for the stop/exit reason before restarting anything.

Root cause: A stopped orchestration/sync service breaks the path between desired state and hardware programming on that leaf.

Fix: Start the service and confirm it reconciles state to hardware. Note: restarting core containers can disrupt forwarding depending on version and warm-restart configuration — do it in a maintenance window on production.

⚠ Hardware resource issue (conceptual)

Injected: Conceptual scenario (not faithfully reproducible on SONiC-VS): the ASIC route/next-hop or FDB table is driven to exhaustion by excess scale.

Symptom: New routes/MACs are present in FRR/APPL_DB but stop appearing in the ASIC-facing state; forwarding for new objects silently fails while existing entries keep working.

Isolation:

  • Inspect CRM: `crm show resources all` shows a specific table at/near its limit.
  • Check syncd logs for table-full / SAI insufficient-resources status when new objects are pushed.
  • Confirm the new objects exist in APPL_DB but are missing from the ASIC-facing state.

Root cause: A hardware table (routes, next-hops, FDB, or ACL/TCAM) is exhausted, so syncd cannot program further objects — a resource ceiling, not a protocol bug. On SONiC-VS this is conceptual because VS does not model real ASIC table limits.

Fix: Reduce demand (route summarization, ACL consolidation, ECMP reduction) or re-profile the ASIC table allocation for the switch's role; re-verify programming on real hardware, since VS will not reproduce the limit.

Capstone report template

Copy this into your notes and fill it in as you go.

report template (Markdown)
# SONiC EVPN/VXLAN Capstone Report

> Fill in each section with your own evidence. Every claim about hardware
> forwarding should cite a counter or capture, not just a database entry —
> ASIC_DB is the sync-path representation of what syncd pushed, not proof the
> silicon forwards. Note the image version/distribution and whether each
> result came from SONiC-VS or physical hardware.

## Topology
- Diagram / description of the two-leaf fabric and two hosts.
- Underlay: routed P2P links, per-leaf VTEP loopbacks, MTU (with VXLAN headroom).
- Overlay: BGP EVPN (l2vpn evpn), Vlan10 <-> L2VNI 10010, host subnet.
- Image version and distribution per leaf; virtual vs physical.

## Configuration summary
- Underlay addressing and loopbacks.
- BGP design (eBGP vs iBGP+RR), ASNs, address families.
- VTEP source, VLAN-to-VNI mapping, route-distinguishers and route-targets.
- EVPN advertisement (type-2 MAC/IP, type-3 IMET).

## Validation evidence
- Underlay/loopback reachability.
- BGP and EVPN session state.
- Tunnel and VNI state; remote MAC/IP learned.
- Host-to-host forwarding proof (ping, large-frame DF test, VXLAN capture).

## Route trace
- FRR (zebra/bgpd) selection -> APPL_DB ROUTE_TABLE -> swss/orchagent ->
  syncd/SAI -> ASIC-facing state -> forwarding.
- Cite the evidence at each hop; confirm forwarding with counters.

## Packet trace
- One MAC end to end: local FDB learning -> EVPN type-2 advertisement ->
  reception/import on the remote leaf -> remote-MAC installation ->
  ASIC-facing FDB/tunnel object.
- Outer frame (Ethernet / IP / UDP-4789 / VXLAN+VNI) -> inner frame, from the
  underlay capture.

## Failure findings
- For each injected failure: symptom observed and where you localized it on
  the diagnostic ladder.

## Root causes
- The root cause for each failure (intent, config, service, protocol,
  dependency, hardware, etc.).

## Corrective actions
- The fix applied for each failure and how you verified recovery.

## Open-source vs enterprise recommendation
- Recommendation for the fictional org (Meridian Freight) and its constraints.
- Justification: operational risk, staffing/skills, vendor support, feature and
  HWSKU certification, lifecycle/upgrade cadence, and cost.
- Trade-offs of the option not chosen.

## Lessons learned
- What transferred directly from CCNP-level knowledge.
- What was genuinely new (database-driven state, containers, SAI/ASIC path).
- Where virtual-lab behavior diverged from (or could not represent) real
  hardware, and what you would only certify on physical silicon.