SONiC
Module 3 · ASICs, Switching Pipelines, SAI & Forwarding

Control Plane Healthy, Forwarding Broken

60 minLabTroubleshootingCorrelationVirtualOpen-source

BGP is established and the route is visible, yet traffic fails. Use a layered flow — following the packet from control-plane presence down through APPL_DB, ASIC_DB, next hop, neighbor, and counters — to isolate the exact break, proving that a healthy control plane does not guarantee forwarding.

Safety: Caution

May change state — take a snapshot first.

If self-injecting a fault, do it on a disposable VS and keep a snapshot; do not practice fault injection on shared systems.

VS reproduces the methodology, not real ASIC fault behavior — validate hardware-specific faults on hardware (Lab 3.6).

VS with an eBGP peer and a target prefix

  peer ══ eBGP ══ SONiC-VS ── Ethernet? ─→ target prefix (traffic FAILS)
             (route present, forwarding broken — isolate why)

Objective

  • • Practice the layered, packet-order isolation method for a forwarding-broken scenario.
  • • Prove which layer holds the break using targeted evidence at each step.
  • • Internalize Course Principle #2: control-plane health ≠ correct forwarding.

Starting state

  • • BGP established; target prefix present in show ip route.
  • • Traffic to the prefix fails (loss/timeout).

Prerequisites: Lessons 3.4, 3.5, and 3.7 complete. · An established BGP session with at least one learned prefix whose traffic currently fails (instructor-injected, or self-injected by removing the neighbor/adjacency for a next hop).

Tasks

  1. 1

    Confirm the control plane really is healthy

    Rule the control plane in as 'healthy' so the rest of the investigation is honest.

    FRR / vtysh
    show ip bgp summary
    SONiC CLI
    show ip route <target-prefix>
    Expected:
    • BGP Established; the target prefix present with a next-hop IP and egress interface.

    What happened internally: This step deliberately establishes that intent is correct — so any failure below is a programming/forwarding issue, not routing.

  2. 2

    Layer 1 — is intent in APPL_DB?

    SONiC DB
    redis-cli -n 0 SCAN 0 MATCH 'ROUTE_TABLE:<target-prefix>' COUNT 100
    SONiC DB
    redis-cli -n 0 HGETALL 'ROUTE_TABLE:<target-prefix>'
    Expected:
    • The prefix present in APPL_DB with a nexthop and ifname.

    What happened internally: If it were missing here, the break would be FRR→APPL_DB (fpmsyncd). Presence pushes the investigation downward.

  3. 3

    Layer 2 — did it reach ASIC_DB?

    SONiC DB
    redis-cli -n 1 SCAN 0 MATCH 'ASIC_STATE:SAI_OBJECT_TYPE_ROUTE_ENTRY*<target-prefix>*' COUNT 500
    Expected:
    • If PRESENT: the break is below (next hop/neighbor/egress).
    • If ABSENT: the break is the swss→syncd path or resource exhaustion (CRM, orchagent/syncd logs).

    What happened internally: This single check splits the problem space in half — the highest-value step in the flow.

  4. 4

    Layer 3 — next hop and neighbor resolution

    SONiC CLIis the adjacency resolved?
    show arp | grep <next-hop-ip>
    SONiC DB
    redis-cli -n 1 SCAN 0 MATCH 'ASIC_STATE:SAI_OBJECT_TYPE_NEIGHBOR_ENTRY*' COUNT 500
    Expected:
    • If the neighbor is unresolved: this is the 'inert route' — route present, no dst MAC to rewrite. Prime suspect.

    What happened internally: The most common healthy-control-plane failure. A present route with an unresolved neighbor forwards nothing.

  5. 5

    Layer 4 — egress port and counters

    SONiC CLI
    show interfaces status | grep <egress-port>
    SONiC CLIare tx/rx and drops moving where expected?
    show interfaces counters
    Expected:
    • Egress port up; counters reveal whether frames leave, or are dropped, or never reach egress.

    What happened internally: Counters localize the final leg: transmitting vs dropping vs never-arriving distinguishes egress/queue problems from upstream ones.

  6. 6

    Name the break and verify the fix

    State the single layer at which the packet's journey stopped, apply the corresponding fix (e.g., restore the adjacency), and re-test end to end.

    Linuxconfirm forwarding restored
    ping -c 3 <target-in-prefix>
    Expected:
    • Traffic succeeds once the identified layer is corrected.

    What happened internally: Closing the loop confirms the isolation was correct — the discipline is 'prove the layer, then fix the layer'.

Troubleshooting branches

If: Route present in APPL_DB but absent in ASIC_DB

Then: swss→syncd stall or exhaustion — check orchagent/syncd logs and CRM (Lessons 3.4/3.8).

If: Route + next hop in ASIC_DB but neighbor unresolved

Then: Adjacency/ARP problem — the inert-route pattern; fix resolution, not routing.

If: Everything programmed but counters show egress drops

Then: Egress/queue/buffer or an egress ACL — inspect port and queue counters, not the RIB.

Cleanup / rollback
  • If you self-injected the fault, restore the neighbor/config; revert to 'baseline' if in doubt.

Lab evidence checklist

  • ☐ Evidence the control plane is healthy (BGP established, route present).
  • ☐ APPL_DB presence check captured.
  • ☐ ASIC_DB presence check captured (the halving step).
  • ☐ Neighbor-resolution evidence captured.
  • ☐ Egress/counter evidence captured.
  • ☐ A one-line statement of the exact layer that held the break.

Reflection

  • Which single check split the problem space most effectively, and why?
  • How did this scenario demonstrate that a healthy control plane does not guarantee forwarding?
  • If you had started at the route table instead of following the packet, how much longer would isolation have taken?
Lab notes & observations