SONiC

Module 3

ASICs, Switching Pipelines, SAI & Forwarding

How a logical route, MAC, ACL, or tunnel becomes real behavior in silicon.

~8 h4.5 h lecture3 h labs8 lessons6 labs10 check questions

Purpose

Teach how a logical route, MAC entry, ACL, or tunnel becomes actual forwarding behavior inside switch silicon.

This is the central module of the course and receives the strongest visual treatment: pipeline, SAI object model, and full packet walks.

Learning objectives

  • →Explain what a switching ASIC is and how hardware forwarding differs from software routing.
  • →Walk a conceptual switching pipeline and reason about match-action, table types (LPM, exact-match, TCAM/SRAM), and resource sharing.
  • →Describe SAI's object model and how create/remove/set/get map onto routes, next-hops, FDB, ACLs, and tunnels.
  • →Explain SWSS, orchagents, syncd, and the true role of ASIC_DB in the synchronization path.
  • →Trace a full routed packet and a full bridged packet, and reason about CPU-bound exception traffic and hardware resource limits.

Lessons

  1. What a Switching ASIC IsA switching ASIC forwards at line rate using fixed-function and programmable stages, purpose-built lookup tables, and a finite pool of hardware resources. This is fundamentally different from routing packets in software on a CPU, and understanding the difference is the foundation for everything else in the course.50m
  2. The Switching PipelinePackets move through a sequence of match-action stages backed by different memory technologies: exact-match tables in SRAM/hash memory, longest-prefix-match tables for routes, and TCAM for wildcard matches like ACLs. Stages carry metadata forward, can recirculate a packet, and draw on shared resources — which is why turning on one feature can reduce another's capacity. Ordering and capabilities vary by ASIC.55m
  3. SAI — The Switch Abstraction InterfaceSAI is the vendor-neutral API SONiC uses to program forwarding silicon. It models the switch as objects (ports, router interfaces, routes, next hops, neighbors, FDB entries, VLANs, ACLs, tunnels, queues) with attributes, manipulated through create/remove/set/get. SAI standardizes the request; it does NOT make every ASIC identical — supported attributes, limits, and behaviors still vary.55m
  4. SWSS, Orchagents, syncd & ASIC_DBSWSS orchestration agents consume desired state from the databases, resolve dependencies, and issue SAI calls; syncd carries those calls to the vendor SAI implementation and SDK, which programs the chip and reports back. ASIC_DB is the database representation in that synchronization path — a mirror of what has been asked of/acknowledged by SAI — NOT the silicon itself.55m
  5. Full Routed Packet WalkFollow a single routed packet from ingress port to egress transmission: header parse, VLAN context, router-interface association, destination route lookup, next-hop resolution, neighbor adjacency, source/destination MAC rewrite, TTL decrement and checksum update, egress selection, queueing, and transmit. Each step maps to a SAI object you can inspect.55m
  6. Full Bridged Packet WalkFollow an L2 packet: source-MAC learning into the FDB, destination FDB lookup, and the three delivery cases — known unicast (forward to one port), unknown unicast and broadcast (flood within the VLAN), and multicast (constrained or flooded). Egress filtering (split-horizon, STP/VLAN state) prunes where the frame may go. Bridging does not rewrite MACs or touch TTL.50m
  7. Forwarding Exceptions — Traffic to the CPUSome packets cannot or must not be handled by the fast path and are trapped (punted) to the CPU: ARP/ND, routing-protocol packets, packets to the switch itself, TTL-expired packets, and destinations with no resolved adjacency. Control-plane policing (CoPP) rate-limits these traps so the CPU (and control plane) survive. Trapped traffic is the slow path — powerful but scarce.50m
  8. Hardware Resource LimitsEvery hardware table is finite: routes, MACs, neighbors, ACL/TCAM entries, ECMP groups, tunnels, counters, and buffers. Because these draw on a shared, ASIC-specific pool, features interact — enlarging one can shrink another. Exhaustion produces distinctive symptoms (programming failures, silent non-installation, punting) that look nothing like a config error. Knowing the limits and their symptoms is core to operating SONiC at scale.55m

Labs

  1. Route Object CorrelationTake a single installed route and trace it through the SAI object graph — route entry → next hop (or next-hop group) → router interface → neighbor → port — building a relationship graph that shows exactly what must exist for the route to forward.CorrelationVirtualOpen-source60m
  2. MAC Learning and FDBGenerate L2 traffic, watch MACs get learned, inspect the FDB, clear an entry and observe relearning, and compare the software-visible MAC state against forwarding behavior — surfacing the gap between what `show mac` reports and what the data plane does.ObservationPacket tracingVirtualOpen-source55m
  3. Failure Injection: SWSSOn a disposable virtual switch only, baseline the propagation path, disrupt the orchestration service (swss), make a config/route change while it is down, observe exactly where propagation stops, then restore and determine whether the intended state reconciles into hardware.Failure injectionVirtualTroubleshootingOpen-source60m
  4. ASIC_DB ExplorationUse targeted queries (SCAN, never KEYS *) to locate route- and next-hop-related objects in ASIC_DB, follow their OID references, translate virtual OIDs to real ones, and articulate which parts are human-readable versus implementation-oriented — reinforcing that ASIC_DB is a sync-path representation, not the silicon.ObservationCorrelationVirtualOpen-source50m
  5. Control Plane Healthy, Forwarding BrokenBGP is established and the route is visible, yet traffic fails. Use a layered flow — following the packet from control-plane presence down through APPL_DB, ASIC_DB, next hop, neighbor, and counters — to isolate the exact break, proving that a healthy control plane does not guarantee forwarding.TroubleshootingCorrelationVirtualOpen-source60m
  6. Physical Switch ValidationOn dedicated lab hardware only, identify the vendor/model/SONiC distribution/ASIC family, compare the command surface, ASIC/platform visibility, and hardware counters against SONiC-VS, validate a real route and MAC in hardware, and document the concrete differences between virtual and physical SONiC.PhysicalObservationCorrelationEnterprise75m

Module summary

  • SAI standardizes the request; it does not erase hardware differences. ASIC_DB is the synchronization representation, not the hardware itself.
  • Forwarding is where truth lives: when every 'show' looks clean but traffic fails, the intent stopped somewhere between SAI and silicon.

Completion checklist

  • ☐ I correlated a route to its next-hop, router-interface, neighbor, and port objects.
  • ☐ I observed MAC learning and FDB behavior and compared software-visible state with forwarding.
  • ☐ I injected a SWSS disruption and located where propagation stopped.
  • ☐ I explored ASIC_DB with targeted queries and identified object references.
  • ☐ I isolated a 'control plane healthy, forwarding broken' scenario with a layered flow.
Carry this forward

Finish the labs before moving on — later modules assume you can navigate the databases and containers hands-on, not just in theory.