The Switching Pipeline
Packets move through a sequence of match-action stages backed by different memory technologies: exact-match tables in SRAM/hash memory, longest-prefix-match tables for routes, and TCAM for wildcard matches like ACLs. Stages carry metadata forward, can recirculate a packet, and draw on shared resources — which is why turning on one feature can reduce another's capacity. Ordering and capabilities vary by ASIC.
Prerequisites: Lesson 3.1
Learning objectives
- →Describe the match-action model and how stages pass metadata down the pipeline.
- →Distinguish exact-match tables, LPM tables, and TCAM, and the memory each typically uses.
- →Explain recirculation and when a packet makes more than one pass.
- →Explain resource sharing and why enabling one feature can reduce capacity for another.
- →State clearly that stage ordering and capabilities vary by ASIC.
Nearly every forwarding decision, ACL hit, ECMP choice, and resource-exhaustion symptom in the rest of the course maps back to a stage in this pipeline.
Knowing which memory backs which table tells you why routes, MACs, and ACLs scale to very different numbers and why they compete.
What your CCNP already gives you
- • You know ACLs use TCAM and that TCAM is a limited resource on traditional switches.
- • You understand longest-prefix match for route selection.
- • You have seen 'hardware table full' style errors and TCAM carving/templates (e.g., SDM templates) on other platforms.
What is new in SONiC
- • You will see these tables as SAI objects in ASIC_DB and correlate them to pipeline stages.
- • Resource sharing is explicit: profiles and features visibly trade capacity.
- • The pipeline model is vendor-neutral in SONiC's abstraction, but the real chip's ordering still governs behavior.
| In the analogy | In SONiC |
|---|---|
| One station on the line doing a single job | A pipeline stage |
| A traveler card riding along with each item | Metadata carried between stages |
| A bin indexed by an exact part number — instant grab | Exact-match table (SRAM/hash) |
| A sorting station that files by the longest matching zip-code prefix | LPM route table |
| A flexible inspector that matches on patterns with wildcards — powerful but expensive and few in number | TCAM |
| Sending an item back to the top of the line for a second pass | Recirculation |
Where the analogy stops working
A real line has a single fixed order; ASIC pipelines vary — stages may be merged, reordered, or programmable, and the same logical step can live in different physical places on different chips. Treat this lesson as a model for reasoning, not a literal blueprint of your silicon.
Match-action, stage by stage
A pipeline is a chain of match-action stages. Each stage *matches* on some fields (from the packet or from metadata) and *acts* — set a field, choose a next table, drop, mark, or forward. Conceptually, packets flow: parse → ingress processing (VLAN, L2/L3 lookups, ACLs) → forwarding decision → egress processing (rewrites, egress ACLs) → queueing/scheduling → transmit.
The key idea is that stages do not re-parse the whole packet each time. The parser extracts headers once into a structured representation, and stages read/write metadata — a scratchpad traveling with the packet — so a decision made early (e.g., 'this is VLAN 20, routed, next-hop resolved to port Ethernet8') is available to later stages cheaply.
Three table types, three memory technologies
| Table type | What it matches | Typical memory | Scales to | Classic use |
|---|---|---|---|---|
| Exact-match | A full, exact key (e.g., MAC+VLAN, /32 host) | SRAM via hashing | Large (hundreds of thousands) | FDB/MAC table, host routes, neighbors |
| LPM (longest prefix match) | Longest matching prefix of a key | Specialized LPM memory / algorithmic (often SRAM-backed) | Large, but prefix-length dependent | IPv4/IPv6 route table (FIB) |
| TCAM (ternary CAM) | Match with wildcards/masks, any field | TCAM (power- and area-expensive) | Small (thousands) | ACLs, PBR, some classification |
Why TCAM is precious
TCAM can match any bit as 0, 1, or 'don't care', which is exactly what wildcard ACL rules need — but that flexibility costs silicon area and power, so there is comparatively little of it. That is why ACL scale is measured in thousands while MAC/route scale reaches hundreds of thousands. It is also why ACL design (ordering, consolidation, ranges vs exact) has outsized impact on a switch.
Exact-match tables (hash-indexed SRAM) are cheap per entry and huge, which is why MAC and host tables are large. LPM memory is specialized to answer 'longest matching prefix' efficiently; its effective capacity can depend on the mix of prefix lengths (IPv4 vs IPv6, /24s vs /32s).
Metadata and recirculation
Metadata is how stages cooperate without re-doing work — the parsed headers, the resolved VLAN, the forwarding result, drop/trap flags, and the chosen queue all ride along.
Recirculation sends a packet back through (part of) the pipeline for a second pass when a single pass is not enough — for example some tunnel encap/decap, certain double lookups, or features that need to act on a rewritten packet. Recirculation consumes pipeline bandwidth, so heavy use of recirculation-based features can reduce effective throughput. Whether and how a given feature recirculates is ASIC-specific.
Enabling or enlarging one feature can reduce the resources available to another. A bigger ACL profile can shrink route or MAC capacity; more ECMP groups or tunnels can compete with host entries. On many platforms these trades are controlled by profiles/templates chosen at boot — changing them may require a reload.
Ordering and capabilities vary by ASIC
Everything above is a reasoning model. On real silicon the number of stages, their order, which are programmable, how tables are partitioned, and whether a feature recirculates all differ by chip family and generation. Two ASICs can implement the same SONiC feature with different pipeline placements and therefore different interactions and limits.
Practical consequence: when a feature behaves oddly or a scale number surprises you, the explanation often lives in *this specific ASIC's* pipeline, not in a universal rule. Confirm against the platform's documentation and its SAI capability set.
Common misconceptions
Myth: TCAM and SRAM are interchangeable; the ASIC uses whatever is free.
Reality: Different tables are backed by different memories with different match semantics. ACLs need TCAM; you cannot spend spare MAC-table SRAM as ACL space.
Myth: The pipeline order is the same on every switch.
Reality: Ordering and capabilities vary by ASIC. The model in this lesson is for reasoning, not a datasheet for your chip.
Myth: Enabling a feature is free if the config applies cleanly.
Reality: Config acceptance does not reveal resource cost. A feature can apply and quietly reduce headroom elsewhere until something overflows later.
- 'ACL won't program' with routes/MACs fine often means TCAM exhaustion, not a config error — check the ACL/TCAM resource, not CONFIG_DB.
- A scale surprise (fewer routes than expected) can be a profile/template choice that traded route space for another feature at boot.
- Behavioral differences between two platforms running the same config are frequently pipeline-ordering differences — consult the ASIC's docs.
Key takeaways
- ✓Pipelines are chains of match-action stages that pass metadata forward instead of re-parsing.
- ✓Exact-match (SRAM) scales large and cheap; LPM handles routes; TCAM handles wildcard matches but is small and expensive.
- ✓Recirculation gives a second pass at the cost of pipeline bandwidth.
- ✓Tables share a finite pool, so features compete — often governed by boot-time profiles.
- ✓Stage ordering and capabilities vary by ASIC; this is a model, not a blueprint.
Tested against: Concepts, ASIC-independent · Upstream concepts · Model only; real pipelines differ by silicon · last reviewed 2026-07. Exact commands and behavior can vary by SONiC release, image, hardware platform, and enterprise distribution.
Knowledge check
Your ACL rules fail to program while routes and MACs are fine. Which explanation is most consistent with how pipeline memories work?