Hardware Resource Limits
Every hardware table is finite: routes, MACs, neighbors, ACL/TCAM entries, ECMP groups, tunnels, counters, and buffers. Because these draw on a shared, ASIC-specific pool, features interact — enlarging one can shrink another. Exhaustion produces distinctive symptoms (programming failures, silent non-installation, punting) that look nothing like a config error. Knowing the limits and their symptoms is core to operating SONiC at scale.
Prerequisites: Lesson 3.2 · Lesson 3.4
Learning objectives
- →Enumerate the major finite hardware resources and what consumes them.
- →Explain feature interaction: how enabling/enlarging one resource reduces another.
- →Describe how boot-time profiles/templates carve shared memory.
- →Recognize resource-exhaustion symptoms and distinguish them from config errors.
- →Explain why VS cannot be trusted for scale/limit behavior.
Scale problems are silent and nasty: the config is accepted, but entries never reach hardware, so forwarding is wrong for reasons invisible in CONFIG_DB.
Capacity planning and 'why did adding X break Y' questions both reduce to the shared-resource model in this lesson.
What your CCNP already gives you
- • You know TCAM/FIB/MAC scale limits and SDM-style templates from traditional switches.
- • You have seen 'hardware resource exceeded' style failures and their subtlety.
What is new in SONiC
- • Resource usage is observable (e.g., CRM — the Critical Resource Monitor — tracks utilization and thresholds).
- • The trades are explicit and often profile-driven, and you can watch programming fail at the SAI/ASIC_DB boundary.
- • VS does not model real capacities — scale lessons must be validated on hardware (Lab 3.6).
| In the analogy | In SONiC |
|---|---|
| Shelves of fixed total size | Hardware tables |
| Different product lines competing for shelves | Routes / MACs / ACLs / tunnels |
| The shelving plan chosen when the warehouse opens | Boot-time profile/template |
| A pallet that arrives with nowhere to go — left on the dock (never installed) | Exhaustion |
Where the analogy stops working
Real ASICs partition memory in complex, sometimes algorithmic ways (shared banks, hash efficiency, prefix-length effects), so 'shelves' is a simplification. Exact capacities and partitioning are vendor/platform-specific — always consult the platform's numbers, not a generic figure.
The finite resources and what consumes them
| Resource | Consumed by | Exhaustion tends to look like |
|---|---|---|
| Route table (LPM) | IPv4/IPv6 prefixes; IPv6 and long prefixes cost more | New routes not installed in hardware though present in FRR/APPL_DB |
| Host / neighbor table | /32 & /128 hosts, ARP/ND entries | Neighbors fail to program; intermittent reachability to hosts |
| MAC / FDB table | Learned + static MACs (per VLAN) | Learning stalls; excessive flooding as entries can't be added |
| ACL / TCAM | ACL rules, PBR, some classification | ACL rules not programmed; policy silently not enforced |
| ECMP / next-hop groups | Number of groups and members | ECMP falls back to fewer paths; imbalance or blackhole on scale-out |
| Tunnels | VXLAN/IP tunnel endpoints, tunnel maps | Overlay endpoints not installed; EVPN paths missing |
| Counters | Per-object flex counters (ports, queues, ACLs) | Counter allocation limits; enabling more polling costs resources/CPU |
| Buffers | Ingress/egress buffer pools, per-queue reservations | Tail drops, PFC/PG issues under congestion; not a 'count' but a pool |
Feature interaction and boot-time profiles
Because tables share on-chip memory, they interact. Common trades: a larger ACL profile reduces route/MAC space; enabling many tunnels or large ECMP consumes shared entries; turning on more counters/flex-counter polling costs allocation and CPU. Some of these are dynamic; many are governed by a profile/template selected at boot that carves the shared memory a particular way. Changing that carving usually requires a reload and is a deliberate capacity-planning decision, not a runtime tweak.
This is why 'I enabled feature X and unrelated feature Y broke' is a routine, expected class of problem on ASICs — and why you plan the resource budget as a whole, not feature by feature.
Symptoms of exhaustion (and why they fool people)
Exhaustion rarely announces itself as a clean error at the CLI. The config is accepted (it is valid), CONFIG_DB and often APPL_DB show the intent, but the object never lands in ASIC_DB / hardware — or lands with a failure logged deep in swss/syncd. The result is a forwarding gap with a healthy-looking control plane: exactly the trap Course Principle #2 warns about.
Tell-tale signs: routes/MACs/ACLs present in software but missing in ASIC_DB; syslog messages from orchagent/syncd about SAI failures or table-full; forwarding that works up to a scale threshold then degrades; or a new feature's objects failing to install while old ones are fine. SONiC's CRM (Critical Resource Monitor) exists precisely to watch utilization and warn before you hit the wall.
Illustrative: checking resource utilization
# Critical Resource Monitor — used vs available per resource:
crm show resources all
# Resource Used Available
# ipv4_route 48213 ...
# ipv6_route 9022 ...
# fdb_entry 31110 ...
# ipv4_neighbor 12004 ...
# acl_entry 384 ...
# nexthop_group ... ...
# CRM thresholds (when to warn) — illustrative:
crm show thresholds all
# Cross-check: is a route in FRR/APPL_DB but NOT in ASIC_DB?
redis-cli -n 1 SCAN 0 MATCH 'ASIC_STATE:SAI_OBJECT_TYPE_ROUTE_ENTRY*' COUNT 500 | wc -lIllustrative; exact command surface and thresholds vary by image/ASIC. CRM tracks used/available and can log/alert on thresholds.
SONiC-VS emulates behavior and does not reproduce a physical ASIC's table sizes, partitioning, or exhaustion behavior. You can learn the *concepts* and the *observation workflow* on VS, but any real scale/limit claim must be validated on the intended hardware (Lab 3.6). Do not size a production design from VS numbers.
Common misconceptions
Myth: If the CLI accepted it, it's in hardware.
Reality: Acceptance writes intent. Under exhaustion the object may never reach ASIC_DB/hardware. Verify installation, not just acceptance.
Myth: Each feature has its own independent capacity.
Reality: Features share on-chip memory. Enlarging one can shrink another, often governed by a boot-time profile.
Myth: One capacity number fits the platform family.
Reality: Capacities and partitioning vary by ASIC, generation, and profile. Use the specific platform's numbers and CRM utilization.
Go deeperIPv6 and prefix-length effects on route scale
Route capacity is not a single scalar. Longer prefixes and IPv6 often consume more table space per entry than IPv4 aggregates, and some LPM implementations are sensitive to the *distribution* of prefix lengths. A table rated for N IPv4 routes may hold far fewer IPv6 or many-/32 routes. When planning scale, budget by the actual prefix mix and validate on hardware — a generic 'route count' can badly mislead.
- Route/MAC/ACL present in software but missing in ASIC_DB → suspect exhaustion; check CRM utilization and swss/syncd logs for SAI/table-full errors.
- 'Enabling X broke unrelated Y' → shared-resource trade or a profile that recarved memory; treat as capacity planning, not a bug.
- Forwarding that degrades past a scale threshold points at a table limit, not a config change.
Key takeaways
- ✓Routes, hosts/neighbors, MACs, ACL/TCAM, ECMP groups, tunnels, counters, and buffers are all finite and often shared.
- ✓Features interact: enlarging one resource can shrink another, frequently via boot-time profiles.
- ✓Exhaustion is quiet — intent is accepted but objects never reach hardware; use CRM and ASIC_DB cross-checks to catch it.
- ✓Capacities and partitioning are ASIC-specific; VS does not model them — validate scale on real hardware.
Tested against: Concepts; figures are ASIC-specific · Upstream; capacities vary by vendor/platform · Real limits only on hardware; VS does not model them faithfully · last reviewed 2026-07. Exact commands and behavior can vary by SONiC release, image, hardware platform, and enterprise distribution.
Knowledge check
No config syntax errors; control plane healthy.
After enabling a large ACL profile, some newly advertised routes stop being installed in hardware even though FRR and APPL_DB show them and the CLI accepted everything. Most likely cause?