Control Plane vs Data Plane
A route can be perfectly valid in BGP and still forward nothing. This lesson catalogs the reasons that gap opens in SONiC: unresolved recursive next-hops, missing neighbor/interface dependencies, ASIC resource exhaustion, orchagent/syncd sync failures, and stale or inconsistent state — and how to tell them apart.
Prerequisites: Lesson 2.3 (route lifecycle). · Lesson 2.2 (route-state layers).
Learning objectives
- →Explain the structural reasons a control-plane route may not become a forwarding entry.
- →Diagnose recursive next-hop resolution failures.
- →Recognize hardware-resource exhaustion and CRM/threshold signals.
- →Distinguish a genuine software sync failure from expected transient skew.
'BGP looks fine but traffic is black-holed' is one of the highest-value skills in a disaggregated NOS. The control plane lying to you is the whole reason to inspect lower layers.
Each cause has a different fix. Misreading resource exhaustion as a BGP problem wastes an outage window.
What your CCNP already gives you
- • Recursive next-hop resolution and the idea of an unresolved next-hop.
- • TCAM/FIB scale limits from traditional platforms — the concept transfers.
- • The habit of separating 'route present' from 'route usable.'
What is new in SONiC
- • Resource limits are observable via CRM (Critical Resource Monitor) thresholds and counters, not just vendor syslog.
- • Sync failures live at the orchagent/syncd boundary and surface in logs and DB divergence.
- • Stale state is a first-class failure mode because multiple stores must converge.
| In the analogy | In SONiC |
|---|---|
| Perfectly good written directions | Valid BGP route |
| Directions that say 'take the highway' but you never found the on-ramp | Unresolved recursive next-hop |
| The road exists but the bridge to it is out | Missing neighbor/interface |
| The nav unit's memory is full — it refuses to store one more route | ASIC table full |
| The installer accepted the job but never finished writing it to the unit | Sync failure |
| Old directions still loaded after the road changed | Stale state |
Where the analogy stops working
Directions on paper are inert; SONiC's layers actively retry. Many of these conditions self-heal when the missing neighbor resolves or resources free up, so a snapshot showing a gap is not automatically a permanent fault — you must observe whether it *persists* and whether retries are progressing. Also, hardware-resource exhaustion has no meaningful analog on SONiC-VS; you need real silicon (and CRM counters) to see it.
Five reasons a valid route does not forward
| Cause | What you see | Where to look | Typical fix |
|---|---|---|---|
| Recursive next-hop unresolved | Route in zebra/APPL_DB; next-hop has no resolving route/neighbor; no/incomplete ASIC object | show ip route <nh>, NEIGH_TABLE, APPL_DB route flags | Restore the route/interface that resolves the next-hop |
| Neighbor / interface dependency | Route pending; NEIGH entry missing or interface oper-down | show ip interfaces, ip neigh, STATE_DB PORT/NEIGH | Fix L2/ARP/interface; neighbor resolution then unblocks the route |
| Hardware resource exhaustion | APPL_DB has the route; ASIC_DB missing it; CRM thresholds exceeded; syncd errors | crm show resources all, syslog/syncd, ASIC_DB counts | Reduce scale, adjust route summarization, or move to higher-scale HWSKU |
| Software sync failure | Persistent APPL_DB ↔ ASIC_DB divergence; orchagent/syncd errors or restart loops | docker ps, orchagent/syncd logs, show logging | Investigate the failing service; may need warm/cold restart per runbook |
| Stale / inconsistent state | Old entry persists after withdrawal, or mismatched entries across stores | Compare zebra vs APPL_DB vs ASIC_DB for the prefix | Identify the store that failed to converge; targeted reconcile/restart |
Recursive next-hop resolution, the usual suspect
BGP routes frequently point at a next-hop that is itself reachable only via another route (classic IBGP next-hop, or a leaked/aggregate route). SONiC must resolve that recursion: the route's immediate next-hop IP has to map to a directly connected next-hop with a resolved neighbor before a hardware next-hop object can exist. If the resolving route disappears or its neighbor never resolves, the dependent route stays valid in BGP but cannot be programmed. Confirm by resolving the next-hop yourself: show ip route <next-hop-ip> and check whether *that* has a real adjacency.
Hardware resource exhaustion is real and observable
Merchant silicon has finite route/next-hop/neighbor/ECMP-group capacity, and different HWSKUs and ASIC families differ widely — do not assume a scale number from one platform applies to another. SONiC exposes the CRM (Critical Resource Monitor), which tracks used/available counts and can raise threshold alarms per resource (IPv4/IPv6 routes, next-hops, neighbors, next-hop groups, ACLs, FDB, and more). When a resource is exhausted, orchagent may hold routes or SAI create calls fail — the route sits in APPL_DB with no ASIC_DB object. This failure mode simply does not appear on SONiC-VS; you need real hardware to test it.
Check hardware resource headroom (read-only, illustrative)
crm show resources all
# Resource Name Used Available
# ipv4_route 48213 1787 <- nearly full
# ipv4_nexthop 2011 2085
# ipv4_neighbor 2011 2085
# nexthop_group 1015 9
# nexthop_group_member 8120 72
crm show thresholds all # how alarms are configured
show logging | grep -i crm # threshold-exceeded eventsCRM shows used vs available per resource and current thresholds. Numbers are platform-specific and illustrative here.
Sync failure vs. expected transient skew
During convergence, APPL_DB and ASIC_DB briefly disagree — that is normal. The tell for a genuine software sync failure is *persistence and non-progress*: the divergence does not close, and you find orchagent or syncd throwing errors, restart-looping, or a full queue. Check container health (docker ps), the orchagent/syncd logs, and whether the ASIC_DB object count is moving. A restart-looping syncd will leave large, stable gaps between intent and hardware.
Route-scale limits, whether stale entries linger, and how quickly things reconcile are platform-, HWSKU-, version-, and configuration-dependent (including warm-restart features). Never present a single observed behavior as how 'SONiC always behaves.' Record the exact image, HWSKU, and feature flags with any finding.
Common misconceptions
Myth: If show ip bgp and show ip route look right, forwarding must be fine.
Reality: Those are control-plane views. Resource exhaustion, sync failure, or unresolved recursion can still leave no hardware entry. Verify at ASIC_DB and with a data-plane test.
Myth: APPL_DB and ASIC_DB differing means something is broken.
Reality: Transient skew during convergence is expected. Only persistent, non-progressing divergence indicates a fault.
Myth: Route scale limits are basically the same across SONiC boxes.
Reality: They vary by ASIC family, HWSKU, and profile. Always check CRM on the specific platform.
- Prove or rule out recursive next-hop resolution first — it is the most common cause and cheap to check.
- On real hardware, check CRM before blaming BGP for a route that won't program.
- Distinguish transient skew from sync failure by watching whether the gap persists and whether services are healthy.
Key takeaways
- ✓A valid control-plane route does not forward until every dependency in the lifecycle resolves and hardware is programmed.
- ✓Five recurring causes: unresolved recursive next-hop, neighbor/interface dependency, resource exhaustion, software sync failure, stale/inconsistent state.
- ✓CRM makes hardware-resource exhaustion observable — but only on real silicon.
- ✓Failure behavior is platform/version/config dependent; never universalize it.
Tested against: SONiC 202305 / 202311 lineage · Upstream unless noted; enterprise images may add resource telemetry · Real ASIC required to observe hardware-resource failure modes · last reviewed 2026-07. Exact commands and behavior can vary by SONiC release, image, hardware platform, and enterprise distribution.
Knowledge check
Which of these can leave a route valid in BGP but not forwarding in hardware? (Select all that apply.)
After stopping and restarting the bgp container, a colleague claims 'SONiC always keeps forwarding while bgp is down.' How should you respond?