Module 2 · Control Plane, FRR & Route Lifecycle

Broken Next-Hop Scenario

55 minLabTroubleshootingFailure injectionVirtual

Create the classic 'valid in BGP, not forwarding' condition on purpose: a route whose next-hop cannot be fully resolved, so it lives happily in the control plane but never programs a hardware entry. Your job is to diagnose the broken dependency using the lifecycle and layer-tracing skills from this module — then fix it.

Safety: Caution

May change state — take a snapshot first.

Caution: introduces a deliberately broken route. Confine changes to the lab test prefix and topology.

Reuses Lab 2.1, with a recursive/unresolvable next-hop introduced

  sonic-a learns 10.9.9.0/24 via next-hop that cannot resolve
  route present in BGP/zebra, no ASIC entry, no forwarding

Objective

• Reproduce a control-plane-present / data-plane-absent route.
• Diagnose the broken dependency by walking the lifecycle layers.
• Identify the specific unresolved next-hop / neighbor / interface condition.
• Apply and verify the fix.

Starting state

• eBGP Established on the Lab 2.1 topology.
• Instructor or self-injected condition present (see Task 1).

Prerequisites: Labs 2.1–2.2 (you can trace a route across layers). · Lessons 2.3 and 2.4.

Tasks

1
Inject the broken next-hop (setup)
Choose one injection so the peer advertises a prefix whose next-hop cannot be resolved for forwarding. For example, advertise 10.9.9.0/24 with a next-hop on a subnet that has no resolvable neighbor, or set next-hop to an address not reachable over an oper-up L3 interface.
FRR / vtysh Disruptiveon the peer; combine with an unresolvable next-hop condition
```
vtysh -c 'conf t' -c 'router bgp 65002' -c ' address-family ipv4 unicast' -c '  network 10.9.9.0/24' -c 'end'  # then manipulate next-hop / disable the resolving path
```
Expected:
- sonic-a receives 10.9.9.0/24 in BGP, but its next-hop will not fully resolve.
What happened internally: The goal is a route valid at bgpd/zebra whose next-hop lacks a resolved neighbor/interface — the 'held pending' state from Lesson 2.3.
2
Confirm the symptom
SONiC CLI
```
show ip route 10.9.9.0/24
```
Linuxexpected to fail
```
ping -c 3 10.9.9.1
```
Expected:
- Route appears present in the control plane, but forwarding fails.
What happened internally: This is the paradox the lab is about: 'the route is there' yet nothing forwards.
3
Diagnose: is it in BGP/zebra?
FRR / vtysh
```
vtysh -c 'show ip bgp 10.9.9.0/24'
```
FRR / vtyshnote any 'inactive'/unresolved next-hop marker
```
vtysh -c 'show ip route 10.9.9.0/24'
```
Expected:
- Path present in bgpd; in zebra the next-hop may be flagged unresolved/inactive.
What happened internally: If zebra marks the next-hop unresolved, the route will not be exported for programming — the fault is upstream of APPL_DB.
4
Diagnose: resolve the next-hop yourself
FRR / vtyshdoes the next-hop have a real, connected, resolved path?
```
vtysh -c 'show ip route <next-hop-ip>'
```
SONiC DB
```
sonic-db-cli APPL_DB HGETALL 'NEIGH_TABLE:Ethernet0:<next-hop-ip>'
```
SONiC CLIis the egress interface L3 and oper-up?
```
show ip interfaces
```
Expected:
- You find the break: no resolving route for the next-hop, or no resolved neighbor, or the egress interface is down/not L3.
What happened internally: This is recursive next-hop resolution in practice: the dependent route can only program once its next-hop maps to a directly connected, resolved adjacency.
5
Diagnose: confirm the gap at APPL_DB / ASIC_DB
SONiC DBmay be absent or present-without-programming
```
sonic-db-cli APPL_DB HGETALL 'ROUTE_TABLE:10.9.9.0/24'
```
SONiC DBIllustrative outputexpect no route-entry for 10.9.9.0/24
```
sonic-db-cli ASIC_DB EVAL "return redis.call('SCAN',0,'MATCH','ASIC_STATE:SAI_OBJECT_TYPE_ROUTE_ENTRY:*','COUNT',200)" 0
```
Expected:
- No hardware route object for the prefix — consistent with an unresolved dependency.
What happened internally: Confirms the failure localizes to dependency resolution, not to BGP path selection.
6
Fix and verify
Repair the broken dependency you identified: restore the resolving route, fix the neighbor/ARP, or bring the egress interface up as L3.
shell
```
# e.g. bring the interface up / restore the resolving route / fix addressing
```
SONiC CLI
```
show ip route 10.9.9.0/24
```
Linux
```
ping -c 3 10.9.9.1
```
Expected:
- Next-hop resolves; route programs to APPL_DB and ASIC_DB; forwarding succeeds.
What happened internally: Fixing the dependency lets orchagent re-evaluate the held route and complete the lifecycle to hardware.

Troubleshooting branches

If: Route programs even though you meant to break it

Then: Your next-hop still resolves via another path; make the next-hop genuinely unreachable over an oper-up L3 interface.

If: Next-hop looks resolved but still no forwarding

Then: Re-check the egress interface L3/oper state and CRM resources (on hardware); rule out a second dependency.

If: Fix applied but route still pending

Then: Give orchagent a moment to re-evaluate; confirm the neighbor is now resolved in NEIGH_TABLE.

Cleanup / rollback

Remove the injected 10.9.9.0/24 advertisement and restore the topology to the Lab 2.1/2.2 baseline.

Lab evidence checklist

☐ Symptom captured (route present, ping fails).
☐ The specific broken dependency identified (unresolved next-hop / neighbor / interface).
☐ APPL_DB/ASIC_DB gap captured.
☐ Fix applied and forwarding verified.
☐ A one-line root-cause statement written.

Reflection

Which single command most quickly pointed at the broken dependency, and why?
How does this scenario differ from a resource-exhaustion failure, and how would you tell them apart on real hardware?

Lab notes & observations