Full Routed Packet Walk
Follow a single routed packet from ingress port to egress transmission: header parse, VLAN context, router-interface association, destination route lookup, next-hop resolution, neighbor adjacency, source/destination MAC rewrite, TTL decrement and checksum update, egress selection, queueing, and transmit. Each step maps to a SAI object you can inspect.
Prerequisites: Lesson 3.2 · Lesson 3.3 · Lesson 3.4
Learning objectives
- →Trace the end-to-end sequence of a routed packet through the pipeline.
- →Map each step to the SAI object(s) that must exist and be resolved for it to succeed.
- →Explain the L3 rewrite (src/dst MAC, TTL, checksum) precisely.
- →Identify, at each step, what a failure would look like and where to look.
Routing troubleshooting on SONiC is the art of asking 'which step failed?'. This walk is the checklist behind that question.
Every object category from Lesson 3.3 appears here in the exact role it plays for a live packet.
What your CCNP already gives you
- • You already know routed forwarding: dstIP lookup, next hop, ARP for the next hop, rewrite L2, decrement TTL, forward.
- • You know TTL decrement forces an IP header checksum update.
What is new in SONiC
- • Each step corresponds to a concrete SAI object you can query in ASIC_DB.
- • A break at any step is diagnosable by which object is missing or unresolved.
- • The 'ARP' step is a SAI neighbor entry keyed by (RIF, IP) — visible, not hidden.
| In the analogy | In SONiC |
|---|---|
| Receiving reads the item's label | Ingress port + parse |
| Deciding which department (routing context) owns it | VLAN / RIF context |
| Choosing the destination dock and the route to it | Route + next-hop lookup |
| The exact address label to slap on for the next hop | Neighbor adjacency |
| Relabeling the box and marking one leg of the journey used | MAC rewrite + TTL |
| Staging at the dock and loading the truck | Queue + transmit |
Where the analogy stops working
The order and merging of these steps vary by ASIC; some happen in parallel or in different pipeline positions. This is a model for reasoning and troubleshooting, not a literal per-chip sequence.
The routed packet walk (ingress to transmit)
- Ingress port: the packet arrives on a front-panel port (SAI port object). Port must be admin/oper up; ingress counters increment.
- Header parse: the parser extracts L2/L3 (and deeper) headers into metadata — dst/src MAC, EtherType, dst/src IP, TTL — once, for later stages to reuse.
- VLAN / bridge-domain context: ingress VLAN is determined (port default or tagged). This decides whether the packet is bridged (Lesson 3.6) or routed.
- Router-interface (RIF) association: if the destination MAC is the switch's router MAC on this VLAN/port, the packet is destined to be routed. The matching RIF object supplies the L3 context and source MAC for rewrite, and binds to a virtual router (VRF).
- Destination route lookup (LPM): the dst IP is looked up in the route table (SAI route entry, LPM). Result: a packet action and a reference to a next hop or next-hop group (ECMP).
- Next-hop resolution: the next-hop object (or a hashed member of the next-hop group) identifies the egress RIF and the next-hop IP. ECMP hashing selects one member deterministically from header fields.
- Neighbor adjacency: the (RIF, next-hop IP) neighbor entry supplies the destination MAC. If unresolved, the packet is punted/trapped to trigger ARP/ND (Lesson 3.7) and does not yet forward in hardware.
- Source MAC rewrite: the L2 source MAC is set to the egress RIF's router MAC.
- Destination MAC rewrite: the L2 destination MAC is set to the neighbor's MAC (the next hop).
- TTL decrement: the IPv4 TTL (or IPv6 hop limit) is decremented by one. TTL reaching zero is trapped to CPU instead of forwarded (Lesson 3.7).
- IP checksum update: because the TTL changed, the IPv4 header checksum is recomputed in hardware (IPv6 has no header checksum).
- Egress port selection: the egress port is derived from the chosen next hop/RIF (a physical port, or a LAG member selected by hashing).
- Queueing / scheduling: the packet is enqueued on the egress port's queue per its traffic class; scheduler/buffer profiles (SAI queue/scheduler objects) govern ordering and drops under congestion.
- Transmission: the rewritten packet is serialized out the egress port; egress counters increment.
Step to SAI object to failure signature
| Step | SAI object(s) | If it fails, you see… |
|---|---|---|
| RIF association | Router interface, virtual router | Traffic to router MAC not routed; wrong/absent RIF; L3 not enabled on the interface |
| Route lookup | Route entry (LPM) | No route / default only; dropped or sent the wrong way; missing prefix in ASIC_DB |
| Next-hop resolution | Next hop / next-hop group | ECMP imbalance or blackhole; next hop points at down/unresolved path |
| Neighbor adjacency | Neighbor entry (RIF, IP → MAC) | Punts to CPU, ARP incomplete; route present but 'inert' — no dst MAC |
| MAC/TTL rewrite | (pipeline action, driven by RIF/neighbor) | Malformed frames, TTL not decrementing (loops), or checksum errors |
| Egress/queue | Port, queue, scheduler, buffer | Tail drops, congestion, or port down at egress |
The most common routed-forwarding failure on a healthy control plane: the route entry exists in ASIC_DB and points at a next hop, but the neighbor entry is unresolved (no ARP/ND). The route is programmed yet forwards nothing because there is no destination MAC to rewrite. Always check the neighbor, not just the route.
Why the order of your checks should follow the packet
When routed traffic fails, resist jumping to the route table first. Follow the packet: is the ingress port up and are ingress counters moving? Is the packet even being routed (RIF/dst-MAC), or is it being bridged/dropped at L2? Only then the route, then the next hop, then the neighbor, then egress. Checking in packet order localizes the fault in the fewest steps — this is exactly the layered method Lab 3.5 drills.
Common misconceptions
Myth: If the route is in the FIB, the packet forwards.
Reality: Forwarding also needs a resolved next hop and neighbor (dst MAC), an up egress port, and available queue. The route is necessary, not sufficient.
Myth: The switch decrements TTL only sometimes.
Reality: Routed (L3) forwarding decrements TTL every hop and recomputes the IPv4 checksum. Bridged (L2) forwarding does not — that distinction is diagnostic.
Myth: ECMP picks a random member each packet.
Reality: ECMP hashes fixed header fields so a flow stays on one member (avoiding reordering). Imbalance usually means hash input or resolved-member issues, not randomness.
- Diagnose routed failures in packet order: ingress → RIF → route → next hop → neighbor → egress/queue.
- Route present + neighbor unresolved = 'inert route'; fix adjacency (ARP/ND), not routing.
- TTL not decrementing or checksum errors point at the rewrite stage / a bridging-vs-routing mix-up, not the RIB.
Key takeaways
- ✓A routed packet is parsed, given VLAN/RIF context, looked up (route → next hop → neighbor), rewritten (src/dst MAC, TTL, checksum), then queued and transmitted.
- ✓Each step is backed by a specific SAI object you can inspect in ASIC_DB.
- ✓A route with an unresolved neighbor is programmed but inert.
- ✓Routed forwarding decrements TTL and updates the IPv4 checksum; bridging does not.
- ✓Troubleshoot by following the packet's order, not by starting at the route table.
Tested against: Model; ASIC ordering varies · Upstream concepts · Reasoning model; confirm details per ASIC · last reviewed 2026-07. Exact commands and behavior can vary by SONiC release, image, hardware platform, and enterprise distribution.
Knowledge check
Order the steps a routed packet takes from ingress to transmission.
- 1Rewrite src/dst MAC, decrement TTL, update IPv4 checksum
- 2Parse headers into metadata
- 3Next-hop resolution → neighbor adjacency (dst MAC)
- 4Select egress port and enqueue
- 5Destination route lookup (LPM) → next hop
- 6Ingress port receives the packet
- 7Transmit on the egress port
- 8Determine VLAN / router-interface (RIF) context
You want the highest-value next step, not a scattershot of checks.
BGP is Established, show ip route shows the prefix with a next hop, but traffic to it fails. After confirming the control plane is healthy, which single check most efficiently splits the problem in half?