Module 3 · ASICs, Switching Pipelines, SAI & Forwarding

Failure Injection: SWSS

60 minLabFailure injectionVirtualTroubleshootingOpen-source

On a disposable virtual switch only, baseline the propagation path, disrupt the orchestration service (swss), make a config/route change while it is down, observe exactly where propagation stops, then restore and determine whether the intended state reconciles into hardware.

Safety: Disruptive — lab only

Disrupts forwarding or services. Run only in a dedicated virtual lab you can roll back. Never on production or shared physical switches.

DISRUPTIVE and VS-ONLY. Never stop swss/syncd on production or shared physical switches — it reprograms the ASIC and interrupts forwarding.

Use a dedicated, disposable VS with a known-good snapshot; be ready to revert.

Do not assume this procedure is safe on hardware just because it is safe on VS (see Lab 3.6).

Single disposable VS node

  SONiC-VS (DISPOSABLE)
    CONFIG_DB → *Mgr → APPL_DB → [swss/orchagent] → syncd → ASIC_DB
                                     ▲ disrupt here

Objective

• See, concretely, that intent stalls when the orchestration layer is down.
• Identify the exact boundary where propagation stops (APPL_DB present, ASIC_DB not updated).
• Determine whether SONiC reconciles the change after the service is restored.

Starting state

• VS healthy; all containers up; a known 'baseline' snapshot exists.
• You can add a test route/config and observe APPL_DB and ASIC_DB.

Prerequisites: Lessons 3.4 and 3.5 complete. · A DEDICATED, disposable SONiC-VS with a 'baseline' snapshot. This lab must not run on anything shared.

Tasks

1
Baseline the propagation path
Record a working change flowing all the way to ASIC_DB, so you know what 'healthy' looks like.
Linux
```
docker ps | grep -E 'swss|syncd'
```
SONiC CLI
```
show ip route summary
```
SONiC DBcount route objects in hardware view
```
redis-cli -n 1 SCAN 0 MATCH 'ASIC_STATE:SAI_OBJECT_TYPE_ROUTE_ENTRY*' COUNT 500 | wc -l
```
Expected:
- swss and syncd running; a baseline count of ASIC_DB route objects.
What happened internally: Establishes the healthy end-to-end path and a countable ASIC_DB baseline you will compare against after disruption.
2
Disrupt the orchestration service
Linux Disruptivestops the orchestration layer on THIS disposable VS only
```
sudo systemctl stop swss
```
Linux
```
docker ps | grep -E 'swss|syncd'
```
Expected:
- swss no longer running; note the effect on dependent services.
What happened internally: With orchagents down, nothing translates APPL_DB changes into SAI calls. This is the deliberate break.
3
Make a change while orchestration is down
Add a static route (or change an interface) and watch it land in APPL_DB but NOT reach ASIC_DB.
SONiC CLI Disruptive
```
sudo config route add prefix 10.99.99.0/24 nexthop <reachable-nh>
```
SONiC DBAPPL_DB — intent present
```
redis-cli -n 0 SCAN 0 MATCH 'ROUTE_TABLE:10.99.99.0/24' COUNT 100
```
SONiC DBASIC_DB — expect NOT present
```
redis-cli -n 1 SCAN 0 MATCH 'ASIC_STATE:SAI_OBJECT_TYPE_ROUTE_ENTRY*10.99.99.0/24*' COUNT 500
```
Expected:
- The new prefix appears in APPL_DB (intent recorded).
- The new prefix does NOT appear in ASIC_DB — propagation stopped at the orchestration boundary.
What happened internally: This is the whole lesson made tangible: intent can be perfectly recorded and still never reach hardware when a middle service is down.
4
Restore the service and test reconciliation
Linuxrestart orchestration
```
sudo systemctl start swss
```
Linux
```
docker ps | grep -E 'swss|syncd'
```
SONiC DB
```
redis-cli -n 1 SCAN 0 MATCH 'ASIC_STATE:SAI_OBJECT_TYPE_ROUTE_ENTRY*10.99.99.0/24*' COUNT 500
```
Expected:
- swss (and syncd, if it restarted) come back; the pending prefix now appears in ASIC_DB as orchagents reconcile APPL_DB against hardware.
What happened internally: On restart, orchagents reconcile desired state (APPL_DB) with the ASIC view — the design intent is convergence to desired state. Note whether syncd also restarted and whether a full reprogram occurred (impactful on real hardware).
5
Revert to baseline
SONiC CLI
```
sudo config route del prefix 10.99.99.0/24 nexthop <reachable-nh>
```
Expected:
- Test route removed; state matches baseline (or revert the VM snapshot).
What happened internally: Clean teardown so the disposable VS returns to a known state.

Troubleshooting branches

If: Stopping swss also disturbs syncd/other containers

Then: Expected — services are dependency-linked. This is why the lab is VS-only and snapshot-protected.

If: After restart, the route still isn't in ASIC_DB

Then: Confirm the next hop is reachable (neighbor resolves) and check orchagent/syncd logs; reconciliation needs resolvable dependencies.

If: VS won't return to healthy

Then: Revert the 'baseline' snapshot — the fastest clean recovery for a disposable VS.

Cleanup / rollback

Remove the test route.
Preferably revert to the 'baseline' snapshot to guarantee a clean, fully-reconciled state.

Lab evidence checklist

☐ Healthy baseline (services up, ASIC_DB route count) captured.
☐ Proof the change reached APPL_DB but NOT ASIC_DB while swss was down.
☐ Proof of reconciliation after restart (prefix now in ASIC_DB).
☐ A note on whether syncd restarted and whether a reprogram occurred.

Reflection

Exactly which boundary did propagation stop at, and how did you prove it?
Did SONiC reconcile to desired state on restart? What does that tell you about the system's design?
Why would running this same failure injection on production or shared physical hardware be dangerous?

Lab notes & observations