Why your Cisco configs drift (and what to do about it)

You shipped a clean config to all 47 switches. Six months later, three of them have an extra ACL line, two have lost the NTP server, and one has syslog pointed at a server that doesn't exist. None of these were authorized changes. They just happened.

This is config drift, and it's one of the most common operational problems in any Cisco shop with more than 5 devices. The fix isn't harder than the diagnosis.

What "drift" actually means

Config drift is the divergence between the config you intended to run on a device and the config that's actually there. The intended state lives in your template, your git repo, or your memory. The actual state is whatever's in NVRAM right now.

The two are not the same. They drift because of:

• Cabling changes that prompted an "I'll add this temporary ACL" line that never got removed
• Outage fixes that left a config-only change undocumented because the fix was supposed to be "temporary"
• Vendor changes (TACACS, monitoring) that pushed config via SSH at 2 AM and never wrote back to your source of truth
• Handovers where the new engineer cleaned up a config in a way that wasn't propagated
• Bugs and edge cases in automation that didn't catch all the fields

In every shop I've seen with more than 20 devices, at least 10% of devices have at least one drift line within 6 months of a clean config rollout.

How to detect it (the cheap way)

You don't need fancy tooling to detect drift. The simplest approach:

1. Pick a baseline. Your "golden config" for a given device class. Could be the config you shipped six months ago, could be a template, could be the config of the cleanest device in the fleet.
2. Pull current configs from every device. Netmiko/Ansible/PacketPilot Config, whatever you have.
3. Diff each current config against the baseline. Ignore trivial diffs (timestamps, RSA keys, last-change markers). Look for actual config changes.
4. Triage each diff: was it authorized? If not, what's the impact?

This works at fleet sizes up to a few hundred devices. Above that, you want automation that runs the diffs continuously and alerts on unexpected changes.

How to detect it (the continuous way)

If you want drift detection to run automatically:

• Nightly config capture + diff against yesterday's backup. Catches changes within 24 hours.
• Diff against a baseline (golden config) in addition to yesterday. Catches "drift back to a clean state" if a fix gets reverted.
• Alert on large diffs. If 50% of the config changed overnight, someone probably did something big — and they probably should have told you.
• Track per-line changes in a database. Over time, you can see which devices drift most, which engineers introduce the most drift, which lines are most often changed.

Tools that do this:

• PacketPilot Config — built for this, multi-vendor, scheduled backups + diff + audit trail
• Oxidized — open-source network device backup + diff (requires Ruby ops to set up)
• Nautobot + Golden Config plugin — for shops already running Nautobot
• RANCID — old-school but still works, plain text diffs

For most teams, the answer is "whatever I already have running backups, plus a daily diff cron." You don't need a new tool, you need a habit.

How to fix drift (the remediation playbook)

Once you've detected drift, here's the order of operations:

1. Triage by risk, not by volume. A 200-line diff that's just timestamp updates is fine. A 3-line diff that disables AAA on a core switch is an emergency.
2. Decide: revert or document. Either the drift is unauthorized (revert it) or it was authorized but undocumented (document it in your source of truth, then leave the running config alone).
3. For unauthorized drift, do a root-cause analysis. Someone added that ACL line for a reason. If you just delete it, they'll add it again next time they hit whatever problem prompted it. Talk to them first.
4. Re-broadcast the clean state. Once you've triaged, push the canonical config to all similar devices. Now they're all consistent again.
5. Update the process. Drift happens because your process allows it. The fix is usually one of:
   • Source of truth that automation pulls from (not config that's edited directly on devices)
   • Change control that requires all changes to go through the source of truth
   • Better tooling (PacketPilot Config, Oxidized) so the manual cron-script excuse goes away

How to prevent drift (the structural fix)

The single most effective thing you can do: make it easier to update the source of truth than to update the device directly.

Concretely:

• Config lives in git. Every change is a PR. PRs get reviewed.
• Netmiko or similar pulls from git and pushes to devices. Engineers never SSH into a switch to change config — they update git, automation pushes.
• New ACL line? It's a PR. The PR is reviewed by a peer. The reviewer asks "why." Then the automation pushes.
• Drift becomes a "the automation failed" problem, not a "nobody documented the change" problem.

This is a 3-6 month project to set up properly. Worth it past 10 devices.

For shops that aren't ready for that level of automation, the simpler version is: nightly backups + diff + a Slack alert when something unexpected changes. The alert is your prompt to ask "who did this and why?"

Related

• Cisco IOS configuration backup: the complete guide — the operational foundation
• Cisco archive vs kron: which config backup mechanism wins — the device-side backup mechanisms
• show running-config vs show startup-config: what actually matters — what you're actually diffing

Try it

If you've been running config backups from a cron job you wrote in 2019 and praying — there's a faster way. PacketPilot Config runs scheduled backups with diff-style compare, one-click restore, and a timestamped audit trail. Multi-vendor from day one (Cisco, Juniper, Arista, HPE/Aruba). 30-day machine-bound trial, no credit card.

Download PacketPilot Config

— Ole