Why Validation Gates Matter
Network changes carry inherent risk. A routing protocol misconfiguration can black-hole traffic. An ACL typo might expose critical systems. A botched VLAN assignment creates mysterious connectivity gaps that take hours to diagnose. Traditional change processes rely on peer review and testing in lab environments, but these controls don't guarantee production safety.
Pre-change validation establishes a known-good baseline. It captures current state before modifications begin: which routes exist, what neighbours are established, how traffic flows through the fabric. Post-change validation confirms the network behaves as intended after the change. Together, these gates create a safety net that catches errors before users notice problems.
Automation transforms validation from optional checklist to enforced requirement. When checks run automatically as part of the change workflow, teams get consistent verification without manual effort. Failed validations block deployment, preventing bad configurations from reaching production. The approach shifts risk left, catching issues during implementation rather than during incident response.
Pre-Change Baseline Capture
Effective validation starts with capturing the right baseline data. The specific checks depend on the change type, but certain elements prove universally useful. BGP neighbour states and route counts establish routing health. Interface status and error counters reveal link quality. ARP and MAC address table sizes indicate endpoint connectivity patterns.
Structured data beats text scraping. Collect information in formats that support programmatic comparison: JSON from REST APIs, YANG models via NETCONF, or parsed output from CLI commands. Store these snapshots with timestamps and change ticket identifiers for audit trails. The baseline becomes evidence that the network was healthy before the change began.
- BGP/OSPF/EIGRP neighbour counts and states across affected devices
- Route table size and specific prefix presence for critical networks
- Interface operational status, speed/duplex settings, and error rates
- Spanning-tree topology and root bridge identity for VLAN changes
- Current firewall session counts and NAT translation table size
- Multicast group memberships for environments running PIM
Post-Change Verification Tests
Post-change checks verify the intended outcome while confirming nothing unexpected broke. Start with basic reachability: can the automation system still connect to managed devices? SSH or HTTPS API failures indicate serious problems requiring immediate rollback.
Compare post-change state against the pre-change baseline. Route counts should match or reflect the expected delta from the change. Interface states should remain consistent unless the change specifically targeted them. New BGP neighbours should appear up; existing ones should maintain their sessions. Significant deviations signal problems even when the change technically completed.
Application-layer validation provides the strongest confidence. Synthetic transactions prove end-to-end functionality: HTTP requests to web services, database query latency checks, DNS resolution tests. These tests confirm users can actually do their work, not just that routing protocols converged. Path analysis tools can trace flows through the fabric, verifying traffic takes expected routes and passes through required security controls.
Automating the Validation Workflow
Pipeline-driven automation enforces validation as a mandatory step. The workflow begins when an engineer commits configuration changes to version control. The pipeline retrieves current network state as the pre-change baseline, then applies configurations to devices. A waiting period allows protocols to reconverge before post-change checks run.
Comparison logic detects meaningful differences while ignoring expected noise. Uptime counters will obviously change. OSPF hello timers create minor variations. The validation code needs thresholds: route count deviations beyond five percent warrant investigation, but smaller fluctuations are normal. Interface error increments are acceptable during the change window but shouldn't increase significantly afterward.
Structured reporting helps teams make quick decisions. Validation results should clearly state pass/fail status, highlight specific deviations from baseline, and provide enough context for troubleshooting. When checks fail, the pipeline can automatically trigger rollback procedures or alert on-call engineers. Success notifications close the loop, documenting that changes deployed safely.
- Use configuration management tools (Ansible, Nornir) to orchestrate baseline capture, change deployment, and verification in sequence
- Implement wait timers based on protocol convergence characteristics (30 seconds for OSPF, 2-3 minutes for BGP)
- Store validation results alongside change tickets for compliance documentation
- Create exception mechanisms for expected differences like route count changes during subnet additions
- Build rollback automation that restores previous configurations when validation fails
Advanced Validation Techniques
Stateful path analysis goes beyond control plane checks. Tools like Batfish model the network's forwarding behaviour, simulating packet flows through the infrastructure. These simulations can verify that traffic from specific source networks reaches intended destinations while avoiding prohibited paths. Path analysis catches subtle policy violations that simple interface and routing checks might miss.
Compliance validation ensures changes don't drift from standards. Automated checks can verify naming conventions, confirm required interface descriptions exist, validate VLAN assignments against documentation, and ensure spanning-tree priority settings follow design specifications. These tests maintain network hygiene over time as multiple engineers make modifications.
Some organisations implement gradual deployment with validation gates between stages. A multi-site change might deploy to a pilot location first, run extended validation, then proceed to remaining sites only if results look clean. This approach limits blast radius while providing real-world verification before full rollout. YS Infomatics has built validation frameworks that integrate with existing GitOps workflows, providing teams with pipeline-driven safety nets that catch configuration errors before they impact production environments.