Validate SD-WAN and Panorama Upgrade

1. Verify the plugin installation and activate the commit. Check the config diff. The difference contains SD-WAN plugin phash change and any other new feature schema changes. Validate and commit to Panorama.
2. Before any push, check show vpn ike-sa on the hub to determine which serial number (lower or higher of the HA peer) is being used for IKE.
3. Start the push operation. Examine the preview changes for all branch and hub firewalls within each cluster. Within this preview, check for pre-shared-key -AQ and ike_keyid_serial values. If the values in config diff are same, follow step 4. If these values don’t match in config diff, this would indicate a network interruption thus follow step 5.
4. Once all pre-validation checks complete successfully, push the configuration changes to all SD-WAN firewalls.
   The push process should be executed in a phased manner to minimize disruption and allow for immediate rollback if unforeseen issues arise.
   1. Push to hub firewall first. If there are multiple hubs, push to only one. If there are multiple clusters, start with the cluster that has the fewest devices.
   2. If hub firewalls are configured as HA pairs, push to the passive hub first. Monitor the progress and status of this push carefully. Ensure the push completes successfully on all targeted passive devices. Immediately address and resolve any failures or errors before proceeding.
   3. Push the configuration to all passive branches. Monitor the progress and status carefully. Ensure the push completes successfully on all targeted passive devices. Immediately address and resolve any failures or errors before proceeding to the next phase.
   4. After pushing to all passive devices, start the push to the active hub. Monitor the progress and status carefully, ensuring the push completes successfully. Immediately address and resolve any failures or errors before proceeding. Following the configuration push, verify that no issues are observed by checking network connectivity, VPN tunnel status, security policy enforcement, and overall device health. Promptly investigate and rectify any anomalies or performance degradation.
   5. If no issues are observed after the push to the active hub, push the configuration to the active branches. Monitor the progress and status carefully, ensuring the push completes successfully. Immediately address and resolve any failures or errors before proceeding. Following the configuration push, verify that no issues are observed by checking network connectivity, VPN tunnel status, security policy enforcement, and overall device health. Promptly investigate and rectify any anomalies or performance degradation.
   6. Once all preceding steps are completed, push the configuration to the remaining hubs in the cluster. If additional clusters exist, repeat the same steps for those clusters.
   7. Once the push to all hub is complete, repeat sub-steps 2 through 6 for all branches. If the SD-WAN cluster is a mesh topology, perform sub-steps 2 through 6 for all branches simultaneously.
5. Once the system completes all pre-validation checks and observes no issues, it can push configuration changes to all SD-WAN devices. The operations team must execute the push process in a phased manner to minimize disruption and allow for immediate rollback if unforeseen issues arise.
   1. First, push the configuration to the hub. If multiple hubs exist, push to only one of them. If multiple clusters exist, start with the cluster that has the fewest devices.
   2. If hub firewalls are configured as HA pairs, push the configuration to the passive hub first. Carefully monitor the progress and status of this push. Ensure that the push completes successfully on all targeted passive devices. Immediately address and resolve any failures or errors at this stage before proceeding to the next phase.
   3. From Panorama or directly from the active firewall, initiate the suspension process for the currently active hubs to force a failover.
   4. Check Panorama and the devices to confirm their suspended or inactive state. Next, verify that network traffic flows smoothly through the newly active devices, and monitor critical services to ensure continuous operation.
   5. On the Panorama push window, find and select only the currently suspended hubs. Once selected, push the configuration to these specific devices. Monitor the push operation closely to ensure successful completion for all suspended devices, with no errors in the commit logs.
   6. After the successful configuration push, activate the suspended devices by selecting unsuspend, bring online, or enable. Once all devices are active, verify that Panorama reflects the same status. Repeat above steps B through F for all remaining branches. If the SD-WAN cluster is a mesh topology, then follow sub-steps 2 through 6 for all branches simultaneously.
6. Once all steps in either 4 or 5 complete successfully, perform a thorough set of checks:
   1. On Panorama, verify that all SD-WAN firewalls are fully synchronized (both DG and template). Check for any out-of-sync warnings.
   2. Confirm all SD-WAN tunnels are up and stable, and check for errors or latency. Test traffic flow through critical tunnels.
   3. Conduct extensive traffic verification by testing access to critical applications and services from different network segments. Confirm all connectivity paths are functional and traffic flows as expected. Review firewall and Panorama logs for unusual patterns, errors, or security events.