Upgrade Panorama High Availability Pair (Active/Passive) Leveraging SD-WAN Plugin

Panorama HA Pair: Upgrade SD-WAN Plugin 1.0.4 to 2.2.7 Version

1. Upgrade your Panorama management server version.

   1. From Panorama 9.1.x, download and install Panorama 10.0.7-h3 on both active and passive Panorama.
   2. From Panorama 10.0.7-h3, download and install the latest Panorama 10.1 release on both active and passive Panorama.
   3. After the Panorama is upgraded to the latest 10.1 release, check if the active Panorama remains as active and the passive Panorama remains as passive. If there is no change in the HA states, then the upgrade is successful. Otherwise, you need to perform a force switch over to maintain the state of the HA pairs that it was before the upgrade.
      To perform the force switchover, execute the following CLI commands in the same order from the current active HA peer.

      admin > request high-availbility state suspend

      admin > request high-availbility state functional

      
2. Monitor the configd logs.

   (In administrator mode) Before upgrading the SD-WAN plugin to 2.2.7, start monitoring the configd log on both the Panorama HA pairs.

   admin> tail follow yes mp-log configd.log

   If you see the below error message on executing tail follow yes mp-log configd.log command, the Mongo DB of the active and passive Panorama has become out of sync.

   

   To resolve this issue:

   1. (In administrator mode) Drop the whole database pan_oplog on both the active and passive Panorama.

      admin > debug mongo drop database pan_oplog instance mdb 

   2. (In administrator mode) Restart configd on both the active and passive Panorama.

      admin > debug software restart process configd

   

   Once the configd is restarted, refresh the respective web interface and command line interface. After restart, you won't be seeing the mongo pan_oplog error on any of the commit processes.

   We recommend you to monitor the configd logs during the whole upgrade process.
3. Download and install the SD-WAN plugin version 2.2.7 on both active and passive Panorama.
4. (Mandatory) (In configuration mode) Commit the changes from the active Panorama.
5. Monitor the HA synchronization job on the passive Panorama HA peer. If you encounter the following error message, then execute debug plugins sd_wan mongo-db sync-db-to-peer CLI command from the active Panorama HA peer manually.

   

   If you encounter the following error message, then add a new VPN address pool.

   IP address from vpn address pool subnet/subnets are exhausted.
6. Perform HA synchronization job from active Panorama HA peer:

   • By executing request high-availability sync-to-remote running-config CLI command, or
   • In the Panorama Dashboard (under High Availability), select Sync Now for the Running config.
7. Check the following after Panorama HA upgrade.

   1. Perform a selective push to branch devices first, followed by the hub devices from active Panorama.
   2. Select PanoramaManaged DevicesSummary and verify if the device group and templates are in synchronization on both active and passive Panorama under the devices summary page.
   3. Verify if the SD-WAN configurations such as, Tunnel, BGP, Key ID, and traffic are as expected.
      After successful upgrade of the Panorama HA pair, the Key ID, PSK, IP cache, IPSec tunnel cache, and subnet cache will be refreshed which will not affect the functionalities of SD-WAN.
8. (Recommended) Upgrade the connected firewalls.

   Once the Panorama HA pair upgrade is successful, the connected hub and branch devices can be upgraded one-by-one starting with the branch firewalls followed by hub firewalls (the branch and hub firewalls can be standalone firewalls or HA pairs).

   We recommend you to check the SD-WAN configuration and functionality after upgrading each firewall.

   1. Introduce a minor change on all the templates by modifying or adding the comment for an interface on the template, followed by a Commit and Push to Devices. This is just a verification activity to ensure that the configuration is good and the upgrade is working.

      

   2. Check the SD-WAN configuration and functionalities.
   3. Upgrade the branch firewall one-by-one till all the branches are upgraded.
   4. Follow the below steps for branch firewalls first.
      1. Start upgrading a pair of branch HA or standalone devices from Panorama version 9.1.x to 10.0.7-h3, and then to the latest Panorama 10.1 release.
      2. Introduce a minor change in the comment of an interface from the particular firewall template from the active Panorama where the upgrade was performed, Commit, and Push to Devices. Once the Commit All is completed, check the SD-WAN configurations and functionalities. This is just a verification activity to ensure that the configuration is good and the upgrade is working after the firewall is upgraded.
   5. Follow the below steps for the hub firewalls. It's important that you complete the upgrade of the branch firewalls and then start the upgrade of the hub firewalls.
      1. Start upgrading a pair of hub HA or standalone devices from Panorama version 9.1.x to 10.0.7-h3, and then to the latest Panorama 10.1 release.
      2. Introduce a minor change in the comment of an interface from the particular firewall template from the active Panorama where the upgrade was performed, Commit, and Push to Devices. Once the Commit All is completed, check the SD-WAN configurations and functionalities.
         This is just a verification activity to ensure that the configuration is good and the upgrade is working after the firewall is upgraded.
   6. Select PanoramaManaged DevicesSummary and verify if the device group and templates are in synchronization on both active and passive Panorama under the devices summary page.
   7. After the upgrade is complete, verify the changes after the upgrade.