Recover from Split Brain in Panorama HA Deployments
Table of Contents
Expand all | Collapse all
-
- Determine Panorama Log Storage Requirements
-
- Setup Prerequisites for the Panorama Virtual Appliance
- Perform Initial Configuration of the Panorama Virtual Appliance
- Set Up The Panorama Virtual Appliance as a Log Collector
- Set Up the Panorama Virtual Appliance with Local Log Collector
- Set up a Panorama Virtual Appliance in Panorama Mode
- Set up a Panorama Virtual Appliance in Management Only Mode
-
- Preserve Existing Logs When Adding Storage on Panorama Virtual Appliance in Legacy Mode
- Add a Virtual Disk to Panorama on an ESXi Server
- Add a Virtual Disk to Panorama on vCloud Air
- Add a Virtual Disk to Panorama on AWS
- Add a Virtual Disk to Panorama on Azure
- Add a Virtual Disk to Panorama on Google Cloud Platform
- Add a Virtual Disk to Panorama on KVM
- Add a Virtual Disk to Panorama on Hyper-V
- Mount the Panorama ESXi Server to an NFS Datastore
-
- Increase CPUs and Memory for Panorama on an ESXi Server
- Increase CPUs and Memory for Panorama on vCloud Air
- Increase CPUs and Memory for Panorama on AWS
- Increase CPUs and Memory for Panorama on Azure
- Increase CPUs and Memory for Panorama on Google Cloud Platform
- Increase CPUs and Memory for Panorama on KVM
- Increase CPUs and Memory for Panorama on Hyper-V
- Complete the Panorama Virtual Appliance Setup
-
- Convert Your Evaluation Panorama to a Production Panorama with Local Log Collector
- Convert Your Evaluation Panorama to a Production Panorama without Local Log Collector
- Convert Your Evaluation Panorama to VM-Flex Licensing with Local Log Collector
- Convert Your Evaluation Panorama to VM-Flex Licensing without Local Log Collector
- Convert Your Production Panorama to an ELA Panorama
-
- Register Panorama
- Activate a Panorama Support License
- Activate/Retrieve a Firewall Management License when the Panorama Virtual Appliance is Internet-connected
- Activate/Retrieve a Firewall Management License when the Panorama Virtual Appliance is not Internet-connected
- Activate/Retrieve a Firewall Management License on the M-Series Appliance
- Install the Panorama Device Certificate
- Install the Device Certificate for a Dedicated Log Collector
-
- Migrate from a Panorama Virtual Appliance to an M-Series Appliance
- Migrate a Panorama Virtual Appliance to a Different Hypervisor
- Migrate from an M-Series Appliance to a Panorama Virtual Appliance
- Migrate from an M-100 Appliance to an M-500 Appliance
- Migrate from an M-100 or M-500 Appliance to an M-200 or M-600 Appliance
-
- Configure an Admin Role Profile
- Configure an Access Domain
-
- Configure a Panorama Administrator Account
- Configure Local or External Authentication for Panorama Administrators
- Configure a Panorama Administrator with Certificate-Based Authentication for the Web Interface
- Configure an Administrator with SSH Key-Based Authentication for the CLI
- Configure RADIUS Authentication for Panorama Administrators
- Configure TACACS+ Authentication for Panorama Administrators
- Configure SAML Authentication for Panorama Administrators
-
- Add a Firewall as a Managed Device
-
- Add a Device Group
- Create a Device Group Hierarchy
- Create Objects for Use in Shared or Device Group Policy
- Revert to Inherited Object Values
- Manage Unused Shared Objects
- Manage Precedence of Inherited Objects
- Move or Clone a Policy Rule or Object to a Different Device Group
- Push a Policy Rule to a Subset of Firewalls
- Manage the Rule Hierarchy
- Manage the Master Key from Panorama
- Redistribute User-ID Information to Managed Firewalls
-
- Plan the Transition to Panorama Management
- Migrate a Firewall to Panorama Management and Reuse Existing Configuration
- Migrate a Firewall to Panorama Management and Push a New Configuration
- Migrate a Firewall HA Pair to Panorama Management and Reuse Existing Configuration
- Migrate a Firewall HA Pair to Panorama Management and Push a New Configuration
- Load a Partial Firewall Configuration into Panorama
- Localize a Panorama Pushed Configuration on a Managed Firewall
-
- Add Standalone WildFire Appliances to Manage with Panorama
- Configure Basic WildFire Appliance Settings on Panorama
- Remove a WildFire Appliance from Panorama Management
-
-
- Configure a Cluster and Add Nodes on Panorama
- Configure General Cluster Settings on Panorama
- Remove a Cluster from Panorama Management
- Configure Appliance-to-Appliance Encryption Using Predefined Certificates Centrally on Panorama
- Configure Appliance-to-Appliance Encryption Using Custom Certificates Centrally on Panorama
- View WildFire Cluster Status Using Panorama
- Upgrade a Cluster Centrally on Panorama with an Internet Connection
- Upgrade a Cluster Centrally on Panorama without an Internet Connection
-
-
- Manage Licenses on Firewalls Using Panorama
-
- Supported Updates
- Schedule a Content Update Using Panorama
- Upgrade Log Collectors When Panorama Is Internet-Connected
- Upgrade Log Collectors When Panorama Is Not Internet-Connected
- Upgrade Firewalls When Panorama Is Internet-Connected
- Upgrade Firewalls When Panorama Is Not Internet-Connected
- Upgrade a ZTP Firewall
- Revert Content Updates from Panorama
-
- Preview, Validate, or Commit Configuration Changes
- Enable Automated Commit Recovery
- Compare Changes in Panorama Configurations
- Manage Locks for Restricting Configuration Changes
- Add Custom Logos to Panorama
- Use the Panorama Task Manager
- Reboot or Shut Down Panorama
- Configure Panorama Password Profiles and Complexity
-
-
- Verify Panorama Port Usage
- Resolve Zero Log Storage for a Collector Group
- Replace a Failed Disk on an M-Series Appliance
- Replace the Virtual Disk on an ESXi Server
- Replace the Virtual Disk on vCloud Air
- Migrate Logs to a New M-Series Appliance in Log Collector Mode
- Migrate Logs to a New M-Series Appliance in Panorama Mode
- Migrate Logs to a New M-Series Appliance Model in Panorama Mode in High Availability
- Migrate Logs to the Same M-Series Appliance Model in Panorama Mode in High Availability
- Migrate Log Collectors after Failure/RMA of Non-HA Panorama
- Regenerate Metadata for M-Series Appliance RAID Pairs
- Troubleshoot Registration or Serial Number Errors
- Troubleshoot Reporting Errors
- Troubleshoot Device Management License Errors
- Troubleshoot Automatically Reverted Firewall Configurations
- Complete Content Update When Panorama HA Peer is Down
- View Task Success or Failure Status
- Restore an Expired Device Certificate
- Downgrade from Panorama 9.1
Recover from Split Brain in Panorama HA Deployments
When Panorama is configured in a high availability (HA)
setup, the managed firewalls are connected to both the active and
passive Panorama HA peers. When the connection between the active
and the passive Panorama peers fails, before the passive Panorama
takes over as the active peer it checks whether any firewall is
connected to both the active and the passive peer. If even one firewall
is connected to both peers, the failover is not triggered.
In the rare event that a failover is triggered when a set of
firewalls are connected to the active peer and a set of firewalls
are connected to the passive peer, but none of the firewalls are
connected to both peers, it is called a split brain. When a split
brain occurs, the following conditions occur:
- Neither Panorama peer is aware of the state nor the HA role of the other peer.
- Both Panorama peers become active and manage a unique set of firewalls.
To resolve a split brain, debug your network issues and restore
connectivity between the Panorama HA peers.
However, if you need to make configuration changes to your firewalls
without restoring the connection between the peers, here are a couple
of options:
- Manually add the same configuration changes on both Panorama peers. This ensures that when the link is reestablished the configuration is synchronized.
- If you need to add/change the configuration at only one Panorama location, make the changes and synchronize the configuration (make sure that you initiate the synchronization from the peer on which you made the changes) when the link between the Panorama peers is re-established. To synchronize the peers, select theDashboardtab and click theSync to peerlink in the High Availability widget.
- If you need to add/change the configuration for only the connected firewalls at each location, you can make configuration changes independently on each Panorama peer. Because the peers are disconnected, there is no replication and each peer now has a completely different configuration file (they are out of sync). Therefore, to ensure that the configuration changes on each peer are not lost when the connection is restored, you cannot allow the configuration to be automatically re-synchronized. To solve this problem, export the configuration from each Panorama peer and manually merge the changes using an external diff and merge tool. After the changes are integrated, you can import the unified configuration file on the primary Panorama and then synchronize the imported configuration file with the peer.