Recover from Split Brain in Panorama HA Deployments
When Panorama is configured in a high availability (HA)
setup, the managed firewalls are connected to both the active and
passive Panorama HA peers. When the connection between the active
and the passive Panorama peers fails, before the passive Panorama
takes over as the active peer it checks whether any firewall is
connected to both the active and the passive peer. If even one firewall
is connected to both peers, the failover is not triggered.
In the rare event that a failover is triggered when a set of
firewalls are connected to the active peer and a set of firewalls
are connected to the passive peer, but none of the firewalls are
connected to both peers, it is called a split brain. When a split
brain occurs, the following conditions occur:
Neither Panorama peer is aware of the state nor the HA
role of the other peer.
Both Panorama peers become active and manage a unique set
of firewalls.
To resolve a split brain, debug your network issues and restore
connectivity between the Panorama HA peers.
However, if you need to make configuration changes to your firewalls
without restoring the connection between the peers, here are a couple
of options:
Manually add the same configuration changes on both Panorama
peers. This ensures that when the link is reestablished the configuration
is synchronized.
If you need to add/change the configuration at only one Panorama
location, make the changes and synchronize the configuration (make
sure that you initiate the synchronization from the peer on which
you made the changes) when the link between the Panorama peers is
re-established. To synchronize the peers, select the
Dashboard
tab
and click the
Sync to peer
link in the High
Availability widget.
If you need to add/change the configuration for only the
connected firewalls at each location, you can make configuration
changes independently on each Panorama peer. Because the peers are disconnected,
there is no replication and each peer now has a completely different
configuration file (they are out of sync). Therefore, to ensure
that the configuration changes on each peer are not lost when the
connection is restored, you cannot allow the configuration to be
automatically re-synchronized. To solve this problem, export the
configuration from each Panorama peer and manually merge the changes
using an external diff and merge tool. After the changes are integrated,
you can import the unified configuration file on the primary Panorama
and then synchronize the imported configuration file with the peer.