High availability

Prisma Cloud High Availability is designed specifically for customers running Console
outside
of a cluster manager. If you’re running Console in a cluster, such as Kubernetes, Prisma Cloud HA should not be used. Instead, utilize the native availability features of your cluster manager.
When you deploy Console on multiple hosts, Prisma Cloud HA ensures that one Console is always available, even if a host fails. Prisma Cloud HA creates a high availability clusters from redundant hosts running Console, and automatically manages cluster membership, leader election, and failover.
Prisma Cloud HA offers the following features:
  • Automatic database replication and failover.
  • Automatic Console failover.
Defender can continue to operate autonomously, even if communication with Console is temporarily severed. Therefore, it might not be necessary to deploy Prisma Cloud HA in your environment.
Keep in mind that HA and data backup are two separate issues. Even if you run Prisma Cloud in HA mode, you should still routinely backup and archive Prisma Cloud to some place outside of your production environment. Prisma Cloud supports disaster recovery and a method to quickly restore Console from a backup. By default, backups are taken every 24 hours, and can be written to durable storage, such as S3 buckets exposed as volumes. To learn more about backups, see Disaster recovery.
Prisma Cloud HA is not compatible with Projects. If you want to enable Projects, consider using a cluster manager rather than High Availability to oversee the uptime of the Console container.

Architecture

The Prisma Cloud HA cluster is set up in an active/passive configuration, where each host is a fully redundant node. Only one node at a time, the primary node, provides services. Redundant (secondary) nodes are only brought online when the primary node fails.
The following diagram shows a setup with three Console nodes organized into an HA cluster. Console nodes run behind a load balancer. At the top, there are three nodes running Defender. Defenders connect to the load balancer to access Console services.
Each Console has a local instance of the database that runs inside the Console container. Under the covers, we are using standard MongoDB clustering technology to manage data replication and leader election. When a node is primary, Console opens all Prisma Cloud ports (management port, web socket) to provide services. All other nodes close their Prisma Cloud ports and appear offline. In the background, secondary nodes periodically query the cluster to sync data and verify state. If the primary node fails, the remaining members elect a new primary, and Console on the new primary opens all Prisma Cloud ports to offer continued access to Console services.

Load balancer

Before you deploy Prisma Cloud HA, you must set up a load balancer. Numerous options are possible, including Amazon Elastic Load Balancer (ELB), nginx, and Round Robin DNS. If you deploy a load balancer, such as ELB and nginx, it will act as a reverse proxy that routes traffic to the primary node. Configure your load balancer to periodically health check the nodes in the cluster with Prisma Cloud’s ping API:
$ curl -u admin:Password https://console-ip:8083/api/v1/_ping
Most load balancing setups distribute traffic to all nodes. With Prisma Cloud HA, however, the two secondary nodes appear offline, and the load balancer sends all traffic to the primary node. In this setup, the logic for routing traffic resides in the load balancer.
An alternative setup is Round Robin DNS. Round Robin DNS responds to DNS requests with a list of IP addresses for all nodes in the cluster. Defender (or your browser, if you are accessing the Console UI) tries all IP addresses until it successfully connects to the primary node. In this case, the logic for routing traffic to the primary node resides in the client.

Setting up Prisma Cloud HA

This procedure shows you how to set up an HA cluster with three nodes. Adding cluster nodes is a simple process that can be done entirely from the Console UI.
Don’t use Prisma Cloud HA if you’re running Console inside a cluster, such as Kubernetes, Docker Swarm, or Amazon ECS. In such cases, use the native capabilities of the cluster manager to ensure Console is always available.
Although it is possible to design a custom service that runs multiple instances of Console in an active/active configuration, it is not tested or supported configuration.
Prerequisites:
  • Prisma Cloud only supports
    three
    nodes in an HA cluster setup. One primary Console and two secondary Consoles.
  • Port 27017 must be available on all cluster nodes. This is a MongoDB requirement.
  • All nodes must be on the same local, non-routed network.
  • You have set up a load balancer.
  1. Install Primary Console
    1. Download the latest Prisma Cloud release, and unpack the tarball.
    2. Open
      twistlock.cfg
      for editing.
    3. In the section for High availability settings, set
      HIGH_AVAILABILITY_ENABLED
      to
      true
      .
      ############################################# # High availability settings ############################################# HIGH_AVAILABILITY_ENABLED=true HIGH_AVAILABILITY_STATE=PRIMARY HIGH_AVAILABILITY_PORT=8086
  2. Install a secondary Console
    1. Open the primary Console. In a browser, go to <CONSOLE_PRIMARY_NODE>:8083, and login as admin.
    2. Go to
      Manage > System > High Availability
      .
      You should see your current host set as the primary Console.
      You cannot see the
      High Availability
      tab in the Console UI unless you have enabled HIGH_AVAILABILITY_ENABLED and installed Console with your updated
      twistlock.cfg
      file.
    3. Copy the script to install a secondary Console.
    4. On a different host, designated as secondary, run the script you just copied.
      Console is installed on your secondary host.
      This is the only step required to install Console on a secondary node. This script installs Console on the host, and joins the cluster. Do not run the procedure documented in the Install Prisma Cloud article on any secondary cluster nodes.
      In your primary Console, you will the secondary host waiting to join the cluster in the
      REQUEST_JOIN
      state.
    5. Follow the steps a-d to install another secondary Console.
      You now have a total of three Consoles. One primary Console and two secondary Consoles.
    6. When you see the
      REQUEST_JOIN
      , state for both secondary Consoles, click
      APPLY
      . After a few moments, the state changes to
      SECONDARY
      .
      Do not click
      APPLY
      until you see both secondary Consoles appear in the primary Console with the
      REQUEST_JOIN
      state.
  3. Test the cluster by taking the primary Console offline.
    1. Shut down the host running the primary Console
    2. Open Console in a browser.
      The secondary console will not become active until the primary console is disabled / disconnected. You will not be able to log in to other consoles in HA cluster as long as your primary console is up and running.
      Your load balancer should detect the failure and reroute traffic to the new primary in the cluster. The Console UI should open in your browser window:
    3. Go to
      Configure > SYSTEM > HIGH AVAILABILITY
      .
      You will see that the state of the nodes has changed. The previous primary node is now marked as unavailable, and another node has been promoted to primary.

Upgrading an HA cluster

To upgrade an HA cluster, first remove all secondary nodes. Then upgrade the primary node, and redeploy Prisma Cloud to the secondary nodes. For more information, see Upgrade Prisma Cloud

Recommended For You