High Availability Overview
Overview of high availability in Cortex XSOAR, including
information about the different deployment architectures. Elasticsearch.
Load balancing.
High availability is intended to keep
your systems running even if one of the components in the system
fails. It provides redundancy for the different components so if
a problem occurs, there is minimal effect on your system. High availability
is an active-active failover configuration, and is different from Live Backup,
which is active-passive and requires manually enabling a backup server.
Live Backup is
not available for high availability deployments.
Out of the box, Cortex XSOAR is installed with the app server
and database on the same machine, but for a high availability deployment,
the app servers and database must be separated.
The app server processes all of the requests like running playbooks
or creating custom content, while the database stores all of the
data, including the content (custom and OOTB), indicators, incidents,
etc.
In Cortex XSOAR there are different degrees of
redundancy. The first, and also requisite degree, is database redundancy.
To enable any level of high availability (HA) in Cortex XSOAR, you
must migrate your database to Elasticsearch. This means that the
app server and database server are located on different machines.
In addition, you can implement high availability
by installing additional app servers, and if you are using Cortex
XSOAR Multi-tenant, also configure HA Groups.
Flow
Implementing a high availability deployment requires
you to prepare your environment with the following:
- Migrate your existing database to Elasticsearch or, for new customers, install a new instance of Elasticsearch.
- Configure a load balancer.
- Configure a shared file server. For single-instance deployments, this needs to be done between the app servers, and for multi-tenant deployments, this needs to be done for the main hosts, as well as each HA group.
- Proceed with the implementation of the high availability configuration.
Architecture
Depending on the configuration of your
system, single instance or multi-tenant, you can achieve high availability
using one of the following architectures.
Single instance deployment
In a single-instance deployment, the application
server is installed on a dedicated machine and connects to an Elasticsearch
database server. The Elasticsearch database server automatically
provides redundancy in accordance with how you have configured Elasticsearch.
In addition, you can also install multiple application
servers behind a load balancer. Requests are managed by the load
balancer, most commonly, but not required, using a round-robin methodology.
Also, the app servers use a shared file system
to ensure that all of the necessary files are available to all of
the application servers in the cluster.
Multi-Tenant Deployment
Similar to a single instance, in a multi-tenant
configuration you must first migrate your data to an Elasticsearch
database, separating the main account server from the database server.
Elasticsearch, depending on how it is configured, provides the database
redundancy.
To achieve full high availability, you can then
install multiple main account servers behind a load balancer, which
also uses an NFS server to share the required files, and communicate
with the designated index in Elasticsearch.
Once the main host servers are highly available, you can
no longer host new accounts on those servers. Existing accounts
on the Main host will still exist, but will not be highly available.
Therefore, Cortex XSOAR recommends that you move the accounts from
the Main host to an HA group.
In addition, Cortex XSOAR v6.1 enables you to create high availability
groups (HA groups), which form a cluster of hosts that provide redundancy
for all of the accounts on those hosts. The HA groups also use an
NFS server to share the required files, and communicate with the
designated index in Elasticsearch.
HA groups provide your system with:
- Redundancy for every account in the group
- Redundancy for every host machine in the group
- Performance improvements for every account in the group
All hosts in the HA Group must have the same hardware specifications.
In the full high availability architecture, you
can still have hosts that are not part of an HA group and work with
Elasticsearch, and you can also maintain hosts that use the out-of-the-box
configuration where the application server and database are on the
same machine, using a Bolt/Bleve database.
High Availability and Remote Repositories
The remote
repositories feature in the UI is not supported on development
environments that run as High Availability (multi-app servers). You can still use a
development > staging > production set up, where development is a single server (not
High availability), but production can be High Availability. In this setup, both
staging and production pull from the same git repository. If your development
environment runs as High Availability, use the CI/CD Solution.
Load Balancing
You can install multiple application servers behind
a load balancer. The load balancer must be configured to use sticky
sessions and request timeouts. The load balancer should use a health
check to verify that the App Server is up and running. Cortex XSOAR
recommends you use the /health route and verify that you receive
a 200 HTTP response.
Load balancing within HA groups for Multi-Tenant Deployment
Load balancing within HA groups uses an internal mechanism that
distributes requests between all available hosts in the group. By
default, the method used is Round Robin, but you can change it to
random using the
ha.host.selection.alg
server
configuration. When a host is alive and reachable, the requested account needs
to be alive and reachable as well for the request to be directed
to it:
- On the host level, as long as at least 1 host in an HA group is alive and reachable, the host and all of the accounts within that group are fully available.
- On the account level, as long as an account is alive and reachable on any of the hosts within that group, requests will be directed to it, offering full availability for that account
- Load balancing on the Main account level is handled by the external load balancer.
Engines and Load balancing
Before you install an engine, ensure that you set both the
Base
URL
and the External Host Name
to
the IP address of the Load Balancer. ( ). The Base URL enables engines to connect to the Load Balancer.
Setting the external host name to the load balancer enables the
engines to connect to the Load Balancer even if one of the app servers
fails. You can also add or remove app servers but the engines remain
connected.
When you upgrade to version 6.1 from a previous version,
you need to create either a new engine or edit the configuration file.
