Migrate a Multi-Tenant Deployment for High Availability

Instructions for migrating your Cortex XSOAR Multi-tenant deployment to enable High Availability.
To migrate a multi-tenant deployment from BoltDB to Elasticsearch with High Availability, there are two steps. First, you use the Cortex XSOAR migration tool to migrate the databases. Then you set up additional app servers to achieve high availability.

Migration

The migration tool migrates Cortex XSOAR objects to an Elasticsearch database. When you run the migration tool, the contents of the Cortex XSOAR database are read, and a corresponding object is created in Elasticsearch. The migration tool is run from the main machine and each host machine. Migrate the main account on the main host, then the tenants on the main host, and then proceed to other hosts and tenant accounts. For complex environments with multiple host machines, we recommend migrating all tenants on a host before proceeding to the next host.
You cannot run more than one migration tool process at a time.
When you run the migration tool, parameter values identified in the
demisto.conf
file override values supplied for tool flags and default values. If no value exists in the
demisto.conf
file, values supplied in the tool flags override default values, but do not write the values to the
demisto.config
file. For example, if the db-path is identified in the configuration file, the tool will use the value in that file, not the value supplied or the default value, when running the tool.
If you are upgrading from Cortex XSOAR 6.0 to 6.1 and you have indicators and audits stored in Elasticsearch in Cortex XSOAR 6.0, upgrade from Cortex XSOAR 6.0 to 6.1 before the migration. Do NOT start the server. Then follow the migration instructions below.
Before you begin, ensure the following:
  • All app servers have network access to port 9200 for communication with Elasticsearch.
  • All app servers have network access to each other over port 443.
Download the migration tool
To download the migration tool, append
downloadName=elasticsearch_migration_tool_6_1_0
to the end of the download link that you received.
Configuration file parameters
The elasticsearch object should be a top-level object in the
demisto.config
configuration file (within the main curly brackets).
Migration tool flags
Flag
Type
Description
Required
accounts
(multi-tenant only)
String
A comma-separated list of accounts to migrate. If not specified, all accounts are migrated.
Optional
config-path
String
The path to the configuration file for the server. Default: /etc/demisto.conf.
Optional
db-path
String
The path to the database directory. Default: /var/lib/demisto.
Optional
elasticsearch-batch-size
integer
The number of indicators per batch to write to Elasticsearch indexes. Default: 700.
Optional
elasticsearch-index-prefix
String
The index prefix used in Elasticsearch.
Optional
elasticsearch-key
String
The API key to connect to Elasticsearch.
Required (unless a username and password are used)
elasticsearch-password
String
The password to connect to Elasticsearch.
required(unless API key is used)
elasticsearch-url
String
The URL of your Elasticsearch environment. Default: http://localhost:9200.
Required
elasticsearch-username
String
The username to connect to Elasticsearch.
required(unless API key is used)
ignore-ids-path
String
The path to the file with the IDs to ignore, per object.
Optional
log-level
String
The log level to display. Default: info.
Optional
logfile
String
The location of the log file.Default: /var/log/demisto/elastic_migration.log
Optional
migrate-all
true
By default, the Elasticsearch tool checks existing indexes and migrates only the ones that are new. Using this flag, the Elasticsearch tool migrates all indexes even if they currently exist. This is useful, for example, if there was an error or invalid data that was fixed. When used, the objects-to-migrate and objects-to-ignore flags are ignored.
Optional
objects-to-ignore
String
Comma-separated list of objects not to migrate. When the migrate-all flag is used, this flag is ignored.
Optional
objects-to-migrate
String
Comma-separated list of objects to migrate. When the migrate-all flag is used, this flag is ignored.
Optional
partitions
String
Comma-separated list of partitions to migrate. If no partitions are specified, all partitions are migrated.
Optional
previous-results
Show results of the previous migration.
Optional
skip-accounts
false
Does not migrate multi-tenant accounts. When set to true, only the main tenant or host database are migrated.
Optional
skip-existing-indicators
false
Existing indicators are not modified during the migration.
Optional
skip-mt-main
false
Does not migrate the main host or host database.
Optional
version
Prints the migration tool version.
Optional
y
false
Answers yes to all questions, unless there is an error.
Optional
In the BoltDB, data related to incidents and indicators is stored in partitions by month. To minimize downtime during the migration, we recommend you create a copy of the database, then migrate data that is older than three months from the copy, while continuing to work in your current environment. Once the initial migration is completed, you should then migrate the last three months.
  1. Copy the
    demisto.lic
    file from the
    /usr/local/demisto
    directory on the main host to the
    /var/lib/demisto
    directory.
  2. Add the following entry in the
    demisto.conf
    file:
    license.file.path:"/var/lib/demisto"
  3. Follow the migration instructions for multi-tenant deployments.
    If you used Elasticsearch to store indicators and audits in Cortex XSOAR 6.0 prior to upgrade, ensure the
    demisto.conf
    file is up-to-date with the Elasticsearch object before migration. To avoid overwriting indicators that might already exist in Elasticsearch, you must run the migration with the
    -objects-to-ignore "newInsights"
    flag. If you already migrated audits in a previous version, you must run the migration with the
    -objects-to-ignore "newInsights, audits"
    flag.

High Availability - Main Host

The main account cannot be High Availability if it has tenant accounts. Move any tenant accounts from the main host to another host before proceeding.
  1. Edit the
    demisto.conf
    file to set the app server’s hostname:
    Under the Server object, set the value of the inClusterHostname parameter to the desired hostname.
  2. (
    Optional
    ) Install additional app servers.
    Verify that the clocks on the app servers are synchronized with an NTP server.

High Availability - Hosts

Add hosts to a HA group:
All hosts in the HA Group must have the same hardware specifications.
  1. Configure a shared directory, using the network file sharing solution of your choice, on each host. Make sure that the
    /var/lib/demisto
    directory is configured as a shared folder. If you are using a location that is different from the default
    /var/lib/demisto
    , you must install the additional hosts using the
    -data-dir
    flag. For the existing host, follow instructions to Move Data Folders to Another Location on the Server.
  2. In the Main host, navigate to the Hosts page, under
    Settings
    Account Management
    . A list of all of the hosts, including those you have migrated, will be displayed.
  3. Select the Host for which you want to create an HA group and click
    Make Host Highly Available
    .
  4. In the window that appears, click
    Download Host Installer
    .
  5. Use this dedicated installer for installing additional hosts to this HA group. The new hosts will automatically be configured to connect to the relevant Elasticsearch index.

Recommended For You