Migrate an Existing Elasticsearch Deployment

Migrate a Cortex XSOAR deployment that already uses Elasticsearch for managing indicators.

To use Cortex XSOAR version 6.2 with Elasticsearch, you must migrate all of your existing content. It is no longer possible to use Elasticsearch to manage only indicators.

In the BoltDB, data related to incidents and indicators is stored in partitions by month. To minimize downtime during the migration, we recommend you create a copy of the database, then migrate data that is older than three months from the copy, while continuing to work in your current environment. Once the initial migration is completed, you should then migrate the last three months.

If you are working in an environment with remote repositories, you must run the migration separately on each environment. For example, if both your development and production environments are going to be used with an Elasticsearch database, you must migrate each of those environments, and each environment must use a different index prefix.

Run all commands from the Cortex XSOAR server machine. To migrate your data, you need to use the migration tool.

You cannot run more than one migration tool process at a time.

Always migrate older data before newer data. Migrating partitions out of order can cause duplicate incident ids.

Upgrade the Cortex XSOAR server to 6.1, but do not start the server.

Download the migration tool by appending

downloadName=elasticsearch_migration_tool_6_1_0

to the end of the download link that you received, when installing Cortex XSOAR.

Copy your database and migrate data from the copy database to Elasticsearch.

It is recommended to copy your data up to the last 3 months, without any downtime. If you do not want to do this, go to step 4.

Copy the Cortex XSOAR database by doing one of the following:

Take a snapshot of the database.

Manually create a copy of the

/var/lib/demisto/data

directory.

Copy the

demisto.conf

file.

Edit the copy of the

demisto.conf

file, by adding your Elasticsearch configuration.

Ensure that

elasticsearch

is the top-level object in the

demisto.config

file (within the main curly brackets).

Using

demisto

sudo

permissions, run the following command:

sudo ./elasticMigrator -config-path <file path-of-copy-of-demisto.conf> -db-path

<path-of-the-copy-of-the-demisto-database> -<flags>

For a full list of the flags, see Migration Tool Flags. For example, to exclude the last 3 partitions from the migration, add the

-partitions-to-ignore

flag and value to the command by typing the following:

sudo ./elasticMigrator -config-path /usr/local/dev/copy_of_demisto.conf -db-path /usr/local/dev/lib_demisto_copy/data -partitions-to-ignore '042021,052021,062021'

When you run the migration tool, parameter values specified in the

demisto.conf

file override values supplied for tool flags and default values. If no value exists in the

demisto.conf

file, values supplied in the tool flags override default values, but do not write the values to the

demisto.config

file.

To avoid overwriting indicators that might already exist in Elasticsearch, you must run the migration with the

-objects-to-ignore "newInsights"

flag. If you already migrated audits in a previous version, you must run the migration with the

-objects-to-ignore "newInsights, audits"

flag.

Example:

sudo ./elasticMigrator -config-path /usr/local/dev/copy_of_demisto.conf -db-path /usr/local/dev/lib_demisto_copy/data -objects-to-ignore “newInsights” -partitions-to-ignore '042021,052021,062021'

Complete steps 1 to 3 in Validate the migration.

After the migration of the data is complete and validated, migrate your data from the active database to Elasticsearch.

Create a backup copy of the

demisto.conf

file for your active database.

Edit the

demisto.conf

as needed.

Stop the Cortex XSOAR server.

CentOS:

sudo systemctl stop demisto

Ubuntu:

sudo service demisto stop

Migrate the remaining months from the active database to Elasticsearch. Use the

-partitions

flag to specify the remaining partitions.

To avoid overwriting indicators that might already exist in Elasticsearch, you must run the migration with the

-objects-to-ignore "newInsights"

flag. If your already migrated audits in a previous version, you must run the migration with the

-objects-to-ignore "newInsights, audits"

flag.

Example:

sudo ./elasticMigrator -partitions '042021,052021,062021' -objects-to-ignore “newInsights”

Validate the migration (all steps).

Document:Cortex XSOAR Administrator’s Guide

Migrate an Existing Elasticsearch Deployment

Migrate an Existing Elasticsearch Deployment

Recommended For You

Document:Cortex XSOAR Administrator’s Guide

Migrate an Existing Elasticsearch Deployment

Migrate an Existing Elasticsearch Deployment

Recommended For You

Recommended Videos