End-of-Life (EoL)

Migrate an Existing Elasticsearch Deployment

Migrate a Cortex XSOAR deployment that already uses Elasticsearch for managing indicators.
To use Cortex XSOAR version 6.2 with Elasticsearch, you must migrate all of your existing content. It is no longer possible to use Elasticsearch to manage only indicators.
In the BoltDB, data related to incidents and indicators is stored in partitions by month. To minimize downtime during the migration, we recommend you create a copy of the database, then migrate data that is older than three months from the copy, while continuing to work in your current environment. Once the initial migration is completed, you should then migrate the last three months.
If you are working in an environment with remote repositories, you must run the migration separately on each environment. For example, if both your development and production environments are going to be used with an Elasticsearch database, you must migrate each of those environments, and each environment must use a different index prefix.
Run all commands from the Cortex XSOAR server machine. To migrate your data, you need to use the migration tool.
You cannot run more than one migration tool process at a time.
Always migrate older data before newer data. Migrating partitions out of order can cause duplicate incident ids.
  1. Upgrade the Cortex XSOAR server to 6.1, but do not start the server.
  2. Download the migration tool by appending
    downloadName=elasticsearch_migration_tool_6_1_0
    to the end of the download link that you received, when installing Cortex XSOAR.
  3. Copy your database and migrate data from the copy database to Elasticsearch.
    It is recommended to copy your data up to the last 3 months, without any downtime. If you do not want to do this, go to step 4.
    1. Copy the Cortex XSOAR database by doing one of the following:
    2. Copy the
      demisto.conf
      file.
    3. Edit the copy of the
      demisto.conf
      file, by adding your Elasticsearch configuration.
      Ensure that
      elasticsearch
      is the top-level object in the
      demisto.config
      file (within the main curly brackets).
    4. Using
      demisto
      or
      sudo
      permissions, run the following command:
      sudo ./elasticMigrator -config-path
      <file path-of-copy-of-demisto.conf>
      -db-path
      <path-of-the-copy-of-the-demisto-database>
      -
      <flags>
      For a full list of the flags, see Migration Tool Flags. For example, to exclude the last 3 partitions from the migration, add the
      -partitions-to-ignore
      flag and value to the command by typing the following:
      sudo ./elasticMigrator -config-path /usr/local/dev/copy_of_demisto.conf -db-path /usr/local/dev/lib_demisto_copy/data -partitions-to-ignore '042021,052021,062021'
      When you run the migration tool, parameter values specified in the
      demisto.conf
      file override values supplied for tool flags and default values. If no value exists in the
      demisto.conf
      file, values supplied in the tool flags override default values, but do not write the values to the
      demisto.config
      file.
      To avoid overwriting indicators that might already exist in Elasticsearch, you must run the migration with the
      -objects-to-ignore "newInsights"
      flag. If you already migrated audits in a previous version, you must run the migration with the
      -objects-to-ignore "newInsights, audits"
      flag.
      Example:
      sudo ./elasticMigrator -config-path /usr/local/dev/copy_of_demisto.conf -db-path /usr/local/dev/lib_demisto_copy/data -objects-to-ignore “newInsights” -partitions-to-ignore '042021,052021,062021'
    5. Complete steps 1 to 3 in Validate the migration.
  4. After the migration of the data is complete and validated, migrate your data from the active database to Elasticsearch.
    1. Create a backup copy of the
      demisto.conf
      file for your active database.
    2. Edit the
      demisto.conf
      as needed.
    3. Stop the Cortex XSOAR server.
      • CentOS:
        sudo systemctl stop demisto
      • Ubuntu:
        sudo service demisto stop
    4. Migrate the remaining months from the active database to Elasticsearch. Use the
      -partitions
      flag to specify the remaining partitions.
      To avoid overwriting indicators that might already exist in Elasticsearch, you must run the migration with the
      -objects-to-ignore "newInsights"
      flag. If your already migrated audits in a previous version, you must run the migration with the
      -objects-to-ignore "newInsights, audits"
      flag.
      Example:
      sudo ./elasticMigrator -partitions '
      042021,052021,062021
      ' -objects-to-ignore “newInsights”

Recommended For You