End-of-Life (EoL)

Migrate Cortex XSOAR Objects to Elasticsearch for a Single Server

Migrate Cortex XSOAR objects to Elasticsearch for a single server. Migration tool, flags.
You should migrate Cortex XSOAR objects to Elasticsearch if you plan to ingest a large amount of objects.
In the BoltDB, data related to incidents and indicators is stored by month in partitions. To minimize downtime during migration, it is recommended to create a copy of the database and migrate data that is older than three months from the copied database. This enables you to continue to work in your current environment. As soon as the initial migration completes, migrate the remaining months.
If you are working in an environment with remote repositories, you must run the migration separately on each environment. For example, if both your development and production environments are going to be used with an Elasticsearch database, you must migrate each of those environments, and each environment must use a different index prefix.
All commands are run from the Cortex XSOAR server machine.
To migrate your data, use the migration tool. You cannot run more than one migration tool process at a time.
Always migrate older data before newer data. Migrating partitions out of order can cause duplicate incident ids.
  1. Download the migration tool by appending
    downloadName=elasticsearch_migration_tool_6_1_0
    to the end of the download link that you received, when installing Cortex XSOAR.
  2. Copy your database and migrate data from the copy database to Elasticsearch.
    It is recommended to copy your data up to the last 3 months, without any downtime. If you do not want to do this, go to step 3.
    1. Copy the Cortex XSOAR database by doing one of the following:
    2. Copy the
      demisto.conf
      file.
    3. Edit the copy of the
      demisto.conf
      file, by adding your Elasticsearch configuration.
      Ensure that
      elasticsearch
      is the top-level object in the
      demisto.config
      file (within the main curly brackets).
    4. Using
      demisto
      or
      sudo
      permissions, run the following command:
      sudo ./elasticMigrator -config-path
      <file path-of-copy-of-demisto.conf>
      -db-path
      <path-of-the-copy-of-the-demisto-database>
      -
      <flags>
      For a full list of the flags, see Migration Tool Flags. For example, to exclude the last 3 partitions from the migration, add the
      -partitions-to-ignore
      flag and value to the command by typing the following:
      sudo ./elasticMigrator -config-path /usr/local/dev/copy_of_demisto.conf -db-path /usr/local/dev/lib_demisto_copy/data -partitions-to-ignore '042021,052021,062021'
      When you run the migration tool, parameter values specified in the
      demisto.conf
      file override values supplied for tool flags and default values. If no value exists in the
      demisto.conf
      file, values supplied in the tool flags override default values, but do not write the values to the
      demisto.config
      file.
    5. Complete steps 1 to 3 in Validate the migration.
  3. After the migration of the data is complete and validated, migrate your data from the active database to Elasticsearch.
    1. Create a backup copy of the
      demisto.conf
      file for your active database.
    2. Edit the
      demisto.conf
      for your active database to add your Elasticsearch configuration.
    3. Stop the Cortex XSOAR server by running one of the following commands:
      • CentOS:
        sudo systemctl stop demisto
      • Ubuntu:
        sudo service demisto stop
    4. Using
      demisto
      or
      sudo
      permissions, run the following command:
      sudo ./elasticMigrator -config-path
      <file path-of-demisto.conf>
      -db-path
      <path-of-the-demisto-database>
      -
      <flag>
      If you have any remaining data (such as the last 3 months partitions), migrate the remaining months from the active database to Elasticsearch, by adding the
      -partitions
      flag to the
      elasticMigrator
      command.
      For example,
      sudo ./elasticMigrator -partitions '
      042021,052021,062021
      '
      .
      When you run the migration tool, parameter values specified in the
      demisto.conf
      file override values supplied for tool flags and default values. If no value exists in the
      demisto.conf
      file, values supplied in the tool flags override default values, but do not write the values to the
      demisto.config
      file.

Migration Tool Flags

Flag
Type
Description
Required
accounts
(multi-tenant only)
String
A comma-separated list of accounts to migrate. If not specified, all accounts are migrated.
Optional
config-path
String
The path to the configuration file for the server. Default: /etc/demisto.conf.
Optional
db-path
String
The path to the database directory. Default: /var/lib/demisto.
Optional
elastic-batch-size
integer
The number of indicators per batch to write to Elasticsearch indexes. Default: 700.
Optional
elastic-index-prefix
String
The index prefix used in Elasticsearch.
Optional
elastic-key
String
The API key to connect to Elasticsearch.
Required (unless a username and password are used)
elastic-password
String
The password to connect to Elasticsearch.
required (unless API key is used)
elastic-url
String
The URL of your Elasticsearch environment. Default: http://localhost:9200.
Required
elastic-username
String
The username to connect to Elasticsearch.
required (unless API key is used)
ignore-ids-path
String
The path to the file with the IDs to ignore, per object.
Optional
log-level
String
The log level to display. Default: info.
Optional
logfile
String
The location of the log file.Default: /var/log/demisto/elastic_migration.log
Optional
migrate-all
Boolean
By default, the Elasticsearch tool checks existing indexes and migrates only the ones that are new. Using this flag, the Elasticsearch tool migrates all indexes even if they currently exist. This is useful, for example, if there was an error or invalid data that was fixed. When used, the objects-to-migrate and objects-to-ignore flags are ignored.
Optional
objects-to-ignore
String
Comma-separated list of objects not to migrate. When the migrate-all flag is used, this flag is ignored.
Optional
objects-to-migrate
String
Comma-separated list of objects to migrate. When the migrate-all flag is used, this flag is ignored.
Optional
partitions
String
Comma-separated list of partitions to migrate. If no partitions are specified, all partitions are migrated.
Optional
partitions-to-ignore
String
Comma-separated list of partitions to exclude.
Optional
previous-results
Show results of the previous migration.
Optional
skip-existing-indicators
false
Existing indicators are not modified during the migration.
Optional
version
Prints the migration tool version.
Optional
y
false
Answers yes to all questions, unless there is an error.
Optional

Recommended For You