Migrate Cortex XSOAR Objects to Elasticsearch for Multi-Tenant

Migrate Cortex XSOAR objects to Elasticsearch for a multi-tenant environment.
The migration tool migrates Cortex XSOAR objects to an Elasticsearch database. When you run the migration, the contents of the Cortex XSOAR database are read and a corresponding object is created in Elasticsearch.
You run the migration tool from the main machine and each host machine. Migrate the main account on the main host, then the tenants on the main host, and then proceed to the other hosts and tenant accounts. For complex environments with multiple host machines, we recommend migrating all tenants on the host before proceeding to the next host.
In the BoltDB, data related to incidents and indicators is stored in partitions by month. To minimize downtime during the migration, we recommend you create a copy of the database, then migrate data that is older than three months from the copy, while continuing to work in your current environment. Once the initial migration is completed, you should then migrate the last three months.
If you are working in an environment with remote repositories, you must run the migration separately on each environment. For example, if both your development and production environments are going to be used with an Elasticsearch database, you must migrate each of those environments, and each environment must use a different index prefix.
When you run the migration tool, parameter values identified in the
etc/demisto.conf
file override values supplied for tool flags and default values. If no value exists in the
demisto.conf
file, values supplied in the tool flags override default values, but do not write the values to the
demisto.conf
file. For example, if the
db-path
is identified in the configuration file, the tool will use the value in that file, not the value supplied or the default value, when running the tool.
You cannot run more than one migration tool process at a time.
Always migrate older data before newer data. Migrating partitions out of order can cause duplicate incident ids.
By default, the migration tool skips over objects larger than 100 megabytes. After the migration process runs, you can view the skipped large objects and determine whether to migrate them. For more information, see Validate the Migration to Elasticsearch.
  1. Download the migration tool by appending
    downloadName=elasticsearch_migration_tool_6_5_0
    to the end of the download link that you received, when installing Cortex XSOAR.
  2. For each host you want to migrate copy your database and migrate data from the copy database to Elasticsearch.
    1. Copy the Cortex XSOAR database by doing one of the following:
    2. Copy the
      demisto.conf
      file.
    3. Edit the copy of the
      demisto.conf
      file, by adding your Elasticsearch configuration.
      Ensure that
      elasticsearch
      is the top-level object in the
      demisto.config
      file (within the main curly brackets).
    4. From the host, using the
      demisto
      or
      sudo
      permissions, run the following command: command with either
      demisto
      or
      sudo
      permissions. Specify the following:
      sudo ./elasticMigrator -config-path <
      path-of-copy-of-demisto.conf
      > -db-path <
      path-of-the-copy-of-the-demisto-data
      > -
      <flags>
      For a full list of flags, see Migration Tool Flags.
      For example, to exclude the last 3 months partitions, add the
      -only-old-partitions
      and
      -partitions-back flags
      :
      sudo ./elasticMigrator -config-path /usr/local/dev/copy_of_demisto.conf -db-path /usr/local/dev/lib_demisto_copy/data -only-old-partitions -partitions-back 3
      By default, the migration migrates the entire contents of the database and the tenants on the host.
      • To migrate only the host, use the
        skip-accounts
        flag.
      • To migrate all accounts, use the
        skip-main
        flag.
      • To migrate specific accounts, use the
        accounts
        and
        skip-main
        flags.
    5. Complete steps 1 to 3 in Validate the migration
  3. After the migration of the data is complete and validated, migrate your data from the active database to Elasticsearch.
    1. Create a backup copy of the
      demisto.conf
      file for your active database.
    2. Edit the demisto.conf for your active database to add your Elasticsearch configuration.
    3. On the host (main or any other) stop the Cortex XSOAR server by running one of the following commands:
      • CentOS:
        sudo systemctl stop demisto
      • Ubuntu:
        sudo service demisto stop
    4. From the host, migrate the remaining months from the active database to Elasticsearch. Run the
      sudo ./elasticMigrator
      command with either
      demisto
      or
      sudo
      permissions. Use the
      -partitions-back
      flag to specify the remaining partitions.
      For example,
      sudo ./elasticMigrator -partitions-back 3
      migrates the last three partitions, which would include the current month and the previous two months, as well as the main partition.
      By default, the migration migrates the entire contents of the database and the tenants on the host.
      • To migrate only the host, use the
        skip-accounts
        flag.
      • To migrate all accounts, use the
        skip-main
        flag.
      • To migrate specific accounts, use the
        accounts
        and
        skip-main
        flags.
    5. For any additional hosts that you want to migrate, repeat the steps above.

Migration Tool Flags

Flag
Type
Description
Required
accounts
(multi-tenant only)
String
A comma-separated list of accounts to migrate. If not specified, all accounts are migrated.
Optional
config-path
String
The path to the configuration file for the server. Default:
/etc/demisto.conf
.
Optional
db-path
String
The path to the database directory. Default:
/var/lib/demisto
.
Optional
elastic-batch-size
Integer
The number of indicators per batch to write to Elasticsearch indexes. Default:
700
.
Optional
elastic-index-prefix
String
The index prefix used in Elasticsearch.
Optional
elastic-key
String
The API key to connect to Elasticsearch.
Required (unless a username and password are used)
elastic-password
String
The password to connect to Elasticsearch.
Required (unless API key is used)
elastic-url
String
The URL of your Elasticsearch environment. Default:
http://localhost:9200
.
Required
elastic-username
String
The username to connect to Elasticsearch.
Required (unless API key is used)
ignore-ids-path
String
The path to the file with the IDs to ignore, per object.
Optional
log-level
String
The log level to display. Default:
info
.
Optional
logfile
String
The location of the log file. Default:
/var/log/demisto/elastic_migration.log
.
Optional
log-failed-items
Integer
Log individual failed items, either in a single meta file, or file per item failure. Values:
0
: Does not log failed items (default).
1
: Logs failed items metadata in a single file.
2
: Logs each failed item in an individual file under the log folder. For more information, see Troubleshoot Elasticsearch
Optional
migrate-all
Boolean
By default, the Elasticsearch tool checks existing indexes and migrates only the ones that are new. Using this flag, the Elasticsearch tool migrates all indexes even if they currently exist. This is useful, for example, if there was an error or invalid data that was fixed. When used, the objects-to-migrate and objects-to-ignore flags are ignored. Values:
true
(default)
false
Optional
objects-to-ignore
String
Comma-separated list of objects not to migrate. When the migrate-all flag is used, this flag is ignored.
Optional
objects-to-migrate
String
Comma-separated list of objects to migrate. When the migrate-all flag is used, this flag is ignored.
Optional
partitions
String
Comma-separated list of partitions to migrate. If no partitions are specified, all partitions are migrated.
Optional
partitions-to-ignore
String
Comma-separated list of partitions to exclude.
Optional
previous-results
N/a
Show results of the previous migration.
Optional
skip-accounts
Boolean
Does not migrate multi-tenant accounts. When set to true, only the main tenant or host database are migrated. Values:
true
false
(default)
Optional
skip-existing-indicators
Boolean
Existing indicators are not modified during the migration. Values:
true
false
(default)
Optional
skip-mt-main
Boolean
Does not migrate the main host or host database. Values:
true
false
(default)
Optional
object-max-size
Integer
The maximum size, in megabytes, of objects that will be migrated to Elasticsearch. The default is 100 MB.
Optional
retry-large-objects
Boolean
Retry the migration of large objects when re-running the migration tool. With this flag, the entire bucket that contains the skipped large object is migrated again, which may include data that was previously migrated. If new data has been added in Elasticsearch since the earlier migration, this data will be overwritten.
Optional
force-main
Boolean
When the partitions flag is used, the
-force-main
flag forces the migration of the main partition as well as the specific list of partitions.
Optional
partitions-back
Integer
Provides an option to migrate X number of partitions back. For example, migrating 3 partitions back migrates the current month and the previous two months. If set to 0 or not used, all partitions are migrated.
Optional
only-old-partitions
Boolean
Can only be used with the partitions-back flag. Migrates all partitions other than the most recent partitions specified with partitions-back. For example,
-partitions-back 3 -only-old-partitions
migrates all partitions besides the current month and the two previous months. Should be used within the same calendar month as the previous migration.
version
N/a
Prints the migration tool version.
Optional
y
N/a
Answers yes to all questions, unless there is an error.
Optional

Recommended For You