Migrate Objects to Elasticsearch for a Distributed Database
Follow these instructions to migrate Cortex XSOAR objects to Elasticsearch from a distributed database environment.
You should migrate Cortex XSOAR objects to Elasticsearch if you plan to ingest a large amount of objects.
When you run the migration tool, the contents of the Cortex XSOAR database are read, and a corresponding object is created in Elasticsearch. The migration tool is run from the main database machine and from each database node.
In the BoltDB, data related to incidents and indicators is stored in partitions by month. To minimize downtime during the migration, we recommend you create a copy of the database, then migrate data that is older than three months from the copy, while continuing to work in your current environment. Once the initial migration is completed, you should then migrate the last three months.
All commands are run from the Cortex XSOAR server machine.
To migrate your data, use the migration tool. You cannot run more than one migration tool process at a time.
Always migrate older data before newer data. Migrating partitions out of order can cause duplicate incident ids.
- For the main database, copy the Cortex XSOAR database either by taking a snapshot OR manually create a copy of the/var/lib/demisto/datadirectory and thedemisto.conffile. Then follow the same procedure for each node.
- Download the migration tool by appendingdownloadName=elasticsearch_migration_tool_6_2_0to the end of the download link that you received, when installing Cortex XSOAR.
- Copy your database and migrate data from the copy database to Elasticsearch.
- Copy the Cortex XSOAR database by doing one of the following:
- Copy thedemisto.conffile.
- Edit the copy of thedemisto.conffile, by adding your Elasticsearch configuration.Ensure that the same Elasticsearch object exists in thedemisto.confon the app server and the main database and hatelasticsearchis the top-level object in thedemisto.configfile (within the main curly brackets).Edit the copies ofdemisto.conffor the main database and for each node to add your Elasticsearch configuration.
- For the main database and for each node, usingdemistoorsudopermissions, run the following command:sudo ./elasticMigrator -config-path<file path-of-copy-of-demisto.conf>-db-path<path-of-the-copy-of-the-demisto-database>-<flags>For a full list of the flags, see Migration Tool Flags. For example, to exclude the last 3 partitions from the migration, add the-partitions-to-ignoreflag and value to the command by typing the following:sudo ./elasticMigrator -config-path /usr/local/dev/copy_of_demisto.conf -db-path /usr/local/dev/lib_demisto_copy/data -partitions-to-ignore '042021,052021,062021'When you run the migration tool, parameter values specified in thedemisto.conffile override values supplied for tool flags and default values. If no value exists in thedemisto.conffile, values supplied in the tool flags override default values, but do not write the values to thedemisto.configfile.
- Complete steps 1 to 3 in Validate the migration.
- After the migration of the data is complete and validated, migrate your data from the active database to Elasticsearch.
- Create a backup copy of thedemisto.conffile for your active database.
- Edit the originaldemisto.conffile (not a copy) for the main database and for each node to add your Elasticsearch configuration. Ensure that the same Elasticsearch object exists in thedemisto.confon the app server and the main database.
- For the main database and for each node, stop the Cortex XSOAR server.
- CentOS:sudo systemctl stop demisto
- Ubuntu:sudo service demisto stop
- For the main database and for each node, run thesudo ./elasticMigratorcommand with eitherdemistoorsudopermissions. Use the-partitionsflag to specify the remaining partitions.Example:sudu ./elasticMigrator -partitions '042021,052021,062021'
- Validate the migration (all steps).
Migration Tool Flags
A comma-separated list of accounts to migrate. If not specified, all accounts are migrated.
The path to the configuration file for the server. Default:
The path to the database directory. Default:
The number of indicators per batch to write to Elasticsearch indexes. Default:
The index prefix used in Elasticsearch.
The API key to connect to Elasticsearch.
Required (unless a username and password are used)
The password to connect to Elasticsearch.
Required (unless API key is used)
The URL of your Elasticsearch environment. Default:
The username to connect to Elasticsearch.
Required (unless API key is used)
The path to the file with the IDs to ignore, per object.
The log level to display. Default:
The location of the log file. Default:
By default, the Elasticsearch tool checks existing indexes and migrates only the ones that are new. Using this flag, the Elasticsearch tool migrates all indexes even if they currently exist. This is useful, for example, if there was an error or invalid data that was fixed. When used, the objects-to-migrate and objects-to-ignore flags are ignored. Values:
Comma-separated list of objects not to migrate. When the migrate-all flag is used, this flag is ignored.
Comma-separated list of objects to migrate. When the migrate-all flag is used, this flag is ignored.
Comma-separated list of partitions to exclude.
Comma-separated list of partitions to migrate. If no partitions are specified, all partitions are migrated.
Show results of the previous migration.
Existing indicators are not modified during the migration. Values:
Prints the migration tool version.
Answers yes to all questions, unless there is an error.
Recommended For You
Recommended videos not found.