Migrate Cortex XSOAR Objects to Elasticsearch for Multi-Tenant
Migrate Cortex XSOAR objects to Elasticsearch
for a multi-tenant environment.
The migration tool migrates Cortex XSOAR objects
to an Elasticsearch database. When you run the migration, the contents
of the Cortex XSOAR database are read and a corresponding object
is created in Elasticsearch.
You run the migration tool from
the main machine and each host machine. Migrate the main account
on the main host, then the tenants on the main host, and then proceed
to the other hosts and tenant accounts. For complex environments
with multiple host machines, we recommend migrating all tenants
on the host before proceeding to the next host.
In the BoltDB,
data related to incidents and indicators is stored in partitions
by month. To minimize downtime during the migration, we recommend
you create a copy of the database, then migrate data that is older
than three months from the copy, while continuing to work in your
current environment. Once the initial migration is completed, you
should then migrate the last three months.
If you are
working in an environment with remote repositories, you must run
the migration separately on each environment. For example, if both
your development and production environments are going to be used
with an Elasticsearch database, you must migrate each of those environments,
and each environment must use a different index prefix.
When
you run the migration tool, parameter values identified in the
etc/demisto.conf
file
override values supplied for tool flags and default values. If no
value exists in the demisto.conf
file, values
supplied in the tool flags override default values, but do not write
the values to the demisto.conf
file. For example,
if the db-path
is identified in the configuration
file, the tool will use the value in that file, not the value supplied
or the default value, when running the tool.You cannot
run more than one migration tool process at a time.
Always
migrate older data before newer data. Migrating partitions out of
order can cause duplicate incident ids.
By default,
the migration tool skips over objects larger than 100 megabytes.
After the migration process runs, you can view the skipped large
objects and determine whether to migrate them. For more information,
see Validate the Migration to Elasticsearch.
- Download the migration tool by appendingdownloadName=elasticsearch_migration_tool_6_5_0to the end of the download link that you received, when installing Cortex XSOAR.
- For each host you want to migrate copy your database and migrate data from the copy database to Elasticsearch.
- Copy the Cortex XSOAR database by doing one of the following:
- Take a snapshot of the database.
- Manually create a copy of the/var/lib/demisto/directory.
- Copy thedemisto.conffile.
- Edit the copy of thedemisto.conffile, by adding your Elasticsearch configuration.Ensure thatelasticsearchis the top-level object in thedemisto.configfile (within the main curly brackets).
- From the host, using thedemistoorsudopermissions, run the following command: command with eitherdemistoorsudopermissions. Specify the following:sudo ./elasticMigrator -config-path <path-of-copy-of-demisto.conf> -db-path <path-of-the-copy-of-the-demisto-data> -<flags>For a full list of flags, see Migration Tool Flags.For example, to exclude the last 3 months partitions, add the-only-old-partitionsand-partitions-back flags:sudo ./elasticMigrator -config-path /usr/local/dev/copy_of_demisto.conf -db-path /usr/local/dev/lib_demisto_copy/data -only-old-partitions -partitions-back 3By default, the migration migrates the entire contents of the database and the tenants on the host.
- To migrate only the host, use theskip-accountsflag.
- To migrate all accounts, use theskip-mainflag.
- To migrate specific accounts, use theaccountsandskip-mainflags.
- Complete steps 1 to 3 in Validate the migration
- After the migration of the data is complete and validated, migrate your data from the active database to Elasticsearch.
- Create a backup copy of thedemisto.conffile for your active database.
- Edit the demisto.conf for your active database to add your Elasticsearch configuration.
- On the host (main or any other) stop the Cortex XSOAR server by running one of the following commands:
- CentOS:sudo systemctl stop demisto
- Ubuntu:sudo service demisto stop
- From the host, migrate the remaining months from the active database to Elasticsearch. Run thesudo ./elasticMigratorcommand with eitherdemistoorsudopermissions. Use the-partitions-backflag to specify the remaining partitions.For example,sudo ./elasticMigrator -partitions-back 3migrates the last three partitions, which would include the current month and the previous two months, as well as the main partition.By default, the migration migrates the entire contents of the database and the tenants on the host.
- To migrate only the host, use theskip-accountsflag.
- To migrate all accounts, use theskip-mainflag.
- To migrate specific accounts, use theaccountsandskip-mainflags.
- Validate the migration (all steps).For any additional hosts that you want to migrate, repeat the steps above.
Migration Tool Flags
Flag | Type | Description | Required |
---|---|---|---|
accounts (multi-tenant
only) | String | A comma-separated list of accounts to migrate.
If not specified, all accounts are migrated. | Optional |
config-path | String | The path to the configuration file for the
server. Default: /etc/demisto.conf . | Optional |
db-path | String | The path to the database directory. Default: /var/lib/demisto . | Optional |
elastic-batch-size | Integer | The number of indicators per batch to write
to Elasticsearch indexes. Default: 700 . | Optional |
elastic-index-prefix | String | The index prefix used in Elasticsearch. | Optional |
elastic-key | String | The API key to connect to Elasticsearch. | Required (unless a username and password are used) |
elastic-password | String | The password to connect to Elasticsearch. | Required (unless API key is used) |
elastic-url | String | The URL of your Elasticsearch environment.
Default: http://localhost:9200 . | Required |
elastic-username | String | The username to connect to Elasticsearch. | Required (unless API key is used) |
ignore-ids-path | String | The path to the file with the IDs to ignore,
per object. | Optional |
log-level | String | The log level to display. Default: info . | Optional |
logfile | String | The location of the log file. Default: /var/log/demisto/elastic_migration.log . | Optional |
log-failed-items | Integer | Log individual failed items, either in a single
meta file, or file per item failure. Values: 0 : Does
not log failed items (default).1 : Logs
failed items metadata in a single file.2 : Logs
each failed item in an individual file under the log folder. For
more information, see Troubleshoot Elasticsearch | Optional |
migrate-all | Boolean | By default, the Elasticsearch tool checks existing
indexes and migrates only the ones that are new. Using this flag,
the Elasticsearch tool migrates all indexes even if they currently
exist. This is useful, for example, if there was an error or invalid
data that was fixed. When used, the objects-to-migrate and objects-to-ignore
flags are ignored. Values: true (default)false | Optional |
objects-to-ignore | String | Comma-separated list of objects not to migrate.
When the migrate-all flag is used, this flag is ignored. | Optional |
objects-to-migrate | String | Comma-separated list of objects to migrate.
When the migrate-all flag is used, this flag is ignored. | Optional |
partitions | String | Comma-separated list of partitions to migrate.
If no partitions are specified, all partitions are migrated. | Optional |
partitions-to-ignore | String | Comma-separated list of partitions to exclude. | Optional |
previous-results | N/a | Show results of the previous migration. | Optional |
skip-accounts | Boolean | Does not migrate multi-tenant accounts. When
set to true, only the main tenant or host database are migrated.
Values: true false (default) | Optional |
skip-existing-indicators | Boolean | Existing indicators are not modified during
the migration. Values: true false (default) | Optional |
skip-mt-main | Boolean | Does not migrate the main host or host database.
Values: true false (default) | Optional |
object-max-size | Integer | The maximum size, in megabytes, of objects
that will be migrated to Elasticsearch. The default is 100 MB. | Optional |
retry-large-objects | Boolean | Retry the migration of large objects when re-running
the migration tool. With this flag, the entire bucket that contains
the skipped large object is migrated again, which may include data
that was previously migrated. If new data has been added in Elasticsearch
since the earlier migration, this data will be overwritten. | Optional |
force-main | Boolean | When the partitions flag is used, the -force-main flag
forces the migration of the main partition as well as the specific
list of partitions. | Optional |
partitions-back | Integer | Provides an option to migrate X number of partitions
back. For example, migrating 3 partitions back migrates the current
month and the previous two months. If set to 0 or not used, all
partitions are migrated. | Optional |
only-old-partitions | Boolean | Can only be used with the partitions-back flag.
Migrates all partitions other than the most recent partitions specified
with partitions-back. For example, -partitions-back 3 -only-old-partitions migrates
all partitions besides the current month and the two previous months.
Should be used within the same calendar month as the previous migration. | |
version | N/a | Prints the migration tool version. | Optional |
y | N/a | Answers yes to all questions, unless there
is an error. | Optional |
Recommended For You
Recommended Videos
Recommended videos not found.