End-of-Life (EoL)
Elasticsearch Sizing Requirements
There are several ways you can deploy Elasticsearch,
and each one has specific sizing requirements.
We recommend using Elasticsearch if you
plan to exceed at least one of the following maximum capacities
for BoltDB. For information regarding migrating indicators to Elasticsearch,
see Migrate Cortex XSOAR Objects to Elasticsearch.
The Cortex XSOAR indicators used to test the sizing requirements
did not contain a significant number of additional fields or custom
fields. The maximum size of the indicators we tested had 20 additional
or custom fields and a random string between 1-16 characters. Therefore,
the indicators size tested were approximately 0.5KB. If you plan
to have additional or custom fields for indicators, the maximum numbers
should be reduced.
Maximum indicator capacity and disk usage comparison
The following table compares the maximum total indicator
capacity and disk usage for BoltDB and Elasticsearch. The maximum
indicator capacity value was determined when testing the system.
Benchmark | BoltDB | Elasticsearch |
|---|---|---|
Maximum indicator capacity (total) | 5-7 million (Requires up to 10 seconds for
a complex query) | 100 million (Requires approximately 40 seconds
for a complex query) |
Disk usage | 23 million (~ 69 GB) | 100 million (~ 70 GB) |
If performance is poor, or you know in advance that you will
need more than the maximum number of indicators, you should consider
scaling BoltDB or moving to Elasticsearch. If you are already in
Elasticsearch, you can scale it as well. For both BoltDB and Elasticsearch,
you can scale by either adding engines for one or more feed integrations
or increasing the resources (CPU, RAM, Disk IOPS) of the Cortex XSOAR
server. For Elasticsearch, you can also increase the Elasticsearch
cluster size from 1 server to 2 or more servers.
Maximum number of indicators in a single fetch
The following table compares the maximum number of indicators
in a single fetch for BoltDB and Elasticsearch.
Benchmark | BoltDB | Elasticsearch |
|---|---|---|
Maximum number of indicators per single fetch
(one-time) | 1 million | 3 million |
Single feed fetch comparison
The following table compares the number of indicators,
time to ingestion, and disk usage for BoltDB and Elasticsearch.
Number of Indicators | Database | Time to Ingestion | Disk Usage |
|---|---|---|---|
30k | BoltDB | 16s | 1.3 GB |
Elasticsearch | 11s | 1.08 GB + 161 MB (Elasticsearch index) | |
50k | BoltDB | 33s | 1.45 GB |
Elasticsearch | 25s | 1.08 GB + 26.7 MB (Elasticsearch index) | |
100k | BoltDB | 1m8s | 2.1 GB |
Elasticsearch | 49s | 1.08 GB + 53 MB (Elasticsearch index) | |
1M | BoltDB | 12m21s | 13.5 GB |
Elasticsearch | 7m25s | 1.08 GB + 570MB (Elasticsearch index) | |
2M | BoltDB | 22m27s | 32 GB |
Elasticsearch | 22m20s | 1.23 GB + 1 GB (Elasticsearch index) |
Elasticsearch server
Component | Dev Environment Minimum | Production Minimum |
|---|---|---|
CPU | 8 CPU Cores | 16 CPU cores |
Memory | 16 GB RAM | 32 GB RAM |
Storage | 500 GB SSD | 1 TB SSD with minimum 3k dedicated IOPS |
Additional Configurations
It is recommended that you implement the following Elasticsearch configurations
in Cortex XSOAR.
Set the number of shards for an index
This server configuration enables you to set the number of shards
for a specific index upon creation, where
<common-indicator>
is
the name of the index. You should set the value to the number of
CPU cores in the Elasticsearch dedicated server. The default is
1.elasticsearch.shards.
<common-indicator>
Set the number of replica shards for an index
This server configuration enables you to set the number of replica
shards for a specific index upon creation, where
<common-indicator>
is
the name of the index. You should set the value to the number of
backups you require. The default is 0.elasticsearch.replicas.
<common-indicator>
Recommended For You
Recommended Videos
Recommended videos not found.
