End-of-Life (EoL)

Elasticsearch Sizing Requirements

There are several ways you can deploy Elasticsearch, and each one has specific sizing requirements.
We recommend using Elasticsearch if you plan to exceed at least one of the following maximum capacities for BoltDB. For information regarding migrating indicators to Elasticsearch, see Migrate Cortex XSOAR Objects to Elasticsearch.
The Cortex XSOAR indicators used to test the sizing requirements did not contain a significant number of additional fields or custom fields. The maximum size of the indicators we tested had 20 additional or custom fields and a random string between 1-16 characters. Therefore, the indicators size tested were approximately 0.5KB. If you plan to have additional or custom fields for indicators, the maximum numbers should be reduced.

Maximum indicator capacity and disk usage comparison

The following table compares the maximum total indicator capacity and disk usage for BoltDB and Elasticsearch. The maximum indicator capacity value was determined when testing the system.
Benchmark
BoltDB
Elasticsearch
Maximum indicator capacity (total)
5-7 million
(Requires up to 10 seconds for a complex query)
100 million
(Requires approximately 40 seconds for a complex query)
Disk usage
23 million (~ 69 GB)
100 million (~ 70 GB)
If performance is poor, or you know in advance that you will need more than the maximum number of indicators, you should consider scaling BoltDB or moving to Elasticsearch. If you are already in Elasticsearch, you can scale it as well. For both BoltDB and Elasticsearch, you can scale by either adding engines for one or more feed integrations or increasing the resources (CPU, RAM, Disk IOPS) of the Cortex XSOAR server. For Elasticsearch, you can also increase the Elasticsearch cluster size from 1 server to 2 or more servers.

Maximum number of indicators in a single fetch

The following table compares the maximum number of indicators in a single fetch for BoltDB and Elasticsearch.
Benchmark
BoltDB
Elasticsearch
Maximum number of indicators per single fetch (one-time)
1 million
3 million

Single feed fetch comparison

The following table compares the number of indicators, time to ingestion, and disk usage for BoltDB and Elasticsearch.
Number of Indicators
Database
Time to Ingestion
Disk Usage
30k
BoltDB
16s
1.3 GB
Elasticsearch
11s
1.08 GB + 161 MB (Elasticsearch index)
50k
BoltDB
33s
1.45 GB
Elasticsearch
25s
1.08 GB + 26.7 MB (Elasticsearch index)
100k
BoltDB
1m8s
2.1 GB
Elasticsearch
49s
1.08 GB + 53 MB (Elasticsearch index)
1M
BoltDB
12m21s
13.5 GB
Elasticsearch
7m25s
1.08 GB + 570MB (Elasticsearch index)
2M
BoltDB
22m27s
32 GB
Elasticsearch
22m20s
1.23 GB + 1 GB (Elasticsearch index)

Elasticsearch server

Component
Dev Environment Minimum
Production Minimum
CPU
8 CPU Cores
16 CPU cores
Memory
16 GB RAM
32 GB RAM
Storage
500 GB SSD
1 TB SSD with minimum 3k dedicated IOPS

Additional Configurations

It is recommended that you implement the following Elasticsearch configurations in Cortex XSOAR.
Set the number of shards for an index
This server configuration enables you to set the number of shards for a specific index upon creation, where
<common-indicator>
is the name of the index. You should set the value to the number of CPU cores in the Elasticsearch dedicated server. The default is 1.
elasticsearch.shards.
<common-indicator>
Set the number of replica shards for an index
This server configuration enables you to set the number of replica shards for a specific index upon creation, where
<common-indicator>
is the name of the index. You should set the value to the number of backups you require. The default is 0.
elasticsearch.replicas.
<common-indicator>

Recommended For You