Elasticsearch System Requirements - Administrator Guide - 6.6 - Cortex XSOAR - Cortex - Security Operations

Cortex XSOAR Administrator Guide

Product
Cortex XSOAR
Version
6.6
Creation date
2022-09-29
Last date published
2024-03-21
End_of_Life
EoL
Category
Administrator Guide
Abstract

There are several options for Elasticsearch deployment, each with specific sizing requirements. Requirements for implementing Cortex XSOAR with Elasticsearch.

This topic provides information about the system requirements for implementing Cortex XSOAR with Elasticsearch.

Elasticsearch server

The information in the following table is per Elasticsearch node, and assumes that the node is assigned all Elasticsearch node roles (for example, which data is written to disk and when).

Component

Dev Minimum

Prod Minimum

Prod Recommended

CPU

8 CPU Cores

16 CPU cores

36 CPU Cores

Memory

16 GB RAM

32 GB RAM

64 GB RAM

Storage

250 GB SSD

500 GB SSD with minimum 3k dedicated IOPS

1 TB SSD

Note

We recommend starting with the production minimum. If the production minimum is not sufficient for your needs (for cases such as managing many tenants in a multi-tenant environment, high memory use, large number of processes, etc.) we recommend upgrading to the production recommended specifications.

Elasticsearch user permissions are available in the security guidelines.

Note

You must ensure that between the Elasticsearch and Cortex XSOAR servers, and between Elasticsearch servers, latency should not exceed 100 MS. Latency that exceeds 100 MS can cause serious performance degradation. For optimal performance, we recommend 10 MS or lower.

Supported Versions
  • Elasticsearch: Cortex XSOAR supports Elasticsearch versions 7.4 to 7.17 including minor versions.

  • OpenSearch: Cortex XSOAR supports OpenSearch versions 1.0 to 1.2 including minor versions.

Elasticsearch in the Cloud

Cortex XSOAR supports using Elasticsearch with all the major cloud service providers, Amazon Web Services, Azure, and Google Cloud Platform.

Note

For Opensearch, ensure that the AWS instance type supports a maximum HTTP payload of 100 MB, which is sufficient for production usage. For more information, see Amazon OpenSearch Instance Limits.

You can use Elasticsearch as a service provided by your cloud provider, or install Elasticsearch on a server in the cloud.

The hardware requirements for Elasticsearch in the cloud similar to those posted above. To achieve this with your cloud provider, Cortex XSOAR recommends you use the machines based on your intentions. For example:

  • When the Elasticsearch server functions as a data node, we recommend you use storage optimized machines, such as the AWS i3.2xlarge machine. Alternatively, you can use a memory optimized machine, such as the AWS r3.2xlarge machine. On managed Opensearch, we recommend using c5.4xlarge.search as a minimum, or c5.9xlarge.search for higher scale.

  • When the Elasticsearch server is used for any other function (such as master mode), we recommend that you use a Compute optimized machine, such as the AWS c4.2xlarge machine. On managed Opensearch, we recommend using m5.xlarge.search.

You can configure your cloud environment to work with different regions provided that you can maintain the minimum latency requirements noted above.

General Configurations

It is recommended that you implement the following Elasticsearch configurations in Cortex XSOAR. The value of the shards and replica shards should match the sum total of Elasticsearch nodes that you have.

Set the number of shards for an index

This server configuration enables you to set the number of shards for a specific index upon creation, where <common-indicator> is the name of the index. The default is 1.

To improve the write-performance, you can increase the number of shards and decrease the number of replica shards.

elasticsearch.shards.<common-indicator>

Set the number of replica shards for an index

This server configuration enables you to set the number of replica shards for a specific index upon creation, where <common-indicator> is the name of the index. To increase search performance and data redundancy, you should set the value to the number of Elasticsearch nodes that you have. The default is 1.

elasticsearch.replicas.<common-indicator>

Maximum indicator capacity and disk usage comparison

The following table compares the maximum total indicator capacity and disk usage for BoltDB and Elasticsearch. The maximum indicator capacity value was determined when testing the system.

We recommend using Elasticsearch if you plan to exceed at least one of the following maximum capacities for BoltDB.

The Cortex XSOAR indicators used to test the sizing requirements did not contain a significant number of additional fields nor custom fields. The maximum size of the indicators we tested had 20 additional or custom fields and a random string between 1-16 characters. Therefore, the indicators size tested were approximately 0.5KB. If you plan to have additional or custom fields for indicators, the maximum numbers should be reduced.

Benchmark

BoltDB

Elasticsearch

Maximum indicator capacity (total)

5-7 million

(Requires up to 10 seconds for a complex query)

100 million

(Requires approximately 40 seconds for a complex query)

Disk usage

5 million (~ 30 GB)

100 million (~ 70 GB)

If performance is poor, or you know in advance that you will need more than the maximum number of indicators, you should consider scaling BoltDB or moving to Elasticsearch. If you are already in Elasticsearch, you can scale it as well. For both BoltDB and Elasticsearch, you can scale by either adding engines for one or more feed integrations or increasing the resources (CPU, RAM, Disk IOPS) of the Cortex XSOAR server. For Elasticsearch, you can also increase the Elasticsearch cluster size from 1 server to 2 or more servers.

Incident disk usage comparison

The following table compares the disk usage for BoltDB and Elasticsearch.

Number of Incidents

BoltDB

Elasticsearch

10,000

28GB

13GB

40,000

112GB

52GB

100,000

280GB

130GB

Single feed fetch comparison

The following table compares the number of indicators, time to ingestion, and disk usage for BoltDB and Elasticsearch.

Number of Indicators

Database

Time to Ingestion

Disk Usage

30k

BoltDB

16s

1.3 GB

Elasticsearch

11s

1.08 GB + 161 MB (Elasticsearch index)

50k

BoltDB

33s

1.45 GB

Elasticsearch

25s

1.08 GB + 26.7 MB (Elasticsearch index)

100k

BoltDB

1m8s

2.1 GB

Elasticsearch

49s

1.08 GB + 53 MB (Elasticsearch index)

1M

BoltDB

12m21s

13.5 GB

Elasticsearch

7m25s

1.08 GB + 570MB (Elasticsearch index)

2M

BoltDB

22m27s

32 GB

Elasticsearch

22m20s

1.23 GB + 1 GB (Elasticsearch index)