Sizing for Cortex Data Lake Storage

Learn how to properly size storage for Cortex Data Lake.
Cortex Data Lake is a cloud-based service for secure storage of Palo Alto Networks firewall logs regardless of form factor, location, or scale. When purchasing Palo Alto Networks devices or services, log storage is an important consideration. Ensuring sufficient log retention enables operations by ensuring data is available to administrators for troubleshooting and incident response. Maintaining a healthy backlog of data allows you to fully utilize various Palo Alto Networks products.

Sizing Considerations

When planning a log collection infrastructure, there are some considerations that dictate how much storage needs to be provided:
  • Average size of a log.
  • Log rate for NGFWs.
  • Throughput and number of users for Prisma Access.
  • Desired retention period.

Log Sizes

All firewall logs (including Traffic, Threat, URL, etc.) have an average size of 1500 bytes when stored in Cortex Data Lake. This number may change as new features and log fields are introduced. When this happens, the CDL Estimator will be updated to reflect the current status.

Log Rate

For both physical and virtual firewall platforms, there are several methods for calculating log rate based on predefined connections-per-second.

Throughput and Users

Occasionally, it is not practical to directly measure or estimate what the log rate will be. Examples of these cases are when sizing for Prisma Access. Different use cases, such as remote networks and mobile users, use different metrics, like throughput and the number of users.

Log Retention

There are several, mostly regulatory, factors that drive log storage requirements. Users may need to meet compliance requirements for HIPPA, PCI, or Sarbanes-Oxley:
There may be other governmental or industry standards, including some internal standards within your company.

Methods for Sizing

You can size storage for Cortex Data Lake using three different methods:
  1. Based on log rate: This will be the most accurate method.
  2. Based on throughput: This is used when sizing storage for Prisma Access (Remote Networks).
  3. Based on user count: This is used when sizing storage for Prisma Access (Mobile Users).

Calculate Storage with the Cortex Data Lake Estimator

You can use this app to estimate the amount of Cortex Data Lake storage you may need to purchase.
Select which products you will be using in your network, and enter the necessary metrics mentioned above, to estimate your recommended purchase for sufficient log retention.

Next-Generation Firewall

The Next-Generation Firewall section allows you to size based on Log Rate:
This is a traditional log-rate based estimator for firewalls. The only input required is log rate and desired retention date (in days).
If you are unable to calculate your own log rate, select
I don’t know the log rate
to estimate your log rate using the number of deployed firewalls and their utilization percentages.

Prisma Access (Remote Networks)

The Prisma Access (Remote Networks) section allows you to size based on bandwidth:
This option requires more data to provide an accurate number. Prisma Access (Remote Networks) is sold according to throughput. When 100Mbps is purchased and allocated to a location, it's not likely that the link will see 100% utilization all of the time. In addition to entering the throughput purchased, the estimator requires desired retention period (in days) and utilization data for production and non-production hours.

Prisma Access (Mobile Users)

The Prisma Access (Mobile Users) section allows you to determine how much storage you need based on the number of mobile users:
The only input required is the number of users and desired retention period (in days).

Cortex XDR Agents

The Cortex XDR Agents section allows you to determine how much storage you need based on XDR Agent deployments:
The quota allocation calculator for Cortex XDR Agents reflects the estimated size of a given deployment scenario as derived from the number of Cortex XDR Agents and the allocated storage size. The only input required is the total XDR Pro endpoints forwarding Cortex XDR data and desired retention period (in days).

Cortex XDR

The Cortex XDR section allows you to determine how much storage you need based on Cortex XDR utilization:
Cortex XDR increases storage demand across all other products. It requires Enhanced Application Logs, which are streamed to XDR for storage and Network Traffic Analysis. When you select this option, the estimator automatically calculates the increase in storage demand for all other sections highlighted.

IoT Security

The IOT Security section allows you to determine how much storage you need based on Cortex XDR utilization:
IoT Security increases storage demand across firewalls. It requires Enhanced Application Logs, which are streamed in order to discover IoT/OT devices, identify risks, security threats, and anomalies, and to perform analytics. When you select this option, the estimator automatically calculates the increase in storage demand for all other sections highlighted.
For a traditional NGFW deployment, log rate will still yield the most accurate numbers for log storage. In cases where measuring or estimating the log rate isn't practical, you can size based on bandwidth using the Prisma Access (Remote Networks) section.

Recommended For You