What’s New in Cortex Data Lake

Here are the new features in Cortex Data Lake.
Feature
Description
Easy Activation
September 2021
Cortex Data Lake now features a simplified activation flow to help you get up and running with the product quickly and easily. After you purchase a Cortex Data Lake license, you now receive an email with a link that takes you to a step-by-step process for activating your product.
India Regional Support
August 2021
To comply with data privacy regulations that require you to keep data within India regional boundaries, you can now select India as a host region when you activate Cortex Data Lake.
Saved and Shared Filters
August 2021
You can now save log queries and share them with other users. Save log queries to avoid re-entering long, complex, or frequently used queries each time you want to see a particular set of logs. Share queries to quickly present the logs to a team member, support technician, or anyone whom you want to see them.
Saved Log Viewer Profiles
August 2021
In the log viewer, you can now create profiles that save preferences so that you can quickly change to a set of preferences for a particular use case or user.
These preferences include the Cloud Identity Engine (CIE) tenant, the time zone in which logs appear, and the columns you’ve chosen to display as well as their order.
Query Builder Enhancements
July 2021
The character limit for queries has increased to 4096, and queries now wrap to the next line when the field is filled. This enables you to form longer queries and view their contents at a glance.
Time Zone Selection
July 2021
You can now choose to view logs in different time zones. This helps you correlate logs generated by different products that may use a different time zone from the timezone of your browser.
Millisecond-Level Queries
July 2021
You can now create queries with the
time_generated_high_res
field
equal_to
a time in milliseconds. This enables you to correlate logs with events from other systems at a millisecond level.
Default User Preferences
July 2021
You can now restore preferences, such as column order and time zone, to the preferences set when you first started the app. This enables you to quickly undo any changes you’ve made if you are no longer satisfied with your preferences.
Log Viewer Admin Role
July 2021
Cortex Data Lake now has a new role that only grants permission to view the
Explore
tab and export log data. If one of your users only needs to view logs, this enables you to maintain a good security posture by only granting them the permissions they need.
Germany Regional Support
July 2021
To comply with data privacy regulations that require you to keep data within German regional boundaries, you can now select Germany as a host region when you activate Cortex Data Lake.
Filter Query Parentheses Support
June 2021
The log viewer filter now supports parentheses to determine the order in which it evaluates terms in queries so you can more precisely identify the logs you’re looking for.
Firewall Data Retention Toggle
June 2021
For better control over your log data, you can now disable log retention for each of your firewalls from the
Inventory
tab in the Cortex Data Lake app. To do this, set
Store Log Data
to
Off
for the firewalls whose data you do not want to retain.
Device Certificate for Cortex Data Lake
June 2021
(
PAN-OS 10.1 or later
) To reduce the number of certificates you need to install and manage to connect to Palo Alto Networks cloud services, you can now authenticate to Cortex Data Lake using a device certificate. This enables you to authenticate to Cortex Data Lake using the same certificate that you would use to connect to Cortex XDR, IoT Security, and Enterprise Data Loss Prevention.
Devices using a device certificate follow a new process to onboard to Cortex Data Lake. Make sure to follow the onboarding process appropriate for your PAN-OS version and deployment style.
Self-Signed Certificate Support
April 2021
You can now get started forwarding logs from Cortex Data Lake more quickly, easily, and cost-effectively by using a self-signed certificate to authenticate your syslog or HTTPS receiver. After installing the certificate on your receiver, you can upload the private CA or self-signed certificate as part of your syslog or HTTPS forwarding profiles.
Log Forwarding Certificate Validation Enhancement
March 2021
To ensure your log data arrives safely to its intended destination, Cortex Data Lake now more rigorously inspects the validity of server certificates.
Log Forwarding Connection Check
March 2021
To help you verify that you can connect to the syslog server to which you want to forward logs, Cortex Data Lake Log Forwarding now features a
Test Connection
button in Syslog and HTTPS profile configuration. When you click this button, you will see that the connection either succeeded or failed and why.
HTTPS Log Forwarding
March 2021
For compatibility with services that receive events through HTTPS, such as Splunk HTTP Event Collector (HEC), Cortex Data Lake now supports forwarding logs through HTTPS.
Common Event Format (CEF) Support
March 2021
Enabling you to forward logs to Microfocus ArcSight Enterprise Security Manager, Cortex Data Lake now supports CEF as an option when you select the log format for a syslog forwarding profile.
No Data Retention
March 2021
For better control over your log data, Cortex Data Lake now does not retain logs at all if you set log storage
Quota
or
Max Retention Days
to 0 in
Storage
Configuration
. If you do want to store logs, ensure that
Quota
is greater than 0 and
Max Retention Days
is not set to 0.
Related Log Events
February 2021
Certain network logs—Traffic, Threat, URL, File—now show you the other events logged during the same session.
Without leaving the context of the log you’re interested in, you can see the sequence of related events for the session. Related logs are displayed chronologically, top to bottom—the log with the earliest timestamp is listed first.
Select a related log to investigate the details for that event.
Log Format Updates
February 2021
To take advantage of these features, you must edit and resubmit your log forwarding profiles.
New Log Fields
—To support the transport of richer data about your network traffic, Cortex Data Lake now processes new log fields from PAN-OS: device group (DG) hierarchy and secure web gateway (SWG) fields. The DG hierarchy field helps you identify which firewall Device Group generated a log, and SWG fields provide more detailed user Authentication information.
New Email Log Format
—For better consistency across log outputs, the log fields in email log forwarding now more closely resemble other supported formats, such as LEEF and the format used in
Explore
. This does not affect email forwarding profiles that were migrated from an older version of
Log Forwarding
.
Log Field Modification
—For better consistency with other log fields, the
ProfileToken
field now has the first letter capitalized. If you reference this field in automation scripts, ensure that it reads
ProfileToken
.
Log Forwarding Filter Updates
February 2021
To take advantage of these features, you must edit and resubmit your log forwarding profiles.
Editable Migrated Filters
—You now have the flexibility to modify the queries in log forwarding filters that you may have retained from an earlier version of the Log Forwarding app.
Migrated filters will not tell you if a query that you entered is valid. To validate a query, create a new filter and test it there. When you determine the query works, then paste it into the migrated filter.
Filter Deletion Confirmation
—To prevent you from accidentally deleting log forwarding filters, filter deletion is now a two-step process.
In-App Device Connection Management
January 2021
For smoother device onboarding, you can now view a list of your available Panorama and firewall devices and generate onboarding keys for them within the app.
Redesigned UI
January 2021
To provide a more consistent experience across Palo Alto Networks platforms, Cortex Data Lake now features a new user interface that you may recognize from products such as Prisma Access.
Explore Integration
January 2021
Instead of switching to a different app, you can now search, filter, and export logs directly within the Cortex Data Lake app. Select
Explore
in the app’s new sidebar to get started.
Australia Regional Support
December 2020
To comply with data privacy regulations that require you to keep data within Australian regional boundaries, you can now select Australia as a host region when you activate Cortex Data Lake.
Log Forwarding Integration
November 2020
You can now forward logs from within the Cortex Data Lake app, enabling you to conveniently manage onboarding, storage, and log transmission in a single application. In moving to the Cortex Data Lake app, the log forwarding interface now has a new, simplified design that makes it easier to begin configuring Syslog and email profiles to forward your Cortex Data Lake log data.
Log Filter Query Support
November 2020
When creating your log forwarding profiles in Cortex Data Lake, you can now use the same query language from Explore to define precise log filters based on time, device serial number, IP address, and more.
LEEF Format Support for IBM QRadar
November 2020
You can now forward logs in Log Extended Event Format (LEEF) for use with IBM QRadar SIEM.
Combined Log Types
November 2020
To simplify the list of available log types for log forwarding, the
tunnel
log type now includes
GTP
logs, and
Threat
logs now include
WildFire
logs.
Because log forwarding profiles can only include one filter per log type, if you had a log forwarding profile with a log filter for both of the formerly separate log types, you will now see a new log forwarding profile. This profile contains the log filter that could not be duplicated in the original profile.
For example, a log forwarding profile with filters for both
tunnel
and
GTP
logs now appears as two profiles, each with a
tunnel
filter. One of the profiles will continue filtering
tunnel
logs and the other will filter
GTP
logs, which are now included in tunnel logs. The new profile will be called
<
original name
> - GTP
or, in the case of
Threat
and
WildFire
,
<
original name
> - WildFire
.
Non-Editable Log Forwarding Filters
November 2020
Some log filters created in the previous Log Forwarding app can no longer be edited. If you would like to change such filters, you must delete them and create new ones.
Because some fields in the migrated filters are no longer available, you may not be able to recreate an identical filter if you delete it.
Scheduled Reports for Cortex Data Lake
November 2020
(
PAN-OS 10.0.2 or later and Cloud Services plugin 1.8.0 or later
) From Panorama, you can now generate scheduled reports on Cortex Data Lake data.
Japan Regional Support
September 2020
To comply with data privacy regulations that require you to keep data within Japanese regional boundaries, you can now select Japan as a host region when you activate Cortex Data Lake.
Canada Regional Support
July 2020
To comply with data privacy regulations that require you to keep data within Canadian regional boundaries, you can now select Canada as a host region when you activate Cortex Data Lake.
To choose Canada as your host region, select
Canada
at activation. The
Americas
region represents the United States only.
Proxy Support
July 2020
(
PAN-OS 10.0 or later
) You can now configure the firewall to forward logs to Cortex Data Lake through a proxy server. This enables you to send log data to Cortex Data Lake from a network without a default gateway.
UK and Singapore Regional Support
July 2020
For compliance with regulations that require you to keep data within regional boundaries, you can now select the UK or Singapore as a host region when you activate Cortex Data Lake.
Quota Manager Enhancements
June 2020
The quota manager now features a detailed breakdown of firewall log types and a simpler method of allocating remaining storage to help you more easily manage your Cortex Data Lake log storage.
Instead of a single Detailed log type, the quota manager now displays the firewall log types individually. The Infrastructure & Audit log type now appears as System and Config logs.
To allocate all remaining storage to one or more log types, you can now leave the quota percentage of log types blank and the quota manager will automatically assign them the unallocated space.
New Quota Manager UI
April 2020
To help you more easily allocate log storage and visualize the data you're storing in Cortex Data Lake, the Cortex Data Lake app now features a completely redesigned quota manager.
The quota manager now visually displays your total storage capacity as a bar, with color-coded segments representing different log sources so you can instantly identify how much storage a service uses and adjust if necessary.
New Minimum PAN-OS Version for Cortex Data Lake Without Panorama
March 2020
To authenticate using the new G2 certificate chain, firewalls that you want to onboard to Cortex Data Lake without using Panorama must now run PAN-OS 9.0.6 or later.
Cortex Data Lake Without Panorama
July 2019
Until PAN-OS 9.0.3, Panorama was required to onboard firewalls to Cortex Data Lake, and to view logs stored in Cortex Data Lake. Now, firewalls running PAN-OS 9.0.3 and later can securely connect and log to Cortex Data Lake, without Panorama. The new app, Explore, allows you to see and interact with the log data stored in Cortex Data Lake.
New App-ID for Palo Alto Networks Shared Services
May 2019
For better application visibility and control, you now have a new App-ID for paloalto-shared-services, in addition to the App-ID for the palo-alto-logging-service. The paloalto-shared-services App-ID identifies traffic for any shared services that are used by Palo Alto Networks including Directory Sync Service, Logging Service, and Magnifier; any paloalto-shared-services traffic that was earlier identified as ssl, web-browsing will now be identified as paloalto-shared-services.
If you have a Palo Alto Networks next-generation firewall between the Panorama appliance and the internet, you must add a security policy rule on the firewall to allow the paloalto-logging-service and paloalto-shared-services App-IDs from the Panorama appliance to the internet. These applications allow SSL-secured communication to the Cortex Data Lake that the Panorama appliance uses to query logs, and enable communication to the shared services and the Cortex Data Lake for performing certificate status and revocation checks.
Connection Status Reporting Improvements
September 2018
To help with visibility on the status and connectivity to the Cortex Data Lake, the Cloud Services plugin 1.2 provides details on the connection status between Panorama and the Cortex Data Lake. On
Panorama
Cloud Services
Status
Status
, you can now verify that Panorama appliance was able to successfully retrieve the Logging Service certificate, view the Customer Identification number and the region in which your Cortex Data Lake instance is deployed, and confirm that the Panorama appliance is connected to the Logging Service. If any of these checks fail, the Status is reported as an error.
New App-ID for Palo Alto Networks Shared Services
September 2018
For better application visbility and control, you now have a new App-ID for paloalto-shared-services, in addition to the App-ID for the palo-alto-logging-service. The paloalto-shared-services App-ID identifies traffic for any shared services that are used by Palo Alto Networks including Directory Sync Service, Logging Service, and Magnifier; any paloalto-shared-services traffic that was earlier identified as ssl, web-browsing will now be identified as paloalto-shared-services.
If you have a Palo Alto Networks next-generation firewall between the Panorama appliance and the internet, you must add a security policy rule on the firewall to allow the paloalto-logging-service and paloalto-shared-services App-IDs from the Panorama appliance to the internet. These applications allow SSL-secured communication to the Cortex Data Lake that the Panorama appliance uses to query logs, and enable communication to the shared services and the Cortex Data Lake for performing certificate status and revocation checks.
Expand Log Storage Capacity for Traps Logs
April 2018
You can now activate the Cortex Data Lake Auth code from the cloud services portal to upgrade the Traps Included Storage of 100GB to a Cortex Data Lake license with larger storage capacity.
Log Quota Management on the hub
March 2018
Starting March 19, 2018, you must use the cloud services portal to manage the log quota for logs stored on the Cortex Data Lake.
Log in to the cloud services portal using your Customer Support Portal credentials, and then refer to the Logging Service Getting Started Guide for instructions on activating licenses and deploying this service.

Recommended For You