Start Sending Logs to Cortex Data Lake (Individually Managed)
Table of Contents
Expand all | Collapse all
-
- Cortex Data Lake for Panorama-Managed Firewalls
- Start Sending Logs to a New Cortex Data Lake Instance
- Configure Panorama in High Availability for Cortex Data Lake
- Allocate Storage Based on Log Type
- View Cortex Data Lake Status
- View Logs in Cortex Data Lake
- TCP Ports and FQDNs Required for Cortex Data Lake
- Sizing for Cortex Data Lake Storage
-
- Forward Logs from Cortex Data Lake to a Syslog Server
- Forward Logs from Cortex Data Lake to an HTTPS Server
- Forward Logs from Cortex Data Lake to an Email Server
- Log Record Formats
- Create Log Filters
- Server Certificate Validation
- List of Trusted Certificates for Syslog and HTTPS Forwarding
- Log Forwarding Errors
Start Sending Logs to Cortex
Data Lake (Individually Managed)
Cortex
Data Lake
(Individually Managed)Follow these steps to send logs from your firewalls to
Cortex
Data Lake
.Before you start sending logs to Cortex™ Data
Lake, you must:
The following task describes
how to start forwarding logs to
Cortex
Data Lake
from firewalls
that are not managed by Panorama™. You’ll specify the log types
you want to forward and also take steps to make sure that the traffic
between the firewall and Cortex
Data Lake
remains secure.Sending
log data to
Cortex
Data Lake
from other sources, including from Panorama-managed
firewalls, requires a different workflow:Log Source | See... |
---|---|
Panorama-managed firewalls | |
Prisma Access | |
How you activate and implement
Cortex
Data Lake
varies depending on the products and services you’re using. Learn
more about how to get started with
.Cortex
Data Lake
was previously called Logging Service so you might continue to
see references to Logging Service in the firewall web interface.- If you haven’t done so already, Activate and onboard firewalls to .
- Configure NTP so that the firewall stays in sync withCortex Data Lake.On the firewall, selectand set it to the sameDeviceSetupServicesNTPNTP Server Addressyou configured on Panorama. For example:pool.ntp.org.
- (Optional) If you do not want to use the management interface to forward logs toCortex Data Lake, enable the firewall to send traffic through a different interface.Beginning with content release version 8067, you can use thepaloalto-shared-servicesApp-ID™, thepaloalto-logging-serviceApp-ID, and thepanoramaApp-ID. to safely enable traffic between the firewalls andCortex Data Lake. You also must create a Security policy rule that allows this traffic on any firewalls that are between the firewalls sending the logs and the internet. If the upstream firewalls are not Palo Alto Networks firewalls, you must enable access to the TCP Ports and FQDNs Required for Cortex Data Lake.Consider that a Panorama™ appliance or firewall running PAN-OS®9.1 and earlier versions cannot connect toCortex Data Lakefrom behind a proxy (Cortex Data Lakerequires mutual authentication).You can, however, enable proxy communication on PAN-OS 10.0 and later versions:
- Configure a service route for Palo Alto Networks Services.
- Create a Security policy rule that enables the firewalls to communicate withCortex Data Lake.This is required if you are using the Palo Alto Networks Services service route instead of the management interface to forward logs toCortex Data Lake. To create this rule, set theApplicationtopaloalto-shared-services(requires content release version 8066 or a later version) andpaloalto-logging-service, andpanorama(not required after content release version 8290). The paloalto-shared-services app covers the common traffic for different Palo Alto Networks services and is a dependency for the paloalto-logging-service app.Make sure you position this rule before any rule that allows web-browsing and SSL traffic to the internet. If you have a firewall between Panorama and the internet, you must also add a rule that allows paloalto-shared-services and paloalto-logging-service traffic on that firewall. The paloalto-logging-service app enables the firewalls and Panorama to connect toCortex Data Lakeon ports 444 and 3978—the defaults ports for this communication.If that intermediate firewall is not a Palo Alto Networks firewall, then you must create a Security policy rule on that firewall that allows outbound SSL traffic to the internet, which allows the TCP ports and FQDNs required for Cortex Data Lake so that the internet gateway firewall does not block traffic between Panorama andCortex Data Lake.
- Specify the log types to forward toCortex Data Lake.
- To forward System, Configuration, User-ID, and HIP Match logs:
- Select.DeviceLog Settings
- For each log type that you want to forward toCortex Data Lake,Adda match list filter. Give it aName, optionally define aFilter, selectLogging Service, and clickOK.
- To forward log types that are generated when a policy match occurs—Traffic, Threat, WildFire®Submission, URL Filtering, Data Filtering, and Authentication logs—create and attach a Log Forwarding profile to each policy rule for which you want to forward logs.
- SelecttoObjectsLog ForwardingAdda profile. In the log forwarding profile match list, add each log type that you want to forward.If you enabled the Enhanced Application Logs feature, then fullyEnable enhanced application logging to Cortex Data Lakeon the firewall to forward these log types. When you enable this feature, the match lists that specify the log types required for enhanced application logging are automatically added to the profile.
- SelectLogging Serviceas the Forward Method to enable the firewalls in the device group to forward the logs toCortex Data Lake. You can monitor the logs and generate reports from Panorama.
- If you haven’t already done so, create basic Security policy rules.Until the firewall has interfaces and zones and a basic Security policy, it will not let any traffic through and, by default, only traffic that matches a Security policy rule will be logged.
- For each rule you create, selectActionsand select the Log Forwarding profile that allows the firewall to send logs toCortex Data Lake.
- (PA-7000 Series firewalls only) Configure a log card interface to perform log forwarding.As of PAN-OS 10.1, you can no longer forward system logs using the Management interface or using service routes through the Data Plane interfaces. The only way to forward system logs from a PA-7000 Series firewall running PAN-OS 10.1 or later is by configuring a Log Forwarding Card (LFC).
- Selectand clickNetworkInterfacesEthernetAdd Interface.
- Select theSlotandInterface Name.
- Set theInterface TypetoLog Card.
- Enter theIP Address,Default Gateway, and (for IPv4 only)Netmask.
- SelectAdvancedand specify theLink Speed,Link Duplex, andLink State.These fields default toauto, which specifies that the firewall automatically determines the values based on the connection. However, the minimum recommendedLink Speedfor any connection is1000(Mbps).
- ClickOKto save your changes.
- Commityour changes.
- Verify that the firewall logs are forwarded toCortex Data Lake.
- On a firewall, enter the CLI commandshow logging-status:
Look for the----------------------------------------------------------------------------------------------------------------------------- Type Last Log Created Last Log Fwded Last Seq Num Fwded Last Seq Num Acked Total Logs Fwded ----------------------------------------------------------------------------------------------------------------------------- > CMS 0 Not Sending to CMS 0 > CMS 1 Not Sending to CMS 1 >Log Collection Service 'Log Collection log forwarding agent' is active and connected to xx.xxx.xxx.xx config 2017/07/26 16:33:20 2017/07/26 16:34:09 323 321 2 system 2017/07/31 12:23:10 2017/07/31 12:23:18 13634645 13634637 84831 threat 2014/12/01 14:47:52 2017/07/26 16:34:24 557404252 557404169 93 traffic 2017/07/28 18:03:39 2017/07/28 18:03:50 3619306590 3619306590 1740 hipmatch Not Available Not Available 0 0 0 gtp-tunnel Not Available Not Available 0 0 0 userid Not Available Not Available 0 0 0 auth Not Available Not Available 0 0 0‘Log collection log forwarding agent’ is active and connected to <IP_address>line. You can also see that CMS 0 and CMS (the Log Collectors) are not receiving logs.Show Status() to verify that the firewall is connected and sending logs toDeviceSetupManagementCortex Data LakeCortex Data Lake.
- Next steps:
- Use Explore to search, filter, and export log data. This app offers you critical visibility into the network activity in your enterprise by enabling you to easily examine network and endpoint log data.
- ArchiveCortex Data Lakelogs by forwarding logs from to a Syslog server or email server for long-term storage, SOC, or internal audit.