Start Sending Logs to Cortex Data Lake (Individually Managed)

If you haven’t done so already, Activate Cortex Data Lake and onboard firewalls to Cortex Data Lake.
Configure NTP so that the firewall stays in sync with Cortex Data Lake.
On the firewall, select
Device
Setup
Services
NTP
and set it to the same
NTP Server Address
you configured on Panorama. For example:
pool.ntp.org
.
(
Optional
) If you do not want to use the management interface to forward logs to Cortex Data Lake, enable the firewall to send traffic through a different interface.
Beginning with content release version 8067, you can use the
paloalto-shared-services
App-ID™, the
paloalto-logging-service
App-ID, and the
panorama
App-ID. to safely enable traffic between the firewalls and Cortex Data Lake. You also must create a Security policy rule that allows this traffic on any firewalls that are between the firewalls sending the logs and the internet. If the upstream firewalls are not Palo Alto Networks firewalls, you must enable access to the TCP Ports and FQDNs Required for Cortex Data Lake.
Consider that a Panorama™ appliance or firewall running PAN-OS
®
9.1 and earlier versions cannot connect to Cortex Data Lake from behind a proxy (Cortex Data Lake requires mutual authentication).
You can, however, enable proxy communication on PAN-OS 10.0 and later versions:
Configure a service route for Palo Alto Networks Services.

Create a Security policy rule that enables the firewalls to communicate with Cortex Data Lake.
This is required if you are using the Palo Alto Networks Services service route instead of the management interface to forward logs to Cortex Data Lake. To create this rule, set the
Application
to
paloalto-shared-services
(requires content release version 8066 or a later version) and
paloalto-logging-service
, and
panorama
(not required after content release version 8290). The paloalto-shared-services app covers the common traffic for different Palo Alto Networks services and is a dependency for the paloalto-logging-service app.

Make sure you position this rule before any rule that allows web-browsing and SSL traffic to the internet. If you have a firewall between Panorama and the internet, you must also add a rule that allows paloalto-shared-services and paloalto-logging-service traffic on that firewall. The paloalto-logging-service app enables the firewalls and Panorama to connect to Cortex Data Lake on ports 444 and 3978—the defaults ports for this communication.
If that intermediate firewall is not a Palo Alto Networks firewall, then you must create a Security policy rule on that firewall that allows outbound SSL traffic to the internet, which allows the TCP ports and FQDNs required for Cortex Data Lake so that the internet gateway firewall does not block traffic between Panorama and Cortex Data Lake.
Specify the log types to forward to Cortex Data Lake.
To forward System, Configuration, User-ID, and HIP Match logs:

Select
Device
Log Settings
.
For each log type that you want to forward to Cortex Data Lake,
Add
a match list filter. Give it a
Name
, optionally define a
Filter
, select
Logging Service
, and click
OK
.

To forward log types that are generated when a policy match occurs—Traffic, Threat, WildFire

®

Submission, URL Filtering, Data Filtering, and Authentication logs—create and attach a Log Forwarding profile to each policy rule for which you want to forward logs.

Select

Objects

Log Forwarding

to

Add

a profile. In the log forwarding profile match list, add each log type that you want to forward.

If you enabled the Enhanced Application Logs feature, then fully

Enable enhanced application logging to Cortex Data Lake

on the firewall to forward these log types. When you enable this feature, the match lists that specify the log types required for enhanced application logging are automatically added to the profile.

Select

Logging Service

as the Forward Method to enable the firewalls in the device group to forward the logs to Cortex Data Lake. You can monitor the logs and generate reports from Panorama.

If you haven’t already done so, create basic Security policy rules.

Until the firewall has interfaces and zones and a basic Security policy, it will not let any traffic through and, by default, only traffic that matches a Security policy rule will be logged.

For each rule you create, select

Actions

and select the Log Forwarding profile that allows the firewall to send logs to Cortex Data Lake.


(
PA-7000 Series firewalls only
) Configure a log card interface to perform log forwarding.
As of PAN-OS 10.1, you can no longer forward system logs using the Management interface or using service routes through the Data Plane interfaces. The only way to forward system logs from a PA-7000 Series firewall running PAN-OS 10.1 or later is by configuring a Log Forwarding Card (LFC).
Select
Network
Interfaces
Ethernet
and click
Add Interface
.
Select the
Slot
and
Interface Name
.
Set the
Interface Type
to
Log Card
.
Enter the
IP Address
,
Default Gateway
, and (
for IPv4 only
)
Netmask
.
Select
Advanced
and specify the
Link Speed
,
Link Duplex
, and
Link State
.
These fields default to
auto
, which specifies that the firewall automatically determines the values based on the connection. However, the minimum recommended
Link Speed
for any connection is
1000
(Mbps).
Click
OK
to save your changes.
Commit
your changes.
Verify that the firewall logs are forwarded to Cortex Data Lake.
Log in to Explore, available in the hub, so that you can view and filter Cortex Data Lake logs.
On a firewall, enter the CLI command
show logging-status
:
-----------------------------------------------------------------------------------------------------------------------------



      Type      Last Log Created        Last Log Fwded       Last Seq Num Fwded  Last Seq Num Acked         Total Logs Fwded



-----------------------------------------------------------------------------------------------------------------------------



> CMS 0



        Not Sending to CMS 0



> CMS 1



        Not Sending to CMS 1







>Log Collection Service



'Log Collection log forwarding agent' is active and connected to xx.xxx.xxx.xx







    config   2017/07/26 16:33:20   2017/07/26 16:34:09                      323                 321                        2



    system   2017/07/31 12:23:10   2017/07/31 12:23:18                 13634645            13634637                    84831



    threat   2014/12/01 14:47:52   2017/07/26 16:34:24                557404252           557404169                       93



   traffic   2017/07/28 18:03:39   2017/07/28 18:03:50               3619306590          3619306590                     1740



  hipmatch         Not Available         Not Available                        0                   0                        0



gtp-tunnel         Not Available         Not Available                        0                   0                        0



    userid         Not Available         Not Available                        0                   0                        0



      auth         Not Available         Not Available                        0                   0                        0
Look for the
‘Log collection log forwarding agent’ is active and connected to <IP_address>
line. You can also see that CMS 0 and CMS (the Log Collectors) are not receiving logs.
Show Status
(
Device
Setup
Management
Cortex Data Lake
) to verify that the firewall is connected and sending logs to Cortex Data Lake.
Next steps:
Use Explore to search, filter, and export log data. This app offers you critical visibility into the network activity in your enterprise by enabling you to easily examine network and endpoint log data.
Archive Cortex Data Lake logs by forwarding logs from Cortex Data Lake to a Syslog server or email server for long-term storage, SOC, or internal audit.