Configure the Firewalls to Forward Logs to the Cortex Data Lake

Use these steps to learn how to begin forwarding logs from your hardware-based and VM-Series firewalls to the Cortex Data Lake.
Panorama does not forward any logs to the Cortex Data Lake (formerly called the Logging Service). The firewalls must connect directly to the Cortex Data Lake to forward logs.
To configure your hardware-based and VM-Series firewalls to forward logs to the Cortex Data Lake, you must add the firewall as a managed device on Panorama. The firewalls must be managed by Panorama so that Panorama can provision the certificate that the firewall needs to securely connect to the Cortex Data Lake. The firewalls and Panorama must be running PAN-OS 8.0.6 or later to use the Cortex Data Lake.
You can then enable the Cortex Data Lake on the firewall and configure the log forwarding settings to forward the specific log types you want to send to the Cortex Data Lake. You can configure these settings directly on the firewall, or configure device groups and templates with the settings and push them to the firewalls. In addition, you must enable the firewalls to communicate with the Cortex Data Lake. If you have configured a service route for the Palo Alto Networks Cloud Services, you must create a security policy rule to enable traffic between the firewalls and the Cortex Data Lake. Beginning with content release version 8067, you can use the paloalto-shared-services and paloalto-logging-service App-IDs to safely enable traffic between the firewalls and the Cortex Data Lake. You will also need to create a security policy rule to allow this traffic on any firewalls between the firewalls sending the logs and the internet. If the upstream firewalls are not Palo Alto Networks firewalls, you must enable access to the TCP Ports and FQDNs Required for Cortex Data Lake. Keep in mind that the firewalls and the Cortex Data Lake use mutual certificate authentication and therefore cannot be decrypted and you cannot connect through a proxy server.
If you want to be able to archive the logs you send to the Cortex Data Lake for long-term storage, SOC, or internal audit directly from the Cortex Data Lake, you can use the Log Forwarding app, which is included with your Cortex Data Lake (formerly called Logging Service) license. This app enables log forwarding from the Cortex Data Lake to an external destination such as a Syslog server or an email server. Refer to the Log Forwarding App Getting Started Guide for more information. Alternatively, you continue to forward logs directly from the firewalls to your Syslog receiver.
Use the following workflow to configure your hardware-based and VM-Series firewalls to log to the Cortex Data Lake. For instructions on how to enable the firewalls deployed within your GlobalProtect cloud service infrastructure to log to the Cortex Data Lake, refer to the GlobalProtect Cloud Service Getting Started Guide. For instructions on how to enable Traps to log to the Cortex Data Lake, refer to the Traps Administrator’s Guide.
  1. Add the firewall as a managed device on Panorama. Before you add the firewall as a managed device, you must configure NTP so that the firewall can stay in sync with the Cortex Data Lake.
    On the firewall, select DeviceSetupServicesNTP and set it to the same NTP Server Address you configured on Panorama, for example pool.ntp.org.
  2. Retrieve and push the Cortex Data Lake licenses for the managed firewalls.
    1. From Panorama, select PanoramaDevice DeploymentLicense.
    2. Click Refresh and then select the firewalls from the list. Panorama retrieves the licenses, deploys them to the selected firewalls, and updates the licensing status on the Panorama web interface.
      Make sure you see that Panorama successfully installed the Cortex Data Lake license on the firewall.
      Do not click Refresh again until the first refresh completes. When the refresh completes, you will see Status shows Completed and Progress is 100%, along with some Details about whether the refresh succeeded.
      license-refresh-panorama-fw.png
  3. From Panorama, create a template and a device group to push log forwarding settings to the firewalls that you want to log to the Cortex Data Lake.
  4. Enable the firewalls in the template to send logs to the Cortex Data Lake and select the region where you want the logs stored.
    If some firewalls in your deployment are sending logs to dedicated Log Collectors or to Panorama with a local Log Collector, only firewalls that belong to the template with the Enable Cortex Data Lake option selected can send logs to the Cortex Data Lake.
    1. Select DeviceSetupManagement.
    2. Select the Template that contains the firewalls you want to log to the Cortex Data Lake.
    3. Edit the Cortex Data Lake settings.
      logging-service-settings.png
    4. Select the Enable Cortex Data Lake check box.
      For firewalls running PAN-OS 8.1.x, you can opt to send logs to both the Cortex Data Lake and to your Panorama and on premise log collection setup when you select Enable Duplicate Logging (Cloud and On-Premise). When enabled, the firewalls that belong to the selected Template will save a copy of the logs to both locations.
      You can also Enable Enhanced Application Logging to allow the firewall to collect data for apps running the Palo Alto Networks Cloud Services environment. These logs provide Palo Alto Networks Cloud services apps increased visibility into network activity, and in some cases, are required to support app features.
    5. Select the Region where you want to forward logs for the firewalls associated with this template and then click OK.
    6. (Panorama 9.0 only) Specify the Connection count to Cortex Data Lake for PA-7000s and PA-5200s.
      Specify the number of connections that are established between the firewalls and the Cortex Data Lake (range is 1 to 20; default is 5) to forward logs to the Cortex Data Lake.
    7. Configure interfaces and zones in the template.
  5. (Optional) If you do not want to use the management interface to forward logs to the Cortex Data Lake, enable the firewall to send traffic through a different interface and create the policy to allow the traffic.
    1. Configure a service route for Palo Alto Networks Services.
      palo-alto-networks-services-service-route.png
    2. Create a security policy rule that enables the firewalls to communicate with the Cortex Data Lake.
      This is required if you are using the Palo Alto Networks Services service route instead of the management interface to forward logs to the Cortex Data Lake. To create this rule, set the Application to paloalto-shared-services (requires content release version 8066 or later) and paloalto-logging-service (requires content release version 8033 or later). The paloalto-shared-services covers the common traffic for different Palo Alto Networks services and is a dependency for the paloalto-logging-service.
      logging-service-app-id.png
      Make sure you place this rule above any rule that allows the web-browsing and SSL traffic to the internet. In addition, if you have a firewall between Panorama and the internet, you must also add a rule that allows paloalto-shared-services and paloalto-logging service traffic on that firewall. The paloalto-logging-service app enables the firewalls and Panorama to connect to the Cortex Data Lake on ports 444 and 3978, the defaults ports for this communication.
      If that firewall is not a Palo Alto Networks firewall, create a security policy rule on that firewall that allows outbound SSL traffic to the internet to allow the TCP Ports and FQDNs Required for Cortex Data Lakeso that the internet gateway firewall does not block traffic between Panorama and the Cortex Data Lake.
      The firewalls and Panorama need access to the domain 8.0.0 on port 3978 in order to forward logs to the Cortex Data Lake. This is true even if you are using the paloalto-logging-service App-ID to safely enable Cortex Data Lake traffic.
  6. Specify the log types to forward to the Cortex Data Lake.
    The way you enable forwarding depends on the log type. For logs that are generated based on a policy match, use a log forwarding profile within a device group. For other log types, use the Log Settings configuration within a template.
    1. To configure forwarding of System, Configuration, User-ID, and HIP Match logs:
      1. Select DeviceLog Settings.
      2. Select the Template that contains the firewalls you want to forward logs to the Cortex Data Lake.
      3. For each log type that you to forward to the Cortex Data Lake, Add a match list filter. Give it a Name, optionally define a Filter, select the Panorama/Logging Service check box, and click OK.
        logging-service-forward-system-config-logs.png
    2. To configure forwarding of all other log types that are generated when a policy match occurs ,such as Traffic, Threat, WildFire Submission, URL Filtering, Data Filtering, and Authentication logs, create and attach a Log Forwarding profile to each policy rule for which you want to forward logs.
      1. Select the Device Group, and then select ObjectsLog Forwarding to Add a profile. In the log forwarding profile match list, add each log type that you want to forward.
        If you have already turned on Enhanced Application Logs, fully enable the firewall to forward these log types by selecting Enable enhanced application logging to Cortex Data Lake. Notice that when you select this option, match lists that specify the logs types required for enhanced application logging are automatically added to the profile.
      2. Select Panorama/Cortex Data Lake as the Forward Method to enable the firewalls in the device group to forward the logs to the Cortex Data Lake. You will be able to monitor the logs and generate reports from Panorama.
        logging-service-log-forwarding-profile.png
      3. Until the firewall has interfaces and zones and a basic security policy, it will not let any traffic through, and only traffic that matches a security policy rule will be logged (by default).
      4. For each rule you create, select Actions and select the Log Forwarding profile that allows the firewall to send logs to the Cortex Data Lake.
        logging-service-lf-profile-to-policy.png
  7. Commit your changes to Panorama and push them to the template and device group you created.
  8. Verify that the firewall logs are being forwarded to the Cortex Data Lake.
    • On Panorama 8.1.7 or later, select MonitorLogs and find the column From Logging Service to identify whether the logs that you view on Panorama are stored on the Cortex Data Lake. The value yes indicates that the logs are saved on the Cortex Data Lake.
      logging-service-90-column.png
      Use the CLI command request logging-service-forwarding status for detailed information on the connectivity status to the Cortex Data Lake and to verify whether you have enabled duplicate log forwarding and Enhanced Application Logs.
    • On a firewall (PAN-OS 8.0.6 or later), enter the CLI command show logging-status:
      -----------------------------------------------------------------------------------------------------------------------------
      
      
      
            Type      Last Log Created        Last Log Fwded       Last Seq Num Fwded  Last Seq Num Acked         Total Logs Fwded
      
      
      
      -----------------------------------------------------------------------------------------------------------------------------
      
      
      
      > CMS 0
      
      
      
              Not Sending to CMS 0
      
      
      
      > CMS 1
      
      
      
              Not Sending to CMS 1
      
      
      
      
      
      
      
      >Log Collection Service
      
      
      
      'Log Collection log forwarding agent' is active and connected to xx.xxx.xxx.xx
      
      
      
      
      
      
      
          config   2017/07/26 16:33:20   2017/07/26 16:34:09                      323                 321                        2
      
      
      
          system   2017/07/31 12:23:10   2017/07/31 12:23:18                 13634645            13634637                    84831
      
      
      
          threat   2014/12/01 14:47:52   2017/07/26 16:34:24                557404252           557404169                       93
      
      
      
         traffic   2017/07/28 18:03:39   2017/07/28 18:03:50               3619306590          3619306590                     1740
      
      
      
        hipmatch         Not Available         Not Available                        0                   0                        0
      
      
      
      gtp-tunnel         Not Available         Not Available                        0                   0                        0
      
      
      
          userid         Not Available         Not Available                        0                   0                        0
      
      
      
            auth         Not Available         Not Available                        0                   0                        0
      
      
      
      
      Look for the ‘Log collection log forwardingagent’ is active and connected to <IP_address> line. You can also see that CMS 0 and CMS (the Log Collectors) are not receiving logs.
      On firewall running PAN-OS 8.1.7 or later, select the Show Status link on DeviceSetupManagementCortex Data Lake to verify that the firewall is connected and sending logs to the Cortex Data Lake.
  9. Use the ACC on Panorama to monitor network activity.
    You can also use MonitorManage Custom Reports and generate Run Now reports on summary logs. You cannot generate scheduled reports or generate reports on detailed logs stored on the Cortex Data Lake.

Related Documentation