Start Sending Logs to Cortex Data Lake

Before you can start sending logs to Cortex Data Lake, you must:
  • Activate your Cortex Data Lake instance and
  • connect the firewall to Cortex Data Lake.
The following steps describe how to start forwarding logs to Cortex Data Lake, from firewalls that are not managed by Panorama. You’ll specify the log types you want to forward, and also take steps to make sure that the traffic between the firewall and Cortex Data Lake remains secure.
If you’re using PAN-OS 9.0.2 or earlier release versions, Panorama-managed firewalls, or Traps
:
  • PAN-OS 9.0.2 or earlier
    —Direct firewall onboarding to Cortex Data Lake is supported for firewalls running PAN-OS 9.0.3 and later. If your firewall is running PAN-OS 9.0.2 or an earlier release version, you either need Panorama or you need to upgrade to PAN-OS 9.0.3 to support a Cortex Data Lake deployment.
  • Panorama
    —Follow this workflow instead to enable Panorama-managed firewalls to start forwarding logs to Cortex Data Lake.
  • Traps
    —To enable Traps to log to the Cortex Data Lake, see the Traps Administrator’s Guide.
How you activate and implement Cortex Data Lake varies depending on the products and services you’re using. Learn more about what you should do to get started with Cortex Data Lake based on the product you’re using.
Cortex Data Lake was previously called the Logging Service; you might continue to see references to the Logging Service in the firewall web interface.
  1. Configure NTP so that the firewall can stay in sync with Cortex Data Lake.
    On the firewall, select
    Device
    Setup
    Services
    NTP
    and set it to the same
    NTP Server Address
    you configured on Panorama, for example
    pool.ntp.org
    .
  2. (Optional)
    If you do not want to use the management interface to forward logs to the Cortex Data Lake, enable the firewall to send traffic through a different interface.
    Beginning with content release version 8067, you can use the paloalto-shared-services and paloalto-logging-service App-IDs to safely enable traffic between the firewalls and the Cortex Data Lake. You will also need to create a security policy rule to allow this traffic on any firewalls between the firewalls sending the logs and the internet. If the upstream firewalls are not Palo Alto Networks firewalls, you must enable access to the TCP Ports and FQDNs Required for Cortex Data Lake. Keep in mind that the firewalls and the Cortex Data Lake use mutual certificate authentication, so they cannot be decrypted and you cannot connect through a proxy server.
    1. Configure a service route for Palo Alto Networks Services.
      palo-alto-networks-services-service-route.png
    2. Create a security policy rule that enables the firewalls to communicate with the Cortex Data Lake.
      This is required if you are using the Palo Alto Networks Services service route instead of the management interface to forward logs to the Cortex Data Lake. To create this rule, set the
      Application
      to
      paloalto-shared-services
      (requires content release version 8066 or later) and
      paloalto-logging-service
      (requires content release version 8033 or later). The paloalto-shared-services covers the common traffic for different Palo Alto Networks services and is a dependency for the paloalto-logging-service.
      logging-service-app-id.png
      Make sure you place this rule above any rule that allows web-browsing and SSL traffic to the internet. In addition, if you have a firewall between Panorama and the internet, you must also add a rule that allows paloalto-shared-services and paloalto-logging service traffic on that firewall. The paloalto-logging-service app enables the firewalls and Panorama to connect to the Cortex Data Lake on ports 444 and 3978, the defaults ports for this communication.
      If that firewall is not a Palo Alto Networks firewall, create a security policy rule on that firewall that allows outbound SSL traffic to the internet to allow the TCP Ports and FQDNs Required for Cortex Data Lake so that the internet gateway firewall does not block traffic between Panorama and the Cortex Data Lake.
      The firewalls and Panorama need access to the domain 8.0.0 on port 3978 in order to forward logs to the Cortex Data Lake. This is true even if you are using the paloalto-logging-service App-ID to safely enable Cortex Data Lake traffic.
  3. Specify the log types to forward to Cortex Data Lake.
    1. To forward System, Configuration, User-ID, and HIP Match logs:
      1. Select
        Device
        Log Settings
        .
      2. For each log type that you want to forward to the Cortex Data Lake,
        Add
        a match list filter. Give it a
        Name
        , optionally define a
        Filter
        , select the
        Logging Service
        check box, and click
        OK
        .
    2. To forward log types that are generated when a policy match occurs—Traffic, Threat, WildFire Submission, URL Filtering, Data Filtering, and Authentication logs—create and attach a Log Forwarding profile to each policy rule for which you want to forward logs.
      1. Select
        Objects
        Log Forwarding
        to
        Add
        a profile. In the log forwarding profile match list, add each log type that you want to forward.
        If you have already turned on Enhanced Application Logs, fully enable the firewall to forward these log types by selecting
        Enable enhanced application logging to Cortex Data Lake
        . Notice that when you select this option, match lists that specify the logs types required for enhanced application logging are automatically added to the profile.
      2. Select
        Logging Service
        as the Forward Method to enable the firewalls in the device group to forward the logs to the Cortex Data Lake. You will be able to monitor the logs and generate reports from Panorama.
      3. If you haven’t already done so, Create basic security policy rules now.
        Until the firewall has interfaces and zones and a basic security policy, it will not let any traffic through, and only traffic that matches a security policy rule will be logged (by default).
      4. For each rule you create, select
        Actions
        and select the Log Forwarding profile that allows the firewall to send logs to the Cortex Data Lake.
        logging-service-lf-profile-to-policy.png
  4. Commit
    your changes.
  5. Verify that the firewall logs are being forwarded to the Cortex Data Lake.
    • Log in to Explore, available in the Cortex hub, to view and filter Cortex Data Lake logs.
    • On a firewall, enter the CLI command
      show logging-status
      :
      ----------------------------------------------------------------------------------------------------------------------------- Type Last Log Created Last Log Fwded Last Seq Num Fwded Last Seq Num Acked Total Logs Fwded ----------------------------------------------------------------------------------------------------------------------------- > CMS 0 Not Sending to CMS 0 > CMS 1 Not Sending to CMS 1 >Log Collection Service 'Log Collection log forwarding agent' is active and connected to xx.xxx.xxx.xx config 2017/07/26 16:33:20 2017/07/26 16:34:09 323 321 2 system 2017/07/31 12:23:10 2017/07/31 12:23:18 13634645 13634637 84831 threat 2014/12/01 14:47:52 2017/07/26 16:34:24 557404252 557404169 93 traffic 2017/07/28 18:03:39 2017/07/28 18:03:50 3619306590 3619306590 1740 hipmatch Not Available Not Available 0 0 0 gtp-tunnel Not Available Not Available 0 0 0 userid Not Available Not Available 0 0 0 auth Not Available Not Available 0 0 0
      Look for the
      ‘Log collection log forwardingagent’ is active and connected to <IP_address>
      line. You can also see that CMS 0 and CMS (the Log Collectors) are not receiving logs.
      Select the
      Show Status
      link on
      Device
      Setup
      Management
      Cortex Data Lake
      to verify that the firewall is connected and sending logs to the Cortex Data Lake.
  6. Next steps:
    • Use Explore to search, filter, and export log data. This app offers you critical visibility into your enterprise's network activities by allowing you to easily examine network and endpoint log data.
    • Use the Log Forwarding app to forward logs stored in Cortex Data Lake to a Syslog or email destination.
      If you want to be able to archive the logs you send to the Cortex Data Lake for long-term storage, SOC, or internal audit directly from the Cortex Data Lake, you can use the Log Forwarding app, which is included with your Cortex Data Lake (formerly called Logging Service) license. This app enables log forwarding from the Cortex Data Lake to an external destination such as a Syslog server or an email server. Refer to the Log Forwarding App Getting Started Guide for more information. Alternatively, you continue to forward logs directly from the firewalls to your Syslog receiver.

Related Documentation