Forward Logs to Cortex Data Lake (Panorama-Managed)

Learn how to forward logs to Cortex Data Lake from your Panorama-managed firewalls.
To send logs from Panorama™-managed firewalls to Cortex™ Data Lake, you must:Activating Cortex Data Lake includes provisioning the certificate that the firewalls need to securely connect to Cortex Data Lake. Only after you activate Cortex Data Lake can you enable Panorama-managed firewalls to forward logs.
The following task describes how to start forwarding logs. First, you’ll enable firewalls to communicate with Cortex Data Lake and then you can specify the log types that you want to send. You can then use Panorama device groups and templates to push these settings to managed firewalls.
If you’re using:
  • Firewalls without Panorama
    —To forward logs to Cortex Data Lake from firewalls that are not managed by Panorama, follow these steps, instead.
  • Cortex XDR
    —To enable Cortex XDR to send logs to Pro or Prevent.
How you activate and implement Cortex Data Lake varies depending on the products and services you’re using. Learn more about how to get started with Cortex Data Lake based on the products you’re using.
  1. Add the firewall as a managed device on Panorama. Before you add the firewall as a managed device, you must configure NTP so that the firewall stays in sync with Cortex Data Lake.
    On the firewall, select
    Device
    Setup
    Services
    NTP
    and set it to the same
    NTP Server Address
    you configured on Panorama. For example:
    pool.ntp.org
    .
    1. (
      Optional, Panorama 10.0 or and later releases
      ) To configure Panorama to connect to Cortex Data Lake through a proxy server, select
      Panorama
      Setup
      Services
      edit-cog.png
      and
      Use proxy to send logs to Cortex Data Lake
      .
  2. Retrieve and push the Cortex Data Lake licenses for managed firewalls.
    1. From Panorama, select
      Panorama
      Device Deployment
      License
      .
    2. First
      Refresh
      and then select the firewalls from the list. Panorama retrieves the licenses, deploys them to the selected firewalls, and updates the licensing status on the Panorama web interface.
      Make sure you see that Panorama successfully installed the Cortex Data Lake license on the firewall.
      Do not
      Refresh
      again until the first refresh completes. When the refresh completes, you will see that Status shows Completed and Progress is 100%. There are also Details about whether the refresh succeeded.
      license-refresh-panorama-fw.png
  3. From Panorama, create a template and a device group to push log forwarding settings to the firewalls from which you want to forward logs to Cortex Data Lake.
  4. Enable the firewalls in the template to send logs to Cortex Data Lake and select the region where you want the logs stored.
    If some firewalls in your deployment are sending logs to dedicated Log Collectors or to Panorama with a local Log Collector, only firewalls that belong to the template with the Enable Cortex Data Lake option selected can send logs to Cortex Data Lake.
    1. Select
      Device
      Setup
      Management
      .
    2. Select the
      Template
      that contains the firewalls from which you want to forward logs to Cortex Data Lake.
    3. Edit the Cortex Data Lake settings.
      logging-service-settings.png
    4. Enable either of the following two options:
      • Enable Logging Service
        —Send and save logs to Cortex Data Lake only. With this option, you can use Explore to see and interact with your log data.
      • Enable Duplicate Logging
        —For firewalls running PAN-OS 8.1 and later releases, you can send and save logs both to Cortex Data Lake and to your Panorama and log collection setup. Firewalls save a copy of all log data to both Panorama and Cortex Data Lake except for system and config logs, which are sent to Panorama only.
      To forward logs to Cortex Data Lake with Duplicate Logging enabled, you must add the firewalls with the option enabled to a Collector Group.
    5. Enable Enhanced Application Logging to allow the firewall to collect data for apps running the Palo Alto Networks Cloud Services environment. These logs provide Palo Alto Networks Cloud services apps increased visibility into network activity and, in some cases, are required to support app features.
    6. Select the
      Region
      where you want to forward logs for the firewalls associated with this template and then click
      OK
      .
      Starting with PAN-OS 9.0.2, there is an option to
      Onboard Without Panorama
      . This setting is used only for firewalls that are not managed by Panorama; there’s no need to populate it when you’re enabling Panorama-managed firewalls to forward logs to Cortex Data Lake.
    7. (
      Panorama 9.0 or later releases only
      ) Specify the
      Connection count to Cortex Data Lake for PA-7000s and PA-5200s
      .
      Specify the number of connections that are established between the firewalls and Cortex Data Lake for forwarding logs to Cortex Data Lake (range is 1 to 20; default is 5).
  5. Set the
    Palo Alto Networks Services
    service route to use either the management interface or a data interface.
    • Follow these steps to use the management interface for activation. Otherwise, skip to configuring a data interface.
      1. Select
        Device
        Setup
        Services
        Global
        .
        Global
        on a firewall without multiple virtual system (multi-vsys) capability.
      2. Under Services Features, click
        Service Route Configuration
        .
      3. Select
        Customize
        .
      4. Under Service, click
        Palo Alto Networks Services
        .
      5. For
        Source Interface
        , select
        MGT
        .
      6. Click
        OK
        to exit the Service Route Source dialog and click
        OK
        again to exit Service Route Configuration.
    After activation, you can configure a different interface to forward logs to Cortex Data Lake (see how to start sending logs to Cortex Data Lake).
    • If you chose not to use the management interface for activation, use a data interface by configuring destination service routes for the following FQDNs:
      • api.paloaltonetworks.com
      • apitrusted.paloaltonetworks.com
      • lic.lc.prod.us.cs.paloaltonetworks.com
      1. Select
        Device
        Setup
        Services
        Global
        .
        Global
        on a firewall without multiple virtual system (multi-vsys) capability.
      2. Under Services Features, click
        Service Route Configuration
        .
      3. Select
        Customize
        .
      4. Under Service, select the following:
        • Palo Alto Networks Services
        • CRL status
        • DNS
        • HTTP
        • NTP
      5. Set Selected Service Routes
        .
      6. Select the
        Source Interface
        you want to use for activation and then select a
        Source Address
        from that interface.
        key-set-routes.png
      7. Click
        OK
        .
      8. Select
        Destination
        .
      9. Add
        a destination.
      10. Enter any of the FQDNs above as
        Destination
        .
        key-destination-fqdn.png
      11. Select the same
        Source Interface
        and
        Source Address
        that you selected for activation.
      12. Click
        OK
        .
      13. Add
        two more destinations for the same interface using the remaining two FQDNs.
        key-destinations.png
      14. Click
        OK
        again to exit Service Route Configuration.
  6. Specify the log types to forward to Cortex Data Lake.
    The way you enable forwarding depends on the log type. For logs that are generated based on a policy match, use a log forwarding profile within a device group. For other log types, use the Log Settings configuration within a template.
    1. To configure forwarding of System, Configuration, User-ID, and HIP Match logs:
      1. Select
        Device
        Log Settings
        .
      2. Select the
        Template
        that contains the firewalls from which you want to forward logs to Cortex Data Lake.
      3. For each log type that you want to forward to Cortex Data Lake,
        Add
        a match list filter. Give it a
        Name
        , optionally define a
        Filter
        , select
        Panorama/Logging Service
        , and click
        OK
        .
        logging-service-forward-system-config-logs.png
    2. To configure forwarding of all other log types that are generated when a policy match occurs, such as Traffic or Threat logs, create and attach a Log Forwarding profile to each policy rule for which you want to forward logs.
      1. Select the
        Device Group
        and then select
        Objects
        Log Forwarding
        to
        Add
        a profile. In the log forwarding profile match list, add each log type that you want to forward.
        If you enabled the Enhanced Application Logs feature, then fully
        Enable enhanced application logging to Cortex Data Lake
        on the firewall to forward these log types. When you select this option, match lists that specify the log types required for enhanced application logging are automatically added to the profile.
      2. Select
        Panorama/Cortex Data Lake
        as the Forward Method to enable the firewalls in the device group to forward logs so you can monitor the logs and generate reports from Panorama.
        logging-service-log-forwarding-profile.png
      3. Until the firewall has interfaces and zones and a basic Security policy, it will not let any traffic through and, by default, will log only traffic that matches a Security policy rule.
      4. For each rule you create, select
        Actions
        and select the Log Forwarding profile that allows the firewall to send logs to Cortex Data Lake.
        logging-service-lf-profile-to-policy.png
  7. Commit your changes to Panorama and push them to the template and device group you created.
  8. Verify that the firewall logs are forwarded to Cortex Data Lake.
    • On Panorama 8.1.7 and later releases, select
      Monitor
      Logs
      and review the From Logging Service column to identify whether the logs that you view on Panorama are stored on Cortex Data Lake—
      yes
      indicates that the logs are saved to Cortex Data Lake.
      logging-service-90-column.png
      Use the CLI command
      request logging-service-forwarding status
      for detailed information on the connectivity status to Cortex Data Lake and to verify whether you enabled Duplicate Log Forwarding or Enhanced Application Logs.
    • On a firewall, enter the CLI command
      show logging-status
      :
      ----------------------------------------------------------------------------------------------------------------------------- Type Last Log Created Last Log Fwded Last Seq Num Fwded Last Seq Num Acked Total Logs Fwded ----------------------------------------------------------------------------------------------------------------------------- > CMS 0 Not Sending to CMS 0 > CMS 1 Not Sending to CMS 1 >Log Collection Service 'Log Collection log forwarding agent' is active and connected to xx.xxx.xxx.xx config 2017/07/26 16:33:20 2017/07/26 16:34:09 323 321 2 system 2017/07/31 12:23:10 2017/07/31 12:23:18 13634645 13634637 84831 threat 2014/12/01 14:47:52 2017/07/26 16:34:24 557404252 557404169 93 traffic 2017/07/28 18:03:39 2017/07/28 18:03:50 3619306590 3619306590 1740 hipmatch Not Available Not Available 0 0 0 gtp-tunnel Not Available Not Available 0 0 0 userid Not Available Not Available 0 0 0 auth Not Available Not Available 0 0 0
      Look for the
      ‘Log collection log forwarding agent’ is active and connected to <IP_address>
      line. You can also see that CMS 0 and CMS (the Log Collectors) are not receiving logs.
      On firewalls running PAN-OS 8.1.7 and later releases, you can
      Show Status
      Device
      Setup
      Management
      Cortex Data Lake
      ) to verify that the firewall is connected and sending logs to Cortex Data Lake.
  9. Use the
    ACC
    on Panorama to monitor network activity.
    You can also use
    Monitor
    Manage Custom Reports
    and
    Run Now
    to generate reports on summary logs. You cannot generate scheduled reports or generate reports on detailed logs stored on Cortex Data Lake.
  10. Archive Cortex Data Lake logs.
    If you want to be able to archive the logs you send to Cortex Data Lake for long-term storage, SOC, or internal audit directly from Cortex Data Lake, use the Log Forwarding app, which is included with your Cortex Data Lake (formerly called Logging Service) license. This app enables log forwarding from Cortex Data Lake to an external destination, such as a Syslog server or an email server. (Refer to the Log Forwarding App Getting Started Guide for more information.) Alternatively, continue to forward logs directly from the firewalls to your syslog receiver.

Recommended For You