1. Home
    2. Security Operations
    3. Cortex Data Lake
    4. Cortex Data Lake Getting Started
    5. Forwarding Logs from Cortex Data Lake
    6. Forward Logs from Cortex Data Lake to an HTTPS Server

    Forward Logs from Cortex Data Lake to an HTTPS Server

    Learn how to forward logs from Cortex Data Lake to an HTTPS server.
    To meet your long-term storage, reporting and monitoring, or legal and compliance needs, you can configure Cortex Data Lake to forward logs to an HTTPS server or to the following SIEMs:
    • Splunk HTTP Event Collector (HEC)
    • Microsoft Sentinel
    • Google Chronicle
    For successful log transmission, ensure that your HTTPS receiver:
    • Accepts and parses the correct log format. The format in which
      Cortex Data Lake
       forwards logs depends on the HTTPS receiver:
      Splunk HEC
      (Stacked JSON)
      {
  "time": 1437522387,
  "host": "stream-logfwd20",
  "source": "Palo Alto Networks FLS LF",
  "event": {
    "LogType": "THREAT",
    "Severity": "Critical",
    ...
  }
}
{
  "time": 1437522387,
  "host": "stream-logfwd20",
  "source": "Palo Alto Networks FLS LF",
  "event": {
    "LogType": "THREAT",
    "Severity": "Info",
    ...
  }
}
      Microsoft Sentinel
      (JSON Array)
      [
  {
    "time": 1437522387,
    "host": "stream-logfwd20",
    "source": "Palo Alto Networks FLS LF",
    "event": {
      "message": "Something happened",
      "severity": "INFO"
    }
  },
  {
    "time": 1437522387,
    "host": "stream-logfwd20",
    "source": "testapp",
    "event": {
      "message": "Something happened",
      "severity": "INFO"
    }
  }
]
      Google Chronicle
      (JSON Array)
      {
  "customer_id": "xxxxx-xxxxx-xxxxx-xxxxx-xxxxx",
  "log_type": "ARCSIGHT_CEF",
  "entries": [
    {
      "log_text": "CEF:0|palo alto networks|LF|2.0|DNS|realtime_dns_query|3|ProfileToken=hello-cef dtz=UTC deviceExternalID=xxxxx rt=Jun 09 2022 18:08:31 startTime=Dec 09 1654  PanOSRecordType=null PanOSCloudDNSClientIP= PanOSDNSResolverIP=9.9.9.9 PanOSThreatID=109010001 PanOSDNSCategory=phishing cat=Phishing:blockchain.ppckite.com src= cs4= act= cs5= duser= DestinationDNSDomain="
    },
    {
      "log_text": "CEF:0|palo alto networks|LF|2.0|DNS|realtime_dns_query|3|ProfileToken=hello-cef dtz=UTC deviceExternalID=xxxxx rt=Jun 09 2022 18:08:31 startTime=Dec 09 1654  PanOSRecordType=null PanOSCloudDNSClientIP= PanOSDNSResolverIP=9.9.9.9 PanOSThreatID=109010001 PanOSDNSCategory=phishing cat=Phishing:blockchain.ppckite.com src= cs4= act= cs5= duser= DestinationDNSDomain="
    },
    {
      "log_text": "CEF:0|palo alto networks|LF|2.0|DNS|realtime_dns_query|3|ProfileToken=hello-cef dtz=UTC deviceExternalID=xxxxx rt=Jun 09 2022 18:08:31 startTime=Dec 09 1654  PanOSRecordType=null PanOSCloudDNSClientIP= PanOSDNSResolverIP=9.9.9.9 PanOSThreatID=109010001 PanOSDNSCategory=phishing cat=Phishing:blockchain.ppckite.com src= cs4= act= cs5= duser= DestinationDNSDomain="
    }
  ]
}
      For more information about HTTPS log format, see the Log Forwarding Schema Reference.
    • Accepts and decompresses GZIP HTTPS payloads. Cortex Data Lake compresses the JSON data using GZIP when forwarding through HTTPS.
      Microsoft Sentinel does not accept GZIP, so you must deploy a web app to decompress the data.
    • Accepts a batch size of 500 logs, or 2.25 MB (500 logs x 4500B/log = 2.25MB). This does not apply to Google Chronicle, which accepts only 1 MB of data at a time, or a batch size of 250 logs.
    For each instance of Cortex Data Lake, you can forward logs to up to 200 destinations.
    Cortex Data Lake communicates with the receiver using TLS 1.2 and Java 8 default cipher suites (except GCM ciphers, which are not currently supported). Upon connection Cortex Data Lake validates that the receiver has a certificate signed by a trusted root CA or a private CA. To complete the TLS handshake and establish the connection, the receiver must present all the certificates from the chain of trust.
    If the connection with the HTTPS server fails,
    Cortex Data Lake
     will wait 60 seconds and retry. If the connection times out,
    Cortex Data Lake
     waits 20 seconds and drops the connection with the server before establishing a new one.
    1. Enable communication between Cortex Data Lake and your HTTPS receiver.
      Ensure that your HTTPS receiver can connect to Cortex Data Lake and can present a valid certificate to complete the connection request.
      • Allow an inbound TLS feed to your HTTPS receiver from the IP address range for your
        Cortex Data Lake
         region.
      • Obtain a certificate from a well-known, public CA.
        Because Cortex Data Lake validates the server certificate to establish a connection, you must verify that the receiver is configured to properly send the TLS certificate chain to Cortex Data Lake. If the app cannot verify that the certificate of the receiver and all CAs in the chain are trustworthy, the connection cannot be established. See the list of trusted certificates.
    2. Sign In
       to the hub at https://apps.paloaltonetworks.com/.
    3. Select the Cortex Data Lake instance that you want to configure for HTTPS forwarding.
      If you have multiple Cortex Data Lake instances, click the Cortex Data Lake tile and select an instance from the list of those available.
    4. Select
      Log Forwarding
      Add
       to add a new HTTPS forwarding profile.
    5. Enter a descriptive
      Name
       for the profile.
    6. Enter the
      URL
       for the HTTPS receiver.
      The URL entered must begin with
      https
      . At the end of the top-level domain, you can specify a port number. The default port is 443.
      For Splunk HEC, ensure that the URL ends with
      /event
      .
      If you are forwarding logs to Google Chronicle, you must enter the URL that corresponds to your
      Cortex Data Lake
       region:
      US
      https://malachiteingestion-pa.googleapis.com/v2/unstructuredlogentries:batchCreate
      EU
      https://europe-malachiteingestion-pa.googleapis.com/v2/unstructuredlogentries:batchCreate
      Asia
      https://asia-southeast1-malachiteingestion-pa.googleapis.com/v2/unstructuredlogentries:batchCreate
    7. Configure Client Authorization.
      Do this if you are forwarding logs to a cloud HTTPS receiver, or if your HTTPS server requires a username and password.
      1. Select the
        Type
         of client authorization and enter the necessary credentials.
        HTTPS Receiver
        Type
        Credentials
        Splunk HEC
        Splunk Authorization
        Microsoft Sentinel
        Sentinel Authorization
        • Workspace ID
        • Primary Key
        Google Chronicle
        Chronicle Authorization
        • Service Account
          This should be a properly formatted JSON document. If you don’t know your Service Account credentials, contact Google Chronicle support.
        HTTPS Server (Password-protected)
        Basic Authorization
        • Username
        • Password
        HTTPS Server
        None
        None
    8. Test Connection
       to ensure that Cortex Data Lake can communicate with the receiver.
      This sends an empty log to the configured destination to verify that transmission is possible.
      If the test fails, you will not be able to proceed.
    9. Click
      Next
      .
    10. (
      Optional
      ) To receive a
      STATUS NOTIFICATION
       when Cortex Data Lake is unable to connect to the HTTPS receiver, enter the email address at which you’d like to receive the notification.
      You will continue to receive these notifications every 60 minutes until connectivity is restored. If the connectivity issue is addressed within 72 hours, no logs will be lost. However, any log older than 72 hours following the service disconnection could be lost.
    11. Enter a unique PROFILE TOKEN if your receiver needs to distinguish logs coming from different tenants.
    12. Select the logs you want to forward.
      1. Add
         a new log filter.
      2. Select the log type.
        The Threat log type does not include URL logs or Data logs. If you wish to forward these log types, you must add them individually.
      3. (Optional)
         Create a log filter to forward only the logs that are most critical to you.
        You can either write your own queries from scratch or use the query builder. You can also select the query field to choose from among a set of common predefined queries.
        Log filters function like queries in Explore, with the following differences:
        • Double quotes (
          “”
          ) are not supported.
        • Subnet masks are not supported in log forwarding filters. To return IP addresses with subnets, use the
          LIKE
           operator. Example:
          src_ip.value LIKE “192.1.1.%”
          .
        If you want to forward all logs of the type you selected, do not enter a query. Instead, proceed to the next step.
      4. Save
         your changes.
    13. Save
       your changes.
    14. Verify that the
      Status
       of your HTTPS forwarding profile is
      Running
       ( ).
      Immediately after creating or editing your profile, the
      Status
       may be
      Provisioning
       for up to 10 minutes.
    15. Verify that you can view logs on the HTTPS receiver.
      For Splunk Common Information Model (CIM) fields and Enterprise Security, follow the guide at https://splunk.paloaltonetworks.com to install the PANW Splunk Addon and create a Splunk HEC input.

    Recommended For You

    Recommended Videos

    Recommended videos not found.

    © 2023 Palo Alto Networks, Inc. All rights reserved.

    Techdocs Logo