Forward Logs from Cortex Data Lake to an HTTPS Server

Learn how to forward logs from Cortex Data Lake to an HTTPS server.
To meet your long-term storage, reporting and monitoring, or legal and compliance needs, you can configure Cortex Data Lake to forward logs to an HTTPS server or to the following SIEMs:
  • Splunk HTTP Event Collector (HEC)
  • Microsoft Sentinel
  • Google Chronicle
For successful log transmission, ensure that your HTTPS receiver:
  • Accepts and parses the correct log format. The format in which
    Cortex Data Lake
    forwards logs depends on the HTTPS receiver:
    Splunk HEC
    (Stacked JSON)
    { "time": 1437522387, "host": "stream-logfwd20", "source": "Palo Alto Networks FLS LF", "event": { "LogType": "THREAT", "Severity": "Critical", ... } } { "time": 1437522387, "host": "stream-logfwd20", "source": "Palo Alto Networks FLS LF", "event": { "LogType": "THREAT", "Severity": "Info", ... } }
    Microsoft Sentinel
    (JSON Array)
    [ { "time": 1437522387, "host": "stream-logfwd20", "source": "Palo Alto Networks FLS LF", "event": { "message": "Something happened", "severity": "INFO" } }, { "time": 1437522387, "host": "stream-logfwd20", "source": "testapp", "event": { "message": "Something happened", "severity": "INFO" } } ]
    Google Chronicle
    (JSON Array)
    { "customer_id": "xxxxx-xxxxx-xxxxx-xxxxx-xxxxx", "log_type": "ARCSIGHT_CEF", "entries": [ { "log_text": "CEF:0|palo alto networks|LF|2.0|DNS|realtime_dns_query|3|ProfileToken=hello-cef dtz=UTC deviceExternalID=xxxxx rt=Jun 09 2022 18:08:31 startTime=Dec 09 1654 PanOSRecordType=null PanOSCloudDNSClientIP= PanOSDNSResolverIP=9.9.9.9 PanOSThreatID=109010001 PanOSDNSCategory=phishing cat=Phishing:blockchain.ppckite.com src= cs4= act= cs5= duser= DestinationDNSDomain=" }, { "log_text": "CEF:0|palo alto networks|LF|2.0|DNS|realtime_dns_query|3|ProfileToken=hello-cef dtz=UTC deviceExternalID=xxxxx rt=Jun 09 2022 18:08:31 startTime=Dec 09 1654 PanOSRecordType=null PanOSCloudDNSClientIP= PanOSDNSResolverIP=9.9.9.9 PanOSThreatID=109010001 PanOSDNSCategory=phishing cat=Phishing:blockchain.ppckite.com src= cs4= act= cs5= duser= DestinationDNSDomain=" }, { "log_text": "CEF:0|palo alto networks|LF|2.0|DNS|realtime_dns_query|3|ProfileToken=hello-cef dtz=UTC deviceExternalID=xxxxx rt=Jun 09 2022 18:08:31 startTime=Dec 09 1654 PanOSRecordType=null PanOSCloudDNSClientIP= PanOSDNSResolverIP=9.9.9.9 PanOSThreatID=109010001 PanOSDNSCategory=phishing cat=Phishing:blockchain.ppckite.com src= cs4= act= cs5= duser= DestinationDNSDomain=" } ] }
    For more information about HTTPS log format, see the Log Forwarding Schema Reference.
  • Accepts and decompresses GZIP HTTPS payloads. Cortex Data Lake compresses the JSON data using GZIP when forwarding through HTTPS.
    Microsoft Sentinel does not accept GZIP, so you must deploy a web app to decompress the data.
  • Accepts a batch size of 500 logs, or 2.25 MB (500 logs x 4500B/log = 2.25MB). This does not apply to Google Chronicle, which accepts only 1 MB of data at a time, or a batch size of 250 logs.
For each instance of Cortex Data Lake, you can forward logs to up to 200 destinations.
Cortex Data Lake communicates with the receiver using TLS 1.2 and Java 8 default cipher suites (except GCM ciphers, which are not currently supported). Upon connection Cortex Data Lake validates that the receiver has a certificate signed by a trusted root CA or a private CA. To complete the TLS handshake and establish the connection, the receiver must present all the certificates from the chain of trust.
If the connection with the HTTPS server fails,
Cortex Data Lake
will wait 60 seconds and retry. If the connection times out,
Cortex Data Lake
waits 20 seconds and drops the connection with the server before establishing a new one.
  1. Enable communication between Cortex Data Lake and your HTTPS receiver.
    Ensure that your HTTPS receiver can connect to Cortex Data Lake and can present a valid certificate to complete the connection request.
    • Allow an inbound TLS feed to your HTTPS receiver from the IP address range for your
      Cortex Data Lake
      region.
    • Obtain a certificate from a well-known, public CA.
      Because Cortex Data Lake validates the server certificate to establish a connection, you must verify that the receiver is configured to properly send the TLS certificate chain to Cortex Data Lake. If the app cannot verify that the certificate of the receiver and all CAs in the chain are trustworthy, the connection cannot be established. See the list of trusted certificates.
  2. Sign In
    to the hub at https://apps.paloaltonetworks.com/.
  3. Select the Cortex Data Lake instance that you want to configure for HTTPS forwarding.
    If you have multiple Cortex Data Lake instances, click the Cortex Data Lake tile and select an instance from the list of those available.
  4. Select
    Log Forwarding
    Add
    to add a new HTTPS forwarding profile.
  5. Enter a descriptive
    Name
    for the profile.
  6. Enter the
    URL
    for the HTTPS receiver.
    The URL entered must begin with
    https
    . At the end of the top-level domain, you can specify a port number. The default port is 443.
    For Splunk HEC, ensure that the URL ends with
    /event
    .
    If you are forwarding logs to Google Chronicle, you must enter the URL that corresponds to your
    Cortex Data Lake
    region:
    US
    https://malachiteingestion-pa.googleapis.com/v2/unstructuredlogentries:batchCreate
    EU
    https://europe-malachiteingestion-pa.googleapis.com/v2/unstructuredlogentries:batchCreate
    Asia
    https://asia-southeast1-malachiteingestion-pa.googleapis.com/v2/unstructuredlogentries:batchCreate
  7. Configure Client Authorization.
    Do this if you are forwarding logs to a cloud HTTPS receiver, or if your HTTPS server requires a username and password.
    1. Select the
      Type
      of client authorization and enter the necessary credentials.
      HTTPS Receiver
      Type
      Credentials
      Splunk HEC
      Splunk Authorization
      Microsoft Sentinel
      Sentinel Authorization
      • Workspace ID
      • Primary Key
      Google Chronicle
      Chronicle Authorization
      • Service Account
        This should be a properly formatted JSON document. If you don’t know your Service Account credentials, contact Google Chronicle support.
      HTTPS Server (Password-protected)
      Basic Authorization
      • Username
      • Password
      HTTPS Server
      None
      None
  8. Test Connection
    to ensure that Cortex Data Lake can communicate with the receiver.
    This sends an empty log to the configured destination to verify that transmission is possible.
    If the test fails, you will not be able to proceed.
  9. Click
    Next
    .
  10. (
    Optional
    ) To receive a
    STATUS NOTIFICATION
    when Cortex Data Lake is unable to connect to the HTTPS receiver, enter the email address at which you’d like to receive the notification.
    You will continue to receive these notifications every 60 minutes until connectivity is restored. If the connectivity issue is addressed within 72 hours, no logs will be lost. However, any log older than 72 hours following the service disconnection could be lost.
  11. Enter a unique PROFILE TOKEN if your receiver needs to distinguish logs coming from different tenants.
  12. Select the logs you want to forward.
    1. Add
      a new log filter.
    2. Select the log type.
      The Threat log type does not include URL logs or Data logs. If you wish to forward these log types, you must add them individually.
    3. (Optional)
      Create a log filter to forward only the logs that are most critical to you.
      You can either write your own queries from scratch or use the query builder. You can also select the query field to choose from among a set of common predefined queries.
      Log filters function like queries in Explore. However, double quotes (
      “”
      ) are not supported.
      If you want to forward all logs of the type you selected, do not enter a query. Instead, proceed to the next step.
    4. Save
      your changes.
  13. Save
    your changes.
  14. Verify that the
    Status
    of your HTTPS forwarding profile is
    Running
    ( ).
    Immediately after creating or editing your profile, the
    Status
    may be
    Provisioning
    for up to 10 minutes.
  15. Verify that you can view logs on the HTTPS receiver.
    For Splunk Common Information Model (CIM) fields and Enterprise Security, follow the guide at https://splunk.paloaltonetworks.com to install the PANW Splunk Addon and create a Splunk HEC input.

Recommended For You