Cortex Data Lake
    : Forward Logs from Cortex Data Lake to an HTTPS Server
    Focus
    Download PDF
    Focus
    1. Home
    2. Security Operations
    3. Cortex Data Lake
    4. Cortex Data Lake
    5. Forwarding Logs from Cortex Data Lake
    6. Forward Logs from Cortex Data Lake to an HTTPS Server
    Download PDF

    Cortex Data Lake

    Forward Logs from Cortex Data Lake to an HTTPS Server

    Table of Contents

    Forward Logs from
    Cortex Data Lake
     to an HTTPS Server

    Learn how to forward logs from
    Cortex Data Lake
     to an HTTPS server.
    To meet your long-term storage, reporting and monitoring, or legal and compliance needs, you can configure
    Cortex Data Lake
     to forward logs to an HTTPS server or to the following SIEMs:
    • Exabeam
    • Google Chronicle
    • Microsoft Sentinel
    • Splunk HTTP Event Collector (HEC)
    For successful log transmission, ensure that your HTTPS receiver:
    • Accepts and parses the correct log format. The format in which
      Cortex Data Lake
       forwards logs depends on the HTTPS receiver:
      Google Chronicle
      (JSON Array)
      {
  "customer_id": "xxxxx-xxxxx-xxxxx-xxxxx-xxxxx",
  "log_type": "ARCSIGHT_CEF",
  "entries": [
    {
      "log_text": "CEF:0|palo alto networks|LF|2.0|DNS|realtime_dns_query|3|ProfileToken=hello-cef dtz=UTC deviceExternalID=xxxxx rt=Jun 09 2022 18:08:31 startTime=Dec 09 1654  PanOSRecordType=null PanOSCloudDNSClientIP= PanOSDNSResolverIP=9.9.9.9 PanOSThreatID=109010001 PanOSDNSCategory=phishing cat=Phishing:blockchain.ppckite.com src= cs4= act= cs5= duser= DestinationDNSDomain="
    },
    {
      "log_text": "CEF:0|palo alto networks|LF|2.0|DNS|realtime_dns_query|3|ProfileToken=hello-cef dtz=UTC deviceExternalID=xxxxx rt=Jun 09 2022 18:08:31 startTime=Dec 09 1654  PanOSRecordType=null PanOSCloudDNSClientIP= PanOSDNSResolverIP=9.9.9.9 PanOSThreatID=109010001 PanOSDNSCategory=phishing cat=Phishing:blockchain.ppckite.com src= cs4= act= cs5= duser= DestinationDNSDomain="
    },
    {
      "log_text": "CEF:0|palo alto networks|LF|2.0|DNS|realtime_dns_query|3|ProfileToken=hello-cef dtz=UTC deviceExternalID=xxxxx rt=Jun 09 2022 18:08:31 startTime=Dec 09 1654  PanOSRecordType=null PanOSCloudDNSClientIP= PanOSDNSResolverIP=9.9.9.9 PanOSThreatID=109010001 PanOSDNSCategory=phishing cat=Phishing:blockchain.ppckite.com src= cs4= act= cs5= duser= DestinationDNSDomain="
    }
  ]
}
      Microsoft Sentinel
      (JSON Array)
      [
  {
    "time": 1437522387,
    "host": "stream-logfwd20",
    "source": "Palo Alto Networks FLS LF",
    "event": {
      "message": "Something happened",
      "severity": "INFO"
    }
  },
  {
    "time": 1437522387,
    "host": "stream-logfwd20",
    "source": "testapp",
    "event": {
      "message": "Something happened",
      "severity": "INFO"
    }
  }
]
      Splunk HEC
      (Stacked JSON)
      {
  "time": 1437522387,
  "host": "stream-logfwd20",
  "source": "Palo Alto Networks FLS LF",
  "event": {
    "LogType": "THREAT",
    "Severity": "Critical",
    ...
  }
}
{
  "time": 1437522387,
  "host": "stream-logfwd20",
  "source": "Palo Alto Networks FLS LF",
  "event": {
    "LogType": "THREAT",
    "Severity": "Info",
    ...
  }
}
      Exabeam
      (JSON Array)
      {
  "DestinationDeviceCategory": "N-Phone",
  "DestinationDeviceOSFamily": "H1511",
  "DestinationDeviceOSVersion": "Android v7",
  "SourceDeviceOS": null,
  "NATDestination": "xxx.xx.x.xxx",
  "ApplicationSubcategory": "database",
  "VendorName": "Palo Alto Networks",
  "Protocol": "tcp",
  "IsServertoClient": false,
  "PacketID": 0,
  "NSSAINetworkSliceType": "fc",
  "DirectionOfAttack": "client to server",
  "EndpointSerialNumber": "SG0000001",
  "ApplicationTechnology": "network-protocol",
  "VendorSeverity": "Critical",
  "SubType": "data",
  "DeviceSN": "xxxxxxxxxx",
  "ConfigVersion": "10.2",
  "IsMptcpOn": false,
  "SourceDeviceModel": "Nexus",
  "InboundInterface": "ethernet1/1",
  "LogExported": false,
  "ParentSessionID": 0,
  "CloudHostname": "PA-VM-E2E-PCL-TEST",
  "SourcePort": 25195,
  "CaptivePortal": false,
  "LogSource": "firewall",
  "LogType": "THREAT",
  "InboundInterfaceDetailsType": "ethernet",
  "Severity": "Critical",
  "TimeReceived": "2023-07-22T08:49:30.000000Z",
  "SourceUserDomain": "paloaltonetwork",
  "IsDuplicateLog": false,
}
      For more information about HTTPS log format, see the Log Forwarding Schema Reference.
    • Accepts and decompresses GZIP HTTPS payloads.
      Cortex Data Lake
       compresses the JSON data using GZIP when forwarding through HTTPS.
      Microsoft Sentinel does not accept GZIP, so you must deploy a web app to decompress the data.
    • Accepts a batch size of 500 logs, or 2.25 MB (500 logs x 4500 B = 2.25 MB). This does not apply to Google Chronicle, which accepts only 1 MB of data at a time, or a batch size of 250 logs.
    For each instance of
    Cortex Data Lake
    , you can forward logs to up to 200 destinations.
    Cortex Data Lake
     communicates with the receiver using TLS 1.2 and Java 8 default cipher suites (except GCM ciphers, which CDL does not currently support). Upon connection,
    Cortex Data Lake
     validates that the receiver has a certificate signed by a trusted root CA or a private CA. To complete the TLS handshake and establish the connection, the receiver must present all the certificates from the chain of trust.
    If the connection with the HTTPS server fails,
    Cortex Data Lake
     will wait 60 seconds and retry. If the connection times out,
    Cortex Data Lake
     waits 20 seconds and drops the connection with the server before establishing a new one.
    1. Enable communication between
      Cortex Data Lake
       and your HTTPS receiver.
      Ensure that your HTTPS receiver can connect to
      Cortex Data Lake
       and can present a valid certificate to complete the connection request.
      • Allow an inbound TLS feed to your HTTPS receiver from the IP address range for your
        Cortex Data Lake
         region.
      • Obtain a certificate from a well-known, public CA.
        Because
        Cortex Data Lake
         validates the server certificate to establish a connection, you must verify that you have configured the receiver to properly send the TLS certificate chain to
        Cortex Data Lake
        . If the app can't verify that the certificate of the receiver and all CAs in the chain are trustworthy, it can't establish a connection. See the list of trusted certificates.
    2. Sign In
       to the hub.
    3. Select the
      Cortex Data Lake
       instance that you want to configure for HTTPS forwarding.
      If you have multiple
      Cortex Data Lake
       instances, click the
      Cortex Data Lake
       tile and select an instance from the list of those available.
    4. Select
      Log Forwarding
      Add
       to add a new HTTPS forwarding profile.
    5. Enter a descriptive
      Name
       for the profile.
    6. Enter the
      URL
       for the HTTPS receiver.
      The URL entered must begin with
      https
      . At the end of the top-level domain, you can specify a port number. The default port is 443.
      For Splunk HEC, ensure that the URL ends with
      /event
      .
      If you're forwarding logs to Google Chronicle, you must enter the URL that corresponds to your
      Cortex Data Lake
       region:
      US
      https://malachiteingestion-pa.googleapis.com/v2/unstructuredlogentries:batchCreate
      EU
      https://europe-malachiteingestion-pa.googleapis.com/v2/unstructuredlogentries:batchCreate
      Asia
      https://asia-southeast1-malachiteingestion-pa.googleapis.com/v2/unstructuredlogentries:batchCreate
    7. Configure Client Authorization.
      Do this if you're forwarding logs to a cloud HTTPS receiver, or if your HTTPS server requires a username and password.
      1. Select the
        Type
         of client authorization and enter the necessary credentials.
        HTTPS Receiver
        Type
        Credentials
        Splunk HEC
        Splunk Authorization
        HTTPS Server (Password-protected)
        Basic Authorization
        • Username
        • Password
        Exabeam
        Exabeam Authorization
        Microsoft Sentinel
        Sentinel Authorization
        • Workspace ID
        • Primary Key
        Google Chronicle
        Chronicle Authorization
        • Service Account
          This should be a properly formatted JSON document. If you don’t know your Service Account credentials, contact Google Chronicle support.
        HTTPS Server
        None
        None
    8. Test Connection
       to ensure that
      Cortex Data Lake
       can communicate with the receiver.
      This sends an empty log to the configured destination to verify that transmission is possible.
      If the test fails, you won't be able to proceed.
    9. Click
      Next
      .
    10. (
      Optional
      ) To receive a
      STATUS NOTIFICATION
       when
      Cortex Data Lake
       is unable to connect to the HTTPS receiver, enter the email address at which you’d like to receive the notification.
      You will continue to receive these notifications every 60 minutes until the service restores connectivity. If the connectivity issue resolves within 72 hours, the service won't lose logs. However, the service could lose any log older than 72 hours following the service disconnection.
    11. Enter a unique PROFILE TOKEN if your receiver needs to distinguish logs coming from different tenants.
    12. Select the logs you want to forward.
      1. Add
         a new log filter.
      2. Select the log type.
        The Threat log type does not include URL logs or Data logs. If you wish to forward these log types, you must add them individually.
      3. (
        Optional
        ) Create a log filter to forward only the logs that are most critical to you.
        You can either write your own queries from scratch or use the query builder. You can also select the query field to choose from among a set of common predefined queries.
        Log filters function like queries in Explore, with the following differences:
        • No double quotes (
          “”
          ).
        • No subnet masks. To return IP addresses with subnets, use the
          LIKE
           operator. Example:
          src_ip.value LIKE “192.1.1.%”
          .
        If you want to forward all logs of the type you selected, don't enter a query. Instead, proceed to the next step.
      4. Save
         your changes.
    13. Save
       your changes.
    14. Verify that the
      Status
       of your HTTPS forwarding profile is
      Running
       ( ).
      Immediately after creating or editing your profile, the
      Status
       might be
      Provisioning
       for up to 10 minutes.
    15. Verify that you can view logs on the HTTPS receiver.
      For Splunk Common Information Model (CIM) fields and Enterprise Security, follow the guide at https://splunk.paloaltonetworks.com to install the Palo Alto Networks Splunk add-on and create a Splunk HEC input.
    16. (Optional)
       You can use the running HTTPS forwarding profile to forward past logs spanning up to 3 days.

    Recommended For You

    © 2023 Palo Alto Networks, Inc. All rights reserved.