: Forward Logs from Cortex Data Lake to an HTTPS Server
Focus
Focus

Forward Logs from Cortex Data Lake to an HTTPS Server

Table of Contents

Forward Logs from
Cortex Data Lake
to an HTTPS Server

Learn how to forward logs from
Cortex Data Lake
to an HTTPS server.
To meet your long-term storage, reporting and monitoring, or legal and compliance needs, you can configure
Cortex Data Lake
to forward logs to an HTTPS server or to the following SIEMs:
  • Exabeam
  • Google Chronicle
  • Microsoft Sentinel
  • Splunk HTTP Event Collector (HEC)
For successful log transmission, ensure that your HTTPS receiver:
  • Accepts and parses the correct log format. The format in which
    Cortex Data Lake
    forwards logs depends on the HTTPS receiver:
    Google Chronicle
    (JSON Array)
    { "customer_id": "xxxxx-xxxxx-xxxxx-xxxxx-xxxxx", "log_type": "ARCSIGHT_CEF", "entries": [ { "log_text": "CEF:0|palo alto networks|LF|2.0|DNS|realtime_dns_query|3|ProfileToken=hello-cef dtz=UTC deviceExternalID=xxxxx rt=Jun 09 2022 18:08:31 startTime=Dec 09 1654 PanOSRecordType=null PanOSCloudDNSClientIP= PanOSDNSResolverIP=9.9.9.9 PanOSThreatID=109010001 PanOSDNSCategory=phishing cat=Phishing:blockchain.ppckite.com src= cs4= act= cs5= duser= DestinationDNSDomain=" }, { "log_text": "CEF:0|palo alto networks|LF|2.0|DNS|realtime_dns_query|3|ProfileToken=hello-cef dtz=UTC deviceExternalID=xxxxx rt=Jun 09 2022 18:08:31 startTime=Dec 09 1654 PanOSRecordType=null PanOSCloudDNSClientIP= PanOSDNSResolverIP=9.9.9.9 PanOSThreatID=109010001 PanOSDNSCategory=phishing cat=Phishing:blockchain.ppckite.com src= cs4= act= cs5= duser= DestinationDNSDomain=" }, { "log_text": "CEF:0|palo alto networks|LF|2.0|DNS|realtime_dns_query|3|ProfileToken=hello-cef dtz=UTC deviceExternalID=xxxxx rt=Jun 09 2022 18:08:31 startTime=Dec 09 1654 PanOSRecordType=null PanOSCloudDNSClientIP= PanOSDNSResolverIP=9.9.9.9 PanOSThreatID=109010001 PanOSDNSCategory=phishing cat=Phishing:blockchain.ppckite.com src= cs4= act= cs5= duser= DestinationDNSDomain=" } ] }
    Microsoft Sentinel
    (JSON Array)
    [ { "time": 1437522387, "host": "stream-logfwd20", "source": "Palo Alto Networks FLS LF", "event": { "message": "Something happened", "severity": "INFO" } }, { "time": 1437522387, "host": "stream-logfwd20", "source": "testapp", "event": { "message": "Something happened", "severity": "INFO" } } ]
    Splunk HEC
    (Stacked JSON)
    { "time": 1437522387, "host": "stream-logfwd20", "source": "Palo Alto Networks FLS LF", "event": { "LogType": "THREAT", "Severity": "Critical", ... } } { "time": 1437522387, "host": "stream-logfwd20", "source": "Palo Alto Networks FLS LF", "event": { "LogType": "THREAT", "Severity": "Info", ... } }
    Exabeam
    (JSON Array)
    { "DestinationDeviceCategory": "N-Phone", "DestinationDeviceOSFamily": "H1511", "DestinationDeviceOSVersion": "Android v7", "SourceDeviceOS": null, "NATDestination": "xxx.xx.x.xxx", "ApplicationSubcategory": "database", "VendorName": "Palo Alto Networks", "Protocol": "tcp", "IsServertoClient": false, "PacketID": 0, "NSSAINetworkSliceType": "fc", "DirectionOfAttack": "client to server", "EndpointSerialNumber": "SG0000001", "ApplicationTechnology": "network-protocol", "VendorSeverity": "Critical", "SubType": "data", "DeviceSN": "xxxxxxxxxx", "ConfigVersion": "10.2", "IsMptcpOn": false, "SourceDeviceModel": "Nexus", "InboundInterface": "ethernet1/1", "LogExported": false, "ParentSessionID": 0, "CloudHostname": "PA-VM-E2E-PCL-TEST", "SourcePort": 25195, "CaptivePortal": false, "LogSource": "firewall", "LogType": "THREAT", "InboundInterfaceDetailsType": "ethernet", "Severity": "Critical", "TimeReceived": "2023-07-22T08:49:30.000000Z", "SourceUserDomain": "paloaltonetwork", "IsDuplicateLog": false, }
    For more information about HTTPS log format, see the Log Forwarding Schema Reference.
  • Accepts and decompresses GZIP HTTPS payloads.
    Cortex Data Lake
    compresses the JSON data using GZIP when forwarding through HTTPS.
  • (
    For Microsoft Sentinel Only
    ) - Microsoft Sentinel does not accept GZIP, so you must deploy a web app to decompress the data.
  • Accepts a batch size of 500 logs, or 2.25 MB (500 logs x 4500 B = 2.25 MB). This does not apply to Google Chronicle, which accepts only 1 MB of data at a time, or a batch size of 250 logs. For each instance of
    Cortex Data Lake
    , you can forward logs to up to 200 destinations.
  • Can connect to
    Cortex Data Lake
    -
    Cortex Data Lake
    communicates with the receiver using TLS 1.2 and Java 8 default cipher suites (except GCM ciphers, which
    Cortex Data Lake
    does not currently support). Upon connection,
    Cortex Data Lake
    validates that the receiver has a certificate signed by a trusted root CA or a private CA. To complete the TLS handshake and establish the connection, the receiver must present all the certificates from the chain of trust.
    • Allow an inbound TLS feed to your HTTPS receiver from the IP address range for your
      Cortex Data Lake
      region.
    • Obtain a certificate from a well-known, public CA.
      Because
      Cortex Data Lake
      validates the server certificate to establish a connection, you must verify that you have configured the receiver to properly send the TLS certificate chain to
      Cortex Data Lake
      . If the app can't verify that the certificate of the receiver and all CAs in the chain are trustworthy, it can't establish a connection. See the list of trusted certificates.
    If the connection with the HTTPS server fails,
    Cortex Data Lake
    will wait 60 seconds and retry. If the connection times out,
    Cortex Data Lake
    waits 20 seconds and drops the connection with the server before establishing a new one.
  1. Sign In
    to the hub.
  2. Select the
    Cortex Data Lake
    instance that you want to configure for HTTPS forwarding.
    If you have multiple
    Cortex Data Lake
    instances, click the
    Cortex Data Lake
    tile and select an instance from the list of those available.
  3. Select
    Log Forwarding
    Add
    to add a new HTTPS forwarding profile.
  4. Enter a descriptive
    Name
    for the profile.
  5. Enter the
    URL
    that you copied in step 13 for the HTTPS receiver.
    The URL entered must begin with
    https
    . At the end of the top-level domain, you can specify a port number. The default port is 443.
    For Splunk HEC, ensure that the URL ends with
    /event
    .
    If you're forwarding logs to Google Chronicle, you must enter the URL that corresponds to your
    Cortex Data Lake
    region:
    US
    https://malachiteingestion-pa.googleapis.com/v2/unstructuredlogentries:batchCreate
    EU
    https://europe-malachiteingestion-pa.googleapis.com/v2/unstructuredlogentries:batchCreate
    Asia
    https://asia-southeast1-malachiteingestion-pa.googleapis.com/v2/unstructuredlogentries:batchCreate
  6. (
    Optional
    ) Upload a self-signed certificate if you do not want to use a publicly signed certificate.
  7. (
    Optional
    )
    Upload
    the private Root CA and intermediate CAs (If an intermediate CA exists). Do not upload the certificate issued for the https server—only CA certificates are needed to verify the chain from the https server.
    Only do this if you installed a private CA-signed, self-signed certificate on your receiver, or the public CA is not in the list of trusted CAs. The file containing the certificates must be in PEM format.
  8. (
    Optional
    ) Enable client authentication.
    Do this if company or regulatory policy requires client authentication when forwarding logs to your server.
    1. Download
      the certificate chain.
    2. Upload the certificate chain to your server.
      Refer to the documentation for your server management software to find out how to do this.
  9. Configure Client Authorization.
    Do this if you're forwarding logs to a cloud HTTPS receiver, or if your HTTPS server requires a username and password.
    1. Select the
      Type
      of client authorization and enter the necessary credentials.
      HTTPS Receiver
      Type
      Credentials
      Splunk HEC
      Splunk Authorization
      HTTPS Server (Password-protected)
      Basic Authorization
      • Username
      • Password
      Exabeam
      Exabeam Authorization
      Microsoft Sentinel
      Sentinel Authorization
      • Workspace ID
      • Primary Key
      Google Chronicle
      Chronicle Authorization
      • Service Account
        This should be a properly formatted JSON document. If you don’t know your Service Account credentials, contact Google Chronicle support.
      HTTPS Server
      None
      None
  10. Test Connection
    to ensure that
    Cortex Data Lake
    can communicate with the receiver.
    This sends an empty log to the configured destination to verify that transmission is possible.
    If the test fails, you won't be able to proceed.
  11. Click
    Next
    .
  12. (
    Optional
    ) To receive a
    STATUS NOTIFICATION
    when
    Cortex Data Lake
    is unable to connect to the HTTPS receiver, enter the email address at which you’d like to receive the notification.
    You will continue to receive these notifications every 60 minutes until the service restores connectivity. If the connectivity issue resolves within 72 hours, the service won't lose logs. However, the service could lose any log older than 72 hours following the service disconnection.
  13. Enter a unique PROFILE TOKEN if your receiver needs to distinguish logs coming from different tenants.
  14. Select the logs you want to forward.
    1. Add
      a new log filter.
    2. Select the log type.
      The Threat log type does not include URL logs or Data logs. If you wish to forward these log types, you must add them individually.
    3. (
      Optional
      ) Create a log filter to forward only the logs that are most critical to you.
      You can either write your own queries from scratch or use the query builder. You can also select the query field to choose from among a set of common predefined queries.
      Log filters function like queries in Explore, with the following differences:
      • No double quotes (
        “”
        ).
      • No subnet masks. To return IP addresses with subnets, use the
        LIKE
        operator. Example:
        src_ip.value LIKE “192.1.1.%”
        .
      If you want to forward all logs of the type you selected, don't enter a query. Instead, proceed to the next step.
    4. Save
      your changes.
  15. Save
    your changes.
  16. Verify that the
    Status
    of your HTTPS forwarding profile is
    Running
    ( ).
    Immediately after creating or editing your profile, the
    Status
    might be
    Provisioning
    for up to 10 minutes.
  17. Verify that you can view logs on the HTTPS receiver.
    For Splunk Common Information Model (CIM) fields and Enterprise Security, follow the guide at https://splunk.paloaltonetworks.com to install the Palo Alto Networks Splunk add-on and create a Splunk HEC input.
  18. (
    Optional
    ) You can use the running HTTPS forwarding profile to forward past logs spanning up to 3 days.

Create and Deploy a Agent Web Application

Microsoft Sentinel does not accept GZIP to compress the data when forwarding through HTTPS receiver. To enable this, you must deploy a web application.
  1. Log in to your Microsoft Azure account, and create a log analytics workspace in your Sentinel.
  2. Install Visual Studio Code version 1.64.1 or a later version.
  3. Install the Azure Tools and Azure App Service extensions in Visual Studio Code.
  4. Obtain the agent web application’s code from GitHub - git clone https://github.com/PaloAltoNetworks/cdl-decompress-proxy-sentinel-ingest.git
    Download and extract the ZIP folder if you did not install Git. https://github.com/PaloAltoNetworks/cdl-decompress-proxy-sentinel-ingest/archive/refs/heads/master.zip
  5. Open the
    cdl-decompress-proxy-sentinel-ingest
    folder in Visual Studio Code.
  6. Sign in to Azure and click
    Resources
    your subscription
    App Services
    .
  7. Right click and select
    Create New Web App
    . Select the advanced option if you want to make use of previously created Azure resources.
  8. Enter a name, choose the
    Python 3.9
    runtime stack, and select an appropriate pricing tier.
  9. Right click the new agent web app and select
    Deploy to Web App
    .
  10. Connect the web app to the Log Analytics workspace - In Azure, navigate to the desired Log Analytics workspace, and select
    Agents management
    Linux servers
    . Copy the Workspace ID and Primary Key values. These details are used while adding a new setting in your agent web application.
  11. (
    Optional
    ) In Azure, navigate to agent web app and select
    Settings
    Identity
    System assigned
    , change
    Status
    to
    On
    .
  12. Access or create an Azure Key Vault to store the workspace ID and primary key values as secrets in the key vault. If you don’t have an Azure key vault, you can add the values as settings directly in the agent web app.
    • To create or import a key vault:
      1. Click
        Settings > Secrets > and click Generate or Import
        .
      2. Enter the name for the secret where you will store the Workspace ID value you collected earlier, enter the value, and click Create.
      3. Copy the secret URI you just created. For example,
        https://<key vault name>.vault.azure.net/secrets/<secret name>
      4. Click
        Settings > Access policies
        and click
        Add Access Policy
        .
      5. Select
        Get
        from the
        Secret permissions
        drop-down.
      6. Click
        None selected
        , select your agent web application and click
        Add
        .
      7. Save the access policy.
    • Add Value in agent web application:
      1. In Visual Studio, open the agent web application, right-click
        Application Settings
        and select
        Add New Settings
        .
      2. Enter WORKSPACE_ID and press enter.
        If you have stored the values as secrets in a Key Vault, enter the Secret URI, that represents the Workspace ID you collected earlier, in this format -
        @Microsoft.KeyVault(SecretUri=https://<key vault name>.vault.azure.net/secrets/<secret name>)
        If you did not store the value as a secret in a Key Vault, enter in the WORKSPACE_ID value you copied down earlier ((step 10) while connecting to Log Analytics workspace).
      3. Follow the same process to add another new setting named SHARED_KEY with the Primary Key value collected earlier, or enter the SecretURI if you have this value stored in a Key Vault.
      4. Right click the web app and select
        Restart
        .
  13. In Azure, copy the URL from your web app (App Services). This URL is used in the URL field while adding a HTTPS forwarding profile.

Recommended For You