Forward Logs from Cortex Data Lake to an HTTPS Server
Learn how to forward logs from Cortex Data Lake to an
HTTPS server.
To meet your long-term storage, reporting
and monitoring, or legal and compliance needs, you can configure
Cortex Data Lake to forward logs to an HTTPS server or to the following
SIEMs:
- Splunk HTTP Event Collector (HEC)
- Microsoft Sentinel
- Google Chronicle
For successful
log transmission, ensure that your HTTPS receiver:
- Accepts and parses the correct log format. The format in whichCortex Data Lakeforwards logs depends on the HTTPS receiver:Splunk HEC(Stacked JSON){ "time": 1437522387, "host": "stream-logfwd20", "source": "Palo Alto Networks FLS LF", "event": { "LogType": "THREAT", "Severity": "Critical", ... } } { "time": 1437522387, "host": "stream-logfwd20", "source": "Palo Alto Networks FLS LF", "event": { "LogType": "THREAT", "Severity": "Info", ... } }Microsoft Sentinel(JSON Array)[ { "time": 1437522387, "host": "stream-logfwd20", "source": "Palo Alto Networks FLS LF", "event": { "message": "Something happened", "severity": "INFO" } }, { "time": 1437522387, "host": "stream-logfwd20", "source": "testapp", "event": { "message": "Something happened", "severity": "INFO" } } ]Google Chronicle(JSON Array){ "customer_id": "xxxxx-xxxxx-xxxxx-xxxxx-xxxxx", "log_type": "ARCSIGHT_CEF", "entries": [ { "log_text": "CEF:0|palo alto networks|LF|2.0|DNS|realtime_dns_query|3|ProfileToken=hello-cef dtz=UTC deviceExternalID=xxxxx rt=Jun 09 2022 18:08:31 startTime=Dec 09 1654 PanOSRecordType=null PanOSCloudDNSClientIP= PanOSDNSResolverIP=9.9.9.9 PanOSThreatID=109010001 PanOSDNSCategory=phishing cat=Phishing:blockchain.ppckite.com src= cs4= act= cs5= duser= DestinationDNSDomain=" }, { "log_text": "CEF:0|palo alto networks|LF|2.0|DNS|realtime_dns_query|3|ProfileToken=hello-cef dtz=UTC deviceExternalID=xxxxx rt=Jun 09 2022 18:08:31 startTime=Dec 09 1654 PanOSRecordType=null PanOSCloudDNSClientIP= PanOSDNSResolverIP=9.9.9.9 PanOSThreatID=109010001 PanOSDNSCategory=phishing cat=Phishing:blockchain.ppckite.com src= cs4= act= cs5= duser= DestinationDNSDomain=" }, { "log_text": "CEF:0|palo alto networks|LF|2.0|DNS|realtime_dns_query|3|ProfileToken=hello-cef dtz=UTC deviceExternalID=xxxxx rt=Jun 09 2022 18:08:31 startTime=Dec 09 1654 PanOSRecordType=null PanOSCloudDNSClientIP= PanOSDNSResolverIP=9.9.9.9 PanOSThreatID=109010001 PanOSDNSCategory=phishing cat=Phishing:blockchain.ppckite.com src= cs4= act= cs5= duser= DestinationDNSDomain=" } ] }For more information about HTTPS log format, see the Log Forwarding Schema Reference.
- Accepts and decompresses GZIP HTTPS payloads. Cortex Data Lake compresses the JSON data using GZIP when forwarding through HTTPS.Microsoft Sentinel does not accept GZIP, so you must deploy a web app to decompress the data.
- Accepts a batch size of 500 logs, or 2.25 MB (500 logs x 4500B/log = 2.25MB). This does not apply to Google Chronicle, which accepts only 1 MB of data at a time, or a batch size of 250 logs.
For
each instance of Cortex Data Lake, you can forward logs to up to
200 destinations.
Cortex Data Lake communicates with the receiver using
TLS 1.2 and Java 8 default cipher suites (except GCM
ciphers, which are not currently supported). Upon connection Cortex Data Lake
validates that the receiver has a certificate signed by a trusted root CA or a
private CA. To complete the TLS handshake and establish the connection, the receiver
must present all the certificates from the chain of trust.
If
the connection with the HTTPS server fails,
Cortex
Data Lake
will
wait 60 seconds and retry. If the connection times out, Cortex
Data Lake
waits 20 seconds and drops the connection
with the server before establishing a new one.- Enable communication between Cortex Data Lake and your HTTPS receiver.Ensure that your HTTPS receiver can connect to Cortex Data Lake and can present a valid certificate to complete the connection request.
- Allow an inbound TLS feed to your HTTPS receiver from the IP address range for yourCortex Data Lakeregion.
- Obtain a certificate from a well-known, public CA.Because Cortex Data Lake validates the server certificate to establish a connection, you must verify that the receiver is configured to properly send the TLS certificate chain to Cortex Data Lake. If the app cannot verify that the certificate of the receiver and all CAs in the chain are trustworthy, the connection cannot be established. See the list of trusted certificates.
- Sign Into the hub at https://apps.paloaltonetworks.com/.
- Select the Cortex Data Lake instance that you want to configure for HTTPS forwarding.If you have multiple Cortex Data Lake instances, click the Cortex Data Lake tile and select an instance from the list of those available.
- Selectto add a new HTTPS forwarding profile.Log ForwardingAdd
- Enter a descriptiveNamefor the profile.
- Enter theURLfor the HTTPS receiver.The URL entered must begin withhttps. At the end of the top-level domain, you can specify a port number. The default port is 443.For Splunk HEC, ensure that the URL ends with/event.If you are forwarding logs to Google Chronicle, you must enter the URL that corresponds to yourCortex Data Lakeregion:UShttps://malachiteingestion-pa.googleapis.com/v2/unstructuredlogentries:batchCreateEUhttps://europe-malachiteingestion-pa.googleapis.com/v2/unstructuredlogentries:batchCreateAsiahttps://asia-southeast1-malachiteingestion-pa.googleapis.com/v2/unstructuredlogentries:batchCreate
- Configure Client Authorization.Do this if you are forwarding logs to a cloud HTTPS receiver, or if your HTTPS server requires a username and password.
- Select theTypeof client authorization and enter the necessary credentials.HTTPS ReceiverTypeCredentialsGoogle ChronicleChronicle Authorization
- Service AccountThis should be a properly formatted JSON document. If you don’t know your Service Account credentials, contact Google Chronicle support.
HTTPS Server (Password-protected)Basic Authorization- Username
- Password
HTTPS ServerNoneNone
- Test Connectionto ensure that Cortex Data Lake can communicate with the receiver.This sends an empty log to the configured destination to verify that transmission is possible.If the test fails, you will not be able to proceed.
- ClickNext.
- (Optional) To receive aSTATUS NOTIFICATIONwhen Cortex Data Lake is unable to connect to the HTTPS receiver, enter the email address at which you’d like to receive the notification.You will continue to receive these notifications every 60 minutes until connectivity is restored. If the connectivity issue is addressed within 72 hours, no logs will be lost. However, any log older than 72 hours following the service disconnection could be lost.
- Enter a unique PROFILE TOKEN if your receiver needs to distinguish logs coming from different tenants.
- Select the logs you want to forward.
- Adda new log filter.
- Select the log type.The Threat log type does not include URL logs or Data logs. If you wish to forward these log types, you must add them individually.
- (Optional)Create a log filter to forward only the logs that are most critical to you.You can either write your own queries from scratch or use the query builder. You can also select the query field to choose from among a set of common predefined queries.Log filters function like queries in Explore, with the following differences:
- Double quotes (“”) are not supported.
- Subnet masks are not supported in log forwarding filters. To return IP addresses with subnets, use theLIKEoperator. Example:src_ip.value LIKE “192.1.1.%”.
If you want to forward all logs of the type you selected, do not enter a query. Instead, proceed to the next step. - Saveyour changes.
- Saveyour changes.
- Verify that theStatusof your HTTPS forwarding profile isRunning(
).
Immediately after creating or editing your profile, theStatusmay beProvisioningfor up to 10 minutes. - Verify that you can view logs on the HTTPS receiver.For Splunk Common Information Model (CIM) fields and Enterprise Security, follow the guide at https://splunk.paloaltonetworks.com to install the PANW Splunk Addon and create a Splunk HEC input.
Most Popular
Recommended For You
Recommended Videos
Recommended videos not found.