Forward Logs from Cortex Data Lake to an HTTPS Server
Table of Contents
Expand all | Collapse all
-
- Cortex Data Lake for Panorama-Managed Firewalls
- Start Sending Logs to a New Cortex Data Lake Instance
- Configure Panorama in High Availability for Cortex Data Lake
- Allocate Storage Based on Log Type
- View Cortex Data Lake Status
- View Logs in Cortex Data Lake
- TCP Ports and FQDNs Required for Cortex Data Lake
- Sizing for Cortex Data Lake Storage
-
- Forward Logs from Cortex Data Lake to a Syslog Server
- Forward Logs from Cortex Data Lake to an HTTPS Server
- Forward Logs from Cortex Data Lake to an Email Server
- Log Record Formats
- Create Log Filters
- Server Certificate Validation
- List of Trusted Certificates for Syslog and HTTPS Forwarding
- Log Forwarding Errors
Forward Logs from Cortex Data Lake to an HTTPS Server
Learn how to forward logs from Cortex Data Lake to an
HTTPS server.
To meet your long-term storage, reporting
and monitoring, or legal and compliance needs, you can configure
Cortex Data Lake to forward logs to an HTTPS server or to the following
SIEMs:
- Exabeam
- Google Chronicle
- Microsoft Sentinel
- Splunk HTTP Event Collector (HEC)
For successful
log transmission, ensure that your HTTPS receiver:
- Accepts and parses the correct log format. The format in whichCortex Data Lakeforwards logs depends on the HTTPS receiver:Google Chronicle(JSON Array){ "customer_id": "xxxxx-xxxxx-xxxxx-xxxxx-xxxxx", "log_type": "ARCSIGHT_CEF", "entries": [ { "log_text": "CEF:0|palo alto networks|LF|2.0|DNS|realtime_dns_query|3|ProfileToken=hello-cef dtz=UTC deviceExternalID=xxxxx rt=Jun 09 2022 18:08:31 startTime=Dec 09 1654 PanOSRecordType=null PanOSCloudDNSClientIP= PanOSDNSResolverIP=9.9.9.9 PanOSThreatID=109010001 PanOSDNSCategory=phishing cat=Phishing:blockchain.ppckite.com src= cs4= act= cs5= duser= DestinationDNSDomain=" }, { "log_text": "CEF:0|palo alto networks|LF|2.0|DNS|realtime_dns_query|3|ProfileToken=hello-cef dtz=UTC deviceExternalID=xxxxx rt=Jun 09 2022 18:08:31 startTime=Dec 09 1654 PanOSRecordType=null PanOSCloudDNSClientIP= PanOSDNSResolverIP=9.9.9.9 PanOSThreatID=109010001 PanOSDNSCategory=phishing cat=Phishing:blockchain.ppckite.com src= cs4= act= cs5= duser= DestinationDNSDomain=" }, { "log_text": "CEF:0|palo alto networks|LF|2.0|DNS|realtime_dns_query|3|ProfileToken=hello-cef dtz=UTC deviceExternalID=xxxxx rt=Jun 09 2022 18:08:31 startTime=Dec 09 1654 PanOSRecordType=null PanOSCloudDNSClientIP= PanOSDNSResolverIP=9.9.9.9 PanOSThreatID=109010001 PanOSDNSCategory=phishing cat=Phishing:blockchain.ppckite.com src= cs4= act= cs5= duser= DestinationDNSDomain=" } ] }Microsoft Sentinel(JSON Array)[ { "time": 1437522387, "host": "stream-logfwd20", "source": "Palo Alto Networks FLS LF", "event": { "message": "Something happened", "severity": "INFO" } }, { "time": 1437522387, "host": "stream-logfwd20", "source": "testapp", "event": { "message": "Something happened", "severity": "INFO" } } ]Splunk HEC(Stacked JSON){ "time": 1437522387, "host": "stream-logfwd20", "source": "Palo Alto Networks FLS LF", "event": { "LogType": "THREAT", "Severity": "Critical", ... } } { "time": 1437522387, "host": "stream-logfwd20", "source": "Palo Alto Networks FLS LF", "event": { "LogType": "THREAT", "Severity": "Info", ... } }Exabeam(JSON Array){ "DestinationDeviceCategory": "N-Phone", "DestinationDeviceOSFamily": "H1511", "DestinationDeviceOSVersion": "Android v7", "SourceDeviceOS": null, "NATDestination": "xxx.xx.x.xxx", "ApplicationSubcategory": "database", "VendorName": "Palo Alto Networks", "Protocol": "tcp", "IsServertoClient": false, "PacketID": 0, "NSSAINetworkSliceType": "fc", "DirectionOfAttack": "client to server", "EndpointSerialNumber": "SG0000001", "ApplicationTechnology": "network-protocol", "VendorSeverity": "Critical", "SubType": "data", "DeviceSN": "xxxxxxxxxx", "ConfigVersion": "10.2", "IsMptcpOn": false, "SourceDeviceModel": "Nexus", "InboundInterface": "ethernet1/1", "LogExported": false, "ParentSessionID": 0, "CloudHostname": "PA-VM-E2E-PCL-TEST", "SourcePort": 25195, "CaptivePortal": false, "LogSource": "firewall", "LogType": "THREAT", "InboundInterfaceDetailsType": "ethernet", "Severity": "Critical", "TimeReceived": "2023-07-22T08:49:30.000000Z", "SourceUserDomain": "paloaltonetwork", "IsDuplicateLog": false, }For more information about HTTPS log format, see the Log Forwarding Schema Reference.
- Accepts and decompresses GZIP HTTPS payloads. Cortex Data Lake compresses the JSON data using GZIP when forwarding through HTTPS.Microsoft Sentinel does not accept GZIP, so you must deploy a web app to decompress the data.
- Accepts a batch size of 500 logs, or 2.25 MB (500 logs x 4500 B = 2.25 MB). This does not apply to Google Chronicle, which accepts only 1 MB of data at a time, or a batch size of 250 logs.
For
each instance of Cortex Data Lake, you can forward logs to up to
200 destinations.
Cortex Data Lake communicates with the receiver using
TLS 1.2 and Java 8 default cipher suites (except GCM
ciphers, which CDL does not currently support). Upon connection, Cortex Data Lake
validates that the receiver has a certificate signed by a trusted root CA or a
private CA. To complete the TLS handshake and establish the connection, the receiver
must present all the certificates from the chain of trust.
If
the connection with the HTTPS server fails,
Cortex
Data Lake
will
wait 60 seconds and retry. If the connection times out, Cortex
Data Lake
waits 20 seconds and drops the connection
with the server before establishing a new one.- Enable communication between Cortex Data Lake and your HTTPS receiver.Ensure that your HTTPS receiver can connect to Cortex Data Lake and can present a valid certificate to complete the connection request.
- Allow an inbound TLS feed to your HTTPS receiver from the IP address range for yourCortex Data Lakeregion.
- Obtain a certificate from a well-known, public CA.Because Cortex Data Lake validates the server certificate to establish a connection, you must verify that you have configured the receiver to properly send the TLS certificate chain to Cortex Data Lake. If the app can't verify that the certificate of the receiver and all CAs in the chain are trustworthy, it can't establish a connection. See the list of trusted certificates.
- Sign Into the hub at https://apps.paloaltonetworks.com/.
- Select the Cortex Data Lake instance that you want to configure for HTTPS forwarding.If you have multiple Cortex Data Lake instances, click the Cortex Data Lake tile and select an instance from the list of those available.
- Selectto add a new HTTPS forwarding profile.Log ForwardingAdd
- Enter a descriptiveNamefor the profile.
- Enter theURLfor the HTTPS receiver.The URL entered must begin withhttps. At the end of the top-level domain, you can specify a port number. The default port is 443.For Splunk HEC, ensure that the URL ends with/event.If you're forwarding logs to Google Chronicle, you must enter the URL that corresponds to yourCortex Data Lakeregion:UShttps://malachiteingestion-pa.googleapis.com/v2/unstructuredlogentries:batchCreateEUhttps://europe-malachiteingestion-pa.googleapis.com/v2/unstructuredlogentries:batchCreateAsiahttps://asia-southeast1-malachiteingestion-pa.googleapis.com/v2/unstructuredlogentries:batchCreate
- Configure Client Authorization.Do this if you're forwarding logs to a cloud HTTPS receiver, or if your HTTPS server requires a username and password.
- Select theTypeof client authorization and enter the necessary credentials.HTTPS ReceiverTypeCredentialsHTTPS Server (Password-protected)Basic Authorization
- Username
- Password
Google ChronicleChronicle Authorization- Service AccountThis should be a properly formatted JSON document. If you don’t know your Service Account credentials, contact Google Chronicle support.
HTTPS ServerNoneNone
- Test Connectionto ensure that Cortex Data Lake can communicate with the receiver.This sends an empty log to the configured destination to verify that transmission is possible.If the test fails, you won't be able to proceed.
- ClickNext.
- (Optional) To receive aSTATUS NOTIFICATIONwhen Cortex Data Lake is unable to connect to the HTTPS receiver, enter the email address at which you’d like to receive the notification.You will continue to receive these notifications every 60 minutes until the service restores connectivity. If the connectivity issue resolves within 72 hours, the service won't lose logs. However, the service could lose any log older than 72 hours following the service disconnection.
- Enter a unique PROFILE TOKEN if your receiver needs to distinguish logs coming from different tenants.
- Select the logs you want to forward.
- Adda new log filter.
- Select the log type.The Threat log type does not include URL logs or Data logs. If you wish to forward these log types, you must add them individually.
- (Optional)Create a log filter to forward only the logs that are most critical to you.You can either write your own queries from scratch or use the query builder. You can also select the query field to choose from among a set of common predefined queries.Log filters function like queries in Explore, with the following differences:
- No double quotes (“”).
- No subnet masks. To return IP addresses with subnets, use theLIKEoperator. Example:src_ip.value LIKE “192.1.1.%”.
If you want to forward all logs of the type you selected, don't enter a query. Instead, proceed to the next step. - Saveyour changes.
- Saveyour changes.
- Verify that theStatusof your HTTPS forwarding profile isRunning(
).
Immediately after creating or editing your profile, theStatusmight beProvisioningfor up to 10 minutes. - Verify that you can view logs on the HTTPS receiver.For Splunk Common Information Model (CIM) fields and Enterprise Security, follow the guide at https://splunk.paloaltonetworks.com to install the Palo Alto Networks Splunk add-on and create a Splunk HEC input.