Syslog and stdout integration
You can configure Prisma Cloud to send audit event records (audits) to syslog and/or stdout for Console and Defender based on whether you have Prisma Cloud Compute Edition or Prisma Cloud Enterprise Edition.
For the Prisma Cloud Enterprise Edition, we operate and monitor the Console for you. Therefore, the Console does not generate syslog events that you can reference.
The Defenders generate syslog messages that you can ingest for runtime and firewall events. In addition, you can configure Prisma Cloud to append a custom string to all Defender syslog messages.
Sending syslog messages to a network endpoint
Writing to /dev/log sends logs to the local host’s syslog daemon. The syslog daemon can then be optionally configured to forward those logs to a remote syslog or SIEM server. If you don’t have access to the underlying host, you can configure Prisma Cloud Console to send log messages directly to your remote system.
In most cases, you won’t need to specify a network endpoint in order to send syslog messages to your SIEM tool. If you already have log collectors on your hosts, simply enable syslog. Your log collectors will stream Prisma Cloud syslog messages to your SIEM tool.
Some things to keep in mind:
- Console sends logs directly to your remote server. When configuring Console with the remote server, validate that the address you enter is actually reachable from the host where Console runs. Otherwise, you risk losing log messages.
- Because Console sends messages directly to your remote server, and not through the local syslog daemon, you don’t get some of syslog’s built-in benefits, such as buffering, which protects against network outages and service failures.
- The classic syslog implementation sends logs over UDP. This is considered a bad practice if your logs have any value. UDP is connectionless. Packets are sent to their destination without confirming that they were received. TCP’s stateful connections and retransmission capabilities make it more appropriate for shuttling logs to a SIEM.
- Log into Console.
- Go toManage > Alerts > Logging.
- InSend syslog messages over the network to, clickEdit, and then specify a destination.
Appending custom strings to syslog messages
You can configure Prisma Cloud Compute to append a custom string to all Defender syslog messages.
Custom strings are set in the event message as a key-value pair, where the key is "id", and the value is your custom string. The following screenshot shows a Defender event, where the custom string is "koko".
Configuring a custom string is useful when you have multiple Prisma Cloud Compute deployments (i.e. multiple Compute Consoles) and you’re aggregating all messages in a single log management system. The custom string serves as a marker that lets you correlate specific events to specific deployments.
- Open Console.
- Go toManage > Alerts > Logging.
- ForIdentifier, clickEdit, and enter a string.
Both Console and Defender emit messages. Console syslog messages are tagged as Twistlock-Console in the logs.
The data emitted to syslog and stdout is exactly the same.
Console syslog event types
The following table describes each message type and sub-type.
This represents an image scan.
This represents any Compliance findings within the image scan.
This represents any Vulnerability findings within the image scan.
This represents a Container scan.
This represents any Compliance findings within the container scan.
This represents a VM scan.
This represents any Compliance findings within the vm scan.
This represents any Vulnerability findings within the vm scan.
This represents a Host scan.
This represents any Compliance findings within the host scan.
This represents any Vulnerability findings within the host scan.
This represents a scan summary. The type of summary is dependent upon subtype below.
This represents a summary of image Vulnerability and Compliance issues.
This represents a summary of container Vulnerability and Compliance issues.
This represents a summary of vm Vulnerability and Compliance issues.
This represents a summary of host Vulnerability and Compliance issues.
This represents a summary of code repository Vulnerability and Compliance issues.
This represents a summary of registry Vulnerability and Compliance issues.
This represents a summary of cloud accounts with Compute Compliance issues.
This represents any management audit. This is broken out in the subtypes listed below.
This represents a login audit.
This represents a profile state change audit.
This represents a settings change audit.
This represents a rule change audit.
This represents a user change audit.
This represents a group change audit.
This represents a credential change audit.
This represents a tag change audit.
This represents a Kubernetes audit.
This represents an Admission Controller audit.
This represents a Serverless runtime audit.
This represents a Serverless WAAS audit.
This represents an app embedded runtime audit.
This represents an app embedded WAAS audit.
This represents when a Defender is disconnected.
Records when Prisma Cloud scans an image.
Example image scan message:
Jul 30 18:51:32 aqsa-root Twistlock-Console: time="2019-07-30T18:51:32.214136319Z" type="scan_summary" log_type="image" image_id="sha256:cd14cecfdb3a657ba7d05bea026e7ac8b9abafc6e5c66253ab327c7211fa6281" image_name="aqsa/internal:tag5" vulnerabilities="297" compliance="1"
Records when Prisma Cloud scans a container.
Example container scan message:
Jul 30 22:06:15 aqsa-root Twistlock-Console: time="2019-07-30T22:06:15.804842461Z" type="container_scan" log_type="container" container_id="d29ac3222f430ccf6a7d730db5cec3363d4c608680de881e26e13f9011e36d13" container_name="twistlock_console" image_name="twistlock/private:console_19_07_353" compliance="6"
Records when Prisma Cloud scans a host. Defenders scan the hosts they run on.
Example host scan:
Jul 30 22:09:53 aqsa-root Twistlock-Console: time="2019-07-30T22:09:53.390680962Z" type="scan_summary" log_type="host" hostname="aqsa-root.c.cto-sandbox.internal" vulnerabilities="89" compliance="17"
Code repository scan
Records when Prisma Cloud scans a code repository.
Jul 7 23:34:09 ip-172-31-55-106 Twistlock-Console: time="2020-07-07T23:34:09.25109843Z" type="scan_summary" last_update_time="2020-07-07 23:21:00.203 +0000 UTC" log_type="code_repository_scan" source="github" repository_name="jerryso/apper" vulnerable_files="1" vulnerabilities="25" collections="All"
Changes to any settings (including previous and new values), changes to any rules (create, modify, or delete), and all logon activity (success and failure) are logged. For every event, both the user name and source IP are captured.
Example admin acitivty audit:
Jul 30 21:58:16 aqsa-root Twistlock-Console: time="2019-07-30T21:58:16.80522678Z" type="management_audit" log_type="login" username="aqsa" source_ip="188.8.131.52" api="/api/v1/authenticate" status="successful login attempt"
Defender syslog messages are tagged as Twistlock-Defender in logs. The data emitted to syslog and stdout is exactly the same.
App-embedded, Serverless, and Windows Defenders do not support Syslog.
Defender syslog event types
The following table describes each event type and sub-type.
This represents a Container Runtime Audit. Details of Audit type is listed as subtype below.
This represents a Container process runtime audit.
This represents a Container network runtime audit.
This represents a Container filesystem runtime audit.
This represents a Host activity audit.
This represents a Host WAAS audit.
This represents a Container WAAS audit.
This represents a Host Runtime Audit. Each audit type is listed as subtype below.
This represents a Host process runtime audit.
This represents a Host network runtime audit.
This represents a Host Kubernetes runtime audit.
This represents a Host filesystem runtime audit.
This represents an Incident. Host and Container incidents are differentiated by "host" or "container_id".
Container runtime audit
Activity that breaches your runtime rules or the automatically generated allow lists in your models generates audits. The log_type field specifies the runtime sensor that detected the anomaly (filesystem, processes. syscalls, or network).
Example container runtime audit: The following process audit shows that busybox was unexpectedly launched, and an alert was raised.
Jul 30 22:41:25 aqsa-root Twistlock-Defender: time="2019-07-30T22:41:25.448709847Z" type="container_runtime_audit" container_id="73c2e8267f9b80ea152403c36c377476d24e43e211bb098300a317b3d1c472e4" container_name="/dreamy_rosalind" image_id="sha256:94e814e2efa8845d95b2112d54497fbad173e45121ce9255b93401392f538499" image_name="ubuntu:18.04" effect="alert" msg="High rate of reg file access events, reporting aggregation started; last event: /usr/lib/apt/methods/gpgv wrote a suspicious file to /tmp/apt.conf.2ZH7tP. Command: /usr/lib/apt/methods/gpgv" log_type="filesystem" custom_labels="io.kubernetes.pod.namespace:default" account_id="prisma-cloud-compute" cluster="cluster1"
Host runtime audit
Activity that breaches your runtime rules or the automatically generated allow lists in your host services models generates audits.
Example host runtime audit:
Jul 30 22:47:12 aqsa-root Twistlock-Defender: time="2019-07-30T22:47:12.325487039Z" type="host_runtime_audit" service_name="ssh" effect="alert" msg="Outbound connection by /usr/lib/apt/methods/http to an unexpected port: 80 IP: 184.108.40.206. Low severity audit, event is automatically added to the runtime model" log_type="network" account_id="prisma-cloud-compute" cluster="cluster1"
Docker commands run on hosts protected by Defender.
With user access events, you can determine who performed an action, and on which resource.
- [Bruce] [started container X] in the [DEV environment] (allowed).
- [Bruce] [stopped container Y] in the [PROD environment] (denied).
All Docker commands issued to the Docker daemon are intercepted and inspected by Defender to determine if they comply with the policy set in Console.
The following diagram illustrates how Defender operates on the management plane:
- Bruce, a developer, issues a command, docker -H.
- Defender checks the command against the policies defined in the Console. If the command is allowed, Defender forwards it to the Docker daemon for execution. If the command is denied, the user is notified.
- An event is recorded in syslog.
Access audits have the following fields:
- user=[String] Identity of the person who ran the command
- action=[String] Docker command requested - API invoked
- action_type=[String] Action type
- allow=[Boolean] true/false - Action was allowed or not.
- rule=[String] Rule matched
Jul 30 23:02:23 aqsa-root Twistlock-Defender: time="2019-07-30T23:02:23.179494498Z" type="access_audit" user="aqsa" action="docker_ping" action_type="docker" allow="true" rule="Default - allow all"
App firewall audit (WAAS)
All events associated with WAAS (Web-Application and API Security) rules for container, hosts and app-embedded generate audits.
WAAS serverless events are not registered in the syslog. Events audits will be registered to the syslog in future releases.
WAAS Container and Host rule audits are written to the Defender host’s syslog. WAAS App-Embedded rule audits are written to the console’s host’s syslog.
Message fields for WAAS audit would change based on the deployment type as follows:
- container_id=[String]Container id in which the event triggered
- container_name=[String]Container name on which the action was performed
- image_name=[String]Image name on which the action was performed
- custom_labels=[String]User-defined Alert Labels (Mange > Alerts > Alert Labels)
- cluster=[String]Cluster name in which the event triggered
- hostname=[String]host in which the event triggered
- cluster=[String]Cluster name in which the event triggered
- app_id=[String]app_id in which the event triggered
- time=[String]request timestamp
- type=[String]type of app_firewall_audit
- effect=[String]"alert", "prevent", "ban"
- msg=[String]Audit message detailing the event
- log_type=[String]Attack Type
- source_ip=[String]source IP address from the request originated
- source_country=[String]country associated with source IP address
- request_method=[String]HTTP Request Method
- request_host=[String]HTTP hostname in the request
- request_url=[String]request url
- request_path=[String]request path
- request_query=[String]request query string
- request_header_names=[String]ordered list of HTTP request headers
- response_header_names=[String]ordered list of HTTP response headers
- status_code=[String]HTTP response status code in the server response
In addition, message structure is subject for the following changes:
- Fields containing empty values are omitted from the message i.e. if a HTTP message does not contain a query field the request_query field will not be present in the message.
- status_code- present only for audits created for the "Track Server Error Response Codes" and "Detect Information Leakage" protections
- response_header_names- present only for audits created for the "Track Server Error Response Codes" and "Detect Information Leakage" protections.
- source_country- present only if resolution was successful.
- container_name- will be replaced byhost_idorfunction_id
Jul 16 20:10:16 cnaf-nightly-build Twistlock-Defender: time="2020-07-16T20:10:16.706085135Z" type="container_app_firewall_audit" container_id="0a16b4e4dbefc6ef8cc6a08d038e775a8523ad053416730f01eafbf2dee2e693" container_name="/nginx" image_name="nginx:latest" effect="prevent" msg="Client exceeded violations within 1m. Banning client for 5m" log_type="violations exceeded" source_ip="220.127.116.11" source_country="IL" connecting_ips="18.104.22.168" request_method="HEAD" request_user_agents="curl/7.54.0" request_host="www.example.com" request_url="www.example.com/?id=../etc/passwd" request_path="/" request_query="id=../etc/passwd" request_header_names="X-Forwarded-For,User-Agent,Accept" response_header_names="Set-Cookie,Date,Content-Type,Content-Length X-Frame-Options" status_code="404"
Process activity audit
Records all processes spawned in a container.
Process audits are only recorded when
Detailed output of all runtime process activityis enabled in
Manage > Alerts > Logging.
Note that process activity that breaches your runtime policy is separately audited. For more infomration, see the container runtime audit section.
This audit has the following fields:
- pid=Process ID
- path=Path to the executable in the container file system
- interactive=Whether the process was spawned from a shell session: true or false
- container-id=Container ID
Example: This audit shows that busybox was spawned in the container with ID 8c5b3fe0037d.
Jul 30 22:06:03 aqsa-root Twistlock-Defender: time="2019-07-30T22:06:03.515319204Z" type="process" pid="20859" path="/bin/df" interactive="false" container_id="3491b03544a51c60e176e54a5077161f14dbc850bf069cf7a096db028e9981de"
Incidents are logical groupings of events, related by context, that reveal known attack patterns.
Example container incident:
Jul 30 22:41:24 aqsa-root Twistlock-Defender: time="2019-07-30T22:41:24.987209676Z" type="incident" container_id="73c2e8267f9b80ea152403c36c377476d24e43e211bb098300a317b3d1c472e4" image_name="ubuntu:18.04" host="aqsa-root.c.cto-sandbox.internal" incident_category="hijackedProcess" custom_labels="io.kubernetes.pod.namespace:default" account_id="prisma-cloud-compute" cluster="cluster1"
Example host incident:
Mar 5 00:26:42 itay-ThinkPad-P50 Twistlock-Defender: time="2018-03-05T00:26:42.894707831+02:00" type="incident" service_name="http-service" host="itay-ThinkPad-P50" incident_category="serviceViolation" audit_ids="5a9c72a223d020590de74db5" account_id="prisma-cloud-compute" cluster="cluster1"
Depending on your configuration, Prisma Cloud can produce a lot of logs, especially in environments with many hosts, images, and containers. By default, most syslog daemons throttle logging with a rate limiter.
If you have a large environment (hundreds of Defenders with tens of images per host) AND you have configured Prisma Cloud for verbose syslog output, you will need to tune the rate limiter. Otherwise, you might find that logs are missing.
For example, on RHEL 7, you must tune both systemd-journald’s RateLimitInterval and RateLimitBurst settings and rsyslog’s imjournalRatelimitInterval and imjournalRatelimitBurst settings. For more information about RedHat settings, see How to disable log rate-limiting in Red Hat Enterprise Linux 7.
Truncated log messages
Very long syslog events can get truncated. For example, changing settings in Console generates management_audits events, which show a diff between old settings and new settings. For policies changes, the diff can be big. Linux log managers limit the number of characters logged per line, and so long messages, such as management audits, can be truncated.
If you’ve got truncated log messages, increase the log manager’s default string size limit. There are several types log managers, but rsyslog is popular with most distributions. For rsyslog, the default log string size is 1024 characters per line. To increase it, open /etc/rsyslog.conf and set the maximum message size: