Runtime defense for containers
Runtime defense is the set of features that provide both predictive and threat-based active protection for running containers.
For example, predictive protection includes capabilities like determining when a container runs a process not included in the origin image or creates an unexpected network socket.
Threat-based protection includes capabilities like detecting when malware is added to a container or when a container connects to a botnet.
Prisma Cloud Compute has distinct sensors for file system, network, and process activity.
Each sensor is implemented individually, with its own set of rules and alerting.
The runtime defense architecture is unified to both simplify the administrator experience and to show more detail about what Prisma Cloud automatically learns from each image.
Runtime defense has two principle object types: models and rules.
Container Models
Models are the results of the autonomous learning that Prisma Cloud performs every time we see a new image in an environment.
A model is the ‘allow list’ for what a given container image should be doing, across all runtime sensors.
Models are automatically created and maintained by Prisma Cloud and provide an easy way for administrators to view and understand what Prisma Cloud has learned about their images.
For example, a model for an Apache image would detail the specific processes that should run within containers derived from the image and what network sockets should be exposed.
Navigate to
Monitor > Runtime > Container Models
.
Click on the image to view the model.There is a 1:1 relationship between models and images; every image has a model and every model applies to a single unique image.
For each image, a unique model is created and mapped to the image digest.
So, even if there are multiple images with the same tags, Prisma Cloud creates a unique model for each image.
Models are built from both static analysis, such as building a hashed process map based on parsing an init script in a Dockerfile ENTRYPOINT), and from dynamic behavioral analysis, such as observing actual process activity during early runtime of the container.
Models can be in one of 3 modes: Active, Archived, or Learning.
For containers in Kubernetes clusters, Prisma Cloud considers the image, namespace, cluster, and deployment (YAML) file when creating models.
- When the same image runs in multiple different clusters, Prisma Cloud creates separate models for each image in each cluster.
- When the same image runs in multiple different namespaces, Prisma Cloud creates separate models for each image in each namespace.
- When there are multiple running instances of an image in the same namespace, Prisma Cloud creates a single model.
- When there are multiple running instances of an image in the same namespace, but started from different deployment (YAML) files, Prisma Cloud creates multiple container models, one for each deployment.
Prisma Cloud shows you how models map to specific images.
Go to
Monitor > Runtime > Container Models
, click a model in the table, and click the General
tab.
Capabilities
Some containers are difficult to model.
For example, Jenkins containers dynamically build and run numerous processes, and the profile of those processes changes depending on what’s being built.
Constructing accurate models to monitor processeses in containers that build, run, test, and deploy software is impractical, although other aspects of the model can still have utility.
Prisma Cloud automatically detects known containers, and overrides one more aspects of the model with capabilities.
Capabilities are discrete enhancements to the model that tune runtime behaviors for specific apps and configurations.
Rather than changing what’s learned in the model, they modify how Prisma Cloud acts on observed behaviors.
For example, the following model for the Jenkins container is enhanced with the capability for writing and executing binaries.
Learning mode
Learning mode is the phase in which Prisma Cloud performs either static or dynamic analysis.
Because the model depends on behavioral inputs, images stay in learning mode for 1 hour to complete the model.
After this 1 hour, Prisma Cloud enters a 'dry run' period for 24 hours to ensure there are no behavioral changes and the model is complete.
If during this 24 hours period, behavioral changes are observed, the model goes back to Learning mode for additional 24 hours.
The behavioral model uses a combination of machine learning techniques and typically requires less than 1 hour of cumulative observation time for a given image (it might comprise of a single container running the entire learning period or multiple containers running for some time slice where the sum of the slices is 1 hour).
During this period, only threat based runtime events (malicious files or connections to high risk IPs) are logged.
Prisma Cloud automatically detects when new images are added anywhere in the environment and automatically puts them in learning mode.
- Extend Learning: You can relearn an existing model by clicking theExtend Learningoption in theActionsmenu. This is an additive process, and any existing static and behavioral modeling remains in place.
- Manual Relearning: You can manually alter the duration of learning at any time by starting and stopping theManual Relearningoption in theActionsmenu. This should be done with discretion because the model may or may not complete within the time period due to manual interruption. There is no time limit for manual relearning and depends on user’s choice.
Active mode
Active mode is the phase in which Prisma Cloud is actively enforcing the model and looking for anomalies that violate it.
Active mode begins after the initial 1 hour that the Learning mode takes to create a model.
Because models are explicit allow lists, in enforcing mode, Prisma Cloud is simply looking for variances against the model.
For example, if a model predicted that a given image should only run the foo process and Prisma Cloud observes the bar process has spawned, it would be an anomaly.
Prisma Cloud automatically transitions models from learning mode into enforcing mode after the model is complete.
During this period, runtime events are logged.
During the initial dry run period (the first 24 hours), model may switch automatically from Active mode to Learning mode depending on the behavioral changes observed, as mentioned above.
This automatic switching only happens during the first 24 hours of model initiation. If violations are observed later on, they are logged as runtime alerts under Monitor > Runtime.
Archived mode
Archived mode is a phase that models are transitioned into when a container is no longer actively running them.
Models persist in archived mode for 24 hours after being archived, after which point they’re automatically removed.
Archived mode serves as a 'recycle bin' for models, ensuring that a given image does not need go through learning mode again if it frequently starts and stops while also ensuring that the list of models does not continuously grow over time.
Models display all the learned data across each of the runtime sensors to make it easy to understand exactly what Prisma Cloud has learned about an image and how it will protect it.
However, what if you need to customize the protection for a given image, set of images, or containers?
That’s the job of rules.
Rules
Rules control how Prisma Cloud uses the autonomously generated models to protect an environment.
For example, if Prisma Cloud’s model for the Apache image includes the process httpd, but you know that process bar will eventually run and you want to ensure that process foo never runs, you can create a rule that applies to all images named httpd, add bar to the allowed process list, and add foo to the blocked process list.
The following screenshot shows how the scope of the rule is set with collections:
The following screenshot shows how allowed and blocked process activity is set in the rule:
Rules let you explicitly allow and block activity by sensor.
Rules and models are evaluated together to create a resultant policy as follows:
model
(which contains only allowed activity) + allowed activity from rule(s)
- blocked activity from rule(s)
= resultant policy
The resultant policy from the previous example:
model (
httpd
) + allowed activity from rule (process bar
) - blocked activity from rule (process foo
) = httpd and bar are allowed and foo always is an anomaly regardless of the modelBy default, Prisma Cloud ships with an empty container runtime policy.
An empty policy disables runtime defense entirely.
To enable runtime defense, create a rule.
New runtime rules can be created in Console in
Defend > Runtime > Container policy
.As with every other subsystem in Prisma Cloud, you can customize how it works by creating rules, scoping rules to desired objects with filtering and pattern matching, and properly ordering the rules in the policy.
Rules are evaluated sequentially from top to bottom.
Once a match is found for the scope, the actions in the rule are executed and enforced.
Only a single rule is ever enforced for a given event.
While rules work in conjunction with models as described above, rules themselves are never combined.
Refine your policy by creating rules that target specific resources, enabling or disabling protection features, and defining exceptions to the automatically generated allow-list models.
Discrete blocking
Prisma Cloud lets you create runtime rules that block discrete processes inside a container.
It is an alternative to stopping an entire container when the violation of a runtime rule is detected.
Blocked containers
Prisma Cloud’s runtime defense system compares the state of a running container to the predictive model created for it during its learning period.
When abnormal activity is detected, such as executing an unknown process, Prisma Cloud can:
- Raise an alert by generating an audit. Audits are shown underMonitor > Events > Container Audits. If you have an alert channel configured, such as email or Slack, audits are forwarded there too. Alert is the default action for new runtime rules.
- Block the container by stopping it altogether. To enable blocking, create a new runtime rule.
- Prevent just the discrete process or file system write (not the entire container).
Blocking action
Blocking stops potentially compromised containers from running in your environment.
Prisma Cloud blocks containers under the following conditions:
- A container violates its runtime model, and you’ve installed a runtime rule with the action set to block. For example, if an attacker infiltrates a container and tries to run a port scan using nc, then the container would be blocked if nc weren’t a known, allowed process.
- A newly started container violates a vulnerability or compliance rule, and those rules have the action set to block. Prisma Cloud scans all images before they run, to enforce policies about what’s allowed to execute in your environment. For example, you policy might call for blocking any container with critical severity vulnerabilities.
Runtime rules can be created under Defend > Runtime > Container Policy.
Vulnerability rules can be created under Defend > Vulnerabilities > Policy, and compliance rules can be created under Defend > Compliance > Policy.
Viewing blocked containers
Blocking immediately stops a container, taking it out of service.
Blocked containers are never restarted.
To see a list of blocked containers, go to the container audits page under
Monitor > Events > Container Audits
.
When a container is stopped, Prisma Cloud takes no further action to keep it stopped.
Orchestrators, such as Kubernetes and Docker Swarm, start a fresh container in the blocked container’s place.
Orchestrators have their own mechanism for maintaining a set point, so they ignore the restart policy defined in the image’s Dockerfile.
There is an exception when you run containers in a Docker-only environment (no orchestrator) and Prisma Cloud blocks a container.
In this case, Prisma Cloud must take additional action to keep the container blocked.
To prevent the container from automatically restarting, Prisma Cloud modifies the container’s restart policy to always unless stopped.
If you want to unblock a container, connect to the node with the blocked container, and manually modify the container’s Docker configuration.
Blocked container artifacts
Forensic investigators can inspect a blocked container’s artifacts to determine why it was stopped.
You can capture all of the container’s contents, including its file system data, with the docker export command.
Go to the node with the blocked container and run: