Runtime defense for containers

Runtime defense is the set of features that provide both predictive and threat-based active protection for running containers. For example, predictive protection includes capabilities like determining when a container runs a process not included in the original image or creates an unexpected network socket. Threat-based protection includes capabilities like detecting when malware is added to a container or when a container connects to a botnet.
Prisma Cloud Compute has distinct sensors for the filesystem, network, and process activity. Each sensor is implemented individually, with its own set of rules and alerts. The runtime defense architecture is unified to both simplify the administrator experience and to show more detail about what Prisma Cloud automatically learns from each image. Runtime defense has two principal object types: models and rules.

Container Models

Models are the results of the autonomous learning that Prisma Cloud performs every time we see a new image in an environment. A model is the "allow list" for what a given container image should be doing, across all runtime sensors. Models are automatically created and maintained by Prisma Cloud and provide an easy way for administrators to view and understand what Prisma Cloud has learned about their images. For example, a model for an Apache image would detail the specific processes that should run within containers derived from the image and what network sockets should be exposed.
Navigate to
Monitor > Runtime > Container Models
. Click on the image to view the model.
There is a 1:1 relationship between models and images; every image has a model and every model applies to a single unique image. For each image, a unique model is created and mapped to the image digest. So, even if there are multiple images with the same tags, Prisma Cloud will create unique models for each image.
Models are built from both static analysis (such as building a hashed process map based on parsing an init script in a Dockerfile ENTRYPOINT) and dynamic behavioral analysis (such as observing actual process activity during early runtime of the container). Models can be in one of 3 modes: Active, Archived, or Learning.
For containers in Kubernetes clusters, Prisma Cloud considers the image, namespace, and cluster when creating models.
  • When the same image runs in multiple different clusters, Prisma Cloud creates separate models for each image in each cluster.
  • When the same image runs in multiple different namespaces, Prisma Cloud creates separate models for each image in each namespace.
  • When there are multiple running instances of an image in the same namespace, Prisma Cloud creates a single model.
  • When clusters are not applicable, Prisma cloud considers the image and namespace to create models.
Prisma Cloud shows you how models map to specific images. Go to
Monitor > Runtime > Container Models
, click a model in the table, and click the
General
tab.

Capabilities

Some containers are difficult to model. For example, Jenkins containers dynamically build and run numerous processes, and the profile of those processes changes depending on what’s being built. Constructing accurate models to monitor processes in containers that build, run, test, and deploy software is impractical, although other aspects of the model can still have utility. Prisma Cloud automatically detects known containers and overrides one more aspect of the model with capabilities.
Capabilities are discrete enhancements to the model that tune runtime behaviours for specific apps and configurations. Rather than changing what’s learned in the model, they modify how Prisma Cloud acts on observed behaviours.
For example, the following model for the Jenkins container is enhanced with the capability for writing and executing binaries.

Learning mode

Learning mode is the phase in which Prisma Cloud performs either static or dynamic analysis. Because the model depends on behavioral inputs, images stay in learning mode for 1 hour to complete the model. After this 1 hour, Prisma Cloud enters a 'dry run' period for 24 hours to ensure there are no behavioral changes and the model is complete. If during these 24 hours, behavioral changes are observed, the model goes back to Learning mode for an additional 24 hours. The behavioral model uses a combination of machine learning techniques and typically requires less than 1 hour of cumulative observation time for a given image (it might comprise of a single container running the entire learning period or multiple containers running for some time slice where the sum of the slices is 1 hour). During this period, only threat-based runtime events (malicious files or connections to high-risk IPs) are logged. Prisma Cloud automatically detects when new images are added anywhere in the environment and automatically puts them in learning mode.
  • Relearn: You can relearn an existing model by clicking the
    Relearn
    button in the
    Actions
    menu. This is an additive process, so any existing static and behavioral modeling remain in place.
  • Manual Learning: You can manually alter the duration of learning at any time by starting and stopping the
    Manual Learning
    option in the
    Actions
    menu. This should be done with discretion because the model may or may not complete within the time-period due to manual interruption. There is no time limit for manual learning. It depends on the user’s selection.

Active mode

Active mode is the phase in which Prisma Cloud is actively enforcing the model and looking for anomalies that violate it. Active mode begins after the initial 1 hour that the learning mode takes to create a model. Because models are explicit allow lists, in enforcing mode, Prisma Cloud is simply looking for variances against the model. For example, if a model predicted that a given image should only run the foo process and Prisma Cloud observes the bar process has spawned, it would be an anomaly. Prisma Cloud automatically transitions models from learning mode into enforcing mode after the model is complete. During this period, runtime events are logged.
During the initial dry run period (the first 24 hours), model may switch automatically from Active mode to learning mode depending on the behavioral changes observed, as mentioned above. This automatic switching only happens during the first 24 hours of model initiation. If violations are observed later on, they are logged as runtime alerts under Monitor > Runtime.

Archived mode

Archived mode is a phase that models are transitioned into when a container is no longer actively running them. Models persist in an archived mode for 24 hours after being archived, after which point they’re automatically removed. Archived mode serves as a 'recycle bin' for models, ensuring that a given image does not need to go through learning mode again if it frequently starts and stops, while also ensuring that the list of models does not continuously grow over time.
Models display all the learned data across each of the runtime sensors to make it easy to understand exactly what Prisma Cloud has learned about an image and how it will protect it. However, what if you need to customize the protection for a given image, set of images, or containers? That’s the job of rules.

Rules

Rules control how Prisma Cloud uses autonomously generated models to protect the environment. For example, if Prisma Cloud’s model for the Apache image includes the process httpd, but you know that process bar will eventually run, and you want to ensure that process foo never runs, you can create a rule that applies to all images named httpd, add bar to the allowed process list, and add foo to the blocked process list.
The following screenshot shows how the scope of the rule is set with collections:
The Container runtime policy rules allow individual effects per protection, such as. anti-malware, crypto miners, reverse shell attacks, etc. for each section - Processes, Networking, File System, and Anti-malware. The effect includes the following options: Disabled/Alert/Prevent/Block according to the supported effects for each detection.
Rules let you explicitly allow/alert/prevent/block activity by a sensor. Rules and models are evaluated together to create a resultant policy as follows:
model
(which contains only allowed activity) +
allowed activity from rule(s)
-
blocked activity from rule(s)
=
resultant policy
The resultant policy from the previous example:
model (
httpd
) + allowed activity from rule (
process bar
) - blocked activity from rule (
process foo
) = httpd and bar are allowed and foo always is an anomaly regardless of the model
By default, Prisma Cloud ships with an empty container runtime policy. An empty policy disables runtime defense entirely. To enable runtime defense, create a rule. New runtime rules can be created in Console in
Defend > Runtime > Container policy
.
As with every other subsystem in Prisma Cloud, you can customize how it works by creating rules, scoping rules to desired objects with filtering and pattern matching, and properly ordering the rules in the policy. Rules are evaluated sequentially from top to bottom. Once a match is found for the scope, the actions in the rule are executed and enforced. Only a single rule is ever enforced for a given event. While rules work in conjunction with models as described above, rules themselves are never combined.
Refine your policy by creating rules that target specific resources, enabling or disabling protection features, and defining exceptions to the automatically generated allow-list models.

Discrete blocking

Prisma Cloud lets you create runtime rules that block discrete processes inside a container using the
Prevent
effect. It is an alternative to stopping an entire container when a violation of a runtime rule is detected.

Blocked containers

Prisma Cloud’s runtime defense system compares the state of a running container to the predictive model created for it during its learning period. When abnormal activity is detected, such as executing an unknown process, Prisma Cloud can:
  • Raise an alert by generating an audit. Audits are shown under
    Monitor > Events > Container Audits
    . If you have an alert channel configured, such as email or Slack, audits are forwarded there too. Alert is the default action for new runtime rules.
  • Block the container by stopping it altogether. To enable blocking, create a new runtime rule.
  • Prevent just the discrete process or file system write (not the entire container).

Blocking action

Blocking stops potentially compromised containers from running in your environment.
Prisma Cloud blocks containers under the following conditions:
  • A container violates its runtime model, and you’ve installed a runtime rule with the action set to block. For example, if an attacker infiltrates a container and tries to run a port scan using nc, then the container would be blocked if nc weren’t a known, allowed process.
  • A newly started container violates a vulnerability or compliance rule, and those rules have the action set to block. Prisma Cloud scans all images before they run, to enforce policies about what’s allowed to execute in your environment. For example, your policy might call for blocking any container with critical severity vulnerabilities.
Runtime rules can be created under Defend > Runtime > Container Policy. Vulnerability rules can be created under Defend > Vulnerabilities > Policy, and compliance rules can be created under Defend > Compliance > Policy.

Viewing blocked containers

Blocking immediately stops a container, taking it out of service. Blocked containers are never restarted. To see a list of blocked containers, go to the container audits page under
Monitor > Events > Container Audits
.
When a container is stopped, Prisma Cloud takes no further action to keep it stopped. Orchestrators, such as Kubernetes and OpenShift, start a fresh container in the blocked container’s place. Orchestrators have their own mechanism for maintaining a set point, so they ignore the restart policy defined in the image’s Dockerfile.
There is an exception when you run containers in a Docker-only environment (no orchestrator) and Prisma Cloud blocks a container. In this case, Prisma Cloud must take additional action to keep the container blocked. To prevent the container from automatically restarting, Prisma Cloud modifies the container’s restart policy to always unless stopped. If you want to unblock a container, connect to the node with the blocked container, and manually modify the container’s Docker configuration.

Blocked container artifacts

Forensic investigators can inspect a blocked container’s artifacts to determine why it was stopped. You can capture all the container’s contents, including its file system data, with the docker export command. Go to the node with the blocked container and run:
$ docker export [container_id] > /path/filename.tar

VMware Tanzu Application Service (TAS)

Runtime rules for VMware TAS apps are scoped by app name and space ID. Specify values for app name and space ID in the
Labels
field of the relevant collection. This field is auto-populated with values from your environment.
tas-application-name:<value> tas-space-id:<value>

Best practices

One key goal is minimizing the amount of work you’re required to do to manage runtime defense. Leverage the models that Prisma Cloud can automatically create and manage. Because behavioral learning for model creation is a mature technology for Prisma Cloud, in most cases, you won’t need to create auxiliary rules to augment model behavior. There will be some exceptions. For example, a long-running container that changes its behavior throughout its lifecycle might need some manually created rules to fully capture all valid behaviors. This is atypical for most environments, however, as containers that need to be upgraded are typically destroyed and reprovisioned with new images.
If you do need to create runtime rules, here are some best practices for doing so: