Enable Horizontal Pod Autoscaling on the CN-Series
Focus
Focus
CN-Series

Enable Horizontal Pod Autoscaling on the CN-Series

Table of Contents

Enable Horizontal Pod Autoscaling on the CN-Series

Where Can I Use This?
What Do I Need?
  • CN-Series Firewall
    deployment
  • CN-Series 10.1.x or above Container Images
  • Panorama
    running PAN-OS 10.1.x or above version
  • Helm 3.6 or above version client
    for CN-Series deployment with helm chart
The horizontal pod autoscaler (HPA) is a Kubernetes resource available in all cloud environments that automatically scales the number of CN-MGMT and CN-NGFW pods in a deployment based on monitored metrics. HPA uses two standard metrics across all cloud environments—CPU and memory utilization—as well as custom metrics specific to each cloud environment. As such, each cloud requires specific yaml files to enable HPA in AKS, EKS, and GKE.
HPA uses a cloud-specific metric adapter to retrieve metrics data from a monitoring adapter in the cloud environment, such as CloudWatch in EKS, to determine when to scale up or down based on the thresholds you define. You must modify the necessary yaml files to set the minimum and maximum number of replicas, the thresholds for each metric, and which metric are used in autoscaling your firewalls.
In PAN OS 10.1, if you use CN-MGMT pods HPA scaling, you may scale many CN-MGMT pods with no DP pods connected to them. It is recommended to create maximum replicas of CN-MGMT pods to prevent unnecessary scaling.
Cloud Environment
Metrics
Average Value
AKS, EKS, and GKE
CN-MGMT
panloggingrate
Log count
pandataplaneslots
Dataplane Slot count
CN-NFGW
dataplanecpuutilizationpct
Percent of CN-NGFW CPU utilization
dataplanepacketbufferutilization
Percent of CN-NGFW packet buffer utilization
pansessionactive
Number of sessions active on the CN-NGFW
pansessionutilization
Percent of session utilization
pansessionsslproxyutilization
Percent of session SSL proxy utilization
panthroughput
Throughput in kbps
panpacketrate
Packet rate in packets per second (pps)
panconnectionspersecond
Connections per second
In the example below is the pan-cn-hpa-dp.yaml file for EKS. This example is using the dataplane CPU utilization percentage to autoscale the CN-NGFW pods. At 25 percent, the cluster will scale up. If the CPU utilization reaches 50 percent, the cluster will deploy one additional pod. If the CPU utilization were to reach 75 percent, the cluster would deploy two additional pods. This is determined by dividing the total metric by the metric threshold and then deploying enough pods to bring the metric down to the configure threshold across all CN-NGFW pods in the cluster. However, the cluster will not deploy more CN-NGFW pods than the maxReplicas. If more than one metric exceed the threshold at the same time, the cluster will deploy the necessary number of pods to address the higher metric.
By default, the HPA adapter polls the metrics adapter every 15 seconds. If the metrics you have specified exceed the configured threshold for 60 seconds, the cluster will deploy an additional CN-NGFW pod. The cluster then waits for 300 seconds (five minutes) before deciding whether or not additional CN-NGFW pods are required. By default, one pod is deployed at a time. The cluster then checks the metric (in this case, CPU utilization) after 300 seconds. If the utilization has dropped down to a level where a pod is no longer needed, the cluster will delete a pod. The cluster will then wait an additional 60 seconds before deciding if another pod can be removed.
All the values shown below and for any metric can be modified to best fit your deployment.
kind: HorizontalPodAutoscaler apiVersion: autoscaling/v2beta2 metadata: name: hpa-dp-eks namespace: kube-system spec: scaleTargetRef: apiVersion: apps/v1beta1 kind: Deployment name: pan-ngfw-dep minReplicas: 1 maxReplicas: 10 behavior: scaleDown: stabilizationWindowSeconds: 300 policies: - type: Pods value: 1 periodSeconds: 60 - type: Percent value: 1 periodSeconds: 60 selectPolicy: Max scaleUp: stabilizationWindowSeconds: 60 policies: - type: Pods value: 1 periodSeconds: 300 # assuming 5 mins for dp to be ready - type: Percent value: 1 periodSeconds: 300 # assuming 5 mins for dp to be ready selectPolicy: Max metrics: - type: External external: metric: name: dataplaneCpuUtilizationPct target: type: Value value: 25

AKS

  1. Deploy an Azure Application Insights instance in your the cluster. You must provide the required Azure Application Insights Instrumentation Key and Azure Application Insight APP ID API key as a K8s secret.
  2. Download the AKS-specific HPA yaml files from the Palo Alto Networks GitHub repository.
  3. If your CN-MGMT is deployed in a custom namespace, update pan-cn-adapater.yaml with the custom namespace. The default namespace is
    kube-system
    .
  4. If you have not already done so, update the HPA parameters in the AKS-specific
    pan-cn-mgmt-configmap.yaml
    .
    #PAN_CLOUD: "AKS" #HPA_NAME: "<name>" #unique name to identify hpa resource per namespace or per tenant #PAN_INSTRUMENTATION_KEY: "<>" #Azure APP Insight Instrumentation Key #PUSH_INTERVAL: "15" #time interval to publish metrics to azure app insight
  5. Edit the
    pan-cn-hpa-secret.yaml
    .
    appinsights-appid: "<Azure App Insight Application ID obtained from API Access>" appinsights-key: "<Azure App Insight API Key created under API Access>" azure-client-id: "<Azure SP APP ID associated with corresponding resource group with monitoring reader access>" azure-client-secret: "<Azure SP Password associated with corresponding resource group with monitoring reader access>" azure-tenant-id: "<Azure SP tenant ID associated with corresponding resource group with monitoring reader access>"
  6. Add the HPA name you created above to the appropriate places in
    pan-cn-custommetrics.yaml
    .
  7. Modify
    pan-cn-hpa-dp.yaml
    and
    pan-cn-hpa-mp.yaml
    .
    1. Enter the minimum and maximum number of replicas.
    2. (
      Optional
      ) Change the scale down and scale up frequency values to suit your deployment. If you do not change these values, the default values are used.
    3. Copy the following section for each metric you want to use for scaling.
      - type: Pods pods: metric: name: pansessionactive target: type: AverageValue averageValue: 30
    4. Change the name the metric you want to use and set the
      averageValue
      to the threshold described in the table above. If you do not change these values, the default values are used.
    5. Save you changes.
  8. Deploy the HPA yaml files. The files must be deployed in the order described below.
    1. Use Kubectl to run the pan-cn-hpa-secret.yaml
      kubectl apply -f pan-cn-hpa-secret.yaml
    2. Use Kubectl to run the pan-cn-adapter.yaml
      kubectl apply -f pan-cn-adapter.yaml
    3. Use Kubectl to run the pan-cn-custommetrics.yaml
      kubectl apply -f pan-cn-custommetrics.yaml
    4. Use Kubectl to run the pan-cn-hpa-dp.yaml
      kubectl apply -f pan-cn-hpa-dp.yaml
    5. Use Kubectl to run the pan-cn-hpa-mp.yaml
      kubectl apply -f pan-cn-hpa-mp.yaml
  9. Verify your deployment.
    • Use kubectl to verify that the custom metrics adapter pod in the custom metrics namespace.
      kubectl get pods -n custom-metrics
    • Use kubectl to check for the HPA resource.
      kubectl get hpa -n kube-system
      kubectl describe hpa <hpa-name> -n kube-system

EKS

  1. Deploy the Amazon CloudWatch Metrics Adapter for Kubernetes in your CN-Series as a Service cluster. You must allow CloudWatch complete access to both IAM roles associated with your Kubernetes pods and clusters. To publish the custom metrics to CloudWatch, the worker nodes’ role must have the AWS managed policy
    CloudWatchAgentServerPolicy
    so that the HPA can retrieve them.
  2. Download the EKS-specific HPA yaml files from the Palo Alto Networks GitHub repository.
  3. If your CN-MGMT is deployed in a custom namespace, update pan-cn-adapater.yaml with the custom namespace. The default namespace is
    kube-system
    .
  4. Modify
    pan-cn-hpa-dp.yaml
    and
    pan-cn-hpa-mp.yaml
    .
    1. Enter the minimum and maximum number of replicas.
    2. (
      Optional
      ) Change the scale down and scale up frequency values to suit your deployment. If you do not change these values, the default values are used.
    3. Copy the following section for each metric you want to use for scaling.
      - type: Pods pods: metric: name: pansessionactive target: type: AverageValue averageValue: 30
    4. Change the name the metric you want to use and set the
      averageValue
      to the threshold described in the table above. If you do not change these values, the default values are used.
    5. Save you changes.
  5. Deploy the HPA yaml files. The files must be deployed in the order described below.
    1. Use Kubectl to run the pan-cn-adapter.yaml
      kubectl apply -f pan-cn-adapter.yaml
    2. Use Kubectl to run the pan-cn-externalmetrics.yaml
      kubectl apply -f pan-cn-externalmetrics.yaml
    3. Use Kubectl to run the pan-cn-hpa-dp.yaml
      kubectl apply -f pan-cn-hpa-dp.yaml
    4. Use Kubectl to run the pan-cn-hpa-mp.yaml
      kubectl apply -f pan-cn-hpa-mp.yaml
  6. Verify your deployment.
    • Use kubectl to verify that the custom metrics adapter pod in the custom metrics namespace.
      kubectl get pods -n custom-metrics
    • Use kubectl to check for the HPA resource.
      kubectl get hpa -n kube-system
      kubectl describe hpa <hpa-name> -n kube-system

Recommended For You