Step 1: Review the incident details and record the affected ION and the time the fan failure was detected.

Step 2: Visually inspect the device’s external fan openings and ventilation areas. Confirm they are not blocked by dust, cables, or other objects. Do not touch the fan blades or open the chassis.

Step 3: If the ION is online, SSH to the device and run:

dump sensor type=fan

Step 4: Save the output. If the fan is not operating or the incident remains active, attach the incident details and command output to a Palo Alto Networks Support case. The fan assembly may require replacement.

Step 5: Do not manually spin, remove, reseat, or replace the fan unless instructed by Palo Alto Networks Support.

Step 1: Review the incident details, identify the affected interface, and determine whether it is expected to be in use.

Step 2: SSH to the affected ION and run the following commands, replacing <port number> with the affected interface number:

dump interface status <port number>

dump interface config <port number>

Step 3: Review the output to confirm the interface’s Admin State and whether its operational State is up or down.

Step 4: If the interface is intentionally unused, click the incident Entity, change Admin Up to Down, and save the configuration.

Step 5: If the interface should be operational, keep it Admin Up and verify the cable, SFP/transceiver, and the connected port on the neighboring device.

Step 6: Run dump interface status <port number> again and confirm that the interface state is up. If it remains down after verifying the physical connection and configuration, save both command outputs and contact Palo Alto Networks Support.

Step 1: Review the incident details and identify the affected ION interface.

Step 2: SSH to the ION and save the current interface statistics:

inspect interface stats <interface ID> details

Step 3: For ION software release 5.6.1 or later, run:

dump dpdk port status port=<port>

Confirm that the link is detected and that the speed and duplex match the connected device.

Step 4: Inspect the cable and connections. Replace the cable with a known-good cable of the correct type.

Step 5: If the interface uses a transceiver, verify that the correct compatible SFP is installed and test with a known-good SFP. If a patch panel is used, try another patch-panel port or temporarily bypass it.

Step 6: Inspect the connected device and its port. If operationally safe, test another ION port.

Step 7: Run inspect interface stats <interface ID> details again and confirm that the error counters are no longer increasing.

Step 8: If the errors continue, run dump dpdk stats on release 5.6.1 or later. Attach this output, the before-and-after interface statistics, the port-status output, and the incident details to a Palo Alto Networks Support case.

Step 1: Review the incident details and confirm that the affected device is an ION 9000 running ION software Release 5.6.x.

Step 2: If the device is running a later release, do not run the legacy firmware-update command. Record the incident details and software version, and contact Palo Alto Networks Support.

Step 3: If the device is running Release 5.6.x, schedule a maintenance window. The firmware update takes approximately 20 minutes and automatically reboots the ION, causing a temporary service interruption.

Step 4: SSH to the affected ION and stop the FC process:

debug process stop name=fc

Confirm the operation when prompted.

Step 5: Run the following firmware-update command:

ion9000 upgrade-x722-fw

Step 6: Accept the confirmation prompt and allow the update to finish. Do not log out, press Ctrl+C, shut down the device, or disconnect power while the update is in progress.

Step 7: After the ION automatically reboots, confirm that it reconnects to the controller, ports 9–12 are operational, and the incident clears.

Step 8: If the command is unavailable, the update fails, or the incident remains active, save the command output and contact Palo Alto Networks Support.

Step 1: Review the incident details and identify the affected PSU. On a device with dual PSUs, record which PSU reported the loss of power.

Step 2: Check the affected PSU’s status LED. Confirm that its power cord is securely connected to both the PSU and a working outlet or PDU.

Step 3: Reseat the power cord. If power is not restored, test a different working power source and replace the cord with a known-good compatible power cord.

Step 4: Confirm that the affected PSU is fully seated. If necessary, reseat the PSU.

Step 5: Confirm that the PSU status LED is green and that the incident clears. Continue monitoring the device.

Step 6: If the PSU still does not receive power, contact Palo Alto Networks Support to obtain the correct replacement PSU for the affected ION model.

Step 1: Review the incident details and identify the affected PSU and reported reason. This event applies to ION 5200 and ION 9200 devices:

psu_not_present: The PSU is not detected.

ac_lost: The PSU is installed, but AC power is not detected.

Step 2: For psu_not_present, confirm that both PSUs are installed and fully seated.

Step 3: For ac_lost, confirm that the affected PSU’s power cord is securely connected to a working outlet or PDU. Test a different power source and a known-good compatible power cord if necessary.

Step 4: Check the PSU status LEDs and confirm that both PSUs are operational and the incident clears.

Step 5: If the PSU is correctly installed and receiving AC power but the incident remains active, reseat the affected PSU. If the issue persists, contact Palo Alto Networks Support.

Step 1: Review the incident details and record the affected ION, the sensor reporting the high temperature, and the incident time.

Step 2: Go to Insights > Prisma SD-WAN > ION Devices > Device List > Device Activity and select the affected ION.

Step 3: Review the Device Temperature chart around the incident time. Review all available sensor readings and identify which sensor exceeded its temperature threshold. The available sensors vary by ION model.

Step 4: Inspect the device environment. Confirm that the ambient temperature is within the device’s supported operating range and that the device is away from direct sunlight and other heat sources. Ensure that all air intake, exhaust, and ventilation openings are unobstructed.

Step 5: SSH to the affected ION and run:

dump sensor type=temperature

Save the output and identify any sensors that remain above their normal operating range.

Step 6: If the affected ION has cooling fans, run:

dump sensor type=fan

Confirm that the fans are operating normally and save the output.

Step 7: Correct any environmental, cooling, or airflow issue. Continue monitoring the Device Temperature chart and confirm that the sensor readings return to normal and the incident clears.

Step 8: If the device shuts down because of the thermal condition, correct the environmental issue and allow the device to cool before powering it back on. Confirm that it reconnects to the controller and that its temperature remains within the normal range.

Step 9: If the temperature remains high or the incident recurs, attach the incident details, Device Temperature chart, CLI outputs, device model, and software version to a Palo Alto Networks Support case.

Step 1: Review the incident details and record the affected site, ION serial, incident time, software version, and the cellular interface that raised the incident.

Step 2: In SCM, go to Configuration > Prisma SD-WAN > ION Devices > Claimed > [Device] > Interfaces > Cellular Interfaces and review the firmware selection mode (Host, Modem, or Custom) configured for the affected interface.

Step 3: Run dump cellular status all to review the firmware version currently loaded on the modem and any available firmware images on the device.

Step 4: If a custom firmware version is configured and that version is not available on the device, update the firmware selection mode or select an available firmware version in SCM and push the configuration to the device.

Step 5: Monitor the cellular interface status to confirm the incident clears.

Step 6: If the incident persists or the firmware cannot be applied, run dump-support all file=<descriptive_filename> (Release 6.4.1 and later) or dump-support outputs file=<descriptive_filename> (earlier releases), where <descriptive_filename> is a user-selected name such as event_name_ion-name_YYYYMMDDTHHMM.

Step 1: Review the incident details and record the affected site, ION, incident time, software version, and any error details reported with the incident.

Step 2: Run dump cellular status all to review the current modem status, firmware version, registration state, and SIM status.

Step 3: The system may attempt automatic recovery. Monitor the cellular interface status to confirm whether the incident clears automatically within a few minutes.

Step 4: Review related cellular incidents and alerts at the same site during the same period.

Step 5: If the incident persists or the cellular link does not recover, run dump-support all file=<descriptive_filename> (Release 6.4.1 and later) or dump-support outputs file=<descriptive_filename> (earlier releases), where <descriptive_filename> is a user-selected name such as event_name_ion-name_YYYYMMDDTHHMM.

Step 1: Review the incident details and record the affected site, ION, incident time, software version, and any related events.

Step 2: Run dump cellular status all to check the current modem detection and registration state.

Step 3: Monitor the device to see whether modem detection recovers automatically within a few minutes.

Step 4: Verify that no physical SIM or hardware change was made around the incident time.

Step 5: If modem detection does not recover, run dump-support all file=<descriptive_filename> (Release 6.4.1 and later) or dump-support outputs file=<descriptive_filename> (earlier releases), where <descriptive_filename> is a user-selected name such as event_name_ion-name_YYYYMMDDTHHMM.

Step 1: Review the incident details and record the affected site, ION, incident time, software version, and any reported temperature details.

Step 2: Run dump cellular status all to review the current modem status.

Step 3: Run dump cellular stats all to review signal and operational statistics while the temperature condition is active.

Step 4: Inspect the ION's physical installation. Verify that the device has adequate ventilation, is not exposed to direct sunlight or radiant heat sources, and has unobstructed air vents.

Step 5: Monitor the cellular interface to confirm the incident clears as the device temperature returns to an acceptable level.

Step 6: If the modem shuts down or the incident persists, run dump-support all file=<descriptive_filename> (Release 6.4.1 and later) or dump-support outputs file=<descriptive_filename> (earlier releases), where <descriptive_filename> is a user-selected name such as event_name_ion-name_YYYYMMDDTHHMM.

Step 1: Review the incident details and record the affected site, ION, incident time, software version, and affected cellular interface.

Step 2: Run dump cellular status all to review the SIM status, PIN state, and remaining PIN retry count.

Step 3: Verify the remaining PIN retry count before another attempt. After three failed PIN attempts, the SIM is blocked and requires the PUK; exhausting PUK attempts can permanently disable the SIM.

Step 4: In SCM, go to Configuration > Prisma SD-WAN > ION Devices > Claimed, select the device, select Configure the Device > Interfaces, select the cellular module, and in Main Configuration open the ellipsis next to the SIM and select Enter PIN.

Step 5: Enter the verified SIM PIN and save the configuration. Do not guess the PIN or continue attempting PINs when the remaining retry count is low.

Step 6: Monitor the cellular interface and confirm that the PIN is accepted and the incident clears.

Step 7: If the SIM is blocked, follow the PUK procedure for DEVICE_CELLULAR_SIM_PUK_ERROR. If the incident persists, collect a support bundle and open a Palo Alto Networks Support case.

Step 1: Review the incident details and record the affected site, ION, incident time, software version, cellular interface, and remaining PUK retry count.

Step 2: Run dump cellular status all to confirm that the SIM is blocked and requires a PUK.

Step 3: Obtain the correct PUK from the carrier or SIM administrator. Do not guess the PUK; exhausting the allowed PUK attempts can permanently disable the SIM.

Step 4: In SCM, go to Configuration > Prisma SD-WAN > ION Devices > Claimed, select the device, select Configure the Device > Interfaces, select the cellular module, and in Main Configuration open the ellipsis next to the active SIM and select Unblock SIM.

Step 5: Enter the verified PUK and a new SIM PIN, save the configuration, and monitor the cellular interface.

Step 6: Confirm that the SIM is unblocked, registers with the carrier, and the incident clears.

Step 7: If the PUK is rejected or no attempts remain, contact the carrier for SIM replacement. If the incident persists after the SIM is unblocked, collect a support bundle and open a Palo Alto Networks Support case.

Step 1: Review the incident details and record the affected site, ION, incident time, software version, and whether the SIM was inserted or removed.

Step 2: Confirm whether the SIM insertion or removal was planned or authorized.

Step 3: Run dump cellular status all to verify the current SIM status, active SIM slot, carrier registration, and connection state.

Step 4: If the SIM removal was unintentional, check the physical SIM slot to confirm the SIM card is properly seated and secured.

Step 5: Monitor the cellular interface to confirm the incident clears after the SIM is properly detected and registered.

Step 6: If the cellular interface does not recover or the SIM appears physically damaged, run dump-support all file=<descriptive_filename> (Release 6.4.1 and later) or dump-support outputs file=<descriptive_filename> (earlier releases), where <descriptive_filename> is a user-selected name such as event_name_ion-name_YYYYMMDDTHHMM.

Step 1: Review the incident details and record the affected site, ION serial, incident time, and software version.

Step 2: Run dump poe system status to review the PoE subsystem state, port status, power usage, and any reported fault conditions.

Step 3: Run dump poe system config to review the current PoE configuration for reference.

Step 4: Verify whether connected powered devices (PDs) are operating normally and note whether the PoE fault coincides with a hardware or power event.

Step 5: If PoE ports remain non-functional, collect a support bundle and open a Palo Alto Networks Support case. Support can determine whether a reload, power cycle, or hardware replacement is required. Run dump-support all file=<descriptive_filename> for Release 6.4.1 and later or dump-support outputs file=<descriptive_filename> for earlier releases.

Step 1: Review the incident details and record the affected site, ION, incident time, software version, and total power usage relative to the configured threshold.

Step 2: Run dump poe system status to review current power usage per port, total consumption, and the configured threshold.

Step 3: Run dump poe system config to confirm the configured system power threshold and per-port power settings.

Step 4: Identify connected PoE devices with high power consumption. Determine whether the current set of powered devices is appropriate for the ION's total PoE power budget.

Step 5: If total power consumption consistently exceeds the threshold, reduce the number of powered devices, connect higher-power devices to a separate power source, or increase the threshold if supported by the hardware's total PoE power budget.

Step 6: Monitor PoE system status to confirm the incident clears after adjustments.

Step 7: If power usage remains unexpectedly high or the incident persists, run dump-support all file=<descriptive_filename> (Release 6.4.1 and later) or dump-support outputs file=<descriptive_filename> (earlier releases), where <descriptive_filename> is a user-selected name such as event_name_ion-name_YYYYMMDDTHHMM.

Step 1: Review the incident details and record the affected site, ION, incident time, software version, and the specific PoE port that exceeded the threshold.

Step 2: Run dump poe system status to review the power consumption and status for all PoE ports.

Step 3: Run dump poe system config to confirm the configured per-port power threshold for the affected port.

Step 4: Identify the powered device connected to the affected port and confirm whether its power draw is within the expected range.

Step 5: If the powered device's power consumption is legitimately high, consider increasing the port power threshold in SCM or replacing the device with a lower-power alternative.

Step 6: Monitor the PoE port status to confirm the incident clears after the adjustment.

Step 7: If the port power remains unexpectedly high or the incident persists, run dump-support all file=<descriptive_filename> (Release 6.4.1 and later) or dump-support outputs file=<descriptive_filename> (earlier releases), where <descriptive_filename> is a user-selected name such as event_name_ion-name_YYYYMMDDTHHMM.

Step 1: Review the incident details and record the affected site, ION, incident time, software version, and the affected PoE port.

Step 2: Run dump poe system status to review the current power status for all PoE ports.

Step 3: Identify the powered device connected to the affected port and confirm whether it is operating normally.

Step 4: Verify that the cable connection on the affected port is secure and that the powered device is functional.

Step 5: Run dump poe system config to confirm the port is enabled and the power allocation is correct.

Step 6: Monitor the PoE port to confirm it returns to normal operation.

Step 7: If the PoE port status does not recover or the connected device remains offline, run dump-support all file=<descriptive_filename> (Release 6.4.1 and later) or dump-support outputs file=<descriptive_filename> (earlier releases), where <descriptive_filename> is a user-selected name such as event_name_ion-name_YYYYMMDDTHHMM.

Step 1: Review the incident details and record the affected site, cluster name, unavailable device, active device, incident time, and software versions.

Step 2: Run dump spoke-ha status on both devices, when accessible. Confirm the active state, Peer Connected state, Base Priority, Effective Priority, Priority Adjustment reason, and last update time.

Step 3: Run dump spoke-ha config and review the HA control interface, advertisement interval, preemption setting, Base Priority, tracked interfaces, WAN reachability tracking, and configured priority reductions.

Step 4: Determine whether a tracked interface or tracked WAN became unavailable. If a Priority Adjustment is displayed, check the reported interface using dump interface status interface <interface>. Confirm whether the configured priority reduction caused the peer to become active.

Step 5: Review the configuration and audit history for changes to Base Priority, priority adjustments, tracked interfaces, WAN reachability tracking, or preemption. A priority change can change the active device, particularly when preemption is enabled. Make any configuration corrections through SCM.

Step 6: Review related DEVICEHW_INTERFACE_DOWN, DEVICEHW_POWER_LOST, DEVICESW_SYSTEM_BOOT, DEVICESW_CRITICAL_PROCESSSTOP, DEVICESW_GENERAL_PROCESSSTOP, DEVICESW_GENERAL_PROCESSRESTART, and DEVICESW_DISCONNECTED_FROM_CONTROLLER incidents. Check for a site power outage, loss of power to the unavailable device, an unexpected reboot, or a critical service failure on the previously active device.

Step 7: If Peer Connected is false, verify the HA control interface and physical connection between the IONs. Check cabling, switch ports, VLAN connectivity, interface errors, and any firewall or access-control device between the peers. Confirm that HA advertisement or keepalive messages are not being dropped.

Step 8: For Release 6.6.1 and later, run dump spoke-ha failover-timeline to determine when the device became active and whether interface programming, route installation, traffic forwarding, or VPN recovery was delayed.

Step 9: In SCM, go to Configuration > Prisma SD-WAN > Branch Sites, open the site actions menu, and select HA Groups and verify the cluster configuration and device membership. Address the identified interface, WAN-reachability, power, device, software-service, priority, or HA peer-connectivity issue.

Step 10: Monitor the cluster and confirm that both devices are operational, peer connectivity is restored, the expected device is active, and the incident clears.

Step 11: If the degraded state persists, collect a support bundle from both devices when possible and open a Palo Alto Networks Support case. Run dump-support all file=<descriptive_filename> for Release 6.4.1 and later or dump-support outputs file=<descriptive_filename> for earlier releases.

Step 1: Review the incident details and record the affected site, cluster name, both device identifiers, incident time, software versions, and customer impact.

Step 2: Determine whether either device is powered on and physically accessible. Because both devices may be disconnected from the controller, use direct SSH through a reachable local or out-of-band network, or use the physical console.

Step 3: On each accessible device, run dump spoke-ha status and dump spoke-ha config. Review the active state, Peer Connected state, Base and Effective Priority, Priority Adjustment reason, HA control interface, tracked interfaces, WAN reachability tracking, advertisement interval, and preemption setting.

Step 4: Check whether a tracked interface or tracked WAN failed on one or both devices and reduced their Effective Priority. Run dump interface status interface <interface> for the tracked and HA control interfaces.

Step 5: Review the configuration and audit history for recent changes to Base Priority, priority adjustments, tracked interfaces, WAN reachability tracking, preemption, device membership, or the HA control interface.

Step 6: Review related DEVICEHW_POWER_LOST, DEVICEHW_INTERFACE_DOWN, DEVICESW_SYSTEM_BOOT, DEVICESW_CRITICAL_PROCESSSTOP, DEVICESW_GENERAL_PROCESSSTOP, DEVICESW_GENERAL_PROCESSRESTART, and DEVICESW_DISCONNECTED_FROM_CONTROLLER incidents on both devices. Determine whether a common power outage, upstream network failure, simultaneous reboot, hardware failure, or critical software-service failure affected both devices.

Step 7: Verify the HA control connection between the IONs. Check cabling, switch ports, VLAN connectivity, interface errors, peer reachability, and any device that could drop HA advertisement or keepalive messages.

Step 8: For Release 6.6.1 and later, run dump spoke-ha failover-timeline on each accessible device to review any recorded state transitions and recovery stages.

Step 9: Restore power, physical connectivity, or network access to at least one device. Make configuration corrections through SCM after controller connectivity is restored. Do not change the HA configuration locally through the CLI.

Step 10: Confirm that at least one device becomes active, traffic forwarding resumes, and the incident clears. Then verify that the second device recovers and rejoins the cluster.

Step 11: If neither device can be recovered, collect a support bundle from each accessible device and open a Palo Alto Networks Support case. Run dump-support all file=<descriptive_filename> for Release 6.4.1 and later or dump-support outputs file=<descriptive_filename> for earlier releases.

Step 1: Review the incident details and record the affected site, cluster name, both device identifiers, incident time, software versions, and customer impact.

Step 2: Run dump spoke-ha status on both devices. Confirm that both devices report Active and review Peer Connected, Base Priority, Effective Priority, Priority Adjustment reason, and last update time.

Step 3: Run dump spoke-ha config on both devices. Confirm that the cluster ID, HA control interface, advertisement interval, preemption setting, Base Priority, tracked-interface configuration, WAN reachability tracking, and priority adjustments are consistent with the intended design.

Step 4: If Peer Connected is false, investigate the HA control connection first. Verify cabling, switch ports, VLAN configuration, interface state, interface errors, peer reachability, and any firewall or access-control device between the IONs. Determine whether HA advertisement or keepalive messages are being delayed or dropped.

Step 5: Determine whether one device lost power, rebooted, experienced an active-device or critical-service failure, and then returned while peer communication was unavailable. Review DEVICEHW_POWER_LOST, DEVICESW_SYSTEM_BOOT, DEVICESW_CRITICAL_PROCESSSTOP, DEVICESW_GENERAL_PROCESSSTOP, and DEVICESW_GENERAL_PROCESSRESTART incidents.

Step 6: Check whether a tracked-interface failure, WAN-reachability failure, or recent Base Priority or Priority Adjustment change contributed to the state transition. Review the configuration and audit history and make any necessary corrections through SCM.

Step 7: For Release 6.6.1 and later, run dump spoke-ha failover-timeline on both devices and compare the state-transition timestamps.

Step 8: In SCM, go to Configuration > Prisma SD-WAN > Branch Sites, open the site actions menu, and select HA Groups and verify the HA configuration and device membership. Do not make local HA configuration changes through the CLI.

Step 9: Monitor the cluster and confirm that peer connectivity is restored, HA advertisements are exchanged, and the cluster converges to one active and one backup device.

Step 10: If both devices remain active, collect a support bundle from both devices and open a Palo Alto Networks Support case. Run dump-support all file=<descriptive_filename> for Release 6.4.1 and later or dump-support outputs file=<descriptive_filename> for earlier releases.

Step 1: Review the incident details and record the affected site, cluster name, device that changed state, previous and new states, incident time, software version, and whether the transition was expected.

Step 2: Run dump spoke-ha status on both devices. Confirm the current active and backup states, Peer Connected state, Base Priority, Effective Priority, Priority Adjustment reason, and last update time.

Step 3: Run dump spoke-ha config and review the HA control interface, advertisement interval, preemption setting, Base Priority, tracked interfaces, WAN reachability tracking, and configured priority reductions.

Step 4: If the state change was planned, confirm that the expected device is active, its tracked interfaces and WAN paths are operational, peer connectivity is established, and traffic is forwarding normally.

Step 5: If the state change was not expected, determine whether a tracked interface or tracked WAN became unavailable and reduced the device’s Effective Priority. Check the reported interface using dump interface status interface <interface>.

Step 6: Review the configuration and audit history for changes to Base Priority, priority adjustments, tracked interfaces, WAN reachability tracking, or preemption. Confirm whether a configuration change caused the election of a different active device.

Step 7: Review related DEVICEHW_INTERFACE_DOWN, DEVICEHW_POWER_LOST, DEVICESW_SYSTEM_BOOT, DEVICESW_CRITICAL_PROCESSSTOP, DEVICESW_GENERAL_PROCESSSTOP, DEVICESW_GENERAL_PROCESSRESTART, and DEVICESW_DISCONNECTED_FROM_CONTROLLER incidents. Determine whether the previous active device lost power, went down, rebooted, or experienced a critical software-service failure.

Step 8: If Peer Connected is false, verify the HA control interface, peer reachability, cabling, switch ports, VLAN configuration, and interface errors. Confirm that HA advertisement or keepalive messages are not being dropped between the IONs.

Step 9: For Release 6.6.1 and later, run dump spoke-ha failover-timeline to review the chronological HA events and recovery stages.

Step 10: Monitor the cluster and confirm that it stabilizes with one active and one backup device and that traffic forwarding is normal.

Step 11: If the transition was unintended, repeatedly occurs, or the cluster does not stabilize, collect a support bundle from both devices and open a Palo Alto Networks Support case. Run dump-support all file=<descriptive_filename> for Release 6.4.1 and later or dump-support outputs file=<descriptive_filename> for earlier releases.

Step 1: Review the incident details and identify the affected ION device and Claim Certificate expiration date.

Step 2: Log in to the ION and run the following commands:

dump overview

dump cert-operation-state info

inspect certificate cic

Review Certificate Validity, Renewal Window, Renewal State, and Info. Also review any earlier Claim Certificate incidents that may explain why automatic renewal did not complete.

Step 3: If the ION is offline or the device time is incorrect, restore controller connectivity or time synchronization.

Step 4: If the Info field specifically identifies an external CA or SCEP issue, engage the PKI administrator to investigate that issue.

Step 5: If the Claim Certificate continues to approach expiration, collect the incident details and command outputs and contact Palo Alto Networks Support immediately. Support must investigate the renewal failure and manually trigger CIC renewal from the controller or backend, if required.

Step 1: Review the incident details and identify the affected ION device and Claim Certificate expiration date.

Step 2: Log in to the ION and review its controller connection, device time, certificate validity, and renewal state.

dump overview

dump cert-operation-state info

inspect certificate cic

Review Certificate Validity, Renewal Window, Renewal State, and Info. Also check for earlier Claim Certificate incidents that may explain why automatic renewal did not complete.

Step 3: If the ION is offline or the device time is incorrect, restore controller connectivity or time synchronization.

Step 4: If the tenant uses SCEP, verify that the SCEP service is available and that the Certificate Authority is issuing valid certificates with an appropriate validity period.

Step 5: If the Claim Certificate continues to approach expiration, collect the incident details and command outputs and contact Palo Alto Networks Support immediately.

Step 6: Support must investigate the renewal failure and, if appropriate, manually trigger CIC renewal from the controller or backend.

Step 1: Review the incident details and identify the affected ION device.

Step 2: Log in to the ION and run the following commands:

dump overview

dump cert-operation-state info

inspect certificate cic

Review Certificate Validity, Renewal Window, Renewal State, and Info. Compare the certificate validity period and expiration date with the renewal window shown in the output.

Step 3: Confirm that the ION is connected to the controller and the device time is correct.

Step 4: If the Info field specifically identifies an invalid or expired certificate from an external CA, engage the PKI administrator to investigate that issue.

Step 5: If repeated renewal attempts continue, collect the incident details and command outputs and contact Palo Alto Networks Support. Support must investigate the certificate validity, renewal-window configuration, and controller or backend renewal process.

Step 1: Review the incident details and any related Claim Certificate incidents to identify the reported renewal failure reason.

Step 2: Log in to the affected ION and run the following commands:

dump overview

dump cert-operation-state info

inspect certificate cic

Confirm that the controller connection is Up [CIC] and the device time is correct. Under the CIC certificate-operation state, review Certificate Validity, Renewal Window, Renewal State, and Info.

Step 3: If the ION is offline or the device time is incorrect, restore controller connectivity or time synchronization and check the renewal state again.

Step 4: If the Info field specifically identifies an external CA, SCEP, or CSR rejection issue, engage the PKI administrator and Palo Alto Networks Support for further investigation.

Step 5: If no actionable issue is identified or certificate renewal continues to fail, collect the incident details and command outputs and contact Palo Alto Networks Support. Support must investigate the certificate-issuance, CSR, controller, or backend renewal process.

Step 1: Review the preceding Claim Certificate incidents to determine why the earlier renewal attempts failed.

Step 2: Log in to the affected ION and run the following commands:

dump overview

dump cert-operation-state info

inspect certificate cic

Confirm that the controller connection is Up [CIC] and the device time is correct. Under the CIC certificate-operation state, review Certificate Validity, Renewal Window, Renewal State, and Info.

Step 3: Resolve any identified controller-connectivity or time-synchronization issue.

Step 4: If the Info field specifically identifies an external CA or SCEP issue, engage the PKI administrator to investigate that issue.

Step 5: Because the renewal retry limit has been exceeded, collect the incident details and command outputs and contact Palo Alto Networks Support immediately. Support must investigate the repeated failures and restore automatic renewal or manually trigger CIC renewal from the controller or backend, if required.

Step 1: Review the incident details and record the affected ION, incident time, and duration. If the incident has already cleared and does not recur, continue monitoring.

Step 2: Confirm that the ION is online and connected to the controller. If the ION is offline or has a general controller-connectivity issue, troubleshoot DEVICESW_DISCONNECTED_FROM_CONTROLLER instead.

Step 3: SSH to the ION and run:

dump overview

dump controller status

Confirm the controller connection is Connected. Save the output if it is Partially Connected or Not Connected.

Step 4: Verify controller reachability through the active controller-facing interface:

debug controller reachability <interface>

Correct any reported IP addressing, gateway, certificate, or reachability issue.

Step 5: Verify DNS resolution and HTTPS connectivity to the controller services:

nslookup locator.cgnx.net

tcpping <src_interface> locator.cgnx.net:443

Ensure that upstream firewalls or proxies do not block outbound TCP port 443.

Step 6: Verify that at least one available Internet circuit is allowed to be used for controller connections. In the circuit configuration, set Controller Connections to Yes or Use Circuit Category Setting as appropriate. Do not exclude every available circuit from controller connections.

Step 7: If the incident persists or recurs, collect a support bundle before opening a Support case.

For ION software Release 6.4.1 or later, run:

dump-support all file=<file_name>

For earlier releases that support dump-support, run:

dump-support outputs file=<file_name>

The all option collects CLI outputs, syslogs, and core files. It was introduced in Release 6.4.1; dump-support requires the Super role. You can then use file list to locate the generated file and export it for the case.

Attach the incident details, command outputs, and logs to a Palo Alto Networks Support case. Do not restart processes or reboot the ION unless Support instructs you to do so.

Step 1: Review the incident details and record the affected ION, site, disconnection time, software version, and the WAN interface expected to provide internet and controller connectivity.

Step 2: Because the ION is disconnected from the controller, the Remote CLI Toolkit will not be available, and configuration changes made in SCM will not be pushed to the device. Connect directly to the ION through SSH from the local network or through the console port.

Step 3: Check the ION and controller-connection status:

dump overview

dump controller status

Confirm that the device is disconnected and review the reported state of the controller sessions.

Step 4: Verify the configuration and operational state of the controller-facing WAN interface:

dump interface config <WAN-interface-number>

dump interface status <WAN-interface-number>

Confirm that the interface is Admin Up, operationally up, has a valid IP address, and has the expected default gateway and DNS configuration.

Step 5: Verify Layer 3 internet reachability through the affected WAN interface:

ping <WAN-interface-number> 8.8.8.8

If the test fails, investigate the local interface, IP addressing, default gateway, routing, upstream router, ISP connectivity, cabling, or modem.

Step 6: Verify DNS resolution and reachability using the controller locator hostname:

nslookup locator.cgnx.net

ping <WAN-interface-number> locator.cgnx.net

Confirm that locator.cgnx.net resolves to an IP address. If the public-IP ping succeeds but the hostname test fails, investigate the configured DNS servers and DNS reachability.

Step 7: Verify Layer 4 TCP connectivity to the controller locator service over port 443:

tcpping <WAN-interface-number> locator.cgnx.net:443

If DNS resolution works but the TCP test fails, confirm that upstream firewalls, proxies, security policies, and the ISP allow outbound TCP port 443 to the Palo Alto Networks controller services.

Step 8: Verify the complete controller connection, including Layer 7 and certificate validation:

debug controller reachability <WAN-interface-number>

Review each test result to identify a DNS, routing, TCP 443, TLS, certificate-validation, or controller-service failure. Confirm that the ION system time is correct if certificate validation fails.

Step 9: Correct the identified network, DNS, routing, firewall, or time-synchronization issue locally. After connectivity is restored, run:

dump controller status

Confirm that Controller Connection shows Connected and that the controller sessions are established. Verify that the ION returns online in SCM and that the incident clears.

Step 10: If Layer 3, DNS, and TCP 443 connectivity succeed but the ION remains disconnected, collect the incident details and all command outputs and contact Palo Alto Networks Support. After controller connectivity is restored, collect a support bundle if requested:

For Release 6.4.1 or later:

dump-support all file=controller-disconnected

For earlier supported releases:

dump-support outputs file=controller-disconnected

Do not restart processes or reboot the ION unless instructed by Palo Alto Networks Support.

Step 1: Review the incident details and record the affected site, ION, incident time, duration, software version, and any related controller, analytics, interface, or circuit incidents.

Step 2: Navigate to Configuration > Prisma SD-WAN > Devices > Claimed Devices and hover over the affected device’s status. Review Config and Events, Analytics, and Flows.

If the Device State or Config and Events connection is Offline, check for an active DEVICESW_DISCONNECTED_FROM_CONTROLLER incident and follow its linked remediation. If the device and Config and Events connection are Online but Flows is Offline, continue with the flow-specific troubleshooting below.

Step 3: Navigate to Insights > Prisma SASE > Branch Sites > Flows and review the affected site. Record the timestamp of the most recent flow record and determine whether the flow-visibility gap corresponds with the incident time.

Step 4: Access the ION through the Remote CLI Toolkit, SSH, or console and run:

dump overview

dump controller status

dump cgnxinfra status

dump cgnxinfra status live

Confirm the Flows Connection status and destination FQDN, the flows WebSocket state, the source interface used for the connection, the last flow-message timestamp, and whether the flows_live TCP session is established on port 443.

Step 5: Identify the flow-connection source interface from the command output and verify its status and configuration:

dump interface status <source_interface>

dump interface config <source_interface>

Confirm that the interface is operational and has a valid IP address, default route, gateway, and DNS server.

Step 6: In the dump overview output, locate the Flows Connection line and copy the displayed hostname. Test DNS resolution and TCP 443 connectivity to that hostname:

ping <source_interface> <hostname_shown_under_Flows_Connection>

tcpping <source_interface> <hostname_shown_under_Flows_Connection>:443

A resolved IP address confirms that DNS resolution is working. A failed ICMP response alone does not confirm that the service is unreachable because ICMP may be blocked. Use the TCP 443 result as the primary connectivity test.

Step 7: If TCP 443 connectivity fails, verify routing, DNS, NAT, and upstream firewall policies. Confirm that outbound TCP port 443 is allowed from the selected source interface to the flow-service FQDN.

Step 8: After remediation, run the commands again and confirm that Flows Connection is Up, the flows WebSocket is connected, the flows_live TCP session is established, and the last flow-message timestamp is updating. Confirm that SCM shows Flows as Online, new flow records appear in Flow Browser, and the incident clears automatically.

Step 9: If the Flows connection remains down while the device and Config and Events connection remain Online and TCP 443 connectivity succeeds, collect a support bundle:

dump-support all file=flows-controller-disconnect

For earlier releases that do not support the all option:

dump-support outputs file=flows-controller-disconnect

Attach the incident details, CLI outputs, connectivity-test results, flow-visibility gap, and support bundle to a Palo Alto Networks Support case. Do not restart any process or reboot the ION unless instructed by Support.

Step 1: Review the incident details and record the affected site, virtual ION name, incident time, duration, software version, device model, branch or data center deployment type, and any recent license renewal, tenant allocation, virtual ION deployment, replacement, or decommissioning activity.

Step 2: Go to System Settings > Subscription Usage > Prisma SD-WAN > Subscription Usage. Review the purchased and consumed subscriptions, any subscription highlighted as over-consumed, the subscription type and quantity, and the license expiration date under Purchase History. Confirm that the subscription assigned to the tenant is valid and covers the affected virtual ION. For data center or per-device branch subscriptions, each ION requires its own entitlement.

Step 3: Confirm that the virtual ION and its subscription are allocated to the correct Tenant Service Group (TSG). In a multitenant deployment, verify that the required subscription has not remained at the parent tenant or been allocated to a different child tenant. To allocate available capacity to a child tenant, go to Subscription > Subscription List, open the subscription’s ellipsis menu, select Activate Cloud Tenant, select the intended tenant, specify the quantity, and activate the subscription.

Step 4: If all available capacity is already consumed, activate or purchase enough additional subscription capacity, reallocate unused capacity from another tenant, or return the entitlement from a virtual ION that has been permanently decommissioned and is no longer carrying traffic. Do not unclaim, shut down, or remove an active ION simply to free a license. License purchases, reallocations, and device decommissioning must follow administrator approval and normal change-control procedures.

Step 5: After correcting the subscription, allow time for the updated entitlement to reach the Prisma SD-WAN tenant. For a newly allocated virtual ION, go to Configuration > Prisma SD-WAN > ION Devices > Unclaimed Devices. Newly allocated devices can take approximately 15–60 minutes to appear.

Step 6: Go to Configuration > Prisma SD-WAN > Devices > Claimed Devices and hover over the affected ION’s status. Confirm that Device State and Config and Events are Online. Verify that the incident clears and that traffic and applications using the virtual ION are operating normally.

Step 7: If the virtual machine is powered off, start it through the supported hypervisor or cloud platform after the license becomes available. Do not reboot a running virtual ION as a licensing troubleshooting step.

Step 8: If Subscription Usage shows a valid, unexpired subscription with sufficient capacity allocated to the correct tenant, but license verification still fails, open a Palo Alto Networks Support case. Include screenshots of Subscription Usage, the subscription expiration and allocation details, the affected TSG ID, and the virtual ION identity. If the ION CLI remains accessible, collect a support bundle:

dump-support all file=<descriptive_filename>

For earlier releases that do not support the all option:

dump-support outputs file=<descriptive_filename>

Use a filename containing the ION name and collection time, such as license_verification_failed_branch-vion1_20260818T1430.

Step 1: Review the alert details and record the affected site, ION, alert time, software version, current MIC expiration date, and any related certificate or controller-connectivity incidents.

Step 2: Go to Configuration > Prisma SD-WAN > Devices > Claimed Devices and hover over the affected ION’s status. Confirm that Device State and Config and Events are Online. If either connection is Offline, first follow the remediation for DEVICESW_DISCONNECTED_FROM_CONTROLLER.

Step 3: Run the following read-only commands:

dump overview

inspect certificate mic

In dump overview, verify that Controller is Up, confirm that Time Now is correct, and record the MIC Certificate expiration date. In inspect certificate mic, review the certificate issuer, serial number, and Not Before and Not After validity dates.

Step 4: MIC renewal occurs automatically. There is no documented customer CLI command to manually renew or replace the MIC. If the MIC expiration date is extended and the alert clears, continue monitoring for recurrence.

Step 5: If the expiration date does not change, the alert remains active or repeatedly returns, the certificate expires, or DEVICESW_MFGCERT_RENEWAL_FAILED occurs, collect a support bundle and open a Palo Alto Networks Support case:

dump-support all file=<descriptive_filename>

For releases earlier than 6.4.1 that do not support the all option:

dump-support outputs file=<descriptive_filename>

Use a filename containing the ION name and collection time, such as mfgcert_expiry_warning_branch-ion1_20260818T1430.

Step 1: Review the incident details and record the affected site, ION, incident time, duration, software version, current MIC expiration date, and any related certificate or controller-connectivity incidents.

Step 2: Go to Configuration > Prisma SD-WAN > Devices > Claimed Devices and hover over the affected ION’s status. Confirm that Device State and Config and Events are Online. If either connection is Offline, first follow the remediation for DEVICESW_DISCONNECTED_FROM_CONTROLLER. Continue with the remaining steps because Support must still investigate the MIC renewal failure after connectivity is restored.

Step 3: Run the following read-only commands:

dump overview

inspect certificate mic

In dump overview, verify that Controller is Up, confirm that Time Now is correct, and record the MIC Certificate expiration date. In inspect certificate mic, review the certificate issuer, serial number, and Not Before and Not After validity dates.

Step 4: MIC renewal is automatic and cannot be manually restarted using documented customer CLI commands. Collect a support bundle and open a Palo Alto Networks Support case. Include the incident details and command outputs from Step 3:

dump-support all file=<descriptive_filename>

For releases earlier than 6.4.1 that do not support the all option:

dump-support outputs file=<descriptive_filename>

Use a filename containing the ION name and collection time, such as mfgcert_renewal_failed_branch-ion1_20260818T1430.

Step 5: After Palo Alto Networks Support resolves the renewal failure, run dump overview and inspect certificate mic again. Confirm that the MIC has a new expiration date, the incident clears, and the ION remains connected to the controller.

Step 1: Review the failed deployment and record the virtual ION model, deployment method, deployment start time, incident time, software image or version, and the reported validation error.

Step 2: In SCM, go to System Settings > vION Subscription Management. Select the applicable virtual ION model and create a new authorization token.

Step 3: Select the appropriate token use type:

Select Single Use when deploying one virtual ION.

Select Multi Use when deploying multiple virtual IONs of the same model using the same deployment workflow.

Copy the newly generated ION Key and Secret Key. These values are sensitive and should not be included in screenshots, logs, incident notes, or support cases.

Step 4: Enter the new ION Key and Secret Key in the token fields of the deployment metadata or initialization file required by the applicable cloud or hypervisor deployment method. Confirm that the values are complete and do not contain leading or trailing spaces.

Step 5: Repeat the virtual ION deployment using the updated deployment metadata or initialization file.

Step 6: Confirm that token validation succeeds, the incident clears, and the virtual ION connects to the Prisma SD-WAN controller. After successful registration, the virtual ION should appear in the ION device inventory as Unclaimed and Online-Restricted.

Step 7: If validation fails again with a newly generated token, open a Palo Alto Networks Support case. Include the virtual ION model, software image or version, deployment method, deployment and failure times, token use type, and the exact validation error. Do not include the ION Key or Secret Key.

Step 1: Review the incident details and record the affected site, ION, incident time, software version, and the cellular interface that is out of sync.

Step 2: In SCM, go to Configuration > Prisma SD-WAN > ION Devices > Claimed > [Device] > Interfaces > Cellular Interfaces and confirm the current configuration matches the intended settings.

Step 3: Run dump cellular config all to review the interface configuration currently applied on the device and compare it with the SCM configuration.

Step 4: If the configuration in SCM is correct and there is a discrepancy, push the configuration update to the device.

Step 5: Monitor the interface to confirm the incident clears after the configuration synchronizes.

Step 6: If the incident persists after the configuration push, run dump-support all file=<descriptive_filename> (Release 6.4.1 and later) or dump-support outputs file=<descriptive_filename> (earlier releases), where <descriptive_filename> is a user-selected name such as event_name_ion-name_YYYYMMDDTHHMM.

Step 1: Review the incident details and record the affected site, ION, incident time, software version, and the module configuration that is out of sync.

Step 2: In SCM, go to Configuration > Prisma SD-WAN > ION Devices > Claimed > [Device] > Interfaces > Cellular Interfaces and confirm the SIM PIN configuration and module settings match the intended values.

Step 3: Run dump cellular config all to review the current module configuration applied on the device and compare it with the SCM configuration.

Step 4: Correct any configuration discrepancy in SCM and push the update to the device.

Step 5: Monitor the cellular interface to confirm the incident clears after the configuration synchronizes.

Step 6: If the incident persists, run dump-support all file=<descriptive_filename> (Release 6.4.1 and later) or dump-support outputs file=<descriptive_filename> (earlier releases), where <descriptive_filename> is a user-selected name such as event_name_ion-name_YYYYMMDDTHHMM.

Step 1: Review the incident details and record the affected site, ION, incident time, software version, and any related SIM or cellular incidents.

Step 2: Run dump cellular status all to review the current SIM status and any error details.

Step 3: Monitor the cellular interface to see whether the system recovers automatically.

Step 4: If the incident persists or recurs frequently, run dump-support all file=<descriptive_filename> (Release 6.4.1 and later) or dump-support outputs file=<descriptive_filename> (earlier releases), where <descriptive_filename> is a user-selected name such as event_name_ion-name_YYYYMMDDTHHMM.

Step 1: Review the incident details and note the affected site, ION, incident time and duration, excluded interface or circuit, and any related circuit or controller-connectivity incidents.

Step 2: Go to Configuration > Prisma SD-WAN > Devices > Claimed Devices and locate the affected ION. Review Device State and Config and Events. If either connection is Offline, check for DEVICESW_DISCONNECTED_FROM_CONTROLLER and follow the DEVICESW_DISCONNECTED_FROM_CONTROLLER remediation.

Step 3: In the Remote CLI Toolkit, run:

dump overview

dump controller status

Confirm that the controller connection is established. In dump controller status, review the source IP address used by the active controller connections.

For each interface that is expected to carry controller connections, run:

dump interface status <interface>

debug controller reachability <interface>

Replace <interface> with the interface name or number associated with the circuit under the affected site’s configuration. Also check the excluded interface identified in the incident, if available. Verify that the intended interface is Up, has a valid IP address, default route, and DNS server, and can reach the controller.

Step 4: Go to Configuration > Prisma SD-WAN > Sites/Data Centers > Configurations and select the affected site. Under Connectivity and Circuits > Internet Circuits, select Change Internet Circuits, edit each circuit, and review Controller Connections:

Yes allows the circuit to carry device-initiated controller connections.

No excludes the circuit from controller connections.

Use Circuit Category Setting inherits the setting from its circuit category.

The circuit-level setting takes precedence over the circuit-category setting.

Step 5: If the excluded circuit is intentionally reserved as a last-resort path, leave it excluded and restore connectivity on at least one circuit that is allowed to reach the controller. Check the circuit, interface, ISP connection, routing, DNS, firewall, proxy, and NAT path as applicable.

If the circuit was excluded by mistake, correct the Controller Connections setting through SCM based on the intended design.

Step 6: If the circuit inherits its setting from a circuit category, go to Configuration > Prisma SD-WAN > Resources > Circuit Categories. Edit the category and verify Use For Controller Connections. Enable it only when circuits in that category are intended to carry device-initiated controller connections.

Step 7: After restoring an allowed path or correcting the configuration, repeat dump controller status and confirm that the controller connections use the source IP address of an allowed interface. Verify that Device State and Config and Events remain Online and that the incident clears.

Step 8: If permitted interfaces are Up and can reach the controller but the ION continues to use an excluded path, collect a support bundle and open a case with Palo Alto Networks Support.

dump-support all file=excluded_controller_path_<ion_name>

For earlier releases that do not support the all option:

dump-support outputs file=excluded_controller_path_<ion_name>

Replace <ion_name> with the affected ION’s name as shown on the Claimed Devices page.

Step 1: Determine whether the reboot was expected. Correlate the incident time with any administrator-initiated reboot, scheduled software upgrade, site maintenance, or known power interruption.

Step 2: Run the following read-only command:

dump overview

Review the Uptime and Last Reboot Reason fields. Also confirm the software version, device and HA states, controller connection, statistics connection, flows connection, and operational interfaces.

Step 3: If the reboot was associated with an approved administrative action or software upgrade, confirm that the reported software version is the expected version and that the ION has reconnected to the controller. If the device is operational and no related incidents remain active, no further action is required.

Step 4: If the reboot was unexpected, review the incidents generated during at least the 30 minutes preceding the reboot. Determine whether a critical process stop, general process stop, process restart, temperature, memory, controller-connectivity, or interface incident occurred immediately before DEVICESW_SYSTEM_BOOT. Also check the site’s power and UPS monitoring for a power interruption at the same time.

Step 5: Go to Insights > ION Devices > Device Activity and review the period beginning at least 30 minutes before the reboot through recovery. Check CPU utilization, free memory, free disk space, network utilization, interface utilization, dropped packets, interface errors, New TCP Flows, New UDP Flows, and Concurrent Flows. Determine whether an abnormal resource, bandwidth, or flow spike occurred immediately before the reboot.

Step 6: Go to Insights > Applications > TCP Connection Stats and review TCP initiation failures during the same period. Check Top 10 Apps by Initiation Failure and determine whether a significant increase in failed connections corresponds with the network-utilization or flow spike.

Confirm that utilization, flow counts, TCP initiation failures, and expected application traffic have returned to normal after the reboot. For an HA site, confirm that both IONs have returned to their expected active and standby states.

Step 7: If Last Reboot Reason is unknown, the reboot was unexpected, a process-stop or process-restart incident preceded the reboot, the ION repeatedly reboots, abnormal resource or flow activity is identified, or customer impact continues, collect a support bundle and open a Palo Alto Networks Support case:

dump-support all file=<descriptive_filename>

For releases earlier than 6.4.1 that do not support the all option:

dump-support outputs file=<descriptive_filename>

Use a filename containing the ION name and collection time, such as system_boot_branch-ion1_20260818T1430.

Contact support if this incident recurs frequently or remains unresolved.

Step 1: Review the incident details and record the affected ION, the upgrade time, and the source and target software versions.

Step 2: Go to Configuration > Prisma SD-WAN > ION Devices > Claimed Devices.

Step 3: Locate the affected device and click Upgrade in the Software column.

Step 4: Select the required software version and complete the upgrade. After the upgrade completes, repeat the procedure and downgrade the device to the target version.

Step 5: If the target version is the latest version and the incident persists, upgrade the device to the version immediately before the latest version, and then upgrade it to the latest version again.

Step 6: If the incident persists, contact Palo Alto Networks Support.

Step 1: Review the incident details and record the affected ION and the time the FRU SSD became unavailable.

Step 2: If the ION is online, run the following command and save the output:

dump device info

Step 3: Generate a support bundle:

dump-support file=fru-ssd-unavailable

Step 4: Attach the incident details, command output, and dump-support bundle to a Palo Alto Networks Support case.

Step 5: Do not reboot the ION unless instructed by Palo Alto Networks Support.

Step 1: Review the incident details and record the affected ION and the time the disk utilization reached the threshold.

Step 2: SSH to the affected ION and run:

dump disk info

Step 3: Review the output to identify the attached volume with low available space.

Step 4: Save the command output and contact Palo Alto Networks Support to clear the utilized volume.

Step 1: Review the affected ION under Insights > ION Devices > Device Activity. For the time surrounding the incident, review free memory, CPU utilization, and interface bandwidth utilization. Look for sustained utilization or a pattern that correlates with the memory increase.

Step 2: For the same period, review New TCP Flows, New UDP Flows, and Concurrent Flows. Check for an unusual increase or sustained flow load.

Step 3: If the flow metrics are elevated, use Flow Browser to identify the applications and source or destination IP addresses generating excessive sessions. Address any unexpected traffic or misbehaving host, and then confirm that the flow count and memory utilization return to normal.

Step 4: If the device utilization and flow metrics are normal, or memory remains high, SSH to the ION and run:

inspect memory summary

inspect process status

debug process status all

Step 5: Review MemAvailable, SwapTotal, and SwapFree. Check for processes consuming unusually high CPU or memory, and save the service-status output. Some unused services may normally appear as stopped.

Step 6: If memory returns to normal and the incident clears, continue monitoring for recurrence.

Step 7: If memory remains high, the incident recurs, or the CLI output indicates a possible process or service issue, attach the Device Activity findings, flow information, CLI outputs, incident time, and software version to a Palo Alto Networks Support case.

Step 8: Do not stop or restart services, processes, or the ION unless instructed by Palo Alto Networks Support.

Step 1: Review the incident details and record the affected ION, incident time, and event code.

Step 2: Open Device Activity for the affected ION. Review New TCP Flows, New UDP Flows, and Concurrent Flows around the incident time. Determine whether there is an unusual increase or sustained flow load.

Step 3: In the Applications view, review the TCP Connection Stats chart for initiation failures during the same time period. Compare the trend with New TCP Flows, New UDP Flows, and Concurrent Flows.

Step 4: If initiation failures correlate with increased flow activity, filter the chart by Top 10 Apps by Initiation Failure. Identify the applications contributing the most failures, then use Flow Browser to review the associated source and destination IP addresses. Check for unexpected traffic, scanning activity, or a misbehaving host or application.

Step 5: SSH to the ION and run:

dump flow count-summary

dump device conntrack count

Step 6: Review the active TCP, UDP, and total-flow counts. For the conntrack command, compare the current count with the reported maximum value.

Step 7: If unexpected traffic is causing the high flow count, correct or contain the source of the traffic. If the traffic is expected and sustained, validate whether the ION is appropriately sized for the site.

Step 8: Confirm that the flow and conntrack counts return below the applicable threshold and that the incident clears.

Step 9: If the condition persists or recurs, collect a support bundle and contact Palo Alto Networks Support.

For Release 6.4.1 or later:

dump-support all file=flow-capacity-limit

For earlier supported releases:

dump-support outputs file=flow-capacity-limit

Step 1: Review the incident details and record the affected ION, incident time, and event code.

Step 2: Open Device Activity for the affected ION. Review New TCP Flows, New UDP Flows, and Concurrent Flows around the incident time. Determine whether there is an unusual increase or sustained flow load.

Step 3: In the Applications view, review the TCP Connection Stats chart for initiation failures during the same time period. Compare the trend with New TCP Flows, New UDP Flows, and Concurrent Flows.

Step 4: If initiation failures correlate with increased flow activity, filter the chart by Top 10 Apps by Initiation Failure. Identify the applications contributing the most failures, then use Flow Browser to review the associated source and destination IP addresses. Check for unexpected traffic, scanning activity, or a misbehaving host or application.

Step 5: SSH to the ION and run:

dump flow count-summary

dump device conntrack count

Step 6: Review the active TCP, UDP, and total-flow counts. For the conntrack command, compare the current count with the reported maximum value.

Step 7: If unexpected traffic is causing the high flow count, correct or contain the source of the traffic. If the traffic is expected and sustained, validate whether the ION is appropriately sized for the site.

Step 8: Confirm that the flow and conntrack counts return below the applicable threshold and that the incident clears.

Step 9: If the condition persists or recurs, collect a support bundle and contact Palo Alto Networks Support.

For Release 6.4.1 or later:

dump-support all file=flow-capacity-limit

For earlier supported releases:

dump-support outputs file=flow-capacity-limit

Step 1: Review the incident details and record the affected ION, incident time, and event code.

Step 2: Open Device Activity for the affected ION. Review New TCP Flows, New UDP Flows, and Concurrent Flows around the incident time to identify unusual spikes or sustained flow load.

Step 3: In the Applications view, review the TCP Connection Stats chart for initiation failures during the same time period. Compare the trend with New TCP Flows, New UDP Flows, and Concurrent Flows.

Step 4: If initiation failures correlate with increased flow activity, filter the chart by Top 10 Apps by Initiation Failure. Identify the applications contributing the most failures, then use Flow Browser to review the associated source and destination IP addresses. Check for unexpected traffic, scanning activity, or a misbehaving host or application.

Step 5: SSH to the affected ION and run:

dump flow count-summary

dump device conntrack count

dump nat counters all

If the site uses IPv6 NAT and the ION runs Release 6.4.2 or later, also run:

dump nat6 counters all

Step 6: Review the active TCP, UDP, and total-flow counts. Compare the current conntrack count with the reported maximum value. Review NAT counters for rules with unusually high packet or byte counts that correlate with the flow increase.

Step 7: If unexpected traffic is causing the high flow count, correct or contain the traffic source. If the traffic is expected and sustained, validate whether the ION is appropriately sized for the site.

Step 8: Confirm that flow and conntrack counts return below the applicable threshold and that the incident clears.

Step 9: If the condition persists or recurs, collect a support bundle and contact Palo Alto Networks Support.

For Release 6.4.1 or later:

dump-support all file=flow-capacity-limit

For earlier supported releases:

dump-support outputs file=flow-capacity-limit

Step 1: Review the incident details and record the affected ION, reported process name, incident time, and whether the alert is recurring.

Step 2: In the Remote CLI Toolkit, run:

dump overview

debug controller reachability <interface>

Verify that the ION is connected to the controller and that controller reachability succeeds on the controller-facing interface.

Step 3: Review the period immediately before and during the process restart in System Health and Device Activity. Check CPU and memory utilization, interface errors or drops, network utilization, New TCP Flows, New UDP Flows, and Concurrent Flows.

Step 4: In the Applications view, review TCP Connection Stats, including initiation failures and unusual changes in successful TCP session activity.

Step 5: Correlate these trends with the incident time to identify any abnormal condition before the restart. For example, determine whether increased TCP or UDP flows coincided with high CPU or memory use, interface drops, high traffic utilization, or initiation failures.

Step 6: Check the process status:

debug process status all

inspect process status

If the incident identifies a specific process, also run:

debug process status name=<process_name>

Confirm that the process is running and remains stable.

Step 7: If the review identifies a clear traffic, resource, or interface issue, correct or contain that condition and monitor for recurrence. If no clear cause is identified, or if the process restart recurs or affects traffic, collect a support bundle and open a Palo Alto Networks TAC case. TAC can review the support bundle and the logs related to the affected service to determine the root cause.

For Release 6.4.1 or later:

dump-support all file=critical-process-restart

For earlier supported releases:

dump-support outputs file=critical-process-restart

Step 8: Do not manually restart the process or reboot the ION unless instructed by TAC.

Step 1: Review the incident details and record the affected ION, reported process name, incident time, and any reported traffic or application impact.

Step 2: Determine whether the event caused a service impact. Verify that the affected site remains online and that traffic and critical business applications are operating normally. If the site uses HA, verify that the peer ION is active and carrying traffic.

Step 3: SSH to the affected ION and run:

dump overview

debug controller reachability <interface>

Verify that the ION is fully connected to the controller.

Step 4: Review the period immediately before and during the process stop in System Health and Device Activity. Check CPU and memory utilization, interface errors or drops, network utilization, New TCP Flows, New UDP Flows, and Concurrent Flows.

Step 5: In the Applications view, review TCP Connection Stats, including initiation failures and unusual changes in successful TCP session activity.

Step 6: Correlate these trends with the incident time to identify any abnormal condition before the stop. For example, determine whether increased TCP or UDP flows coincided with high CPU or memory use, interface drops, high traffic utilization, or initiation failures.

Step 7: Check the process status:

debug process status all

inspect process status

If the incident identifies a process name, also run:

debug process status name=<process_name>

Step 8: If the review identifies an abnormal condition that may have contributed to the process event, such as unexpected traffic, high resource utilization, or interface errors, address the underlying condition where possible and monitor the ION for recurrence.

Step 9: Collect device details, check for core files, and collect a support bundle:

file list core

For Release 6.4.1 or later:

dump-support all file=critical-process-stop

For earlier supported releases:

dump-support outputs file=critical-process-stop

Step 10: If no clear cause is identified, or if the process stop recurs or affects traffic, open a Palo Alto Networks Support case immediately. Attach the incident details, process-status output, dump overview output, controller-status output, Device Activity findings, support bundle, and any core-file names.

Step 11: Do not manually restart the process or reboot the ION unless instructed by TAC.

Step 1: Review the incident details and record the affected ION, incident time, software version, hardware model, and any related concurrent-flow, conntrack, CPU, memory, or process incidents.

Step 2: Open Device Activity for the affected ION and review New TCP Flows, New UDP Flows, and Concurrent Flows for the incident period. Determine whether the event was caused by a short traffic burst or a sustained increase in new flows.

Step 3: In Application Insights, review the New Flows chart to identify the applications contributing to the increase. Also review TCP Connection Stats for increased initiation failures during the same period.

Step 4: Use Flow Browser with a time range surrounding the incident. Identify the source and destination IP addresses, applications, destination ports, protocols, and traffic direction generating the largest number of new flows. Check whether a single source, application, or subnet is generating an abnormal number of short-lived sessions.

Step 5: Access the ION through the Remote CLI Toolkit or SSH and run:

dump flow count-summary

Review the TCP, UDP, other, and active-flow counts and the FPS in previous second value. Because this value represents only the previous second, run the command several times while the incident is active to determine whether the high flow rate is continuing.

If a suspicious source IP was identified, review its currently active flows:

inspect flow detail srcv4=<source_ip>

For an IPv6 source:

inspect flow detail srcv6=<source_ipv6>

Step 6: If Concurrent Flows are also approaching the device limit, or a related DEVICESW_CONCURRENT_FLOWLIMIT_EXCEEDED or DEVICESW_CONCURRENT_FLOW_SOFTLIMIT_EXCEEDED incident is active, follow the linked remediation for that event.

Step 7: If the high flow rate is unexpected, investigate the identified source for scanning activity, malware, a connection loop, or a misbehaving application. Contain or rate-limit the traffic at the source, access switch, firewall, or other appropriate enforcement point.

If the traffic is generated by an authorized scanner, consider defining it as a custom application and enabling Network Scan App under Configuration > Prisma SD-WAN > Resources > Applications. For a Network Scan App, disable Use Unreachability Detection and set Path Affinity to None. This classification helps protect legitimate application sessions, but it does not replace rate limiting at the scanner when its flow rate is excessive.

Step 8: Confirm that New TCP and UDP Flows and the CLI FPS value have returned to normal, initiation failures are no longer increasing, and the incident clears.

Step 9: If the condition persists or recurs without an identifiable traffic source, collect a support bundle:

dump-support all file=fps-limit-exceeded

For earlier releases that do not support the all option:

dump-support outputs file=fps-limit-exceeded

Attach the incident details, Device Activity findings, Flow Browser results, CLI outputs, device model, software version, and support bundle to a Palo Alto Networks Support case. Do not clear flows, restart processes, or reboot the ION unless instructed by Support.

Step 1: Review the incident details and record the affected ION, process name, incident time, software version, and whether the process has restarted more than once.

Step 2: Review related incidents and determine whether the restart caused any site, application, management, or traffic impact. If the restart occurred once, the process recovered, and no impact is observed, continue monitoring.

Step 3: Review System Health and Device Activity for the period surrounding the restart. Check CPU and memory utilization, interface errors and drops, bandwidth utilization, New TCP Flows, New UDP Flows, and Concurrent Flows. Also check for a recent software upgrade, device reboot, or configuration change.

Step 4: If the incident recurs or an impact is observed, access the ION through the Remote CLI Toolkit or SSH and run:

dump overview

debug process status all

inspect process status

Check the affected process specifically:

debug process status name=<process_name>

Confirm that the process is running and review its uptime, CPU usage, and memory consumption. Some unused or optional services may normally appear as stopped in the complete process list.

Step 5: If the process remains stable and the incident does not recur, continue monitoring. If the process repeatedly restarts, consumes abnormal resources, or affects functionality, collect a support bundle:

dump-support all file=general-process-restart

For earlier releases that do not support the all option:

dump-support outputs file=general-process-restart

Attach the incident details, process-status output, Device Activity findings, software version, and support bundle to a Palo Alto Networks Support case. Do not manually restart the process or reboot the ION unless instructed by Support.

Step 1: Review the incident details and record the affected ION, process name, incident time, software version, and any related process or system-resource incidents.

Step 2: Determine which device function is provided by the stopped process and verify whether there is any site, application, management, monitoring, or traffic impact.

Step 3: Review System Health and Device Activity for the period immediately before and during the process stop. Check CPU and memory utilization, interface errors and drops, bandwidth utilization, New TCP Flows, New UDP Flows, and Concurrent Flows. Also check for a recent software upgrade, device reboot, or configuration change.

Step 4: Access the ION through the Remote CLI Toolkit or SSH and run:

dump overview

debug process status all

inspect process status

Check the affected process specifically:

debug process status name=<process_name>

Confirm whether the process remains stopped or has recovered. Some unused or optional services may normally appear as stopped, so evaluate the process identified in the incident rather than relying only on the complete process list.

Step 5: Check whether a core file was generated:

file list core

Do not manually start or restart the stopped process.

Step 6: If the process has recovered, verify that the affected functionality is operational and monitor for recurrence. If the process remains stopped, the incident recurs, or functionality is affected, collect a support bundle:

dump-support all file=general-process-stop

For earlier releases that do not support the all option:

dump-support outputs file=general-process-stop

Step 7: Open a Palo Alto Networks Support case and attach the incident details, process-status output, Device Activity findings, any core-file names, software version, and support bundle. Do not manually restart the process or reboot the ION unless instructed by Support.

Step 1: Review the incident details and record the affected site, ION, incident time, software version, and the reported CPU temperature.

Step 2: Run dump poe system status to confirm the PoE shutdown state and any reported temperature information.

Step 3: Inspect the ION's physical installation. Verify that the device has adequate ventilation, air vents are unobstructed, and the device is not exposed to excessive ambient heat or direct sunlight.

Step 4: Review related DEVICEHW_TEMPERATURE_SENSOR incidents at the same site and time.

Step 5: Monitor device temperature. PoE operations resume automatically when the temperature returns to a value below the threshold.

Step 6: If the CPU temperature remains elevated or the thermal event recurs, collect a support bundle and open a Palo Alto Networks Support case for hardware assessment. Run dump-support all file=<descriptive_filename> for Release 6.4.1 and later or dump-support outputs file=<descriptive_filename> for earlier releases.

Step 1: Review the incident details and record the affected site, ION, incident time, software version, source identifier, observed flow count, and configured threshold.

Step 2: In SCM, go to Configuration > Prisma SD-WAN > Profiles and Templates > Site Protection and open the profile applied to the affected site.

Step 3: Identify the source generating the flows and determine whether the activity is expected business traffic or an anomaly such as scanning or a misconfigured application.

Step 4: If the flow volume is legitimate, review the per-source threshold and adjust it only within the supported concurrent-flow capacity of the ION model.

Step 5: If the flow volume is unexpected, contain or correct the source and preserve relevant security and flow evidence.

Step 6: Monitor the per-source flow count and confirm that the incident clears.

Step 7: If the condition persists or affects service, collect a support bundle and open a Palo Alto Networks Support case.

Step 1: Review the incident details and record the affected site, ION, incident time, software version, NAT stack, NAT set, rule, traffic selector, and static NAT pool.

Step 2: In SCM, review the affected NAT policy rule and compare the number of addresses in the matching traffic selector with the number of addresses in the static NAT pool.

Step 3: Expand the static NAT pool so that every address in the matched range can be mapped one-to-one, or narrow the traffic selector to the addresses that require static mapping.

Step 4: Save and push the corrected configuration, then confirm that the mapping sizes are compatible and the incident clears.

Step 5: If the configuration cannot be corrected within the available address space or the incident persists, collect a support bundle and open a Palo Alto Networks Support case.

Step 1: Review the incident details and record the affected site, ION, incident time, software version, Path Policy stack, policy set, and rules identified as dropped.

Step 2: In SCM, go to Configuration > Prisma SD-WAN > Policies > Paths > Path Stacks.

For a simple stack, select Simple and open the applicable Path Policy stack. For an advanced stack, select Advanced, open the applicable Path Policy stack and Path Set, and review the rules identified in the incident.

Step 3: Review the applications and source and destination prefixes configured in each affected rule. Remove unnecessary applications or prefixes to reduce the number of rule permutations while preserving the required policy behavior.

Step 4: Save the changes through SCM and confirm that the corrected Path Policy stack is bound to the affected site.

Step 5: Monitor the affected site and confirm that all Path Policy rules remain active and the incident clears.

Step 6: If the policy cannot be simplified without losing required behavior, or rules continue to be dropped, collect a support bundle and open a Palo Alto Networks Support case:

dump-support all file=<descriptive_filename>

For releases earlier than 6.4.1 that do not support the all option:

dump-support outputs file=<descriptive_filename>

Use a filename containing the event, ION name, and collection time, such as network_policy_rule_dropped_branch-ion1_YYYYMMDDTHHMM.

Step 1: Review the incident details and record the affected site, ION, incident time, software version, QoS stack, QoS set, and dropped rules.

Step 2: In SCM, go to Configuration > Prisma SD-WAN > Policies > QoS > QoS Stacks. For an advanced stack, select Advanced > QoS Sets and open the affected set.

Step 3: Run inspect priority-policy dropped and compare the reported rules and entry counts with the SCM configuration.

Step 4: Simplify affected rules by reducing or dividing applications, source prefixes, or destination prefixes while preserving the intended QoS behavior.

Step 5: Save and push the corrected configuration, run inspect priority-policy dropped again, and confirm that all required rules are active and the incident clears.

Step 6: If the rule cannot be reduced within supported limits, collect a support bundle and open a Palo Alto Networks Support case.

Step 1: Review the incident details and record the affected site, ION, incident time, software version, Security stack and set, and the reported limit type and value.

Step 2: In SCM, go to Configuration > Prisma SD-WAN > Policies > Security > Security Stacks and open the affected Simple or Advanced policy. For an advanced stack, review the applicable Security Set.

Step 3: Run inspect security-policy size to review the compiled security-policy size and resource usage reported by the ION.

Step 4: Identify rules or objects that can be consolidated, narrowed, or removed without weakening the required security behavior.

Step 5: Save and push the corrected policy, run inspect security-policy size again, and confirm that the policy is within the reported limit and the incident clears.

Step 6: If the policy cannot be reduced within supported limits, collect a support bundle and open a Palo Alto Networks Support case.

Step 1: Review the incident details and record the affected site, ION, incident time, software version, and current CPU utilization relative to the configured threshold.

Step 2: In SCM, go to Configuration > Prisma SD-WAN > Policies > Performance > Performance Sets, open the applicable System Health rule, and review the CPU utilization threshold configured for the affected ION.

Step 3: Review CPU utilization trends to determine whether the exceedance is a temporary burst or a sustained condition.

Step 4: Check for related DEVICESW_CRITICAL_PROCESSRESTART, DEVICESW_CRITICAL_PROCESSSTOP, or DEVICESW_CONCURRENT_FLOWLIMIT_EXCEEDED incidents at the same site and time.

Step 5: Identify whether the high CPU usage coincides with high concurrent flow counts, a software event, or an unexpected traffic volume.

Step 6: Monitor the CPU utilization to confirm it returns below the threshold.

Step 7: If the CPU utilization remains high or service is affected, run dump-support all file=<descriptive_filename> (Release 6.4.1 and later) or dump-support outputs file=<descriptive_filename> (earlier releases), where <descriptive_filename> is a user-selected name such as event_name_ion-name_YYYYMMDDTHHMM.

Step 1: Review the incident details and record the affected site, ION, incident time, software version, and current disk utilization relative to the configured threshold.

Step 2: In SCM, go to Configuration > Prisma SD-WAN > Policies > Performance > Performance Sets, open the applicable System Health rule, and review the disk utilization threshold configured for the affected ION.

Step 3: Review disk utilization trends to determine whether the exceedance is a temporary condition or a sustained trend.

Step 4: Check for related DEVICEHW_DISKUTIL_PARTITIONSPACE or DEVICEHW_DISKUTIL_FRUSSD incidents at the same site.

Step 5: Monitor disk utilization to confirm it returns below the threshold.

Step 6: If disk utilization remains persistently high, run dump-support all file=<descriptive_filename> (Release 6.4.1 and later) or dump-support outputs file=<descriptive_filename> (earlier releases), where <descriptive_filename> is a user-selected name such as event_name_ion-name_YYYYMMDDTHHMM.

Step 1: Review the incident details and record the affected site, ION, incident time, software version, and current memory utilization relative to the configured threshold.

Step 2: In SCM, go to Configuration > Prisma SD-WAN > Policies > Performance > Performance Sets, open the applicable System Health rule, and review the memory utilization threshold configured for the affected ION.

Step 3: Run inspect memory summary to review the current memory utilization breakdown.

Step 4: Review memory utilization trends and check for related DEVICESW_CONCURRENT_FLOWLIMIT_EXCEEDED or DEVICESW_CRITICAL_PROCESSRESTART incidents at the same site and time.

Step 5: Identify whether the high memory usage coincides with high concurrent flow counts, a software event, or an unexpected traffic volume.

Step 6: Monitor memory utilization to confirm it returns below the threshold.

Step 7: If memory utilization remains persistently high or service is affected, run dump-support all file=<descriptive_filename> (Release 6.4.1 and later) or dump-support outputs file=<descriptive_filename> (earlier releases), where <descriptive_filename> is a user-selected name such as event_name_ion-name_YYYYMMDDTHHMM.