SD-WAN
Troubleshoot KeyID Mismatch
Table of Contents
Expand All
|
Collapse All
SD-WAN Docs
-
- SD-WAN Deployment Workflow
-
- Add SD-WAN Branch or Hub Firewall
- Configure Certificate-based Authentication for Strong Security
- Quickly Add Multiple SD-WAN Devices with Bulk Import
- Configure SD-WAN Devices in HA Mode
- Onboard PAN-OS Firewalls to Prisma Access for Cloud-based Security
- Plan Your Topology for SD-WAN with Auto VPN
- Create a Full Mesh VPN Cluster with DDNS Service
- Create a Static Route for SD-WAN
- Configure Advanced Routing for SD-WAN
Troubleshoot KeyID Mismatch
Learn why KeyID mismatch error occurs and how to resolve or prevent keyID mismatch
error in the full mesh or hub and spoke topology.
| Where Can I Use This? | What Do I Need? |
|---|---|
|
You may encounter a keyID mismatch error in your SD-WAN deployments like, hub and spoke
or full mesh deployments. This error commonly occurs when you replace a HA pair in the
SD-WAN cluster, upgrade a HA pair, or clear the SD-WAN plugin cache:
- For deployments earlier to SD-WAN plugin version 2.2.5, 3.0.8, 3.1.3, 3.2.1, or 3.3.0, this issue is seen after the RMA process (that is, after replacing a faulty firewall).
- For the newer SD-WAN plugin versions, this issue is seen when replacing one of the HA pairs with a serial number that’s lower than either of the pairs prior to replacement.
- For the same plugin versions, this issue is seen when clearing the SD-WAN plugin cache. It is not recommended to clear the SD-WAN cache; however, if the cache is cleared, a commit and push to all members of the cluster is mandatory.
Now, let us understand why the keyID error occurs in these cases. When you commit and
push to a few of the SD-WAN devices in the SD-WAN cluster after making these changes,
the SD-WAN device will fail to establish all the SD-WAN tunnels throwing the keyID
mismatch error. The SD-WAN firewall logs the keyID mismatch error in an IKE or system
log file.
The SD-WAN firewall uses the following mechanism to generate the keyID and other
attributes of the SD-WAN tunnel:
(For versions earlier to SD-WAN plugin 2.2.5, 3.0.8, 3.1.3, 3.2.1, and 3.3.0)
During autoprovisioning, the SD-WAN hub firewall uses the serial number of the branch
firewall in the VPN cluster to generate IKE gateway name, IPSec tunnel name, and keyID
values. Similarly, the branch firewall uses the serial number of the hub firewall to
generate the IKE gateway name, IPSec tunnel name, and keyID values.
(For SD-WAN plugin 2.2.5, 3.0.8, 3.1.3, 3.2.1, and 3.3.0 versions) The SD-WAN
firewall uses the lower serial number among the HA pair to generate the IKE gateway
name, IPSec tunnel name, and keyID values.
Any changes to the firewall's serial number in the VPN cluster will impact the SD-WAN
tunnel establishment, as the firewall's serial number is associated with the keyID and
other attributes of the SD-WAN tunnel.
Consider an example hub and spoke topology, where a hub site is connected with a HA
branch site. Let's see how the hub uses the branch firewall's serial number and the
branch uses the connected hub's serial number in its SD-WAN tunnel attributes. Note the
following hub and branch firewall states that show the usage of serial numbers in the
tunnel names.
Hub (in working
state):
admin@SDWAN-hub> show system state | match info.serial sys.s1.info.serial: 007251000362133 admin@SDWAN-hub> show vpn flow total tunnels configured: 3 filter - type IPSec, state any total IPSec tunnel configured: 3 total IPSec tunnel shown: 3 id name state monitor local-ip peer-ip tunnel-i/f mode -- -------------- ----- ------- -------- ------- ---------- ---- 1 tl_0101_007251000362138_0101 active up 192.0.2.2 172.16.20.2 tunnel.912 tunnel 4 tl_0101_007251000362138_0102 active up 192.0.2.2 172.16.40.2 tunnel.913 tunnel admin@SDWAN-hub# show template network ike | match "local-id\|peer-id" set network ike gateway gw_0101_007251000362138_0101 local-id id 00725100036213801010072510003621330101 set network ike gateway gw_0101_007251000362138_0101 peer-id id 00725100036213801010072510003621330101 set network ike gateway gw_0101_007251000362138_0102 local-id id 00725100036213801020072510003621330101 set network ike gateway gw_0101_007251000362138_0102 peer-id id 00725100036213801020072510003621330101
Branch (in working state):
admin@SDWAN-branch(active)> show system state | match info.serial sys.s1.info.serial: 007251000362138 peer.sys.s1.info.serial: 007251000362134 admin@SDWAN-branch(active)> show vpn flow total tunnels configured: 2 filter - type IPSec, state any total IPSec tunnel configured: 2 total IPSec tunnel shown: 2 id name state monitor local-ip peer-ip tunnel-i/f mode -- -------------- --- ------- -------- ------- ----------- ---- 1 tl_0101_007251000362133_0101 active up 198.51.100.1 192.0.2.2 tunnel.900 tunnel 4 tl_0102_007251000362133_0101 active up 198.51.100.8 192.0.2.2 tunnel.901 tunnel
In the above example, the hub firewall references the serial of the active branch device
in its tunnel name convention. Similarly, the branch firewall references the serial of
the hub device in the tunnel name. Further, the keyID configuration on each peer is
derived from the respective serial numbers. For example, the CLI output from the hub
firewall above shows the local and peer keyID for the tunnel gateway
gw_0101_007251000362138_0101 as:
00725100036213801010072510003621330101 and
00725100036213801010072510003621330101 respectively.
Now, consider that the SD-WAN plugin managing the hub and spoke topology is upgraded to
version 3.0.8. This results in a recalculation of the serial numbers and redetermining
the lower of the two serials for the HA peer: that is, the serial number of the passive
branch device (007251000362134) will be used for the IKE gateway name, IPSec tunnel name
as well as the keyID. After the upgrade, if a commit push was executed only to the hub
device, then the new tunnel parameters, including a new keyID will be generated; but
since there was no simultaneous commit and push to the branch and as the branch firewall
is still using the old keyID, it will lead to a keyID mismatch.
After upgrade, the tunnel configurations and hub tunnel attribute have changed as
follows.
Branch (in broken state):
admin@SDWAN-hub> show vpn flow total tunnels configured: 3 filter - type IPSec, state any total IPSec tunnel configured: 3 total IPSec tunnel shown: 3 id name state monitor local-ip peer-ip tunnel-i/f mode -- -------------- ----- ------- -------- ------- ---------- ---- 1 tl_0101_007251000362134_0101 inactiv down 192.0.2.2 172.16.20.2 tunnel.912 tunnel 4 tl_0101_007251000362134_0102 inactiv down 192.0.2.2 172.16.40.2 tunnel.913 tunnel admin@SDWAN-hub# show template network ike | match "local-id\|peer-id" set network ike gateway gw_0101_007251000362134_0101 local-id id 00725100036213401010072510003621330101 set network ike gateway gw_0101_007251000362134_0101 peer-id id 00725100036213401010072510003621330101 set network ike gateway gw_0101_007251000362134_0102 local-id id 00725100036213401020072510003621330101 set network ike gateway gw_0101_007251000362134_0102 peer-id id 00725100036213401020072510003621330101
Following is the sample IKE log of the hub firewall with keyID mismatch error.
2025-01-21 19:54:41.680 -0800 [PERR]: { 1: }: 198.51.100.1[500] - 192.0.2.2[500]:0x556c1e76bde0 received ID_I (type keyid [00725100036213801010072510003621330101]) does not match peers id
2025-01-21 19:54:41.680 -0800 [INFO]: { 1: }: 192.0.2.2[500] - 198.51.100.1[500]:(nil) closing IKEv2 SA gw_0101_007251000362134_0101:74, code 13
2025-01-21 19:54:41.680 -0800 [PNTF]: { 1: }: ====> IKEv2 IKE SA NEGOTIATION FAILED AS RESPONDER, non-rekey; gateway gw_0101_007251000362134_0101 <====While the hub firewall is using the new keyID, the branch firewall is still using the old
keyID, which resulted in the keyID mismatch.
To avoid keyID mismatch error, ensure the following:
- If the devices in the SD-WAN cluster have direct reachability to Panorama outside of the SD-WAN tunnels, commit and push to all devices in the cluster. This ensures that every device has the matching keyID parameters.
- In both the hub-and-spoke and full mesh topologies, override the gateway
configuration on one of the branch (or hub) devices and set the correct keyID values
to match the values of the corresponding peer followed by a commit. After
reestablishing connectivity to Panorama, revert the IKE gateway changes by selecting
revert after enabling the IKE gateway
name (NetworkNetwork ProfilesIKE Gateways). Do not commit after reverting. Instead, commit and push from
Panorama and ensure that Merge with Device Candidate Config
is selected in the Push Selection Scope.The local identification and peer identification may not always be the same value. Therefore, when setting the local and peer identification, be sure to copy the values exactly. Copy the local identification value from one gateway and then on the corresponding overriden gateway, paste this value into the peer identification field; similarly, copy the peer identification value from the gateway and paste it into the local identification field on the corresponding overriden gateway.