CN-Series
Test Case: CN-NGFW Failure Handling
Table of Contents
Test Case: CN-NGFW Failure Handling
| Where Can I Use This? | What Do I Need? |
|---|---|
|
|
This test evaluates CN-NGFW failure handling.
CN-NGFW failure can happen under the following circumstances.
- Node issues
- CN-NGFW pod crashes and restarts
- Node and CN-NGFW pod are okay but pan_task crashes
- CN-NGFW is removed from cluster membership when:
- IPsec monitoring over Eth0 interface fails
- Cluster Interconnect (CI) link is broken
- Traffic Interconnect (TI) link is broken
In this scenario, the SSH session between client and server is installed on CN-NGFW
1. If the CN-NGFW 1 is down, the SSH session must be kept alive by failover to
another CN-NGFW.
- From the Panorama CLI, enter show clusters name
<cluster-name> to view CN-NGFW, CN-DB, and CN-GW pods connected
to the CN-MGMT pod.Cluster: cluster-002 Creation time: 2022/11/22 04:56:46 CN-MGMT pods: 87F87FE94CBBB03 (active, pan-mgmt-sts-0.cluster-002, connected, In Sync) Slot-ID PodName Type Version ---------------------------------------------------------------------------------------- 1 pan-gw-dep-5cd5c87d76-przjx CN-GW 11.0.1-c156.dev_e_rel 6 pan-db-dep-d6fb496b-jf2ms CN-DB 11.0.1-c156.dev_e_rel 5 pan-ngfw-dep-5cd8f55848-dbhwh CN-NGFW 11.0.1-c156.dev_e_rel 8 pan-ngfw-dep-5cd8f55848-slk5l CN-NGFW 11.0.1-c156.dev_e_rel 7 pan-db-dep-d6fb496b-hfmlp CN-DB 11.0.1-c156.dev_e_rel 9 pan-ngfw-dep-5cd8f55848-pq6ks CN-NGFW 11.0.1-c156.dev_e_rel 2 pan-gw-dep-5cd5c87d76-4kbfk CN-GW 11.0.1-c156.dev_e_rel 11 pan-ngfw-dep-5cd8f55848-rsbqn CN-NGFW 11.0.1-c156.dev_e_rel
- View cluster membership details of CN-MGMT pod an-mgmt-sts-0 using the command show cluster-membership show-slot-info slot all.
MP leader status: Leader Slot-id Type CI-IP TI-IP State CI-State TI-State ======================================================================================== 1 CN-GW 192.168.23.100 192.168.24.80 UP UP UP 11 CN-NGFW 192.168.23.87 192.168.24.93 UP UP UP 2 CN-GW 192.168.23.101 192.168.24.100 UP UP UP 7 CN-DB 192.168.23.102 :: UP UP NA 6 CN-DB 192.168.23.104 :: UP UP NA 5 CN-NGFW 192.168.23.103 192.168.24.86 UP UP UP 8 CN-NGFW 192.168.23.105 192.168.24.84 UP UP UP 9 CN-NGFW 192.168.23.82 192.168.24.81 UP UP UPAll interfaces of the ethernetx/3 subnet must be in the same zone. Similarly, all interfaces of the ethernetx/4 subnet must be in the same zone.- Use show session all filter application ssh to view all SSH sessions.
For every session, there are two flows for Client to Server and Server to Client directions.-------------------------------------------------------------------------------- ID Application State Type Flag Src[Sport]/Zone/Proto (translated IP[Port]) Vsys Dst[Dport]/Zone (translated IP[Port]) -------------------------------------------------------------------------------- 1342177294 ssh ACTIVE FLOW 192.168.200.100[48702]/untrust_ei1/6 (192.168.200.100[48702]) vsys1 192.168.250.100[22]/trust_ei2 (192.168.250.100[22]) admin@pan-mgmt-sts-1.cluster-001> show session id 1342177294 Session 1342177294 c2s flow: source: 192.168.200.100 [untrust_ei1] dst: 192.168.250.100 proto: 6 sport: 48702 dport: 22 state: ACTIVE type: FLOW src user: unknown dst user: unknown s2c flow: source: 192.168.250.100 [trust_ei2] dst: 192.168.200.100 proto: 6 sport: 22 dport: 48702 state: ACTIVE type: FLOW src user: unknown dst user: unknown Slot : 11 DP : 0 index(local): : 14 start time : Mon Nov 21 21:30:02 2022 timeout : 3600 sec time to live : 3542 sec total byte count(c2s) : 3887 total byte count(s2c) : 4501 layer7 packet count(c2s) : 23 layer7 packet count(s2c) : 20 vsys : vsys1 application : ssh rule : allow_inside-to-outside service timeout override(index) : False session to be logged at end : True session in session ager : True session updated by HA peer : False layer7 processing : completed URL filtering enabled : True URL category : any session via syn-cookies : False session terminated on host : False session traverses tunnel : False session terminate tunnel : False captive portal session : False ingress interface : ethernet1/3 egress interface : ethernet1/4 session QoS rule : N/A (class 4) tracker stage l7proc : ctd decoder done end-reason : unknownThe session owner is Slot 11.You can view the filtered cluster flow details using the following example command.show cluster-flow all filter source-port 22Output:-------------------------------------------------------------------------------- Slot 5 -------------------------------------------------------------------------------- Id State Type Src[Sport]/Proto Dst[Dport] -------------------------------------------------------------------------------- 536870940 ACTIVE FLOW 192.168.250.100[22]/6 192.168.200.100[48702] -------------------------------------------------------------------------------- Slot 6 -------------------------------------------------------------------------------- Id State Type Src[Sport]/Proto Dst[Dport] -------------------------------------------------------------------------------- 671088668 ACTIVE FLOW 192.168.250.100[22]/6 192.168.200.100[48702]show cluster-flow all filter destination-port 22Output:-------------------------------------------------------------------------------- Slot 5 -------------------------------------------------------------------------------- Id State Type Src[Sport]/Proto Dst[Dport] -------------------------------------------------------------------------------- 536870939 ACTIVE FLOW 192.168.200.100[48702]/6 192.168.250.100[22] -------------------------------------------------------------------------------- Slot 6 -------------------------------------------------------------------------------- Id State Type Src[Sport]/Proto Dst[Dport] -------------------------------------------------------------------------------- 671088667 ACTIVE FLOW 192.168.200.100[48702]/6 192.168.250.100[22]- Delete the pod on Slot 11 using the command kubectl -n kube-system delete pod pan-ngfw-dep-5cd8f55848-rsbqn.
Output:pod "pan-ngfw-dep-5cd8f55848-rsbqn" deletedThe session owned by the CN-NGFW pod in Slot 11 is now marked as orphan.admin@pan-mgmt-sts-1.cluster-001> set system setting target-dp s5dp0 Session target dp changed to s6dp0 admin@pan-mgmt-sts-1.cluster-001> show cluster-flow id 536870939 Flow 536870939 start time : Mon Nov 21 21:30:02 2022 timeout : 3600 sec source : 192.168.200.100 sport : 48702 dest : 192.168.250.100 dport : 22 proto : 6 zone : 1 type : FLOW state : ACTIVE ipver : 4 fidx : 28 cid : 0 gft : 0 gft' : 1 predict : 0 orphan : 1 flag_inager : 0 ager_thread : 3 flags : 0 flow-data : type: l7 app-id: 25 startlog: 1 endlog: 1 denied: 0 admin@pan-mgmt-sts-1.cluster-001> set system setting target-dp s6dp0 Session target dp changed to s6dp0 admin@pan-mgmt-sts-1.cluster-001> show cluster-flow id 671088667 Flow 671088667 start time : Mon Nov 21 21:30:02 2022 timeout : 3600 sec source : 192.168.200.100 sport : 48702 dest : 192.168.250.100 dport : 22 proto : 6 zone : 1 type : FLOW state : ACTIVE ipver : 4 fidx : 28 cid : 0 gft : 1 gft' : 0 predict : 0 orphan : 1 flag_inager : 0 ager_thread : 4 flags : 0 flow-data : type: l7 app-id: 25 startlog: 1 endlog: 1 denied: 0- Access the SSH session using the command show session all filter application ssh.
The firewall will failover to an available CN-NGFW pod to handle the orphaned flow. The new session owner is Slot 7.-------------------------------------------------------------------------------- ID Application State Type Flag Src[Sport]/Zone/Proto (translated IP[Port]) Vsys Dst[Dport]/Zone (translated IP[Port]) -------------------------------------------------------------------------------- 805306374 ssh ACTIVE FLOW 192.168.200.100[48702]/untrust_ei1/6 (192.168.200.100[48702]) vsys1 192.168.250.100[22]/trust_ei2 (192.168.250.100[22]) admin@pan-mgmt-sts-1.cluster-001> show session id 805306374 Session 805306374 c2s flow: source: 192.168.200.100 [untrust_ei1] dst: 192.168.250.100 proto: 6 sport: 48702 dport: 22 state: ACTIVE type: FLOW src user: unknown dst user: unknown s2c flow: source: 192.168.250.100 [trust_ei2] dst: 192.168.200.100 proto: 6 sport: 22 dport: 48702 state: ACTIVE type: FLOW src user: unknown dst user: unknown Slot : 7 DP : 0 index(local): : 6 start time : Mon Nov 21 21:43:27 2022 timeout : 3600 sec time to live : 3581 sec total byte count(c2s) : 1350 total byte count(s2c) : 1506 layer7 packet count(c2s) : 17 layer7 packet count(s2c) : 11 vsys : vsys1 application : ssh rule : Promoted-session service timeout override(index) : False session to be logged at end : True session in session ager : True session updated by HA peer : False layer7 processing : completed URL filtering enabled : True URL category : any session via syn-cookies : False session terminated on host : False session traverses tunnel : False session terminate tunnel : False captive portal session : False ingress interface : ethernet1/3 egress interface : ethernet1/4 session QoS rule : N/A (class 4) tracker stage l7proc : fastpath state none end-reason : unknownNo change in cluster flow.admin@pan-mgmt-sts-1.cluster-001> set system setting target-dp s5dp0 Session target dp changed to s5dp0 admin@pan-mgmt-sts-1.cluster-001> show cluster-flow id 536870939 Flow 536870939 start time : Mon Nov 21 21:30:02 2022 timeout : 3600 sec source : 192.168.200.100 sport : 48702 dest : 192.168.250.100 dport : 22 proto : 6 zone : 1 type : FLOW state : ACTIVE ipver : 4 fidx : 12 cid : 7 gft : 0 gft' : 1 predict : 0 orphan : 0 flag_inager : 0 ager_thread : 3 flags : 0 flow-data : type: l7 app-id: 25 startlog: 1 endlog: 1 denied: 0 admin@pan-mgmt-sts-1.cluster-001> set system setting target-dp s6dp0 Session target dp changed to s6dp0 admin@pan-mgmt-sts-1.cluster-001> show session id 805306374 Session 805306374 Bad Key: c2s: 'c2s' Bad Key: s2c: 's2c' index(local): : 6 admin@pan-mgmt-sts-1.cluster-001> show cluster-flow id 671088667 Flow 671088667 start time : Mon Nov 21 21:30:02 2022 timeout : 3600 sec source : 192.168.200.100 sport : 48702 dest : 192.168.250.100 dport : 22 proto : 6 zone : 1 type : FLOW state : ACTIVE ipver : 4 fidx : 12 cid : 7 gft : 1 gft' : 0 predict : 0 orphan : 0 flag_inager : 0 ager_thread : 4 flags : 0 flow-data : type: l7 app-id: 25 startlog: 1 endlog: 1 denied: 0Results: No impact to existing or new sessions. Cluster membership updated on Panorama.
- View cluster membership details of CN-MGMT pod an-mgmt-sts-0 using the command show cluster-membership show-slot-info slot all.