CN-Series
Test Case: CN-NGFW Failure Handling
Table of Contents
Test Case: CN-NGFW Failure Handling
| Where Can I Use This? | What Do I Need? |
|---|---|
|
|
This test evaluates CN-NGFW failure handling.
CN-NGFW failure can happen under the following circumstances.
- Node issues
- CN-NGFW pod crashes and restarts
- Node and CN-NGFW pod are okay but pan_task crashes
- CN-NGFW is removed from cluster membership when:
- IPsec monitoring over Eth0 interface fails
- Cluster Interconnect (CI) link is broken
- Traffic Interconnect (TI) link is broken
In this scenario, the SSH session between client and server is installed on CN-NGFW
1. If the CN-NGFW 1 is down, the SSH session must be kept alive by failover to
another CN-NGFW.
- From the Panorama CLI, enter show clusters name <cluster-name> to view CN-NGFW, CN-DB, and CN-GW pods connected to the CN-MGMT pod.Cluster: cluster-002 Creation time: 2022/11/22 04:56:46 CN-MGMT pods: 87F87FE94CBBB03 (active, pan-mgmt-sts-0.cluster-002, connected, In Sync) Slot-ID PodName Type Version ---------------------------------------------------------------------------------------- 1 pan-gw-dep-5cd5c87d76-przjx CN-GW 11.0.1-c156.dev_e_rel 6 pan-db-dep-d6fb496b-jf2ms CN-DB 11.0.1-c156.dev_e_rel 5 pan-ngfw-dep-5cd8f55848-dbhwh CN-NGFW 11.0.1-c156.dev_e_rel 8 pan-ngfw-dep-5cd8f55848-slk5l CN-NGFW 11.0.1-c156.dev_e_rel 7 pan-db-dep-d6fb496b-hfmlp CN-DB 11.0.1-c156.dev_e_rel 9 pan-ngfw-dep-5cd8f55848-pq6ks CN-NGFW 11.0.1-c156.dev_e_rel 2 pan-gw-dep-5cd5c87d76-4kbfk CN-GW 11.0.1-c156.dev_e_rel 11 pan-ngfw-dep-5cd8f55848-rsbqn CN-NGFW 11.0.1-c156.dev_e_relView cluster membership details of CN-MGMT pod an-mgmt-sts-0 using the command show cluster-membership show-slot-info slot all.MP leader status: Leader Slot-id Type CI-IP TI-IP State CI-State TI-State ======================================================================================== 1 CN-GW 192.168.23.100 192.168.24.80 UP UP UP 11 CN-NGFW 192.168.23.87 192.168.24.93 UP UP UP 2 CN-GW 192.168.23.101 192.168.24.100 UP UP UP 7 CN-DB 192.168.23.102 :: UP UP NA 6 CN-DB 192.168.23.104 :: UP UP NA 5 CN-NGFW 192.168.23.103 192.168.24.86 UP UP UP 8 CN-NGFW 192.168.23.105 192.168.24.84 UP UP UP 9 CN-NGFW 192.168.23.82 192.168.24.81 UP UP UPAll interfaces of the ethernetx/3 subnet must be in the same zone. Similarly, all interfaces of the ethernetx/4 subnet must be in the same zone.Use show session all filter application ssh to view all SSH sessions.For every session, there are two flows for Client to Server and Server to Client directions.-------------------------------------------------------------------------------- ID Application State Type Flag Src[Sport]/Zone/Proto (translated IP[Port]) Vsys Dst[Dport]/Zone (translated IP[Port]) -------------------------------------------------------------------------------- 1342177294 ssh ACTIVE FLOW 192.168.200.100[48702]/untrust_ei1/6 (192.168.200.100[48702]) vsys1 192.168.250.100[22]/trust_ei2 (192.168.250.100[22]) admin@pan-mgmt-sts-1.cluster-001> show session id 1342177294 Session 1342177294 c2s flow: source: 192.168.200.100 [untrust_ei1] dst: 192.168.250.100 proto: 6 sport: 48702 dport: 22 state: ACTIVE type: FLOW src user: unknown dst user: unknown s2c flow: source: 192.168.250.100 [trust_ei2] dst: 192.168.200.100 proto: 6 sport: 22 dport: 48702 state: ACTIVE type: FLOW src user: unknown dst user: unknown Slot : 11 DP : 0 index(local): : 14 start time : Mon Nov 21 21:30:02 2022 timeout : 3600 sec time to live : 3542 sec total byte count(c2s) : 3887 total byte count(s2c) : 4501 layer7 packet count(c2s) : 23 layer7 packet count(s2c) : 20 vsys : vsys1 application : ssh rule : allow_inside-to-outside service timeout override(index) : False session to be logged at end : True session in session ager : True session updated by HA peer : False layer7 processing : completed URL filtering enabled : True URL category : any session via syn-cookies : False session terminated on host : False session traverses tunnel : False session terminate tunnel : False captive portal session : False ingress interface : ethernet1/3 egress interface : ethernet1/4 session QoS rule : N/A (class 4) tracker stage l7proc : ctd decoder done end-reason : unknownThe session owner is Slot 11.You can view the filtered cluster flow details using the following example command.show cluster-flow all filter source-port 22Output:-------------------------------------------------------------------------------- Slot 5 -------------------------------------------------------------------------------- Id State Type Src[Sport]/Proto Dst[Dport] -------------------------------------------------------------------------------- 536870940 ACTIVE FLOW 192.168.250.100[22]/6 192.168.200.100[48702] -------------------------------------------------------------------------------- Slot 6 -------------------------------------------------------------------------------- Id State Type Src[Sport]/Proto Dst[Dport] -------------------------------------------------------------------------------- 671088668 ACTIVE FLOW 192.168.250.100[22]/6 192.168.200.100[48702]show cluster-flow all filter destination-port 22Output:-------------------------------------------------------------------------------- Slot 5 -------------------------------------------------------------------------------- Id State Type Src[Sport]/Proto Dst[Dport] -------------------------------------------------------------------------------- 536870939 ACTIVE FLOW 192.168.200.100[48702]/6 192.168.250.100[22] -------------------------------------------------------------------------------- Slot 6 -------------------------------------------------------------------------------- Id State Type Src[Sport]/Proto Dst[Dport] -------------------------------------------------------------------------------- 671088667 ACTIVE FLOW 192.168.200.100[48702]/6 192.168.250.100[22]Delete the pod on Slot 11 using the command kubectl -n kube-system delete pod pan-ngfw-dep-5cd8f55848-rsbqn.Output:pod "pan-ngfw-dep-5cd8f55848-rsbqn" deletedThe session owned by the CN-NGFW pod in Slot 11 is now marked as orphan.admin@pan-mgmt-sts-1.cluster-001> set system setting target-dp s5dp0 Session target dp changed to s6dp0 admin@pan-mgmt-sts-1.cluster-001> show cluster-flow id 536870939 Flow 536870939 start time : Mon Nov 21 21:30:02 2022 timeout : 3600 sec source : 192.168.200.100 sport : 48702 dest : 192.168.250.100 dport : 22 proto : 6 zone : 1 type : FLOW state : ACTIVE ipver : 4 fidx : 28 cid : 0 gft : 0 gft' : 1 predict : 0 orphan : 1 flag_inager : 0 ager_thread : 3 flags : 0 flow-data : type: l7 app-id: 25 startlog: 1 endlog: 1 denied: 0 admin@pan-mgmt-sts-1.cluster-001> set system setting target-dp s6dp0 Session target dp changed to s6dp0 admin@pan-mgmt-sts-1.cluster-001> show cluster-flow id 671088667 Flow 671088667 start time : Mon Nov 21 21:30:02 2022 timeout : 3600 sec source : 192.168.200.100 sport : 48702 dest : 192.168.250.100 dport : 22 proto : 6 zone : 1 type : FLOW state : ACTIVE ipver : 4 fidx : 28 cid : 0 gft : 1 gft' : 0 predict : 0 orphan : 1 flag_inager : 0 ager_thread : 4 flags : 0 flow-data : type: l7 app-id: 25 startlog: 1 endlog: 1 denied: 0Access the SSH session using the command show session all filter application ssh.The firewall will failover to an available CN-NGFW pod to handle the orphaned flow. The new session owner is Slot 7.-------------------------------------------------------------------------------- ID Application State Type Flag Src[Sport]/Zone/Proto (translated IP[Port]) Vsys Dst[Dport]/Zone (translated IP[Port]) -------------------------------------------------------------------------------- 805306374 ssh ACTIVE FLOW 192.168.200.100[48702]/untrust_ei1/6 (192.168.200.100[48702]) vsys1 192.168.250.100[22]/trust_ei2 (192.168.250.100[22]) admin@pan-mgmt-sts-1.cluster-001> show session id 805306374 Session 805306374 c2s flow: source: 192.168.200.100 [untrust_ei1] dst: 192.168.250.100 proto: 6 sport: 48702 dport: 22 state: ACTIVE type: FLOW src user: unknown dst user: unknown s2c flow: source: 192.168.250.100 [trust_ei2] dst: 192.168.200.100 proto: 6 sport: 22 dport: 48702 state: ACTIVE type: FLOW src user: unknown dst user: unknown Slot : 7 DP : 0 index(local): : 6 start time : Mon Nov 21 21:43:27 2022 timeout : 3600 sec time to live : 3581 sec total byte count(c2s) : 1350 total byte count(s2c) : 1506 layer7 packet count(c2s) : 17 layer7 packet count(s2c) : 11 vsys : vsys1 application : ssh rule : Promoted-session service timeout override(index) : False session to be logged at end : True session in session ager : True session updated by HA peer : False layer7 processing : completed URL filtering enabled : True URL category : any session via syn-cookies : False session terminated on host : False session traverses tunnel : False session terminate tunnel : False captive portal session : False ingress interface : ethernet1/3 egress interface : ethernet1/4 session QoS rule : N/A (class 4) tracker stage l7proc : fastpath state none end-reason : unknownNo change in cluster flow.admin@pan-mgmt-sts-1.cluster-001> set system setting target-dp s5dp0 Session target dp changed to s5dp0 admin@pan-mgmt-sts-1.cluster-001> show cluster-flow id 536870939 Flow 536870939 start time : Mon Nov 21 21:30:02 2022 timeout : 3600 sec source : 192.168.200.100 sport : 48702 dest : 192.168.250.100 dport : 22 proto : 6 zone : 1 type : FLOW state : ACTIVE ipver : 4 fidx : 12 cid : 7 gft : 0 gft' : 1 predict : 0 orphan : 0 flag_inager : 0 ager_thread : 3 flags : 0 flow-data : type: l7 app-id: 25 startlog: 1 endlog: 1 denied: 0 admin@pan-mgmt-sts-1.cluster-001> set system setting target-dp s6dp0 Session target dp changed to s6dp0 admin@pan-mgmt-sts-1.cluster-001> show session id 805306374 Session 805306374 Bad Key: c2s: 'c2s' Bad Key: s2c: 's2c' index(local): : 6 admin@pan-mgmt-sts-1.cluster-001> show cluster-flow id 671088667 Flow 671088667 start time : Mon Nov 21 21:30:02 2022 timeout : 3600 sec source : 192.168.200.100 sport : 48702 dest : 192.168.250.100 dport : 22 proto : 6 zone : 1 type : FLOW state : ACTIVE ipver : 4 fidx : 12 cid : 7 gft : 1 gft' : 0 predict : 0 orphan : 0 flag_inager : 0 ager_thread : 4 flags : 0 flow-data : type: l7 app-id: 25 startlog: 1 endlog: 1 denied: 0
Results: No impact to existing or new sessions. Cluster membership updated on Panorama.