View WildFire Cluster Status Using the CLI
Focus
Focus
Advanced WildFire

View WildFire Cluster Status Using the CLI

Table of Contents

View WildFire Cluster Status Using the CLI

Where Can I Use This?
What Do I Need?
  • WildFire Appliance
  • WildFire License
To confirm that you WildFire cluster is running within normal operating parameters, you must execute the following show commands:
  • show cluster controller
    —Displays the status of active/passive WildFire cluster nodes.
  • show cluster all-peers
    —Displays information about all of the members in a given WildFire cluster.
  • show cluster membership
    —Displays WildFire appliance information for cluster and standalone nodes.
  • show cluster data-migration-status
    —Displays the current status of the data migration process.
  • show log system
    —Displays the WildFire event log, including system status details.
  1. On a WildFire appliance controller node, run:
    admin@WF-500(active-controller)>show clustercontroller
    A healthy WildFire cluster displays the following details:
    • The name of the cluster the appliance has been enrolled in and its configured role.
    • The
      K/V API online status
      indicates
      True
      when the internal cluster interface is functioning properly. A status of
      False
      can indicate an improperly configured node or a network issue.
    • Task processing
      indicates
      True
      on active-controllers (primary) and
      False
      on passive-controllers (backup).
    • The IP addresses for all WildFire nodes in the cluster are listed under
      App Service Avail
      .
    • Up to three
      Good Core Servers
      . The number of
      Good Core Servers
      depends on the number of nodes running in the cluster. If you have a third node operating within a cluster, it automatically get configured as a server node to maximize cluster integrity.
    • No
      Suspended Nodes
      .
    • The
      Current Task
      provides background information about cluster-level operations, such as reboot, decommission, and suspend tasks.
    The following example shows the output from an active controller configured in a 2-node WildFire cluster operating in a healthy state:
    Cluster name: WildFire_Cluster K/V API online: True Task processing: on Active Controller: True DNS Advertisement: App Service DNS Name: App Service Avail: 2.2.2.14, 2.2.2.15 Core Servers: 009701000026: 2.2.2.15 009701000043: 2.2.2.14 Good Core Servers: 2 Suspended Nodes: Current Task: * Showing latest completed task Request: startup from qa14 (009701000043/80025) at 2017-09-18 21:43:34 UTC null Response: permit by qa15 at 2017-09-18 21:45:15 UTC 1/2 core servers available. Finished: success at 2017-09-18 21:43:47 UTC
  2. On a WildFire appliance controller node, run:
    admin@WF-500>
    show cluster all-peers
    A healthy WildFire cluster displays the following details:
    • The general information about the WildFire nodes in the cluster are listed under
      Address
      ,
      Mode
      ,
      Server
      ,
      Node
      , and
      Name
      .
    • All WildFire cluster nodes are running the
      wfpc
      service, an internal file sample analysis service.
    • Nodes operating as an active, passive, or server display
      Serverrole applied
      next to
      Status
      . If the node has been configured to be a server, but isn’t operating as a server, the
      status
      displays
      Serverrole assigned
      .
      In a 3-node deployment, the third server node is categorized as a worker.
    • Recently removed nodes might be present but displays as
      Disconnected
      . It can take several days for a disconnected node to be removed from the cluster node list.
    • The active controller node displays
      siggen-db:ReadyMaster
      .
    • The passive controller node displays
      siggen-db:ReadySlave
      .
      For more information about general WildFire application and service status details, refer to WildFire Application States and WildFire Service States.
    • The
      Diag report
      displays cluster system events and error messages:
      Error Message
      Description
      Unreachable
      The node was never reachable from the cluster controller.
      Unexpected member
      The node is not part of the cluster configuration. The node might have recently deleted from the cluster configuration or the result of misconfiguration.
      Left cluster
      The node is no longer reachable from the cluster controller.
      Incorrect cluster name
      The node has an incorrectly configured cluster name.
      Connectivity unstable
      The node’s connection to the cluster controller is unstable.
      Connectivity lost
      The node’s connectivity to the cluster controller has been lost.
      Unexpected server serial number
      The unexpected presence of a server node has been detected.
    The following example shows a 3-node WildFire cluster operating in a healthy state:
    Address Mode Server Node Name ------- ---- ------ --------- 2.2.2.15 controller Self True qa15 Service: infra signature wfcore wfpc Status: Connected, Server role applied Changed: Mon, 18 Sep 2017 15:37:40 -0700 WF App: global-db-service: JoinedCluster wildfire-apps-service: Stopped global-queue-service: JoinedCluster wildfire-management-service: Done siggen-db: ReadySlave 2.2.2.14 controller Peer True qa14 Service: infra signature wfcore wfpc Status: Connected, Server role applied Changed: Mon, 18 Sep 2017 15:37:40 -0700 WF App: global-db-service: commit-lock wildfire-apps-service: Stopped global-queue-service: ReadyStandalone wildfire-management-service: Done siggen-db: ReadyMaster 2.2.2.16 worker True wf6240 Service: infra wfcore wfpc Status: Connected, Server role applied Changed: Wed, 22 Feb 2017 11:11:15 -0800 WF App: wildfire-apps-service: Ready global-db-service: JoinedCluster global-queue-service: JoinedCluster local-db-service: DataMigrationFailed Diag report: 2.2.2.14: reported leader '2.2.2.15', age 0. 2.2.2.15: local node passed sanity check.
  3. On a WildFire appliance controller node, run:
    admin@WF-500>
    show cluster membership
    A healthy WildFire cluster displays the following details:
    • The general WildFire appliance configuration details, such as the cluster name, IP address of the appliance, serial number, etc.
    • Server role
      indicates whether or not the WildFire appliance is operating as a cluster server. Cluster servers operate additional infrastructure applications and services. You can have a maximum of three servers per cluster.
    • Node mode
      describes the role of a WildFire appliance. WildFire appliances enrolled in a cluster can be either a
      controller
      or
      worker
      node depending on your configuration and the number of nodes in your deployment. Appliances that are not a part of a cluster displays
      stand_alone
      .
    • Operates the following
      Services
      based on the cluster node role:
      Node Type
      Services Operating on the Node
      Controller Node (Active or Passive)
      • infra
      • wfpc
      • signature
      • wfcore
      Server Node
      • infra
      • wfpc
      • wfcore
      Worker Node
      • infra
      • wfpc
    • HA priority
      displays primary or secondary depending on its configured role, however this setting is independent of the current HA state of the appliance.
    • Work queue status
      shows the sample analysis backlog as well as samples that are currently being analyzed. This also indicates how much load a particular WildFire appliance receives.
    For more information about WildFire application and service status details, refer to WildFire Application States and WildFire Service States.
    The following example shows a WildFire controller operating in a healthy state:
    Service Summary: wfpc signature Cluster name: qa-auto-0ut1 Address: 2.2.2.15 Host name: qa15 Node name: wfpc-009701000026-internal Serial number: 009701000026 Node mode: controller Server role: True HA priority: secondary Last changed: Fri, 22 Sep 2017 11:30:47 -0700 Services: wfcore signature wfpc infra Monitor status: Serf Health Status: passing Agent alive and reachable Service 'infra' check: passing Application status: global-db-service: ReadyLeader wildfire-apps-service: Ready global-queue-service: ReadyLeader wildfire-management-service: Done siggen-db: Ready Work queue status: sample anaysis queued: 0 sample anaysis running: 0 sample copy queued: 0 sample copy running: 0 Diag report: 2.2.2.14: reported leader '2.2.2.15', age 0. 2.2.2.15: local node passed sanity check.
  4. On a WildFire appliance controller node, run:
    admin@WF-500(active-controller)>show clusterdata-migration-status
    The WildFire appliance displays the following data migration details:
    • Do not forward files to the WildFire appliance cluster when data migration is in progress. When data migration is finishes, the completion timestamp displays.
    • Topology changes to the WildFire cluster (for examples, adding or removing nodes and changing node roles) triggers data migration events.
    • Data migration can occur upon upgrade to a new version of WildFire. After upgrading, be sure to check the operational status of your WildFire cluster to verify proper functionality.
    The following example shows the progress of data migration in a WildFire appliance cluster:
    admin@WF-500(active-controller)>: show data-migration-status 100% completed on Mon Sep 9 21:44:48 PDT 2019
  5. On a WildFire appliance active, passive, and server nodes, run:
    admin@WF-500(active-controller)>show log systemsubtype direction equal backward
    This command displays all WildFire logged events categorized as a wildfire-appliance subtype from newest to oldest.
    • You must issue this command to all nodes in a cluster. For example, if you are operating a 3-node cluster, you must verify the status on the active controller, passive controller, and the server node.
    • The log messages returned by the WildFire appliance CLI can include numerous subtypes. You can filter the logs based on a common subtype keyword. Use the following command argument to filter based on a specific component:
      • global-queue—
        matchqueue
        , for example
        show log system directionequal backward | match queue
      • global-database—
        match global
        , for example
        show log system direction equal backward | matchglobal
      • signature-generation—
        match signature
        , for example
        show log system direction equal backward| match signature
    • WildFire appliance clusters operating normally return the following status readouts for each node in a 2-node cluster. Healthy WildFire cluster nodes have differing status readouts based on the role of an appliance.
      Use the following checklist to verify that the WildFire appliance services are running correctly in your cluster deployment.
      • Active Controller
        Component
        Active Controller Status
        global-queue
        • infowildfire cluster 0 Global queue (rabbitmq) cluster formation succeeded withstatus ReadyLeader
        • info general general 0 Setup policy for global-queue service
        global-database
        • infogeneral general 0 I'm cluster leader, bootstrap for global-db service
        • info general general 0 Setup policy for global-queue service
        signature-generation
        • infowildfir cluster 0 Signature generation service status set to ReadyMaster
        • info wildfir cluster 0 Signature generationservice status set to ReadyMaster
        The log messages returned by the WildFire appliance(s) are shown from newest to oldest. If you do not use the
        direction equal backward
        command argument as shown in the above procedure, the WildFire appliance CLI returns the log messages from oldest to newest.
      • Passive Controller
        Component
        Passive Controller Status Example
        global-queue
        • infogeneral general 0 Setup policy for global-queue service
        • info wildfire cluster 0 Global queue (rabbitmq)cluster formation succeeded with status JoinedCluster
        • info general general 0 Join cluster for global-queueservice - succeeded
        • info general general 0 Setup policy for global-queue service
        global-database
        • infogeneral general 0 Setup policy for global-queue service
        • info general general 0 Restore applications:done, For global-db bootstrap and join cluster
        • info general general 0 Start vm_mgr, For global-dbbootstrap and join cluster
        • info general general 0 Start uwsgi, For global-dbbootstrap and join cluster
        • info general general 0 Start wf_services, Forglobal-db bootstrap and join cluster
        • info general general 0 Suspend applications:done, For global-db bootstrap and join cluster
        • info general general 0 Stop vm_mgr, For global-dbbootstrap and join cluster
        • info general general 0 Stop uwsgi, For global-dbbootstrap and join cluster
        • info general general 0 Stop wf_services, Forglobal-db bootstrap and join cluster
        • info general general 0 Bootstrap and join clusterfor global-db service
        signature-generation
        • infowildfir cluster 0 Signature generation service status set to ReadySlave
        • info wildfir cluster 0 Signature generationservice status set to ReadySlave
        The log messages returned by the WildFire appliance(s) are shown from newest to oldest. If you do not use the
        direction equal backward
        command argument as shown in the above procedure, the WildFire appliance CLI returns the log messages from oldest to newest.
    • WildFire appliance clusters operating normally return the following status readouts for each node in a 3-node cluster. Healthy WildFire cluster nodes have differing status readouts based on the role of an appliance.
      Use the following checklist to verify that the WildFire appliance services are running correctly in your cluster deployment.
      • Active Controller
        Component
        Active Controller Status
        global-queue
        • infowildfire cluster 0 Global queue (rabbitmq) cluster formation succeeded withstatus JoinedCluster
        • info general general 0 Join cluster for global-queueservice - succeeded
        • info general general 0 Setup policy for global-queue service
        global-database
        • infogeneral general 0 Restore applications: done, For global-db bootstrap andjoin cluster
        • info general general 0 Start vm_mgr, For global-dbbootstrap and join cluster
        • info general general 0 Start uwsgi, For global-dbbootstrap and join cluster
        • info general general 0 Start wf_services, Forglobal-db bootstrap and join cluster
        • info general general 0 Suspend applications:done, For global-db bootstrap and join cluster
        • info general general 0 Stop vm_mgr, For global-dbbootstrap and join cluster
        • info general general 0 Stop uwsgi, For global-dbbootstrap and join cluster
        • info general general 0 Stop wf_services, Forglobal-db bootstrap and join cluster
        • 2019/07/19 14:40:19 info general general 0Bootstrap and join cluster for global-db service
        signature-generation
        • infowildfire cluster 0 Signature generation service status set to ReadyMaster
        The log messages returned by the WildFire appliance(s) are shown from newest to oldest. If you do not use the
        direction equal backward
        command argument as shown in the above procedure, the WildFire appliance CLI returns the log messages from oldest to newest.
      • Passive Controller
        Component
        Passive Controller Status
        global-queue
        • infogeneral general 0 Setup policy for global-queue service
        • info general general 0 Setup policy for global-queue service
        • info wildfire cluster 0 Global queue (rabbitmq)cluster formation succeeded with status ReadyLeader
        • info general general 0 Setup policy for global-queue service
        global-database
        • infogeneral general 0 I'm cluster leader, bootstrap for global-db service
        • info general general 0 Setup policy for global-queue service
        signature-generation
        • infowildfire cluster 0 Signature generation service status set to ReadySlave
        • info wildfire cluster 0 Signature generationservice status set to ReadySlave
        The log messages returned by the WildFire appliance(s) are shown from newest to oldest. If you do not use the
        direction equal backward
        command argument as shown in the above procedure, the WildFire appliance CLI returns the log messages from oldest to newest.
      • Server Node
        Component
        Server Node Status
        global-queue
        • infowildfire cluster 0 Global queue (rabbitmq) cluster formation succeeded withstatus JoinedCluster
        • info general general 0 Join cluster for global-queueservice - succeeded
        • info general general 0 Setup policy for global-queue service
        • info wildfire cluster 0 Global queue (rabbitmq)cluster formation succeeded with status StandbyAsWorker
        global-database
        • infogeneral general 0 Restore applications: done, For global-db bootstrap andjoin cluster
        • info general general 0 Start vm_mgr, For global-dbbootstrap and join cluster
        • info general general 0 Start uwsgi, For global-dbbootstrap and join cluster
        • info general general 0 Start wf_services, Forglobal-db bootstrap and join cluster
        • info general general 0 Suspend applications:done, For global-db bootstrap and join cluster
        • info general general 0 Stop vm_mgr, For global-dbbootstrap and join cluster
        • info general general 0 Stop uwsgi, For global-dbbootstrap and join cluster
        • info general general 0 Stop wf_services, Forglobal-db bootstrap and join cluster
        • 2019/07/19 14:32:50 info general general 0Promote worker node and join cluster for global-db service
        signature-generation
        • infowildfire cluster 0 Signature generation service status set to Stopped
        • critical wildfire cluster 0 Signature DataMigrationDone
        The log messages returned by the WildFire appliance(s) are shown from newest to oldest. If you do not use the
        direction equal backward
        command argument as shown in the above procedure, the WildFire appliance CLI returns the log messages from oldest to newest.

Recommended For You