Migrate Logs to the Same M-Series Appliance Model in Panorama Mode in High Availability

If you need to replace an M-700, M-600, M-500, M-300, M-200, or M-100 appliance deployed in high availability (HA) configuration in Panorama mode (Panorama management server) with the same M-Series appliance as the M-Series appliance being replaced, you can migrate the logs it collected from firewalls by moving its RAID disks to the new M-Series appliance. Moving the disks enables you to recover logs after a system failure on the M-Series appliance.

This migration procedure covers the following scenarios:

• One Panorama HA peer has a managed collector (Log Collector) in a Collector Group.

• Both Panorama HA peers have managed collectors that belong to a single Collector Group. For details, see Multiple Local Log Collectors Per Collector Group.
• Both Panorama HA peers have a managed collector and each is assigned to a separate Collector Group. For details, see Single Local Log Collector Per Collector Group.

1. Forward any logs on the SSD of the old M-Series appliance to an external destination if you want to preserve them.

   The SSD stores the System and Config logs that Panorama and Log Collectors generate. You cannot move the SSD between M-Series appliances.

   Configure Log Forwarding from Panorama to External Destinations.
2. (RMA of Active Primary HA peer only) Reconfigure the high availability configuration of the Panorama HA peers to make the Secondary HA peer the Primary HA peer during the RMA process.

   This step is required if you are replacing the Primary HA peer in an A/P HA configuration to help ensure the reset secure communication status and the certificate authority (CA) of the new M-Series appliance are not unintentionally synchronized to the existing peer in the HA configuration. When replacing the Primary HA peer, reconfiguring the HA election settings ensures Panorama management of devices remains uninterrupted during the RMA process.

   Skip this step if you are replacing the Secondary HA peer in an A/P HA configuration.

   1. Log in to the Panorama web interface of the Primary HA peer.
   2. Select PanoramaHigh Availability and edit the Election Settings.
   3. For the Priority, select secondary and click OK.
   4. Select Commit and Commit to Panorama.

      The Panorama HA peer you are replacing is now the Secondary HA peer.
   5. Log in to the Panorama web interface of the Secondary HA peer.
   6. Select PanoramaHigh Availability and edit the Election Settings.
   7. For the Priority, select primary and click OK.
   8. Select Commit and Commit to Panorama.

      The previously Secondary Panorama is now the Primary HA peer.
3. Suspend HA functionality on the Panorama HA peer you are replacing.

   Step is required to help ensure the reset secure communication status and the certificate authority (CA) of the new M-Series appliance are not unintentionally synchronized to the existing peer in the HA configuration during the RMA process. This puts the Panorama HA peer in a suspended state.

   1. Log in to the Panorama web interface of the Panorama HA peer you are replacing.
   2. Select PanoramaHigh Availability and Suspend local Panorama for high availability.
   3. Click OK to confirm suspending HA on Panorama HA peer.
4. Reset the secure connection setting on the Panorama HA peer you are replacing.

   1. Log in to the Panorama CLI of the Panorama HA peer you are replacing.
   2. Reset the secure connection state.

      This command resets all managed device connections and is irreversible.

      admin> request sc3 reset

   3. Restart the management server on the Panorama HA peer you are replacing.

      admin> debug software restart process management-server

5. Remove the RAID disks from the old M-Series appliance.

   1. Power off the old M-Series appliance by pressing the Power button until the system shuts down.
   2. Remove the disk pairs. For details, refer to the disk replacement procedure in the M-Series Appliance Hardware Reference Guides.
6. Perform initial setup of the new M-Series appliance.

   1. Rack mount the M-Series appliance. Refer to the M-Series Appliance Hardware Reference Guides for instructions.
   2. Perform Initial Configuration of the M-Series Appliance.

      If the old M-Series appliance used interfaces other than the MGT interface for Panorama services (such as log collection), you must define those interfaces during initial configuration of the new M-Series appliance (PanoramaSetupInterfaces).
   3. Register Panorama.
   4. Purchase and activate a Panorama support license or transfer licenses as follows only if the new M-Series appliance is the same hardware model as the old M-Series appliance. If the new M-Series appliance is a different model than the old M-Series appliance, you must purchase new licenses.

      1. Log in to the Palo Alto Networks Customer Support web site.
      2. Select the Assets tab and click the Spares link.
      3. Click the Serial Number of the new M-Series appliance.
      4. Click Transfer Licenses.
      5. Select the old M-Series appliance and click Submit.
   5. Activate a firewall management license. If you are migrating from an M-100 appliance to an M-500 appliance, enter the auth-code associated with the migration license.
   6. Install Content and Software Updates for Panorama. For important details about software versions, see Panorama, Log Collector, Firewall, and WildFire Version Compatibility.
   7. Perform any additional configuration changes as needed.

      If the old M-Series appliance used interfaces other than the MGT interface for Panorama services (such as log collection), you must define those interfaces on the new M-Series appliance (PanoramaSetupInterfaces).
   8. Set Up HA on Panorama. The new M-Series appliance must have the same priority as the HA peer you are replacing.

      The new M-Series appliance must be added to the HA configuration as the Secondary HA peer. Adding the new M-series as the Primary HA peer forces synchronization of the reset secure communication setting status to the existing M-Series appliance, resulting in the interruption of Panorama management of devices.
7. Insert the disks into the new M-Series appliance. For details, refer to the disk replacement procedure in the M-Series Appliance Hardware Reference Guides.

   The disk carriers of the M-100 appliance are incompatible with those of the M-500 appliance. Therefore, when migrating between these hardware models, you must unscrew each disk from its old carrier and insert the disk in the new carrier before inserting the disk in the new appliance.

   You must maintain the disk pair association. Although you can place a disk pair from slot A1/A2 on the old appliance into slot B1/B2 on the new appliance, you must keep the disks together in the same slot; otherwise, Panorama might not restore the data successfully.
8. If the M-Series appliance was part of a Collector Group, verify that the decommissioned M-Series appliance serial number is still part of the correct Collector Group:

   debug log-collector-group show name <Log CollectorGroup name>
9. Prepare the disks for migration.

   Generating the metadata for each disk pair rebuilds the indexes. Therefore, depending on the data size, this process can take a long time to complete. To expedite the process, you can launch multiple CLI sessions and run the metadata regeneration command in each session to complete the process simultaneously for every pair. For details, see Regenerate Metadata for M-Series Appliance RAID Pairs.

   1. Enable the disk pairs by running the following CLI command for each pair:

      admin> request system raid add <slot> force no-format

      For example:

      admin> request system raid add A1 force no-format 
      admin> request system raid add A2 force no-format 

      The force and no-format arguments are required. The force argument associates the disk pair with the new appliance. The no-format argument prevents reformatting of the drives and retains the logs stored on the disks.
   2. Generate the metadata for each disk pair.

      This step may take up to 60 hours to complete depending on the volume of log data on the disks. The new M-Series appliance must be in an active state. Panorama may experience issues getting the assigned Log Collector ID if Panorama is in a suspended state.

      If the new M-Series appliance is the passive HA peer, log in to the Panorama web interface of the currently active HA peer and select Panorama > High Availability and Suspend local Panorama for high availability.

      After you successfully migrate the logs, Make local Panorama functional for high availability.

      admin> request metadata-regenerate slot <slot_number>

      For example:

      admin> request metadata-regenerate slot 1

10. Configure the local Log Collector on the new M-Series appliance.

    For all steps with commands that require a serial number, you must type the entire serial number; pressing the Tab key won’t complete a partial serial number.

    Don’t enable the disks on the new M-Series appliance at this point. When you successfully migrate the logs, Panorama automatically enables the disks.

    1. Configure the local Log Collector as a managed collector using the Panorama web interface or using the following CLI commands:

       admin> configure 
       admin# set log-collector <log-collector_SN> deviceconfig system hostname <log-collector-hostname>  
       admin# exit 

    2. Commit your changes to Panorama. Don’t commit the changes to the Collector Group just yet.

       admin> configure 
       admin# commit 

    3. Verify that the local Log Collector is connected to Panorama and that the status of its disk pairs is present/available.

       admin> show log-collector serial-number <log-collector_SN>

       The disk pairs will display as disabled at this stage of the restoration process.
11. Add a Log Collector with no disks to a Collector Group.

    From this point, only commits that are required to complete the migration process on Panorama and the Log Collectors. Hold off making any other changes.

    1. Access the Panorama CLI.
    2. Overwrite Panorama restriction to allow Log Collector with no disk to be added to a Collector Group: request log-migration-set-start
    3. Commit the overwritten restriction:

       admin> configure 
       admin# commit force  

12. Migrate the logs.

    The existing M-Series appliance must be the active HA peer before you can begin migrating logs. If the existing M-Series appliance is the passive HA peer, log in to the Panorama web interface of the currently active HA peer and select Panorama > High Availability and Suspend local Panorama for high availability.

    After you successfully migrate the logs, Make local Panorama functional for high availability.

    1. Access the Panorama CLI.
    2. Add the new local Log Collector as a member of the Collector Group and commit your changes to Panorama.

       admin# set log-collector-group <collector_group_name> logfwd-setting collectors <SN_managed_collector> 
       admin# commit 
       admin# exit 

       The old local Log Collector still appears in the list of members, because you haven’t deleted it from the configuration.
    3. For each disk pair, migrate the logs to the new appliance.

       Verify that the existing Panorama is in an active HA state before you begin migrating logs. This is required to successfully migrate logs.

       admin> request log-migration from <old_LC_serial_number> old-disk-pair <log_disk_pair> to <new_LC_serial_number> new-disk-pair <log_disk_pair> 

       For example:

       admin> request log-migration from 003001000010 old-disk-pair A to 00300100038 new-disk-pair A 

    4. Commit the changes to Panorama.

       admin> configure 
       admin# commit 

13. Reconfigure the Collector Group.

    1. Use the web interface to assign the new Log Collector to the firewalls that forward logs (PanoramaCollector GroupsDevice Log Forwarding). Give the new Log Collector the same priority in the firewall preference lists as the old Log Collector.

       You cannot use the CLI change the priority assignments of firewall preference lists.
    2. Delete the old Log Collector from the Collector Group.

       admin# delete log-collector-group <group_name> logfwd-setting collectors <old_LC_serial_number>

       For example:

       admin# delete log-collector-group DC-Collector-Group logfwd-setting collectors 003001000010

    3. Delete the old Log Collector from the Panorama configuration and commit your changes to Panorama.

       admin# delete log-collector <old_LC_serial_number> 
       admin# commit 
       admin# exit 

    4. Synchronize the configuration of the M-Series appliance HA peers.

       admin> request high-availability sync-to-remote running-config

    5. Commit the Collector Group changes so that the managed firewalls can send logs to the new Log Collector.

       admin> commit-all log-collector-config log-collector-group <collector_group_name>

       For example:

       admin> commit-all log-collector-config log-collector-group DC-Collector-Group

14. Generate new keys on the new Log Collector.

    This command is required in order to add the new Log Collector to the Collector Group and should only be run for the Collector Group of the Log Collector being replaced. This step deletes the existing RSA keys and allows Panorama to create new RSA keys.

    1. Access the Panorama CLI.
    2. Delete all RSA keys on the new Log Collector:

       request logdb update-collector-group-after-replace collector-group <collector-group-name>

       The process can take up to 10 minutes to completed.
15. Confirm that SearchEngine Status is Active for all Log Collectors in the Collector Group.

    Do not continue until SearchEngine Status is Active for all Log Collectors in the Collector Group. This will result in purging of logs from the Log Collector being replaced.

    1. Access the Panorama CLI.
    2. Show the Log Collector details by running the following commands either:

       • On Panorama for all Log Collectors:
         show log-collector all

       Alternatively, you can run the following command on each Dedicated Log Collector:

       show log-collector detail

    3. Confirm that SearchEngine Status is Active.

       Redistribution status:       none

       Last commit-all: commit succeeded, current ring version 1

       SearchEngine status:     Active

       md5sum 4e5055a359f7662fab8f8c4f57e24525 updated at 2017/06/14 09:58:19

16. On the new Log Collector, replace previous Log Collector serial number with the new Log Collector serial number.

    You must replace the old Log Collector serial number with the new Log Collector serial number so that the new Log Collector will not run in to purging issues, resulting in the Log Collector being unable to purge old data from the migrated logs when necessary.

    (PAN-OS 11.1 and later releases) When the Log Collector serial number changes, you must erase all Elasticsearch data before executing the logger update command. If you skip this step, Elasticsearch will not start correctly after the migration.

    1. Access the Log Collector CLI.
    2. Erase all existing Elasticsearch data on the new Log Collector:

       debug elasticsearch erase data
    3. Replace old Log Collector serial number with new Log Collector serial number:

       request log-migration-update-logger from <old-log-collector-serial-number> to <new-log-collector-serial-number>
    4. Reingest raw logs for each required log type and time window. Repeat the command for each log type and monitor progress before initiating the next. debug logdb migrate-lc start log-type <log-type> time period start-date <YYYY/MM/DD> end-date <YYYY/MM/DD>
17. Restore HA functionality for the suspended Panorama HA peer to force the reset secure communication state changes to the Secondary HA peer.

    1. Log in to the Panorama web interface of the new (Secondary) Panorama HA peer.
    2. Select PanoramaHigh Availability and Make local Panorama functional for high availability.
    3. Click OK to confirm restoring HA functionality on the new Panorama HA peer.
18. Restart the management server on the new Panorama HA peer.

    1. Log in to the Panorama CLI of the new (Secondary) Panorama HA peer.
    2. Restart the management server.

       admin> debug software restart process management-server

19. (RMA of Active Primary HA peer only) Restore the high availability configuration of the Panorama HA peers.

    Skip this step if you are replaced the Secondary HA peer in an A/P HA configuration.

    1. Log in to the Panorama web interface of the Primary HA peer.
    2. Select PanoramaHigh Availability and edit the Election Settings.
    3. For the Priority, select secondary and click OK.
    4. Select Commit and Commit to Panorama.
    5. Log in to the Panorama web interface of the Secondary HA peer.
    6. Select PanoramaHigh Availability and edit the Election Settings.
    7. For the Priority, select primary and click OK.
    8. Select Commit and Commit to Panorama.