Free up Disk Space with Data Archiving

Free up disk space by archiving Cortex XSOAR folders to condense the unused data within them.
Cortex XSOAR supports full archiving of incidents, entries and indicators by month. Data from incidents, insights (indicators), and entries are stored in folders on a monthly basis.
If disk space needs to be freed up, you can archive folders to condense unused data within them. It is recommended to archive folders and not delete them permanently.
Locate the folders that reside in the following location (where Cortex XSOAR is installed),
/var/lib/demisto/data/
.
Although the folders reside in
/var/lib/demisto/data/
,
Do Not
save the backup folders under
/var/lib/demisto/
.
The following data folder and files can be found in this folder:
  • demisto.db
    : The database for all playbooks and automation (not anything related to incidents and insights).
  • demistoidx
    : Indexing of the system.
  • partitionsData
    : Data of incidents, insights, and entries separated by month resolution.
    If you archive indexes you need to archive the matching partitions, so they are not rebuilt.
The following is an example of how the folders and filenames will appear in your system.
$ tree /var/lib/demisto/data ├── demisto.db ├── demistoidx │ ├── accounts │ │ ├── index_meta.json │ │ └── store ... │ ├── entries_082017 │ │ ├── index_meta.json │ │ └── store │ ├── entries_092017 │ │ ├── index_meta.json │ │ └── store │ ├── entries_102017 │ │ ├── index_meta.json │ │ └── store │ ├── evidences │ │ ├── index_meta.json │ │ └── store │ ├── incidents_082017 │ │ ├── index_meta.json │ │ └── store │ ├── incidents_092017 │ │ ├── index_meta.json │ │ └── store │ ├── incidents_102017 │ │ ├── index_meta.json │ │ └── store │ ├── investigations_082017 │ │ ├── index_meta.json │ │ └── store │ ├── investigations_092017 │ │ ├── index_meta.json │ │ └── store │ ├── investigations_102017 │ │ ├── index_meta.json │ │ └── store ... │ ├── newInsights_082017 │ │ ├── index_meta.json │ │ └── store │ ├── newInsights_092017 │ │ ├── index_meta.json │ │ └── store │ ├── newInsights_102017 │ │ ├── index_meta.json │ │ └── store │ ├── playbooks │ │ ├── index_meta.json │ │ └── store ... └── partitionsData ├── demisto_082017.db ├── demisto_092017.db └── demisto_102017.db
Follow these steps to free up disk space by archiving folders.
In a distributed database deployment, first stop the app server and then the databases. Then run this procedure on each database that contains incidents. By default, you only need to run this procedure on the secondary nodes. However, in distributed database deployments that were converted from a single server deployment, there is incident data in the main database. Therefore, you also need to run this procedure on the main database.
  1. Stop the Cortex XSOAR service using the following command.
    $ sudo service demisto stop
  2. Create the following directories:
    mkdir /var/lib/demisto-archive
    mkdir /var/lib/demisto-archive/archived-2019
  3. Navigate to the
    /var/lib/demisto-archive/
    filepath using the following command.
    cd /var/lib/demisto-archive/
  4. Move the data you want to archive to the archive directory using the following command. The following command moves all folders that have a
    mmyyyy
    suffix.
    mv /var/lib/demisto/data/**/*_<date_to_archive>* /var/lib/demisto-archive/archived-2019
    For example:
    mv /var/lib/demisto/data/**/*_092019* /var/lib/demisto-archive/
    If the
    mv /var/lib/demisto/data/**/*_<date_to_archive>* /var/lib/demisto-archive/archived-2019
    command does not work in your environment, archive the indicies and partition separately. First create the following directories:
    mkdir /var/lib/demisto-archive/archived-2019/demistoidx/
    mkdir /var/lib/demisto-archive/archived-2019/partitionsData/
    Then, archive each index and the partition that has a mmyyyy suffix. The following are the commands to archive the current indices and the partition:
    sudo mv /var/lib/demisto/data/demistoidx/entries_082019 /var/lib/demistoarchive/archived-2019/demistoidx/
    sudo mv /var/lib/demisto/data/demistoidx/evidences_082019 /var/lib/demistoarchive/archived-2019/demistoidx/
    sudo mv /var/lib/demisto/data/demistoidx/incidents_082019 /var/lib/demistoarchive/archived-2019/demistoidx/
    sudo mv /var/lib/demisto/data/demistoidx/invTaskIdx_082019 /var/lib/demistoarchive/archived-2019/demistoidx/
    sudo mv /var/lib/demisto/data/demistoidx/investigations_082019 /var/lib/demistoarchive/archived-2019/demistoidx/
    sudo mv /var/lib/demisto/data/demistoidx/newInsights_082019 /var/lib/demistoarchive/archived-2019/demistoidx/
    sudo mv /var/lib/demisto/data/demistoidx/todosTask_082019 /var/lib/demistoarchive/archived-2019/demistoidx/
    sudo mv /var/lib/demisto/data/partitionsData/demisto_082019.db /var/lib/demisto-archive/archived-2019/partitionsData/demisto_082019.db
  5. Create the compressed archive of your selected files and folders using the following tarball command.
    $ tar -cvzf demisto-2019-archive.tar.gz /var/lib/demisto-archive/archived-2019
  6. Start the Cortex XSOAR service using the following command.
    $ sudo service demisto start

Recommended For You