Back up

Enabling automatic backups

About automatic backups

If you have deployed the multi-container Microsegmentation Console, you can enable automatic snapshots of the following volumes.
The auto-backup pods are run as cron jobs, using an auto-backup service account. You must associate the auto-backup service account with a workload identity to give the service account permissions to use the cloud provider’s snapshot API.
We only support automatic backups in the following environments at this time.
  • AWS
  • GCP
If you require an alternate cloud provider, contact us for assistance. Refer to the /opt/aporeto/voila/flavors/k8s/helpers/auto-backup/README.md file for more details.

Enabling backups

If you are upgrading or did not enable backups when you created your Voila environment, use the following command to enable them.
set_value global.backup.enabled true override
Then run doit to deploy the backup jobs.

Configuring backups on AWS

Create an IAM role with the required permissions. An example follows.
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "ec2:CreateSnapshot", "ec2:CreateSnapshots", "ec2:DeleteSnapshot", "ec2:DescribeInstances", "ec2:DescribeVolumes", "ec2:DescribeSnapshots", "ec2:EnableFastSnapshotRestores", "ec2:DescribeFastSnapshotRestores", "ec2:DisableFastSnapshotRestores", "ec2:CopySnapshot", "ec2:ModifySnapshotAttribute", "ec2:DescribeSnapshotAttribute" ], "Resource": "*" }, { "Effect": "Allow", "Action": [ "ec2:CreateTags" ], "Resource": "arn:aws:ec2:*::snapshot/*" } ] }
Refer to the AWS documentation for more information. Once you have added the required role, annotate the auto-backup service account to use this role.
k annotate serviceaccount auto-backup \ eks.amazonaws.com/role-arn=arn:aws:iam::[PROJECT_ID]:role/[ROLE_NAME]

Configuring backups on Google Cloud

Create a service account with a role that allows it to manage storage and snapshot. Make sure the cluster is leveraging workload identity. Then bind the auto-backup service account to the role.
gcloud iam service-accounts add-iam-policy-binding \ --role roles/iam.workloadIdentityUser \ --member "serviceAccount:[PROJECTID].svc.id.goog[default/auto-backup]" [SERVICE_ACCOUNT]@[PROJECTID].iam.gserviceaccount.com
Annotate the service account.
k annotate serviceaccount auto-backup \ iam.gke.io/gcp-service-account=[SERVICE_ACCOUNT]@[PROJECTID].iam.gserviceaccount.com
Refer to the GKE documentation for more information.

Modifying auto-backup defaults

The auto-backup jobs use the following default configuration.
  • Runs at 11AM UTC each day
  • Two snapshots retained
  • No disaster recovery region set
You can modify these settings from within Voila.
Example
auto-backup job --auto --keep 3 --copy-to us-west --copy-keep 4 | kubectl apply -f -
Syntax
auto-backup job|run --auto --keep <N> --copy-to <REGION> --copy-keep <N> --copy-kind partial|full
Run auto-backup -h for more details.

Restoring a snapshot

The rewind tool allows you to create new set of persistent volume claims for a running service from current snapshotted volumes. Simply call rewind --service redis for instance and follow the instruction written on the output.
To cleanup dangling resources and their disk you can use rewind --cleanup.
You can also check the file /opt/aporeto/voila/flavors/k8s/helpers/rewind/README.md for more details.
Supported providers:
  • AWS: Elastic Block Store (EBS)
  • GCP: persistent disk (PD)

Restoring mongo in another namespace

You can also restore a snapshot in a different namespace to take a look at it or for a selective restoration. We will leverage rewind to restore a mongodb cluster on another namespace to restore or explore a snapshot. This can be useful to restore only a few documents instead of rolling back the entire cluster.
Create a temporary namespace and create persistent volume claims from snapshots for the primary node only.
k create ns test rewind --target-namespace test --service mongo --pvc "mongodb-shard-(config-0|data-[0-2]-0)" --last
Should end up with:
VOILA_K8S_NAMESPACE=test k get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE snap-mongodb-shard-config-0 Bound sandbox-mongodb-shard-config-0-1610653775 50Gi RWO fast-xfs 2m9s snap-mongodb-shard-data-0-0 Bound sandbox-mongodb-shard-data-0-0-1610653775 1000Gi RWO fast-xfs 2m1s snap-mongodb-shard-data-1-0 Bound sandbox-mongodb-shard-data-1-0-1610653775 1000Gi RWO fast-xfs 89s snap-mongodb-shard-data-2-0 Bound sandbox-mongodb-shard-data-2-0-1610653775 1000Gi RWO fast-xfs 57s
Deploy mongo to that namespace with the following options:
VOILA_K8S_NAMESPACE=test deploy i mongodb-shard \ --set global.resources.enabled=false \ --set nodeAffinity.operator=NotIn \ --set nodeAffinity.value=mongodb \ --set volumeClaimTemplateName=snap \ --set 'shard.shards[0].replicas=1' \ --set 'shard.shards[1].replicas=0' \ --set config.replicas=1 \ --set router.replicas=1 \ --set useNamespacedFQDN=false \ --set recoveryMode=true
Because we are using disk-based snapshots that are using the XFS file system we cannot mount those on the same node that are running the snapshotted disk. There is a conflict in XFS UID (the pod stays at container creating and barks a about bad mount option…​). You might need to pop more nodes like the MongoDB ones.
Should end up as:
VOILA_K8S_NAMESPACE=test k get pod NAME READY STATUS RESTARTS AGE mongodb-shard-config-0 2/2 Running 0 26s mongodb-shard-data-0-0 2/2 Running 0 26s mongodb-shard-data-1-0 2/2 Running 0 26s mongodb-shard-data-2-0 2/2 Running 0 26s mongodb-shard-router-0 2/2 Running 0 26s
If there is some error related to the volume mount try VOILA_K8S_NAMESPACE=test restart -s mongodb-shard-data-2 for instance.
Make sure it uses the correct disk:
VOILA_K8S_NAMESPACE=test k get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE snap-mongodb-shard-config-0 Bound sandbox-mongodb-shard-config-0-1610653775 50Gi RWO fast-xfs 3h16m snap-mongodb-shard-data-0-0 Bound sandbox-mongodb-shard-data-0-0-1610653775 1000Gi RWO fast-xfs 3h16m snap-mongodb-shard-data-1-0 Bound sandbox-mongodb-shard-data-1-0-1610653775 1000Gi RWO fast-xfs 3h15m snap-mongodb-shard-data-2-0 Bound sandbox-mongodb-shard-data-2-0-1610653775 1000Gi RWO fast-xfs 3h15m
No new persistent volume claim created means that we are using the one based on the snapshots.
Check that the recovery worked by checking the logs. It should end up with:
[RECOVERY] Recovery operation completed you can update the services to regular mode.
Now we need to update the deployment to start as a cluster:
VOILA_K8S_NAMESPACE=test deploy u mongodb-shard \ --set global.resources.enabled=false \ --set nodeAffinity.operator=NotIn \ --set nodeAffinity.value=mongodb \ --set volumeClaimTemplateName=snap \ --set 'shard.shards[0].replicas=1' \ --set 'shard.shards[1].replicas=0' \ --set config.replicas=1 \ --set router.replicas=1 \ --set useNamespacedFQDN=false \ --set recoveryMode=false
After few seconds check the state of the pods and the state of the cluster with:
VOILA_K8S_NAMESPACE=test mgos status MongoDB status * Sharding status: Shard shard-z0-2 tagged as [z0] members - mongodb-shard-data-2-0.mongodb-shard-data-2:27018 Shard reports-z1-0 tagged as [z1] members - mongodb-shard-reports-0-0.mongodb-shard-reports-0:27018 Shard shard-z0-1 tagged as [z0] members - mongodb-shard-data-1-0.mongodb-shard-data-1:27018 Shard shard-z0-0 tagged as [z0] members - mongodb-shard-data-0-0.mongodb-shard-data-0:27018 * Config node replicaset: mongodb-shard-config-0.mongodb-shard-config:27019 PRIMARY * Data shard 0 mongodb-shard-data node replicaset: mongodb-shard-data-0-0.mongodb-shard-data-0:27018 PRIMARY * Data shard 1 mongodb-shard-data node replicaset: mongodb-shard-data-1-0.mongodb-shard-data-1:27018 PRIMARY * Data shard 2 mongodb-shard-data node replicaset: mongodb-shard-data-2-0.mongodb-shard-data-2:27018 PRIMARY * Data shard 0 mongodb-shard-reports node replicaset: mongodb-shard-reports-0-0.mongodb-shard-reports-0:27018 PRIMARY
You are ready to explore data with:
VOILA_K8S_NAMESPACE=test mgos
To stream data from this cluster to the main one:
# stream data! mgos stream listen Listening for data stream on: 10.64.20.54:42000 Use the above line as streamer target # send data to the listener VOILA_K8S_NAMESPACE=test mgos stream to 10.64.20.54:42000 --db squall --collection=policy Streaming data to 10.64.20.54:42000... with args --db squall --collection=policy 2021-01-20T19:48:47.307+0000 writing squall.policy to archive on stdout 2021-01-20T19:48:47.604+0000 done dumping squall.policy (4567 documents) Transfert is complete

Backing up the Voila environment

A Voila environment is set with a version control system (git).
Each time you exit an activated Voila environment, it will commit its changes and encrypt private information like:
This means that if the Voila environment is
not
activated, you cannot edit those files.
Once activated, the files are decrypted and you can edit them either from the Voila shell, or from the machine that is mounting the Voila environment.
The Voila environment
MUST
be saved as its your entry point to manage Microsegmentation Console upgrades and operations.
The recommended way is to push the Voila environment to a git remote repository like GitHub or GitLab. If you don’t have a git remote repository infrastructure, just make sure to backup the Voila environment to a safe place each time you are done using it.

Recommended For You