Data Backup and Restoration

Instructions to restoration of obsrv from the backups

---

📘 Overview

This guide provides comprehensive instructions to restore Obsrv data using Velero, Redis, and PostgreSQL backups. It also includes procedures for Druid data migration between S3 buckets.

---

🧾 Introduction

This document provides a step-by-step guide for restoring data from backups in an Obsrv deployment. It covers:

• Restoration of critical services (PostgreSQL, Redis, etc.)

• Velero-based system-wide recovery

• Data migration between object storage buckets (e.g., for Druid segments)

---

🗂 Backup Storage Details

Service	S3 Bucket Naming Convention
PostgreSQL	backups-{building_block}-{env}-{account-id}
Denorm Redis	backups-{building_block}-{env}-{account-id}
Dedup Redis	backups-{building_block}-{env}-{account-id}
Dataset Events	{building_block}-{env}-{account-id}
Terraform State	Stored in: AWS_TERRAFORM_BACKEND_BUCKET_NAME
Velero Backups	velero-{building_block}-{env}-{account-id}
Flink Checkpoints	checkpoint-{building_block}-{env}-{account-id}

---

🔄 Velero: Full Restoration Workflow

✅ Prerequisites

Ensure the following tools are installed:

• AWS CLI

• Velero CLI

To install Velero:

wget https://github.com/vmware-tanzu/velero/releases/download/v1.3.2/velero-v1.3.2-linux-amd64.tar.gz
tar -xvf velero-v1.3.2-linux-amd64.tar.gz -C /tmp
sudo mv /tmp/velero-v1.3.2-linux-amd64/velero /usr/local/bin
velero version

Ensure:

• AWS credentials are configured

• Kubernetes cluster is accessible

• Velero is installed and running in the cluster

---

📦 Restoration Procedure

🔁 Restore All Services

velero restore create --from-backup <backup-file-name>
# Example:
velero restore create --from-backup velero-obsrv-daily-backup-20240904133016-20240904190534

🔧 Restore a Specific Service (e.g., PostgreSQL)

velero restore create --from-backup <backup-file-name> --include-namespaces <namespace>

✅ Verify Restoration

velero restore describe <backup-name>

---

🧪 Post-Restoration Checklist

• Ensure all pods are running (PostgreSQL, Redis, Prometheus, etc.)

• Verify data in all critical databases

• Check data processors (Kafka, Flink, Druid) resume from last committed state

• Run test queries to validate dataset access and accuracy

---

💾 Redis & PostgreSQL: Targeted Restoration

Use this when restoring only Redis and PostgreSQL without changes to S3 bucket paths.

---

🔒 Pre-Restoration

Pause these services:

• Flink Jobs

• Web Console

• Obsrv APIs

• Command Service

• Druid

• Superset

# Example to scale down
kubectl scale deployment --all --replicas=0 -n <namespace>

---

🐘 PostgreSQL Restoration

1. Access the Pod

kubectl exec -it postgresql-0 --namespace postgresql -- /bin/bash

2. Pre-Check & Clean Old Data

psql -U postgres -d postgres
DROP DATABASE druid_raw;
DROP DATABASE obsrv;
DROP DATABASE superset;

-- Optional: Create if needed
CREATE DATABASE druid_raw;
CREATE DATABASE obsrv;
CREATE DATABASE superset;

3. Copy Backup File

kubectl cp ./backup.sql postgresql/postgresql-0:/tmp/db.sql

4. Restore

psql -U postgres -f /tmp/db.sql

---

🧠 Redis Restoration

1. Download and Decompress

aws s3 cp s3://<bucket-path>/fulldb-05-07-2024.rdb.bz2 .
bzip2 -dk fulldb-05-07-2024.rdb.bz2

2. Access the Pod

kubectl exec -it pod/obsrv-<instance>-redis-master-0 -n redis -- sh

3. Disable Persistence Temporarily

redis-cli
config set appendonly no
config set save ""
SAVE

4. Replace and Restart

kubectl cp ./dump.rdb redis/obsrv-<instance>-redis-master-0:/data/dump.rdb
kubectl delete pod obsrv-<instance>-redis-master-0 -n redis

5. Verify Keys

kubectl exec -it pod/obsrv-redis-<instance>-master-0 -n redis -- sh -c 'redis-cli info'

6. Revert Configs

redis-cli
config set appendonly yes
config set save "3600 1 300 100 60 10000"
SAVE

---

✅ Final Steps After Restoration

Once PostgreSQL and Redis are restored:

# Resume Flink Jobs
kubectl scale deployment master-data-processor-ext-jobmanager --replicas=1 -n flink
kubectl scale deployment master-data-processor-ext-taskmanager --replicas=1 -n flink
kubectl scale deployment unified-pipeline-jobmanager --replicas=1 -n flink
kubectl scale deployment unified-pipeline-taskmanager --replicas=1 -n flink

# Resume other services
kubectl scale deployment --all --replicas=1 -n flink
kubectl scale deployment --all --replicas=1 -n druid-raw
kubectl scale deployment --all --replicas=1 -n superset
kubectl scale deployment --all --replicas=1 -n dataset-api
kubectl scale deployment --all --replicas=1 -n web-console

Validate all datasets and operational behavior through the Obsrv console.

---

🚚 Druid Data Migration

Use this section to migrate Druid segment data from one S3 bucket to another.

---

📌 Pre-Migration Checklist

• Ensure all segments are copied to the target bucket.

• Scale down Druid components:

kubectl scale deployment --all --replicas=0 -n druid-raw

---

📦 Migration Execution

1. Deploy a temporary Python pod:

apiVersion: v1
kind: Pod
metadata:
  name: python-migration
spec:
  containers:
  - name: python
    image: python:3.9
    command: ["sleep", "3600"]
  restartPolicy: Never

1. Run the migration script inside the pod using the AWS CLI or boto3 script to copy paths from the old bucket to the new bucket.

---

⚠️ Important Notes

• Always take snapshots before migration or restoration.

• Confirm with stakeholders before starting restoration in a production environment.

• Monitor resource usage post-restoration for anomalies.

---