GCP

Instructions to install in GCP infra

1. Minimal Installation:

   • You need a system with a minimum of 16 CPUs. We recommend using 3 nodes with 8 cores each, totalling 48GB.

   • All machines must reside in the same zone to avoid data transfer charges across zones. Our Obsrv installer will automatically create the GKE cluster for you.

2. Networking Environment:

   • The networking requirements change w.r.t the type of cluster we are creating. Follow this link to find out different types of clusters supported by Google Cloud in the region you are installation.

   • By default our setup script creates a Single Zonal Public Cluster and the required VPC’s for it.

Infrastructure Requirements

1. System Specifications

• CPU Requirements:

  • Minimum: 16 CPUs.

  • Recommended: 2 nodes with 8 cores each and 32GB total RAM.

The installation includes both lakehouse and real-time OLAP storage by default. If lakehouse is not needed, the requirements remain the same (16 CPUs, 32GB RAM).

• Zone Constraints: All instances must be in the same GCP zone to avoid inter-zone data transfer costs. The Obsrv installer will automatically create the GKE (Google Kubernetes Engine) cluster.

2. Networking Setup

• Cluster Type: The default setup creates a Single Zonal Public Cluster.

• More Options: Refer to GCP Cluster Types for alternatives.

---

Prerequisites

Before starting installation, install the following CLI tools on your Linux-based system:

Tool	Version	Installation Command	Official Documentation
Terraform	1.5.x or earlier	curl "https://releases.hashicorp.com/terraform/1.5.2/terraform_1.5.2_linux_amd64.zip" -o terraform.zip && unzip terraform.zip && sudo mv terraform /usr/local/bin/ && rm terraform.zip	Terraform Install
Terragrunt	0.48 or later	curl -OL https://github.com/gruntwork-io/terragrunt/releases/download/v0.49.0/terragrunt_linux_amd64 && sudo mv terragrunt_linux_amd64 /usr/local/bin/terragrunt && sudo chmod +x /usr/local/bin/terragrunt	Terragrunt Install
Helm	3.10.2 or later	curl https://get.helm.sh/helm-v3.10.2-linux-amd64.tar.gz -o helm.tar.gz && tar -zxvf helm.tar.gz && sudo mv linux-amd64/helm /usr/local/bin/	Helm Install
Google Cloud SDK	Latest	gcloud init && gcloud auth application-default login Install plugin: gcloud components install gke-gcloud-auth-plugin	gcloud SDK Install

---

Installation Steps

1. Clone the Obsrv Repository

git clone https://github.com/Sanketika-Obsrv/obsrv-automation.git
cd obsrv-automation/terraform/gcp

2. Configure the Kubernetes Cluster

1. Update Configuration File:

   Edit vars/cluster_overrides.tfvars with your GCP-specific values:

   project                     = "your-gcp-project-id"
   building_block              = "education"
   env                         = "dev"
   region                      = "us-central1"
   gke_cluster_location        = "us-central1-a"
   zone                        = "us-central1-a"
   timezone                    = "UTC"
   service_type                = "LoadBalancer"
   gke_node_pool_instance_type = "e2-standard-4"
   gke_node_pool_scaling_config = {
     desired_size = 2
     max_size     = 3
     min_size     = 1
   }

2. Create the GCP Config File:

   Create obsrv.conf inside infra-setup/:

   GOOGLE_PROJECT_ID="your-project-id"
   GOOGLE_TERRAFORM_BACKEND_BUCKET="your-tfstate-bucket-name"
   GOOGLE_TERRAFORM_BACKEND_BUCKET_REGION="asia-south1"

   ℹ️ Bucket will be created automatically if it doesn’t exist.

3. Run the Installation Script

cd ../../infra-setup
chmod +x obsrv.sh
./obsrv.sh install --provider gcp --config ./obsrv.conf

This script:

• Deploys infrastructure via Terragrunt.

• Creates the GKE cluster and VPCs.

• Sets up kubeconfig at terraform/gcp/credentials/<file>.

export KUBECONFIG=$(pwd)/terraform/gcp/credentials/<kubeconfig-filename>

4. Verify the Cluster

kubectl get nodes

All GKE nodes should appear in Ready state.

---

Helm Chart Configuration

1. Navigate to Helm Directory

cd ../helmcharts/

2. Update GCP Cloud Configuration

Edit global-cloud-values-gcp.yaml:

global:
  project_id: <your-gcp-project-id>
  cloud_storage_region: asia-south1
  cloud_storage_config: <client_email, private_key, project_id> # From terraform credentials
  postgresql_backup_cloud_bucket: <bucket-name>
  checkpoint_bucket: <bucket-name>
  velero_backup_cloud_bucket: <bucket-name>

service_accounts:
  config-api: <sa-name>
  dataset-api: <sa-name>
  druid-raw: <sa-name>
  flink-sa: <sa-name>
  postgres: <sa-name>
  secor: <sa-name>
  spark: <sa-name>
  velero: <sa-name>

3. Install Core Services

cd kitchen
export cloud_env=gcp
bash install.sh core-setup

4. Configure Domain Mapping

1. Get the external IP:

   kubectl get svc -n kong

2. Update global-values.yaml:

   domain: "<external-ip>.sslip.io"

5. Install All Obsrv Services

bash install.sh all

6. Validate Installation

• Open in browser: https://<external-ip>.sslip.io/console

• Verify pods:

  kubectl get pods -A

---

Upgrade Steps

1. Pull Latest Code

cd obsrv-automation
git pull

2. Upgrade Infrastructure

cd ../infra-setup
./obsrv.sh install --provider gcp --config ./obsrv.conf

3. Upgrade Services

cd ../helmcharts/kitchen
export cloud_env=gcp
bash install.sh core-setup
bash install.sh all

---

By following the above steps, you can successfully install, configure, and manage Obsrv on Google Cloud Platform (GCP) using best practices for infrastructure automation and service deployment.