Skip to content

AWS Installation Guide

Instructions to install Obsrv on AWS infra

This guide provides detailed, step-by-step instructions for installing and configuring Obsrv on AWS, utilizing Terraform, Terragrunt, and Helm.

Infrastructure Requirements

Section titled “Infrastructure Requirements”

1. System Specifications

Section titled “1. System Specifications”
  • CPU Requirements:
    • Minimum: 19 CPUs.
    • Optimal Configuration: 5 nodes with 4 cores each, totaling 80GB of RAM.

The installation package includes both lakehouse and real-time OLAP storage by default. If the lakehouse component is not required, only the real-time OLAP storage can be installed, reducing requirements to 16 CPUs and 64GB of RAM.

In this case, we recommend using 2 nodes with 8 cores each, totaling 64GB of RAM, by selecting the t2.2xlarge AWS instance type.

  • Availability Zones: All instances should be within the same availability zone to minimize cross-zone data transfer costs. The Obsrv installer will automatically create the EKS (Elastic Kubernetes Service) cluster for you.

2. Networking Setup

Section titled “2. Networking Setup”
  • CIDR Block: Use a /23 CIDR range (512 IPs) for your environment.
    • Example: A VPC with 10.0.0.0/23 provides IPs from 10.0.0.0 to 10.0.1.255.
  • Subnets: Ensure subnets are created in all availability zones within your AWS region.

Prerequisites

Section titled “Prerequisites”

Before beginning the installation, make sure the following tools are installed on your Linux-based system:

ToolVersionInstallation CommandOfficial Documentation
Terraform1.5.x or earliercurl "https://releases.hashicorp.com/terraform/1.5.2/terraform_1.5.2_linux_amd64.zip" -o terraform.zip && unzip terraform.zip && sudo mv terraform /usr/local/bin/ && rm terraform.zipTerraform Install
Terragrunt0.48 or latercurl -OL https://github.com/gruntwork-io/terragrunt/releases/download/v0.49.0/terragrunt_linux_amd64 && sudo mv terragrunt_linux_amd64 /usr/local/bin/terragrunt && sudo chmod +x /usr/local/bin/terragruntTerragrunt Install
Helm3.10.2 or latercurl https://get.helm.sh/helm-v3.10.2-linux-amd64.tar.gz -o helm.tar.gz && tar -zxvf helm.tar.gz && sudo mv linux-amd64/helm /usr/local/bin/Helm Install
AWS CLI2.10 or latercurl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" && unzip awscliv2.zip && sudo ./aws/installAWS CLI Install

Installation Steps

Section titled “Installation Steps”

1. Clone the Obsrv Repository

Section titled “1. Clone the Obsrv Repository”

Start by cloning the Obsrv automation repository and checkout to either latest release tag or master.

Terminal window
git clone https://github.com/Sunbird-Obsrv/obsrv-automation.git

2. Configure the Kubernetes Cluster

Section titled “2. Configure the Kubernetes Cluster”

By executing the following commands which will bring up the kubernetes cluster in the AWS environment of configured region.

  1. Navigate to the Configuration Directory:

    Terminal window
    cd ./obsrv-automation/terraform/aws/vars

  2. Update Configuration Files:

    • Open cluster_overides.tf and modify the configuration values to match your environment.
    Terminal window
    building_block = "obsrv"
    env = "dev"
    region = "us-east-2"
    availability_zones = ["us-east-2a", "us-east-2b", "us-east-2c"]
    timezone = "UTC"
    create_kong_ingress = "true"
    create_vpc = "true"
    create_velero_user = "true"
    eks_node_group_instance_type = ["t2.xlarge"] # Choose depending on your requirements by considering the CPU requirements
    eks_node_group_capacity_type = "ON_DEMAND"
    eks_node_group_scaling_config = { desired_size = 5, max_size = 5, min_size = 1 } # Choose depending on your requirements by considering the CPU requirements
    eks_node_disk_size = 100

  3. Configure S3 for Cluster State:

    • Open obsrv.conf and update your AWS credentials and bucket names.
    Terminal window
    AWS_ACCESS_KEY_ID=<your_access_key_id>
    AWS_SECRET_ACCESS_KEY=<your_secret_access_key>
    AWS_DEFAULT_REGION="us-east-2"
    KUBE_CONFIG_PATH="$HOME/.kube/obsrv-kube-config.yaml"
    AWS_TERRAFORM_BACKEND_BUCKET_NAME="obsrv-tfstate"
    AWS_TERRAFORM_BACKEND_BUCKET_REGION="us-east-2"

3. Run the Installation Script

Section titled “3. Run the Installation Script”

  1. Make the Script Executable:

    Terminal window
    chmod +x ./obsrv.sh

  2. Run the Installation:

    • To start the installation, run the script:
    Terminal window
    ./obsrv.sh install --config ./obsrv.conf --install_dependencies false
    • If you want the installer to automatically handle dependencies, set install_dependencies=true.

4. Verify the Cluster

Section titled “4. Verify the Cluster”

Once the installation completes, verify that your Kubernetes cluster is up and running:

Terminal window
kubectl get nodes

This should show the nodes in your Kubernetes cluster.

Helm Chart Configuration

Section titled “Helm Chart Configuration”

1. Navigate to the Helm Chart Directory

Section titled “1. Navigate to the Helm Chart Directory”
Terminal window
cd ./obsrv-automation/helmcharts/

2. Update AWS Cloud Configuration

Section titled “2. Update AWS Cloud Configuration”

Modify global-cloud-values-aws.yaml with the appropriate values for your environment:

global:
  cloud_storage_provider: "aws"
  cloud_store_provider: "s3"
  cloud_storage_region: "<region>"
  dataset_api_cloud_bucket: "<dataset_bucket_name>"
  config_api_cloud_bucket: "<config_bucket_name>"
  postgresql_backup_cloud_bucket: "<backup_bucket_name>"
  redis_backup_cloud_bucket: "<redis_backup_bucket_name>"
  velero_backup_cloud_bucket: "<velero_backup_bucket_name>"
  cloud_storage_bucket: "<storage_bucket_name>"
  hudi_metadata_bucket: "s3a://<hudi_bucket_name>/hudi"
  cloud_storage_config: |
    '{"identity":"<access-key>","credential":"<secret-key>","region":"<region-name>"}'


  storage_class_name: "gp2"
  checkpoint_bucket: "s3://<checkpoint-bucket-name>"
  s3_access_key: "<aws-access-key>"
  s3_secret_key: "<aws-secret-key>"


kong_annotations:
  service.beta.kubernetes.io/aws-load-balancer-type: nlb
  service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
  service.beta.kubernetes.io/aws-load-balancer-eip-allocations: "<elastic-ip>"
  service.beta.kubernetes.io/aws-load-balancer-subnets: "<subnet-id>"


service_accounts:
  enabled: true
  secor: eks.amazonaws.com/role-arn: "<role-arn>"
  dataset_api: eks.amazonaws.com/role-arn: "<role-arn>"
  config_api: eks.amazonaws.com/role-arn: "<role-arn>"
  druid_raw: eks.amazonaws.com/role-arn: "<role-arn>"
  flink: eks.amazonaws.com/role-arn: "<role-arn>"
  postgresql_backup: eks.amazonaws.com/role-arn: "<role-arn>"
  redis_backup: eks.amazonaws.com/role-arn: "<role-arn>"
  s3_exporter: eks.amazonaws.com/role-arn: "<role-arn>"
  spark: eks.amazonaws.com/role-arn: "<role-arn>"


velero-backup:
  credentials:
    useSecret: true
    secretContents:
      cloud: |
        [default]
        aws_access_key_id="<aws-access-key>"
        aws_secret_access_key="<aws-secret-key>"


trino:
  additionalCatalogs:
    lakehouse: |-
      connector.name=hudi
      hive.metastore.uri=thrift://hudi-hms.hms.svc:9083
      hive.s3.aws-access-key=<aws-access-key>
      hive.s3.aws-secret-key=<aws-secret-key>
      hive.s3.ssl.enabled=false

3. Update Domain Configuration

Section titled “3. Update Domain Configuration”

In global-values.yaml, replace <domain> with your actual domain or Elastic IP:

domain: "<domain>.sslip.io"

4. Update Private and Public Keys

Section titled “4. Update Private and Public Keys”

Follow these steps to generate and configure the private and public keys for the web console and dataset API:

Step 1: Generate a Private Key

Section titled “Step 1: Generate a Private Key”

Run the following command to generate a private key for the web console:

Terminal window
openssl genpkey -algorithm RSA -out private_key.pem -pkeyopt rsa_keygen_bits:2048
  • Open the generated private_key.pem file.
  • Copy its contents and update the USER_TOKEN_PRIVATE_KEY field in the following file:
    obsrv-automation/helmcharts/services/web-console/values.yaml

Example:

USER_TOKEN_PRIVATE_KEY: |-
    <paste-private-key-here>

Step 2: Generate a Public Key

Section titled “Step 2: Generate a Public Key”

Using the private key generated above, create a public key with the following command:

Terminal window
openssl rsa -pubout -in private_key.pem -out public_key.pem
  • Open the generated public_key.pem file.
  • Copy its contents and update the user_token_public_key field in the following file:
    obsrv-automation/helmcharts/services/dataset-api/values.yaml

Example:

user_token_public_key: <paste-public-key-here>

5. Install Obsrv

Section titled “5. Install Obsrv”

Make the script executable and set the environment variables and run the installation:

Terminal window
export cloud_env=aws
export AWS_ACCESS_KEY_ID=<aws-access-key>
export AWS_SECRET_ACCESS_KEY=<aws-secret-key>
export AWS_DEFAULT_REGION=<aws-region>
export KUBE_CONFIG_PATH="$HOME/.kube/obsrv-kube-config.yaml"
export KUBECONFIG="$HOME/.kube/obsrv-kube-config.yaml"
chmod +x ./kitchen/install.sh
./kitchen/install.sh all

Post-Installation Verification

Section titled “Post-Installation Verification”

After completing the installation, follow these steps to verify that all components are running correctly:

1. Check Kubernetes Components

Section titled “1. Check Kubernetes Components”

  1. Verify all pods are running:

    Terminal window
    kubectl get pods -A

    All pods should be in Running state. Common namespaces to check:

    • flink: Core Pipeline
    • monitoring: Monitoring stack
    • dataset-api: Dataset APIs
    • web-console: Dataset Management console

  2. Check Services:

    Terminal window
    kubectl get svc -A

    Verify that essential services have external IPs assigned, particularly the Kong service.

If any component fails these checks, refer to the component-specific logs:

Terminal window
kubectl logs -f <pod-name> -n <namespace>

Upgrade Steps

Section titled “Upgrade Steps”

  1. Pull the Latest Code:

    Terminal window
    cd ./obsrv-automation
    git pull
    cd ./automation-scripts/infra-setup

  2. Update Configurations: Review and update configuration values as needed.

  3. Run Terraform for Upgrade:

    Terminal window
    ./obsrv.sh install --config ./obsrv.conf --install_dependencies false

  4. Upgrade with Updated Cloud Values:

    Terminal window
    export cloud_env=aws
    export AWS_ACCESS_KEY_ID=<aws-access-key>
    export AWS_SECRET_ACCESS_KEY=<aws-secret-key>
    export AWS_DEFAULT_REGION=<aws-region>
    export KUBE_CONFIG_PATH="$HOME/.kube/obsrv-kube-config.yaml"
    export KUBECONFIG="$HOME/.kube/obsrv-kube-config.yaml"
    chmod +x ./kitchen/install.sh
    ./kitchen/install.sh all

By following these steps, you will ensure a successful installation and configuration of Obsrv on AWS.