Chapter 11: Infrastructure as Code, Kubernetes, and Observability

The production outage started at 2:07 AM.

Priya's phone lit up with a PagerDuty alert. latency_p99 > 2000ms. By the time she unlocked her screen, the Slack thread was already moving. The on-call Senior Engineers were in the AWS console, clicking through log groups, trying to find which service had tipped over. Screenshots of CloudWatch. "I think it's the payment service?" "No, checkout is also spiking."

At 2:15 AM, Priya — the Staff Engineer — posted a single message: "Deploy #1842 to production. Reverting now." She ran terraform apply from her phone. By 2:18, the bad deployment was rolled back. By 2:22, she posted the incident summary: root cause, affected services, time to resolve. By 2:30, she was back asleep.

The Senior Engineers were still trying to find the right log group.

This is not a story about intelligence. Priya is not smarter than the Senior Engineers. She is not a 10x developer. She has something they don't: she can see the system. When an alert fires, she knows exactly where to look because she built the infrastructure, she wrote the dashboards, and she instrumented the code. The Senior Engineers can write brilliant Node.js. But they treat infrastructure as someone else's problem — the DevOps team's problem, the SRE team's problem, the "platform" team's problem.

At ₹60 lakh to ₹1 crore, infrastructure is your problem. Not because you'll be on-call forever. Because the engineers who command those packages are the ones who can debug a production issue from alert to root cause without waiting for someone else to tell them which pod crashed.

This chapter gives you that ability. You will learn Infrastructure as Code with Terraform, the Kubernetes primitives that run your services, the CI/CD pipelines that ship them, and the observability stack that lets you sleep through the night. By the end, you will be able to provision infrastructure, deploy to Kubernetes, and instrument a Node.js service so that when something breaks at 2 AM, you are the one who fixes it in ten minutes.

The IaC Mindset: Infrastructure Is Just Another Codebase

Before you write a single line of Terraform, you need to internalize one idea: infrastructure is not a console-clicking exercise. It is a codebase. It gets reviewed, tested, versioned, and rolled back — exactly like your application code.

The old way was manual. Someone clicked through the AWS console to create an EC2 instance, an RDS database, a load balancer. They took screenshots. They wrote a wiki page. Six months later, nobody knew why the security group had that one weird rule. The new way — the ₹1 crore way — is declarative. You write what you want. A tool makes it real. If something changes, you change the code and apply it again. The code is the source of truth.

Terraform: The Industry Standard

Terraform is the lingua franca of cloud infrastructure. It is not the only option — Pulumi is gaining ground, and we will cover it — but Terraform is what you will find at 90% of companies paying ₹60 lakh and above. Learn it first.

Terraform works in three stages: write, plan, apply. You write HCL (HashiCorp Configuration Language) files that declare what infrastructure you want. terraform plan shows you what will change. terraform apply makes those changes real. The state of your infrastructure is stored in a state file — a JSON blob that maps your HCL declarations to real cloud resources.

Here is a minimal Terraform configuration that provisions a Node.js application on AWS ECS:

# main.tf
terraform {
  required_version = ">= 1.5"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
  backend "s3" {
    bucket         = "myapp-terraform-state"
    key            = "production/terraform.tfstate"
    region         = "ap-south-1"
    dynamodb_table = "terraform-locks"
  }
}

provider "aws" {
  region = "ap-south-1"
}

module "vpc" {
  source = "terraform-aws-modules/vpc/aws"
  version = "5.0.0"

  name = "myapp-vpc"
  cidr = "10.0.0.0/16"

  azs             = ["ap-south-1a", "ap-south-1b"]
  private_subnets = ["10.0.1.0/24", "10.0.2.0/24"]
  public_subnets  = ["10.0.101.0/24", "10.0.102.0/24"]

  enable_nat_gateway = true
  single_nat_gateway = true
}

module "ecs" {
  source = "terraform-aws-modules/ecs/aws"
  version = "5.2.0"

  cluster_name = "myapp-cluster"

  fargate_capacity_providers = {
    FARGATE = {
      default_capacity_provider_strategy = {
        weight = 100
      }
    }
  }
}

resource "aws_ecs_task_definition" "api" {
  family                   = "myapp-api"
  network_mode             = "awsvpc"
  requires_compatibilities = ["FARGATE"]
  cpu                      = "512"
  memory                   = "1024"
  execution_role_arn       = aws_iam_role.ecs_execution.arn
  task_role_arn            = aws_iam_role.ecs_task.arn

  container_definitions = jsonencode([{
    name  = "api"
    image = "123456789012.dkr.ecr.ap-south-1.amazonaws.com/myapp-api:latest"
    portMappings = [{
      containerPort = 3000
      protocol      = "tcp"
    }]
    environment = [
      { name = "NODE_ENV", value = "production" },
      { name = "DB_HOST", value = module.rds.db_instance_address }
    ]
    secrets = [
      { name = "DB_PASSWORD", valueFrom = aws_ssm_parameter.db_password.arn }
    ]
    logConfiguration = {
      logDriver = "awslogs"
      options = {
        "awslogs-group"         = "/ecs/myapp-api"
        "awslogs-region"        = "ap-south-1"
        "awslogs-stream-prefix" = "ecs"
      }
    }
  }])
}

This is not a toy example. This is production infrastructure. Let me walk you through what each block is doing, because understanding this is the difference between copying from StackOverflow and actually knowing your infrastructure.

The terraform block declares the required providers and the backend. The backend is where your state file lives. Never store state locally. Never commit it to git. Use an S3 bucket with DynamoDB for locking — this prevents two people from running terraform apply at the same time and corrupting your state. I have seen this happen. The cleanup took three days.

The provider block configures AWS. Notice ap-south-1 — the Mumbai region. If your users are in India, your infrastructure should be in India. Latency matters. A 200ms round trip to us-east-1 versus 20ms to ap-south-1 is the difference between a snappy API and a sluggish one.

The module blocks are where Terraform gets powerful. Instead of writing 200 lines of VPC configuration, you use the official AWS VPC module. Modules are reusable, versioned, and tested. You should write your own modules too — for your standard service, your standard database, your standard queue. At a company paying ₹60 lakh+, you are not provisioning one-off resources. You are building a platform of reusable infrastructure components.

The resource block for the ECS task definition is where your Node.js application meets infrastructure. The container image comes from ECR. Environment variables come from Terraform outputs. Secrets come from AWS SSM Parameter Store — never hardcode secrets in Terraform. The valueFrom syntax tells ECS to inject the secret at runtime.

Terraform State: The Source of All Truth and All Pain

State is the most important concept in Terraform, and the one that causes the most production incidents. The state file maps your HCL declarations to real cloud resource IDs. When you run terraform plan, Terraform reads the state file, compares it to your HCL, and calculates the diff. When you run terraform apply, it updates the state file to reflect the new reality.

If the state file is lost, Terraform does not know which resources it manages. It will try to recreate everything. If the state file is out of sync — because someone changed something in the console — Terraform will either overwrite the manual change or fail with a confusing error.

Here is what good state management looks like:

# Never do this:
# terraform {
#   backend "local" {
#     path = "terraform.tfstate"
#   }
# }

# Always do this:
terraform {
  backend "s3" {
    bucket         = "myapp-terraform-state-${var.environment}"
    key            = "${var.environment}/terraform.tfstate"
    region         = "ap-south-1"
    dynamodb_table = "terraform-locks"
    encrypt        = true
  }
}

The key uses ${var.environment}. This is deliberate. You should have separate state files for each environment — development, staging, production. Never share state across environments. A terraform destroy in the wrong environment should not be possible.

Terraform Workspaces vs. Directory Separation

There are two schools of thought for managing multiple environments in Terraform: workspaces and directory separation. Terraform workspaces let you use the same configuration with different state files. Directory separation means each environment has its own directory with its own configuration.

Workspaces look elegant at first:

terraform workspace new production
terraform workspace new staging
terraform workspace select production
terraform apply

But workspaces have a fundamental problem: they assume every environment is identical. They are not. Production needs more CPU. Staging needs fewer instances. Development can share a database. With workspaces, you end up with conditionals everywhere:

resource "aws_ecs_task_definition" "api" {
  cpu    = terraform.workspace == "production" ? 2048 : 512
  memory = terraform.workspace == "production" ? 4096 : 1024
}

This becomes unmaintainable fast. The ₹1 crore approach is directory separation with shared modules:

infrastructure/
├── modules/
│   ├── service/
│   │   ├── main.tf
│   │   ├── variables.tf
│   │   └── outputs.tf
│   └── database/
│       ├── main.tf
│       ├── variables.tf
│       └── outputs.tf
├── production/
│   ├── main.tf
│   ├── variables.tf
│   └── terraform.tfvars
├── staging/
│   ├── main.tf
│   ├── variables.tf
│   └── terraform.tfvars
└── development/
    ├── main.tf
    ├── variables.tf
    └── terraform.tfvars

Each environment directory calls the same modules with different parameters. The module is the source of truth for how a service is structured. The environment directory is the source of truth for how that service is configured in a specific context.

Here is what a production main.tf looks like with this pattern:

module "api_service" {
  source = "../modules/service"

  environment     = "production"
  container_image = "123456789012.dkr.ecr.ap-south-1.amazonaws.com/myapp-api:${var.image_tag}"
  container_port  = 3000
  cpu             = 2048
  memory          = 4096
  desired_count   = 3
  min_capacity    = 3
  max_capacity    = 10

  vpc_id            = module.vpc.vpc_id
  private_subnet_ids = module.vpc.private_subnets

  db_host     = module.database.endpoint
  db_password_arn = module.database.password_parameter_arn

  health_check_path = "/health"
  alb_listener_arn  = module.alb.https_listener_arn
  domain_name       = "api.myapp.com"
}

This is clean. This is reviewable. This is what infrastructure looks like when it is treated as a codebase.

Pulumi: Infrastructure in Real Code

Pulumi is the alternative worth knowing. Instead of HCL, you write infrastructure in TypeScript, Python, or Go. For a Node.js engineer, this is compelling — you already know the language.

import * as aws from "@pulumi/aws";
import * as awsx from "@pulumi/awsx";

const cluster = new aws.ecs.Cluster("myapp-cluster");

const alb = new awsx.lb.ApplicationLoadBalancer("myapp-alb", {
  internal: false,
});

const repo = new aws.ecr.Repository("myapp-api-repo");

const image = awsx.ecs.Image.fromDockerBuild("myapp-api-image", {
  context: "../api",
  dockerfile: "../api/Dockerfile",
});

const service = new awsx.ecs.FargateService("myapp-api", {
  cluster: cluster.arn,
  taskDefinitionArgs: {
    container: {
      name: "api",
      image: image.imageUri,
      cpu: 512,
      memory: 1024,
      portMappings: [{ containerPort: 3000 }],
      environment: [
        { name: "NODE_ENV", value: "production" },
      ],
    },
  },
  desiredCount: 3,
});

export const url = alb.loadBalancer.dnsName;

Pulumi's advantage is real: loops, conditionals, and abstractions are native to the language. You do not need to learn HCL's limited expression syntax. You can write functions, classes, and modules. You can use npm packages. The downside: Pulumi is newer, the community is smaller, and most companies still use Terraform. Learn Terraform first. Know Pulumi exists. If you join a startup that uses Pulumi, you will pick it up in a week.

Kubernetes: The Operating System of the Cloud

If you work at a company paying ₹60 lakh+, your code almost certainly runs on Kubernetes. Not because Kubernetes is the best solution for every problem — it is not — but because it has won. Every cloud provider offers a managed Kubernetes service. Every observability tool integrates with it. Every CI/CD platform targets it.

Kubernetes is a container orchestrator. You give it container images. It decides which machines to run them on, keeps them running, scales them up and down, routes traffic to them, and replaces them when they die. It is the operating system, and your containers are the processes.

The Primitives You Actually Need

Kubernetes has a hundred resource types. You need about eight. Here they are, in the order you will encounter them when deploying a Node.js service.

Pod — The smallest deployable unit. A pod is one or more containers that share a network namespace and storage. In practice, one pod runs one container. Pods are ephemeral. They die. Kubernetes replaces them. You almost never create pods directly.

Deployment — The resource that manages pods. You declare how many replicas you want, which container image to run, and what resources it needs. The deployment controller makes it real. If a pod dies, the deployment creates a new one. If you update the image, the deployment rolls out the change gradually.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-service
  labels:
    app: api-service
spec:
  replicas: 3
  selector:
    matchLabels:
      app: api-service
  template:
    metadata:
      labels:
        app: api-service
    spec:
      containers:
        - name: api
          image: 123456789012.dkr.ecr.ap-south-1.amazonaws.com/api-service:v1.2.3
          ports:
            - containerPort: 3000
          resources:
            requests:
              cpu: "500m"
              memory: "512Mi"
            limits:
              cpu: "1000m"
              memory: "1024Mi"
          readinessProbe:
            httpGet:
              path: /health
              port: 3000
            initialDelaySeconds: 10
            periodSeconds: 5
          livenessProbe:
            httpGet:
              path: /health
              port: 3000
            initialDelaySeconds: 30
            periodSeconds: 10
          env:
            - name: NODE_ENV
              value: "production"
            - name: DB_HOST
              valueFrom:
                secretKeyRef:
                  name: api-secrets
                  key: db-host

The resources block is not optional. If you do not set CPU and memory requests and limits, the scheduler does not know how to place your pods, and a noisy neighbor can starve your service. Set requests to what your service needs under normal load. Set limits to the maximum it should ever use. If a container exceeds its memory limit, Kubernetes kills it. This is better than letting it take down the node.

The readinessProbe tells Kubernetes when your pod is ready to receive traffic. Without it, Kubernetes sends traffic to your pod the moment the container starts — before your Node.js server is listening. Users get 502 errors. The livenessProbe tells Kubernetes when your pod is dead and needs to be replaced. Without it, a stuck process runs forever, serving no traffic, consuming resources.

Service — A stable network endpoint for a set of pods. Pods come and go. Their IP addresses change. A Service gives you a single DNS name that always routes to healthy pods.

apiVersion: v1
kind: Service
metadata:
  name: api-service
spec:
  selector:
    app: api-service
  ports:
    - port: 80
      targetPort: 3000
      protocol: TCP
  type: ClusterIP

ClusterIP is the default and the right choice for internal services. Your API service does not need a public IP. The ingress controller handles external traffic.

Ingress — Routes external HTTP/S traffic to services based on hostname and path. Think of it as a reverse proxy configured through Kubernetes resources.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: api-ingress
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - api.myapp.com
      secretName: api-tls
  rules:
    - host: api.myapp.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: api-service
                port:
                  number: 80

ConfigMap — Non-sensitive configuration as key-value pairs. Database URLs, feature flags, log levels.

Secret — Sensitive configuration. API keys, database passwords, JWT signing keys. Secrets are base64-encoded by default, which is not encryption. In production, use a secrets management solution: AWS Secrets Manager, HashiCorp Vault, or the External Secrets Operator.

apiVersion: v1
kind: ConfigMap
metadata:
  name: api-config
data:
  NODE_ENV: "production"
  LOG_LEVEL: "info"
  REDIS_URL: "redis://redis-service:6379"
---
apiVersion: v1
kind: Secret
metadata:
  name: api-secrets
type: Opaque
data:
  DB_PASSWORD: <base64-encoded-value>
  JWT_SECRET: <base64-encoded-value>

HorizontalPodAutoscaler (HPA) — Scales pods based on CPU, memory, or custom metrics. This is what keeps your service alive under load without human intervention.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api-service
  minReplicas: 3
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 80

The HPA reads metrics from the Kubernetes metrics server every 15 seconds. When CPU utilization exceeds 70%, it adds pods. When it drops below 70%, it removes them. The minReplicas of 3 ensures you always have enough pods to handle baseline traffic and survive a node failure. The maxReplicas of 20 prevents an infinite scaling loop from bankrupting you.

The Kubernetes Debugging Flow

When something breaks in Kubernetes — and it will — you need a systematic debugging flow. Here is the one Priya used at 2 AM. Memorize it.

Step 1: Check the pods. Are they running? How many restarts?

kubectl get pods -n production
kubectl describe pod api-service-7d8f9c6b5-xk2lm -n production

The describe output tells you the pod's lifecycle: when it was scheduled, whether the image pulled successfully, whether the readiness probe passed, and the last few events before it died.

Step 2: Check the logs. What was the pod doing before it crashed?

kubectl logs api-service-7d8f9c6b5-xk2lm -n production --tail=100
kubectl logs api-service-7d8f9c6b5-xk2lm -n production --previous  # logs from the previous container instance

Step 3: Check the events. What is happening at the cluster level?

kubectl get events -n production --sort-by='.lastTimestamp'

Events show you scheduling failures, image pull errors, probe failures, and OOM kills. They are the first place to look when pods are not starting.

Step 4: Exec into the container. Can you reach the dependencies?

kubectl exec -it api-service-7d8f9c6b5-xk2lm -n production -- sh
# Inside the container:
curl http://db-service:5432  # Can you reach the database?
nslookup redis-service       # Does DNS resolve?
env                          # Are the environment variables set?

Step 5: Check the deployment history. Was there a recent change?

kubectl rollout history deployment/api-service -n production
kubectl rollout undo deployment/api-service -n production --to-revision=3

Most production incidents are caused by a bad deployment. The rollout undo command reverts to the previous revision. This is the fastest way to restore service. Priya knew this. She checked the deployment history, saw the new revision, and rolled it back. Ten minutes. Done.

A Real Kubernetes Debugging Story

Rahul, a Senior Engineer at a Bangalore fintech startup, was on-call when the payment service started returning 504 errors. The pods were running. The logs showed nothing unusual. The database was responsive. He spent 45 minutes checking everything he could think of.

The issue: the service's readinessProbe was configured with a 5-second timeout, but the /health endpoint was checking database connectivity, which under load took 7 seconds. Kubernetes marked every pod as not-ready. The ingress controller stopped routing traffic. The pods were alive, healthy, and serving zero requests.

The fix was two lines:

readinessProbe:
  httpGet:
    path: /health
    port: 3000
  initialDelaySeconds: 10
  periodSeconds: 5
  timeoutSeconds: 10  # This was 1 (default). Changed to 10.

And the /health endpoint was split into two: /health/liveness for the liveness probe (cheap, checks only that the process is alive) and /health/readiness for the readiness probe (checks dependencies). This is a pattern you should adopt in every Node.js service:

const express = require('express');
const app = express();

// Liveness: is the process alive? Cheap. No dependency checks.
app.get('/health/liveness', (req, res) => {
  res.status(200).json({ status: 'alive' });
});

// Readiness: can this instance serve traffic? Check dependencies.
app.get('/health/readiness', async (req, res) => {
  try {
    await Promise.race([
      db.raw('SELECT 1'),
      new Promise((_, reject) =>
        setTimeout(() => reject(new Error('timeout')), 3000)
      ),
    ]);
    res.status(200).json({ status: 'ready' });
  } catch (err) {
    res.status(503).json({ status: 'not ready', reason: err.message });
  }
});

Rahul learned this the hard way at 11 PM on a Friday. You just learned it in two minutes. That is the point of this book.

CI/CD: The Pipeline That Ships While You Sleep

Infrastructure as Code defines what your system looks like. CI/CD defines how changes get there. At ₹60 lakh+, you are not SSH-ing into servers and running git pull. You push to a branch. A pipeline builds, tests, and deploys. You watch it in a dashboard. If it fails, you fix it and push again.

GitHub Actions for Multi-Environment Deployment

Here is a production-grade GitHub Actions workflow that builds a Docker image, pushes it to ECR, and deploys to Kubernetes — with environment-specific configuration:

name: Deploy API Service

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

env:
  AWS_REGION: ap-south-1
  ECR_REPOSITORY: api-service

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: 'npm'
      - run: npm ci
      - run: npm test
      - run: npm run lint

  build-and-push:
    needs: test
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    outputs:
      image_tag: ${{ steps.meta.outputs.version }}
    steps:
      - uses: actions/checkout@v4
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/github-actions-deploy
          aws-region: ${{ env.AWS_REGION }}
      - uses: aws-actions/amazon-ecr-login@v2
      - uses: docker/metadata-action@v5
        id: meta
        with:
          images: ${{ steps.login.outputs.registry }}/${{ env.ECR_REPOSITORY }}
          tags: |
            type=sha,prefix=
            type=ref,event=branch
      - uses: docker/build-push-action@v5
        with:
          context: .
          push: true
          tags: ${{ steps.meta.outputs.tags }}
          cache-from: type=gha
          cache-to: type=gha,mode=max

  deploy-staging:
    needs: build-and-push
    runs-on: ubuntu-latest
    environment: staging
    steps:
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/github-actions-deploy
          aws-region: ${{ env.AWS_REGION }}
      - run: |
          aws eks update-kubeconfig --region ${{ env.AWS_REGION }} --name myapp-staging
      - uses: azure/k8s-set-context@v3
        with:
          method: kubeconfig
          kubeconfig: ${{ env.KUBECONFIG }}
      - run: |
          kubectl set image deployment/api-service \
            api=${{ steps.login.outputs.registry }}/${{ env.ECR_REPOSITORY }}:${{ needs.build-and-push.outputs.image_tag }} \
            -n staging
          kubectl rollout status deployment/api-service -n staging --timeout=5m

  deploy-production:
    needs: deploy-staging
    runs-on: ubuntu-latest
    environment: production
    steps:
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/github-actions-deploy
          aws-region: ${{ env.AWS_REGION }}
      - run: |
          aws eks update-kubeconfig --region ${{ env.AWS_REGION }} --name myapp-production
      - uses: azure/k8s-set-context@v3
        with:
          method: kubeconfig
          kubeconfig: ${{ env.KUBECONFIG }}
      - run: |
          kubectl set image deployment/api-service \
            api=${{ steps.login.outputs.registry }}/${{ env.ECR_REPOSITORY }}:${{ needs.build-and-push.outputs.image_tag }} \
            -n production
          kubectl rollout status deployment/api-service -n production --timeout=5m

Notice the environment key on the deploy jobs. This is a GitHub Actions feature that enforces protection rules. You can require manual approval before production deploys. You can restrict which branches can deploy to production. You can require specific reviewers. This is not optional. Every company paying ₹60 lakh+ has deployment protection on production. If yours does not, implement it. It will save you from a 2 AM rollback.

The pipeline is linear: test, build, deploy to staging, deploy to production. If tests fail, nothing deploys. If staging deploy fails, production is untouched. This is the minimum viable pipeline for a professional engineering organization.

Observability: The Three Pillars

Infrastructure and CI/CD get your code running. Observability tells you what it is doing. The three pillars are logs, metrics, and traces. You need all three. Logs tell you what happened. Metrics tell you how much and how fast. Traces tell you the journey through the system.

Pillar 1: Structured Logging in Node.js

Console.log is not logging. It is debugging that you forgot to remove. Production logging is structured, leveled, and centralized.

const pino = require('pino');

const logger = pino({
  level: process.env.LOG_LEVEL || 'info',
  formatters: {
    level(label) {
      return { level: label };
    },
  },
  // In production, log JSON to stdout. Let the log aggregator handle formatting.
  // In development, pretty-print for humans.
  ...(process.env.NODE_ENV === 'development' && {
    transport: {
      target: 'pino-pretty',
      options: { colorize: true },
    },
  }),
  // Every log line gets these fields automatically
  base: {
    service: 'api-service',
    environment: process.env.NODE_ENV,
    version: process.env.APP_VERSION,
  },
  // Redact sensitive fields
  redact: ['req.headers.authorization', 'req.headers.cookie', 'password', 'token'],
});

// Usage: always log with context
app.post('/orders', async (req, res) => {
  const startTime = Date.now();

  logger.info({ userId: req.user.id, items: req.body.items.length }, 'order created');

  try {
    const order = await orderService.create(req.body);
    const duration = Date.now() - startTime;

    logger.info({
      orderId: order.id,
      userId: req.user.id,
      amount: order.total,
      durationMs: duration,
    }, 'order processed successfully');

    res.status(201).json(order);
  } catch (err) {
    logger.error({
      err,
      userId: req.user.id,
      requestBody: req.body,
      durationMs: Date.now() - startTime,
    }, 'order processing failed');

    res.status(500).json({ error: 'Order processing failed' });
  }
});

Pino is the right choice for Node.js. It is the fastest logger in the ecosystem — 5-10x faster than Winston in benchmarks. At high throughput, logging overhead matters. A slow logger adds latency to every request. Pino logs to stdout as newline-delimited JSON. Your log aggregator — Fluentd, Logstash, or the cloud provider's agent — picks it up and ships it to a central store.

The key pattern here is structured context on every log line. userId, orderId, durationMs. When something breaks, you do not grep for "error" and read stack traces. You query: "show me all log lines for userId=42 in the last 5 minutes." Structured logging makes this possible.

Pillar 2: Metrics with Prometheus and Grafana

Logs are for debugging specific requests. Metrics are for understanding system behavior over time. Prometheus is the standard. It scrapes metrics from your services, stores them as time-series data, and lets you query them with PromQL. Grafana visualizes them.

First, instrument your Node.js service with the prom-client library:

const client = require('prom-client');

// Create a Registry
const register = new client.Registry();

// Enable default metrics (CPU, memory, event loop lag, GC, open handles)
client.collectDefaultMetrics({
  register,
  prefix: 'myapp_',
});

// Custom metrics: HTTP request duration histogram
const httpRequestDuration = new client.Histogram({
  name: 'myapp_http_request_duration_seconds',
  help: 'Duration of HTTP requests in seconds',
  labelNames: ['method', 'route', 'status_code'],
  buckets: [0.01, 0.05, 0.1, 0.5, 1, 2, 5, 10],
});
register.registerMetric(httpRequestDuration);

// Custom metrics: active database connections
const dbConnections = new client.Gauge({
  name: 'myapp_db_connections_active',
  help: 'Number of active database connections',
});
register.registerMetric(dbConnections);

// Custom metrics: order count by status
const ordersTotal = new client.Counter({
  name: 'myapp_orders_total',
  help: 'Total number of orders',
  labelNames: ['status'],
});
register.registerMetric(ordersTotal);

// Middleware to track request duration
app.use((req, res, next) => {
  const end = httpRequestDuration.startTimer();
  res.on('finish', () => {
    end({
      method: req.method,
      route: req.route?.path || req.path,
      status_code: res.statusCode,
    });
  });
  next();
});

// Expose metrics endpoint for Prometheus scraping
app.get('/metrics', async (req, res) => {
  res.set('Content-Type', register.contentType);
  res.end(await register.metrics());
});

// In your business logic, increment counters
app.post('/orders', async (req, res) => {
  try {
    const order = await orderService.create(req.body);
    ordersTotal.inc({ status: 'success' });
    res.status(201).json(order);
  } catch (err) {
    ordersTotal.inc({ status: 'failed' });
    res.status(500).json({ error: 'Order processing failed' });
  }
});

The Histogram is the most important metric type. It tracks the distribution of values — in this case, request duration. It automatically creates three sub-metrics: _count (total requests), _sum (total duration), and _bucket (count per bucket). From these, you can calculate the 50th, 95th, and 99th percentile latencies. The p99 is what your slowest users experience. Optimize for p99, not average.

The buckets array is critical. The default buckets are [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10] — designed for sub-second responses. If your API typically responds in 200ms, these are fine. If you have a batch job that takes 30 seconds, you need custom buckets. Bad buckets produce useless histograms.

Now, the Prometheus configuration to scrape this service:

# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'api-service'
    kubernetes_sd_configs:
      - role: pod
        namespaces:
          names:
            - production
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app]
        action: keep
        regex: api-service
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
        target_label: __address__

Prometheus uses Kubernetes service discovery to find pods. It looks for pods with the annotation prometheus.io/scrape: "true". Your deployment needs this annotation:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-service
spec:
  template:
    metadata:
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "3000"
        prometheus.io/path: "/metrics"

Once Prometheus is scraping, you build Grafana dashboards. The four golden signals for any service:

1. Latency — histogram_quantile(0.99, rate(myapp_http_request_duration_seconds_bucket[5m])) — How long are requests taking?
2. Traffic — rate(myapp_http_request_duration_seconds_count[5m]) — How many requests per second?
3. Errors — rate(myapp_http_request_duration_seconds_count{status_code=~"5.."}[5m]) — What fraction of requests are failing?
4. Saturation — myapp_db_connections_active / 100 — How full is the system?

These four graphs on a single dashboard tell you everything you need to know about a service's health at a glance. Every service you own should have this dashboard. Build it once. Template it. Reuse it.

Pillar 3: Distributed Tracing with OpenTelemetry

Logs and metrics cover individual services. Traces cover the journey across services. When a request hits your API gateway, passes through three microservices, queries two databases, and calls an external payment provider — a trace shows you the entire path, with timing for each hop.

OpenTelemetry is the standard. It is a CNCF project, backed by every major observability vendor. It provides SDKs for instrumentation and a protocol for exporting traces.

const { NodeSDK } = require('@opentelemetry/sdk-node');
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-http');
const { Resource } = require('@opentelemetry/resources');
const { SemanticResourceAttributes } = require('@opentelemetry/semantic-conventions');

const sdk = new NodeSDK({
  resource: new Resource({
    [SemanticResourceAttributes.SERVICE_NAME]: 'api-service',
    [SemanticResourceAttributes.SERVICE_VERSION]: process.env.APP_VERSION,
    [SemanticResourceAttributes.DEPLOYMENT_ENVIRONMENT]: process.env.NODE_ENV,
  }),
  traceExporter: new OTLPTraceExporter({
    url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT || 'http://jaeger:4318/v1/traces',
  }),
  instrumentations: [getNodeAutoInstrumentations()],
});

sdk.start();

// Graceful shutdown
process.on('SIGTERM', async () => {
  await sdk.shutdown();
  process.exit(0);
});

The auto-instrumentation is remarkable. It instruments Express, HTTP, gRPC, Redis, PostgreSQL, MongoDB, and a dozen other libraries — automatically. Every incoming request gets a trace. Every outgoing HTTP call gets a span. Every database query gets a span. You get 80% of the value with zero manual instrumentation.

For the remaining 20%, you add custom spans:

const { trace } = require('@opentelemetry/api');

const tracer = trace.getTracer('order-service');

app.post('/orders', async (req, res) => {
  const span = tracer.startSpan('process_order');

  try {
    span.setAttribute('order.items_count', req.body.items.length);
    span.setAttribute('order.user_id', req.user.id);

    // Add an event for each significant step
    span.addEvent('validating_inventory');
    await inventoryService.validate(req.body.items);

    span.addEvent('calculating_total');
    const total = await pricingService.calculate(req.body.items);

    span.addEvent('charging_payment');
    await paymentService.charge(req.user.id, total);

    span.addEvent('creating_order_record');
    const order = await orderRepository.create({
      userId: req.user.id,
      items: req.body.items,
      total,
    });

    span.setAttribute('order.id', order.id);
    span.setStatus({ code: 1 }); // OK
    res.status(201).json(order);
  } catch (err) {
    span.setStatus({ code: 2, message: err.message }); // ERROR
    span.recordException(err);
    res.status(500).json({ error: 'Order processing failed' });
  } finally {
    span.end();
  }
});

The trace for this request shows: POST /orders (200ms) containing process_order (180ms) containing validating_inventory (30ms), calculating_total (15ms), charging_payment (120ms), and creating_order_record (10ms). You can see instantly that the payment provider is the bottleneck. Without traces, you would be guessing.

Jaeger is the most common trace backend for self-hosted setups. In the cloud, every provider has a managed offering: AWS X-Ray, Google Cloud Trace, Azure Monitor. The OpenTelemetry exporter is the same regardless of backend. Instrument once. Switch backends by changing the exporter URL.

The SRE Mindset: SLIs, SLOs, and Error Budgets

Observability tools are useless without a framework for using them. The SRE (Site Reliability Engineering) mindset — pioneered at Google, adopted everywhere — gives you that framework.

SLI (Service Level Indicator): A measurement of something your users care about. "The p99 latency of the /orders endpoint." "The availability of the checkout flow." "The error rate of the payment API." An SLI is a number. You measure it with your observability stack.

SLO (Service Level Objective): A target for an SLI. "The p99 latency of /orders must be under 500ms." "The checkout flow must be available 99.9% of the time." "The payment API error rate must be under 0.1%." An SLO is a contract — between you and your users, between your service and the services that depend on it.

Error Budget: The amount of failure your SLO allows. If your SLO is 99.9% availability, your error budget is 0.1% — about 43 minutes of downtime per month. As long as you have error budget remaining, you can push changes, take risks, move fast. When the error budget is exhausted, you stop all feature work and fix reliability.

This is the insight that changes everything: reliability is not a goal. It is a constraint. You do not want 100% uptime. 100% uptime means you are not shipping. You want exactly enough reliability to keep users happy, and you want to spend the rest of your error budget on moving faster than your competitors.

Here is how you define SLOs for a Node.js service:

// SLO monitoring: track error budget consumption
const { Counter, Registry } = require('prom-client');

const sloRegistry = new Registry();

const sloRequests = new Counter({
  name: 'slo_requests_total',
  help: 'Total requests counted toward SLO',
  labelNames: ['slo_name', 'result'],
  registers: [sloRegistry],
});

// Middleware: count every request as good or bad per SLO definition
function sloMiddleware(sloName, goodPredicate) {
  return (req, res, next) => {
    res.on('finish', () => {
      const isGood = goodPredicate(req, res);
      sloRequests.inc({
        slo_name: sloName,
        result: isGood ? 'good' : 'bad',
      });
    });
    next();
  };
}

// Define SLOs
app.use('/orders', sloMiddleware('orders-availability', (req, res) => {
  return res.statusCode < 500; // 5xx counts against error budget
}));

app.use('/orders', sloMiddleware('orders-latency', (req, res) => {
  return res.getHeader('X-Response-Time') < 500; // >500ms counts against error budget
}));

// Set response time header
app.use((req, res, next) => {
  const start = Date.now();
  res.on('finish', () => {
    res.setHeader('X-Response-Time', Date.now() - start);
  });
  next();
});

Now you can query your error budget in Prometheus:

# Error budget remaining for orders availability (SLO: 99.9%)
(
  1 - (
    sum(rate(slo_requests_total{slo_name="orders-availability", result="bad"}[28d]))
    /
    sum(rate(slo_requests_total{slo_name="orders-availability"}[28d]))
  )
) / 0.001

If this number drops below 1, you have exhausted your error budget. Stop deploying features. Fix reliability.

Incident Response: The 2 AM Playbook

When an alert fires, you need a process. Not a panic. A process.

1. Acknowledge. The first response is not "fix it." It is "I see it." Acknowledge the page within 5 minutes. If you cannot, escalate. A silent on-call engineer is the most dangerous person in the organization.

2. Triage. Is this a real incident or a false alarm? Check the dashboard. Check the error rate. Check if users are actually affected. If the p99 latency spiked from 200ms to 400ms but no users are complaining, this is a warning, not an incident. Suppress the alert and file a ticket.

3. Mitigate. Do not debug. Mitigate. The goal is to restore service, not to understand the root cause. Roll back the last deployment. Scale up the pods. Fail over to the replica database. Understanding comes later. Every minute you spend debugging during an incident is a minute your users are experiencing errors.

4. Communicate. Post in the incident channel. What is affected? What are you doing? When will you update next? A silent incident channel creates panic. An active one creates confidence.

5. Resolve. Service is restored. Users are happy. Now — and only now — you debug.

6. Postmortem. Within 24 hours, write a blameless postmortem. What happened? What was the impact? What was the root cause? How was it fixed? What will prevent it from happening again? The postmortem is not about blame. It is about learning. Every incident is a free lesson in how your system fails. Do not waste it.

Here is a postmortem template you can use:

# Incident Postmortem: [Title]

**Date:** 2026-07-27
**Duration:** 02:07 - 02:22 IST (15 minutes)
**Severity:** SEV2 — Payment processing degraded
**Author:** Priya Sharma

## Summary
Deploy #1842 introduced a misconfigured database connection pool (max 5 connections instead of 50). Under load, connections exhausted, causing payment API timeouts.

## Timeline (IST)
- 02:07 — PagerDuty alert: payment API p99 latency > 2000ms
- 02:08 — Acknowledged. Started investigating.
- 02:10 — Identified deploy #1842 as the change. Checked rollout status — pods were crash-looping.
- 02:12 — Rolled back to deploy #1841.
- 02:15 — Latency returned to normal.
- 02:18 — Verified payment processing recovered.
- 02:22 — Posted incident summary.

## Root Cause
The `DB_POOL_MAX` environment variable was changed from 50 to 5 in deploy #1842. The connection pool exhausted under normal load, causing all database queries to queue and time out.

## Impact
- 8 minutes of degraded payment processing
- ~120 failed payment attempts
- 0 successful payments lost (idempotency keys saved retries)

## Action Items
- [ ] Add connection pool utilization metric to Prometheus (P0)
- [ ] Add pre-deploy check: `DB_POOL_MAX` must be >= 20 (P0)
- [ ] Add integration test that verifies concurrent connection handling (P1)
- [ ] Review all environment variable changes in PR diff during code review (P1)

This postmortem is specific, blameless, and actionable. It does not say "Priya made a mistake." It says "the system allowed a misconfiguration to reach production." The action items prevent the class of error, not the specific instance.

The Infrastructure Stack of a ₹1 Crore Engineer

Let me show you what this all looks like together. Here is the infrastructure stack of a ₹60 lakh to ₹1 crore Node.js engineer at an Indian product company:

┌─────────────────────────────────────────────────────────┐
│                    GitHub Actions                        │
│  test → build → push → deploy-staging → deploy-prod     │
└──────────────────────┬──────────────────────────────────┘
                       │
┌──────────────────────▼──────────────────────────────────┐
│                 Amazon EKS (Kubernetes)                  │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐              │
│  │  API Pod │  │  API Pod │  │  API Pod │  HPA: 3-20   │
│  └──────────┘  └──────────┘  └──────────┘              │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐              │
│  │Worker Pod│  │Worker Pod│  │Worker Pod│  HPA: 2-10   │
│  └──────────┘  └──────────┘  └──────────┘              │
└──────────────────────┬──────────────────────────────────┘
                       │
┌──────────────────────▼──────────────────────────────────┐
│                   Observability                          │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐              │
│  │Prometheus│  │  Grafana  │  │  Jaeger   │              │
│  │ (metrics)│  │(dashboards│  │ (traces)  │              │
│  │          │  │  alerts)  │  │           │              │
│  └──────────┘  └──────────┘  └──────────┘              │
│  ┌──────────────────────────────────────┐               │
│  │     Elasticsearch + Kibana (logs)     │               │
│  └──────────────────────────────────────┘               │
└─────────────────────────────────────────────────────────┘

This is not aspirational. This is the standard stack at companies like Razorpay, Swiggy, Zerodha, and Freshworks. The engineers earning ₹60 lakh+ at these companies can explain every box in this diagram. They can provision it, debug it, and improve it.

Practice: Build Your Infrastructure Stack

You cannot learn infrastructure by reading. You learn it by building. Here is your assignment:

Week 1: Terraform. Provision a three-tier architecture on AWS using Terraform: a VPC with public and private subnets, an EKS cluster, and an RDS PostgreSQL instance. Use modules. Store state in S3. Use workspaces or directory separation for dev and prod environments. Destroy everything when you are done — AWS bills are real.

Week 2: Kubernetes. Deploy a Node.js application to your EKS cluster. Write the Deployment, Service, Ingress, ConfigMap, and HPA manifests. Configure readiness and liveness probes. Simulate a failure: kill a pod, watch it restart. Scale the deployment manually, then configure HPA and load-test it with k6 or autocannon.

Week 3: Observability. Install Prometheus and Grafana on your cluster using Helm. Instrument your Node.js application with prom-client. Build the four golden signals dashboard. Add OpenTelemetry tracing with Jaeger. Generate some traffic and explore the traces.

Week 4: CI/CD. Write a GitHub Actions workflow that builds your Docker image, pushes it to ECR, and deploys to your EKS cluster. Add environment protection rules. Simulate a bad deployment: push code that fails the readiness probe. Watch the pipeline catch it. Roll back.

This is four weeks of focused work. It will be frustrating. You will read documentation that assumes you know things you do not. You will encounter YAML indentation errors that take an hour to debug. This is normal. This is how everyone learns infrastructure. The difference is that you are doing it deliberately, with a plan, instead of picking it up in panicked fragments during production incidents.

The Bridge

Infrastructure is the foundation. Terraform, Kubernetes, CI/CD, observability — these are the tools that let you build systems that serve millions of users without waking you up at 2 AM. They are table stakes for ₹60 lakh+ roles. You must know them.

But the next frontier — the one that is creating ₹1 crore+ roles right now, the one that has every CTO in Bangalore rewriting their hiring plans — is not about running systems. It is about building systems that think. AI engineering. Large language models. Vector databases. Agents that reason, plan, and act. The engineers who can bridge traditional backend engineering with AI are the ones writing their own tickets.

That is where we go next.