Provisioning a Kubernetes cluster is easy to demonstrate and difficult to operationalize. A proof of concept can create a Google Kubernetes Engine (GKE) cluster, apply a Deployment, expose a Service, and appear complete. A production platform has a much broader responsibility. It must be repeatable, secure, observable, upgradeable, and understandable by engineers who did not create the first version.
I originally wrote about automating GKE with Terraform in 2020. The central idea still holds: infrastructure as code should make the platform reproducible. The implementation patterns, however, deserve a modern treatment. This article revisits that design and focuses on the boundaries and safeguards that make GKE automation sustainable.
Start by separating responsibilities
The first architectural decision is deciding what Terraform should own.
Terraform is well suited to resources with infrastructure lifecycles:
- projects, APIs, networks, subnets, and secondary IP ranges;
- GKE clusters and node pools;
- service accounts and Identity and Access Management (IAM) bindings;
- logging, monitoring, and policy configuration; and
- supporting cloud resources such as Artifact Registry repositories.
Application workloads usually change more frequently. Deployments, Services, autoscaling policies, and application configuration are often better managed through a delivery system such as GitOps or a continuous delivery pipeline.
Keeping these lifecycles separate reduces the blast radius of changes. A routine application release should not require a Terraform plan against the cluster itself. A node pool change should not depend on the state of every application manifest.
A practical repository structure
A clear repository layout makes ownership visible:
platform/
environments/
development/
staging/
production/
modules/
network/
gke-cluster/
node-pool/
workload-identity/
applications/
base/
overlays/
development/
staging/
production/
The exact directories are less important than the separation. Platform code creates the environment. Application configuration declares what runs inside it.
Model the cluster and node pools independently
For GKE Standard, I prefer managing the cluster and node pools as separate Terraform resources. This allows node pools to evolve without coupling their lifecycle to the control plane configuration.
resource "google_container_cluster" "platform" {
name = var.cluster_name
location = var.region
remove_default_node_pool = true
initial_node_count = 1
deletion_protection = true
release_channel {
channel = "REGULAR"
}
workload_identity_config {
workload_pool = "${var.project_id}.svc.id.goog"
}
}
resource "google_container_node_pool" "general" {
name = "general"
location = var.region
cluster = google_container_cluster.platform.name
autoscaling {
min_node_count = 1
max_node_count = 6
}
node_config {
machine_type = "e2-standard-4"
service_account = google_service_account.gke_nodes.email
oauth_scopes = ["https://www.googleapis.com/auth/cloud-platform"]
}
}
This is an illustrative starting point, not a complete production module. Network topology, maintenance policy, binary authorization, logging, monitoring, encryption, and organization policy should be selected for the environment rather than copied blindly.
Release channels provide a managed upgrade path. A dedicated node service account provides a clearer permission boundary than relying on broad defaults. Deletion protection helps prevent an accidental destroy operation from becoming an outage.
Use identity instead of service account keys
Workloads often need access to Google Cloud APIs. Long-lived service account keys stored as Kubernetes Secrets create rotation and exposure risks.
Workload Identity Federation for GKE allows a Kubernetes workload identity to receive narrowly scoped Google Cloud permissions without distributing key files. Enabling the feature does not grant access by itself. The platform team must still define explicit IAM policies for each workload.
This pattern improves both security and operability:
- credentials are short lived;
- applications do not carry static cloud keys;
- access can be scoped by workload identity; and
- audit trails map more clearly to the calling workload.
Treat networking as an early design decision
Cluster networking is difficult to change after adoption. Plan the following before creating production clusters:
- VPC and subnet ownership;
- secondary IP ranges for Pods and Services;
- private nodes and control plane access;
- authorized administrative paths;
- outbound connectivity and network address translation;
- load balancer exposure; and
- network policy enforcement.
IP capacity deserves particular attention. Underestimating Pod address space can constrain future cluster growth. Overly broad control plane or node exposure can create unnecessary attack paths.
Deliver workloads with explicit health behavior
A Deployment creates and replaces Pods, but availability depends on the behavior expressed in the workload specification.
apiVersion: apps/v1
kind: Deployment
metadata:
name: web
spec:
replicas: 3
selector:
matchLabels:
app: web
template:
metadata:
labels:
app: web
spec:
containers:
- name: web
image: REGION-docker.pkg.dev/PROJECT/apps/web:VERSION
ports:
- containerPort: 8080
startupProbe:
httpGet:
path: /health/startup
port: 8080
failureThreshold: 30
periodSeconds: 5
readinessProbe:
httpGet:
path: /health/ready
port: 8080
periodSeconds: 5
livenessProbe:
httpGet:
path: /health/live
port: 8080
periodSeconds: 10
failureThreshold: 3
resources:
requests:
cpu: 250m
memory: 256Mi
limits:
memory: 512Mi
Each probe answers a different question. Startup determines whether initialization completed. Readiness controls whether the Pod should receive traffic. Liveness determines whether the container is unable to recover without a restart.
Aggressive liveness probes can make an incident worse by restarting containers during temporary load or dependency failures. Probe endpoints should be inexpensive, intentional, and tested under realistic failure conditions.
Add autoscaling only after defining resource intent
Autoscaling is not a substitute for capacity planning. The Horizontal Pod Autoscaler needs meaningful resource requests or application metrics. Cluster autoscaling needs node pools with appropriate minimums, maximums, machine types, and scheduling constraints.
The operating sequence is connected:
- Resource requests communicate expected workload demand.
- Pod autoscaling changes replica count based on measured demand.
- The scheduler places the new Pods when capacity exists.
- Cluster autoscaling adds nodes when Pods cannot be scheduled.
If requests are inaccurate, each later decision is built on weak input. Start with measured workload behavior, then tune autoscaling against load tests and production telemetry.
Build a safe Terraform delivery workflow
Running Terraform manually from a laptop is useful for learning, but it is not a durable production control plane. A team workflow should include:
- remote state with restricted access and versioning;
- provider and module version constraints;
- formatting, validation, and policy checks;
- a saved plan reviewed before apply;
- environment-specific approval controls;
- short-lived CI credentials; and
- serialized applies for each state file.
A typical pipeline is intentionally simple:
pull request
-> terraform fmt and validate
-> security and policy checks
-> terraform plan
-> human review
-> approved apply
-> post-deployment verification
The plan is not just an output file. It is the review boundary that shows whether a small configuration change will replace a node pool, modify a firewall rule, or affect a protected cluster.
Operate the platform after provisioning
Successful creation is the beginning of the platform lifecycle. Production readiness also requires:
- control plane and node upgrade policies;
- logging, metrics, and alerting for both workloads and cluster components;
- backup and recovery expectations for stateful workloads;
- vulnerability and configuration scanning;
- cost allocation labels and capacity reviews;
- tested incident procedures; and
- documented ownership for platform and application layers.
I also monitor the automation itself. Failed plans, stale modules, provider drift, state lock contention, and repeated manual changes are platform signals. They reveal where the declared operating model and actual engineering behavior have diverged.
What I would change from the original implementation
My original approach proved the value of automating cluster creation and workload deployment. A production version should go further:
- use a regional design when the availability requirement justifies it;
- manage node pools separately from the cluster;
- use Workload Identity Federation instead of service account keys;
- keep Terraform state remote and tightly controlled;
- separate infrastructure provisioning from application delivery;
- define startup, readiness, and liveness probes independently;
- use immutable image versions from a controlled registry; and
- design upgrades, observability, security, and recovery before launch.
These changes shift the goal from creating a cluster to operating a dependable platform.
Conclusion
Terraform can make GKE infrastructure consistent and reviewable, but production readiness comes from the operating model around the code. Clear ownership boundaries, secure workload identity, deliberate networking, meaningful health checks, and controlled delivery are what turn an automated cluster into a platform teams can trust.
The best infrastructure automation is not the configuration with the most options. It is the smallest clear system that teams can change safely, understand during an incident, and improve without rebuilding everything around it.