Modern applications rarely fail in simple, isolated ways. A slow request may cross several services, depend on a shared platform, and degrade long before a conventional health check turns red. Logs can explain an individual event, metrics can show a trend, and traces can reveal a request path, but each signal is incomplete on its own.
I recently worked on an observability platform for large-scale applications running across Google Cloud Run and Google Kubernetes Engine (GKE). The goal was not simply to install dashboards. It was to create a dependable telemetry system that helped engineering teams understand behavior, detect risk earlier, and reduce the time required to move from a symptom to a cause.
This article describes the architecture, the decisions behind it, and the lessons that matter when observability moves from a proof of concept into production.
The challenge: visibility across different compute models
Cloud Run and GKE provide very different operational models. Cloud Run abstracts most of the underlying infrastructure, while GKE gives teams deeper control over scheduling, networking, and runtime behavior. The observability platform needed to provide a consistent experience across both without erasing those differences.
The core requirements were to:
- collect metrics, traces, and logs from high-volume applications;
- process and route telemetry without coupling applications to a single backend;
- store and query each signal efficiently;
- present service-level indicators and key performance indicators clearly;
- detect anomalies and reliability risks early; and
- help engineers correlate signals during troubleshooting.
The scale of the applications made efficiency part of the design. Telemetry is production data: it consumes CPU, memory, network bandwidth, and storage. A solution that collects everything without controlling cardinality, retention, or sampling will eventually become expensive or unreliable precisely when it is needed most.
Design principles
Before choosing tools, I used a few principles to guide the architecture.
Standardize collection, not every backend
Application teams should emit telemetry through a consistent, vendor-neutral interface. The platform can then route signals to backends suited to their data model and operational requirements.
Keep application code independent from routing decisions
Applications should describe what happened. Collectors and platform configuration should decide where telemetry goes, how it is enriched, and what is sampled or filtered.
Design around questions and service objectives
A dashboard is useful only when it helps answer a real operational question. The most valuable views start with service-level objectives (SLOs), user journeys, latency, traffic, errors, and saturation, not with every metric a library happens to expose.
Treat the telemetry pipeline as a production service
Collectors, storage systems, dashboards, and alerting paths need capacity planning, monitoring, ownership, and failure testing of their own.
Architecture overview
The platform combined OpenTelemetry, Prometheus, Grafana, and Google Cloud's native operations services.
Applications on Cloud Run and GKE
│
▼
OpenTelemetry SDKs and collectors
│
enrich, batch, sample
│
┌──────┼──────────┐
▼ ▼ ▼
Metrics Traces Logs
│ │ │
Prometheus Google Cloud operations backends
│
▼
Grafana dashboards and alerting
This separation made the platform easier to evolve. Instrumentation remained stable even when routing, retention, dashboards, or backend choices changed.
OpenTelemetry as the instrumentation layer
OpenTelemetry provided a common model for metrics, distributed traces, logs, and resource metadata. Applications used OpenTelemetry SDKs and automatic instrumentation where appropriate, while collectors handled batching, enrichment, filtering, and export.
Consistent resource attributes were especially important. Fields such as service name, environment, version, region, cluster, and workload identity made it possible to move between signals without guessing whether two records referred to the same deployment.
The collector layer also created a control point outside application code. It allowed the platform to adjust sampling, remove noisy attributes, protect backends from spikes, and route telemetry differently by environment.
Prometheus for operational metrics
Prometheus was deployed on GKE for high-value operational metrics and time-series queries. Its pull-based model, Kubernetes service discovery, and PromQL query language made it a strong fit for cluster and application metrics.
The biggest design concern was metric cardinality. Labels such as unbounded user IDs, request IDs, or raw URLs can create explosive growth in time series. We established label conventions and treated cardinality as an engineering budget rather than discovering the problem through an overloaded metrics backend.
Recording rules were useful for frequently evaluated queries and service-level indicators. They reduced dashboard latency and kept important reliability calculations consistent across teams.
Grafana as the operational interface
Grafana provided a shared interface for dashboards and alerting. The focus was on a small number of purposeful views:
- service health and SLO attainment;
- request rate, error rate, and latency distributions;
- resource saturation and capacity signals;
- deployment and version comparisons; and
- drill-down paths from a high-level symptom to supporting telemetry.
Dashboards were organized around services and user journeys rather than infrastructure alone. This helped connect platform symptoms to application behavior and business impact.
Google Cloud integration
Google Cloud's operations services complemented the open-source components. Cloud-native logging, monitoring, and tracing integrations provided visibility into managed services and created a natural path for correlating application telemetry with platform events.
The important architectural decision was to route each signal intentionally. Prometheus remained focused on metrics. Logs and traces were sent to backends designed for their respective query patterns and retention needs. This avoided forcing three different data models into one storage system.
End-to-end telemetry flow
The resulting flow was straightforward from an application team's perspective:
- Applications deployed to Cloud Run or GKE emitted telemetry through OpenTelemetry.
- Collectors received telemetry, added consistent resource context, batched data, and applied filtering or sampling policies.
- Metrics were exposed or exported for Prometheus collection and querying.
- Traces and logs were routed to their designated Google Cloud operations backends.
- Grafana presented service-focused dashboards and evaluated actionable alerts.
- Engineers used shared identifiers and resource attributes to move between metrics, traces, logs, deployments, and infrastructure events.
Alerting without creating more noise
Alerting was designed around symptoms that required action. High CPU usage may be interesting, but sustained user-visible latency or SLO burn is usually a better reason to page someone.
Useful alerts included clear ownership, severity, supporting context, and a next action. Where possible, alerts linked directly to the relevant dashboard or troubleshooting guidance. This made the alert part of the diagnostic workflow instead of merely another notification.
Outcomes
The platform improved several aspects of day-to-day engineering:
- Faster troubleshooting: Engineers could move from a service-level symptom to supporting metrics, traces, and logs with consistent context.
- Earlier detection: SLO-oriented alerts surfaced reliability risks before they became widespread incidents.
- Better performance decisions: Latency distributions and resource signals made bottlenecks and inefficient capacity easier to identify.
- Shared operational language: Standard instrumentation and dashboards gave application and platform teams a common view of system behavior.
- More deliberate cost control: Cardinality, sampling, and retention became explicit platform decisions.
Lessons learned
Instrumentation quality matters more than telemetry volume
More data does not automatically produce better answers. Consistent naming, resource attributes, and semantic conventions are more valuable than collecting every possible signal.
Sampling must be intentional
Uniform trace sampling can discard the rare requests that matter most. Sampling policies should preserve errors, high-latency requests, and other diagnostically valuable traces while controlling normal traffic volume.
Observability needs product thinking
The platform serves internal users. Documentation, onboarding, sensible defaults, dashboard templates, and feedback loops matter as much as the underlying tools.
Correlation should be designed from the beginning
Trace IDs in logs, exemplars connecting metrics to traces, deployment metadata, and consistent service identity dramatically reduce investigative friction. Adding them later is much harder.
The observability platform must observe itself
Collector queue depth, dropped telemetry, export failures, scrape health, ingestion latency, and backend capacity are essential signals. A silent gap in telemetry can be more dangerous than an explicit application error.
Conclusion
The most important result of this project was not a particular dashboard or tool. It was a repeatable way to turn telemetry into operational understanding across different Google Cloud compute platforms.
OpenTelemetry created a consistent instrumentation layer. Prometheus and Grafana provided powerful metrics analysis and visualization. Google Cloud's operations services added native platform context and appropriate backends for additional signals. Together, they formed an architecture that could evolve without requiring every application team to reinvent observability.
For engineering leaders building a similar platform, I would start with service objectives and diagnostic workflows, establish telemetry standards early, and treat cost and reliability as first-class requirements. The tools matter, but the operating model determines whether observability becomes a collection of dashboards or a durable engineering capability.