Observability lets you understand system behavior from external outputs. Metrics, logs, and traces form the three pillars of observability.
The Three Pillars#
Metrics:
- Numerical measurements over time
- CPU, memory, request counts, latencies
- Aggregated and sampled
Logs:
- Discrete events with context
- Errors, requests, business events
- Detailed but voluminous
Traces:
- Request flow across services
- Timing and dependencies
- End-to-end visibility
Metrics with Prometheus#
Loading code block...
Business Metrics#
Loading code block...
Structured Logging#
Loading code block...
Distributed Tracing#
Loading code block...
Error Tracking#
Loading code block...
Health Checks#
Loading code block...
Alerting Rules#
Loading code block...
Dashboard Queries#
# Request rate
sum(rate(http_requests_total[5m])) by (path)
# Error rate
sum(rate(http_requests_total{status=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m]))
# P50, P95, P99 latency
histogram_quantile(0.50, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))
histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))
histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))
# Active connections
active_connections
# Memory usage
process_resident_memory_bytes
Best Practices#
Metrics:
✓ Use consistent naming conventions
✓ Add relevant labels
✓ Set appropriate buckets
✓ Monitor cardinality
Logging:
✓ Use structured logging
✓ Include correlation IDs
✓ Log at appropriate levels
✓ Don't log sensitive data
Tracing:
✓ Propagate context across services
✓ Add meaningful span names
✓ Include relevant attributes
✓ Sample appropriately
Alerting:
✓ Alert on symptoms, not causes
✓ Include runbooks
✓ Avoid alert fatigue
✓ Test alerts regularly
Conclusion#
Observability requires metrics, logs, and traces working together. Start with basic health checks and metrics, add structured logging, then implement tracing for complex systems. Good observability reduces mean time to resolution and improves system reliability.