Prometheus & Grafana

Monitoring systems with Prometheus metrics and Grafana dashboards.

Quick Start

docker run -p 9090:9090 -v prometheus.yml:/etc/prometheus/prometheus.yml -d prom/prometheus
docker run -p 3000:3000 -d grafana/grafana
# Grafana: localhost:3000 admin/admin

When to Use

• Collecting time-series metrics from services
• Dashboards for latency, errors, and saturation
• Alerting on thresholds and SLO burn
• Kubernetes and infra monitoring

Best Practices

Instrumentation

• Expose `/metrics` in Prometheus format
• Use counters (increasing), gauges, histograms
• Label metrics moderately (avoid cardinality explosion)
• Use client libraries for the language

Scraping & Storage

• Set sane scrape intervals (15-60s)
• Use `service_discovery` or static targets
• Limit retention and use TSDB best practices
• Configure recording rules for hot queries

Alerting

• Define alerts with PromQL thresholds
• Use Alertmanager for routing and dedup
• Alert on symptoms (SLOs), not just metrics
• Set for, `pending`, and proper severity

Dashboards

• Build focused dashboards per service
• Use RED (Rate, Errors, Duration) and USE patterns
• Add panels with meaningful units
• Track SLOs with burn-rate panels

Dependencies

# Python client
pip install prometheus-client
# or Node:
npm i prom-client

Examples

# prometheus.yml
global:
  scrape_interval: 15s
scrape_configs:
  - job_name: app
    static_configs:
      - targets: ["app:8080"]
rule_files:
  - alerts.yml

# alerts.yml
groups:
  - name: app
    rules:
      - alert: HighErrorRate
        expr: rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m]) > 0.05
        for: 10m
        labels: { severity: critical }
        annotations: { summary: "Error rate above 5%" }

from prometheus_client import Counter, Histogram, start_http_server
import random, time

REQUESTS = Counter("http_requests_total", "Total HTTP requests", ["status"])
LATENCY = Histogram("http_request_duration_seconds", "Request latency")

start_http_server(8080)
while True:
    REQUESTS.labels(status="200").inc()
    with LATENCY.time():
        time.sleep(random.uniform(0.01, 0.2))

# PromQL: error ratio per service
sum(rate(http_requests_total{status=~"5.."}[5m]))
  / sum(rate(http_requests_total[5m]))

Step-by-Step

1. Instrument services to expose `/metrics`.

2. Configure Prometheus scraping and storage.

3. Add alerting rules and Alertmanager.

4. Connect Grafana as the data source.

5. Build dashboards with RED/USE patterns.

6. Track SLOs with burn-rate alerts.

7. Secure with auth and TLS.

8. Monitor the monitor: alert on Prometheus itself.

Validation

1. `/metrics` exposed and scraped successfully

2. Dashboards show correct metrics and units

3. Alerts fire and route correctly

4. Cardinality stays within limits

5. Retention matches your needs

Troubleshooting

• Missing metrics: check the scrape target and path.
• High cardinality: reduce dynamic label values.
• No data in Grafana: verify the data source and time range.