Observability Basics: Logs, Metrics, Traces — What's Each For?

"Observability" is the word everyone uses; filling it in is harder. Most teams install a tool (Prometheus, Datadog, ELK), build three dashboards, and call it "observability." Then the first real incident exposes that they don't have observability — they have tooling.

Observability has three fundamental pillars: log, metric, trace. Each answers a different question. When you use one in place of another, the cost shows up — you're pointing the wrong tool at the wrong question.

Log: answers "what happened?"

Logs are written records of events in your system. They're what you reach for to trace a single request end-to-end, to see the details of an error, to understand the steps of a business process.

Where logs are strong:

• Full stack trace of an error
• Step-by-step flow of a business action
• What a user did (audit trail)
• Clues to reproduce a specific issue

Where logs are weak:

• Aggregating ("how many errors?" across millions of log lines is expensive)
• Visual summaries (100 log lines vs. 1 chart — the chart wins by 100×)
• Real-time alerting (log-pattern alerts aren't real-time)

Rule: if you're trying to measure something with logs, you probably needed a metric.

Metric: answers "how much?"

Metrics are numeric measurements. Prometheus, Datadog, CloudWatch are built for this.

Where metrics are strong:

• Aggregation (request count, avg latency, 99th percentile)
• Charts (time series, trend analysis)
• Alerting (threshold breach)
• Scale (millions of data points, sub-second queries)

Where metrics are weak:

• Understanding why something happened
• Diagnosing one specific user's problem
• Detailed stack traces

Rule: metrics tell you something is happening — they don't tell you why. For that, you need trace or log.

Trace: answers "where was the time spent?"

Traces follow a request's journey through your system. User clicks a button → load balancer → API gateway → service A → service B → database → cache → response. That full journey is one trace.

Where traces are strong:

• Finding where latency is ("3s response time — 80% of it was in the DB call")
• Spotting bottlenecks across a microservice chain
• In distributed systems, "which service is the culprit?"
• Seeing retry patterns and downstream error cascades

Where traces are weak:

• Aggregate analysis (each trace is separate; aggregating needs a different tool)
• Storage cost (traces are fat, usually sampled)

Using all three together: an example

A customer reports: "The app is slow."

Check metrics: p99 latency over the past 30 minutes went from 800ms to 2.3s. Now you know how bad.

Go to trace: pull a slow request's trace, see where time is spent. "Game server API → auth service" took 1.8s. Now you know where.

Go to logs: look at auth service logs for that window. "Database connection pool exhausted" errors everywhere. Now you know what.

All three needed. Skip one and half the story is missing.

Common mistakes

Mistake 1: logging everything

"Just to be safe, we log it all" → storage cost explodes, finding meaning in logs gets harder, and the team stops looking at them.

Fix: log levels (DEBUG/INFO/WARN/ERROR). Store INFO+ in production, enable DEBUG only during incidents.

Mistake 2: logging and metric-ing the same thing

Writing logger.info("user logged in") and metric.increment("login_count") both is double the cost.

Fix: treat metric as the source of truth for measurement. Use logs only when metric isn't enough detail.

Mistake 3: treating tracing as optional

"Tracing is expensive, performance cost" → kept off. When you need it during an incident, too late to enable.

Fix: sample tracing from the start (e.g. 1% baseline, 100% on errors). This adds no meaningful performance cost.

Alerting

You can alert from any of the three, but metric is clearly the right tool:

• Log-based alerts — pattern matching, slow, noisy
• Trace-based alerts — rarely set up, complex
• Metric-based alerts — fast, crisp, measurable

Hook your alerts to metrics. "Error rate above 5% in last 5 minutes" is a metric alert. "Log X appeared 10 times" alerts have a high false-positive rate.

Practical recommendation

Minimum observability set for a new service:

• Metric: request count, latency (p50/p95/p99), error rate
• Log: structured log per request, stack trace for each error, INFO log per major business step
• Trace: every request traced (1% sampling baseline, 100% on error)

Without this set, shipping to production is flying blind. You'll see the events but won't understand them.

Takeaway

Observability isn't "install a tool" — it's wiring three different tools to three different questions. Log, metric, and trace don't replace each other. They each carry different information; together they show the whole picture.

Incidents are where good observability earns itself. A team with bad observability solves an incident in 4 hours; one with good observability in 40 minutes. That's the delta.