“We have logs” is the most common answer when I ask a team how they debug production. It is also, almost always, the reason their incidents take longer than they should.
Logs are necessary. They are not sufficient. The three signals - logs, metrics, traces - solve different problems, and using one in place of another is a recipe for slow debugging and bloated bills.
Metrics answer “how is the system behaving in aggregate, right now and over time?” They are cheap to store, cheap to query, and easy to alert on. They are the signal you watch when you do not know whether anything is wrong.
Traces answer “for this one request, what happened?” They are the signal you reach for once you know something is wrong and you need to know where. A good trace turns “the checkout endpoint is slow” into “the checkout endpoint spends 400ms waiting on the fraud-check service, which spends 380ms in a single Postgres query.”
Logs answer “what specifically happened at this point in the code?” They carry context that does not fit in a metric label or a span attribute - the actual values, the error chain, the decision the code made. They are the signal you read once you know which line of code you care about.
The mistake is using logs for all three. Counting how often something happens by grepping logs is a metric pretending to be text. Reconstructing a request path by correlating timestamps across services is a trace pretending to be text. Both work. Both are slow, expensive, and lossy.
If you are starting from “we have logs,” the highest-leverage additions are, in order:
The RED metrics for every service. Rate, errors, duration. One histogram and one counter per endpoint. This alone will tell you within seconds whether the problem is “everything is slow,” “one endpoint is slow,” or “one endpoint is erroring.” You do not need a dashboard yet. You need the data.
Request-scoped trace IDs propagated end to end. Generate a trace ID at the edge, stamp it onto every log line in the request’s path, propagate it across service boundaries via headers. Even without a tracing backend, this turns “find all logs related to this request” from a 20-minute exercise into a single grep.
A sampled tracing backend. Once trace IDs exist, point them at OpenTelemetry plus any compatible backend. Sample aggressively - 1% of normal traffic is usually enough, and you can sample 100% of errors. The point is not to keep every span. The point is to keep enough of them to debug typical and pathological cases.
Structured logs with consistent fields. level, ts, service, trace_id, span_id, error, plus whatever is specific to the operation. JSON or logfmt, pick one and never mix them. The single biggest unforced error in logging is letting each service invent its own format.
After those four, the marginal value of each new piece of instrumentation drops sharply. A team that has these four and uses them well will out-debug a team that has a wall of fancy dashboards and no trace IDs.
Every metrics system has the same failure mode: you add a label that takes on too many values and your storage costs explode. The classic offenders are user_id, request_id, path (when the path includes IDs), and error_message (when the message includes dynamic content).
A few rules that have held up:
The same restraint applies to logs. A service that logs at INFO for every request, with the full request body, will spend more on log storage than on compute. A few principles:
Metrics drive alerts. Not logs. Not traces. An alert on “an error log appeared” is the path to alert fatigue, because errors of varying severity all look the same in the log stream. An alert on “the error rate for service X exceeded 1% for 5 minutes” is actionable.
The rule of thumb: every alert is tied to an SLO or a known-bad condition, and every alert has a runbook. Anything else is noise that will eventually be ignored, including by you at 2am.
Logs, metrics, traces - three signals, three jobs. Use each for what it is good at, instrument the small set of things that pay back the most, and treat cardinality and log volume as the budgets they are. Most teams do not need more telemetry. They need the telemetry they already have to be the right shape.