Production Incidents: The First 15 Minutes Matter Most

The first 15 minutes of an incident determine the next 3 hours. The phrase sounds like a cliché, but after many incidents I can tell you cleanly: 15 minutes of good decisions and the incident is actually solved in 30; 15 minutes of bad decisions and that same incident drags on for 6 hours.

This post is a production incident playbook — built up over years in an iGaming platform, from actual on-call runs.

Minutes 1–3: "What's happening?"

The first signal is usually an alarm: Grafana crossed a threshold, a new Sentry pattern, or someone in Slack saying "site is slow."

At this stage do one thing only: figure out the scale and severity. No action yet.

• How many users affected? (rate) → Critical
• Which services affected? (scope) → Critical
• Is it getting worse minute by minute, or stable? (trend) → Critical
• Is data being lost? (severity) → Critical

Answer those four inside three minutes. Otherwise the next 12 minutes solve the wrong problem.

Minutes 3–5: Open the channel

Open an incident Slack channel (e.g. #inc-YYYYMMDD-HHmm) right away. This is where the incident crew talks. All pings route here. Everyone watches this channel.

Quick update to stakeholders:

"Error rate up on the game server, ~15% of users affected. Investigating. Status update in 10 minutes."

Don't go into detail. Don't speculate. Don't try to diagnose in the stakeholder message. Share the situation and commit to an update.

Minutes 5–10: Diagnosis

Use fast observation tools:

Grafana, past 24 hours:

• Error rate — when did it start?
• Latency — did it slow down?
• Traffic — did traffic change?
• Resources — CPU/RAM/disk?

Recent deploys:

"Anything deployed in the past hour?"

This single question solves roughly 40% of incidents as "oh, the last deploy was the cause." Simple check: when was the last deploy, when did the alarm start — if close, discuss rollback first, diagnose later.

Recent logs:

• What error pattern?
• Concentrating on a specific endpoint?
• A downstream call failing?

Minutes 10–15: Make the call

At this point you branch:

Path A: Fast rollback possible?

• Is the last deploy a one-command rollback? → Do it now.
• Can a feature flag be flipped off? → Flip it now.
• Is a config change reversible? → Revert it now.

Rollback's beauty: it removes the likely cause immediately, the incident stops, and you analyze calmly afterward.

Important: don't flinch from rolling back. Don't say "but the new feature goes away." It goes away in four hours too — meanwhile 10,000 users have a bad experience.

Path B: Rollback not possible

Stabilize the situation:

• Reduce traffic to the hot service (load balancer)
• Turn off non-critical paths (feature flag)
• Scale out (sometimes more pods help)
• Tolerate downstream errors gracefully (circuit breaker)

Goal: stop the incident from growing. Solving it comes later.

After the first 15: root cause

Once the system is stable, the real work begins:

• Build a timeline — what happened, when, what did each person see
• Dive into logs — which specific code path produced this behavior
• Write a fix — proper, not a hot patch
• Deploy + verify

But that path is only short if the first 15 minutes were managed well.

Three things not to do during an incident

1. "I'm bored, let me try something" — panic moves. Random pod restarts, random config tweaks. Every random action makes diagnosis harder.

2. Sharing speculation as fact — "I think the DB connection pool is exhausted" → goes to Slack → leadership panics → the actual cause turns out to be Redis timeout → now you're managing two problems at once.

3. Working alone — every incident is a two-person job. One coordinates, the other does the technical work. If you're alone, find the second person immediately.

Post-incident: blameless retro

Within 48 hours of closing, a blameless retrospective:

• What happened?
• How did we detect it?
• Who was affected, and how much?
• How did we resolve it?
• How can we prevent it?
• What monitoring/alarm was missing?

The last question is the most important. Every incident reveals blind spots; closing those blind spots prevents the next one.

Takeaway

Managing production incidents is a skill. It isn't innate; it forms. The first five to ten incidents you'll panic; by the twentieth, it flows almost automatically. That's why on-call rotation has value — everyone should get reps in.

The 15-minute rule: don't rush, but don't slow down either. Observe, communicate, decide, act. The rest resolves.