Engineering On-Call: Surviving It and Improving It

The on-call alert hits: 03:14 AM. Phone wakes you up, the system is alarming, users are affected. You roll out of bed and open the laptop.

That's the scene I've lived through hundreds of variations of, across six years on the on-call rotation of an iGaming platform. The difference between a healthy on-call culture and a damaging one decides how long the team lasts.

Here's what I've learned.

Why on-call exists

First: on-call isn't optional, it's a duty. Production systems require someone who can respond 24/7. That's the price of keeping things up.

But it doesn't mean "one person rotates being awake 24/7." A good on-call:

• Mostly stays quiet (no alerts firing)
• When called, you face it with preparation, not uncertainty
• The practice continually improves itself (every incident reduces future incidents)

Rotation hygiene

Properties of a good rotation:

Enough people. Minimum four. With three, when one is on vacation the other two are nearly always on. Burnout follows. With four, each does about a week per month. Less than that is dangerous.

Explicit handoffs. When the week ends, the outgoing person hands off in writing to the incoming one. What happened, which alarms are flaky, which services need attention. A 5-minute doc prevents a 30-minute incident.

Compensated. Whoever is on-call gets paid for it (extra pay, comp time, days off). It acknowledges the cost. Free on-call gradually becomes "why is it always me?" grumbling.

Severity-aware. Not every alarm is a siren. Tier 1 (revenue-impacting, immediate response): phone. Tier 2 (user-impacting, hours): email. Tier 3 (future risk): morning review. Spamming alarms with the same urgency makes the team numb — alert fatigue.

A starter kit for new on-call

There should be a runbook for new on-call rotators. The list:

• Top-5 most common alarms and their first response steps
• A simple architecture diagram (just the flow)
• Service owners (who do you call at night?)
• Rollback procedures (which service rolls back how?)
• "Help! What's this?" links (Grafana, Sentry, log system)

Each on-call person updates this doc once during their week. Six months in, it gets richer on its own.

The first 5 minutes when paged

The phone rang. To do:

1. Coffee or water — 30 seconds. Wake the brain, don't act in panic.
2. Read the alarm — what, how bad, when did it start? 30 seconds.
3. Open Slack incident channel — "#inc-...", say "I have the alarm, looking now." 30 seconds.
4. Check dashboards — error rate, latency, traffic. What's the trend? 2 minutes.
5. Decide: can I solve it in 15 minutes, or do I escalate? 30 seconds.

These 5 minutes shape the next hour. Don't rush — the wrong action is worse than the right one taken slowly.

When to escalate

Escalating instead of panicking and acting wrong is professionalism. When:

• You don't know the system (new joiner, never touched this service)
• Impact is large and growing (revenue, multi-service)
• 30 minutes of debugging, no progress
• Decision impact is large (DB rollback, region failover)

Escalation isn't shame. The wrong action is shame.

Post-incident — protected time

When the incident closes, post-mortem. Not optional, mandatory. Lessons from the field, when not documented, vanish.

Rules for the post-mortem:

• Blameless: not "who screwed up" but "how did the system allow this"
• Within 48 hours (memory still fresh)
• Action items have owners + dates
• Tracked monthly: action items are reviewed; open ones get pushed

Post-mortem docs are committed to the repo. Years later when "did this happen before?" gets searched, they'll show up.

How a rotation improves

End of every week, the outgoing rotator writes:

• How many alerts came in
• Which were false positives (alert fired but no real issue)
• Which were new alerts (never seen before)
• Which alarms have wrong thresholds (constantly triggering, no real problem)

This data goes to the team. Monthly:

• False-positive thresholds get raised
• New alarms enter the runbook
• For repeated incidents, automation gets written

Without this discipline, on-call is just "putting out fires." With it, the system gets sturdier over time.

Burnout signals

On-call burnout signals:

• Physical reaction to the phone ringtone (heart rate jump)
• "Just leave me alone" tone in personal communication
• Weekend paranoia, "something will fail again"
• Monday morning, no motivation

If these show up, talk. With manager, with team. Maybe rotation is unbalanced, maybe a specific alarm always hits the same person, maybe the system itself is too fragile.

You can't fix burnout alone. Caught early, rotation can be balanced, system improved, person recovers. Caught late, the person leaves.

Closing

On-call done right is where engineers grow fastest. Watching production behavior in real time teaches things no book can.

Done wrong, it burns people out and pushes them out of the industry. The difference isn't tech — it's rotation hygiene + post-incident discipline + human awareness.

Build those three and five years on the rotation is sustainable. Skip them and one year is hard.