✓ MTTR cut from ~3 hours to ~20 minutes
✓ Alert noise down 90% (400 → 40 actionable alerts/day)
✓ Uptime lifted from 99.6% to 99.95%
Before → after
| Metric | Before | After |
|---|---|---|
| Mean time to recover (MTTR) | ~3h | ~20 min |
| Alerts per day | 400 | 40 actionable |
| Uptime | 99.6% | 99.95% |
A SaaS team was flying blind. When something broke, they learned about it from customers, then spent hours digging through logs to find out why.
The challenge
- No dashboards, so nobody knew the system’s real state.
- 400 alerts a day, almost all of them noise.
- Incidents took ~3 hours to recover because there were no runbooks.
- SLOs didn’t exist, so “good enough” was never defined.
What we built
- SLOs defined the reliability targets that actually matter to customers.
- Dashboards built real-time views of latency, errors, and saturation for every service.
- Alerting cut the noise, alerts now fire only when an SLO is about to break.
- Runbooks wrote step-by-step recovery guides for the common failure modes.
- On-call wired a clean escalation path so the right person is paged the first time.
The results
Mean time to recover dropped from ~3 hours to ~20 minutes, alert noise fell 90% (from 400 to 40 actionable alerts a day), and uptime lifted from 99.6% to 99.95%.
The takeaway
Observability isn’t more charts, it’s knowing what “healthy” means, alerting only when it’s at risk, and having a runbook when it breaks.
Stack Cloud MonitoringCloud LoggingSLOsAlertingRunbooks
“
We went from 400 alerts a day to 40 that actually matter, and recovery takes minutes instead of hours.