Detection time is the first metric
You can't fix what you haven't noticed. Most reliability work starts with shortening the gap between broken and known.
- 10 Nov 2026
- 1 min
When a system depends on something outside your control, such as a third-party API, an upstream team or a vendor feed, it will break in ways you can't prevent. What you control is how long it stays broken without anyone knowing.
That gap, from broken to known, is where most of the damage happens. Bad data keeps flowing and downstream systems keep trusting it. By the time someone notices, the cleanup is far bigger than the original fault.
So before asking "how do we stop this failing?", I find it more useful to ask:
- What does working mean, as a number? That is the SLI.
- What number is unacceptable? That is the SLO.
- Who finds out when it crosses the line, and how quickly?
Prevention is worth doing, but it comes second. A system that fails loudly within minutes is usually in better shape than one that fails rarely and silently.