Failure Modes: Why You Need To Know Them

Every system has failure modes but the most common and nasty ones are introduced by bimodal behaviors. A mode is a distinct setting within a machine interface, in which the same user input will produce perceived results different from those that it would in other circumstances. If a Kubernetes pod normally calls a cluster local service, but can fallback to an external service under a certain condition, that’s a biodal behavior. If you call your database every 5 minutes (happy path) but in case of failure, you retry every 100 milliseconds, that's a bimmodal behavior of the system.