Quarantine removes a component from live service the moment it looks unhealthy — a rising error rate, a leak, or persistent flapping. Rather than let it keep serving errors and slow responses, the system detaches it, lets the healthy remainder carry the load, and holds the suspect out until it is diagnosed, repaired, or restarted, then reintroduced.

Bass groups it under Prevent Faults. Removing a component on a schedule to scrub latent faults — memory leaks, fragmentation — is the proactive variant, software rejuvenation.

How It Works

  • Watch per-instance health signals: error rate, latency, resource growth, or health-check failures.
  • When a signal crosses its threshold, stop routing new work to the instance and drain in-flight requests.
  • Detach it from the pool and hold it out — no traffic — while diagnostics, a restart, or a repair run.
  • Reintroduce once it passes health checks, ideally on shadow or canary traffic before taking full load.

Failure Modes

  • Quarantining under load with no spare shifts the removed traffic onto the rest, which saturate and fail in turn.
  • Over-sensitive thresholds pull healthy instances, so the pool flaps in and out and capacity churns.
  • Abrupt detachment without draining drops the instance’s in-flight requests and any session state it held.

Verification

  • Fault injection: push an instance past its error threshold; assert it drains and detaches within the target window.
  • Confirm reintroduction: a repaired instance rejoins only after passing health checks, and load rebalances without an error spike.
  • Track pool size and quarantine in/out events; a steady decline or rapid flapping both signal mis-tuned thresholds.
  • Software rejuvenation is the proactive form: recycle an instance on a schedule to clear latent faults before they surface.
  • A circuit breaker is the client-side counterpart — it stops calls to a bad dependency; quarantine removes the bad instance from the pool.

References