3 ms·

Could you point us to any specific past instances? I'd be interested to read about them.

• jeffreyq 6mo ago
  https://blog.railway.com/p/incident-report-february-11-2026 https://blog.railway.com/p/incident-report-february-11-2026
  • x0x0 6mo ago
    "we did not have the monitoring or controls to prevent our anti-fraud from hard killing 3% of workloads, including many instances of pg" Oof.
    • rzmmm 6mo ago
      Needs an anti-anti-fraud service which terminates malfunctioning anti-fraud services.
      • x0x0 6mo ago
        When I've written similar services, there was a (low) hard cap on how many fraud decisions they could action before they quit and paged. If we were getting hit with a wave of something, a human had to temporarily bump that limit.