5 ms·

that’s not a ddos. it’s just called not being able to handle a spike in traffic.

• uncledave 6y ago
  USDOS - Unexpected Success Denial Of Service. I woke up one Monday morning to crashed servers and angry phone calls once in that situation. Unfortunately it wasn't success on my part but what I had done the Friday before was so crap it couldn't handle a hundred users. But we blamed it loudly on unexpected success, quickly patched the mistake and everyone was happy. YMMV :)
  • nitrogen 6y ago
    Lesson learned: never ship on Friday
• Daniel_sk 6y ago
  The application has a retry mechanism that will keep trying until a connection succeeds (with an exponential backoff), but it doesn't handle a case where the server is basically rejecting the messages on purpose (due to capacity issues). Suddenly all applications will start retrying the connection at once and there is no way to turn if off. So it will make the problem even much worse than it already is. They have added a server feature flag yesterday to change the maximum backoff time and they also added the handling of HTTP 508 error response.
  • williamdclt 6y ago
    I'm wondering if another way to handle that would be the server accepting the request, but not actually servicing it. Basically let it timeout so that the client doesn't retry (at least, does not retry before timeout)
  • Daniel_sk 6y ago
    Commits from yesterday: Feature flag for maximum retry backoff time: https://github.com/signalapp/Signal-Android/commit/93e9dd6425b22db13baaf7e7f780a93bda0a457e https://github.com/signalapp/Signal-Android/commit/93e9dd642... Add jitter to backoff time: https://github.com/signalapp/Signal-Android/commit/8f7fe5c3eeb693e132b3c7d8bc692546bd70d27d https://github.com/signalapp/Signal-Android/commit/8f7fe5c3e... Handle ServerRejectedException (HTTP 508): https://github.com/signalapp/Signal-Android/commit/c95f0fce6ee3b78ff82fde865b2ee49288e1303f https://github.com/signalapp/Signal-Android/commit/c95f0fce6... Feature flag for automatic session reset: https://github.com/signalapp/Signal-Android/commit/a3c7e7e552f35751c43e79481357980bb36a8404 https://github.com/signalapp/Signal-Android/commit/a3c7e7e55...
    • LurkersWillLurk 6y ago
      Can somebody clarify if I'm correct in believing that feature flags are client features that can be activated server-side (as opposed to an app update)?
      • Daniel_sk 6y ago
        These can be set by the server, you don't need to update the app. "A location for flags that can be set locally and remotely. These flags can guard features that are not yet ready to be activated." Here is the file with javadoc on top: https://github.com/signalapp/Signal-Android/blob/2c1c6fab3560ef12996030cf7c84cc0d895e329e/app/src/main/java/org/thoughtcrime/securesms/util/FeatureFlags.java https://github.com/signalapp/Signal-Android/blob/2c1c6fab356...
      • jimsparkman 6y ago
        That is correct. Typically used to enable or modify a feature in the client on your own time table vs. App Store approval.
    • erdeszt 6y ago
      Problem: whatever Solution: TRY-CATCH
      • fctorial 6y ago
        WHILE TRY CATCH
  • ericbarrett 6y ago
    The technical term for this is “thundering herd”
• m-p-3 6y ago
  Which is basically what a DDoS will do with its botnet. Nthis case it was an involuntary one without a C&C.