3 ms·

This is why you need to implement exponential backoff on retries for anything at scale https://en.wikipedia.org/wiki/Exponential_backoff https://en.wikipedia.or

This is why you need to implement exponential backoff on retries for anything at scale https://en.wikipedia.org/wiki/Exponential_backoff https://en.wikipedia.org/wiki/Exponential_backoff

hnarn 6y ago
Is this what's commonly seen as "Retrying connection in:" with a number of seconds increasing like 5 -> 10 -> 20 -> 40 -> 80 etc?
- whycombagator 6y ago
  Could be. If implementing exponential backoff it’s always a good idea to add some jitter (randomness) to the backoff, otherwise you can end up with multiple processes retrying all at the same time, all backing off again, and then all retrying at the same time etc etc
  - manojlds 6y ago
    Called Thundering Herd problem.
fitblipper 6y ago
This is why you need some form of network analytics to figure out if the new version you are rolling out is breaking anything or even seeing of something _is_ breaking. I get the "we don't want to compromise security even an inch", but how secure is a messaging app that cannot send messages? Even more critical since anyone can write clients that function with the server and thus a malicious actor could attach them without them knowing how to identify it or stop it.
- zamadatix 6y ago
  I'm not sure it had anything to do with "a new version rolling out", the issue has always been there it just never had a trigger (massive enough load) up until this point. I suppose the app that can't send messages is the most secure of all! All joking aside if their focus is security over anything else then this is acceptable. Unfortunately I don't think most of the users have the same order of priorities so it's bound to create some tension over time in multiple ways such as testing analytics, features that don't get added, or just general friction due to always requiring "most secure". The last part I agree is probably the most concerning though, it seems like Signal's centralized services aren't ready to be battle tested from attacks bound to come due to it's popularity growth. I'm not sure matrix is the silver bullet to that problem either... email isn't resilient to DoS because it's distributed it's resilient because the key centralized players like Gmail can withstand constant attack without affecting service due to their scale. Neither Signal nor Matrix are ready for the attention that comes with serving billions from that perspective. It's something I think will come with growing pains though.
whycombagator 6y ago
Looks like they had exponential backoff but they didn’t have any jitter (among other things) Helpful comment from this thread: https://news.ycombinator.com/item?id=25803251 https://news.ycombinator.com/item?id=25803251
- nabla9 6y ago
  Exponential backoff retry time slots can be purchased using bitcoin.

This is why you need to implement exponential backoff on retries for anything at scale https://en.wikipedia.org/wiki/Exponential_backoff https://en.wikipedia.org/wiki/Exponential_backoff

hnarn 6y ago

Is this what's commonly seen as "Retrying connection in:" with a number of seconds increasing like 5 -> 10 -> 20 -> 40 -> 80 etc?

whycombagator 6y ago
Could be. If implementing exponential backoff it’s always a good idea to add some jitter (randomness) to the backoff, otherwise you can end up with multiple processes retrying all at the same time, all backing off again, and then all retrying at the same time etc etc
- manojlds 6y ago
  Called Thundering Herd problem.

fitblipper 6y ago

This is why you need some form of network analytics to figure out if the new version you are rolling out is breaking anything or even seeing of something _is_ breaking. I get the "we don't want to compromise security even an inch", but how secure is a messaging app that cannot send messages? Even more critical since anyone can write clients that function with the server and thus a malicious actor could attach them without them knowing how to identify it or stop it.

zamadatix 6y ago
I'm not sure it had anything to do with "a new version rolling out", the issue has always been there it just never had a trigger (massive enough load) up until this point. I suppose the app that can't send messages is the most secure of all! All joking aside if their focus is security over anything else then this is acceptable. Unfortunately I don't think most of the users have the same order of priorities so it's bound to create some tension over time in multiple ways such as testing analytics, features that don't get added, or just general friction due to always requiring "most secure". The last part I agree is probably the most concerning though, it seems like Signal's centralized services aren't ready to be battle tested from attacks bound to come due to it's popularity growth. I'm not sure matrix is the silver bullet to that problem either... email isn't resilient to DoS because it's distributed it's resilient because the key centralized players like Gmail can withstand constant attack without affecting service due to their scale. Neither Signal nor Matrix are ready for the attention that comes with serving billions from that perspective. It's something I think will come with growing pains though.

whycombagator 6y ago

Looks like they had exponential backoff but they didn’t have any jitter (among other things) Helpful comment from this thread: https://news.ycombinator.com/item?id=25803251 https://news.ycombinator.com/item?id=25803251

nabla9 6y ago
Exponential backoff retry time slots can be purchased using bitcoin.