5 ms·

>I think the @signalapp apps DDoS'ed the server. and so the chickens come home to roost. Moxies vehement rejection of a distributed design seems less and less

>I think the @signalapp apps DDoS'ed the server. and so the chickens come home to roost. Moxies vehement rejection of a distributed design seems less and less tenable each outage. Last time it was what...verification numbers that werent getting sent? and FWIW the divinations from the community are exceedingly helpful in a time when not even the signal website seems to confirm or deny any sort of outage. Signups are still being taken and the outage page at signal.org is still as static as ever. Even the twitter hasnt seen an update in nearly a day. Is anyone at Signal foundation at the controls? As a signal user myself I know this is going to sound rude but at this point other than the endorsements from musk and dorsey, why would any new user consider this service at all if its been down for nearly two days? theres never any postmortem, and communication is generally evangelical or solicitous in nature for either installs or donations.

rglullis 6y ago
In some ways, it seems like we are watching a warp-speed demonstration of how evolutionary processes work. Environmental pressure, lots of contenders coming with slightly mutations and all of them passing through some fitness filter. It's just too bad that those arguing for decentralized systems fail to spread the meme more effectively. We federalists are like cockroaches.
LurkersWillLurk 6y ago
Respectfully, how exactly would federation have helped with this outage? Signal's own official clients failed to properly back off from spamming the server with incessant requests. I'm not entirely sure how more third party clients would have helped with this issue.
- bilal4hmed 6y ago
  My question exactly, if majority of the users have an account on the @matrix instance and if that fails, wont the same issue happen ?
  - rglullis 6y ago
    The instance dies, not the whole system. Synapse (Matrix reference server implementation) had for a long time severe performance issues when dealing with large rooms. Clients would try to enter a room and would take literally forever to be able to get messages for that room. Also, the natural reaction from people upon seeing how some rooms from bigger instances were causing problem was to create their own instances and their own rooms. Federated protocols are anti-fragile.
- Matumio 6y ago
  It would not have prevented the outage. But there would have been other servers where people could register and communicate during the outage. At least with Mastodon this seems to be working in practice. Over-populated instances usually switch to invite-only registration.
- Youden 6y ago
  Federation would, technically, decentralize the network and ensure there is no single point of failure. Federation would mean Matrix is about as likely to have a network-wide outage as email (or, for a more apt comparison, Mastodon). Realistically, I understand that the Matrix.org homeserver is an enormous part of the network and an outage there would be comparable to this one. However I believe that to be an orthogonal problem that also should be solved. I'm actually curious, why does Matrix promote the Matrix.org homeserver so strongly? Why isn't there an easy link to community-run homeservers? If you look at Mastodon for example [0], their onboarding process directs you to a registry of community servers [1], which actively promotes diversity in the network and therefore network resilience and reliability. Matrix meanwhile, just links you to element.io, which creates an account on the matrix.org homeserver. [0]: https://joinmastodon.org/ https://joinmastodon.org/ [1]: https://joinmastodon.org/communities https://joinmastodon.org/communities
- xorcist 6y ago
  "Oh no! Ten million people set up their own web page yesterday! The web broke. We just have wait a bit while the Elders of the Internet sort this one out." Said no one ever.
hayst4ck 6y ago
You seem to make a lot of statements which imply you have more knowledge than I would expect. Yesterday you seemed to have insider details on how they were managing (or not) their capacity. Where are you getting your information on the inner workings of signal? I am definitely getting strong signal that you are the type of person who has no perspective on what they are talking about, but I'm open to the idea I'm wrong and would love to be proven so. I think your expectations are out of line for the size and maturity of the company.
- rglullis 6y ago
  FWIW, (pre-Facebook acquisition) WhatsApp's systems were serving tens if not hundreds of millions of people and they never failed so catastrophically. There was a difference in approach, though. WhatsApp was using battle-tested technology (ejabberd, XMPP) and removing things that didn't make sense to them (federation). Moxie had one trick (end-to-end encryption) and wanted to build a whole messaging platform around that. Instead of standing in shoulders of giants, he chose to ignore all the lessons from the past and do everything from scratch out of the belief that he can do better. At some point, these systems break. In Signal's case, this point has been reached.
  - hayst4ck 6y ago
    While actual technology used and its effect on failure is an interesting debate, I think the growth curve involved has a vastly larger influence on stability. My potentially wrong understanding is they experienced a ~10x growth in a ~2 week period. Technology does not exist absent its context. One might use MySQL, but is it sharded? are reads critical? Lets say you go from 1 database to 8 databases and then from 8 databases to 128? How much tooling do you think is involved in those changes (machine configuration, service discovery, shading behavior, machine replacement behavior, how machines are monitored). At what point do assumptions made under a small system completely fail in large systems? I can manually configure 8 machines by hand no problem. I can add 2 more no problem. But 100 machines? That territory starts to require automation. You can potentially look at 100 individual graphs for 100 different machines, but what about 1000 graphs? All of a sudden aggregation and drill down become necessary. None of us (I think) have enough first hand information to understand why scaling up 10x was complicated for them. What I can almost promise is that if you have 10 servers and you go to 100, you are going to run into all types of problems you did not have at 10. Coordinated behavior across 100 machines is going to be a lot less forgiving than coordinated behavior across 10. Imagine a relationship where you have a every frontend talking to every data store. With 5 data stores, random failure isn't all that likely, but with 1000, it's much more likely. If every frontend talks to every data store, a single data store slowing down could result in ALL frontend stuck failing to make requests to it. Machine connection limits can be reached, interrupts from the NIC can bottleneck, service discovery can fail. Does technology have an effect? Yes. Is lack of exponential backoff somewhat embarrassing? Yeah. Is failing spectacularly to keep up with exponential growth over a short period unexpected? No. I would expect any website or service already at scale to spectacularly fail to service 10x growth in a 2 week period.

rglullis 6y ago

In some ways, it seems like we are watching a warp-speed demonstration of how evolutionary processes work. Environmental pressure, lots of contenders coming with slightly mutations and all of them passing through some fitness filter. It's just too bad that those arguing for decentralized systems fail to spread the meme more effectively. We federalists are like cockroaches.

LurkersWillLurk 6y ago

Respectfully, how exactly would federation have helped with this outage? Signal's own official clients failed to properly back off from spamming the server with incessant requests. I'm not entirely sure how more third party clients would have helped with this issue.

bilal4hmed 6y ago
My question exactly, if majority of the users have an account on the @matrix instance and if that fails, wont the same issue happen ?
- rglullis 6y ago
  The instance dies, not the whole system. Synapse (Matrix reference server implementation) had for a long time severe performance issues when dealing with large rooms. Clients would try to enter a room and would take literally forever to be able to get messages for that room. Also, the natural reaction from people upon seeing how some rooms from bigger instances were causing problem was to create their own instances and their own rooms. Federated protocols are anti-fragile.
Matumio 6y ago
It would not have prevented the outage. But there would have been other servers where people could register and communicate during the outage. At least with Mastodon this seems to be working in practice. Over-populated instances usually switch to invite-only registration.
Youden 6y ago
Federation would, technically, decentralize the network and ensure there is no single point of failure. Federation would mean Matrix is about as likely to have a network-wide outage as email (or, for a more apt comparison, Mastodon). Realistically, I understand that the Matrix.org homeserver is an enormous part of the network and an outage there would be comparable to this one. However I believe that to be an orthogonal problem that also should be solved. I'm actually curious, why does Matrix promote the Matrix.org homeserver so strongly? Why isn't there an easy link to community-run homeservers? If you look at Mastodon for example [0], their onboarding process directs you to a registry of community servers [1], which actively promotes diversity in the network and therefore network resilience and reliability. Matrix meanwhile, just links you to element.io, which creates an account on the matrix.org homeserver. [0]: https://joinmastodon.org/ https://joinmastodon.org/ [1]: https://joinmastodon.org/communities https://joinmastodon.org/communities
xorcist 6y ago
"Oh no! Ten million people set up their own web page yesterday! The web broke. We just have wait a bit while the Elders of the Internet sort this one out." Said no one ever.

hayst4ck 6y ago

You seem to make a lot of statements which imply you have more knowledge than I would expect. Yesterday you seemed to have insider details on how they were managing (or not) their capacity. Where are you getting your information on the inner workings of signal? I am definitely getting strong signal that you are the type of person who has no perspective on what they are talking about, but I'm open to the idea I'm wrong and would love to be proven so. I think your expectations are out of line for the size and maturity of the company.

rglullis 6y ago
FWIW, (pre-Facebook acquisition) WhatsApp's systems were serving tens if not hundreds of millions of people and they never failed so catastrophically. There was a difference in approach, though. WhatsApp was using battle-tested technology (ejabberd, XMPP) and removing things that didn't make sense to them (federation). Moxie had one trick (end-to-end encryption) and wanted to build a whole messaging platform around that. Instead of standing in shoulders of giants, he chose to ignore all the lessons from the past and do everything from scratch out of the belief that he can do better. At some point, these systems break. In Signal's case, this point has been reached.
- hayst4ck 6y ago
  While actual technology used and its effect on failure is an interesting debate, I think the growth curve involved has a vastly larger influence on stability. My potentially wrong understanding is they experienced a ~10x growth in a ~2 week period. Technology does not exist absent its context. One might use MySQL, but is it sharded? are reads critical? Lets say you go from 1 database to 8 databases and then from 8 databases to 128? How much tooling do you think is involved in those changes (machine configuration, service discovery, shading behavior, machine replacement behavior, how machines are monitored). At what point do assumptions made under a small system completely fail in large systems? I can manually configure 8 machines by hand no problem. I can add 2 more no problem. But 100 machines? That territory starts to require automation. You can potentially look at 100 individual graphs for 100 different machines, but what about 1000 graphs? All of a sudden aggregation and drill down become necessary. None of us (I think) have enough first hand information to understand why scaling up 10x was complicated for them. What I can almost promise is that if you have 10 servers and you go to 100, you are going to run into all types of problems you did not have at 10. Coordinated behavior across 100 machines is going to be a lot less forgiving than coordinated behavior across 10. Imagine a relationship where you have a every frontend talking to every data store. With 5 data stores, random failure isn't all that likely, but with 1000, it's much more likely. If every frontend talks to every data store, a single data store slowing down could result in ALL frontend stuck failing to make requests to it. Machine connection limits can be reached, interrupts from the NIC can bottleneck, service discovery can fail. Does technology have an effect? Yes. Is lack of exponential backoff somewhat embarrassing? Yeah. Is failing spectacularly to keep up with exponential growth over a short period unexpected? No. I would expect any website or service already at scale to spectacularly fail to service 10x growth in a 2 week period.