4 ms·

You seem to make a lot of statements which imply you have more knowledge than I would expect. Yesterday you seemed to have insider details on how they were mana

You seem to make a lot of statements which imply you have more knowledge than I would expect. Yesterday you seemed to have insider details on how they were managing (or not) their capacity. Where are you getting your information on the inner workings of signal? I am definitely getting strong signal that you are the type of person who has no perspective on what they are talking about, but I'm open to the idea I'm wrong and would love to be proven so. I think your expectations are out of line for the size and maturity of the company.

rglullis 6y ago
FWIW, (pre-Facebook acquisition) WhatsApp's systems were serving tens if not hundreds of millions of people and they never failed so catastrophically. There was a difference in approach, though. WhatsApp was using battle-tested technology (ejabberd, XMPP) and removing things that didn't make sense to them (federation). Moxie had one trick (end-to-end encryption) and wanted to build a whole messaging platform around that. Instead of standing in shoulders of giants, he chose to ignore all the lessons from the past and do everything from scratch out of the belief that he can do better. At some point, these systems break. In Signal's case, this point has been reached.
- hayst4ck 6y ago
  While actual technology used and its effect on failure is an interesting debate, I think the growth curve involved has a vastly larger influence on stability. My potentially wrong understanding is they experienced a ~10x growth in a ~2 week period. Technology does not exist absent its context. One might use MySQL, but is it sharded? are reads critical? Lets say you go from 1 database to 8 databases and then from 8 databases to 128? How much tooling do you think is involved in those changes (machine configuration, service discovery, shading behavior, machine replacement behavior, how machines are monitored). At what point do assumptions made under a small system completely fail in large systems? I can manually configure 8 machines by hand no problem. I can add 2 more no problem. But 100 machines? That territory starts to require automation. You can potentially look at 100 individual graphs for 100 different machines, but what about 1000 graphs? All of a sudden aggregation and drill down become necessary. None of us (I think) have enough first hand information to understand why scaling up 10x was complicated for them. What I can almost promise is that if you have 10 servers and you go to 100, you are going to run into all types of problems you did not have at 10. Coordinated behavior across 100 machines is going to be a lot less forgiving than coordinated behavior across 10. Imagine a relationship where you have a every frontend talking to every data store. With 5 data stores, random failure isn't all that likely, but with 1000, it's much more likely. If every frontend talks to every data store, a single data store slowing down could result in ALL frontend stuck failing to make requests to it. Machine connection limits can be reached, interrupts from the NIC can bottleneck, service discovery can fail. Does technology have an effect? Yes. Is lack of exponential backoff somewhat embarrassing? Yeah. Is failing spectacularly to keep up with exponential growth over a short period unexpected? No. I would expect any website or service already at scale to spectacularly fail to service 10x growth in a 2 week period.
  - rglullis 6y ago
    If you haven't tested or don't know how your systems are going to react when operating for a different scale, then don't let it grow in the first place. By taking more that they could chew, they are burning each one of these 40M+ users. How many of these are going to keep trying other alternatives and how many are going to shrug and say "that shit doesn't work?"
    - hayst4ck 6y ago
      > If you haven't tested or don't know how your systems are going to react when operating for a different scale, In my opinion, this is a pretty unreasonable expectation. Not to mention any load testing is going to be done under a certain set of assumption, a set of assumptions that is almost guaranteed to be violated by actually scaling something. What's the most naive load test you can think of? I am going to see the maximum amount of requests my dev instance can handle. Oh i can handle 10 requests per second. If I run 24 instances on a machine, and run 100 machines, then I must be able to service 24,000 RPS. Ok... Now explain concretely, what kind of load testing would/should have been done that would have surfaced the lack of an exponential backoff without a priori knowledge of it? The category of things you don't know you don't know at 10x scale is large. The justification for spending time "scale testing" is going to be a hard when your manager goes to social media and sees every single person wanting backups for their texts. "Hmm should we apply engineering resources to the feature we know everyone wants, or exploring the potential land of failures that may or may not have payoff, and definitely not immediate payoff." Most of these prioritization situations are directly and solely informed via experiencing outages, just like this one. > then don't let it grow in the first place. This is the type of insight you get from experience, it's also the type of call that I personally think would be very hard to make. As an engineer you don't always know if an outage is going to solve itself, take 10 minutes, 30 minutes, or take 10 hours. Then communicating that to someone with the power to create that policy... How many (social) companies do you know of that haven't had absolutely catastrophic outages/security breeches? This is the type of outage that justifies reliability work and gets resources applied to it. This is the type of thing that results in an organization creating a somewhat rigorous incident response policy that handles the business side (keeping users informed, giving them the best experience possible). This is a growing pain of rapid maturity. I think you can talk the way you do because you have not personally faced this type of engineering situation (a guess). Easy to make comments from the peanut gallery, but try putting yourself in the engineer's or the CEO's shoes and walking through what information you have at the time and what options you have available at the time, without the benefit of hindsight. Imagine you have a pipeline of priorities you look at every day. Is spending time wooing a senior engineer to come work for you a lower priority than having a good story for what happens when you 10x your traffic in 2 weeks? Things can only be black and white in hindsight. Their response to this is infinitely more indicative of the type of organization they are than the actual outage.
exadeci 6y ago
And you are right, on their profile about: > Changing brake pads by day and learning python at night

rglullis 6y ago

FWIW, (pre-Facebook acquisition) WhatsApp's systems were serving tens if not hundreds of millions of people and they never failed so catastrophically. There was a difference in approach, though. WhatsApp was using battle-tested technology (ejabberd, XMPP) and removing things that didn't make sense to them (federation). Moxie had one trick (end-to-end encryption) and wanted to build a whole messaging platform around that. Instead of standing in shoulders of giants, he chose to ignore all the lessons from the past and do everything from scratch out of the belief that he can do better. At some point, these systems break. In Signal's case, this point has been reached.

hayst4ck 6y ago
While actual technology used and its effect on failure is an interesting debate, I think the growth curve involved has a vastly larger influence on stability. My potentially wrong understanding is they experienced a ~10x growth in a ~2 week period. Technology does not exist absent its context. One might use MySQL, but is it sharded? are reads critical? Lets say you go from 1 database to 8 databases and then from 8 databases to 128? How much tooling do you think is involved in those changes (machine configuration, service discovery, shading behavior, machine replacement behavior, how machines are monitored). At what point do assumptions made under a small system completely fail in large systems? I can manually configure 8 machines by hand no problem. I can add 2 more no problem. But 100 machines? That territory starts to require automation. You can potentially look at 100 individual graphs for 100 different machines, but what about 1000 graphs? All of a sudden aggregation and drill down become necessary. None of us (I think) have enough first hand information to understand why scaling up 10x was complicated for them. What I can almost promise is that if you have 10 servers and you go to 100, you are going to run into all types of problems you did not have at 10. Coordinated behavior across 100 machines is going to be a lot less forgiving than coordinated behavior across 10. Imagine a relationship where you have a every frontend talking to every data store. With 5 data stores, random failure isn't all that likely, but with 1000, it's much more likely. If every frontend talks to every data store, a single data store slowing down could result in ALL frontend stuck failing to make requests to it. Machine connection limits can be reached, interrupts from the NIC can bottleneck, service discovery can fail. Does technology have an effect? Yes. Is lack of exponential backoff somewhat embarrassing? Yeah. Is failing spectacularly to keep up with exponential growth over a short period unexpected? No. I would expect any website or service already at scale to spectacularly fail to service 10x growth in a 2 week period.
- rglullis 6y ago
  If you haven't tested or don't know how your systems are going to react when operating for a different scale, then don't let it grow in the first place. By taking more that they could chew, they are burning each one of these 40M+ users. How many of these are going to keep trying other alternatives and how many are going to shrug and say "that shit doesn't work?"
  - hayst4ck 6y ago
    > If you haven't tested or don't know how your systems are going to react when operating for a different scale, In my opinion, this is a pretty unreasonable expectation. Not to mention any load testing is going to be done under a certain set of assumption, a set of assumptions that is almost guaranteed to be violated by actually scaling something. What's the most naive load test you can think of? I am going to see the maximum amount of requests my dev instance can handle. Oh i can handle 10 requests per second. If I run 24 instances on a machine, and run 100 machines, then I must be able to service 24,000 RPS. Ok... Now explain concretely, what kind of load testing would/should have been done that would have surfaced the lack of an exponential backoff without a priori knowledge of it? The category of things you don't know you don't know at 10x scale is large. The justification for spending time "scale testing" is going to be a hard when your manager goes to social media and sees every single person wanting backups for their texts. "Hmm should we apply engineering resources to the feature we know everyone wants, or exploring the potential land of failures that may or may not have payoff, and definitely not immediate payoff." Most of these prioritization situations are directly and solely informed via experiencing outages, just like this one. > then don't let it grow in the first place. This is the type of insight you get from experience, it's also the type of call that I personally think would be very hard to make. As an engineer you don't always know if an outage is going to solve itself, take 10 minutes, 30 minutes, or take 10 hours. Then communicating that to someone with the power to create that policy... How many (social) companies do you know of that haven't had absolutely catastrophic outages/security breeches? This is the type of outage that justifies reliability work and gets resources applied to it. This is the type of thing that results in an organization creating a somewhat rigorous incident response policy that handles the business side (keeping users informed, giving them the best experience possible). This is a growing pain of rapid maturity. I think you can talk the way you do because you have not personally faced this type of engineering situation (a guess). Easy to make comments from the peanut gallery, but try putting yourself in the engineer's or the CEO's shoes and walking through what information you have at the time and what options you have available at the time, without the benefit of hindsight. Imagine you have a pipeline of priorities you look at every day. Is spending time wooing a senior engineer to come work for you a lower priority than having a good story for what happens when you 10x your traffic in 2 weeks? Things can only be black and white in hindsight. Their response to this is infinitely more indicative of the type of organization they are than the actual outage.

exadeci 6y ago

And you are right, on their profile about: > Changing brake pads by day and learning python at night