3 ms·

I'm afraid Signal really dropped the ball on this one. I'd be surprised if any of the new 50M users stick around after this fiasco. Also, this doesn't seem to a

I'm afraid Signal really dropped the ball on this one. I'd be surprised if any of the new 50M users stick around after this fiasco. Also, this doesn't seem to actually be a scaling/hardware issue like they claim. They're running on AWS so they should be able to throw more hardware at it in a matter of minutes, hours tops. As of now, they've been down for 14 hours! Not sure what's going on but it's the absolute worst time to have this happen :/ Edit: worth noting that I've been a Signal user since ~2015, and I've stood by them through multiple amateur-hour mistakes (including one where the iOS app would simply crash on boot for about a week, rendering it unusable to me and every other iOS user). The app was just starting to get into a non-insanely-buggy state that I was quite happy with, so it's upsetting to know that these sorts of mistakes are still being made.

vander_elst 6y ago
I also would have expected a better communication of the outage at least a bit more regularly (at least 1h since it seems to be an extremely large outage). Additionally, the status page, IMO, lacks of some information e.g. what services are up, historical view etc. Anyway, thrilled to see how it will change after this incident. In the meantime, best of luck to the signal sre team.
suyash 6y ago
give them 24-48 hours, it's a small team and they got a massive surge
leipert 6y ago
FWIW I sent messages earlier without a problem. Judging from their jobs page they are just hiring in US time zones, so it’s understandable that they might run out of steam after an incident going on into the middle of the night. HugOps! Hope they get it fully running soon. Maybe they’ll provide a good, public root cause analysis and I hope that they rethink hiring in order to get more TZ coverage.
mike_d 6y ago
> They're running on AWS The client entry point (api.directory.signal.org) seems to be pointing exclusively to Azure and IBM Cloud depending on geography. It looks like just DNS, the website, and maybe some backend stuff are on AWS. > should be able to throw more hardware at it in a matter of minutes, hours tops Who exactly is going to pay for the hundreds of thousands of dollars in cloud spend you propose they turn up?
- superdisk 6y ago
  > Who exactly is going to pay for the hundreds of thousands of dollars in cloud spend you propose they turn up? Their 100M+ loan they got from Acton

vander_elst 6y ago

I also would have expected a better communication of the outage at least a bit more regularly (at least 1h since it seems to be an extremely large outage). Additionally, the status page, IMO, lacks of some information e.g. what services are up, historical view etc. Anyway, thrilled to see how it will change after this incident. In the meantime, best of luck to the signal sre team.

suyash 6y ago

give them 24-48 hours, it's a small team and they got a massive surge

leipert 6y ago

FWIW I sent messages earlier without a problem. Judging from their jobs page they are just hiring in US time zones, so it’s understandable that they might run out of steam after an incident going on into the middle of the night. HugOps! Hope they get it fully running soon. Maybe they’ll provide a good, public root cause analysis and I hope that they rethink hiring in order to get more TZ coverage.

mike_d 6y ago

> They're running on AWS The client entry point (api.directory.signal.org) seems to be pointing exclusively to Azure and IBM Cloud depending on geography. It looks like just DNS, the website, and maybe some backend stuff are on AWS. > should be able to throw more hardware at it in a matter of minutes, hours tops Who exactly is going to pay for the hundreds of thousands of dollars in cloud spend you propose they turn up?

superdisk 6y ago
> Who exactly is going to pay for the hundreds of thousands of dollars in cloud spend you propose they turn up? Their 100M+ loan they got from Acton