8 ms·

Protonmail was down

109 points by Fiveplus 6y ago

philshem 6y ago
> Proton services will be offline for a few minutes for a database intervention. Monday 1st February 2021 08:15:00 https://protonstatus.com/incidents/125 https://protonstatus.com/incidents/125 snapshot of the status page: https://web.archive.org/web/20210201074153/https://protonstatus.com/ https://web.archive.org/web/20210201074153/https://protonsta...
- dathinab 6y ago
  Yes, and I guess it somehow went wrong. Currently it's still a partial outage (due to timeouts). My guess they created a backup system on the fly or similar on part of they system while they try to fix the other part.
  - johnisgood 6y ago
    I saw a 503, then the server was completely down at some point. :D Then it went back up and it was super slow.
raptorraver 6y ago
> We are currently experiencing some technical difficulties with a planned maintenance, and are working to bring the service back online as soon as possible. https://twitter.com/ProtonMail/status/1356145410574852098?s=20 https://twitter.com/ProtonMail/status/1356145410574852098?s=...
deleted 6y ago
[deleted]
marcusfrex 6y ago
It is not for a few minutes but it had been nearly an hour. Seriously considering to switch to another provider. Damn!!
- ternaryoperator 6y ago
  If an hour's downtime would make you 'seriously consider' switching, I doubt you will find any service that meets your requirements. You're asking for 99.96% uptime. I don't think any email provider will guarantee you that.
  - wwn_se 6y ago
    Well gmail has 99.987% currently but only 99.9% is promised according to SLA. https://support.google.com/googlecloud/answer/6056635?hl=en#zippy=%2Chow-reliable-are-g-suite-and-google-cloud-platform https://support.google.com/googlecloud/answer/6056635?hl=en#...
    - another-dave 6y ago
      And Protonmail offer 99.95% according to theirs (as of 2018, latest info I could find) — https://protonmail.com/blog/protonmail-reliability-sla/ https://protonmail.com/blog/protonmail-reliability-sla/
      - pmlnr 6y ago
        99.95% is a high standard already: 365x0.0005x24 is 4.38 hours outage for a year. In our current world, 4.38 hours is certainly not enough time to, say, rebuild a DB from backup, if needed.
        throwaway81523 6y ago
        Yes, you fail over to a hot mirror, or if you really have to rebuild because something went nuts, do it from a recent snapshot.
        another-dave 6y ago
        Yeah, especially when they're saying it's calculated on a calendar month, so you can't even "draw down" on previous months with 100% uptime if you have a serious outage.
hugofloss 6y ago
It seems to be related to a database intervention, but can anyone shed some light on how this results in outages in both incoming and outgoing mail servers?
- detaro 6y ago
  database has user accounts, email server needs to lookup user account information of some kind to handle mail and can't would be the most straight-forward assumption.
kylehotchkiss 6y ago
Oof, incoming mail is down too. Does that mean any emails that you should have received in this downtime are lost?
- sz4kerto 6y ago
  No, because SMTP is a sane protocol, and it'll retry later.
- AntiRush 6y ago
  Probably not. https://tools.ietf.org/html/rfc5321#section-4.5.4 https://tools.ietf.org/html/rfc5321#section-4.5.4
- lifty 6y ago
  Normally the other side should retry delivering several times, with varying degrees of time between the retries (up to days).
marban 6y ago
Curious when Hey will see its first major downtime and the components that'll be involved.
aero-glide2 6y ago
Will I still receive emails sent during this period? Was supposed to receive an important one.
- RL_Quine 6y ago
  Email are in general retried for a short while if delivery fails. Earlier the SMTP server was responding: 550 5.5.1 Protocol error As of now the socket is timing out, so no mail can be delivered at all. Most services will retry this later on if possible, perhaps hours or days into the future so long as the response is not a permanent failure message.
sammorrowdrums 6y ago
Great timing /s. I switched to PM on Friday. I love the combination of features though. I pay for pro so I get catch-all email on custom domains. Really helps to see who sells your info and secure storage and no advertising with good web-mail client are obviously major selling points too. I wish them luck resolving the situation.
- quasirandom 6y ago
  I've used proton for several years now. The only headache I've ever had is being unable to subscribe to the openbsd mailing list. And that's only because openbsd added proton servers to a spam list... a decision I wish they would revisit. Edit: I initially said freebsd mailing list. It was the openbsd mailing list.
  - dathinab 6y ago
    > proton servers to a spam list Wait, what? Which serious organization adds a proper mail service on the spam list?
    - pmlnr 6y ago
      All of them, sadly. Outlook is probably the craziest of all, adding the a /16 of Hetzner, because 1 IP sent spam.
      - vlpwx 6y ago
        So MSFT continues its attempts to extinguish the smaller providers while ostensibly being in favor of "diversity" and open source. How about blocking MSFT in Germany for anti-competitive practices?
      - icefo 6y ago
        I seriously doubt that only one IP of Hetzner sent spam but unfortunately it seems to be pretty common for email providers to soft ban VPS IPs. I don't know how much spam come from cheap hosting services but between the hacked servers and servers rented by spammers it's probably a lot.
deleted 6y ago
[deleted]
philshem 6y ago
I'm now able to log in and access my emails. An incoming "test" email I sent from another email provider during the outage was successfully delivered.
b1476 6y ago
As a paying user I feel that the reliability of their services in recent months has been completely unacceptable for me. That combined with the ridiculously slow rollout of new products (will calendar ever be fully released?) has now made me look at alternatives. I love the product itself and the idea behind it, but the amount of money I pay to host multiple domains for mail and for their VPN service is pretty absurd in itself and now I have to put up with these reliability issues too?
- de6u99er 6y ago
  Your comment could explain why they need to have downtime for maintaining something. Maybe even a pressing security fix, that breaks things. That being said, I don't understand why companies don't have test systems for validation of updade and maintenance procedures. My systems actually always had two test systems. One which was identical with Prod, and another one where test-users could test new features coming out of development system(s).
  - dathinab 6y ago
    Just a guess, but I think they are currently in the process of slowly doing some major infrastructure migrations. And doing such thinks without downtime can be hard, especially if you did some design mistake at the beginning which wasn't obvious and didn't cause problems until they scaled to a much larger customer base and/or provided additional services (storage, calendar) which originally whern't planed/considered when creating the mail service.
- bobsmooth 6y ago
  Consider the unique services offered by Proton, are they available elsewhere at a similar price? They are not Google.
  - nix23 6y ago
    Yes Tutanota...BUT: https://www.cyberscoop.com/germany-court-ruling-tutanota-email-monitoring/ https://www.cyberscoop.com/germany-court-ruling-tutanota-ema...
kilboy 6y ago
(Speaking as a free member) Pretty reliable generally, meets almost all my expectations as a non intrusive mailing service. Surprised that it crashed, but gotta cut them some slack.
ucha 6y ago
I've experienced some serious bugs on ProtonMail, one of them being that a saved draft from the desktop web app subsequently sent from the iOS app doesn't show up in the sent mails. I've since switched to Tutanota which I'm very happy with and is in some ways more secure. It doesn't support third-party clients for security reasons but theirs is open source. And to top it off, it's significantly cheaper...

109 points by Fiveplus 6y ago

> Proton services will be offline for a few minutes for a database intervention. Monday 1st February 2021 08:15:00 https://protonstatus.com/incidents/125 https://protonstatus.com/incidents/125 snapshot of the status page: https://web.archive.org/web/20210201074153/https://protonstatus.com/ https://web.archive.org/web/20210201074153/https://protonsta...

dathinab 6y ago
Yes, and I guess it somehow went wrong. Currently it's still a partial outage (due to timeouts). My guess they created a backup system on the fly or similar on part of they system while they try to fix the other part.
- johnisgood 6y ago
  I saw a 503, then the server was completely down at some point. :D Then it went back up and it was super slow.

raptorraver 6y ago

> We are currently experiencing some technical difficulties with a planned maintenance, and are working to bring the service back online as soon as possible. https://twitter.com/ProtonMail/status/1356145410574852098?s=20 https://twitter.com/ProtonMail/status/1356145410574852098?s=...

deleted 6y ago

[deleted]

marcusfrex 6y ago

It is not for a few minutes but it had been nearly an hour. Seriously considering to switch to another provider. Damn!!

ternaryoperator 6y ago
If an hour's downtime would make you 'seriously consider' switching, I doubt you will find any service that meets your requirements. You're asking for 99.96% uptime. I don't think any email provider will guarantee you that.
- wwn_se 6y ago
  Well gmail has 99.987% currently but only 99.9% is promised according to SLA. https://support.google.com/googlecloud/answer/6056635?hl=en#zippy=%2Chow-reliable-are-g-suite-and-google-cloud-platform https://support.google.com/googlecloud/answer/6056635?hl=en#...
  - another-dave 6y ago
    And Protonmail offer 99.95% according to theirs (as of 2018, latest info I could find) — https://protonmail.com/blog/protonmail-reliability-sla/ https://protonmail.com/blog/protonmail-reliability-sla/
    - pmlnr 6y ago
      99.95% is a high standard already: 365x0.0005x24 is 4.38 hours outage for a year. In our current world, 4.38 hours is certainly not enough time to, say, rebuild a DB from backup, if needed.
      - throwaway81523 6y ago
        Yes, you fail over to a hot mirror, or if you really have to rebuild because something went nuts, do it from a recent snapshot.
      - another-dave 6y ago
        Yeah, especially when they're saying it's calculated on a calendar month, so you can't even "draw down" on previous months with 100% uptime if you have a serious outage.

hugofloss 6y ago

It seems to be related to a database intervention, but can anyone shed some light on how this results in outages in both incoming and outgoing mail servers?

detaro 6y ago
database has user accounts, email server needs to lookup user account information of some kind to handle mail and can't would be the most straight-forward assumption.

kylehotchkiss 6y ago

Oof, incoming mail is down too. Does that mean any emails that you should have received in this downtime are lost?

sz4kerto 6y ago
No, because SMTP is a sane protocol, and it'll retry later.
AntiRush 6y ago
Probably not. https://tools.ietf.org/html/rfc5321#section-4.5.4 https://tools.ietf.org/html/rfc5321#section-4.5.4
lifty 6y ago
Normally the other side should retry delivering several times, with varying degrees of time between the retries (up to days).

marban 6y ago

Curious when Hey will see its first major downtime and the components that'll be involved.

aero-glide2 6y ago

Will I still receive emails sent during this period? Was supposed to receive an important one.

RL_Quine 6y ago
Email are in general retried for a short while if delivery fails. Earlier the SMTP server was responding: 550 5.5.1 Protocol error As of now the socket is timing out, so no mail can be delivered at all. Most services will retry this later on if possible, perhaps hours or days into the future so long as the response is not a permanent failure message.

sammorrowdrums 6y ago

Great timing /s. I switched to PM on Friday. I love the combination of features though. I pay for pro so I get catch-all email on custom domains. Really helps to see who sells your info and secure storage and no advertising with good web-mail client are obviously major selling points too. I wish them luck resolving the situation.

quasirandom 6y ago
I've used proton for several years now. The only headache I've ever had is being unable to subscribe to the openbsd mailing list. And that's only because openbsd added proton servers to a spam list... a decision I wish they would revisit. Edit: I initially said freebsd mailing list. It was the openbsd mailing list.
- dathinab 6y ago
  > proton servers to a spam list Wait, what? Which serious organization adds a proper mail service on the spam list?
  - pmlnr 6y ago
    All of them, sadly. Outlook is probably the craziest of all, adding the a /16 of Hetzner, because 1 IP sent spam.
    - vlpwx 6y ago
      So MSFT continues its attempts to extinguish the smaller providers while ostensibly being in favor of "diversity" and open source. How about blocking MSFT in Germany for anti-competitive practices?
    - icefo 6y ago
      I seriously doubt that only one IP of Hetzner sent spam but unfortunately it seems to be pretty common for email providers to soft ban VPS IPs. I don't know how much spam come from cheap hosting services but between the hacked servers and servers rented by spammers it's probably a lot.

deleted 6y ago

[deleted]

philshem 6y ago

I'm now able to log in and access my emails. An incoming "test" email I sent from another email provider during the outage was successfully delivered.

b1476 6y ago

As a paying user I feel that the reliability of their services in recent months has been completely unacceptable for me. That combined with the ridiculously slow rollout of new products (will calendar ever be fully released?) has now made me look at alternatives. I love the product itself and the idea behind it, but the amount of money I pay to host multiple domains for mail and for their VPN service is pretty absurd in itself and now I have to put up with these reliability issues too?

de6u99er 6y ago
Your comment could explain why they need to have downtime for maintaining something. Maybe even a pressing security fix, that breaks things. That being said, I don't understand why companies don't have test systems for validation of updade and maintenance procedures. My systems actually always had two test systems. One which was identical with Prod, and another one where test-users could test new features coming out of development system(s).
- dathinab 6y ago
  Just a guess, but I think they are currently in the process of slowly doing some major infrastructure migrations. And doing such thinks without downtime can be hard, especially if you did some design mistake at the beginning which wasn't obvious and didn't cause problems until they scaled to a much larger customer base and/or provided additional services (storage, calendar) which originally whern't planed/considered when creating the mail service.
bobsmooth 6y ago
Consider the unique services offered by Proton, are they available elsewhere at a similar price? They are not Google.
- nix23 6y ago
  Yes Tutanota...BUT: https://www.cyberscoop.com/germany-court-ruling-tutanota-email-monitoring/ https://www.cyberscoop.com/germany-court-ruling-tutanota-ema...

kilboy 6y ago

(Speaking as a free member) Pretty reliable generally, meets almost all my expectations as a non intrusive mailing service. Surprised that it crashed, but gotta cut them some slack.

ucha 6y ago

I've experienced some serious bugs on ProtonMail, one of them being that a saved draft from the desktop web app subsequently sent from the iOS app doesn't show up in the sent mails. I've since switched to Tutanota which I'm very happy with and is in some ways more secure. It doesn't support third-party clients for security reasons but theirs is open source. And to top it off, it's significantly cheaper...