12 ms·
ProtonMail Is Down
- ugjka 4y agoDown hard
- futuretaint 4y agoits like blackhawk down but w/ proton mail.
- ugjka 4y agoI mean the connection drops immediately, unlike the other times when it spins forever
- deleted 4y ago[deleted]
- Jaepa 4y agoI'm kind of curious what the source of the outage is. It looks like most of their services are down, including their incident status page¹. The number of things I can think of that so many different services & are not a massively catastrophic failure is pretty small. Though that may be a lack of imagination on my part. 1: https://status.protontech.ch/incidents/195 https://status.protontech.ch/incidents/195
- IG_Semmelweiss 4y agoI'm seeing their status page up now. But they are down hard otherwise.
- jjtheblunt 4y agomaybe the status page is therefore also down, misleadingly showing its last pre-down state
- willis936 4y agoThey have another status page that currently reports everything is up (it isn't). https://status.proton.me/ https://status.proton.me/
- LeifCarrotson 4y agoAnd another that says that some systems (many systems) are experiencing issues/in a state of "partial outage": https://protonstatus.com/ https://protonstatus.com/
- KineticArms 4y ago"An error occurred on client"
- guerrilla 4y agoMaybe it's the same hard drive problem a lot of others (including HN) had.
- 0xbadcafebee 4y agoDoesn't have to be catastrophic failure. Most people just assume that such services are architected properly to be HA, but they usually aren't. They also aren't operated as HA usually, so things like a config change are rolled out globally to everything at once, and then everything breaks and they can't roll back, and you have total outage. Very common. ...and of course, they might not be down at all, it might just be (example) a route from the west coast got fucked in a BGP table so the west coast can't access it, and if that's where the majority of their users are that's all the comments you'll see.
- confounded 4y agoWhat is HA?
- sllabres 4y agoA bit nitpicking but even with a proper implemented HA solution there would be a service interruption.
- krn 4y ago> The number of things I can think of that so many different services & are not a massively catastrophic failure is pretty small. Like, for instance, a simple DNS or BGP misconfiguration that takes the entire network down? Cloudflare (June 2022): https://news.ycombinator.com/item?id=31823132 https://news.ycombinator.com/item?id=31823132 OVH (October 2021): https://news.ycombinator.com/item?id=28849319 https://news.ycombinator.com/item?id=28849319 Facebook (October 2021): https://news.ycombinator.com/item?id=28752131 https://news.ycombinator.com/item?id=28752131 Google (June 2019): https://news.ycombinator.com/item?id=20077421 https://news.ycombinator.com/item?id=20077421
- 29083011397778 4y agoWe don't know for certain, but you may have missed Rogers [0] - which brought down cell phones (every function but wifi), debit transactions, and 911 services. [0] https://en.m.wikipedia.org/wiki/2022_Rogers_Communications_outage https://en.m.wikipedia.org/wiki/2022_Rogers_Communications_o...
- tick_tock_tick 4y agoNo ones it too big to not fuck up BGP or DNS from time to time.
- 0xbadcafebee 4y agohttps://protonstatus.com/incidents/195#update-201 45 minutes ago : Our engineering team is working on a database performance issue which is impacting Proton services Relational database woes. :( Probably re-indexing or slow queries or something. This is why companies with an RDBMS as a SPOF need a good DBA.
- brewdad 4y agoMaybe they installed a whole lot of SSDs on the same day 4.5 years ago.
- Eddy_Viscosity2 4y agoI'm on it now, must be back up?
- hamaluik 4y agoStill down for me :(
- IG_Semmelweiss 4y agoAgreed. I'm back up too
- Eddy_Viscosity2 4y agoCorrection, its down.
- lom 4y agoWas down 5 minutes ago, but is now working again for me
- LeoPanthera 4y agoI've been having trouble with iCloud all day too. (Try Apple ID > Password and Security on your iOS device). It surely can't be related.
- bilkow 4y agoA minute ago StackOverflow was also throwing "Oops! Something Bad Happened!" to me on many different pages. I wonder if that's somehow related or just a coincidence.
- DavideNL 4y ago"Proton Mail mobile apps are are now operational again, and so is Proton Calendar." https://nitter.net/ProtonSupport https://nitter.net/ProtonSupport also RSS feed: https://nitter.net/ProtonSupport/rss https://nitter.net/ProtonSupport/rss
- _8j50 4y agoI thought it was my phone or vpn, I restarted it many times. Where would I be without HN :-)
- Aardwolf 4y ago3 days ago, not very far! https://news.ycombinator.com/item?id=32023848 https://news.ycombinator.com/item?id=32023848 [Hacker News was down]
- willis936 4y agoGeez if both were down at the same time I might have no option but to touch grass.
- hammyhavoc 4y agoAnd if grass isn't available, I guess there's always myself.
- vaylian 4y agofunny that you say that: "HN is up again" https://news.ycombinator.com/item?id=32026571 https://news.ycombinator.com/item?id=32026571 2 days ago
- john-radio 4y ago> Where would I be without HN :-) Reading the regular news like an idiot!
- usr1106 4y agoJust remember, clicking on refresh and searching the net what is going on want make the problem go anyway faster. Few of us urgently depend on email just this hour. An even if you do setting your alarm to some suitable interval and just doing something else is much more productive. Just act like the Arab cliche taxi driver who takes the hands of the steering wheel when it gets tight and say Allah will steer (No insult to anyone, I have never visited any Arab country. Probably the story is not true anyway, but a nice anecdote to accept things you cannot change.) When hackernews was down recently I wanted to report it as big news in our company chat. But it was rather late here and I had already closed all work stuff for the weekend. So I first slept and when I woke up everything was working again. Oh yeah, I missed to share the scoop with my colleagues, but I guess I'll manage without any damage to my professional reputation. Edit: Reading post-mortem afterwards is a different story. If it's well written instead of some corporate communications emptyspeak one can learn something. No mistake happens only once.
- gruez 4y ago>Few of us urgently depend on email just this hour. I agree that you shouldn't be manically refreshing your email every 5 minutes, but my mail service being down for a few hours would definitely be a major inconvenience if I needed to get a 2fa code during that time. Same if I needed to look at my email to view my tickets for an upcoming event.
- usr1106 4y agoThat sounds like an fragile setup if you depend on a single email account to get 2FA codes. I have them configured typically on my office workstation (reachable remotely if everything works), on my laptop, on my phone, and I have the static backup codes in my password manager which is also distributed over several indpendent machines. Of course if you have to respond to incoming tickets within a guaranteed short timeframe that gets a problem if nothing can come in. Make your risk assessment in advance: Can I live without it for a while, in the worst case even until I have an alternative provider? If so then just do so when an outage happens. If that's not acceptable for your case you need to have an independent backup before the first outage.
- mattanimation 4y agogood thing I just bought a 24 month pro sub last week...
- lizardactivist 4y agoIt sure was an excellent purchase -- one of the most secure services available world-wide, operated in the EU with strong privacy and completely outside US jurisdiction. You weren't able to collect your emails for a brief 5 minutes, but you'll live.
- martin8412 4y agoSwitzerland is not in the EU.
- deleted 4y ago[deleted]
- rdevnull 4y agoHello from Switzerland :) We are not in the EU not just "technically" but as a fact. None of these agreement covers the exchange of private emails or metadata with the EU.
- nix23 4y agoBut as a fact with have to comply with EU-Laws -> Schengen.
- motohagiography 4y agoIrony is the people most likey affected by a protonmail outage would be political staffers and strategic PR firms who would suddenly need to use messengers subject to disclosure. As someone taken out by the Rogers outage last week, I'd be concerned the infrastructure of western powers may be too fragile to project power in a seriously contested conflict.
- deleted 4y ago[deleted]
- Trias11 4y agoSlower login, but works
- 2Gkashmiri 4y ago2 things to ask. 1. why don't people selfhost email more? mailinabox is pretty painless to set up and would give you infinitely more security/privacy whatever you want. 2. why is email deliverability still an issue? why do you need to use third party email tools to ensure your email doesn;t go to spam? i am not concerned with newsletters/ads/ that sort of thing but a replacement to gmail/protonmail?? i have personally gone with miab like last year and beyond the first few weeks of gmail putting my emails into spam(then having to inform the other user to unspam it), things have been painless. i get to occasionally spin up the ssh to update the server but its fine. backblaze b2 backups ensure i am somewhat safe(er). can it be better? yes. is it ready to be used in production, aka a replacement to your regular paid/free email? you be the judge but i am sold on this idea
- gobip 4y agoYou replied question 2 with question 1: we don't self host email, -because- of deliverability issues.
- 2Gkashmiri 4y agowhy are they in the first place?
- tick_tock_tick 4y agoSpammers
- skadamat 4y agoDeliverability is an issue in both directions: - when you send emails from MyWebsite.com, you want to make sure that Hotmail, Yahoo, Gmail, Proton Mail, etc users get your email. - when you receive emails to your hi@mywebsite.com email, you don't want spam V1 of email providers ostensibly solved this problem using lots of data as well as "supervised" machine learning (e.g. you telling the provider that a specific email is spam). Self-hosting seems awesome if you're only handing out the email to a few people (probably a few hundred max?), you don't think it will be widely shared, and it's a slightly unique email that makes it hard for a script to guess (e.g. "hello@mywebsite.com" will be easily guessed but "john2@mywebsite.com" may not be)
- devmunchies 4y agoHas to be related to this maintenance announced a couple days ago[1] > "We're conducting a short database intervention on Sunday, July 10th, 2022, between 9:00 - 10:00 am GMT+2. During this time, users will be unable to access all Proton services for 2-4 minutes. We apologize for the inconvenience." [1]: https://twitter.com/ProtonSupport/status/1545698028895584257 https://twitter.com/ProtonSupport/status/1545698028895584257
- amatecha 4y agoRather ironically, the site they link to for the incident status updates ( https://status.protontech.ch/incidents/195 https://status.protontech.ch/incidents/195 ) is not loading for me: "The connection has timed out"
- cyberpunk 4y agoI mean, no one cares if accessing mail is down, but it looks like their MX is down, the status page reports incoming mail has a ‘partial’ outage. I don’t care if I can’t access my email for a while. I very much care if email sent to me isn’t received. They should just be dumping all mail received at their MX’s to maildirs or something if it’s this bad and process them properly later. Losing data is not ok.
- deleted 4y ago[deleted]
- cmeacham98 4y agoSMTP is literally built with this scenario in mind. As long as their servers aren't incorrectly reporting a success the sender will (or at least should) retry.
- runlevel1 4y agoFortunately, retries are built into the spec (rfc5321). Typically, you retry for 3 days.
- shapefrog 4y ago> They should just be dumping all mail received at their MX’s to maildirs Not that it makes all that much difference, but this would break the encryption model for non-pgp inbound emails which are meant to get encrypted with the users public key on arrival. I have no doubt that manual interception of these messages would be trivial with a court order anyway.
- Ancapistani 4y agoSorry about that. I just moved the domain for my small business over to ProtonMail (where I already hosted my personal email). The fact that I did that guaranteed that it would go down. :)
- browda 4y agoI was logged-in -- and composing a message -- a few minutes ago. First there was that cannot reach backup server message followed by a bunch (don't you just love technical jargon?) of messages saying "Could not..." and then lost the message when the site itself apparently went down. (Well, Firefox was seriously unhappy...) I was ready to blame my ISP again but no, this one is on proton mail. (It's pingable and traceroute is happy.) I'm really glad that we didn't move our business over...
- nijave 4y ago>there was no way to roll back the software update oof >During this period, incoming email, outgoing email, and push notifications were also temporarily delayed They say no data was lost but I'm not sure how they can know a sending mail server didn't give up trying to deliver a message to Proton
- haswell 4y agoIt would be surprising if a the SMTP retry timeout of the average sending server was less than the duration of this outage. This is not to say there is no cause for concern, but SMTP was built for scenarios like this, at a time when these things happened far more often.
- yasinaydin 4y agoThis has always been my concern as well. I wonder if there are any global standards, requirements recommendations about m the minimum retry count or timeout.
- valera_rozuvan 4y agoLooks like it's down again: "Some systems are experiencing issues" [1] ------ [1] https://protonstatus.com/ https://protonstatus.com/
- Cxckers 4y agoIt has gone down again today. Their statement yesterday: Yesterday we unfortunately experienced technical difficulties which meant that Proton services were offline intermittently throughout the late afternoon and evening European time. During this period, incoming email, outgoing email, and push notifications were also temporarily delayed. As of 2:45AM Geneva time, all emails and notifications are caught up, but we are continuing to monitor the situation in case of further faults. We can confirm that no data and no emails were lost. Due to our redundant infrastructure, emails that were not immediately delivered were queued and automatically redelivered. We are still investigating the issue, but what we can share so far is that the technical difficulties were caused by a software update that was conducted over the weekend which adversely impacted the performance of certain systems. Unfortunately, this issue only appeared under under high loads which we were not able to fully simulate in testing. Furthermore, because this was a major software and operating system upgrade, there was no way to roll back the software update. This result of this was an extended period of intermittent outages and performance issues while we tried to work around the issue. We have now put a number of safeguards in place, and will continue to improve the reliability of our infrastructure to guard against this type of issue in the future. We apologize deeply for this issue and thank you again for your understanding. Our engineering team is currently conducting a root cause analysis and for transparency, we will be sharing results on the Proton blog after our investigation is complete.
- deleted 4y ago[deleted]
- jrgd 4y agoSeems like it’s not yet completely sorted… despite being impacted, i feel for the guys steering through this outage and bending backward! Good luck guys!