10 ms·
Email Address Disclosures, Preliminary Report
- emilburzo 10y agoHm, I didn't get anything. Did you sign up in the earlier or later stages?
- aaronpk 10y agoIt looks like it's sending them alphabetically. I wonder at what point in the alphabet someone hit the kill switch.
- emilburzo 10y agoMine starts with 'contact' and I didn't get it, so probably before that.
- turbohedgehog 10y ago0-9, A-Z were sent out and part of 'a'
- tgsovlerkhgsel 10y agoSounds like the popular "append e-mail address to e-mail text with each iteration of the loop while keeping the previous ones".
- anonbanker 10y agoNote to self: use a new email account when using this service.
- deleted 10y ago[deleted]
- Pfhreak 10y agoI try to do this for every service I give an email address to.
- logicallee 10y agogmail still drops the ball, because you have to give your realaddress+marker rather than being able to request a marker. The correct behavior is to be able to request a marker when signed into any email account, and on my side set the tag that it gets tagged with in that inbox as a result. The link between marker and inbox should remain secret.
- peteretep 10y agoI pay for a FastMail account which gives me up to 500 aliases. These are then sucked down in to my Gmail account.
- logicallee 10y agoI can't see a better example of Google dropping the ball than you paying for some other service so that you can then consume it from gmail. :) 500 is a reasonable limit I think, in case spamming would be some reason for them not to do this. I don't have an opinion about whether people should be able to send from marker@gmail.com or if it should just end up in a real inbox but without the ability to send from that address.
- nly 10y agoOut of curiosity, why do you do this? FastMail has an amazing webmail interface and Android app. I'd never go back to Gmail.
- peteretep 10y agoThe original thinking was to make it harder for casual snoopers. If FastMail gets compromised, then they'd need to be compromised over time and someone would need to review a lot of my email with a shelf-life of one-hour to understand who I was - I use auto-generated credentials, paid via Bitcoin, and login once a year via VPN to generate more addresses. If Gmail gets compromised, you'd need to be looking for a bunch of Fastmail accounts in To: addresses to link my primary email with those emails. If you wanted to track me down from an email address, you'd need a warrant in Australia (for FastMail), and the US (to find the account using those FastMail credentials), so I'd need to have actually done something wrong (which I haven't), and you'd need to convince judges in two jurisdictions of that. As I said, the threat model is against casual snoopers, rather than a determined state actor with proof of wrong-doing, as I don't think I'm even slightly interesting and I don't think I've done anything that would make me interesting. As it turns out, you could probably just read enough of my FastMail email as it came in (before it gets deleted by Gmail) to figure out who I am, so this is imperfect.
- pred_ 10y agoYou don't even have to specify an email address at all; it's considered optional in ACME.[0] [0] https://github.com/ietf-wg-acme/acme/blob/master/draft-ietf-acme-acme.md#registration-objects https://github.com/ietf-wg-acme/acme/blob/master/draft-ietf-...
- mholt 10y agoTo clarify: the email addresses are in the body of the message, not the To field.
- fvargas 10y agoCurious to see the reply-all thread that ensues.
- RKearney 10y agoIt probably won't be very exciting considering the emails existed only in the Body of the email. The emails themselves were only addressed to individuals. You can see this in the linked screenshot.
- andrewstuart2 10y agoSince both users mentioned were the last in the list of addresses for the email they received, my money's on a trivial mistake like: getEmailBody(users[:i]) instead of getEmailBody(users[i]) I typically prefer a high level of polymorphism in my code/APIs (sensibly handling single inputs vs. arrays) but this is a great counter-example even if not the actual root cause. Every feature is also a liability. Double edged sword. Etc.
- justinpombrio 10y agoSounds like an argument to write for user in users: getEmailBody(user) instead of for i in range(len(users)): getEmailBody(users[i]) (for the languages that let you do so).
- codygman 10y agoA type system can really help here as well. sendUserEmail :: FromEmail -> ToEmail -> [EmailHeader] -> EmailBody -> IO () However that also requires the discipline of wrapping a lower level function that is probably: sendEmails :: FromEmail -> [ToEmail] -> [EmailHeader] -> EmailBody However even a functional language would shy away from using indexing which could be argued to be the source of this problem.
- ubernostrum 10y agoNothing about your suggested solution requires or is exclusive to a statically-typed language. "Replace a send-to-arbitrary-number-of-addresses function with a send-to-exactly-one-address function" is possible in dynamically-typed languages, too, and as you openly admit even a statically-typed language is likely to model email sending in a way that accepts multiple recipient addresses. So your "type system can really help" is really just an irrelevancy you've come up with to try hide your attempt to shove your preferred programming paradigm onto other people. You should probably stop doing that.
- Johnny_Brahms 10y agoi thought of that as well.Went and looked at my mail list programme in Racket, and the mail procedure takes many recipients, but in the form if rest arguments, like: (define (send-mail msg . recipients) ...) passing it a list makes the recipients arg a list with a list of recipients. To actually send to multiple recipients I explicitly have to use the apply procedure that passes all list elements as arguments. as you said, not a staticly typed language. types would have made an eventual error easier to debug though.
- turbohedgehog 10y agoPreliminary report out from Let's Encrypt: https://community.letsencrypt.org/t/email-address-disclosures-preliminary-report-june-11-2016/16867 https://community.letsencrypt.org/t/email-address-disclosure... as far as I know, all emails starting with 0-9, A-Z and at least part of 'a' were exposed. I did not get one starting with 'g', so it's somewhere between 'a' and 'g' that it got stopped. Edit: "7,618 out of approximately 383,000 emails" were sent out
- mholt 10y agoWas just able to confirm, it's up to and including your email address. Mine starts with m so I see 3,761 email addresses. But for me, none lexicographically after my email address are exposed. Edit: Just want to add that I've made a similar mistake before (with a smaller user base). So I understand how easily these bugs occur. Given all their progress in the last few years, I still believe that the privacy and security of such a large portion of the Web could not be in better hands. Props to the LE team for a quick, responsible response.
- turbohedgehog 10y agoYou mean M, not m. (it's in ASCII order). Also giving out the number of addresses you see will allow someone after yours to connect your username with your email address if you weren't aware.
- mholt 10y agoYes, sorry, "M". And if this comment[1] is true, then that number won't reveal much. Their mailer is probably distributed across several nodes. [1] https://news.ycombinator.com/item?id=11881953 https://news.ycombinator.com/item?id=11881953
- turbohedgehog 10y agoMultiple different images and a pastebin I saw posted on twitter showed the same starting set of emails.
- Renner1 10y agoIt's disappointing to see this level of incompetence from a group responsible for such great leaps in web security. Let's Encrypt should take appropriate steps to ensure this never happens again lest they erode users' trust any further.
- steeve 10y agoThings like this happens all the time. Give them a break. They already did what needed to be done. It's a bad bug yes, but lots of people here could have done it.
- mei0Iesh 10y agoGood to know, but really I don't care if they send my email address to every other registrant. I run a public web server, I already receive junk email that must be filtered, so I see no problems. It has zero impact on the free certificate service they provide.
- jaas 10y agoHead of Let's Encrypt here. Our automated mail system had a bug that accidentally exposed about 1.9% of subscriber email addresses to the same 1.9% of recipients. Our sincerest apologies for this mistake. We will be doing a thorough postmortem to determine exactly how this happened and how we can prevent something like this from happening again. There is a preliminary report on the issue here: https://community.letsencrypt.org/t/email-address-disclosures-preliminary-report-june-11-2016/ https://community.letsencrypt.org/t/email-address-disclosure...
- tomjen3 10y agoWhat do you need email for, anyway? Not giving you one makes it pretty easy not to leak it.
- DanBC 10y agoHow big do you estimate your fine would have been if you were operating in the EU?
- dave2000 10y agoFor sharing a few email addresses? $0 I'd imagine. It's not like its anything important like credentials for shopping/banking or other details which could be used in identify theft. Worst case scenario; google's spam filters have to work a little harder. You'd not even notice. Yes, some people have chosen to run their own mail servers for some reason and those people might conceivably get a bit more spam for a while.
- pred_ 10y agoNot of those that did not specify any, I would imagine; you can use the ACME service without providing any email address at all.
- jcoffland 10y agoIt sucks this happened but I don't really care. You guys are providing such an amazing and sorely need service I have no problem cutting you some slack. I hope others will too. Of course those working for companies who's lunch you're eating will likely run with this as far as they can.
- yexponential 10y agoFor the curious this was the content of the email. Pretty generic. "Dear Let's Encrypt Subscriber, We're writing to let you know that we are updating the Let's Encrypt Subscriber Agreement, effective June 30, 2016. You can find the updated agreement (v1.1) as well as the current agreement (v1.0.1) in the "Let's Encrypt Subscriber Agreement" section of the following page: https://letsencrypt.org/repository/ https://letsencrypt.org/repository/ Thank you for helping to secure the Web by using Let's Encrypt"
- _kst_ 10y agoA reminder: If you got a copy of this e-mail (as I did), please don't repost it -- or if you do, don't include the e-mail addresses.
- jboles 10y agoInteresting that since the list of addresses was sequentially prepended to (if I understand the wording of the notice correctly), anyone who anonymously shares the list will incriminate themselves ,though to a smaller and smaller pool of peer customers.
- nilved 10y agoA simple solution to this would be to chop off an arbitrary number of addresses prior to disclosure. The first person can leak any number of emails, and the last can only leak one.
- jorams 10y agoThe list of addresses was prepended to the email, but the addresses were added to the end of the list itself. Thus, every recipient saw their own address as the last item in the list.
- edraferi 10y agoPlanned email blast accidentally cc'd other recipients, allowing users to see each other's email addresses. They caught it after <8,000 emails went out and are fixing the problem.
- DanBC 10y agoSome people are saying the emails were in the message body, not CC field. What's the limit for numbers of addresses in a CC field? Because this is several thousand addresses.
- aroch 10y agoIf you're sending to/from Gmail/Exchange the limit is somewhere around 100 addresses (I believe it's technically the byte-size of the field not strictly the number of addresses). The actual spec though, AFAIK, has no limit.
- jamiesonbecker 10y agoRight. SMTP header lines can't exceed 998 characters, but RFC2822 now allows multi-line headers[1], so there's no limit except one that might be imposed by an MTA (and that would generally only be imposed on sending, not receiving.) 1. https://www.ietf.org/rfc/rfc2822.txt https://www.ietf.org/rfc/rfc2822.txt §2.2.3
- eterm 10y agoIt wasn't a CC, if it was then there wouldn't have been a way to stop after only some had gone out, because that's not how CC works.
- theoh 10y agoIsn't the CC header essentially an instruction for the local MTA? So their local MTA might could have been relatively slowly working through the CC list (contacting each recipient mail server in turn). I'm not saying this is what actually happened.
- peterkelly 10y agoDirectly below the apology for leaking emails addresses, I get this message prominently displayed: > "Hey there! Looks like you're enjoying the discussion, but you're not signed up for an account. > When you create an account, we remember exactly what you've read, so you always come right back where you left off. You also get notifications, here and via email, whenever new posts are made. And you can like posts to share the love." > [Sign up] [Remind me tomorrow] No thanks :)
- mkj 10y agoI wonder if people whose email addresses start with the letter 'a' also get more spam.
- Ghostium 10y agoWhat I'm asking me myself everytime I see a post about data leak, could you sue a company for the leak?
- DanBC 10y agoIf they're in the UK you can report them to the regulator - the Information Commissioner. They tend to take a warn then fine approach.
- icebraining 10y agoYou can sue almost anyone for almost anything. Could you win? IANAL, but I think you'd have to prove damages.
- williamjackson 10y agoThis reminded me exactly of Python's mutable default arguments: http://docs.quantifiedcode.com/python-anti-patterns/correctness/mutable_default_value_as_argument.html http://docs.quantifiedcode.com/python-anti-patterns/correctn...
- frik 10y agoNote to self: keep using HTTP, and provide HTTPS for important website content (like shop payment) and use a SSL/TLS Cert that lasts 1 year.
- wlll 10y ago> keep using HTTP, and provide HTTPS for important website content (like shop payment) What if someone MITM's your site, injects some code so that when the user clicks on the checkout link/button they get sent to a malicious site?
- frik 10y agoHave you asked Amazon the same question? It works fine for them since 1994.
- pfg 10y agoAmazon has deployed HTTPS across all their sites a couple of weeks ago.
- frik 10y agoYes, just last week. And parts of site were down too for some hours. Is MITM something new? No. What is the point? Amazon worked fine since 1994/95. The whole HTTPS-only movement looks very orchestrated.
- wlll 10y agoHTTPS everywhere is a good thing, and if you don't understand why that's fine, I don't have the time to explain to you why you are wrong, but you are. Good luck deploying Internet connected services in the past.
- frik 10y agoThe difference is I understand the Pro and Cons. That's why HTTP and HTTPS is often better.
- appleflaxen 10y agoInteresting. Slightly embarrassing. Not a huge deal. Handled well.
- planetjones 10y agoThe Hyatt hotel in Switzerland did a similar thing a few weeks ago. They sent a mail shot to everyone using the CC function not BCC. I complained and their response was that they'd recalled the mail so 'that was that'. Of course a recall means nothing to the hordes of gmail addresses,etc. that the mail shot was sent to. It's a common problem and a big incentive to use throw away addresses.
- scandox 10y agoThe new head of the IIA (Irish Internet Association) did a cc on the entire membership just a few days ago announcing her arrival. Felt pretty sorry for her. She actually did the bcc correctly the first time but forgot the attachment then correcting that she did a cc. A comedy of errors....
- TallGuyShort 10y agoI once asked to be removed from a list and suggested they use CC instead of BCC. I accidentally did so by way of Reply-All. Boy did I feel stupid. For days. While everyone kept replying to me.
- TallGuyShort 10y agoI once asked to be removed from a list and suggested they use CC instead of BCC. I accidentally did so by way of Reply-All. Boy did I feel stupid. For days. While everyone kept replying to me.
- na85 10y agoThe jaded, bitter part of me hopes this will be another nail in the coffin for XaaS and the recent trend of centralizing everything onto Web services. The rest of me which is more jaded and bitter knows that it won't.
- bovermyer 10y agoWhat would your ideal online world be?
- Cyph0n 10y agoThe reason for this screw up was guessed by a Twitter user, and his theory was confirmed by Josh from Let's Encrypt [1]. The whole mess was caused by the Python `email` package, and specifically the behavior of the `MIMEMultipart` object [2]. When you reuse the same `MIMEMultipart` object for multiple emails, each destination address is appended. The same problem takes place when you use Python 3 [3]. [1]: https://twitter.com/0xjosh/status/741487697059946497 https://twitter.com/0xjosh/status/741487697059946497 [2]: https://docs.python.org/3/library/email.mime.html#email.mime.multipart.MIMEMultipart https://docs.python.org/3/library/email.mime.html#email.mime... [3]: http://i.imgur.com/XwWlUXv.png http://i.imgur.com/XwWlUXv.png
- drdaeman 10y agoI see that it's confrimed, but find it a bit odd that they had originally said "prepended between 0 and 7,618 other email addresses to the body of the email.", as this way it would be just a lot of "To" headers.
- andrewstuart2 10y agoThe design of MIMEMultipart causes duplicate key: values to be printed, rather than comma-separated addresses as specified in RFC 2822. [1] https://tools.ietf.org/html/rfc2822#section-3.6.3 https://tools.ietf.org/html/rfc2822#section-3.6.3
- anderskaseorg 10y agoMultiple occurrences of the To, Cc, and Bcc fields are permitted (though obsolete). https://tools.ietf.org/html/rfc2822#section-4.5.3 https://tools.ietf.org/html/rfc2822#section-4.5.3
- aodin 10y agoI was part of the disclosed list and can confirm that this is exactly what happened (plus an extra newline!)
- chaz6 10y ago
- repox 10y agoI think it's positive that they own up to it and actually apologize. One would also think that most subscribers of this newsletter has a positive attitude towards the general concepts of privacy and security, so I'm also positive in thinking that a list of these disclosed addresses will never see the day of light (hoping I'm not too naive).
- wslh 10y agoSecurity is not just a product.
- joefreeman 10y agoDoes this suggest that the first person got sent 7,618 e-mails?
- 735Tesla 10y agoI received one of these emails (most likely because my address begins with 73 and the emails are sorted alphanumerically). It looks like this: http://pastebin.com/vpPU5sLj http://pastebin.com/vpPU5sLj