12 ms·
Today's Email Incident
- jstanley 14y agoI'm impressed. I thought it much more likely that somebody carelessly pasted a bunch of addresses into a "To" box instead of "Bcc".
- deleted 14y ago[deleted]
- barredo 14y agoyour screenshots are far from un-readable. I'd suggest to remove them. We know there were emails leaked in the 'To' field. We don't need screenshots of them.
- sergiotapia 14y agoThis is bad - real bad. Goes to show you if the 'golden boys' can make mistakes such as these, what about the other 96% of companies handling your info?
- Trufa 14y agoI don't see what's bad about it, well except for the bug itself. Computer software is bound to have bugs, and if you ask me. a company that discloses those bugs, thoroughly explains the cause, addresses how to avoid them the future and apologises to their customers is on the right track. Don't get me wrong, you software should be very stable, but when the inevitable happens, well this is the way you address it.
- deleted 14y ago[deleted]
- markmac 14y agoBad incident, great transparency and handling. Just don't do it again please... :)
- RexM 14y agoWould it make sense to just always put the recipients in the BCC field to ensure you don't accidentally leak email addresses if/when something like this were to happen again?
- swanson 14y ago"putting them in the To: field to enhance deliverability"
- famousactress 14y agoYeah. Is there a strong argument against sending separate emails though? Selfishly curious because that's one of the things we've been doing to prevent this sort of mistake.
- niggler 14y agoWhen they start taxing emails, those separate emails are going to cost a pretty penny :P
- sitharus 14y agoA lot of enterprise spam filters are triggered by a burst of incoming mail from a single sender, sending one email with a bunch of to: is less likely to trigger them. I've had to deal with too many stupid spam filters in my life.
- jleader 14y agoI suspect there are spam-filter considerations. Also, some email clients distinguish between "email addressed directly to me" and "email I happened to get through some mailing list or bcc".
- polemic 14y agoNot really. They thought they were emailing administrators at the one organisation. From the recipients point of view, it would be much easier to deal with if you saw who it was going to. A bunch of BCC recipients within the same organisation is going to lead to confusion.
- deleted 14y ago[deleted]
- pud 14y agoI previously assumed the cause was some marketing intern pasting email addresses into the wrong form. Better training (and a slap on the wrist) would prevent it from happening again. But it's somewhat scarier knowing this was due to a bug triggered by a security patch. Who knows what yet-undiscovered bugs are still in there. That said, there's no such thing as bug-free software and I still find GitHub to be trustworthy.
- 727374 14y agoI don't think anyone doubts github's trustworthiness, but as an enterprise customer, I do wonder if they have tight enough controls. Marketing emails are an area that companies should be spending extra time testing as those screwups tend to be irreversible.
- chadscira 14y agoa little more on the incident http://chadscira.com/post/5148b1f0f93e934b550112be/Github-leaks-1500-emails http://chadscira.com/post/5148b1f0f93e934b550112be/Github-le...
- dbpatterson 14y agoatlassian? :)
- chadscira 14y agoi bet github grinned when they saw that email popup
- jschumacher 14y agoAtlassian builds a JIRA connector for Bitbucket and Github:Enterprise. To test the connector, an installation of Github:Enterprise is required. Internally we use Stash of course ;)
- blarghnox 14y agohttp://pastebin.com/JX1PdiMB http://pastebin.com/JX1PdiMB
- susi22 14y agoYou'd think the .edu addresses would go to bitbucket and get free unlimited hosting.
- amadeuspzs 14y agoNon-technical companies which do this tend not to apologise - it draws attention to the mistake which most of their (non-technical) customers never noticed in the first place. I guess github doesn't qualify for the non-technical crowd :)
- ghuntley 14y agoVery concerning - they accidentally distributed their entire customer list and now I have to handle the internal PR fall out re: my email address and that those of non-technical co-workers (board, accounts, finance, engineering managers) have been distributed globally and then subsequently pastebined. Times like this I'm glad that our code is hosted internally and access to it is controlled internally. https://dl.dropbox.com/u/109905580/Screenshots/qg~abn4fl601.png https://dl.dropbox.com/u/109905580/Screenshots/qg~abn4fl601.... https://dl.dropbox.com/u/109905580/Screenshots/73dw6%7Eq_h4hq.png https://dl.dropbox.com/u/109905580/Screenshots/73dw6%7Eq_h4h...
- eli 14y agoI wouldn't normally consider my email address a secret. I assume the concern is over spear phishing?
- ghuntley 14y agoEmail address isn't secret, the use of certain tools for some however is. Enterprise is a different game than your usual early stage web/product startup. Politics are real, exist and credibility is on the line when you stick your ass out to vouch for a company/product outside of the company standard. Base cost for this software is $5k/yr for one seat pack (20 devs), whereas the competition - Atlasssian is incredibly price competitive (10 users for $10, 25 users for $1,800) - keep in mind competition also exists in the form of TFS (ergh) which is free for MSDN subscribers and included in most enterprise licensing agreements.
- eli 14y agoSorry, still not sure I follow. So the issue is that, for example, the corporate office could discover that a certain department is using what--in their opinion--is an overpriced solution?
- jasonlotito 14y agoAn overpriced insecure solution.
- badgar 14y agoRails security point release breaks apps by changing the ORM. News at 11.
- stanleydrew 14y agoI'm not sure why you found it necessary to post this comment. I found the original article to be a well-written warning about a rather serious bug that might affect a lot of people. But your comment seems to be sarcastically suggesting "move along, nothing to see here."
- mbetter 14y agoI'm not sure why you found it necessary to post that comment, or why I found it necessary to post this one.
- cmccabe 14y agoAnd I can't figure out why either of you two posted. Or why I'm posting. I guess it is what it is.
- jleader 14y agoI don't think the GP is criticizing the article, he's using the article to criticize Rails. Not so much "nothing to see here" as "well, what did you expect from Rails?". (I'm explaining what I think the GP meant, not my position; I'm an old Perl guy, still undecided on the overall trade-offs offered by Rails)
- badgar 14y agoYes, I was pointing out that Rails made backwards-incompatible changes to core functionality in a point release primarily targeting security. I don't know how much more succinctly I could have originally put it, and I thought that it was obvious that this is a Bad Thing. You can't reach everybody.
- 14y ago
- bhauer 14y agoRecognizing that it is not politically correct in these parts to pose this question, I sometimes wonder if the Github team (which seems like a highly-skilled, competent team) regrets building on Rails. It seems they have been able to use their high skill and competence to laboriously steer a bulky and (dare I say it?) heavily-engineered framework to serve Github as the modestly fast and fairly reliable site that it is. But would not they be happier had the efforts exhausted on scale, reliability, and repair been available to spend building features had a faster, less confusing platform and framework been selected? I know, it's antithetical or heretical to imply that Rails is anything but a model of efficiency.
- sitharus 14y agoI've stopped building on Rails because of things like this. Point releases shouldn't break functionality. I got tired of regression testing my apps for framework changes.
- voidlogic 14y agoWhen I read the blog, the first thing I thought was "boy am I glad I don't use rails". I realize all frameworks and toolkits have issues, but Rails seems to be particularly bad. I think the big take away here is the importance of apparently redundant sanity checks....
- ecopoesis 14y agoAnd this is why you shouldn't use ORMs. Magic blackboxes always bite you in the ass eventually.
- drivebyacct2 14y agoYeah, why use SQL or language bindings, just write into the B-trees themselves.
- thrownaway2424 14y agoBecause there's no middle ground between ORMs and packing your own varints into registers.
- drivebyacct2 14y agoI'm not the one that implied all blackboxes are bad.
- AaronBBrown 14y agohttps://yourlogicalfallacyis.com/slippery-slope https://yourlogicalfallacyis.com/slippery-slope
- drivebyacct2 14y agoI'm so excited to see this posted everywhere. It will be just like reddit where everyone makes a loose connection to what they think is a logical fallacy and then just shouts that instead of taking time to actually put thought into a post. And I can just imagine the smugness that goes along with posting that. He implied black boxes are bad. ORMs are black boxes, so are dozens of things that programmers use every day. I understand some of the arguments about why ORMs are bad, but it hardly stems purely from them being a black box. BTW: My post is not an example of a slippery slope fallacy.
- tptacek 14y agoYes I would definitely trust an application much more if each and every one of its SQL queries was lovingly hand crafted by a human being. Because if there's one thing computers are truly bad at, it's transforming high-level things into lower-level things.
- chj 14y agoFrameworks are innocent.
- modarts 14y agoSeing as how this list of customers are about to run out of their trial for Github Enterprise, it'd be funny if atlassian or any other hosted Git providers were to make use these emails for their own gain.
- erikpukinskis 14y agoThis is potentially a pretty big security hole for apps, right? For example: posts = current_user ? current_user.posts : Post.public posts = posts.where(:user_id => params[:user_id]) if params[:user_id] In the old Rails, that code is guaranteed not to show any private posts. But with the new Rails it'll show you everything written by a user. Granted, it's pretty awkward code, but I can imagine situations where people might be inclined to write something like that.
- pjungwir 14y agoI think that sort of code is extremely common. It's a bit less awkward if `current_user` is never nil, but sometimes a non-saved Guest record, letting you write code like this: @posts = current_user.posts @posts = @posts.where(user_id: params[:user_id]) if params[:user_id]
- swampthing 14y agoIn case anyone's interested, I think this is a Github Issue on the same bug: https://github.com/rails/rails/issues/9813 https://github.com/rails/rails/issues/9813
- pjungwir 14y agoThank you for posting that (one of the only informative contributions on this whole thread). That issue description is worrisome because the problem actually isn't limited to `default_scope`, which many people avoid, but also affects just `scope`.
- IgorPartola 14y agoHow would one unit-test an app like this, where you have to talk to an external database and over SMTP at a specific time of day? Seems like in a case like this thorough unit testing needs to be very extensive to catch a bug like this, yet it would have been worthwhile.
- boyter 14y agoYou would only need to unit test the programmer written ORM code against a seeded database to be sure you are getting just the data you expected. You probably wont catch new issues with the ORM (which you should not be testing) but you will ensure you wont put this bug back into production.
- losvedir 14y agoWait, so what exactly is the situation that's at issue here? We upgraded yesterday, too, and have both scopes and class methods that we append onto has_many/belongs_to relations, but haven't experienced anything out of the ordinary. Now I'm terrified we have some unexpected behavior lurking somewhere that's not tested or something. > In this case, when using the scope method to define an Arel scope on Organization, the where clause of the scope is overriding the condition imposed by the Organization#teams association. The part of the WHERE clause meant to restrict the query to Team records related to the Acme organization was dropped. This doesn't happen in all cases -- all the cases where I've done this appears to be fine. And a scope just blowing away an association in all cases would be too obvious to not be caught. Is it because both the scope and the relation select by the same attribute? Does that attribute have to be `id` for this bug to occur? Or does it have to do with the fact that they're nesting a query within the where clause to determine the IDs? (I don't see why this would matter). I guess I'm off to a REPL unless anyone here knows more concretely what the issue is. EDIT TO ADD: I can't reproduce this behavior. I just spun up a new Rails 3.2.13 app, and created those two models. Going through what they do in the console, my results differ on the potentially dangerous part: 1.9.3-p0 :015 > teams = acme.teams.using_octocats_scope Team Load (0.3ms) SELECT "teams".* FROM "teams" WHERE "teams"."organization_id" = 2 AND "teams"."organization_id" IN (SELECT id FROM "organizations" WHERE "organizations"."has_octocats" = 't') => [] I do get an empty array, as is supposed to be the case. I notice that my nested select remains a SQL statement, rather than being evaluated to '(1)' as in their example. So... I don't know if I'm doing something wrong, or if GitHub has other code / configuration settings interacting strangely here. Can anyone else reproduce? EDIT AGAIN: Here are the four files you need to reproduce yourself. https://gist.github.com/losvedir/5202121 https://gist.github.com/losvedir/5202121 ONE MORE EDIT: https://github.com/losvedir/test_github_thing https://github.com/losvedir/test_github_thing There's a repo with my history in the console. Dunno why I'm seeing the behavior I am.
- pjungwir 14y agoI can reproduce it. I'm very worried also because I love using scopes. Here is a repo that shows the problem: https://github.com/pjungwir/scope-error I think it is actually a mite simpler than the gist in your "EDIT AGAIN."
- pjungwir 14y agoI whipped up a quick example Rails project to demonstrate the issue. It's a lot like the Github post, but a little simpler: for instance, there's no real reason to talk about class methods: https://github.com/pjungwir/scope-error https://github.com/pjungwir/scope-error I'm pretty concerned about this bug, because scopes are one of my favorite Rails features.
- teyc 14y agoA modern mail server should have features to prevent too many users from being present in the To: field. This will also prevent a lot of the Reply All problems we see regularly. This can be implemented as part of a Quota management system.
- vinhboy 14y agoWow, I had to scroll a long long long way to finally get a reasonable suggestion on how to fix problems like this. A problem that potentially EVERY software has, regardless of framework or language. Kudos to you.
- pilif 14y agoExim does this since forever (http://www.exim.org/exim-html-current/doc/html/spec_html/ch-main_configuration.html http://www.exim.org/exim-html-current/doc/html/spec_html/ch-... - look for max_recipients). I would assume other MTAs do so too. Problem is, options like these feel nice until they annoy you because you do in fact need more recipients. At that point you build a workaround into your software to send a new mail for every max_recipient users which brings you right back to square one with the additional technical dept caused by potential bugs in your batch sending code (off-by-one errors for example) Be careful what you wish for :)
- teyc 14y agoJust like one can have safe_parse() and unsafe_parse(), there could be an elevated API that could be safely used when vetted. For instance, when the SMTP server rejects a request it could put it in a special queue which requires manual confirmation. This type of problems happen so regularly. In fact, a Japanese trader accidentally punched one too many zeros and lost a lot of money for the company. This type of problems can get very expensive in an instant. As an industry, we tend to think of input validation in passing because we don't pay enough attention to the problem domain.
- Nazzareno 14y agoSending emails always requires more attention than expected. Everyone, even Apple in 2007, made some severe mistakes in delivering emails. By using a professional SMTP relay (disclaimer: we provide SMTP+) this wouldn't happen since, for example, every A:/CC:/CCN: recipient is splitted from the original message by default.
- isarat 14y agoThe excuses are on an on. I admire Github guys. They give a super cool impression to the other outside world and they're doing a great job. But the recent outages and the problems are giving high attention to Github. It's giving negative sign for the people who are interested go for Github way of hosting code. Either their own premise or on github. giving detailed analysis is a good tech thing... it fires developers heart and we are still going admire what they did to resolve this issue but definitely this isn't a good sign for github.
- robertsosinski 14y agoFor quite some time, I have been using Sequel (http://sequel.rubyforge.org http://sequel.rubyforge.org) as an ORM with Postgres instead of ActiveRecord, and have never been happier. Overall, Rails has been a great web-framework, but ActiveRecord has been it's sore spot, at least for me. Also, using the sequel_pg (https://github.com/jeremyevans/sequel_pg https://github.com/jeremyevans/sequel_pg) on Postgres ensures that even raw SQL queries are type-casted properly, plus it supports streaming and cursors, while also being very very quick. If you want to stick with Ruby/Rails and relational databases, but think ActiveRecord might not be your bag, give Sequel a shot. It also fits nicely into lighter frameworks such as Sinatra.
- jrochkind1 14y agoI'm trying to figure out how likely my apps are to be effected by this same bug/behavior change as Github's, and if I should delay installing the security update without more investigation. Not a great situation to be in with a security update.