7 ms·

That took my server down i'm afraid. Good day today. Here's a "cached" version. To all watchers of the libgdx repository: i’m terribly sorry and hope i didn’t

That took my server down i'm afraid. Good day today. Here's a "cached" version. To all watchers of the libgdx repository: i’m terribly sorry and hope i didn’t interfer with your work in any way This is meant as a cautionary tale about using Github’s API on a repository with quite a few watchers (460 in this case). Earlier this year we migrated our code from Google Code to Github. We didn’t have a good migration plan for the 1200 or so issues back then, so we kept them on Google Code. We now have about 1700 issues on the tracker Today i finally wanted to tackle the issue tracker migration, using a Python script [1] i found on Github. The script requires one to specify a Github user account that owns the repository the issues will get migrated to. I did a dry run on a fork of the main repo using my Github account, fixed up some issues in the script, and validated things to the best of my abilities. Things looked good. Then i ran it on the main repository. Luckily i was watching our IRC channel. After about 4 minutes, people started to scream. They each received 789 e-mails from Github. Every single issue i migrated, and every single comment of each issue triggered an e-mail notification to all watchers of the main repository. This wasn’t apparent to me during the dry runs, as i used my own Github account. The script posts all issues/comments with the user account i supplied, so naturally, i did not get any notification mails. I stopped the script after 130 issues (4 minutes), and immediately started sending out apologies and a mail to Github support, to which i haven’t received an answer yet. I send roughly 300k mails through their servers in a matter of minutes. If i hadn’t watched IRC, i’d have send out about 4 million mails to 460 people within an hour. Let me assure you that i’m extremely sorry about this incident. I know that things like this can interrupt daily workflows quite a bit, even if getting rid of those mails is not a Herculean task. I’d be rather upset if a repo maintainer pulled something like this on me. Please accept my deepest apologies. The lesson for Github API users: think hard about the implications of automating tasks through the Github API if you have more than a few watchers. The lesson for Github/API designers: consider safe-guarding against such issues in your API, in case other idiots like me pull off something similar in the future. [1] https://github.com/tgoyne/google-code-issues-migrator https://github.com/tgoyne/google-code-issues-migrator

glimcat 13y ago
While you may be the "visible culprit" - I'd hardly say you're at fault. The action you're attempting is not a "send email" action, even if the API provider has decided that it should send email notifications. The implementation you'd probably want to see is for the API's email notifications to be batched if more than small-n trigger within a short period of time. Then the end user gets a notification that "1700 updates have occured" instead of getting 1700 emails. The API should also be set up with exceptional event detection. Spikes which are several orders of magnitude above normal should get paused in the task queue and flagged for immediate manual review. Users do things you don't expect.
- badlogic 13y ago
  That's indeed a good solution. Here's hoping that Github might adapt something like that in the near future. Manual repo maintainance is quite a burden, so making it harder to shoot yourself in the foot using the API would be much appreciated.
  - borisjabes 13y ago
    We hit this situation a while back on our product and implemented batching as a result (http://blog.meldium.com/home/2013/4/22/dont-spam-your-users-batch-notifications-in-rails http://blog.meldium.com/home/2013/4/22/dont-spam-your-users-...)
txutxu 13y ago
Once upon a time, (about 5years ago) a 256M ram "server" holding a wordpress-mu (mu as in multi-user). There was "agile" methodologies, but management did fuck up with the splits, and there were multiple sysadmins for the same systems working "with their team". I was called in by "performance issues", I was working in other "stuff". There was 47.000 mysql tables. There was triplicated "dumps" from cron. There was no "renice" or "ionice" involved in any of them. There was no ntp, the VM time was with a 7 hours time offset. MySQL was with the "distro defaults" :.( etc ... In resume: human fails happen. You where smart enough to realize and stop it. That puts you over the average.
eksith 13y ago
Ouch. Well, I'm not sure if you're at fault here as much as the Github API itself that bears a lot of the blame (I don't understand why kitchen sink notifications are really necessary for everything). Throttling the event rates or a queue would help in the future. I'm glad this can serve a learning experience and no actual damage was done. Sweeping email, while annoying and little bit disruptive, isn't the end of the world and those who choose to understand, will do so.
driverdan 13y ago
Install W3TC[1], enable all the caches. Test to make sure it doesn't screw up your CSS and JS. Problem solved. [1]: http://wordpress.org/plugins/w3-total-cache/ http://wordpress.org/plugins/w3-total-cache/
- badlogic 13y ago
  Wow, thanks for the tip!
- spion 13y ago
  Why don't they make this the default? Almost every time I see this happening to a blog, its a Wordpress instance...
aidos 13y ago
Wow, bummer. I'm sure the others will understand though - it's really not your fault. Some years ago the company I was at had a policy of sending an email on every error - to every developer, so nobody actually checked them. One day the 3 main servers went down, all at the same time. Transpired that a) someone had introduced a bug into the code that completely 500'd a site b) it got through QA on to live and c) there was a relay chain for the email servers (I have no idea why). By the time we'd worked out what was going on the hard drives on all the machines had reached capacity and all the live systems were down. Of course the timeouts that started happening as the load came on just cascaded the issue. Gotta watch those automated emails :)
- rdixit 13y ago
  This. Is an amazing story.

glimcat 13y ago

While you may be the "visible culprit" - I'd hardly say you're at fault. The action you're attempting is not a "send email" action, even if the API provider has decided that it should send email notifications. The implementation you'd probably want to see is for the API's email notifications to be batched if more than small-n trigger within a short period of time. Then the end user gets a notification that "1700 updates have occured" instead of getting 1700 emails. The API should also be set up with exceptional event detection. Spikes which are several orders of magnitude above normal should get paused in the task queue and flagged for immediate manual review. Users do things you don't expect.

badlogic 13y ago
That's indeed a good solution. Here's hoping that Github might adapt something like that in the near future. Manual repo maintainance is quite a burden, so making it harder to shoot yourself in the foot using the API would be much appreciated.
- borisjabes 13y ago
  We hit this situation a while back on our product and implemented batching as a result (http://blog.meldium.com/home/2013/4/22/dont-spam-your-users-batch-notifications-in-rails http://blog.meldium.com/home/2013/4/22/dont-spam-your-users-...)

txutxu 13y ago

Once upon a time, (about 5years ago) a 256M ram "server" holding a wordpress-mu (mu as in multi-user). There was "agile" methodologies, but management did fuck up with the splits, and there were multiple sysadmins for the same systems working "with their team". I was called in by "performance issues", I was working in other "stuff". There was 47.000 mysql tables. There was triplicated "dumps" from cron. There was no "renice" or "ionice" involved in any of them. There was no ntp, the VM time was with a 7 hours time offset. MySQL was with the "distro defaults" :.( etc ... In resume: human fails happen. You where smart enough to realize and stop it. That puts you over the average.

eksith 13y ago

Ouch. Well, I'm not sure if you're at fault here as much as the Github API itself that bears a lot of the blame (I don't understand why kitchen sink notifications are really necessary for everything). Throttling the event rates or a queue would help in the future. I'm glad this can serve a learning experience and no actual damage was done. Sweeping email, while annoying and little bit disruptive, isn't the end of the world and those who choose to understand, will do so.

driverdan 13y ago

Install W3TC[1], enable all the caches. Test to make sure it doesn't screw up your CSS and JS. Problem solved. [1]: http://wordpress.org/plugins/w3-total-cache/ http://wordpress.org/plugins/w3-total-cache/

badlogic 13y ago
Wow, thanks for the tip!
spion 13y ago
Why don't they make this the default? Almost every time I see this happening to a blog, its a Wordpress instance...

aidos 13y ago

Wow, bummer. I'm sure the others will understand though - it's really not your fault. Some years ago the company I was at had a policy of sending an email on every error - to every developer, so nobody actually checked them. One day the 3 main servers went down, all at the same time. Transpired that a) someone had introduced a bug into the code that completely 500'd a site b) it got through QA on to live and c) there was a relay chain for the email servers (I have no idea why). By the time we'd worked out what was going on the hard drives on all the machines had reached capacity and all the live systems were down. Of course the timeouts that started happening as the load came on just cascaded the issue. Gotta watch those automated emails :)

rdixit 13y ago
This. Is an amazing story.