9 ms·
Visual Studio Code 1.7 overloaded npmjs.org, release reverted
- sync 10y agoShouldn't all these requests be cached by a CDN? What exactly is overloading?
- seldo 10y agoCDNs don't usually cache 404s. VSCode was looking for @types packages for any and every npm package its users were using. Packages that had a type description caused no issue, but most packages don't, so we had a > 1000% spike in 404s. Our workaround before MS did the rollback was to cache 404s for @types packages specifically, and it was effective enough that the registry never really went down.
- akfish 10y agoInteresting. Thanks for sharing this information.
- natuac 10y ago"a > 1000% spike in 404s" overloaded your servers? Such are your generation times? Can I bring the entire NPM ecosystem down from my ADSL line using some silly threaded code to make requests to randomly named packages?
- seldo 10y ago99.9% of our requests are handled by the CDN. The CDN doesn't cache 404s, so 404s are handled by our origin servers, which are much fewer in number and therefore quite easy to overwhelm. You're right that our handling of 404s was naive, and that's definitely something we'll be improving as a result of what we've learned from this incident.
- tankenmate 10y agoIt's a pity, DNS handles negative lookup caching / TTLs (in fact that is exactly what the TTL field in DNS zones are!). So negative lookups can be cached, but you need to do thinking about it ahead of time and set sensible TTLs (preferably configurable) for those negative caches.
- hiou 10y ago> The feature was so great that we started to overload the npmjs.org service. I'm not sure I would call my feature "great" if it could have brought down npm.
- cerebellum42 10y agoI thought that sentence sounded very trump-ish. The feature was so great that npmjs couldn't keep up with it, it was yuuuuuge!
- jessaustin 10y ago...and they made npm Inc. pay for it!
- Aldo_MX 10y agoit was a tremendous overload
- seldo 10y agoI'd just like to say on behalf of npm that Microsoft's handling of this incident was A+. As soon as we alerted them to the issue they were all hands on deck and did a rollback. We've been really pleased that Microsoft chose to put their @types packages into the npm registry rather than a separate, closed system, and in general happy with Microsoft's support of node and npm. We're confident we can make the new features of VSCode work, we just need to work with Microsoft to tweak the implementation a little. This was an honest mistake on their part, and we caught it in time that there was very little impact visible to any npm users. Fun fact: at its peak, VSCode users around the world were sending roughly as many requests to the registry as the entire nation of India.
- paulftw 10y agoSo one day they switched their entire user base to rely on a 3rd party free service without any load testing or heads up? What could possibly go wrong?
- foota 10y agoFwiw if I were building a feature on something that's considered as core a technology as npm is, I likely would not have thought of this either. (Though maybe I would have if I were doing an in depth look into it)
- mohamedhegazy 10y agoWe have been testing this on insider builds of vscode for a few weeks as well as preview builds of visual studio with no issues. We were just notified today by npm that we are flooding their servers.
- ceejbot 10y agoYour installed base is quite large indeed! Your testing load was a drop in the bucket of our daily usage, but once you released to VS users we noticed. Should be straightforward to design something that works for this access pattern and load, now we know what you need. Typeahead package name completion would be a neat feature.
- andyburke 10y agooops
- Arnavion 10y agohttps://github.com/Microsoft/vscode/issues/14889 https://github.com/Microsoft/vscode/issues/14889
- meira 10y agoDamn it, M$! Keep your shit closed. Not everybody wants you messing around.
- deleted 10y ago[deleted]
- sctb 10y agoWe've asked you twice before not to do this, so we have to ban this account. We're happy to unban accounts if you email us at hn@ycombinator.com and we believe you'll not do this in the future.
- deleted 10y ago[deleted]
- akfish 10y agoWhich is more possible? A bug or they just underestimated the volume of traffic that could be caused by ATA in real life?
- hyperliner 10y agoThe latter.
- mavsman 10y agoIf I were on the Azure team I'd be offering tons of free credit to npmjs.org to get them to use Azure. Azure coming to the rescue would be the perfect ending to this story for Microsoft.
- CaveTech 10y agoI don't think most organizations could relaunch their infrastructure on a totally different stack at the drop of a dime. And if it was really a "throw more servers at it" problem then it wouldn't really matter who was hosting them, would it?
- azinman2 10y agoDepends on who's paying
- KayL 10y agoCDN caching globally, please!
- BenjaminCoe 10y agoAs one of the folks on the front-lines helping patch this, I certainly have no hard feelings; and I'm excited to be able to support this feature properly ... also ... not going to lie, this was the first time we've gotten to test several of the checks and balances we have in the npm registry which I was jazzed about :)
- raisedadead 10y agoThanks, Benjamin, Laurie and everyone else for mitigating this, it feels great to know when the community chimes in together for such highly unanticipated scenarios. On that note, however, respectfully I believe that features which have the potential of hitting the registry so bad should first be beta tested on a private registry and moved on to the high traffic serving CDNs of npm. And 10% of the daily traffic is from India??? Whoa, every day is a school day.
- poizan42 10y ago> And 10% of the daily traffic is from India??? Whoa, every day is a school day. Well, 17% of the world's population lives in India, so doesn't seems surprising.
- raisedadead 10y agolol. agreed.
- PudgePacket 10y agoI wonder if any warning was given to npm that they would be getting this potentially huge new source of traffic. It doesn't seem to be mentioned anywhere.
- vonklaus 10y agoEh, NPM is a pretty core service and both sides probably should have done things a bit differently. I don't neccessarily think vscode needed to reach out to NPM to let them know they were going to be consuming their public API. Both teams appear to be in communication as a result however-- which is good. This will likely lead to more fault tolerant systems on both projects and hopefully more collaboration & features in the future.
- CapacitorSet 10y ago>I don't neccessarily think vscode needed to reach out to NPM to let them know they were going to be consuming their public API. VSCode is used by a non-negligible number of users, and seems to rely on npm to operate at its best. It would have been good etiquette to let npm know, even though they couldn't forecast this exact situation.
- vonklaus 10y agoI am not the architect of any large scale system-- that said, I wouldn't expect developers to reach out to GitHub. However, it isn't bad etiquette and I'm sure Microsoft could get in touch with the devs. Interesting thought.
- deleted 10y ago[deleted]
- manojlds 10y agoWould yarn help here? (since FB have their own CDN and registry for it?)
- eugeneionesco 10y agoThey do? yarn uses the npm registry not something else.
- PudgePacket 10y agoIt does, but it also goes though cloudflare as far as I know (which does caching).
- ohitsdom 10y agoThis issue though was with excessive 404s, which aren't cached.
- manojlds 10y agoNo, they mirror the npm registry.
- gremlinsinc 10y agoI'd love to use VSCode but can't until they or someone else rolls out a dockblockr extension that works for php as I'm mostly tied to Laravel right now, and my company requires docblocks and they are not fun to write by hand.
- winsome 10y agoThey have a great extension ecosystem. Why not give writing the extensions a shot yourself?
- gremlinsinc 10y agoI've never really written much software or extension type things... I guess I could take a look at the code and compare it with DockBlockr on Sublime. I'm more of a web app guy.
- smarx007 10y agohttps://github.com/jsdoc3/jsdoc https://github.com/jsdoc3/jsdoc maybe? http://usejsdoc.org/about-getting-started.html http://usejsdoc.org/about-getting-started.html
- z3t4 10y agoThey are probably trolling for a debate about NPM ... I can smell politics.
- markatkinson 10y agoIt is a real foreign feeling being exposed to such an actively and well run project. Every time I see a new release on HN I get a little "wow, that time of the month again." Even this rollback was indicative of how fast they move.
- antrion 10y agoThis is a really cool feature! Is there a similar extension for Atom / Sublime?
- eugeneionesco 10y agoNot really, they don't have the user number for this.
- _pmf_ 10y agoBut they told us that pulling in 1.6 GB for "Hello World" is normal and no big deal.