6 ms·
I'd just like to say on behalf of npm that Microsoft's handling of this incident was A+. As soon as we alerted them to the issue they were all hands on deck and
by seldo 10y ago
I'd just like to say on behalf of npm that Microsoft's handling of this incident was A+. As soon as we alerted them to the issue they were all hands on deck and did a rollback.
We've been really pleased that Microsoft chose to put their @types packages into the npm registry rather than a separate, closed system, and in general happy with Microsoft's support of node and npm. We're confident we can make the new features of VSCode work, we just need to work with Microsoft to tweak the implementation a little.
This was an honest mistake on their part, and we caught it in time that there was very little impact visible to any npm users.
Fun fact: at its peak, VSCode users around the world were sending roughly as many requests to the registry as the entire nation of India.
- paulftw 10y agoSo one day they switched their entire user base to rely on a 3rd party free service without any load testing or heads up? What could possibly go wrong?
- foota 10y agoFwiw if I were building a feature on something that's considered as core a technology as npm is, I likely would not have thought of this either. (Though maybe I would have if I were doing an in depth look into it)
- mohamedhegazy 10y agoWe have been testing this on insider builds of vscode for a few weeks as well as preview builds of visual studio with no issues. We were just notified today by npm that we are flooding their servers.
- ceejbot 10y agoYour installed base is quite large indeed! Your testing load was a drop in the bucket of our daily usage, but once you released to VS users we noticed. Should be straightforward to design something that works for this access pattern and load, now we know what you need. Typeahead package name completion would be a neat feature.
- paulftw 10y agoBefore you start sending couple thousand QPS to any server it's generally not a bad idea to test if that server can handle that much, sometimes it is even worthwhile notifying the other team about the intended change. Overall when you "tested" something, but it still breaks in production and requires a rollback it's usually a sign that your testing strategy isn't could use some improvement - what is the point of testing if it doesn't prevent failures from happening
- Arnavion 10y agoCan you elaborate on what the issue is and how you want it to be fixed? Is it just something like rate-limiting requests or something more fundamental? Edit: Answered at https://news.ycombinator.com/item?id=12861118 https://news.ycombinator.com/item?id=12861118
- seldo 10y agoA VSCode person can (and probably will) answer in more detail, but at heart it's simple: if you want to add type-checking goodness to a library that isn't itself written in TypeScript, you can create a thing called a declaration file: https://github.com/DefinitelyTyped/DefinitelyTyped https://github.com/DefinitelyTyped/DefinitelyTyped Microsoft publishes a list of known good declaration files for popular npm packages to npm, under the scope @types: https://www.npmjs.com/~types https://www.npmjs.com/~types The 1.7 release of VSCode helpfully tries to automatically load type declarations for any npm package you use by requesting the equivalent declaration package under @types. When the package exists this is fine, because it's cached in our CDN. What they forgot to consider is that most CDNs don't cache 404 responses, and since there are 350,000 packages and less than 5000 type declarations, the overwhelming majority of requests from VSCode to the registry were 404s. This hammered the hell out of our servers until we put caching in place for 404s under the @types scope. We didn't start caching 404s for every package, and don't plan to, because that creates annoying race conditions for fresh publishes, which is why most CDNs don't cache 404s in the first place. There are any number of ways to fix this, and we'll work with Microsoft to find the best one, but fundamentally you just need a more network-efficient way of finding out which type declarations exist. At the moment there are few enough that they could fetch a list of all of them and cache it (the public registry lacks a documented API for doing that right now, but we can certainly provide one).
- mappu 10y ago> most CDNs don't cache 404s Sounds like a good CDN-busting DDoS vector.
- chrismorgan 10y ago
- ec109685 10y ago"Many requests to the registry as the entire nation of India" per what time unit?
- oridecon 10y agoApproximately 3 new JS frameworks per hour.
- jug 10y agoFunny. Sounds like the expansion rate of the Javascript ecosystem.
- young_greedo 10y agoIt's not a real HN thread until someone makes this joke...
- seldo 10y agoI was a bit vague :-) India's about 10% of total requests on any given day. VSCode was 10% of requests for a couple of hours.
- lmm 10y agoHow does the time unit make a difference? If they're making the same number of requests/second as India then they're also making the same number of requests/hour or requests/day, no?
- ec109685 10y agoThere wasn't any time unit in the original statement, so it wasn't clear that the request per X from India was the same as the request per X from the IDE.
- pdpi 10y agoDoesn't matter, it's a comparison of request rates. Requests over time where user in India ~= requests over time where user is a vs code user
- thedaniel 10y ago> This was an honest mistake on their part From my outside perspective, it doesn't seem like a mistake on their part at all. Later in the thread you say this accounted for 10% of traffic, mostly 404s. This is (i assume) a hell of a lot of requests, but given npm's position as developer infrastructure, I don't think they could have reasonably expected to melt it. It would have been good of them to give a heads up, but I don't think I'd start assigning blame to the Code team.
- mjpa 10y agoSounds like you're describing a mistake there...
- jfindley 10y agoYeah. It leaves an unpleasant taste in my mouth to hear npm blaming Microsoft for this. As noted elsewhere, 404's are supposed to be very cheap to handle, otherwise DoS attacks become embarrassingly easy. I feel like the npm team have once again failed to own their problems and instead tried to push the blame elsewhere. This is just an outside perspective, but I really feel like it would have been more honest and accurate to at least admit to the possibility that npm isn't perfect, and "blame" (which I'm not sure is even a helpful concept in this instance) is shared between parties more equitably.
- thedaniel 10y agoIndeed: https://whoownsyouravailability.com https://whoownsyouravailability.com
- seldo 10y agoI'm sorry my response looked like I was blaming them, that wasn't my intention. Like I said, it was an honest mistake: these things happen, and they handled it well. Once we determined 404s were the problem we put mitigation in place that worked fine, but the problem of request volume remained: the 10% figure I gave was at a 5% rollout of VSCode. A full rollout would therefore have meant the registry became 3x bigger overnight and two thirds of that would have been 404s to VSCode users. At that point the issue is financial, not technical, which is another reason the rollback happened.
- Fifer82 10y agoAwesome for sharing your thoughts. Don't mind the children here. You could give them gold and they would moan about the purity.