8 ms·

Yarn registry was down

67 points by seancoleman 8y ago

schneidmaster 8y ago
Issue is here: https://github.com/yarnpkg/yarn/issues/5885 https://github.com/yarnpkg/yarn/issues/5885 The problem is that npm just migrated their registry to use CloudFlare. Yarn already used CloudFlare, and had https://registry.yarnpkg.com https://registry.yarnpkg.com CNAME'd to https://registry.npmjs.org https://registry.npmjs.org. CloudFlare has a security restriction that one CloudFlare account cannot have DNS records pointing to another CloudFlare account, so npm moving to CloudFlare broke yarn's existing CNAME to npm's registry (thus breaking yarn).
- jessaustin 8y ago
  Sorry but this is kind of funny. It would have been easy for the default value in the app to have been "https://registry.npmjs.org" https://registry.npmjs.org". This is an example of "every level of indirection can fail". [EDIT: removed unfounded speculation that CF wasn't necessary for yarn's requirements.]
  - foobarbazetc 8y ago
    CF DNS is pretty good for a free service. Lots of annoyances to be sure but saying they’re not focused on it is kind of wrong. This is actually an issue with the CDN portion not the DNS portion.
  - judge2020 8y ago
    Based on the (recent) comment on the issue [0], they use the Cloudflare proxy instead of an actual CNAME /and use workers to do yarn-specific changes on the edge / (see below). [0]: https://github.com/yarnpkg/yarn/issues/5885#issuecomment-392215450 https://github.com/yarnpkg/yarn/issues/5885#issuecomment-392...
    - deleted 8y ago
      [deleted]
    - deleted 8y ago
      [deleted]
    - arcatek 8y ago
      I work on Yarn. The message you're quoting is from the npm CTO. We don't do yarn-specific changes. The main reason we're using a mirror is because it was massively faster than directly querying the npm registry when Yarn got released. The difference decreased during the past year I think, but Cloudflare was still a bit ahead in most scenarios.
- foobarbazetc 8y ago
  You can CNAME across CF accounts but you can’t CNAME a proxied record to another proxied record.
- ryanworl 8y ago
  Fun fact: you actually used to be able to do this and it caused some security problems! There are probably zones out there that are still able to do it today from back when it was allowed.
  - orf 8y ago
    What security problems?
    - asadlionpk 8y ago
      I wonder that too. Circular cnames?
tuananh 8y ago
depending on npm is scary >.<
- mastahmind 8y ago
  well, you can look at it at both ways, if yarn just used npm's registry then there wouldn't be a problem. I am not sure what's their reason to add a registry (a mirror at that) if they don't plan to make their own.
  - jessaustin 8y ago
    ISTM this was a marketing ploy for yarn. If CF could proxy npm's registry faster than npm could serve it directly, yarn could claim to be faster than npm. Even if the apps themselves had been identical, network speed is still speed. As we could expect for any competitive advantage that is purchased by one party, it has now been purchased by another party, so it ceases to be an advantage at all. The Red Queen lives!
    - andrewshifflett 8y ago
      CJ (CTO at npm) explained on Twitter why they switched to Cloudflare: https://twitter.com/ceejbot/status/1000230136900239360 https://twitter.com/ceejbot/status/1000230136900239360 (See: https://developers.cloudflare.com/workers/about/ https://developers.cloudflare.com/workers/about/)
  - asadlionpk 8y ago
    Maybe future proofing? They can always make their own and just point CNAME to the new registry?
- moojd 8y ago
  I think it's probably a good idea to run your own proxy/mirror of whatever package registry your language uses if you heavily depend on it being available. We proxy packagist at work, I'm starting to think we should do the same with npm. Has anyone here had success with any out of the box solutions?
btown 8y ago
And that’s why you never do an infrastructure change EOD Friday before Memorial Day weekend.
- lwhalen 8y ago
  This is why you have a Testing environment you can sandbox this kind of thing out on first ;-) Or, as the old sawhorse goes, "every company has a Testing environment. Some companies are fortunate enough to have a separate production environment as well"
- toomuchtodo 8y ago
  Someone is learning about “read only Fridays” the hard way. Can it wait until the start of the next week? Almost always.
- schneidmaster 8y ago
  True in general, but I don't think this one's that simple. npm made a perfectly reasonable configuration change that incidentally happened to break yarn. Hard to predict since you can't sandbox every internet service that might be unexpectedly depending on you, and arguably not really npm's problem anyway. (Much to their credit, npm jumped on the issue and contacted their CloudFlare rep to help unbreak yarn, but I don't think they're responsible to proactively anticipate things like this.)
- koolba 8y ago
  On the contrary this is exactly why you do it on a holiday weekend. Minimal impact to the world as hopefully your users are yarn installing a beer instead of a package.
  - ceejbot 8y ago
    This was our plan: prod load, but not super load, not a lot of work that we'd disrupt with bug shakeout. I don't have a good way to simulate production load on our staging environment right now (some day) and that wouldn't have shaken this one out anyway.
spdustin 8y ago
"We (aka npm) are in contact with Cloudflare to get this third-party CNAME allowed. I'll update this thread when I hear back." (via [0]) More status on the Github issue [0] than yarn status page [1] [0]: https://github.com/yarnpkg/yarn/issues/5885#issuecomment-392212653 https://github.com/yarnpkg/yarn/issues/5885#issuecomment-392... [1]: https://status.yarnpkg.com/ https://status.yarnpkg.com/
deleted 8y ago
[deleted]
jwalton 8y ago
The official NPM registry isn't fairing well, either: https://github.com/npm/npm/issues/20766 https://github.com/npm/npm/issues/20766
PudgePacket 8y ago
They had other issues earlier this week too. Doesn't help that their status page hasn't been updated in a year :D https://twitter.com/trentbillington/status/998822168036167680 https://twitter.com/trentbillington/status/99882216803616768....

67 points by seancoleman 8y ago

Issue is here: https://github.com/yarnpkg/yarn/issues/5885 https://github.com/yarnpkg/yarn/issues/5885 The problem is that npm just migrated their registry to use CloudFlare. Yarn already used CloudFlare, and had https://registry.yarnpkg.com https://registry.yarnpkg.com CNAME'd to https://registry.npmjs.org https://registry.npmjs.org. CloudFlare has a security restriction that one CloudFlare account cannot have DNS records pointing to another CloudFlare account, so npm moving to CloudFlare broke yarn's existing CNAME to npm's registry (thus breaking yarn).

jessaustin 8y ago
Sorry but this is kind of funny. It would have been easy for the default value in the app to have been "https://registry.npmjs.org" https://registry.npmjs.org". This is an example of "every level of indirection can fail". [EDIT: removed unfounded speculation that CF wasn't necessary for yarn's requirements.]
- foobarbazetc 8y ago
  CF DNS is pretty good for a free service. Lots of annoyances to be sure but saying they’re not focused on it is kind of wrong. This is actually an issue with the CDN portion not the DNS portion.
- judge2020 8y ago
  Based on the (recent) comment on the issue [0], they use the Cloudflare proxy instead of an actual CNAME /and use workers to do yarn-specific changes on the edge / (see below). [0]: https://github.com/yarnpkg/yarn/issues/5885#issuecomment-392215450 https://github.com/yarnpkg/yarn/issues/5885#issuecomment-392...
  - deleted 8y ago
    [deleted]
  - deleted 8y ago
    [deleted]
  - arcatek 8y ago
    I work on Yarn. The message you're quoting is from the npm CTO. We don't do yarn-specific changes. The main reason we're using a mirror is because it was massively faster than directly querying the npm registry when Yarn got released. The difference decreased during the past year I think, but Cloudflare was still a bit ahead in most scenarios.
foobarbazetc 8y ago
You can CNAME across CF accounts but you can’t CNAME a proxied record to another proxied record.
ryanworl 8y ago
Fun fact: you actually used to be able to do this and it caused some security problems! There are probably zones out there that are still able to do it today from back when it was allowed.
- orf 8y ago
  What security problems?
  - asadlionpk 8y ago
    I wonder that too. Circular cnames?

tuananh 8y ago

depending on npm is scary >.<

mastahmind 8y ago
well, you can look at it at both ways, if yarn just used npm's registry then there wouldn't be a problem. I am not sure what's their reason to add a registry (a mirror at that) if they don't plan to make their own.
- jessaustin 8y ago
  ISTM this was a marketing ploy for yarn. If CF could proxy npm's registry faster than npm could serve it directly, yarn could claim to be faster than npm. Even if the apps themselves had been identical, network speed is still speed. As we could expect for any competitive advantage that is purchased by one party, it has now been purchased by another party, so it ceases to be an advantage at all. The Red Queen lives!
  - andrewshifflett 8y ago
    CJ (CTO at npm) explained on Twitter why they switched to Cloudflare: https://twitter.com/ceejbot/status/1000230136900239360 https://twitter.com/ceejbot/status/1000230136900239360 (See: https://developers.cloudflare.com/workers/about/ https://developers.cloudflare.com/workers/about/)
- asadlionpk 8y ago
  Maybe future proofing? They can always make their own and just point CNAME to the new registry?
moojd 8y ago
I think it's probably a good idea to run your own proxy/mirror of whatever package registry your language uses if you heavily depend on it being available. We proxy packagist at work, I'm starting to think we should do the same with npm. Has anyone here had success with any out of the box solutions?

btown 8y ago

And that’s why you never do an infrastructure change EOD Friday before Memorial Day weekend.

lwhalen 8y ago
This is why you have a Testing environment you can sandbox this kind of thing out on first ;-) Or, as the old sawhorse goes, "every company has a Testing environment. Some companies are fortunate enough to have a separate production environment as well"
toomuchtodo 8y ago
Someone is learning about “read only Fridays” the hard way. Can it wait until the start of the next week? Almost always.
schneidmaster 8y ago
True in general, but I don't think this one's that simple. npm made a perfectly reasonable configuration change that incidentally happened to break yarn. Hard to predict since you can't sandbox every internet service that might be unexpectedly depending on you, and arguably not really npm's problem anyway. (Much to their credit, npm jumped on the issue and contacted their CloudFlare rep to help unbreak yarn, but I don't think they're responsible to proactively anticipate things like this.)
koolba 8y ago
On the contrary this is exactly why you do it on a holiday weekend. Minimal impact to the world as hopefully your users are yarn installing a beer instead of a package.
- ceejbot 8y ago
  This was our plan: prod load, but not super load, not a lot of work that we'd disrupt with bug shakeout. I don't have a good way to simulate production load on our staging environment right now (some day) and that wouldn't have shaken this one out anyway.

spdustin 8y ago

"We (aka npm) are in contact with Cloudflare to get this third-party CNAME allowed. I'll update this thread when I hear back." (via [0]) More status on the Github issue [0] than yarn status page [1] [0]: https://github.com/yarnpkg/yarn/issues/5885#issuecomment-392212653 https://github.com/yarnpkg/yarn/issues/5885#issuecomment-392... [1]: https://status.yarnpkg.com/ https://status.yarnpkg.com/

deleted 8y ago

[deleted]

jwalton 8y ago

The official NPM registry isn't fairing well, either: https://github.com/npm/npm/issues/20766 https://github.com/npm/npm/issues/20766

PudgePacket 8y ago

They had other issues earlier this week too. Doesn't help that their status page hasn't been updated in a year :D https://twitter.com/trentbillington/status/998822168036167680 https://twitter.com/trentbillington/status/99882216803616768....