5 ms·

Tell HN: Camelgate NPM Outage (Cloudflare)

122 points by bavarianbob 2y ago

EDIT: Back online?! NPM discussion: https://github.com/npm/cli/issues/8203 https://github.com/npm/cli/issues/8203 NPM incident: https://status.npmjs.org/incidents/hdtkrsqp134s https://status.npmjs.org/incidents/hdtkrsqp134s Cloudflare messaging: https://www.cloudflarestatus.com/incidents/gshczn1wxh74 https://www.cloudflarestatus.com/incidents/gshczn1wxh74 GitHub issue: https://github.com/sindresorhus/camelcase/issues/114 https://github.com/sindresorhus/camelcase/issues/114 Anyone experiencing npm outage that's more than just the referenced camelcase package?

Recursing 2y ago
Any path with the word "camel" seem to trigger this: https://www.npmjs.com/search?q=camel https://www.npmjs.com/search?q=camel | https://registry.npmjs.org/camel123 https://registry.npmjs.org/camel123 | https://registry.yarnpkg.com/camel456 https://registry.yarnpkg.com/camel456 Some discussion here https://github.com/npm/cli/issues/8203 https://github.com/npm/cli/issues/8203 Edit: this is resolved now https://status.npmjs.org/incidents/hdtkrsqp134s https://status.npmjs.org/incidents/hdtkrsqp134s
tom_usher 2y ago
Seems to be a change in Cloudflare's managed WAF ruleset - any site using that will have URLs containing 'camel' blocked due to the 'Apache Camel - Remote Code Execution - CVE:CVE-2025-29891' (a9ec9cf625ff42769298671d1bbcd247) rule. That rule can be overridden if you're having this issue on your own site.
- cbovis 2y ago
  Confirmed here: https://www.cloudflarestatus.com/incidents/gshczn1wxh74 https://www.cloudflarestatus.com/incidents/gshczn1wxh74
- oncallthrow 2y ago
  WAFs are so shit
  - ronsor 2y ago
    WAFs are literally "a pile of regexes can secure my insecure software"
    - mschuster91 2y ago
      To be fair to WAFs, most are more than just a pile of regexes. Things like detecting bot traffic - be it spammers or AI scrapers - are valuable (ESPECIALLY the AI scraper detection, because unlike search engines these things have zero context recognition or respect for robots.txt and will just happily go on and ingest very heavy endpoints), and the large CDN/WAF providers can do it even better because they can spot shit like automated port scanners, Metasploit or similar skiddie tooling across all the services that use them. Honestly what I'd _love_ to see is AWS, GCE, Azure, Fastly, Cloudflare and Akamai band together and share information about such bad actors, compile evidence lists and file abuse reports against their ISP - or in case the ISP is a "bulletproof hoster" or certain enemy states, initiate enforcement actors like governments to get these bad ISPs disconnected from the Internet.
      - randunel 2y ago
        Why would scrapes get blocked, is scrapping illegal?
        eitland 2y ago
        I don't know if it is, but I also don't think we are required to let dumb bots repeatedly assault or web sites if we can find a technical way to get around it.
        Xylakant 2y ago
        It's very often not, but it's still the website owners property and if they choose so, they can show misbehaving guests the door and kindly ask to remain on the other side (aka block them). Large scale scraping puts substantial burden on web properties. I was paged the other night because someone decided it would be a great idea to throw 200 000rq/s for a few minutes at some publicly available volunteer run service.
- internetter 2y ago
  > any site using that will have URLs containing 'camel' blocked What engineer at cloudflare thought this was a good resolution?
  - Raed667 2y ago
    I doubt the system is that simple. No one wrote a rule saying `if url.contains("camel") then block()` it's probably an unintended side-effect
    - keithwhor 2y ago
      If this is a bet, I'll happily take the other side and give you 4:1 on it.
      - dgfitz 2y ago
        Me too.
    - ycombinatrix 2y ago
      Akamai has been doing precisely that for years & years...
      - deleted 2y ago
        [deleted]
      - benoau 2y ago
        I think you can include advertising/privacy block lists in that vein too, although that allows for the users to locally-correct any issues.
    - isbvhodnvemrwvn 2y ago
      Judging by previous outages it was probably a poorly tested overcomplicated regex which matched to much.
  - TacticalCoder 2y ago
deleted 2y ago
[deleted]
nwalters512 2y ago
The npm folks have officially acknowledged an incident now: https://status.npmjs.org/incidents/hdtkrsqp134s https://status.npmjs.org/incidents/hdtkrsqp134s
deleted 2y ago
[deleted]
deleted 2y ago
[deleted]
mplanchard 2y ago
Glad you posted something, thought I was going nuts
klysm 2y ago
This is what you get when you buy security as an add-on product
- troyvit 2y ago
  Some orgs can't afford not to.
drusepth 2y ago
Is this also why unpkg has been up and down all morning?
- ycombinatrix 2y ago
  unpkg barely works even when there's no incident
pvg 2y ago
This is not CF WAF's first rodeo https://news.ycombinator.com/item?id=20421538 https://news.ycombinator.com/item?id=20421538 Cementing its track record as a product that mostly doesn't do anything except for occasionally break the internet here and there to keep things fun and interesting.
- calvinmorrison 2y ago
  we've used it to rescue some vintage appliances that are basically unsecurable.
- AdamJacobMuller 2y ago
  I'm not sure why "WAF has false positives" makes it useless, nor would I say this is anywhere near the scale of "breaking the internet" and I'm not even fan of the concept of WAFs in general.
  - pvg 2y ago
    The last one took out a lot more stuff than this one but the argument is the same - this product is a checkmark thing and when it's not fulfilling its checkmark purpose, it causes outages. Still an amusing bi-modality! I suppose it shares it with DNSSEC.
    - misiek08 2y ago
      Basically CF default WAF settings saved more small and medium companies I can even count to. I’m not CF fan, but WAFs (with rate limiting) do help. Sad that one or two incidents for that complicated and big services make people post such comments, but cmon - it doesn’t have AI in it's name so sheeps have to cry, right?
- lynnesbian 2y ago
  > a product that mostly doesn't do anything except for occasionally break the internet I wouldn't say that. The postmortem you referred to links to another CloudFlare blog post - one about a pretty serious RCE vuln in Microsoft SharePoint that was blocked by their WAF: https://blog.cloudflare.com/stopping-cve-2019-0604/ https://blog.cloudflare.com/stopping-cve-2019-0604/
miyuru 2y ago
Outsourcing WAF is a double-edged sword. I would have thought a large company like GitHub or Microsoft can have their own WAF team for their apps. (NPM is owned by GitHub, and GitHub is owned by Microsoft)
time4tea 2y ago
Scunthorpe problem

122 points by bavarianbob 2y ago

Recursing 2y ago

Any path with the word "camel" seem to trigger this: https://www.npmjs.com/search?q=camel https://www.npmjs.com/search?q=camel | https://registry.npmjs.org/camel123 https://registry.npmjs.org/camel123 | https://registry.yarnpkg.com/camel456 https://registry.yarnpkg.com/camel456 Some discussion here https://github.com/npm/cli/issues/8203 https://github.com/npm/cli/issues/8203 Edit: this is resolved now https://status.npmjs.org/incidents/hdtkrsqp134s https://status.npmjs.org/incidents/hdtkrsqp134s

tom_usher 2y ago

Seems to be a change in Cloudflare's managed WAF ruleset - any site using that will have URLs containing 'camel' blocked due to the 'Apache Camel - Remote Code Execution - CVE:CVE-2025-29891' (a9ec9cf625ff42769298671d1bbcd247) rule. That rule can be overridden if you're having this issue on your own site.

cbovis 2y ago
Confirmed here: https://www.cloudflarestatus.com/incidents/gshczn1wxh74 https://www.cloudflarestatus.com/incidents/gshczn1wxh74
oncallthrow 2y ago
WAFs are so shit
- ronsor 2y ago
  WAFs are literally "a pile of regexes can secure my insecure software"
  - mschuster91 2y ago
    To be fair to WAFs, most are more than just a pile of regexes. Things like detecting bot traffic - be it spammers or AI scrapers - are valuable (ESPECIALLY the AI scraper detection, because unlike search engines these things have zero context recognition or respect for robots.txt and will just happily go on and ingest very heavy endpoints), and the large CDN/WAF providers can do it even better because they can spot shit like automated port scanners, Metasploit or similar skiddie tooling across all the services that use them. Honestly what I'd _love_ to see is AWS, GCE, Azure, Fastly, Cloudflare and Akamai band together and share information about such bad actors, compile evidence lists and file abuse reports against their ISP - or in case the ISP is a "bulletproof hoster" or certain enemy states, initiate enforcement actors like governments to get these bad ISPs disconnected from the Internet.
    - randunel 2y ago
      Why would scrapes get blocked, is scrapping illegal?
      - eitland 2y ago
        I don't know if it is, but I also don't think we are required to let dumb bots repeatedly assault or web sites if we can find a technical way to get around it.
      - Xylakant 2y ago
        It's very often not, but it's still the website owners property and if they choose so, they can show misbehaving guests the door and kindly ask to remain on the other side (aka block them). Large scale scraping puts substantial burden on web properties. I was paged the other night because someone decided it would be a great idea to throw 200 000rq/s for a few minutes at some publicly available volunteer run service.
internetter 2y ago
> any site using that will have URLs containing 'camel' blocked What engineer at cloudflare thought this was a good resolution?
- Raed667 2y ago
  I doubt the system is that simple. No one wrote a rule saying `if url.contains("camel") then block()` it's probably an unintended side-effect
  - keithwhor 2y ago
    If this is a bet, I'll happily take the other side and give you 4:1 on it.
    - dgfitz 2y ago
      Me too.
  - ycombinatrix 2y ago
    Akamai has been doing precisely that for years & years...
    - deleted 2y ago
      [deleted]
    - benoau 2y ago
      I think you can include advertising/privacy block lists in that vein too, although that allows for the users to locally-correct any issues.
  - isbvhodnvemrwvn 2y ago
    Judging by previous outages it was probably a poorly tested overcomplicated regex which matched to much.
- TacticalCoder 2y ago

deleted 2y ago

[deleted]

nwalters512 2y ago

The npm folks have officially acknowledged an incident now: https://status.npmjs.org/incidents/hdtkrsqp134s https://status.npmjs.org/incidents/hdtkrsqp134s

deleted 2y ago

[deleted]

deleted 2y ago

[deleted]

mplanchard 2y ago

Glad you posted something, thought I was going nuts

klysm 2y ago

This is what you get when you buy security as an add-on product

troyvit 2y ago
Some orgs can't afford not to.

drusepth 2y ago

Is this also why unpkg has been up and down all morning?

ycombinatrix 2y ago
unpkg barely works even when there's no incident

pvg 2y ago

This is not CF WAF's first rodeo https://news.ycombinator.com/item?id=20421538 https://news.ycombinator.com/item?id=20421538 Cementing its track record as a product that mostly doesn't do anything except for occasionally break the internet here and there to keep things fun and interesting.

calvinmorrison 2y ago
we've used it to rescue some vintage appliances that are basically unsecurable.
AdamJacobMuller 2y ago
I'm not sure why "WAF has false positives" makes it useless, nor would I say this is anywhere near the scale of "breaking the internet" and I'm not even fan of the concept of WAFs in general.
- pvg 2y ago
  The last one took out a lot more stuff than this one but the argument is the same - this product is a checkmark thing and when it's not fulfilling its checkmark purpose, it causes outages. Still an amusing bi-modality! I suppose it shares it with DNSSEC.
  - misiek08 2y ago
    Basically CF default WAF settings saved more small and medium companies I can even count to. I’m not CF fan, but WAFs (with rate limiting) do help. Sad that one or two incidents for that complicated and big services make people post such comments, but cmon - it doesn’t have AI in it's name so sheeps have to cry, right?
lynnesbian 2y ago
> a product that mostly doesn't do anything except for occasionally break the internet I wouldn't say that. The postmortem you referred to links to another CloudFlare blog post - one about a pretty serious RCE vuln in Microsoft SharePoint that was blocked by their WAF: https://blog.cloudflare.com/stopping-cve-2019-0604/ https://blog.cloudflare.com/stopping-cve-2019-0604/

miyuru 2y ago

Outsourcing WAF is a double-edged sword. I would have thought a large company like GitHub or Microsoft can have their own WAF team for their apps. (NPM is owned by GitHub, and GitHub is owned by Microsoft)

time4tea 2y ago

Scunthorpe problem