6 ms·

Cloudflare automatically fixes Polyfill.io for free sites

152 points by kylehotchkiss 2y ago

skilled 2y ago
> We have taken the exceptional step of using our ability to modify HTML on the fly to replace references to the polyfill.io CDN in our customers’ websites with links to our own, safe, mirror created back in February. I get that they want to do a good thing, but is this something you agree to when you sign up as a Cloudflare customer? If so, that’s kind of crazy. edit: I am talking about both free and paid users. A toggle does not discount my question whatsoever, especially if this is on by default for free users. I am asking specifically about terms of service.
- JohnMakin 2y ago
  > I get that they want to do a good thing, but is this something you agree to when you sign up as a Cloudflare customer? If so, that’s kind of crazy. The article says that if you are a paying customer this is off by default.
  - Mathnerd314 2y ago
    True, but they say "we decided". That implies they could also have decided to turn it on immediately. And they did for the log4j and Shellshock issues.
    - JohnMakin 2y ago
      I can’t think of an enterprise use case where someone would be upset they did that. can you? I’m struggling. I helped triage the log4j incident in a pretty traumatic week and was pretty happy they took the steps they did. It’s like, kind of what you are paying for.
- datadrivenangel 2y ago
  A manual toggle is required for paid customers. Only free customers were automatically turned on. It does show that you have to trust cloudflare, which seems like a safe bet given their desire to keep the internet secure.
  - lolinder 2y ago
    > It does show that you have to trust cloudflare, which seems like a safe bet given their desire to keep the internet secure. This is assuming that Cloudflare's interests will now and always align with your own interests, and that their desire to "keep the internet secure" will never lead to them actually screwing over you and your customers. It's a lot like many classic AI Sci-Fi stories. If their mission ever comes into conflict with your real interests, a mission statement that sounds perfect in theory can suddenly become very very dangerous.
    - madeofpalk 2y ago
      > This is assuming that Cloudflare's interests will now and always align with your own interests If you don't assume this, why are you using Cloudflare?
      - lolinder 2y ago
        I'm not.
    - bastawhiz 2y ago
      > This is assuming that Cloudflare's interests will now and always align with your own interests, and that their desire to "keep the internet secure" will never lead to them actually screwing over you and your customers. This is literally the tradeoff of using any SaaS/PaaS/IaaS vendor. Every single one. It's not unique to cloudflare in any way.
      - lolinder 2y ago
        Correct. I'm just noting that Cloudflare's mission to keep the internet secure doesn't make them a "safe bet" for any given company, no more so than any other provider.
- fblp 2y ago
  People are often using cloud flare to improve the security of their site. I think in this case if they failed to act when they could that would be worse.
- hanniabu 2y ago
  Users of these sites they fixed the polyfill for didn't agree to getting infected with malware.
xnorswap 2y ago
Today cloudflare are re-writing payloads to remove malware. Some grim future who knows if cloudflare will be the ones under new owners re-writing payloads to serve adverts (or worse). edit: I think my comment is being misunderstood, I'm not saying this will happen, there's just a neat symmetry between exploit and mitigation.
- janice1999 2y ago
  They're a US based man-in-the-middle for large parts of the internet at the mercy of the America's secretive FISA Court. There is no way they don't have a Room 641A. https://en.wikipedia.org/wiki/Room_641A https://en.wikipedia.org/wiki/Room_641A
  - xnorswap 2y ago
    They're US based now, but there's no guarantee it'll stay that way, if a buyer were to come along with enough money. I guess if your theory is correct then pressure would be applied to make sure any foreign buyout didn't happen. In any case it's extremely unlikely to happen any time soon, but who knows what could happen 50 years down the line, especially if they were to lose market share or the internet fades from relevance. I'm not accusing cloudflare of anything malicious, I want to be clear. But this polyfill wasn't originally malicious either, it was just eventually bought by a malicious actor. My original comment was just commentary on the symmetry of the exploit and the mitigation, they're essentially the same vector.
deleted 2y ago
[deleted]
0x1ch 2y ago
In other words, Cloudflare is man in the middling their customers and hijacking delivery to customer CDNs? This time it was beneficial...
- jen729w 2y ago
  …and next time when it’s harmful they lose their entire business. Doesn’t seem to make a lot of sense, does it?
  - IggleSniggle 2y ago
    Only if the harm becomes known.
    - makeitdouble 2y ago
      The point of using Cloudflare is the security features. You're entrusting proxying and DDOS management to a third party. If you're at that level of scrutiny, that's not a service for you in the first place.
  - realusername 2y ago
    What doesn't make sense for a business might for a government agency. Cloudflare way too juicy of a target to be left alone.
- asddubs 2y ago
  cloudflares business model is literally to mitm their customers. they incidentally also get access to unencrypted versions of all https traffic to cloudfare enabled sites
curiousfab 2y ago
Setting a dangerous precedent, especially doing this by default (no opt-in) needed. But then again, if the people who carelessly include 3rd party dependencies (i.e. playing with fire) are those who use CF... they probably won't object to it :-)
- hipadev23 2y ago
  Your implied correlation between sloppy devs and customers of cloudflare is a bit rich.
politelemon 2y ago
I totally get the intention behind this and the final outcome is definitely a safer internet. It's also somewhat justified considering the author has mentioned they never controlled its domain, yet the library has been distributed through that domain, correct? This is a reflection of the extremely poor security practices in the web development world. At the same time, there is a wrongness in Cloudflare being able to overwrite content and changing the 'truth'. I'm like that larry david gif, I just don't know how to process this. One more thing to note, if you go to the polyfill repo, they've also mentioned they're using Cloudflare to distribute the library. https://github.com/polyfillpolyfill/polyfill-service/commit/2ba06045ce67295f622ee79f6fa6b43c13e37d27 https://github.com/polyfillpolyfill/polyfill-service/commit/...
- madeofpalk 2y ago
  > At the same time, there is a wrongness in Cloudflare being able to overwrite content and changing the 'truth'. This is one of the many features of Clouldflare though, that you can enable additional features that modify your website in various ways - whether it's image resizing or analytics or security scanning. If you don't trust Cloudflare to deliver your website, then you shouldn't use Cloudflare.
  - dclowd9901 2y ago
    I think GP’s point was more about the truth for the consumer of the site. As a user, I trust a site to deliver the content it intends to deliver. Of course this is just a trust by proxy with Cloudflare because you’re trusting the site’s judgment on who to trust. As a user of the site, it makes me just a little bit more uncomfortable knowing there’s another layer of indirect control over the content coming in. Of course this is all just philosophical anyway. We’ve not even touched on external module usage.
    - paxys 2y ago
      For practical purposes Cloudflare is a lot more trustworthy than a random outsourced IT guy.
jgalt212 2y ago
I understand all the negativity, but to me CF has provided a least bad immediate solution.
ubanholzer 2y ago
CloudFlare is "fixing" all websites using their proxy automatically, meanwhile polyfill.io is using CloudFlare as a CDN. That's funny.
ChrisArchitect 2y ago
More discussion on the attack: https://news.ycombinator.com/item?id=40791829 https://news.ycombinator.com/item?id=40791829
stavros 2y ago
At this point, shouldn't all includes use SRI?
- jsheard 2y ago
  Not in this case, for better or worse. The point of polyfill.io is they tailor the delivered Javascript based on the deficiencies of the user agent making the request, so it's not compatible with SRI.
  - stavros 2y ago
    Ah, wow. I didn't realize this was a thing, very convenient target for a supply chain attack, then.
bastawhiz 2y ago
> We have not outright blocked the domain through any of the mechanisms we have because we are concerned it could cause widespread web outages given how broadly polyfill.io is used with some estimates indicating usage on nearly 4% of all websites. Frankly I find this to be a weak argument. "We'll continue serving malware because broken websites are worse than infected ones" is not a good argument. I'm a cloudflare customer and I think this is a very bad look, even if it does mean websites keep "working."
slowmovintarget 2y ago
Related: uBlock origin blocks Polyfill.io https://news.ycombinator.com/item?id=40802393 https://news.ycombinator.com/item?id=40802393 Polyfill supply chain attack: https://news.ycombinator.com/item?id=40791829 https://news.ycombinator.com/item?id=40791829
Sephr 2y ago
I'm curious as to how Fastly can defend their choice to keep Jake Champion in their employ after the damage he caused by transferring polyfill.io to Funnull[1], triggering this entire incident. The warning signs should have been obvious. There's no profit to be had in this kind of free CDN. 1. https://x.com/JFSIII/status/1761385341951361182 https://x.com/JFSIII/status/1761385341951361182
- pciexpgpu 2y ago
  Why assume bad faith and bring pitch forks out? The actual scammer needs to be out - not some person who maintained it for free. Nobody paid this OSS person - only when there is a problem do we ’accuse’ OSS maintainers not when they were actually doing their job for free.
  - deleted 2y ago
    [deleted]
  - Zafira 2y ago
    I think this touches on an interesting question. What obligation do free or open source project maintainers have? Even if a maintainer slaps on a, “I do what I want with this project. I am not responsible for any damages. There is no support” disclaimer, I am not sure that necessarily removes some social responsibilities.
    - SahAssar 2y ago
      This is not an "open source project", this is a service. When I use a open source project I take it as it is now and take a risk on it not being updated, but any updates are "pull", as in that I willingly take in changes. In this case the service is "push", which is very different. Any website that used polyfill.io can have any changes pushed to it, regardless of if the author even had known about a change being made. If my popular project is replaced with a single poop emoji on NPM any existing user is fine (especially since NPM keeps old versions after the whole left-pad thing) and will find an alternative. If polyfill.io replaces their code with document.documentElement.innerHTML = '' that's not fine, since it affects existing users without any update step. I think that nobody should use these public CDNs at all, including things like unpkg and cdnjs, or at the very least using subresource integrity. Either way this has been something that has been on the horizon for years and similar to the buying of popular webextensions.
sakisv 2y ago
While I appreciate their intention and their transparency on what they're doing, I'll echo other comments here in that I, too, feel a bit uneasy that a 3rd party is rewriting my content. The good news is that a (strict) CSP can help with that - and they do mention it in their blog post that they don't rewrite anything if there's a CSP header. It's also worth noting that a (strict) CSP also prevents them from injecting their analytics JS from being injected to your site.
omoikane 2y ago
This post includes a link to an earlier blog post on "reduce your supply chain risk" in 2024-02-29: https://blog.cloudflare.com/polyfill-io-now-available-on-cdnjs-reduce-your-supply-chain-risk https://blog.cloudflare.com/polyfill-io-now-available-on-cdn... In the earlier thread, there was a link to triblondon's post on 2024-02-25 where he urges users to remove that dependency: https://x.com/triblondon/status/1761852117579427975 https://x.com/triblondon/status/1761852117579427975 Unfortunately, neither of these early warnings seem to have gotten much attention. I wonder if there is some news site or service that would make suspicious ownership transfers of this sort more noticeable? Or maybe this type of supply chain changes actually happen all the time and we just got used to them?
ilaksh 2y ago
One obvious issue to me is that we have over-centralized the internet on Cloudflare. That's great they are fixing it on free sites though.
- dclowd9901 2y ago
  Yeah, I agree, but also, what is the way out of it? By virtue of their value proposition, the only natural progression for a CDN is maximum scale, isn’t it?
danslo 2y ago
It doesn't matter anymore, Namecheap has taken down polyfill[.]io: https://x.com/malwrhunterteam/status/1806074377383121148 https://x.com/malwrhunterteam/status/1806074377383121148
- eastdakota 2y ago
  It matters still in so far as if you turn this feature on your site will continue to work as expected even with Polyfill[.]io offline.
  - tristanperry 2y ago
    Hi Matthew, Cloudflare just charged my card $235 with no prior warning (for a service I intended to cancel when I received the renewal reminder). I did not receive any renewal reminders, no advance notice of payment, no invoice. Nothing. Is this normal operating behaviour for Cloudflare? This seems like very very bad anti-consumer behaviour?
    - jgrahamc 2y ago
      Do you have a support ticket number about this? You can email me (email in profile) and I'll make sure support sees it.
X-Istence 2y ago
Looks like it is down at the .io registry level, I am not getting any NS records for polyfill.io
deleted 2y ago
[deleted]

152 points by kylehotchkiss 2y ago

> We have taken the exceptional step of using our ability to modify HTML on the fly to replace references to the polyfill.io CDN in our customers’ websites with links to our own, safe, mirror created back in February. I get that they want to do a good thing, but is this something you agree to when you sign up as a Cloudflare customer? If so, that’s kind of crazy. edit: I am talking about both free and paid users. A toggle does not discount my question whatsoever, especially if this is on by default for free users. I am asking specifically about terms of service.

JohnMakin 2y ago
> I get that they want to do a good thing, but is this something you agree to when you sign up as a Cloudflare customer? If so, that’s kind of crazy. The article says that if you are a paying customer this is off by default.
- Mathnerd314 2y ago
  True, but they say "we decided". That implies they could also have decided to turn it on immediately. And they did for the log4j and Shellshock issues.
  - JohnMakin 2y ago
    I can’t think of an enterprise use case where someone would be upset they did that. can you? I’m struggling. I helped triage the log4j incident in a pretty traumatic week and was pretty happy they took the steps they did. It’s like, kind of what you are paying for.
datadrivenangel 2y ago
A manual toggle is required for paid customers. Only free customers were automatically turned on. It does show that you have to trust cloudflare, which seems like a safe bet given their desire to keep the internet secure.
- lolinder 2y ago
  > It does show that you have to trust cloudflare, which seems like a safe bet given their desire to keep the internet secure. This is assuming that Cloudflare's interests will now and always align with your own interests, and that their desire to "keep the internet secure" will never lead to them actually screwing over you and your customers. It's a lot like many classic AI Sci-Fi stories. If their mission ever comes into conflict with your real interests, a mission statement that sounds perfect in theory can suddenly become very very dangerous.
  - madeofpalk 2y ago
    > This is assuming that Cloudflare's interests will now and always align with your own interests If you don't assume this, why are you using Cloudflare?
    - lolinder 2y ago
      I'm not.
  - bastawhiz 2y ago
    > This is assuming that Cloudflare's interests will now and always align with your own interests, and that their desire to "keep the internet secure" will never lead to them actually screwing over you and your customers. This is literally the tradeoff of using any SaaS/PaaS/IaaS vendor. Every single one. It's not unique to cloudflare in any way.
    - lolinder 2y ago
      Correct. I'm just noting that Cloudflare's mission to keep the internet secure doesn't make them a "safe bet" for any given company, no more so than any other provider.
fblp 2y ago
People are often using cloud flare to improve the security of their site. I think in this case if they failed to act when they could that would be worse.
hanniabu 2y ago
Users of these sites they fixed the polyfill for didn't agree to getting infected with malware.

xnorswap 2y ago

Today cloudflare are re-writing payloads to remove malware. Some grim future who knows if cloudflare will be the ones under new owners re-writing payloads to serve adverts (or worse). edit: I think my comment is being misunderstood, I'm not saying this will happen, there's just a neat symmetry between exploit and mitigation.

janice1999 2y ago
They're a US based man-in-the-middle for large parts of the internet at the mercy of the America's secretive FISA Court. There is no way they don't have a Room 641A. https://en.wikipedia.org/wiki/Room_641A https://en.wikipedia.org/wiki/Room_641A
- xnorswap 2y ago
  They're US based now, but there's no guarantee it'll stay that way, if a buyer were to come along with enough money. I guess if your theory is correct then pressure would be applied to make sure any foreign buyout didn't happen. In any case it's extremely unlikely to happen any time soon, but who knows what could happen 50 years down the line, especially if they were to lose market share or the internet fades from relevance. I'm not accusing cloudflare of anything malicious, I want to be clear. But this polyfill wasn't originally malicious either, it was just eventually bought by a malicious actor. My original comment was just commentary on the symmetry of the exploit and the mitigation, they're essentially the same vector.

deleted 2y ago

[deleted]

0x1ch 2y ago

In other words, Cloudflare is man in the middling their customers and hijacking delivery to customer CDNs? This time it was beneficial...

jen729w 2y ago
…and next time when it’s harmful they lose their entire business. Doesn’t seem to make a lot of sense, does it?
- IggleSniggle 2y ago
  Only if the harm becomes known.
  - makeitdouble 2y ago
    The point of using Cloudflare is the security features. You're entrusting proxying and DDOS management to a third party. If you're at that level of scrutiny, that's not a service for you in the first place.
- realusername 2y ago
  What doesn't make sense for a business might for a government agency. Cloudflare way too juicy of a target to be left alone.
asddubs 2y ago
cloudflares business model is literally to mitm their customers. they incidentally also get access to unencrypted versions of all https traffic to cloudfare enabled sites

curiousfab 2y ago

Setting a dangerous precedent, especially doing this by default (no opt-in) needed. But then again, if the people who carelessly include 3rd party dependencies (i.e. playing with fire) are those who use CF... they probably won't object to it :-)

hipadev23 2y ago
Your implied correlation between sloppy devs and customers of cloudflare is a bit rich.

politelemon 2y ago

I totally get the intention behind this and the final outcome is definitely a safer internet. It's also somewhat justified considering the author has mentioned they never controlled its domain, yet the library has been distributed through that domain, correct? This is a reflection of the extremely poor security practices in the web development world. At the same time, there is a wrongness in Cloudflare being able to overwrite content and changing the 'truth'. I'm like that larry david gif, I just don't know how to process this. One more thing to note, if you go to the polyfill repo, they've also mentioned they're using Cloudflare to distribute the library. https://github.com/polyfillpolyfill/polyfill-service/commit/2ba06045ce67295f622ee79f6fa6b43c13e37d27 https://github.com/polyfillpolyfill/polyfill-service/commit/...

madeofpalk 2y ago
> At the same time, there is a wrongness in Cloudflare being able to overwrite content and changing the 'truth'. This is one of the many features of Clouldflare though, that you can enable additional features that modify your website in various ways - whether it's image resizing or analytics or security scanning. If you don't trust Cloudflare to deliver your website, then you shouldn't use Cloudflare.
- dclowd9901 2y ago
  I think GP’s point was more about the truth for the consumer of the site. As a user, I trust a site to deliver the content it intends to deliver. Of course this is just a trust by proxy with Cloudflare because you’re trusting the site’s judgment on who to trust. As a user of the site, it makes me just a little bit more uncomfortable knowing there’s another layer of indirect control over the content coming in. Of course this is all just philosophical anyway. We’ve not even touched on external module usage.
  - paxys 2y ago
    For practical purposes Cloudflare is a lot more trustworthy than a random outsourced IT guy.

jgalt212 2y ago

I understand all the negativity, but to me CF has provided a least bad immediate solution.

ubanholzer 2y ago

CloudFlare is "fixing" all websites using their proxy automatically, meanwhile polyfill.io is using CloudFlare as a CDN. That's funny.

ChrisArchitect 2y ago

More discussion on the attack: https://news.ycombinator.com/item?id=40791829 https://news.ycombinator.com/item?id=40791829

stavros 2y ago

At this point, shouldn't all includes use SRI?

jsheard 2y ago
Not in this case, for better or worse. The point of polyfill.io is they tailor the delivered Javascript based on the deficiencies of the user agent making the request, so it's not compatible with SRI.
- stavros 2y ago
  Ah, wow. I didn't realize this was a thing, very convenient target for a supply chain attack, then.

bastawhiz 2y ago

> We have not outright blocked the domain through any of the mechanisms we have because we are concerned it could cause widespread web outages given how broadly polyfill.io is used with some estimates indicating usage on nearly 4% of all websites. Frankly I find this to be a weak argument. "We'll continue serving malware because broken websites are worse than infected ones" is not a good argument. I'm a cloudflare customer and I think this is a very bad look, even if it does mean websites keep "working."

slowmovintarget 2y ago

Related: uBlock origin blocks Polyfill.io https://news.ycombinator.com/item?id=40802393 https://news.ycombinator.com/item?id=40802393 Polyfill supply chain attack: https://news.ycombinator.com/item?id=40791829 https://news.ycombinator.com/item?id=40791829

Sephr 2y ago

I'm curious as to how Fastly can defend their choice to keep Jake Champion in their employ after the damage he caused by transferring polyfill.io to Funnull[1], triggering this entire incident. The warning signs should have been obvious. There's no profit to be had in this kind of free CDN. 1. https://x.com/JFSIII/status/1761385341951361182 https://x.com/JFSIII/status/1761385341951361182

pciexpgpu 2y ago
Why assume bad faith and bring pitch forks out? The actual scammer needs to be out - not some person who maintained it for free. Nobody paid this OSS person - only when there is a problem do we ’accuse’ OSS maintainers not when they were actually doing their job for free.
- deleted 2y ago
  [deleted]
- Zafira 2y ago
  I think this touches on an interesting question. What obligation do free or open source project maintainers have? Even if a maintainer slaps on a, “I do what I want with this project. I am not responsible for any damages. There is no support” disclaimer, I am not sure that necessarily removes some social responsibilities.
  - SahAssar 2y ago
    This is not an "open source project", this is a service. When I use a open source project I take it as it is now and take a risk on it not being updated, but any updates are "pull", as in that I willingly take in changes. In this case the service is "push", which is very different. Any website that used polyfill.io can have any changes pushed to it, regardless of if the author even had known about a change being made. If my popular project is replaced with a single poop emoji on NPM any existing user is fine (especially since NPM keeps old versions after the whole left-pad thing) and will find an alternative. If polyfill.io replaces their code with document.documentElement.innerHTML = '' that's not fine, since it affects existing users without any update step. I think that nobody should use these public CDNs at all, including things like unpkg and cdnjs, or at the very least using subresource integrity. Either way this has been something that has been on the horizon for years and similar to the buying of popular webextensions.

sakisv 2y ago

While I appreciate their intention and their transparency on what they're doing, I'll echo other comments here in that I, too, feel a bit uneasy that a 3rd party is rewriting my content. The good news is that a (strict) CSP can help with that - and they do mention it in their blog post that they don't rewrite anything if there's a CSP header. It's also worth noting that a (strict) CSP also prevents them from injecting their analytics JS from being injected to your site.

omoikane 2y ago

This post includes a link to an earlier blog post on "reduce your supply chain risk" in 2024-02-29: https://blog.cloudflare.com/polyfill-io-now-available-on-cdnjs-reduce-your-supply-chain-risk https://blog.cloudflare.com/polyfill-io-now-available-on-cdn... In the earlier thread, there was a link to triblondon's post on 2024-02-25 where he urges users to remove that dependency: https://x.com/triblondon/status/1761852117579427975 https://x.com/triblondon/status/1761852117579427975 Unfortunately, neither of these early warnings seem to have gotten much attention. I wonder if there is some news site or service that would make suspicious ownership transfers of this sort more noticeable? Or maybe this type of supply chain changes actually happen all the time and we just got used to them?

ilaksh 2y ago

One obvious issue to me is that we have over-centralized the internet on Cloudflare. That's great they are fixing it on free sites though.

dclowd9901 2y ago
Yeah, I agree, but also, what is the way out of it? By virtue of their value proposition, the only natural progression for a CDN is maximum scale, isn’t it?

danslo 2y ago

It doesn't matter anymore, Namecheap has taken down polyfill[.]io: https://x.com/malwrhunterteam/status/1806074377383121148 https://x.com/malwrhunterteam/status/1806074377383121148

eastdakota 2y ago
It matters still in so far as if you turn this feature on your site will continue to work as expected even with Polyfill[.]io offline.
- tristanperry 2y ago
  Hi Matthew, Cloudflare just charged my card $235 with no prior warning (for a service I intended to cancel when I received the renewal reminder). I did not receive any renewal reminders, no advance notice of payment, no invoice. Nothing. Is this normal operating behaviour for Cloudflare? This seems like very very bad anti-consumer behaviour?
  - jgrahamc 2y ago
    Do you have a support ticket number about this? You can email me (email in profile) and I'll make sure support sees it.

X-Istence 2y ago

Looks like it is down at the .io registry level, I am not getting any NS records for polyfill.io

deleted 2y ago

[deleted]