10 ms·

I'm confused about the ToS comment by Cloudflare. The txt is on a website so it is a web content? So robots.txt is not supported by Cloudflare to cache/proxy i

I'm confused about the ToS comment by Cloudflare. The txt is on a website so it is a web content? So robots.txt is not supported by Cloudflare to cache/proxy it? That would be a weird regulation. And I bet everyone violates the Cloudflare ToS then.

andiareso 4y ago
Yeah... That just doesn't seem right. All web content is text...
- naikrovek 4y ago
  text/html is not text/plain but that doesn't matter: it's not a technical limitation that caused cloudflare to draw this line. it's cloudflare deciding to protect "web content" and not videos or .iso images or other things that normally are not commonly served while you browse a contemporary website and read HTML.
  - thaumasiotes 4y ago
    > it's cloudflare deciding to protect "web content" and not videos or .iso images or other things that normally are not commonly served while you browse a contemporary website and read HTML. That's false in two ways: first, text is normally served while you browse a contemporary website; second, so are images, which are explicitly called out as potentially violating this clause. Text is the only data that isn't covered by this clause.
    - naikrovek 4y ago
      - deleted 4y ago
        [deleted]
- jonny_eh 4y ago
  > All web content is text... It's all 1s and 0s too
webstrand 4y ago
I guess they just need to serve it with a minimal html shell
tyingq 4y ago
It's from this tos page: https://www.cloudflare.com/terms/ https://www.cloudflare.com/terms/ 2.8 Limitation on Serving Non-HTML Content ...Use of the Services for serving video or a disproportionate percentage of pictures, audio files, or other non-HTML content is prohibited, unless purchased separately... A huge text/plain artifact, requested often, would seem to fall into that category of "disproportionate percentage" compared to text/html served.
- bornfreddy 4y ago
  Sounds like it is meant to deal with multimedia mostly? But anyway, just rename .txt to .html and you're done.
  - Maxburn 4y ago
    Simple but will it will break all sorts of automation down the line? All the other adlists are txt and I don't know how they would handle other file types, even if the content is unchanged.
    - PaulDavisThe1st 4y ago
      Determining file type from the file name suffix is a fool's game and always was.
      - ethbr0 4y ago
        Is it? Seems superior to arbitrary magic numbers or headers, and God forbid full naive parsing, in most ways.
        mannykannot 4y ago
        I doubt there is any solution that is both robust and simple. In a sense, it is the same problem as that which ad blockers are attempting to solve.
        4y ago
tomudding 4y ago
Cloudflare caches robots.txt by default when proxied (the only .txt-file that they automatically cache), for all other content the following from their ToS probably applies: > Use of the Services for serving video or a disproportionate percentage of pictures, audio files, or other non-HTML content is prohibited, unless purchased separately as part of a Paid Service or expressly allowed under our Supplemental Terms for a specific Service. We will never know the reasoning of the support agent who replied to the EasyList maintainers, but I can imagine that it is indeed disproportionate for EasyList. I really hope that Cloudflare actually sees that they are making a wrong decision here and actually help the EasyList maintainers.
- tyingq 4y ago
  The TOS isn't that you can't serve plain text, it's that it shouldn't be disproportionate in volume to the cached html served.
  - unknownaccount 4y ago
    That could be solved by tacking on 2x the file size worth of pointless html code to the file.
- thaumasiotes 4y ago
  What's the difficulty supposed to be? Serve the same thing with a different MIME type and you're in compliance.
jakear 4y ago
Sounds like they’re just using the wrong service. R2 is designed for object storage, and has 0 egress fees. That’d be the way to go. Not sure why the support engineer didn’t mention it. The standard cloudflare web caching probably doesn’t work well for this use case for whatever reason. The price is only 0.015/GB/mo, so the ~MB(?) of list would be served in perpetuity for less than a dollar.
- stavros 4y ago
  Actually, you're right. How would this work? Is Cloudflare really willing to foot the bill of 20 TB of bandwidth per day for a small text file that costs $0 to store?
  - rvnx 4y ago
    Yes why not ? For reputation and attracting developers it seems to be worth it. If it costs 75K USD/year, that's already paid back with one big enterprise customer only. Though, adblocking is a big business, many actors there are getting large revenue. For example, Eyeo's income was 50 million USD per year last time I checked (and I guess most of it is actually profit), so they can find a solution if they really want.
  - trinovantes 4y ago
    Egress is free but not public i.e. you can't just give anyone an url. You have to use your own server to fetch content from R2 and then serve it to your visitors. Each fetch costs money but first 10 mil reads are free and your own server probably has egress fees.
    - stavros 4y ago
      No, egress is indeed public. Here's an example link, straight to R2: https://img.phantasmagoria.me/img/96XJrjejoHNdrQv7.jpg https://img.phantasmagoria.me/img/96XJrjejoHNdrQv7.jpg Even if you have a private bucket, you can give people a signed link with read access, for up to two weeks, IIRC.
kenmacd 4y ago
Imagine you're trying to block a DDoS attack. If the client is downloading HTML then they likely also have JS enabled giving you a ton of options for running code on their computer to help you decide if the traffic is legitimate. If they're downloading text you can still use the headers, and some tricks around redirects, but overall you have far less data on which to decide.
RunSet 4y ago
> The txt is on a website so it is a web content? Even more wtf- the file extension determines the file content?

andiareso 4y ago

Yeah... That just doesn't seem right. All web content is text...

naikrovek 4y ago
text/html is not text/plain but that doesn't matter: it's not a technical limitation that caused cloudflare to draw this line. it's cloudflare deciding to protect "web content" and not videos or .iso images or other things that normally are not commonly served while you browse a contemporary website and read HTML.
- thaumasiotes 4y ago
  > it's cloudflare deciding to protect "web content" and not videos or .iso images or other things that normally are not commonly served while you browse a contemporary website and read HTML. That's false in two ways: first, text is normally served while you browse a contemporary website; second, so are images, which are explicitly called out as potentially violating this clause. Text is the only data that isn't covered by this clause.
  - naikrovek 4y ago
    - deleted 4y ago
      [deleted]
jonny_eh 4y ago
> All web content is text... It's all 1s and 0s too

webstrand 4y ago

I guess they just need to serve it with a minimal html shell

tyingq 4y ago

It's from this tos page: https://www.cloudflare.com/terms/ https://www.cloudflare.com/terms/ 2.8 Limitation on Serving Non-HTML Content ...Use of the Services for serving video or a disproportionate percentage of pictures, audio files, or other non-HTML content is prohibited, unless purchased separately... A huge text/plain artifact, requested often, would seem to fall into that category of "disproportionate percentage" compared to text/html served.

bornfreddy 4y ago
Sounds like it is meant to deal with multimedia mostly? But anyway, just rename .txt to .html and you're done.
- Maxburn 4y ago
  Simple but will it will break all sorts of automation down the line? All the other adlists are txt and I don't know how they would handle other file types, even if the content is unchanged.
  - PaulDavisThe1st 4y ago
    Determining file type from the file name suffix is a fool's game and always was.
    - ethbr0 4y ago
      Is it? Seems superior to arbitrary magic numbers or headers, and God forbid full naive parsing, in most ways.
      - mannykannot 4y ago
        I doubt there is any solution that is both robust and simple. In a sense, it is the same problem as that which ad blockers are attempting to solve.
      - 4y ago

tomudding 4y ago

Cloudflare caches robots.txt by default when proxied (the only .txt-file that they automatically cache), for all other content the following from their ToS probably applies: > Use of the Services for serving video or a disproportionate percentage of pictures, audio files, or other non-HTML content is prohibited, unless purchased separately as part of a Paid Service or expressly allowed under our Supplemental Terms for a specific Service. We will never know the reasoning of the support agent who replied to the EasyList maintainers, but I can imagine that it is indeed disproportionate for EasyList. I really hope that Cloudflare actually sees that they are making a wrong decision here and actually help the EasyList maintainers.

tyingq 4y ago
The TOS isn't that you can't serve plain text, it's that it shouldn't be disproportionate in volume to the cached html served.
- unknownaccount 4y ago
  That could be solved by tacking on 2x the file size worth of pointless html code to the file.
thaumasiotes 4y ago
What's the difficulty supposed to be? Serve the same thing with a different MIME type and you're in compliance.

jakear 4y ago

Sounds like they’re just using the wrong service. R2 is designed for object storage, and has 0 egress fees. That’d be the way to go. Not sure why the support engineer didn’t mention it. The standard cloudflare web caching probably doesn’t work well for this use case for whatever reason. The price is only 0.015/GB/mo, so the ~MB(?) of list would be served in perpetuity for less than a dollar.

stavros 4y ago
Actually, you're right. How would this work? Is Cloudflare really willing to foot the bill of 20 TB of bandwidth per day for a small text file that costs $0 to store?
- rvnx 4y ago
  Yes why not ? For reputation and attracting developers it seems to be worth it. If it costs 75K USD/year, that's already paid back with one big enterprise customer only. Though, adblocking is a big business, many actors there are getting large revenue. For example, Eyeo's income was 50 million USD per year last time I checked (and I guess most of it is actually profit), so they can find a solution if they really want.
- trinovantes 4y ago
  Egress is free but not public i.e. you can't just give anyone an url. You have to use your own server to fetch content from R2 and then serve it to your visitors. Each fetch costs money but first 10 mil reads are free and your own server probably has egress fees.
  - stavros 4y ago
    No, egress is indeed public. Here's an example link, straight to R2: https://img.phantasmagoria.me/img/96XJrjejoHNdrQv7.jpg https://img.phantasmagoria.me/img/96XJrjejoHNdrQv7.jpg Even if you have a private bucket, you can give people a signed link with read access, for up to two weeks, IIRC.

kenmacd 4y ago

Imagine you're trying to block a DDoS attack. If the client is downloading HTML then they likely also have JS enabled giving you a ton of options for running code on their computer to help you decide if the traffic is legitimate. If they're downloading text you can still use the headers, and some tricks around redirects, but overall you have far less data on which to decide.

RunSet 4y ago

> The txt is on a website so it is a web content? Even more wtf- the file extension determines the file content?