5 ms·

CDN is one part of strategy to deal with load. But it is not the only solution unless your site is exclusively static content. Their search, APIs, submission pi

CDN is one part of strategy to deal with load. But it is not the only solution unless your site is exclusively static content. Their search, APIs, submission pipelines, duplicate detectors and a lot of other things are not going to be powered by CDNs.

motorest 1y ago
Thank you for the insight. It's very easy to prescribe simple solutions when we are oblivious to the actual problems being solved.
coliveira 1y ago
All these services can be throttled to deal with AI. I don't see this as a justification. The idea that a service like arXiv should be run as a startup is, simply put, foolish.
Imustaskforhelp 1y ago
pardon me but cloudflare workers seem better for this approach. If we can get for the fact that we require javascript to run it, aside from that. Cloudflare workers is literally the best single thing to happen at least to me. With a single domain, I have done so many personal projects for problems I found interesting and I built so many projects for literally free, no Credit card. No worries whatsoever. I might ditch writing other languages for server based like golang even though I like golang more just because cloudflare workers exists.
- anelson 1y ago
  I too am impressed by Cloudflare Workers’ potential. However Workers supports WASM so you don’t necessarily have to switch to JavaScript to use it. I wrote some Rust code that I run in Cloudflare Functions, which is a layer on top of Cloudflare Workers which also supports WASM. I wrote up the gory details if you’re interested: https://127.io/2024/11/16/generating-opengraph-image-cards-for-my-zola-static-blog-with-cloudflare-functions-and-rust/ https://127.io/2024/11/16/generating-opengraph-image-cards-f... JavaScript is most definitely the path of least resistance but it’s not the only way.
GTP 1y ago
But, under the assumption that the problem are indeed AI crawlers, of the things you listed only the search would be under increased load.
- specialp 1y ago
  I am sure their pages are not entirely static either. The APIs are used by researchers and AI companies too. Also with search you end up having people trying to use it for RAG with AI. I have dealt with this all and there is no one dead simple solution to deal with things. AI crawlers are one part of things, but they also have increasing submissions, have to deal with AI generated spam papers, and all sorts of stuff. There's always this feeling here on HN that oh it is dead simple you just do "X" as if the people that are dealing with it don't know that.

motorest 1y ago

Thank you for the insight. It's very easy to prescribe simple solutions when we are oblivious to the actual problems being solved.

coliveira 1y ago

All these services can be throttled to deal with AI. I don't see this as a justification. The idea that a service like arXiv should be run as a startup is, simply put, foolish.

Imustaskforhelp 1y ago

pardon me but cloudflare workers seem better for this approach. If we can get for the fact that we require javascript to run it, aside from that. Cloudflare workers is literally the best single thing to happen at least to me. With a single domain, I have done so many personal projects for problems I found interesting and I built so many projects for literally free, no Credit card. No worries whatsoever. I might ditch writing other languages for server based like golang even though I like golang more just because cloudflare workers exists.

anelson 1y ago
I too am impressed by Cloudflare Workers’ potential. However Workers supports WASM so you don’t necessarily have to switch to JavaScript to use it. I wrote some Rust code that I run in Cloudflare Functions, which is a layer on top of Cloudflare Workers which also supports WASM. I wrote up the gory details if you’re interested: https://127.io/2024/11/16/generating-opengraph-image-cards-for-my-zola-static-blog-with-cloudflare-functions-and-rust/ https://127.io/2024/11/16/generating-opengraph-image-cards-f... JavaScript is most definitely the path of least resistance but it’s not the only way.

GTP 1y ago

But, under the assumption that the problem are indeed AI crawlers, of the things you listed only the search would be under increased load.

specialp 1y ago
I am sure their pages are not entirely static either. The APIs are used by researchers and AI companies too. Also with search you end up having people trying to use it for RAG with AI. I have dealt with this all and there is no one dead simple solution to deal with things. AI crawlers are one part of things, but they also have increasing submissions, have to deal with AI generated spam papers, and all sorts of stuff. There's always this feeling here on HN that oh it is dead simple you just do "X" as if the people that are dealing with it don't know that.