9 ms·

Websites and web developers mostly don't care about client-side problems

62 points by zdw 1y ago

decremental 1y ago
It's not that website owners don't care that they're frustrating users, losing visitors and customers, or creating a poor experience. It's an intractable problem for most website owners to combat the endless ways that their sites are being botted and bogged down, and having to pay for resources to handle the 98% of traffic their sites are getting that isn't coming from real users and customers. By all means, solve it and everyone will be happy.
- nottorp 1y ago
  Heh. Who asked those website owners to have laggy scrolling, non existent contrast, hijack my back button, generally run so much javascript that a cluster is needed client side just to display a 3 line LLM generated blog post?
  - nulbyte 1y ago
    This. It seems every website these days needs Javascript enabled just to load static content that could have been loaded between the time I hovered over a link and clicked it.
  - noirscape 1y ago
    To be fair, the sites resorting to extreme anti-bot measures are also often not the ones that are a clientside JavaScript hell. Thinking of the most extreme option (throwing proof of work checks at browsers), the main stuff that jumps to mind is sites like sourcehut, Linux Kernel Archives and so on and the admins for all of those sites have noted that the traffic they get is far outside of expectations[0]. Not whatever blogspam ended up on the top of Google search that day. The badly designed sites are often the ones that don't care about their bandwidth anyways. [0]: https://drewdevault.com/2025/03/17/2025-03-17-Stop-externalizing-your-costs-on-me.html https://drewdevault.com/2025/03/17/2025-03-17-Stop-externali...
  - danaris 1y ago
    You're painting with a pretty damn broad brush there. Do you think that every small personal website is serving nothing but "a 3 line LLM generated blog post"? Do you not think there are some out there that have perfectly reasonable content? Much of it not even monetized? And yet the bots are causing this problem for everyone. They are completely indiscriminate. So before you try to dismiss this as a non-issue, maybe consider that there's more out there being affected by this than the absolute worst case possible to imagine.
    - nottorp 1y ago
      Yes, so there are far worse things out there than the anime cats...
terminalshort 1y ago
How real is this "crawler plague" that the author refers to? I haven't seen it. But that's just as likely to because I don't care, and therefore am not looking, as it is to be because it's not there. Loading static pages from CDN to scrape training data takes such minimal amounts of resources that it's never going to be a significant part of my costs. Are there cases where this isn't true?
- decremental 1y ago
  [dead]
- snowwrestler 1y ago
  Yes, it’s true. Most sites don’t have a forever cache TTL so a crawler that hits every page on a database-backed site is going to hit mostly uncached pages (and therefore the DB). I also have a faceted search that some stupid crawler has spent the last month iterating through. Also mostly uncached URLs.
  - n3storm 1y ago
    Yeah, or an event plugin where spiders walks every day of several years...
- n3storm 1y ago
  My estimation is at least 70% of traffic on small sites 300-3000 daily views, is not human
- danaris 1y ago
  It's very real. It's crashed my site a number of times.
- ApeWithCompiler 1y ago
  The following is the best I could collect quickly to provide backup to the statement. Unfortunally it's not the high quality first instance of raw statistics I would have liked. But from what I have read time to time the crawler acted magnitudes outside of what could have been accepted as just badly configured. https://herman.bearblog.dev/the-great-scrape/ https://herman.bearblog.dev/the-great-scrape/ https://drewdevault.com/2025/03/17/2025-03-17-Stop-externalizing-your-costs-on-me.html https://drewdevault.com/2025/03/17/2025-03-17-Stop-externali... https://lwn.net/Articles/1008897/ https://lwn.net/Articles/1008897/ https://tecnobits.com/en/AI-crawlers-on-Wikipedia-platform-danger/ https://tecnobits.com/en/AI-crawlers-on-Wikipedia-platform-d... https://boston.conman.org/2025/08/21.1 https://boston.conman.org/2025/08/21.1
azeemba 1y ago
The author is suggesting that websites care more about server side issues than client side issues. To the point that they don't realize that users stop using them. I think that statement is way too strong and obviously not true of businesses. It might be true if hobbyist websites where the creator is personally more interested on the server side but it's definitely not true of professional websites. Professional websites that have enough of a budget to care about the server side will absolutely care about the client side and will track usage. If 10% fewer people used the website, the analytics would show that and there would be a fire drill. What I can agree with on the author is more of a nuanced point. Client side problems are a lot harder and have a very long tail due to unique client configurations (OS, browser, extensions, physical hardware). So with thousands of combinations, you end up with some wild and rare issues. It becomes hard to chase all of them down and some you just have to ignore. This can lead to it feeling like websites don't care about client side but it just shows client side is hard.
- carlosjobim 1y ago
  > I think that statement is way too strong and obviously not true of businesses Amazon.com Inc is currently worth 2.4 billion dollars and the only reason is that most businesses insist on giving their customers the worst online experience possible. I wish that I could one day understand the logic, which goes like this: 1. Notice that people are on their phones all the time. 2. And notice that when people are looking to buy something they first go on the computer or on the smart phone. 3. Therefore let's make the most godawful experience on our website possible, to make sure that our potential customers hate us and don't make a purchase. 4. Customers make their purchase on Amazon instead. 5. Profit??
  - danaris 1y ago
    > Amazon.com Inc is currently worth 2.4 billion dollars and the only reason is that most businesses insist on giving their customers the worst online experience possible. This is an incredibly reductive view of how Amazon came to dominate online retail. If you genuinely believe this, I would strongly urge you to research their history and understand how they became the monopoly they are today. I assure you, it's not primarily because they care more about the end user's experience.
jt2190 1y ago
The “client-side problems” Siebenmann is talking about are the various anti-bot measures (CAPTCHAs, rate limiters, etc.) that operators put in place that make the end user experience worse. Operators feel that they have no choice but to keep their servers available, thus they “don’t care”. He makes a statement in an earlier article that I think sums things up nicely: > One thing I've wound up feeling from all this is that the current web is surprisingly fragile. A significant amount of the web seems to have been held up by implicit understandings and bargains, not by technology. When LLM crawlers showed up and decided to ignore the social things that had kept those parts of the web going, things started coming down all over the place. This social contract is, to me, built around the idea that a human will direct the operation of a computer in real time (largely by using a web browser and clicking links) but I think that this approach is extremely inefficient of both the computer’s and the human’s resources (cpu and time, respectively). The promise of technology should not be to put people behind desks staring at a screen all day, so this evolution toward automation must continue. I do wonder what the new social contract will be: Perhaps access to the majority of servers will be gated by micropayments, but what will the “deal” be for those who don’t want to collect payments? How will they prevent abuse while keeping access free? [1] “The current (2025) crawler plague and the fragility of the web”https://utcc.utoronto.ca/~cks/space/blog/web/WebIsKindOfFragile https://utcc.utoronto.ca/~cks/space/blog/web/WebIsKindOfFrag...
- Retr0id 1y ago
  Another implicit social contract is that you can tell whether a request is coming from a commercial or non-commercial source based on the originating ISP. This was always a heuristic but it was more reliable in the past. If 1000 AWS boxes start hammering your API you might raise an eyebrow, but 1000 requests coming from residential ISPs around the world could be an organic surge in demand for your service. Residential proxy services break this - which has been happening on some level for a long time, but the AI-training-set arms race has driven up demand and thus also supply. It's quite easy to block all of AWS, for example, but it's less easy to figure out which residential IPs are part of a commercially-operated botnet.
  - SoftTalker 1y ago
hombre_fatal 1y ago
I don't really get what this article is talking about nor the distinctions that it's trying to draw between server and client. It brings up multiple different things from captcha to actual client performance so it's not clear what "problems" means in the title nor TFA. The author needs to open with a paragraph that establishes better context. They open with a link to another post where they talk about anti-LLM defenses but it doesn't clarify what they are talking about when they compare server problems with client-side problems.
superkuh 1y ago
It's funny because I can no longer view his website's web pages because he implemented a broken by default "anti-bot" system that blocks anything that's not a corporate browser from $latestyear. > You're using a suspiciously old browser >You're probably reading this page because you've attempted to access some part of my blog (Wandering Thoughts) or CSpace, the wiki thing it's part of. Unfortunately you're using a browser version that my anti-crawler precautions consider suspicious, most often because it's too old (most often this applies to versions of Chrome). Unfortunately, as of early 2025 there's a plague of high volume crawlers (apparently in part to gather data for LLM training) that use a variety of old browser user agents, especially Chrome user agents. To reduce the load on Wandering Thoughts I'm experimenting with (attempting to) block all of them, and you've run into this. >If this is in error and you're using a current version of your browser of choice, you can contact me at my current place at the university (you should be able to work out the email address from that). If possible, please let me know what browser you're using and so on, ideally with its exactl User-Agent string. Hopefully I solved his email address riddle.
Terretta 1y ago
The only thing I see is: Safari can't open the page because it couldn't establish a secure connection to the server. Add "webmasters" or "sysadmins" to the list? // An eighth reload worked.
1vuio0pswjnm7 1y ago
No user-agent header required: https://web.archive.org/web/20250823105045if_/https://utcc.utoronto.ca/~cks/space/blog/web/WebsitesDontCareAboutClients https://web.archive.org/web/20250823105045if_/https://utcc.u...
glimmung 1y ago
This is hilarious - I have yet to get to the article, because (on a fresh Ubuntu Mate LTS install with the stock Firefox) I am redirected to: - https://utcc.utoronto.ca/~cks/cspace-generic-ua.html https://utcc.utoronto.ca/~cks/cspace-generic-ua.html ...which complains about an "HTTP User-Agent header value that is too generic or otherwise excessively suspicious. Unfortunately, as of early 2025 there's a plague of high volume crawlers (apparently in part to gather data for LLM training) that behave like this.", and I'm left thinking that the person behind this site does not care about client-side problems...

62 points by zdw 1y ago

It's not that website owners don't care that they're frustrating users, losing visitors and customers, or creating a poor experience. It's an intractable problem for most website owners to combat the endless ways that their sites are being botted and bogged down, and having to pay for resources to handle the 98% of traffic their sites are getting that isn't coming from real users and customers. By all means, solve it and everyone will be happy.

nottorp 1y ago
Heh. Who asked those website owners to have laggy scrolling, non existent contrast, hijack my back button, generally run so much javascript that a cluster is needed client side just to display a 3 line LLM generated blog post?
- nulbyte 1y ago
  This. It seems every website these days needs Javascript enabled just to load static content that could have been loaded between the time I hovered over a link and clicked it.
- noirscape 1y ago
  To be fair, the sites resorting to extreme anti-bot measures are also often not the ones that are a clientside JavaScript hell. Thinking of the most extreme option (throwing proof of work checks at browsers), the main stuff that jumps to mind is sites like sourcehut, Linux Kernel Archives and so on and the admins for all of those sites have noted that the traffic they get is far outside of expectations[0]. Not whatever blogspam ended up on the top of Google search that day. The badly designed sites are often the ones that don't care about their bandwidth anyways. [0]: https://drewdevault.com/2025/03/17/2025-03-17-Stop-externalizing-your-costs-on-me.html https://drewdevault.com/2025/03/17/2025-03-17-Stop-externali...
- danaris 1y ago
  You're painting with a pretty damn broad brush there. Do you think that every small personal website is serving nothing but "a 3 line LLM generated blog post"? Do you not think there are some out there that have perfectly reasonable content? Much of it not even monetized? And yet the bots are causing this problem for everyone. They are completely indiscriminate. So before you try to dismiss this as a non-issue, maybe consider that there's more out there being affected by this than the absolute worst case possible to imagine.
  - nottorp 1y ago
    Yes, so there are far worse things out there than the anime cats...

terminalshort 1y ago

How real is this "crawler plague" that the author refers to? I haven't seen it. But that's just as likely to because I don't care, and therefore am not looking, as it is to be because it's not there. Loading static pages from CDN to scrape training data takes such minimal amounts of resources that it's never going to be a significant part of my costs. Are there cases where this isn't true?

decremental 1y ago
[dead]
snowwrestler 1y ago
Yes, it’s true. Most sites don’t have a forever cache TTL so a crawler that hits every page on a database-backed site is going to hit mostly uncached pages (and therefore the DB). I also have a faceted search that some stupid crawler has spent the last month iterating through. Also mostly uncached URLs.
- n3storm 1y ago
  Yeah, or an event plugin where spiders walks every day of several years...
n3storm 1y ago
My estimation is at least 70% of traffic on small sites 300-3000 daily views, is not human
danaris 1y ago
It's very real. It's crashed my site a number of times.
ApeWithCompiler 1y ago
The following is the best I could collect quickly to provide backup to the statement. Unfortunally it's not the high quality first instance of raw statistics I would have liked. But from what I have read time to time the crawler acted magnitudes outside of what could have been accepted as just badly configured. https://herman.bearblog.dev/the-great-scrape/ https://herman.bearblog.dev/the-great-scrape/ https://drewdevault.com/2025/03/17/2025-03-17-Stop-externalizing-your-costs-on-me.html https://drewdevault.com/2025/03/17/2025-03-17-Stop-externali... https://lwn.net/Articles/1008897/ https://lwn.net/Articles/1008897/ https://tecnobits.com/en/AI-crawlers-on-Wikipedia-platform-danger/ https://tecnobits.com/en/AI-crawlers-on-Wikipedia-platform-d... https://boston.conman.org/2025/08/21.1 https://boston.conman.org/2025/08/21.1

azeemba 1y ago

The author is suggesting that websites care more about server side issues than client side issues. To the point that they don't realize that users stop using them. I think that statement is way too strong and obviously not true of businesses. It might be true if hobbyist websites where the creator is personally more interested on the server side but it's definitely not true of professional websites. Professional websites that have enough of a budget to care about the server side will absolutely care about the client side and will track usage. If 10% fewer people used the website, the analytics would show that and there would be a fire drill. What I can agree with on the author is more of a nuanced point. Client side problems are a lot harder and have a very long tail due to unique client configurations (OS, browser, extensions, physical hardware). So with thousands of combinations, you end up with some wild and rare issues. It becomes hard to chase all of them down and some you just have to ignore. This can lead to it feeling like websites don't care about client side but it just shows client side is hard.

carlosjobim 1y ago
> I think that statement is way too strong and obviously not true of businesses Amazon.com Inc is currently worth 2.4 billion dollars and the only reason is that most businesses insist on giving their customers the worst online experience possible. I wish that I could one day understand the logic, which goes like this: 1. Notice that people are on their phones all the time. 2. And notice that when people are looking to buy something they first go on the computer or on the smart phone. 3. Therefore let's make the most godawful experience on our website possible, to make sure that our potential customers hate us and don't make a purchase. 4. Customers make their purchase on Amazon instead. 5. Profit??
- danaris 1y ago
  > Amazon.com Inc is currently worth 2.4 billion dollars and the only reason is that most businesses insist on giving their customers the worst online experience possible. This is an incredibly reductive view of how Amazon came to dominate online retail. If you genuinely believe this, I would strongly urge you to research their history and understand how they became the monopoly they are today. I assure you, it's not primarily because they care more about the end user's experience.

jt2190 1y ago

The “client-side problems” Siebenmann is talking about are the various anti-bot measures (CAPTCHAs, rate limiters, etc.) that operators put in place that make the end user experience worse. Operators feel that they have no choice but to keep their servers available, thus they “don’t care”. He makes a statement in an earlier article that I think sums things up nicely: > One thing I've wound up feeling from all this is that the current web is surprisingly fragile. A significant amount of the web seems to have been held up by implicit understandings and bargains, not by technology. When LLM crawlers showed up and decided to ignore the social things that had kept those parts of the web going, things started coming down all over the place. This social contract is, to me, built around the idea that a human will direct the operation of a computer in real time (largely by using a web browser and clicking links) but I think that this approach is extremely inefficient of both the computer’s and the human’s resources (cpu and time, respectively). The promise of technology should not be to put people behind desks staring at a screen all day, so this evolution toward automation must continue. I do wonder what the new social contract will be: Perhaps access to the majority of servers will be gated by micropayments, but what will the “deal” be for those who don’t want to collect payments? How will they prevent abuse while keeping access free? [1] “The current (2025) crawler plague and the fragility of the web”https://utcc.utoronto.ca/~cks/space/blog/web/WebIsKindOfFragile https://utcc.utoronto.ca/~cks/space/blog/web/WebIsKindOfFrag...

Retr0id 1y ago
Another implicit social contract is that you can tell whether a request is coming from a commercial or non-commercial source based on the originating ISP. This was always a heuristic but it was more reliable in the past. If 1000 AWS boxes start hammering your API you might raise an eyebrow, but 1000 requests coming from residential ISPs around the world could be an organic surge in demand for your service. Residential proxy services break this - which has been happening on some level for a long time, but the AI-training-set arms race has driven up demand and thus also supply. It's quite easy to block all of AWS, for example, but it's less easy to figure out which residential IPs are part of a commercially-operated botnet.
- SoftTalker 1y ago

hombre_fatal 1y ago

I don't really get what this article is talking about nor the distinctions that it's trying to draw between server and client. It brings up multiple different things from captcha to actual client performance so it's not clear what "problems" means in the title nor TFA. The author needs to open with a paragraph that establishes better context. They open with a link to another post where they talk about anti-LLM defenses but it doesn't clarify what they are talking about when they compare server problems with client-side problems.

superkuh 1y ago

It's funny because I can no longer view his website's web pages because he implemented a broken by default "anti-bot" system that blocks anything that's not a corporate browser from $latestyear. > You're using a suspiciously old browser >You're probably reading this page because you've attempted to access some part of my blog (Wandering Thoughts) or CSpace, the wiki thing it's part of. Unfortunately you're using a browser version that my anti-crawler precautions consider suspicious, most often because it's too old (most often this applies to versions of Chrome). Unfortunately, as of early 2025 there's a plague of high volume crawlers (apparently in part to gather data for LLM training) that use a variety of old browser user agents, especially Chrome user agents. To reduce the load on Wandering Thoughts I'm experimenting with (attempting to) block all of them, and you've run into this. >If this is in error and you're using a current version of your browser of choice, you can contact me at my current place at the university (you should be able to work out the email address from that). If possible, please let me know what browser you're using and so on, ideally with its exactl User-Agent string. Hopefully I solved his email address riddle.

Terretta 1y ago

The only thing I see is: Safari can't open the page because it couldn't establish a secure connection to the server. Add "webmasters" or "sysadmins" to the list? // An eighth reload worked.

1vuio0pswjnm7 1y ago

No user-agent header required: https://web.archive.org/web/20250823105045if_/https://utcc.utoronto.ca/~cks/space/blog/web/WebsitesDontCareAboutClients https://web.archive.org/web/20250823105045if_/https://utcc.u...

glimmung 1y ago

This is hilarious - I have yet to get to the article, because (on a fresh Ubuntu Mate LTS install with the stock Firefox) I am redirected to: - https://utcc.utoronto.ca/~cks/cspace-generic-ua.html https://utcc.utoronto.ca/~cks/cspace-generic-ua.html ...which complains about an "HTTP User-Agent header value that is too generic or otherwise excessively suspicious. Unfortunately, as of early 2025 there's a plague of high volume crawlers (apparently in part to gather data for LLM training) that behave like this.", and I'm left thinking that the person behind this site does not care about client-side problems...