4 ms·
To me it sounds like these people are operating websites that don't work. My home internet (80 down, 5 up) connection hosted website handled 20k+ hits from the
by superkuh 2y ago
To me it sounds like these people are operating websites that don't work. My home internet (80 down, 5 up) connection hosted website handled 20k+ hits from the alibaba ai crawler yesterday without missing a beat. And many thousands more from GPTbot, etc.
I'll grant it can be a problem for super-heavy "application" websites where every GET is a serious computation. So I'm not surprised gitlab is having problems. They've literally the most bloated and heaviest website I've ever seen. Maybe applications shouldn't be websites.
But this spreading social hysteria, this belief that all non-humans are dangerous and must be blocked is a nerd snipe. It really doesn't apply to most situations. Just running a website as we've always run them, public, and letting all user-agents access, is much less resource intensive than these various "mitigations" people are implementing. Mitigations which end up being worse than the problem itself in terms of preventing actual humans from reading text.
- TonyTrapp 2y agoThere's source repository browsers (git/svn) way, way leaner than GitLab that have the same issues. Any repo browser offering a blame view for files can be brought down by those bots' traffic patterns. I have been hosting such repository browsers for 10+ years and it was never an issue until the arrival of these bots.
- superkuh 2y agoIndeed. It's really exposing a major downside to running applications in browser context. It never really made sense. These applications really don't want public traffic like actual websites do. They should remain applications and stay off the web. But more likely is that the web will be destroyed to fit the requirements of the applications. Like what cloudflare, etc, and all this anti-bot social hysteria is doing.