7 ms·
...in the age of AI, does anyone have an actual solution for keeping out bots while preserving the privacy of humans? Obviously this is terrible, but I think t
by Wowfunhappy 5mo ago
...in the age of AI, does anyone have an actual solution for keeping out bots while preserving the privacy of humans?
Obviously this is terrible, but I think there's a possibility it's the least terrible option? Another option is IP reputation, which I think is worse. Or scanning a code with a non-rooted phone, which I think is even worse than that!
- spacedoutman 5mo agoPrivate invite only internets
- arbol 5mo agoLAN parties?!
- fidotron 5mo ago> ...in the age of AI, does anyone have an actual solution for keeping out bots while preserving the privacy of humans? There isn't one, and pretending otherwise is nonsense because humans will always provide their credentials to something to act on their behalf. In the limit you end up with Chinese phone farms.
- tardedmeme 5mo agoRight. Botnet operators love cloudflare because they make so much money renting out compromised machines to pass their tests.
- malka1986 5mo ago> keeping out bot You can forget about it. It is not possible. Simple as that.
- Wowfunhappy 5mo agoLet's say I'm selling concert tickets. How do I prevent bots from buying up all the tickets and scalping them?
- luckylion 5mo agoTie them to the buyer's identity, offer at-value buy-backs until X weeks before event, disallow resale.
- MyMemoryfails 5mo agoI'd simply check filling speed, even with browser's autocomplete humans are slow due needing click submit. Then when it's "processing", do them in bulk and prioritize slower users. There's huge opportunity do bot checks after checkout without affecting user experience. Also on product launches you could add unique field which requires user to input, for example that way bots can't prepare for launches.
- ndriscoll 5mo agoSell them via a Dutch auction. Eliminate the arbitrage opportunity for scalpers and make more money in the process.
- dcrazy 5mo agoThat’s how you wind up with only kids of millionaires at your Taylor Swift concert.
- queenkjuul 5mo agoSo a Taylor Swift concert
- ranguna 5mo ago
- cr125rider 5mo agoAnd identifying a bot that is acting on my behalf. Claude go search this topic is basically the same as Googling something and clicking on the results. Human driven AI searching needs to be in a different box than AI scraping for training data. Which sounds extremely difficult to differentiate
- JoshTriplett 5mo agoHopefully it stays that way; "a bot acting on my behalf" is still a bot. At least it's often a well-behaved bot and uses a user-agent that can be detected and blocked.
- doctorpangloss 5mo agoweb environment integrity
- Wowfunhappy 5mo agoBut that is so clearly worse than WebGL fingerprinting!
- csomar 5mo agoThey are not a problem unless you "believe" it is a problem. I estimate around 20-25K hits to my website from bots per day and I have all cloudflare protections disabled. Any decently optimized server should be able to easily handle that. (it's roughly 1 request every 3 seconds).
- specialp 5mo agoYes and that is just the bot background radiation of the internet. I run a primary source of information site and these botnets are aggressive to a DDOS level. All to do some sort of scraping. Because they have sophisticated enough tactics to DDOS us if they wanted to. However I am not sure their objective as they have wasted enough of our resources to have scraped all our content 1000s of times over. That 25k traffic is a couple of minutes for us. And that adds up. 80-90pct of our traffic is this
- thisislife2 5mo agoTrue. But it still wastes your server resources, right? And it's sad that you have to accept that as part of the "cost" of hosting a site ...
- ndriscoll 5mo agoWhat resources are you concerned about? An n100 minipc should be capable of serving something like a blog at 20k+ requests/second (or saturating its network).
- deleted 5mo ago[deleted]
- thisislife2 5mo agoSure. But 25,000 (bot hits / day) x 50kB (static webpage) = 1.25 GB / day / page x 30 (days) = 37.5 GB of data transfer / month / page. And that's assuming a static resource - if any of the resource is dynamically generated by the server, there's a CPU and memory cost too. Overall, even if you treat the impact on the server as "negligible", it's still an unnecessary waste of resource was the point I was trying to convey. Here's a more real-world projection of the cost and server impact - The Bandwidth Cost of AI Crawlers: What Scraping Really Costs Publishers - https://aipaypercrawl.com/articles/ai-crawler-bandwidth-cost https://aipaypercrawl.com/articles/ai-crawler-bandwidth-cost
- Gander5739 5mo agoYou don't need a non-rooted phone to pass captcha checks, I have a rooted phone and can pass the captchas that ask you to scan a qr code. But I doubt phones without google services would manage.
- HWR_14 5mo agoHow does scanning a QR code prove any kind of captcha?
- Gander5739 5mo agohttps://support.google.com/recaptcha/answer/16609652 https://support.google.com/recaptcha/answer/16609652 - it just launches the verification service.
- thisislife2 5mo agoThe only solution is regulation. If all content created by anyone has a copyright, how does an implicit opt-in (which is what happens if you don't create a robots.txt file for your website) for scraping make any sense? Moreover, even if you have a robots.txt, AI (or whatever) bots often don't respect it (or use workarounds - they outsource scraping of such "restricted" sites to unethical third-parties to get the data; Meta has even resorted to piracy, openly!). So clearly, the logic and the "honour system" has failed. Cloudflare, Google Captcha, HCaptcha etc. are all shitty technical solutions because, as we are all discovering, it comes at the cost of our privacy (i.e. our personal data may monetise these services) and / or our computing resource and time. If current copyright laws aren't sufficient to prevent this, we have to acknowledge the system is broken. The answer could be enhancing it with some kind of Digital Millennium Copyright Act (DMCA) -like laws, but in favour of the creators against BigTech or rogue actors. - Web-scraping and copyright law - https://www.neudata.co/blog/web-scraping-and-copyright-law https://www.neudata.co/blog/web-scraping-and-copyright-law - Why DMCA Claims Against Web Scrapers Face Long Odds - https://capstonedc.com/insights/why-dmca-claims-against-web-scrapers-face-long-odds/ https://capstonedc.com/insights/why-dmca-claims-against-web-...
- ImPostingOnHN 5mo agoI don't think regulation will stop web scraping, not least of which because it can be done from locations outside the jurisdiction of the regulations. > we have to acknowledge the system is broken The system is broken. It probably takes, what, 10 seconds or less to use a residential or foreign proxy, 6+ months to internationally track and prosecute a single offender? So like a million times more effort going the regulatory route.
- thisislife2 5mo agoJust as criminal laws don't end all crimes, copyright laws and anti-scraping regulation won't end all scraping. But it will greatly reduce it and limit it to rogue actors. Two examples I can cite here are the laws against email spams and laws against unsolicited marketing calls - they had a definite impact in reducing both (even in India, from where I am, where implementation of laws are often lax).
- jeroenhd 5mo agoRemote attestation should still be possible with a rooted phone if phone manufacturers weren't so shit. If the attestation happens at hardware level, it doesn't matter what programs or kernels you're running.
- ravenstine 5mo agoOr maybe we can actually start paying for all the things we use on the Web, making it prohibitively expensive to deploy fleets of bots.
- HappMacDonald 5mo agoFeel free to work out a frictionless micropayment method though.
- ashishbijlani15 5mo ago[dead]
- onel 5mo agoUnfortunately, I think the solution will be invite only services, communities, etc. Someone needs to invite you to have access to it. If you host your own blog then that might be okay to have public access, you would want everyone and everything to read it. But if you're hosting your own photos, we might need tailscale like services to only allow certain people to access that.
- neop1x 5mo agoJust implement caches, add indices to your DBs, use CDNs. Servers are very fast nowadays, have quite a lot of RAM and can handle huge amount of clients. No need to implement this anti-bot bullshit, it is mainly marketing, providing solutions to a problem which doesn't exist for most websites.