9 ms·

Please stop externalizing your costs directly into my face

258 points by Tomte 2y ago

iinnPP 2y ago
Assuming you can prove it's a company, doesn't the behavior equate to fraud? Seems to hit all the prongs, but I am no lawyer.
- bigolkevin 2y ago
  It doesn't matter if it's fraud, "AI" is now considered an arms race, if we require them to fairly acquire their content we'll fall behind China or another country and then America might lose the WWIII it is constantly preparing for. You might see a couple of small players or especially egregious executives get a slap on the wrist for bad behavior but in this political climate there's no chance that Republicans or Democrats will put a stop to it.
easton 2y ago
It sucks to say, but maybe the best solution is to pay Cloudflare or another fancy web security/CDN company to get rid of this problem for you? Which is extremely dumb, but when the alternatives are “10x capacity just for stupid bots” or “hire a guy whose job it is to block LLMs”… maybe that’s cheapest? Yes, it sucks for the open web, but if it’s your livelihood I probably would consider it. (Either that or make following robots.txt a legal requirement, but that feels also like stifling hobbyists that just want to scrape a page)
- azornathogron 2y ago
  > Either that or make following robots.txt a legal requirement [...] A legal requirement in what jurisdiction, and to be enforced how and by whom? I guess the only feasible legislation here is something where the victim pursues a case with a regulating agency or just through the courts directly. But how does the victim even find the culprit when the origin of the crawling is being deliberately obscured, with traffic coming from a botnet running on exploited consumer devices?
  - socksy 2y ago
    It wouldn't have to go that deep. If we made not following robots.txt illegal in certain jurisdictions, and blocked all IP addresses not from those jurisdictions, then there would presumably have to be an entity in those jurisdictions, such as a VPN provider, an illegal botnet, or a legal botnet, and you pursue legal action with those. The VPNs and legal botnets would be heavily incentivized to not allow this to happen (and presumably already are doing traffic analysis), and illegal botnets should be shutdown anyway (some grace in the law about being unaware of it happening should of course be afforded, but once you are aware it is your responsibility to prevent your machine from committing crimes).
    - azornathogron 2y ago
      > illegal botnets should be shutdown anyway Illegal botnets aren't new. Are they currently shutdown regularly? (I'm actually asking, I don't know) > If we made not following robots.txt illegal in certain jurisdictions, and blocked all IP addresses not from those jurisdictions That sounds kinda like the balkanization of the internet. It's not without some cost. I don't mean financially, but in terms of eroding the connectedness that is supposed to be one of the internet's great benefits.
  - milch 2y ago
    Maybe people need to add deliberate traps on their websites. You could imagine a provider like Cloudflare injecting a randomly generated code phrase into thousands of sites and making sure to attribute it under a strict license, that is invisible so that no human sees it, and changes every few days. Presumably LLMs would learn this phrase and later be able to repeat it - getting a sufficiently high hit rate should be proof that they used illegitimately obtained data. Kinda like back in the old days when map makers included fake towns, rivers and so on in their maps so that if others copied it they could tell
    - deleted 2y ago
      [deleted]
- rakoo 2y ago
  Or stop defending individually and start attacking collectively ? This is an issue on scrapers side, they have to fix it > make following robots.txt a legal requirement Even better: make them publish their IP addresses so we know what to block, just like robocallers. Every scrape made from another IP is crime.
MathMonkeyMan 2y ago
Good rant! The clever LLM scrapers sound a lot like residential traffic. How does the author know it's not? Behavior? Volume? Unlikely coincidence? > random User-Agents that overlap with end-users and come from tens of thousands of IP addresses – mostly residential, in unrelated subnets, each one making no more than one HTTP request over any time period we tried to measure – actively and maliciously adapting and blending in with end-user traffic and avoiding attempts to characterize their behavior or block their traffic.
- bayindirh 2y ago
  They can be local LLMs doing search, some SETI@Home style distributed work, or else. I host my blog at Mataroa, and the stats show 2K hits for some pages on some days. I don't think my small blog and writings are that popular. Asking ChatGPT about these subjects with my name results in some hilarious hallucinations, pointing to poor training due to lack of data. IOW, my blog is also being scraped and used to train LLMs. Not nice. I really share the sentiment with Drew. I never asked/consented for this.
  - Applejinx 2y ago
    Between Microsoft and Google, my existence AND presence as a community open source developer is being scraped and stolen. I've been trying to write a body of audio code that sounds better than the stuff we got used to in the DAW era, doing things like dithering the mantissa of floating-point words, just experimental stuff ignoring the rules. Never mind if it works: I can think it does, but my objection holds whether it does or not. Firstly, if you rip my stuff halfway it's pointless: without the coordinated intention towards specific goals not corresponding with normally practiced DSP, it's useless. LLMs are not going to 'get' the intention behind what I'm doing while also blending it with the very same code I'm a reaction against, the code that greatly outnumbers my own contributions. So even if you ask it to rip me off it tries to produce a synthesis with what I'm actively avoiding, resulting in a fantasy or parody of what I'm trying to make. Secondly, suppose it became possible to make it hallucinate IN the relevant style, perhaps by training exclusively on my output, so it can spin off variations. That's not so far-fetched: _I_ do that. But where'd the style come from, that you'd spend effort tuning the LLM to? Does it initiate this on its own? Would you let it 'hallucinate' in that direction in the belief that maybe it was on to something? No, it doesn't look like that's a thing. When I've played with LLMs (I have a Mac Studio set up with enough RAM to do that) it's been trying to explore what the thing might do outside of expectation, and it's hard to get anything interesting that doesn't turn out to be a rip from something I didn't know about, but it was familiar with. Not great to go 'oh hey I made it innovate!' when you're mistakenly ripping off an unknown human's efforts. I've tried to explore what you might call 'native hallucination', stuff more inherent to collective humanity than to an individual, and I'm not seeing much facility with that. Not that people are even looking for that! And lastly, as a human trying to explore an unusual position in audio DSP code with many years of practice attempting these things and sharing them with the world around me only to have Microsoft try to reduce me to a nutrient slurry that would add a piquant flavor to 'writing code for people', I turn around and find Google, through YouTube, repeatedly offering to speak FOR me in response to my youtube commenters. I'm sure other people have seen this: probably depends on how interactive you are with your community. YouTube clearly trains a custom LLM on my comment responses to my viewers, that being text they have access to (doubtless adding my very verbose video footnotes) to the point that they're regularly offering to BE ME and save me the trouble. Including technical explanations and helpful suggestions of how to use my stuff, that's not infrequently lies and bizarro world interpretations of what's going on, plus encouraging or self-congratulatory remarks that seem partly drawn from known best practices for being an empty hype beast competing to win the algorithm. I'm not sure whether I prefer this, or the supposed promise of the machines. If it can't be any better than this, I can keep working as I am, have my intentionality and a recognizable consistent sound and style, and be full of sass and contempt for the machines, and that'll remain impossible for that world to match (whether they want to is another question… but purely in marketing terms, yes they'll want to because it'll be a distinct area to conquer once the normal stuff is all a gray paste) If it follows the path of the YouTube suggestions, there will simply be more noise out there, driven by people trying to piggyback off the mindshare of an isolated human doing a recognizable and distinct thing for most of his finite lifetime, with greater and greater volume of hollow mimicry of that person INCLUDING mimicry of his communications and interpersonal tone, the better to shunt attention and literal money to, not the LLMs doing the mimicking, but a third party working essentially in marketing, trying to split off a market segment they've identified as not only relevant, but ripe for plucking because the audience self-identifies as eager to consume the output of something that's not usual and normal. (I should learn French: that rant is structurally identical to an endlessly digressive French expostulation) Today I'm doing a livestream, coding with a small audience as I try for the fourth straight day to do a particular sort of DSP (decrackling) that's previously best served by some very expensive proprietary software costing over two thousand dollars for a license. Ideally I can get some of the results while also being able to honor my intentions for preserving the aspects of the audio I value (which I think can be compromised by such invasive DSP). That's because my intention will include this preservation, these iconoclastic details I think important, the trade-offs I think are right. Meanwhile crap is trained on my work so that a guy who wants money can harness rainforests worth of wasted electrical energy to make programs that don't even work, and a pretend scientist guru persona who can't talk coherently but can and will tell you that he is "a real audio hero who's worked for many years to give you amazing free plugins that really defy all the horrible rules that are ruining music"! Because this stuff can't pay attention, but it can draw all the wrong conclusions from your tone. And if you question your own work and learn and grow from your missteps to have greater confidence in your learned synthesis of knowledge, it can't do that either but it can simultaneously bluster with your confidence and also add 'but who knows maybe I'm totally wrong lol!' And both are forms of lies, as it has neither confidence nor self-doubt. I'm going on for longer than the original article. Sorry.
deleted 2y ago
[deleted]
colordrops 2y ago
I had a friend that said he'd never get a mobile phone, and he did hold out until maybe 2010. He eventually realized the world had changed.
- rambambram 2y ago
  > I had a friend that said he'd never get a mobile phone, and he did hold out... Up until this point you had a good start to a very inspiring story. ;)
  - drcongo 2y ago
    If it helps, I have a friend who still doesn't have a mobile phone.
    - rambambram 2y ago
      Tell me everything about it! Can he do an AMA on HN? ;)
      - JacobKfromIRC 2y ago
        I'm not who the person you replied to is talking about, but I also didn't/don't use a mobile phone for a long time. Technically I had a mobile phone during this time, but it stayed powered off in my room. Now, I carry a mobile phone around but it doesn't exactly work (no network connectivity, not even Wi-Fi or USB tethering) so I don't actually use it except for testing. I carry a laptop almost everywhere, and I even use JMP.chat for sending SMS messages, so it's functionally similar to a mobile phone, just bigger and heavier. It doesn't connect to the cell network but Wi-Fi works fine. I think there's more to a mobile phone than SMS but I'm stuggling to think of what since I don't use it. I have GPS in my car, instead of using GPS in a mobile phone. I don't take pictures very often but when I really needed to I could use my laptop webcam (built-in or USB attached), and recently I bought a camera at a garage sale that seems higher-quality than the webcam and is much more convenient. The ability to call 911 from anywhere is the main thing I think about that I'm missing, which is also one of the reasons I'd like to get a mobile phone working at some point, but it's difficult to gauge how important that is (since the need is rare). My friends might worry about this more than I do. Before college, I did use a mobile phone (but mostly for gaming and taking pictures), and I am only recently out of college, so maybe I will feel more pressure to get a mobile phone in the future, and since I'm not opposed to getting one working at some point, I probably will. I just haven't gotten around to it yet. I've never actually commented on Hacker News as far as I remember, but I'm the same JacobK as <https://codeberg.org/jacobk https://codeberg.org/jacobk> and as jacobk on Libera.Chat. I'm not a bot.
thedevilslawyer 2y ago
Was with drew until the solution: >Please stop legitimizing LLMs or AI image generators or GitHub Copilot or any of this garbage. I am begging you to stop using them, stop talking about them, stop making new ones, just stop. Anyone with any inkling of the current industry understands this is /Old-Man-Yells-At-Cloud. Why offer this as a solution even, I'm unsure. Here's potential solutions: 1) Cloudflare. 2) Own DDoS protecting reverse proxy using captcha etc. 3) Poisoning scraping results (with syntax errors in code lol), so they ditch you as a good source of data.
- deleted 2y ago
  [deleted]
- antiloper 2y ago
  Now ask me how I know that your preferred solution to not stabbed in the chest while walking on the street is to just wear stab vests wherever you go.
  - thedevilslawyer 2y ago
    I hear your idealist's take. The idealist in me agrees 100%. Here' my pragmatist comeback: Is your preferred solution to lack of democracy in middle-east to go and protest in front of the palace?
    - sussmannbaka 2y ago
      to stay with the analogy, people won’t wear the stab-proof vest or at least not stop there. People will stab back. The magnitude of the attack by bad actors in the AI space is potentially life ruining and they shouldn’t expect not to have their lives ruined in retaliation if this keeps up.
fungiblecog 2y ago
Welcome to 2025...
jisnsm 2y ago
I work for a hosting and I know what this is like. And, while I completely respect that you don’t want to give out your resources for free, a properly programmed and/or cached website wouldn’t be brought down by crawlers, no matter how aggressive. Crawlers are hitting our clients sites all the same but you only hear problems from those who have piece of shit websites that take seconds to generate.
- progval 2y ago
  git blame is always expansive to compute; and precomputing (or caching) it for every revision of every file is going to consume a lot of storage.
  - Kinrany 2y ago
    Precomputing git blame should take the same order of magnitude of storage as the repository itself. Same number of lines for each revision, same number of lines changed in every commit.
    - Kinrany 2y ago
      Should be easy to write a script that takes a branch and constructs a mirror branch with git blame output. Then compare storage space used.
    - nurettin 2y ago
      It is more fun to fight LLMs rather than trying to create magical unicorn caches that work for every endpoint.
  - benmmurphy 2y ago
    I guess for computationally expensive things the only real option is to put it behind a login. I’m sure this is something SourceHut doesn’t want to do but maybe it’s their only decent option.
    - progval 2y ago
myaccountonhn 2y ago
I really wonder how to best deal with this issue, almost seems like all web traffic needs to be behind CDNs which is horrible for the open web.
- bayindirh 2y ago
  The internet is evolving to a state that if you don't have a n level deep stack, you're cooked. Which is ugly in so many levels.
- Apreche 2y ago
  I see a future where there are many more digital gated communities. I don’t mean walled gardens in the old sense like AOL or Facebook. I mean people will create private networks with wireguard or something similar. All the services are on the inside. Discoverability is sacrificed to achieve security and privacy.
  - amarcheschi 2y ago
    Dark forest but the very thing we fear could destroy our internet gated communities is big tech/tech companies It sounds reasonable, and i'm not joking
  - Dracophoenix 2y ago
    Sounds like a BBS accessed over encrypted channels.
- JohnFen 2y ago
  I don't think that any CDN is offering adequate protection from these attacks. I wish they did. The only solution I see remains the same -- remove all your stuff from the open web. That's what I've done. I'm constantly looking for some way that I can put everything back up, but nothing has presented itself yet. Plus, if OpenAI's attempts at legalizing what they do are successful, it seems unlikely there will ever be an adequate solution.
h4kor 2y ago
I had to take offline my personal gitea instance as it was regularly spammed, crashing Caddy in the process.
- myaccountonhn 2y ago
  If you don't have public repos you can use gitolite and have ssh only access.
  - h4kor 2y ago
    I used and am now using plain git over ssh. I only hosted gitea to have my code accessible for anyone interested. My "main" projects I want to keep public are moved/cloned to GitHub, the rest I just don't bother with any longer.
kelseydh 2y ago
The internet you knew and mastered no longer exists.
- nh23423fefe 2y ago
  its cool because you ruined the internet for someone older too
  - kelseydh 2y ago
    When I think of the countless software jobs that automated other people out of work perhaps it is poetic justice that we also did it to ourselves.
sussmannbaka 2y ago
just like with crypto, AI culture is fundamentally tainted (see this very thread where people are defending this shit). Legislation hasn’t caught up and doing what’s essentially a crime (functionally this is just a DDoS) is being sold as a cool thing to do by the people who don’t foot the bill. If you ever ask yourself why everybody else hates you and your ilk, this attitude is why. The thing I don’t understand with AI though is that without the host, your parasite is nothing. Shouldn’t it be in your best interest to not kill it?
- amarcheschi 2y ago
  I would like to add that european llm projects made from openly available sources do exist and were previously discussed here on hn. They're much smaller model than the top llm other countries have made, but iirc they're made ethically (and are open source). The culture of "fuck you got mine" is something i despise on a very deep level, not just including ai but most big tech shitfuckery that has been going on since forever
- kermit___ 2y ago
  I hear you, and there are even greater concerns than the “you kids get off of my lawn” vibe, e.g. the massive amounts of water required for cooling data centers. But the “bubble” threat the post mentions is just emotional; things are accelerating so quickly that what is hype one day isn’t hype in a few months. LLM usage is just going to get heavier. What should get better are filters that prevent bots from easily scraping content. Required auth could work. Yes, it breaks things. Yes, it may kill your business. If you can’t deal with that, find another solution, which might be crowdfunding, subscription, or might be giving up and moving on. Working towards a solution makes more sense than getting angry and ranting.
  - bayindirh 2y ago
    The playing fields is very asymmetric. Can you negotiate with these guys? No. They're faceless when it comes to their data operations. Secretive, covert even. Creating a solution requires cooperation. Making these models fit on smaller systems, optimizing them to work with less resources need R&D engineering, which no company wants to spend on right now. Because hype trains need to be built, monies need to be made, moats need to be dug (which is almost impossible, BTW). My 10 year old phone can track 8 people in camera app in real time, with no lag. My A7-III can remember faces, prioritize them, focus on mammals' and humans' eyes, track them and keep them in focus at 30FPS with a small DSP. Building these things is possible, but nobody cares about it right now, because AI Datacenters live in a similar state of ZIRP economy. They're almost free for what they do, even though they harm the environment in irreparable ways just to give you something which is not right. As if people lost their mind about this thing.
me2too 2y ago
He's just right
stavros 2y ago
What's surprising to me is that established companies (OpenAI, Anthropic, etc) would resort to such aggressive scanning to extract a profit. Even faking the user agent and using residential IPs is really scummy.
- bayindirh 2y ago
  They're doing something awesome, why wouldn't they (in their own words), I ask. What they do boils my blood, honestly. Assuming that established companies are automatically ethical and just is not correct. Meta used a "laptop and a client tuned to seed as minimally as possible to torrent 81TBs of copyrighted material" to train their models. For every picture perfect and shiny facade, there's an engineer sitting on the floor in corner and hacking things to prop up this facade.
  - SSLy 2y ago
    > Meta used a "laptop and a client tuned to seed as minimally as possible to torrent 81TBs of copyrighted material" to train their models. Frankly, as a person committing copyright infringement 24/7 I have no problem with that. Especially with them releasing the weights.
    - bayindirh 2y ago
      I'll print, frame and hang this comment. I'm sure you'll have no problem with that, either.
      - SSLy 2y ago
        Be my guest.
      - lcnPylGDnU4H9OF 2y ago
        Should they have a problem with that? What problem would you have with me doing that with your comment?
throwawayffffas 2y ago
> If you personally work on developing LLMs et al, know this: I will never work with you again, and I will remember which side you picked when the bubble bursts. After using Claude code for an afternoon, I have to say I don't think this bubble is going to burst any time soon.
- hobofan 2y ago
  I think there is a good chance of the VC/Startup side of the bubble bursting. However, I think we will never go back to a time without using LLMs, given that you can run useful open-weights models on a local laptop.
  - throwawayffffas 2y ago
    Yeah I totally agree, I also run phi4 mini locally and was thoroughly impressed. The genie is out of the bottle.
    - dingnuts 2y ago
      share your settings and system specs please, I haven't seen anything come out of a local LLM that was useful. if you don't, since you're using a throwaway handle, I'll just assume you're paid to post. it is a little odd that you'd use a throwaway just to post LLM hype. is that you, Sam?
      - eitland 2y ago
        It is a 6 year old throwaway. In practice it only means it is meant to be anonymous and possible to throwaway. Or it was meant for throwaway but they kept using it.
        throwawayffffas 2y ago
        It was meant to be a throwaway but I kept it.
      - 2y ago
InDubioProRubio 2y ago
All things unregulated eventually turn into a mogadishu style eco-system where parasitism is ammunition. Open source are the plants of this brave new disgusting world. To be grazed upon by the animals, that suppressed the protection system of the plants- aka the state. Once the allmende, the grass runs out, things get interesting. We shall see Cyperpunk like computational parasitism plaguing companies and attempts to filter these work out. I guess that is the only way, to really prevent that sort of bot. You take arbitrary pieces of the UOW they want done- and reject them on principal. And depending on the cleverness of the batch algorithm, they will come back with that again and again, identifying themselves via repetition.
petercooper 2y ago
If you personally work on developing LLMs et al, know this: I will never work with you again, and I will remember which side you picked when the bubble bursts. Does this statement have ramifications for sourcehut and the type of projects allowed there? Or is it merely personal opinion?
- ransom_rs 2y ago
  At the top of the article: > This blog post is expressing personal experiences and opinions and doesn’t reflect any official policies of SourceHut.
secondcoming 2y ago
What would happen if these AI crawlers scraped data that was deliberately incorrect? Like a subtly broken algorithm implementation.
- deanc 2y ago
  They’d take that into account when relaying information. Poisoning LLMs is well documented.
- Applejinx 2y ago
  You're assuming it would make a difference: they're already just hoovering up 'usual stuff' and splicing it all sorts of ways. For these purposes there's no difference between 'deliberately incorrect' and 'just not very good at its job'. No point in taking effort to recreate what's already out there as authentic not-goodery :)
- istjohn 2y ago
  They've already scraped my Github.
deleted 2y ago
[deleted]
dalke 2y ago
I have a small static site. I haven't touched it in a couple of years. Even then, I see bot after bot, pulling down about 1/2 GB per day. Like, I distribute Python wheels from my site, with several release versions X several Python versions. I can't understand why ChatGPT, PetalBot, and other bots want to pull down wheels, much less the full contents when the header shows it hasn't changed: Last-Modified: Thu, 25 May 2023 09:07:25 GMT ETag: "8c2f67-5fc80f2f3b3e6" Well, I know the answer to the second question, as DeVault's title highlights - it's cheaper to re-read the data and re-process the content than set up a local cache. Externalizing their costs onto me. I know 1/2 GB/day is not much. It's well under the 500 GB/month I get from my hosting provider. But again, I have a tiny site with only static hosting, and as far as I can tell, the vast majority of transfers from my site are worthless. Just like accessing 'expensive endpoints like git blame, every page of every git log, and every commit in every repo' seems worthless.
- CGamesPlay 2y ago
  This is a pet peeve of Rachel by the Bay. She sets strict limits on her RSS feed based on not properly using the provided caching headers. I wonder if anyone has made a WAF that automates this sort of thing.
  - dalke 2y ago
    I'm pretty sure: "I have to review our mitigations several times per day to keep that number from getting any higher. When I do have time to work on something else, often I have to drop it when all of our alarms go off because our current set of mitigations stopped working. ... it’s not long before we’re complaining about the bots and asking if the other has cracked the code to getting rid of them once and for all." means that no one has managed it.
  - notacoward 2y ago
    Given that they're actively trying to obfuscate their activity (according to Drew's description), identifying and blocking clients seems unlikely to work. I'd be tempted to de-prioritize the more expensive types of queries (like "git blame") and set per repository limits. If a particular repository gets hit too hard, further requests for it will go on the lowest-priority queue and get really slow. That would be slightly annoying for legitimate users, but still better than random outages due to system-wide overload. BTW isn't the obfuscation of the bots' activity a tacit admission by their owners that they know they're doing something wrong and causing headaches for site admins? In the copyright world that becomes wilful infringement and carries triple damages. Maybe it should be the same for DoS perpetrators.
CGamesPlay 2y ago
I feel like I recall someone recently built a simple proof of work CAPTCHA for their personal git server. Would something like that help here? Alternatively, a technique like Privacy Pass might be useful here. Basically, give any user a handful of tokens based on reputation / proof of work, and then make each endpoint require a token to access. This gives you a single endpoint to rate limit, and doesn’t require user accounts (although you could allow known-polite user accounts a higher rate limit on minting tokens).
- akoboldfrying 2y ago
  > This gives you a single endpoint to rate limit Would you be rate-limiting by IP? Because the attacker is using (nearly) unique IPs for each request, so I don't see how that would help. > someone recently built a simple proof of work CAPTCHA for their personal git server As much as everyone hates CAPTCHAs nowadays, I think this could be helpful if it was IP-oblivious, random and the frequency was very low. E.g., once per 100 requests would be enough to hurt abusers making 10000 requests, but would cause minimal inconvenience to regular users.
  - CGamesPlay 2y ago
    I didn’t mention IP at all in my post. You can globally rate limit anonymous requests for everything (except maybe your login endpoint), if that’s the thing that makes sense for you. The nice thing about the proof-of-work approach is that it can be backgrounded for users with normal browsers, just like link prefetching.
    - akoboldfrying 2y ago
      > You can globally rate limit anonymous requests for everything Obviously you can do this, but that will start blocking everyone. How is that a solution? Also what prevents attacker-controlled browsers from backgrounding the PoW too? I took it as read that for something to qualify as a solution to this problem, it needs to affect regular users less badly than attackers.
deleted 2y ago
[deleted]
sirodoht 2y ago
I disagree with the premise of this article, which is: the internet used to be so nice when it wasn't the center of humanity's activities but now everything sucks. I read this: "I am sick and tired of having all of these costs externalized directly into my fucking face", as: "I am sick and tired of doing something for the world and the world not appreciating it because they want to do their own things". Lack of appreciation is underrated as a problem because it's abstracted away due to the mechanics of the free market ("if you don't want it, just don't buy it"). Yet, markets are quite foundational in humanity's activities and a market is place where everybody can put a stall. Now, Drew can't have his stall and he's angry. But anger is rarely part of the solution. His path is that of anger, unfortunately, with some hate added in too: > "If you personally work on developing LLMs et al, know this: I will never work with you again, and I will remember which side you picked when the bubble bursts."
- seba_dos1 2y ago
  There's a significant difference between lack of appreciation and vandalism.
- deleted 2y ago
  [deleted]
OutOfHere 2y ago
The solution here is simple: require user logins for data retrieval. Each user gets a quota as per a standard bucket algorithm. Don't overcomplicate a non-issue.
- pluto_modadic 2y ago
  requiring logins for /everything/ would close up the web. Now... requiring a login for viewing git blame (which is resource intensive) might be a good idea...
ak38Hasg 2y ago
Unsurprisingly, this submission is far removed from the front page. The people stealing and ddos-ing literally do not care. Nobel prizes are handed out for this crap. Politicians support the steal. Thanks Drew for being an honest voice.
- pas 2y ago
  it was in the top 20 for the day on https://hckrnews.com/ https://hckrnews.com/
pentaphobe 2y ago
Is there currently any way to identify ethical (or I guess, semi-ethical or even not actively unethical..) AI companies? Would be nice to see some effort on this front, a la "we don't scrape", or "we use this reputable dataset which scrapes while respecting robots.txt and reasonable caching", or heaven forbid "we train only on a dataset based on public domain content" Even if it was an empty promise (or goodwill ultimately broken) it'd be _something_. If it exists, I'd certainly prioritise any products which proclaimed it Do we need to make one?
- pentaphobe 2y ago
  (Posting as separate comment because wall of text) Also - Honestly don't even understand why so many people would need to scrape for training data. Is it naïveté (thinking it necessary), arrogance (thinking it better than others, and thus justified)? Aren't most advances now primarily focused on either higher level (agents, layered retrieval) or lower level (eg. alternatives to transformers, etc.. which would be easier to prove useful on existing datasets)? Genuine questions, all of these - if I'm off the mark I'm keen to learn!
znpy 2y ago
> All of my sysadmin friends are dealing with the same problems. I'm seeing this as well. Some of the websites my company operates suddenly go through some 10-25x in requests per minute at night. Most often the AS where the ip comes from is from some Microsoft datacenter (meaning, it's most likely OpenAI). At this point i'm starting to consider blocklisting Microsoft AS numbers and/or requiring login (or some kind of api key) when coming from a datacenter network. Reddit does this as well, already, for example. > If you personally work on developing LLMs et al, know this: I will never work with you again, and I will remember which side you picked when the bubble bursts. This sounds... childish (to put it politely) ?
flowerthoughts 2y ago
Have there been efforts to set up something like the SMTP block lists for web scraping bots? Seems like this is something that needs to be crowd sourced. It'll be much easier to find patterns in larger piles of data. A single real user is unlikely to do as much as quickly as a bot.
- johnea 2y ago
  Those are largely ineffective in the modern world. Most bot nets, like he mentions in the blag post, come from 1 IP address only 1 time. With thousands to tens of thousands of IP addresses in the bot net, there is just no way to block by IP address anymore.
closeneough 2y ago
why is this story with 200 votes not on the front page?
shark1 2y ago
We should start generating non sense information to feed up these hungry crawlers. The problem is that they will keep working to find a way to mimic human behaviour to get the real data.

258 points by Tomte 2y ago

Assuming you can prove it's a company, doesn't the behavior equate to fraud? Seems to hit all the prongs, but I am no lawyer.

bigolkevin 2y ago
It doesn't matter if it's fraud, "AI" is now considered an arms race, if we require them to fairly acquire their content we'll fall behind China or another country and then America might lose the WWIII it is constantly preparing for. You might see a couple of small players or especially egregious executives get a slap on the wrist for bad behavior but in this political climate there's no chance that Republicans or Democrats will put a stop to it.

easton 2y ago

It sucks to say, but maybe the best solution is to pay Cloudflare or another fancy web security/CDN company to get rid of this problem for you? Which is extremely dumb, but when the alternatives are “10x capacity just for stupid bots” or “hire a guy whose job it is to block LLMs”… maybe that’s cheapest? Yes, it sucks for the open web, but if it’s your livelihood I probably would consider it. (Either that or make following robots.txt a legal requirement, but that feels also like stifling hobbyists that just want to scrape a page)

azornathogron 2y ago
> Either that or make following robots.txt a legal requirement [...] A legal requirement in what jurisdiction, and to be enforced how and by whom? I guess the only feasible legislation here is something where the victim pursues a case with a regulating agency or just through the courts directly. But how does the victim even find the culprit when the origin of the crawling is being deliberately obscured, with traffic coming from a botnet running on exploited consumer devices?
- socksy 2y ago
  It wouldn't have to go that deep. If we made not following robots.txt illegal in certain jurisdictions, and blocked all IP addresses not from those jurisdictions, then there would presumably have to be an entity in those jurisdictions, such as a VPN provider, an illegal botnet, or a legal botnet, and you pursue legal action with those. The VPNs and legal botnets would be heavily incentivized to not allow this to happen (and presumably already are doing traffic analysis), and illegal botnets should be shutdown anyway (some grace in the law about being unaware of it happening should of course be afforded, but once you are aware it is your responsibility to prevent your machine from committing crimes).
  - azornathogron 2y ago
    > illegal botnets should be shutdown anyway Illegal botnets aren't new. Are they currently shutdown regularly? (I'm actually asking, I don't know) > If we made not following robots.txt illegal in certain jurisdictions, and blocked all IP addresses not from those jurisdictions That sounds kinda like the balkanization of the internet. It's not without some cost. I don't mean financially, but in terms of eroding the connectedness that is supposed to be one of the internet's great benefits.
- milch 2y ago
  Maybe people need to add deliberate traps on their websites. You could imagine a provider like Cloudflare injecting a randomly generated code phrase into thousands of sites and making sure to attribute it under a strict license, that is invisible so that no human sees it, and changes every few days. Presumably LLMs would learn this phrase and later be able to repeat it - getting a sufficiently high hit rate should be proof that they used illegitimately obtained data. Kinda like back in the old days when map makers included fake towns, rivers and so on in their maps so that if others copied it they could tell
  - deleted 2y ago
    [deleted]
rakoo 2y ago
Or stop defending individually and start attacking collectively ? This is an issue on scrapers side, they have to fix it > make following robots.txt a legal requirement Even better: make them publish their IP addresses so we know what to block, just like robocallers. Every scrape made from another IP is crime.

MathMonkeyMan 2y ago

Good rant! The clever LLM scrapers sound a lot like residential traffic. How does the author know it's not? Behavior? Volume? Unlikely coincidence? > random User-Agents that overlap with end-users and come from tens of thousands of IP addresses – mostly residential, in unrelated subnets, each one making no more than one HTTP request over any time period we tried to measure – actively and maliciously adapting and blending in with end-user traffic and avoiding attempts to characterize their behavior or block their traffic.

bayindirh 2y ago
They can be local LLMs doing search, some SETI@Home style distributed work, or else. I host my blog at Mataroa, and the stats show 2K hits for some pages on some days. I don't think my small blog and writings are that popular. Asking ChatGPT about these subjects with my name results in some hilarious hallucinations, pointing to poor training due to lack of data. IOW, my blog is also being scraped and used to train LLMs. Not nice. I really share the sentiment with Drew. I never asked/consented for this.
- Applejinx 2y ago
  Between Microsoft and Google, my existence AND presence as a community open source developer is being scraped and stolen. I've been trying to write a body of audio code that sounds better than the stuff we got used to in the DAW era, doing things like dithering the mantissa of floating-point words, just experimental stuff ignoring the rules. Never mind if it works: I can think it does, but my objection holds whether it does or not. Firstly, if you rip my stuff halfway it's pointless: without the coordinated intention towards specific goals not corresponding with normally practiced DSP, it's useless. LLMs are not going to 'get' the intention behind what I'm doing while also blending it with the very same code I'm a reaction against, the code that greatly outnumbers my own contributions. So even if you ask it to rip me off it tries to produce a synthesis with what I'm actively avoiding, resulting in a fantasy or parody of what I'm trying to make. Secondly, suppose it became possible to make it hallucinate IN the relevant style, perhaps by training exclusively on my output, so it can spin off variations. That's not so far-fetched: _I_ do that. But where'd the style come from, that you'd spend effort tuning the LLM to? Does it initiate this on its own? Would you let it 'hallucinate' in that direction in the belief that maybe it was on to something? No, it doesn't look like that's a thing. When I've played with LLMs (I have a Mac Studio set up with enough RAM to do that) it's been trying to explore what the thing might do outside of expectation, and it's hard to get anything interesting that doesn't turn out to be a rip from something I didn't know about, but it was familiar with. Not great to go 'oh hey I made it innovate!' when you're mistakenly ripping off an unknown human's efforts. I've tried to explore what you might call 'native hallucination', stuff more inherent to collective humanity than to an individual, and I'm not seeing much facility with that. Not that people are even looking for that! And lastly, as a human trying to explore an unusual position in audio DSP code with many years of practice attempting these things and sharing them with the world around me only to have Microsoft try to reduce me to a nutrient slurry that would add a piquant flavor to 'writing code for people', I turn around and find Google, through YouTube, repeatedly offering to speak FOR me in response to my youtube commenters. I'm sure other people have seen this: probably depends on how interactive you are with your community. YouTube clearly trains a custom LLM on my comment responses to my viewers, that being text they have access to (doubtless adding my very verbose video footnotes) to the point that they're regularly offering to BE ME and save me the trouble. Including technical explanations and helpful suggestions of how to use my stuff, that's not infrequently lies and bizarro world interpretations of what's going on, plus encouraging or self-congratulatory remarks that seem partly drawn from known best practices for being an empty hype beast competing to win the algorithm. I'm not sure whether I prefer this, or the supposed promise of the machines. If it can't be any better than this, I can keep working as I am, have my intentionality and a recognizable consistent sound and style, and be full of sass and contempt for the machines, and that'll remain impossible for that world to match (whether they want to is another question… but purely in marketing terms, yes they'll want to because it'll be a distinct area to conquer once the normal stuff is all a gray paste) If it follows the path of the YouTube suggestions, there will simply be more noise out there, driven by people trying to piggyback off the mindshare of an isolated human doing a recognizable and distinct thing for most of his finite lifetime, with greater and greater volume of hollow mimicry of that person INCLUDING mimicry of his communications and interpersonal tone, the better to shunt attention and literal money to, not the LLMs doing the mimicking, but a third party working essentially in marketing, trying to split off a market segment they've identified as not only relevant, but ripe for plucking because the audience self-identifies as eager to consume the output of something that's not usual and normal. (I should learn French: that rant is structurally identical to an endlessly digressive French expostulation) Today I'm doing a livestream, coding with a small audience as I try for the fourth straight day to do a particular sort of DSP (decrackling) that's previously best served by some very expensive proprietary software costing over two thousand dollars for a license. Ideally I can get some of the results while also being able to honor my intentions for preserving the aspects of the audio I value (which I think can be compromised by such invasive DSP). That's because my intention will include this preservation, these iconoclastic details I think important, the trade-offs I think are right. Meanwhile crap is trained on my work so that a guy who wants money can harness rainforests worth of wasted electrical energy to make programs that don't even work, and a pretend scientist guru persona who can't talk coherently but can and will tell you that he is "a real audio hero who's worked for many years to give you amazing free plugins that really defy all the horrible rules that are ruining music"! Because this stuff can't pay attention, but it can draw all the wrong conclusions from your tone. And if you question your own work and learn and grow from your missteps to have greater confidence in your learned synthesis of knowledge, it can't do that either but it can simultaneously bluster with your confidence and also add 'but who knows maybe I'm totally wrong lol!' And both are forms of lies, as it has neither confidence nor self-doubt. I'm going on for longer than the original article. Sorry.

deleted 2y ago

[deleted]

colordrops 2y ago

I had a friend that said he'd never get a mobile phone, and he did hold out until maybe 2010. He eventually realized the world had changed.

rambambram 2y ago
> I had a friend that said he'd never get a mobile phone, and he did hold out... Up until this point you had a good start to a very inspiring story. ;)
- drcongo 2y ago
  If it helps, I have a friend who still doesn't have a mobile phone.
  - rambambram 2y ago
    Tell me everything about it! Can he do an AMA on HN? ;)
    - JacobKfromIRC 2y ago
      I'm not who the person you replied to is talking about, but I also didn't/don't use a mobile phone for a long time. Technically I had a mobile phone during this time, but it stayed powered off in my room. Now, I carry a mobile phone around but it doesn't exactly work (no network connectivity, not even Wi-Fi or USB tethering) so I don't actually use it except for testing. I carry a laptop almost everywhere, and I even use JMP.chat for sending SMS messages, so it's functionally similar to a mobile phone, just bigger and heavier. It doesn't connect to the cell network but Wi-Fi works fine. I think there's more to a mobile phone than SMS but I'm stuggling to think of what since I don't use it. I have GPS in my car, instead of using GPS in a mobile phone. I don't take pictures very often but when I really needed to I could use my laptop webcam (built-in or USB attached), and recently I bought a camera at a garage sale that seems higher-quality than the webcam and is much more convenient. The ability to call 911 from anywhere is the main thing I think about that I'm missing, which is also one of the reasons I'd like to get a mobile phone working at some point, but it's difficult to gauge how important that is (since the need is rare). My friends might worry about this more than I do. Before college, I did use a mobile phone (but mostly for gaming and taking pictures), and I am only recently out of college, so maybe I will feel more pressure to get a mobile phone in the future, and since I'm not opposed to getting one working at some point, I probably will. I just haven't gotten around to it yet. I've never actually commented on Hacker News as far as I remember, but I'm the same JacobK as <https://codeberg.org/jacobk https://codeberg.org/jacobk> and as jacobk on Libera.Chat. I'm not a bot.

thedevilslawyer 2y ago

Was with drew until the solution: >Please stop legitimizing LLMs or AI image generators or GitHub Copilot or any of this garbage. I am begging you to stop using them, stop talking about them, stop making new ones, just stop. Anyone with any inkling of the current industry understands this is /Old-Man-Yells-At-Cloud. Why offer this as a solution even, I'm unsure. Here's potential solutions: 1) Cloudflare. 2) Own DDoS protecting reverse proxy using captcha etc. 3) Poisoning scraping results (with syntax errors in code lol), so they ditch you as a good source of data.

deleted 2y ago
[deleted]
antiloper 2y ago
Now ask me how I know that your preferred solution to not stabbed in the chest while walking on the street is to just wear stab vests wherever you go.
- thedevilslawyer 2y ago
  I hear your idealist's take. The idealist in me agrees 100%. Here' my pragmatist comeback: Is your preferred solution to lack of democracy in middle-east to go and protest in front of the palace?
  - sussmannbaka 2y ago
    to stay with the analogy, people won’t wear the stab-proof vest or at least not stop there. People will stab back. The magnitude of the attack by bad actors in the AI space is potentially life ruining and they shouldn’t expect not to have their lives ruined in retaliation if this keeps up.

fungiblecog 2y ago

Welcome to 2025...

jisnsm 2y ago

I work for a hosting and I know what this is like. And, while I completely respect that you don’t want to give out your resources for free, a properly programmed and/or cached website wouldn’t be brought down by crawlers, no matter how aggressive. Crawlers are hitting our clients sites all the same but you only hear problems from those who have piece of shit websites that take seconds to generate.

progval 2y ago
git blame is always expansive to compute; and precomputing (or caching) it for every revision of every file is going to consume a lot of storage.
- Kinrany 2y ago
  Precomputing git blame should take the same order of magnitude of storage as the repository itself. Same number of lines for each revision, same number of lines changed in every commit.
  - Kinrany 2y ago
    Should be easy to write a script that takes a branch and constructs a mirror branch with git blame output. Then compare storage space used.
  - nurettin 2y ago
    It is more fun to fight LLMs rather than trying to create magical unicorn caches that work for every endpoint.
- benmmurphy 2y ago
  I guess for computationally expensive things the only real option is to put it behind a login. I’m sure this is something SourceHut doesn’t want to do but maybe it’s their only decent option.
  - progval 2y ago

myaccountonhn 2y ago

I really wonder how to best deal with this issue, almost seems like all web traffic needs to be behind CDNs which is horrible for the open web.

bayindirh 2y ago
The internet is evolving to a state that if you don't have a n level deep stack, you're cooked. Which is ugly in so many levels.
Apreche 2y ago
I see a future where there are many more digital gated communities. I don’t mean walled gardens in the old sense like AOL or Facebook. I mean people will create private networks with wireguard or something similar. All the services are on the inside. Discoverability is sacrificed to achieve security and privacy.
- amarcheschi 2y ago
  Dark forest but the very thing we fear could destroy our internet gated communities is big tech/tech companies It sounds reasonable, and i'm not joking
- Dracophoenix 2y ago
  Sounds like a BBS accessed over encrypted channels.
JohnFen 2y ago
I don't think that any CDN is offering adequate protection from these attacks. I wish they did. The only solution I see remains the same -- remove all your stuff from the open web. That's what I've done. I'm constantly looking for some way that I can put everything back up, but nothing has presented itself yet. Plus, if OpenAI's attempts at legalizing what they do are successful, it seems unlikely there will ever be an adequate solution.

h4kor 2y ago

I had to take offline my personal gitea instance as it was regularly spammed, crashing Caddy in the process.

myaccountonhn 2y ago
If you don't have public repos you can use gitolite and have ssh only access.
- h4kor 2y ago
  I used and am now using plain git over ssh. I only hosted gitea to have my code accessible for anyone interested. My "main" projects I want to keep public are moved/cloned to GitHub, the rest I just don't bother with any longer.

kelseydh 2y ago

The internet you knew and mastered no longer exists.

nh23423fefe 2y ago
its cool because you ruined the internet for someone older too
- kelseydh 2y ago
  When I think of the countless software jobs that automated other people out of work perhaps it is poetic justice that we also did it to ourselves.

sussmannbaka 2y ago

just like with crypto, AI culture is fundamentally tainted (see this very thread where people are defending this shit). Legislation hasn’t caught up and doing what’s essentially a crime (functionally this is just a DDoS) is being sold as a cool thing to do by the people who don’t foot the bill. If you ever ask yourself why everybody else hates you and your ilk, this attitude is why. The thing I don’t understand with AI though is that without the host, your parasite is nothing. Shouldn’t it be in your best interest to not kill it?

amarcheschi 2y ago
I would like to add that european llm projects made from openly available sources do exist and were previously discussed here on hn. They're much smaller model than the top llm other countries have made, but iirc they're made ethically (and are open source). The culture of "fuck you got mine" is something i despise on a very deep level, not just including ai but most big tech shitfuckery that has been going on since forever
kermit___ 2y ago
I hear you, and there are even greater concerns than the “you kids get off of my lawn” vibe, e.g. the massive amounts of water required for cooling data centers. But the “bubble” threat the post mentions is just emotional; things are accelerating so quickly that what is hype one day isn’t hype in a few months. LLM usage is just going to get heavier. What should get better are filters that prevent bots from easily scraping content. Required auth could work. Yes, it breaks things. Yes, it may kill your business. If you can’t deal with that, find another solution, which might be crowdfunding, subscription, or might be giving up and moving on. Working towards a solution makes more sense than getting angry and ranting.
- bayindirh 2y ago
  The playing fields is very asymmetric. Can you negotiate with these guys? No. They're faceless when it comes to their data operations. Secretive, covert even. Creating a solution requires cooperation. Making these models fit on smaller systems, optimizing them to work with less resources need R&D engineering, which no company wants to spend on right now. Because hype trains need to be built, monies need to be made, moats need to be dug (which is almost impossible, BTW). My 10 year old phone can track 8 people in camera app in real time, with no lag. My A7-III can remember faces, prioritize them, focus on mammals' and humans' eyes, track them and keep them in focus at 30FPS with a small DSP. Building these things is possible, but nobody cares about it right now, because AI Datacenters live in a similar state of ZIRP economy. They're almost free for what they do, even though they harm the environment in irreparable ways just to give you something which is not right. As if people lost their mind about this thing.

me2too 2y ago

He's just right

stavros 2y ago

What's surprising to me is that established companies (OpenAI, Anthropic, etc) would resort to such aggressive scanning to extract a profit. Even faking the user agent and using residential IPs is really scummy.

bayindirh 2y ago
They're doing something awesome, why wouldn't they (in their own words), I ask. What they do boils my blood, honestly. Assuming that established companies are automatically ethical and just is not correct. Meta used a "laptop and a client tuned to seed as minimally as possible to torrent 81TBs of copyrighted material" to train their models. For every picture perfect and shiny facade, there's an engineer sitting on the floor in corner and hacking things to prop up this facade.
- SSLy 2y ago
  > Meta used a "laptop and a client tuned to seed as minimally as possible to torrent 81TBs of copyrighted material" to train their models. Frankly, as a person committing copyright infringement 24/7 I have no problem with that. Especially with them releasing the weights.
  - bayindirh 2y ago
    I'll print, frame and hang this comment. I'm sure you'll have no problem with that, either.
    - SSLy 2y ago
      Be my guest.
    - lcnPylGDnU4H9OF 2y ago
      Should they have a problem with that? What problem would you have with me doing that with your comment?

throwawayffffas 2y ago

> If you personally work on developing LLMs et al, know this: I will never work with you again, and I will remember which side you picked when the bubble bursts. After using Claude code for an afternoon, I have to say I don't think this bubble is going to burst any time soon.

hobofan 2y ago
I think there is a good chance of the VC/Startup side of the bubble bursting. However, I think we will never go back to a time without using LLMs, given that you can run useful open-weights models on a local laptop.
- throwawayffffas 2y ago
  Yeah I totally agree, I also run phi4 mini locally and was thoroughly impressed. The genie is out of the bottle.
  - dingnuts 2y ago
    share your settings and system specs please, I haven't seen anything come out of a local LLM that was useful. if you don't, since you're using a throwaway handle, I'll just assume you're paid to post. it is a little odd that you'd use a throwaway just to post LLM hype. is that you, Sam?
    - eitland 2y ago
      It is a 6 year old throwaway. In practice it only means it is meant to be anonymous and possible to throwaway. Or it was meant for throwaway but they kept using it.
      - throwawayffffas 2y ago
        It was meant to be a throwaway but I kept it.
    - 2y ago

InDubioProRubio 2y ago

All things unregulated eventually turn into a mogadishu style eco-system where parasitism is ammunition. Open source are the plants of this brave new disgusting world. To be grazed upon by the animals, that suppressed the protection system of the plants- aka the state. Once the allmende, the grass runs out, things get interesting. We shall see Cyperpunk like computational parasitism plaguing companies and attempts to filter these work out. I guess that is the only way, to really prevent that sort of bot. You take arbitrary pieces of the UOW they want done- and reject them on principal. And depending on the cleverness of the batch algorithm, they will come back with that again and again, identifying themselves via repetition.

petercooper 2y ago

If you personally work on developing LLMs et al, know this: I will never work with you again, and I will remember which side you picked when the bubble bursts. Does this statement have ramifications for sourcehut and the type of projects allowed there? Or is it merely personal opinion?

ransom_rs 2y ago
At the top of the article: > This blog post is expressing personal experiences and opinions and doesn’t reflect any official policies of SourceHut.

secondcoming 2y ago

What would happen if these AI crawlers scraped data that was deliberately incorrect? Like a subtly broken algorithm implementation.

deanc 2y ago
They’d take that into account when relaying information. Poisoning LLMs is well documented.
Applejinx 2y ago
You're assuming it would make a difference: they're already just hoovering up 'usual stuff' and splicing it all sorts of ways. For these purposes there's no difference between 'deliberately incorrect' and 'just not very good at its job'. No point in taking effort to recreate what's already out there as authentic not-goodery :)
istjohn 2y ago
They've already scraped my Github.

deleted 2y ago

[deleted]

dalke 2y ago

I have a small static site. I haven't touched it in a couple of years. Even then, I see bot after bot, pulling down about 1/2 GB per day. Like, I distribute Python wheels from my site, with several release versions X several Python versions. I can't understand why ChatGPT, PetalBot, and other bots want to pull down wheels, much less the full contents when the header shows it hasn't changed: Last-Modified: Thu, 25 May 2023 09:07:25 GMT ETag: "8c2f67-5fc80f2f3b3e6" Well, I know the answer to the second question, as DeVault's title highlights - it's cheaper to re-read the data and re-process the content than set up a local cache. Externalizing their costs onto me. I know 1/2 GB/day is not much. It's well under the 500 GB/month I get from my hosting provider. But again, I have a tiny site with only static hosting, and as far as I can tell, the vast majority of transfers from my site are worthless. Just like accessing 'expensive endpoints like git blame, every page of every git log, and every commit in every repo' seems worthless.

CGamesPlay 2y ago
This is a pet peeve of Rachel by the Bay. She sets strict limits on her RSS feed based on not properly using the provided caching headers. I wonder if anyone has made a WAF that automates this sort of thing.
- dalke 2y ago
  I'm pretty sure: "I have to review our mitigations several times per day to keep that number from getting any higher. When I do have time to work on something else, often I have to drop it when all of our alarms go off because our current set of mitigations stopped working. ... it’s not long before we’re complaining about the bots and asking if the other has cracked the code to getting rid of them once and for all." means that no one has managed it.
- notacoward 2y ago
  Given that they're actively trying to obfuscate their activity (according to Drew's description), identifying and blocking clients seems unlikely to work. I'd be tempted to de-prioritize the more expensive types of queries (like "git blame") and set per repository limits. If a particular repository gets hit too hard, further requests for it will go on the lowest-priority queue and get really slow. That would be slightly annoying for legitimate users, but still better than random outages due to system-wide overload. BTW isn't the obfuscation of the bots' activity a tacit admission by their owners that they know they're doing something wrong and causing headaches for site admins? In the copyright world that becomes wilful infringement and carries triple damages. Maybe it should be the same for DoS perpetrators.

CGamesPlay 2y ago

I feel like I recall someone recently built a simple proof of work CAPTCHA for their personal git server. Would something like that help here? Alternatively, a technique like Privacy Pass might be useful here. Basically, give any user a handful of tokens based on reputation / proof of work, and then make each endpoint require a token to access. This gives you a single endpoint to rate limit, and doesn’t require user accounts (although you could allow known-polite user accounts a higher rate limit on minting tokens).

akoboldfrying 2y ago
> This gives you a single endpoint to rate limit Would you be rate-limiting by IP? Because the attacker is using (nearly) unique IPs for each request, so I don't see how that would help. > someone recently built a simple proof of work CAPTCHA for their personal git server As much as everyone hates CAPTCHAs nowadays, I think this could be helpful if it was IP-oblivious, random and the frequency was very low. E.g., once per 100 requests would be enough to hurt abusers making 10000 requests, but would cause minimal inconvenience to regular users.
- CGamesPlay 2y ago
  I didn’t mention IP at all in my post. You can globally rate limit anonymous requests for everything (except maybe your login endpoint), if that’s the thing that makes sense for you. The nice thing about the proof-of-work approach is that it can be backgrounded for users with normal browsers, just like link prefetching.
  - akoboldfrying 2y ago
    > You can globally rate limit anonymous requests for everything Obviously you can do this, but that will start blocking everyone. How is that a solution? Also what prevents attacker-controlled browsers from backgrounding the PoW too? I took it as read that for something to qualify as a solution to this problem, it needs to affect regular users less badly than attackers.

deleted 2y ago

[deleted]

sirodoht 2y ago

I disagree with the premise of this article, which is: the internet used to be so nice when it wasn't the center of humanity's activities but now everything sucks. I read this: "I am sick and tired of having all of these costs externalized directly into my fucking face", as: "I am sick and tired of doing something for the world and the world not appreciating it because they want to do their own things". Lack of appreciation is underrated as a problem because it's abstracted away due to the mechanics of the free market ("if you don't want it, just don't buy it"). Yet, markets are quite foundational in humanity's activities and a market is place where everybody can put a stall. Now, Drew can't have his stall and he's angry. But anger is rarely part of the solution. His path is that of anger, unfortunately, with some hate added in too: > "If you personally work on developing LLMs et al, know this: I will never work with you again, and I will remember which side you picked when the bubble bursts."

seba_dos1 2y ago
There's a significant difference between lack of appreciation and vandalism.
deleted 2y ago
[deleted]

OutOfHere 2y ago

The solution here is simple: require user logins for data retrieval. Each user gets a quota as per a standard bucket algorithm. Don't overcomplicate a non-issue.

pluto_modadic 2y ago
requiring logins for /everything/ would close up the web. Now... requiring a login for viewing git blame (which is resource intensive) might be a good idea...

ak38Hasg 2y ago

Unsurprisingly, this submission is far removed from the front page. The people stealing and ddos-ing literally do not care. Nobel prizes are handed out for this crap. Politicians support the steal. Thanks Drew for being an honest voice.

pas 2y ago
it was in the top 20 for the day on https://hckrnews.com/ https://hckrnews.com/

pentaphobe 2y ago

Is there currently any way to identify ethical (or I guess, semi-ethical or even not actively unethical..) AI companies? Would be nice to see some effort on this front, a la "we don't scrape", or "we use this reputable dataset which scrapes while respecting robots.txt and reasonable caching", or heaven forbid "we train only on a dataset based on public domain content" Even if it was an empty promise (or goodwill ultimately broken) it'd be _something_. If it exists, I'd certainly prioritise any products which proclaimed it Do we need to make one?

pentaphobe 2y ago
(Posting as separate comment because wall of text) Also - Honestly don't even understand why so many people would need to scrape for training data. Is it naïveté (thinking it necessary), arrogance (thinking it better than others, and thus justified)? Aren't most advances now primarily focused on either higher level (agents, layered retrieval) or lower level (eg. alternatives to transformers, etc.. which would be easier to prove useful on existing datasets)? Genuine questions, all of these - if I'm off the mark I'm keen to learn!

znpy 2y ago

> All of my sysadmin friends are dealing with the same problems. I'm seeing this as well. Some of the websites my company operates suddenly go through some 10-25x in requests per minute at night. Most often the AS where the ip comes from is from some Microsoft datacenter (meaning, it's most likely OpenAI). At this point i'm starting to consider blocklisting Microsoft AS numbers and/or requiring login (or some kind of api key) when coming from a datacenter network. Reddit does this as well, already, for example. > If you personally work on developing LLMs et al, know this: I will never work with you again, and I will remember which side you picked when the bubble bursts. This sounds... childish (to put it politely) ?

flowerthoughts 2y ago

Have there been efforts to set up something like the SMTP block lists for web scraping bots? Seems like this is something that needs to be crowd sourced. It'll be much easier to find patterns in larger piles of data. A single real user is unlikely to do as much as quickly as a bot.

johnea 2y ago
Those are largely ineffective in the modern world. Most bot nets, like he mentions in the blag post, come from 1 IP address only 1 time. With thousands to tens of thousands of IP addresses in the bot net, there is just no way to block by IP address anymore.

closeneough 2y ago

why is this story with 200 votes not on the front page?

shark1 2y ago

We should start generating non sense information to feed up these hungry crawlers. The problem is that they will keep working to find a way to mimic human behaviour to get the real data.