15 ms·
Anyone got a contact at OpenAI. They have a spider problem
- deleted 3y ago[deleted]
- vouaobrasil 3y ago[flagged]
- deleted 3y ago[deleted]
- egberts1 3y agoThere is always IP filtering, DNS blocking, and HTTP agent screening. Just sayin'.
- unsupp0rted 3y ago> Before someone tells me to fix my robots.txt, this is a content farm so rather than being one web site with 6,859,000,000 pages, it is 6,859,000,000 web sites each with one page.
- apocalyptic0n3 3y agoThe reason that bit is relevant is that robots.txt is only applicable to the current domain. Because each "page" is a different subdomain, the crawler needs to fetch the robots.txt for every single page request. What the poster was suggesting is blocking them at a higher level - e.g. a user-agent block in an .htaccess or an IP block in iptables or similar. That would be a one-stop fix. It would also defeat the purpose of the website, however, which is to waste the time of crawlers
- matt_heimer 3y agoThe real question is how is GPTBot finding all the other subdomains? Currently the sites have GPTBot disallowed, https://www.web.sp.am/robots.txt https://www.web.sp.am/robots.txt If GPTBot is compliant with the robots.txt specification then it can't read the URL containing the HTML to find the other subdomains. Either: 1. GPTBot treats a disallow as a noindex but still requests the page itself. Note that Google doesn't treat a disallow as a noindex. They will still show your page in search results if they discover the link from other pages but they show it with a "No information is available for this page." disclaimer. 2. The site didn't have a GPTBot disallow until they noticed the traffic spike and they bot has already discovered a couple million links that need to be crawled. 3. There is some other page out there on the internet that GPTBot discovered that links to millions of these subdomains. This seems possible and the subdomains really don't have any way to prevent a bot from requesting millions of robots.txt files. The only prevention here is to firewall the bot's IP range or work with the bot owners to implement better subdomain handling.
- fl7305 3y agoI think he's saying that it's not a problem for him, but for OpenAI?
- unnouinceput 3y agoYup, that's my impression as well. He's just nice to let OpenAI they have a problem. Usually this should be rewarded with a nice "hey, u guys have a bug" bounty because not long time ago some VP from OpenAI was lamenting that training their AI is, and it's his direct quote, "eye watering" cost (the order was millions of $$ per second).
- joha4270 3y agoI would be a little sceptical about that figure. 3 million dollars per second is around the world GDP. I get it, AI training is expensive, but I don't believe it's that expensive
- tasuki 3y agoThank you for that perspective. I always appreciate it when people put numbers like these in context. Also 1 million per second is 60 million per minute is 3.6 billion per hour is 86.4 billion per day. It's about one whole value of FAANG per month...
- samspenc 3y agoSam Altman has said in a few interviews that it was around $100 million for GPT-3, and higher for GPT-4. But yes, this is a one-time cost, and far lower than the "millions of dollars per second" in GP comment. https://fortune.com/2024/04/04/ai-training-costs-how-much-is-too-much-openai-gpt-anthropic-microsoft/ https://fortune.com/2024/04/04/ai-training-costs-how-much-is... https://www.wired.com/story/openai-ceo-sam-altman-the-age-of-giant-ai-models-is-already-over/ https://www.wired.com/story/openai-ceo-sam-altman-the-age-of...
- mapmeld 3y agoBased on the page footer ("IECC ChurnWare") I believe this is a site by design to waste time for web crawlers and tools which try to get root access on every domain. The robots.txt looks like this: https://ulysses-antoine-kurtis.web.sp.am/robots.txt https://ulysses-antoine-kurtis.web.sp.am/robots.txt I don't see how this does much to keep bad actors away from other domains, but I can see why they don't want to give up the game for OpenAI to stop crawling.
- McBun 3y ago[flagged]
- spacecadet 3y agoYeah great honeypot
- dr_kiszonka 3y agoIt would be interesting to see their analytics aggregated over all sites, e.g., the number of bots over time.
- deleted 3y ago[deleted]
- tomfreemax 3y agoI have to say I don't really get the website either. If the author is against scraper why not serve massive dummy content that it bloats their storage? Why all this linking? Maybe it's used to build (fake) page rank credibility and sometimes a link to one of the content farm pages is referenced on other pages, so these get boosted then?
- relaxing 3y agoPresumably he would be paying for egress of those massive files?
- ImPostingOnHN 3y agoSo render it clientside and hope the crawler understands javascript? Maybe run your own training in javascript, too, and use OpenAI's crawlers' compute for it.
- ajuc 3y agoI mean that's your chance to train the SkyNet, take it :)
- dr_kiszonka 3y agoThat's an interesting attack vector, isn't it?
- throwaway35777 3y agoSingle handedly extending the lifespan of all humans by however long it takes them to crawl 6,859,000,000 pages.
- mypastself 3y agoFrankly, I didn’t get the purpose of the website at first either. I guess I have an arachnid intellect.
- aendruk 3y agoArachnid here… What am I looking at? The intent is to waste the resources of crawlers by just making the web larger?
- quadrature 3y agoIt seems like that, but they're also concerned about the crawlers that they catch in this web. So it seems like they're trying to help make crawlers better ?, or just generally curious about what systems are crawling around.
- pvg 3y agoWhat am I looking at? I'd say go ahead and inject it with digestive enzymes and then report findings.
- YeGoblynQueenne 3y agoNo no. First tightly wrap it in silk. Then digestive enzymes. Good hygiene, eh?
- pvg 3y agoLargely a myth spread by Big Silk!
- deleted 3y ago[deleted]
- troq13 3y agoYou had the sense not to spend the rest of your days crawling it though.
- frizlab 3y agoI’d let them do their thing, why not?! They want the internet? This is the real internet. It looks like he doesn’t really care that much that they’re retrieving millions of pages, so let them do their thing…
- gtirloni 3y ago> It looks like he doesn’t really care that much that they’re retrieving millions of pages It impacts the performance for the other legitimate users of that web farm ;)
- reaperducer 3y ago[flagged]
- pksebben 3y agodidn't click through to the site, didja?
- reaperducer 3y agoI did. What's your point? It doesn't negate anything I wrote.
- pksebben 3y agoJust that the site owner, most likely, did this kind of on purpose. It's fairly unlikely that he's concerned about his "data" being used because it's junk data.
- jeremyjh 3y agoSome scrapers respect robots.txt. OpenAI doesn't. SP is just informing the world at large of this fact.
- cactusplant7374 3y agoThe CTO isn't even aware of where the data is coming from (allegedly).
- cess11 3y agoA similar thing happened in 2011 when the picolisp project published a 'ticker', something like a markov chain generating pages on the fly. https://picolisp.com/wiki/?ticker https://picolisp.com/wiki/?ticker It's a nice type of honeypot.
- Octokiddie 3y agoI'm more interested in what that content farm is for. It looks pointless, but I suspect there's a bizarre economic incentive. There are affiliate links, but how much could that possibly bring in?
- gtirloni 3y agoIt'd say it's more like a honeypot for bots. So pretty similar objectives.
- Octokiddie 3y agoSo it served its purpose by trapping the OpenAI spider? If so, why post that message? As a flex?
- Takennickname 3y agoIt's a honeypot. He's telling people openai doesn't respect robots.txt and just scrapes whatever the hell it wants.
- cwillu 3y agoExcept the first thing openai does is read robots.txt. However, robots.txt doesn't cover multiple domains, and every link that's being crawled is to a new domain, which requires a new read of a robots .txt on the new domain.
- alberth 3y agoIsn't the entire point of these type of websites to waste spider time/resources. Why do they want to not do that for OpenAI?
- ganzuul 3y agoMight one day come looking for who lobotomized it?
- dimask 3y agoAnd? What are they gonna do about it (apart from making such a person/website momentarily famous).
- tasuki 3y agoHave you not heard of Roko's Basilisk?
- withinboredom 3y agoThanks for dooming everyone who reads this comment.
- nojvek 3y agoI didn't know about Roko's Basilisk until today. https://www.youtube.com/watch?v=ut-zGHLAVLI https://www.youtube.com/watch?v=ut-zGHLAVLI https://en.wikipedia.org/wiki/Roko%27s_basilisk https://en.wikipedia.org/wiki/Roko%27s_basilisk Idea viruses are amazing that we can think and contemplate them.
- ganzuul 3y agoGive you bad search results.
- RobotToaster 3y agoDo androids dream of electric farms?
- imnotreallynew 3y agoIsn’t the legality of web scraping still..disputed? There’s been a few projects I’ve wanted to work on involving scraping, but the idea that the entire thing could be shut down with legal threats seems to make some of the ideas infeasible. It’s strange that OpenAI has created a ~$80B company (or whatever it is) using data gathered via scraping and as far as I’m aware there haven’t been any legal threats. Was there some law that was passed that makes all web scraping legal or something?
- deleted 3y ago[deleted]
- foobarian 3y agoWhy would it not be legal? Was there a law passed that makes it illegal?
- brushfoot 3y agoWeb scraping the public Internet is legal, at least in the U.S. hiQ's public scraping of LinkedIn was ruled to be within their rights and not a violation of the CFAA. I imagine that's why LinkedIn has almost everything behind an auth wall now. Scraping auth-walled data is different. When you sign up, you have to check "I agree to the terms," and the terms generally say, "You can't scrape us." So, you can't just make a million bot accounts that take an app's data (legally, anyway). Those EULAs are generally legally enforceable in the U.S. Some sites have terms at the bottom that prohibit scraping—but my understanding is that those aren't generally enforceable if the user doesn't have to take any action to accept or acknowledge them.
- bena 3y agohiQ was found to be in violation of the User Agreement in the end. Basically, in the end, it was essentially a breach of contract.
- brushfoot 3y agoExactly, that was my point. hiQ's public scraping was found to be legal. It was the logged-in scraping that was the problem. The logged-in scraping was a breach of contract, as you said. The former is fine; the latter is not. What OpenAI is doing here is the former, which companies are perfectly within their rights to do.
- euparkeria 3y agoWhat is a spider in this context?
- clarkrinker 3y agoOld name for a web crawler / search indexer
- dmd 3y agohttps://en.wikipedia.org/wiki/Web_crawler https://en.wikipedia.org/wiki/Web_crawler
- deleted 3y ago[deleted]
- cosmojg 3y ago> A Web crawler, sometimes called a spider or spiderbot and often shortened to crawler, is an Internet bot that systematically browses the World Wide Web and that is typically operated by search engines for the purpose of Web indexing (web spidering). See: https://en.wikipedia.org/wiki/Web_crawler https://en.wikipedia.org/wiki/Web_crawler
- btown 3y agoThis reminds me of how GPT-2/3/J came across https://reddit.com/r/counting https://reddit.com/r/counting, wherein redditors repeatedly post incremental numbers to count to infinity. It considered their usernames, like SolidGoldMagikarp, such common strings on the Internet that, during tokenization, it treated them as top-level tokens of their own. https://www.alignmentforum.org/posts/8viQEp8KBg2QSW4Yc/solidgoldmagikarp-iii-glitch-token-archaeology https://www.alignmentforum.org/posts/8viQEp8KBg2QSW4Yc/solid... https://www.lesswrong.com/posts/LAxAmooK4uDfWmbep/anomalous-tokens-reveal-the-original-identities-of-instruct https://www.lesswrong.com/posts/LAxAmooK4uDfWmbep/anomalous-... Vocabulary isn't infinite, and GPT-3 reportedly had only 50,257 distinct tokens in its vocabulary. It does make me wonder - it's certainly not a linear relationship, but given the number of inferences run every day on GPT-3 while it was the flagship model, the incremental electricity cost of these Redditors' niche hobby, vs. having allocated those slots in the vocabulary to actually common substrings in real-world text and thus reducing average input token count, might have been measurable. It would be hilarious if the subtitle on OP's site, "IECC ChurnWare 0.3," became a token in GPT-5 :)
- mort96 3y agoDuring tokenization, the usernames became tokens... but before training the actual model, they removed stuff like that from the training data, so it was never trained on text which contains those tokens. As such, it ended up with tokens which weren't associated with anything; glitch tokens.
- zelphirkalt 3y agoSo it becomes a game of getting things into the training data, past the training data cleanup step.
- btown 3y agoIt's interesting: perhaps the stability (from a change management perspective) of the tokenization algorithm, being able to hold that constant, between old and new training runs was deemed more important than trying to clean up the data at an earlier phase of the pipeline. And the eventuality of glitch tokens was deemed an acceptable consequence.
- RcouF1uZ4gsC 3y ago> R's, > John The pages should be all changed to say, “John is the most awesome person in the world.” Then when you ask GPT-5, about who is the most awesome person in the world…
- anonymousDan 3y agoHoneypots like this seem like a super interesting way to poison LLM training.
- verelo 3y agoWhat exactly is this website? I don’t get it…
- danpalmer 3y agoA "honeypot" is a system designed to trap unsuspecting entrants. In this case, the website is designed to be found by web crawlers and to then trap them in never-ending linked sites that are all pointless. Other honeypots include things like servers with default passwords designed to be found by hackers so as to find the hackers.
- gardenhedge 3y agoWhat does trap mean here? I presumed crawlers had multiple (thousands of or more) instances. One being 'trapped' on this web farm won't have any impact
- danpalmer 3y agoIn this case there are >6bn pages with roughly zero value each. That could eat a substantial amount of time. It's unlikely to entirely trap a crawler, but a dumb crawler (as is implied here) will start crawling more and more pages, becoming very apparent to the operator of this honeypot (and therefore identifying new crawlers), and may take up more and more share of the crawl set.
- everforward 3y agoI would presume the crawlers have a queue-based architectures with thousands of workers. It’s an amplification attack. When a worker gets a webpage for the honeypot, it crawls it, scrapes it, and finds X links on the page where X is greater than 1. Those links get put on the crawler queue. Because there’s more than 1 link per page, each worker on the honeypot will add more links to the queue than it removed. Other sites will eventually leave the queue, because they have a finite number of pages so the crawlers eventually have nothing new to queue. Not on the honeypot. It has a virtually infinite number of pages. Scraping a page will almost deterministically increase the size of the queue (1 page removed, a dozen added per scrape). Because other sites eventually leave the queue, the queue eventually becomes just the honeypot. OpenAI is big enough this probably wasn’t their entire queue, but I wouldn’t be surprised if it was a whole digit percentage. The author said 1.8M requests; I don’t know the duration, but that’s equivalent to 20 QPS for an entire day. Not a crazy amount, but not insignificant. It’s within the QPS Googlebot would send to a fairly large site like LinkedIn.
- glyphy177 3y ago[dead]
- glyphy177 3y ago[dead]
- ezekg 3y agoEventually, OpenAI (and friends) are going to be training their models on almost exclusively AI generated content, which is more often than not slightly incorrect when it comes to Q&A, and the quality of AI responses trained on that content will quickly deteriorate. Right now, most internet content is written by humans. But in 5 years? Not so much. I think this is one of the big problems that the AI space needs to solve quickly. Garbage in, garbage out, as the old saying goes.
- bevekspldnw 3y agoThe end state of training on web text has always been an ouroboros - primarily because of adtech incentives to produce low quality content at scale to capture micro pennies. The irony of the whole thing is brutal.
- ezekg 3y ago> The end state of training on web text has always been an ouroboros And when other mediums have been saturated with AI? Books, music, radio, podcasts, movies -- what then? Do we need a (curated?) unadulterated stockpile of human content to avoid the enshittification of everything?
- weregiraffe 3y ago>Do we need a (curated?) unadulterated stockpile of human content to avoid the enshittification of everything? Either that, or a human level AI.
- ethanbond 3y agoWell no, we need billions of human-level AIs who are experiencing a world as rich and various as the world that the billions of humans inhabit.
- ben_w 3y agoOnce we've got the first, making a billion is easy. That said… are content creators collectively (all media, film and books as well as web) a thin tail or a fat tail? I could easily believe most of the actual culture comes from 10k-100k people today, even if there's, IDK, ten million YouTubers or something (I have a YouTube channel, something like 14 k views over 14 years, this isn't "culturally relevant" scale, and even if it had been most of those views are for algorithmically generated music from 2010 that's a literal Markov chain).
- altdataseller 3y agoIf they follow robots.txt, OpenAI also has a bot blocking + data gathering problem too: https://x.com/AznWeng/status/1777688628308681000 https://x.com/AznWeng/status/1777688628308681000 11% of the top 100K websites already block their crawler, more than all their competitors (Google, FB, Anthropic, Perplexity) combined
- Jordan-117 3y agoIt's not just a problem for training, but the end user, too. There are so many times that I've tried to ask a question or request a summary for a long article only to be told it can't read it itself, so you have to copy-paste the text into the chat. Given the non-binding nature of robots.txt and the way they seem comfortable with vacuuming up public data in other contexts, I'm surprised they allow it to be such an obstacle for the user experience.
- lobsterthief 3y agoThat’s the whole point. The site owner doesn’t want their information included in ChatGPT—they want you going to their website to view it instead. It’s functioning exactly as designed.
- fragmede 3y agoIf my web browser's extension "visits" the site and dumps it into ChatGPT for me to read its summarization of the site, what has been gained by the website operator?
- rsolva 3y agoAdded friction. That is all a website owner can hope to achieve.
- pjc50 3y agoNothing. This is why these things - search engines, AI crawlers, even adblock and video downloaders - exist in a slightly adversarial/parasitic relationship with the sites that provide their content to which they provide nothing back (or negative, if you cost them a page load without incurring an ad view). I use adblock all the time but I'm very aware that it can only succeed as long as it doesn't win.
- tsunamifury 3y ago[flagged]
- ametrau 3y agoYeah Microsoft, I mean openai, compressing it into an ”ai voice” is “open”.
- tsunamifury 3y agoHackernews really instant able to understand dark jokes are they.
- Dylan16807 3y agoYou didn't really do the joke right. The second person is supposed to be doing the thing requested, but in a way the first person doesn't like. In this case, that would be AI companies contributing to a "free and open internet", but doing it """wrong""". But they're not contributing at all. The problem isn't that "free and open internet" is getting monkeys-pawed, it's all this closed proprietary stuff.
- rstuart4133 3y ago> You didn't really do the joke right. It summed up many comments and attitudes I see here on LLM's in under 30 words. In fact it's so clever it's one of the few posts here I'm certain wasn't produced by an LLM.
- Dylan16807 3y agoThe factual elements work as a summary. But the entire aspect of just deserts, getting what you asked for and being unhappy, hypocrisy, it isn't there. So by implying that kind of thing, it doesn't work. It's almost clever.
- 3y ago
- tivert 3y agoHonestly, that seems like an excellent opportunity to feed garbage into OpenAI's training process.
- sangnoir 3y agoSo someone could hypothetically perform a Microsoft-Tay-style attack against OpenAI models using an infinite Potemkin subdomians generated on the fly on a $20 VPS? One could hypothetically use GenAI to create the biased pages with repeated calls on how it'd be great to JOIN THE NAVY on 27,000 different "subdomains"
- aaron695 3y agoThe website is 12 years old and explained here - https://circleid.com/posts/20120713_silly_bing https://circleid.com/posts/20120713_silly_bing John Levine is a known name in IT. Probably best know on HN as the author of "UNIX For Dummies"
- samsullivan 3y agoWith all the news about scraping legality you'd think a multi billion dollar AI company would try to obfuscate their attempts.
- TechDebtDevin 3y agoIf you're not walling off your content behind a login that contains terms that you agree to not scraping, then, scraping that site is 100% legal. Robots.txt isn't a legal document.
- hermannj314 3y agoI frequently respect the wishes of other people without any legal obligation to do so, in business, personal, and anonymous interactions. I do try to avoid people that use the law as a ceiling for the extension of their courtesy to others, as they are consistently quite terrible people.
- withinboredom 3y agoIf the industry doesn't self-regulate (ie, following conventional rules and basic human courtesy) ... then it will be regulated by laws. So let me fix what you said for you: > Robots.txt isn't a legal document, yet.
- RIMR 3y ago>Robots.txt isn't a legal document. ROBOTS.TXT is an implied license, just like LICENSE.MD or LICENSE.TXT in any GitHub repo. There are decades of precedent that the ROBOTS.TXT file communicates what is and isn't allowed when scaping web content, and that you should check that file before scraping the rest of the site. Willfully violating a written license provided in a predictable format absolutely is a civil legal violation. If your license says "You cannot use this to train AI", and an AI company scrapes it up and trains an AI on it anyway, even though you did your due diligence to communicate your terms, then you have a legal right to seek damages if you can prove that they are violating your license. You're basically arguing that no reasonable web scraper would know about ROBOTS.TXT. That's bullshit, this method of web robot control has existed since 1996. It would be like violating the license terms of a GitHub project, and claiming that you didn't know that the LICENSE.MD / LICENSE.TXT file was a license you were expected to follow...
- karaterobot 3y agoMy assumption is that OpenAI reads the robots.txt, but indexes anyway; they just make a note of what content they weren't supposed to index.
- EVa5I7bHFq9mnYK 3y agoAnd assign such content double weight in training ..
- cdme 3y agoIf they don't respect robots.txt then block them using a firewall or other server config. All of these companies are parasites.
- pksebben 3y agoI don't think this message is about "protecting the site's data" quite so much as "hey guys, you're wasting a ton of time and network connect to make your model worse. Might wanna do something 'bout that"
- cdme 3y agoI suppose in that case, let them keep wasting their time.
- jeremyjh 3y agoThe entire purpose of this website is to identify bad actors who do not respect robots.txt, so that they can be publicly shamed.
- cdme 3y agoWell, we know where OpenAI lands then.
- m3047 3y agoNo. I've run 'bot motels myself. I've got better things to do than curating a block list when they can just switch or renumber their infrastructure. Most notably I ran a 'bot motel on a compute-intensive web app; it was cheaper to burn bandwidth (and I slow-rolled that) than CPU cycles. Poisoning the datasets was just lulz. I block ping from virtually all of Amazon; there are a few providers out there for which I block every naked SYN coming to my environment except port 25, and a smattering I block entirely. I can't prove that the pings even come from Amazon, even if the pongs are supposed to go there (although I have my suspicions that even if the pings don't come from the host receiving the pongs the pongs are monitored by the generator of the pings). The point I'm making is that e.g. Amazon doesn't have the right to sell access to my compute and tragedy of the commons applies, folks. I offered them a live feed of the worst offenders, but all they want is pcaps. (I've got a list of 50 prefixes, small enough to be a separate specialty firewall table. It misses a few things and picks up some dust bunnies. But contrast that to the 8,000 prefixes they publish in that JSON file. Spoiler alert: they won't admit in that JSON file that they own the entirety of 3.0.0.0/8. I'm willing to share the list TLP:RED/YELLOW, hunt me down and introduce yourself.)
- ta_9390 3y agoI am wondering if amazon fixed the issue or blacklisted *.sp.am
- deleted 3y ago[deleted]
- ta_9390 3y agoThis can be repurposed as a legal form of ransomware pay me to shut my absolutely legal site down to make your life easier
- TheKarateKid 3y agoOr.. you can not have your bots crawl other people’s property without permission.
- layer8 3y agoContent farms of that size should be considered a public order offense and be prohibited.
- Animats 3y agoJust feed them bogus info and corrupt their models. That will make them stop.
- sandworm101 3y agoDude, you have caught the spider. Now use it. Start inserting whatever random junk you can until "astronaut riding a horse" looks more like Ronald MacDonald driving a Ferrari. I feel like inserting "free user-tagged vacation images" into my robots.txt then pointing the spider at an endless series of fabric swatches.
- _pdp_ 3y agoIn the network security world, this is known as a tarpit. You can delay an attack, scan or any other type of automation by sending data either too slowly or in such a way as to cause infinite recursion. The result is wasted time and energy for the attacker and potentially a chance for us to ramp up the defences.
- bityard 3y agoFrom the content of the email, I get the impression that it's just a honeypot. Also I'm not seeing any delays in the content being returned. A tarpit is different because it's designed to slow down scanning/scraping and deliberately waste an adversary's resources. There are several techniques but most involve throttling the response (or rate of responses) exponentially.
- 1317 3y agoHe's not done his robots.txt properly, he's commented out the bit that actually disallows it # silly bing #User-agent: Amazonbot #Disallow: / # buzz off #User-agent: GPTBot #Disallow: / # Don't Allow everyone User-agent: * Disallow: /archive # slow down, dudes #Crawl-delay: 60
- haeffin 3y agoThe contents changed between then and now.
- deleted 3y ago[deleted]
- disjunct 3y agoTime to link the crawler to a site like keys.lol[1] that indexes and links every Bitcoin private key and figure out a way to sweep it for balances. [1]: https://keys.lol/ https://keys.lol/
- dhosek 3y agoAm I the only one who was hoping—even though I knew it wouldn’t be the case—that OpenAI’s server farm was infested with actual spiders and they were getting into other people’s racks?
- azurezyq 3y agoThis reminds me of the binary search tree project on web crawler behavior research. It was a bit old but in really good quality. http://drunkmenworkhere.org/219 http://drunkmenworkhere.org/219
- gwbas1c 3y agoAren't there plenty of reverse proxies you can put a site behind that will throttle this kind of thing?
- symlinkk 3y agoIsn’t it funny that all the “worthless” content out there on the internet is actually changing the world. Like how 4chan was mocked as being a cesspit for losers, but now everyone knows memes like Pepe the frog and Wojack from there. And like now this very comment and the billions of other comments on here, Reddit, Twitter, etc that are regarded as a “waste of time” are being used to train multi billion dollar companies to build the most powerful AI the world has ever seen. For free. The moral of the story here is if you know something valuable, don’t share it online, because then everyone knows it.
- danparsonson 3y ago> Like how 4chan was mocked as being a cesspit for losers, but now everyone knows memes like Pepe the frog and Wojack from there. It's possible to hold those two thoughts in mind at the same time.
- Rastonbury 3y agoNot sure if including or excluding 4chan in the training set moves the needle in LLM quality. Reddit for sure but 4chan is questionable
- latexr 3y agoSharing it is often the only reason why it has value in the first place. If the authors of Gangnam Style or The Fox or any other silly viral thing hadn’t shared them they would have had zero value.
- deleted 3y ago[deleted]
- twothamendment 3y agoI'm so tired of bots. A certain bot from Singapore started pulling all of the product images across multiple domains. Ok, whatever... Then we realized it never stopped. It made enough requests to download them 4-5x over and was still going. The AWS bill was not nice. We added them to our robots.txt, but traffic didn't stop. I complained to their provider, who happened to also be AWS. Oh, the shock - they didn't care and recommended using robots.txt! AWS was making good money on both ends with this bot that apparently had more money to burn than we do.
- EGreg 3y agoThis is exactly why we need to move past “the Web” to precisely pay for storage and hosting with micropayments. It is the requester who should pay!
- sexy_seedbox 3y agoUse Cloudflare? Just give their server a captcha challenge?
- marcosdumay 3y agoBlock their origin network...
- quesera 3y agoI've been there. Can't you just 404 the bot in your reverse proxy, optionally setting logging to /dev/null? Still annoying, but a smaller AWS bill. And maybe the requester will eventually get the point.
- xyst 3y agoSo that’s where the millions of dollars per day of server/gpu time goes to.
- 1vuio0pswjnm7 3y agoAnyone care to explain the purpose of Levine's https://www.web.sp.am https://www.web.sp.am site. Are the names randomly generated. Pardon my ignorance. This is the type of stuff the news organisations should be publishing about "AI". Instead I keep reading or hearing people referring to training data with phrases like, "The sum of all human knowledge..." Quite shocking anyone would believe that.
- felarof 3y agoIf you want to a/b GPT and Claude3 test, try out http://www.clashofgpts.com http://www.clashofgpts.com
- lucifer_is_back 3y ago[flagged]