5 ms·
It’s a shame that the knee-jerk reaction has been to outright block these bots. I think in the future, websites will learn to serve pure markdown to these bots
by andrethegiant 1y ago
It’s a shame that the knee-jerk reaction has been to outright block these bots. I think in the future, websites will learn to serve pure markdown to these bots instead of blocking. That way, websites prevent bandwidth overages like in the article, while still informing LLMs about the services their website provides.
[disclaimer: I run https://pure.md https://pure.md, which helps websites shield from this traffic]
- dmitrygr 1y agoUntil these bots become good citizens (eg: respecting robots.txt), I will be serving them gzipped gibberish that decompresses to terabytes. The ball is in their court. You don’t get to demand civility AFTER being a dick. You apologize and HOPE you’re forgiven.
- AlienRobot 1y agoI thought the AI wars would be fought with bombs vs. bots not with ZIP bombs vs. bots.
- randunel 1y agoWhat do you reckon, does OP in this post respect robots.txt or do they "borrow" content in a similar manner, without respecting such standards?
- Swizec 1y agoIf only there were some way for websites to serve information and provide interactivity in a machine readable format. Like some sort of application programming interface. You could even return different formats based on some sort of 4-letter code at the end of a URL like .html, .json, .xml, etc. And what if there was some standard sort of way for robots to tell your site what they're trying to do with some sort of verb like GET, PUT, POST, DELETE etc. They could even use a standard way to name the resource they're trying to interact with. Like a universal resource finder of some kind. You could even use identifiers to be specific! Like /items/ gives you a list of items and /items/1.json gives you data about a specific item. That would be so awesome. The future is amazing.
- marcusb 1y agoThe only thing that would make that even more perfect would be if there was some way for the site owner to signal to prospective bots which parts of the site are open to the bots to visit. I know this seems really complicated, but I really think it could be expressed in a simple text file.
- mubou 1y agoAccept: and rel="alternate" were literally made for this
- happyzappy 1y agoCool globe graphic on that site :)
- mtlynch 1y ago>I think in the future, websites will learn to serve pure markdown to these bots instead of blocking. That way, websites prevent bandwidth overages like in the article, while still informing LLMs about the services their website provides. Why? There's no value to the website for a bot scraping all of their content and then reselling it with no credit or payment to the original author.
- andrethegiant 1y agoChatGPT and others have some sort of attribution, where they link to the original webpage. How or when they decide to attribute is unclear. But websites are starting to pay attention to GEO (generative engine optimization) so that their brand isn’t entirely ignored by ChatGPT and others.
- genewitch 1y agocould have just left it as SEO and changed the S to "Slop"
- Incipient 1y agoI do agree that LLM-as-a-search is going to likely become more and more prevalent as inference gets cheaper and faster, and people don't too much care about 'minor' hallucinations. What I don't see however is any way this new way of searching will give back. There is some handwaving argument about links, however the entire value prop of an llm is you DON'T need to go to the source content.
- wongarsu 1y agoUnless you're selling something. If you have articles praising your product/service/person and "comparison" articles of the "top 10 X 2025" (your offering happens to be number one) you want the bots to find you. The LLM SEO game has only just begun. Things will only go downwards from here
- sroussey 1y agoOr technical docs. For example: https://bun.sh/llm.txt https://bun.sh/llm.txt
- tough 1y agohow would one serve them .txt instead?
- andrethegiant 1y agoAdd a Cloudflare snippet / some other edge function, and transform the response to convert to plaintext
- detaro 1y agoor you know, AI crawlers could behave and get all that without any extra work for everybody. What makes you think they'll suddenly respect your scheme?
- riffic 1y agoMarkdown over HTTPS reminds me a bit of the gemini protocol: https://en.wikipedia.org/wiki/Gemini_(protocol) https://en.wikipedia.org/wiki/Gemini_(protocol)
- pavel_lishin 1y ago> I think in the future, websites will learn to serve pure markdown to these bots instead of blocking. What for? Why would I serve anything to these leeches?
- randunel 1y agoBecause you, in this case OP, also generates bot traffic to "borrow" content from other websites to serve to their own users. Ironic, no?
- pavel_lishin 1y agoAh. I didn't realize andrethegiant's website was meant to _serve content_ to the bots, instead of _shielding us_ from the bots. I fell for a class "To Serve Man" situation.
- RamblingCTO 1y agoI think you're a bit late to the game ;) I built and sold 2markdown last year, which was then copied by firecrawl/mendable. And then you also have jina reader. Also "compare with" in the footer does nothing.