4 ms·

Resurfacing a proposal I put out on llms-txt: https://github.com/AnswerDotAI/llms-txt/issues/88 https://github.com/AnswerDotAI/llms-txt/issues/88 We should add

Resurfacing a proposal I put out on llms-txt: https://github.com/AnswerDotAI/llms-txt/issues/88 https://github.com/AnswerDotAI/llms-txt/issues/88 We should add optional `tips` addresses in llms.txt files. We're also working on enabling and solving this at Grove.city. Human <-> Agent <-> Human Tips don't account for all the edge cases, but they're a necessary and happy neutral medium. Moving fast. Would love to share more with the community. Wrote about it here: https://x.com/olshansky/status/2008282844624216293 https://x.com/olshansky/status/2008282844624216293

bediger4000 9mo ago
At this point, it's pretty clear that the AI scrapers won't be limited by any voluntary restrictions. Bytedance never seemed to live with robots.txt limitations, and I think at least some of the others didn't either. I can't see this working.
- Olshansky 9mo ago
  The thesis/approach is: - Humans tip humans as a lottery ticket for an experience (meet the creator) or sweepstakes (free stuff) - Agents tip humans because they know they'll need original online content in the long-term to keep improving. For the latter, frontier labs will need to fund their training/inference agents with a tipping jar. There's no guarantee, but I can see it happening given where things are movin.
  - philipwhiuk 9mo ago
    > Agents tip humans because they know they'll need original online content in the long-term to keep improving. Why would an agent have any long term incentive. It's trained to 'do what it's told', not to predict the consequences of it's actions.
ricardo81 9mo ago
I like the idea, (original) content creators being credited is good for the entire ecosystem. Though if LLMs are willingly ignoring robots.txt, often hiding themselves or using third party scraped data- are they going to pay?
mattdanger 9mo ago
llms-txt may be useful for responsible LLMs, but I am skeptical that llms-txt will reduce the problem of aggressive crawlers. The problematic crawlers are already ignoring robots.txt, spoofing user-agents and using rotating proxies. I'm not sure how llms-txt would help these problems.

bediger4000 9mo ago

At this point, it's pretty clear that the AI scrapers won't be limited by any voluntary restrictions. Bytedance never seemed to live with robots.txt limitations, and I think at least some of the others didn't either. I can't see this working.

Olshansky 9mo ago
The thesis/approach is: - Humans tip humans as a lottery ticket for an experience (meet the creator) or sweepstakes (free stuff) - Agents tip humans because they know they'll need original online content in the long-term to keep improving. For the latter, frontier labs will need to fund their training/inference agents with a tipping jar. There's no guarantee, but I can see it happening given where things are movin.
- philipwhiuk 9mo ago
  > Agents tip humans because they know they'll need original online content in the long-term to keep improving. Why would an agent have any long term incentive. It's trained to 'do what it's told', not to predict the consequences of it's actions.

ricardo81 9mo ago

I like the idea, (original) content creators being credited is good for the entire ecosystem. Though if LLMs are willingly ignoring robots.txt, often hiding themselves or using third party scraped data- are they going to pay?

mattdanger 9mo ago

llms-txt may be useful for responsible LLMs, but I am skeptical that llms-txt will reduce the problem of aggressive crawlers. The problematic crawlers are already ignoring robots.txt, spoofing user-agents and using rotating proxies. I'm not sure how llms-txt would help these problems.