5 ms·

What's next for models like GPT now that a lot of sites will outright block CCBot, GPTBot and others? How big of an impact is this going to have on the LLM itse

What's next for models like GPT now that a lot of sites will outright block CCBot, GPTBot and others? How big of an impact is this going to have on the LLM itself? Isn't OpenAI in a bit of a pickle in regards to this? The problem with my question is the following: Content gets syndicated anyway, so if DigitalOcean blocks GPTBot (which it does), pretty much every single one of those tutorials will be syphoned off to other sites, which are unlikely to block GPTBot themselves. How will DigitalOcean (or any other company) address this? It looks to me like it's Catch 22 in every direction you look, and unless you're someone like The New York Times who can afford to outright protect the data with licensing... It's just something thats been on my mind lately but I don't understand the finer details of it.

ChatGTP 3y ago
My guess is the big players hope is to steal an enough content and then build a self training LLM based off synthetic content (rehashed original works) before the theft part matters. Not sure how close they are to achieving but this seems to be a common SV gamble. Steal or do something shady, raise enough money / power so by the time your noticed, you have the money to win in the courts. You already see the propaganda about “none of this should matter because the cure for cancer is on the way courtesy of AI.” I mean maybe it it but it smells fishy to me.
- p1esk 3y ago
  An LLM can be trained to find relevant knowledge online. It doesn’t have to be trained on the entirety of all existing knowledge.
  - ChatGTP 3y ago
    Good luck when everything is paywalled
    - Sindisil 3y ago
      Given the already huge cost of training, and the evident lack of concern the LLM folks seem to have for copyright, why wouldn't the AI groups purchase subs to scrape the paywalled content? The would possibly need to apply some effort to appear human, but that should only throttle the rate, not stop their scraping all together.
      - ChatGTP 3y ago
        It's more difficult to scrape pay walled content no? Clearly, places like Reddit have wised up to this and are making API usages non-free for example, so while it's not impossible, you can see the limitations being put into place already. Twitter is another one. It seems like all this data is now considered gold and people lock up gold?
  - visarga 3y ago
    > An LLM can be trained to find relevant knowledge online. Why do you think chatGPT lost its Web Search plugin lately? Copyright lawsuits. You can't even use copyrighted content in the prompt because it will make the model makers liable.
    - p1esk 3y ago
      But this doesn’t make sense - how is using ChatGPT to find information different from using a search engine? Especially if ChatGPT clearly lists its sources?
chasing 3y ago
Laws. And the lawsuits that arise when such theft is discovered.
- capableweb 3y ago
  Easy to work around. Contract someone outside of the jurisdiction to provide a dataset, then it's up to them to deliver it to you. You "weren't aware" of the data source until people outside the organization starts shouting or the police starts asking questions, but the model "doesn't contain any of the data" so you continue shipping the model/product.
  - pk-protect-ai 3y ago
    You will need to train the model in the same jurisdiction too, to avoid any kind of intervention into the training process. Ideally that would be a shelter company that "sells" you "training services" in that jurisdiction.
  - thechao 3y ago
    I don't know a lot of judges, but shenanigans like that (maybe the second or third time) are a great way to get a summary judgement, and find your way to county for a few days to think about the intent of a law.
- crote 3y ago
  Who is going to enforce those laws? Big publishers are more than happy to settle with AI companies - they just their slice of the pie after all. But who is going to protect, say, your Hacker News comments? Are you going to sue the AI company? Is YC going to sue? Are you going to sue YC for not banning crawlers in their robots.txt?
  - dale_glass 3y ago
    Why would I even bother doing that? I just write comments, it's not some ineffable wisdom. I'm writing them in public. I don't expect to profit from them somehow. Heck, collecting whatever cents I might be owed for being a drop in the ocean is a losing move in my country.
drubio 3y ago
It's going to be a walled garden dystopia, with everyone and everything asking to sign-up, subscribe or whatever to monetize content. Scraping & siphoning content, ad blockers, both sides have been in arms race for years, AI and LLMs will just be the last straw, before almost anything of value is behind a pay/subscription wall.
civilitty 3y ago
Is all the data on the internet from 2010 to 2025 that much more valuable than all the data on the internet from 2010 to 2022? Better data yes, but more data up to the present day? Do we really need that to keep improving AI? GPT is already capable of incredible generalized language understanding and I'd wager we've long since hit diminishing returns from raw internet data. RLHF, fine tuning, and better (and more data efficient) architectures are what we need now.
- radarsat1 3y ago
  People have somehow conflated artificial intelligence with "oracle that knows everything" and so keeping models up to date with recent knowledge has become a must. Of course, that could be done by fine tuning techniques and better architectures that can outsource knowledge retrieval to tools, all very interesting ongoing work on that topic, but even these approaches require to not be "blocked" in their data retrieval tasks for them to work well. I'm sure you can do lots of interesting research using outdated datasets but for companies creating products this will not be sufficient.
  - photonerd 3y ago
    The “somehow” being “it was explicitly marketed that way”
- rhdunn 3y ago
  Language shifts, so NLP models need to understand those. For example, compare the use of "gay" before ~1980 and after. Some words change in spelling, like "to-morrow" present in works around 1800 changing to "tomorrow" in current usage. Some words are also coined, like "woke" or "enshitification". If your data or models don't account for those then it can make mistakes. For example, if a model is only trained on modern sources (and does not know about Early Modern English 2nd person pronouns "thy"/"thine"/etc.) then it can easily get confused when determining parts of speech, which then affects other down-stream processing.
  - cj 3y ago
meiraleal 3y ago
SPAs being difficult to crawl is a feature, after all! :)
blibble 3y ago
I did a random sample of some news sites across the political spectrum: every single one I checked, other than ft.com blocks GPTBot

ChatGTP 3y ago

My guess is the big players hope is to steal an enough content and then build a self training LLM based off synthetic content (rehashed original works) before the theft part matters. Not sure how close they are to achieving but this seems to be a common SV gamble. Steal or do something shady, raise enough money / power so by the time your noticed, you have the money to win in the courts. You already see the propaganda about “none of this should matter because the cure for cancer is on the way courtesy of AI.” I mean maybe it it but it smells fishy to me.

p1esk 3y ago
An LLM can be trained to find relevant knowledge online. It doesn’t have to be trained on the entirety of all existing knowledge.
- ChatGTP 3y ago
  Good luck when everything is paywalled
  - Sindisil 3y ago
    Given the already huge cost of training, and the evident lack of concern the LLM folks seem to have for copyright, why wouldn't the AI groups purchase subs to scrape the paywalled content? The would possibly need to apply some effort to appear human, but that should only throttle the rate, not stop their scraping all together.
    - ChatGTP 3y ago
      It's more difficult to scrape pay walled content no? Clearly, places like Reddit have wised up to this and are making API usages non-free for example, so while it's not impossible, you can see the limitations being put into place already. Twitter is another one. It seems like all this data is now considered gold and people lock up gold?
- visarga 3y ago
  > An LLM can be trained to find relevant knowledge online. Why do you think chatGPT lost its Web Search plugin lately? Copyright lawsuits. You can't even use copyrighted content in the prompt because it will make the model makers liable.
  - p1esk 3y ago
    But this doesn’t make sense - how is using ChatGPT to find information different from using a search engine? Especially if ChatGPT clearly lists its sources?

chasing 3y ago

Laws. And the lawsuits that arise when such theft is discovered.

capableweb 3y ago
Easy to work around. Contract someone outside of the jurisdiction to provide a dataset, then it's up to them to deliver it to you. You "weren't aware" of the data source until people outside the organization starts shouting or the police starts asking questions, but the model "doesn't contain any of the data" so you continue shipping the model/product.
- pk-protect-ai 3y ago
  You will need to train the model in the same jurisdiction too, to avoid any kind of intervention into the training process. Ideally that would be a shelter company that "sells" you "training services" in that jurisdiction.
- thechao 3y ago
  I don't know a lot of judges, but shenanigans like that (maybe the second or third time) are a great way to get a summary judgement, and find your way to county for a few days to think about the intent of a law.
crote 3y ago
Who is going to enforce those laws? Big publishers are more than happy to settle with AI companies - they just their slice of the pie after all. But who is going to protect, say, your Hacker News comments? Are you going to sue the AI company? Is YC going to sue? Are you going to sue YC for not banning crawlers in their robots.txt?
- dale_glass 3y ago
  Why would I even bother doing that? I just write comments, it's not some ineffable wisdom. I'm writing them in public. I don't expect to profit from them somehow. Heck, collecting whatever cents I might be owed for being a drop in the ocean is a losing move in my country.

drubio 3y ago

It's going to be a walled garden dystopia, with everyone and everything asking to sign-up, subscribe or whatever to monetize content. Scraping & siphoning content, ad blockers, both sides have been in arms race for years, AI and LLMs will just be the last straw, before almost anything of value is behind a pay/subscription wall.

civilitty 3y ago

Is all the data on the internet from 2010 to 2025 that much more valuable than all the data on the internet from 2010 to 2022? Better data yes, but more data up to the present day? Do we really need that to keep improving AI? GPT is already capable of incredible generalized language understanding and I'd wager we've long since hit diminishing returns from raw internet data. RLHF, fine tuning, and better (and more data efficient) architectures are what we need now.

radarsat1 3y ago
People have somehow conflated artificial intelligence with "oracle that knows everything" and so keeping models up to date with recent knowledge has become a must. Of course, that could be done by fine tuning techniques and better architectures that can outsource knowledge retrieval to tools, all very interesting ongoing work on that topic, but even these approaches require to not be "blocked" in their data retrieval tasks for them to work well. I'm sure you can do lots of interesting research using outdated datasets but for companies creating products this will not be sufficient.
- photonerd 3y ago
  The “somehow” being “it was explicitly marketed that way”
rhdunn 3y ago
Language shifts, so NLP models need to understand those. For example, compare the use of "gay" before ~1980 and after. Some words change in spelling, like "to-morrow" present in works around 1800 changing to "tomorrow" in current usage. Some words are also coined, like "woke" or "enshitification". If your data or models don't account for those then it can make mistakes. For example, if a model is only trained on modern sources (and does not know about Early Modern English 2nd person pronouns "thy"/"thine"/etc.) then it can easily get confused when determining parts of speech, which then affects other down-stream processing.
- cj 3y ago

meiraleal 3y ago

SPAs being difficult to crawl is a feature, after all! :)

blibble 3y ago

I did a random sample of some news sites across the political spectrum: every single one I checked, other than ft.com blocks GPTBot