4 ms·

there are a bunch of unclear questions about production readiness of this as-is. where does chatgpt get its data? Does it just hijack the data from websites wi

there are a bunch of unclear questions about production readiness of this as-is. where does chatgpt get its data? Does it just hijack the data from websites without referencing them? might work for a limited dataset (commercial or loose licences), but musk already said he won't let it use the Twitter data, for example. Google search is for mutual benefit to publishers that provide the data because it drives traffic. This thing doesn't drive anything, so if I as a publisher found my data in it, I would be getting ready to sue them. there are other questions like, how to you continually retrain as you crawl the bigger web? on a really large number of sources, how do you rank and filter out crap spam from the actual useful, trainable content? so way too early to kill regular search imo.

prepend 4y ago
> Does it just hijack the data from websites without referencing them? How does one hijack data? Are you referring to scraping public websites that publish data for free? I wonder what the appropriate way to reference webscraping everything publicly available. I don’t think Google or other search engines do this, but I like the idea of listing out a massive set of trillions of URLs that have been included in the training along with a scrape date. I’m not sure Google is for mutual benefit as they summarize site content and prevent visits to that site. Would be interesting to measure the pros vs the cons there. I know there have been lawsuits over the years but don’t know the resolution.
- twelve40 4y ago
  yes, I'm talking about scraping. Google drives massive amounts of traffic to web publishers. It's carving out a bit for itself here and there with things like e.g. weather and answers to common questions, but they still drive a ton of traffic, and that's why (pretty much the only reason) the websites tolerate it. Just because a text on a website is "public", it doesn't mean you can copy it and plug it into your product. Nothing is "free" by default, many publishers have a deal with Google what can and cannot be used by Google, specified by robots.txt or in search console.
  - prepend 4y ago
    > Just because a text on a website is "public", it doesn't mean you can copy it and plug it into your product. That’s true but you can for products like training models. The restrictions come about if the content is copyright and republished. So training a model doesn’t republish anything. If content providers don’t want everyone to access, then they should password protect or otherwise limit access. So web scraping is not “hijacking.”
    - wkat4242 4y ago
      Hmm no but offering that model as a service does have a strong possibility of infringing copyright. I think the legal complexity of this will have to be hashed out in court the next few years.

prepend 4y ago

> Does it just hijack the data from websites without referencing them? How does one hijack data? Are you referring to scraping public websites that publish data for free? I wonder what the appropriate way to reference webscraping everything publicly available. I don’t think Google or other search engines do this, but I like the idea of listing out a massive set of trillions of URLs that have been included in the training along with a scrape date. I’m not sure Google is for mutual benefit as they summarize site content and prevent visits to that site. Would be interesting to measure the pros vs the cons there. I know there have been lawsuits over the years but don’t know the resolution.

twelve40 4y ago
yes, I'm talking about scraping. Google drives massive amounts of traffic to web publishers. It's carving out a bit for itself here and there with things like e.g. weather and answers to common questions, but they still drive a ton of traffic, and that's why (pretty much the only reason) the websites tolerate it. Just because a text on a website is "public", it doesn't mean you can copy it and plug it into your product. Nothing is "free" by default, many publishers have a deal with Google what can and cannot be used by Google, specified by robots.txt or in search console.
- prepend 4y ago
  > Just because a text on a website is "public", it doesn't mean you can copy it and plug it into your product. That’s true but you can for products like training models. The restrictions come about if the content is copyright and republished. So training a model doesn’t republish anything. If content providers don’t want everyone to access, then they should password protect or otherwise limit access. So web scraping is not “hijacking.”
  - wkat4242 4y ago
    Hmm no but offering that model as a service does have a strong possibility of infringing copyright. I think the legal complexity of this will have to be hashed out in court the next few years.