3 ms·

> Does it just hijack the data from websites without referencing them? How does one hijack data? Are you referring to scraping public websites that publish dat

> Does it just hijack the data from websites without referencing them? How does one hijack data? Are you referring to scraping public websites that publish data for free? I wonder what the appropriate way to reference webscraping everything publicly available. I don’t think Google or other search engines do this, but I like the idea of listing out a massive set of trillions of URLs that have been included in the training along with a scrape date. I’m not sure Google is for mutual benefit as they summarize site content and prevent visits to that site. Would be interesting to measure the pros vs the cons there. I know there have been lawsuits over the years but don’t know the resolution.

twelve40 4y ago
yes, I'm talking about scraping. Google drives massive amounts of traffic to web publishers. It's carving out a bit for itself here and there with things like e.g. weather and answers to common questions, but they still drive a ton of traffic, and that's why (pretty much the only reason) the websites tolerate it. Just because a text on a website is "public", it doesn't mean you can copy it and plug it into your product. Nothing is "free" by default, many publishers have a deal with Google what can and cannot be used by Google, specified by robots.txt or in search console.
- prepend 4y ago
  > Just because a text on a website is "public", it doesn't mean you can copy it and plug it into your product. That’s true but you can for products like training models. The restrictions come about if the content is copyright and republished. So training a model doesn’t republish anything. If content providers don’t want everyone to access, then they should password protect or otherwise limit access. So web scraping is not “hijacking.”
  - wkat4242 4y ago
    Hmm no but offering that model as a service does have a strong possibility of infringing copyright. I think the legal complexity of this will have to be hashed out in court the next few years.

twelve40 4y ago

yes, I'm talking about scraping. Google drives massive amounts of traffic to web publishers. It's carving out a bit for itself here and there with things like e.g. weather and answers to common questions, but they still drive a ton of traffic, and that's why (pretty much the only reason) the websites tolerate it. Just because a text on a website is "public", it doesn't mean you can copy it and plug it into your product. Nothing is "free" by default, many publishers have a deal with Google what can and cannot be used by Google, specified by robots.txt or in search console.

prepend 4y ago
> Just because a text on a website is "public", it doesn't mean you can copy it and plug it into your product. That’s true but you can for products like training models. The restrictions come about if the content is copyright and republished. So training a model doesn’t republish anything. If content providers don’t want everyone to access, then they should password protect or otherwise limit access. So web scraping is not “hijacking.”
- wkat4242 4y ago
  Hmm no but offering that model as a service does have a strong possibility of infringing copyright. I think the legal complexity of this will have to be hashed out in court the next few years.