3 ms·

Because it started in 2022 and hasn't subsided since? This is just the latest iteration of "AI" scrapers destroying the site, and the worst one yet. https://lw

Because it started in 2022 and hasn't subsided since? This is just the latest iteration of "AI" scrapers destroying the site, and the worst one yet. https://lwn.net/Articles/1008897 https://lwn.net/Articles/1008897 Your nonsense about LWN being a "newsletter" and having "zero valuable data" isn't doing you any favors. It is the prime source of information about Linux kernel development, and Linux development in general. "AI" cancer scraping the same thing over and over and over again is not news for anybody even with a cursory interest in this subject. They've been doing it for years.

NitpickLawyer 9mo ago
> LWN.net is a reader-supported news site I mean... Again, the site is so old that anything worth while is already in cc or any number of crawls. I am not saying they weren't scraped. I'm saying they likely weren't scraped by the bad AI people. And certainly not by AI companies trying to limit others from accessing that data (as the person who I replied to stated).
- MBCook 9mo ago
  Why is it each of your comments seems to include a dig attacking LWN?
- spinningslate 9mo ago
  I’m going to presume good faith rather than trolling. Some questions for you: 1. Coding assistants have emerged as as one of the primary commercial opportunities for AI models. As GP pointed out, LWN is the primary discussion for kernel development. If you were gathering training data for a model, and coding assistance is one of your goals, and you know of a primary sources of open source development expertise, would you: (a) ignore it because it’s in a quaint old format, or (b) slurp up as much as you can? 2. If you’d previously slurped it up, and are now collating data for a new training run, and you know it’s an active mailing list that will have new content since you last crawled it, would you: (a) carefully and respectfully leave it be, because you still get benefit from the previous content even though there’s now more and it’s up to date, or (b) hoover up every last drop because anything you can do to get an edge over your competitors means you get your brief moment of glory in the benchmarks when you release?
  - NitpickLawyer 9mo ago
    I train coding models with RLVR because that's what works. There's ~0.000x good signal in mailing lists that isn't in old mailing lists. (and, since I can't reply to the other person, I mean old as in established, it is in no way a dig to lwn). You seem to be missing my point. There is 0 incentives for AI training companies to behave like this. All that data is already in the common crawls that every lab uses. This is likely from other sources. Yet they always blame big bad AI...
    - dzaima 9mo ago
      Old scrapes can't have data about new things though; have to continously re-scan to not be stuck with ancient info. some scrapers might skip out on already-scraped sources, but easy to imagine that some/many just would not bother (you don't know if it's updated until you've checked, after all). And to some extend you do have to re-scrape, if just to find links to the new stuff.

NitpickLawyer 9mo ago

> LWN.net is a reader-supported news site I mean... Again, the site is so old that anything worth while is already in cc or any number of crawls. I am not saying they weren't scraped. I'm saying they likely weren't scraped by the bad AI people. And certainly not by AI companies trying to limit others from accessing that data (as the person who I replied to stated).

MBCook 9mo ago
Why is it each of your comments seems to include a dig attacking LWN?
spinningslate 9mo ago
I’m going to presume good faith rather than trolling. Some questions for you: 1. Coding assistants have emerged as as one of the primary commercial opportunities for AI models. As GP pointed out, LWN is the primary discussion for kernel development. If you were gathering training data for a model, and coding assistance is one of your goals, and you know of a primary sources of open source development expertise, would you: (a) ignore it because it’s in a quaint old format, or (b) slurp up as much as you can? 2. If you’d previously slurped it up, and are now collating data for a new training run, and you know it’s an active mailing list that will have new content since you last crawled it, would you: (a) carefully and respectfully leave it be, because you still get benefit from the previous content even though there’s now more and it’s up to date, or (b) hoover up every last drop because anything you can do to get an edge over your competitors means you get your brief moment of glory in the benchmarks when you release?
- NitpickLawyer 9mo ago
  I train coding models with RLVR because that's what works. There's ~0.000x good signal in mailing lists that isn't in old mailing lists. (and, since I can't reply to the other person, I mean old as in established, it is in no way a dig to lwn). You seem to be missing my point. There is 0 incentives for AI training companies to behave like this. All that data is already in the common crawls that every lab uses. This is likely from other sources. Yet they always blame big bad AI...
  - dzaima 9mo ago
    Old scrapes can't have data about new things though; have to continously re-scan to not be stuck with ancient info. some scrapers might skip out on already-scraped sources, but easy to imagine that some/many just would not bother (you don't know if it's updated until you've checked, after all). And to some extend you do have to re-scrape, if just to find links to the new stuff.