6 ms·

I think media outlets think way too highly of their contribution to AI. Had they never existed, it had likely not made a dent to the AI development - completel

I think media outlets think way too highly of their contribution to AI. Had they never existed, it had likely not made a dent to the AI development - completely like believing that had they been twice as productive, it had likely neither made a dent to the quality of LLMs.

Freak_NL 7mo ago
How do you think those models get trained? You can only get so far with Wikipedia, Reddit, and non-fiction works like books and academic papers.
- RugnirViking 7mo ago
  How does the entire textual corpus of say, new York times compare to all novels? Each article is a page of text, maybe two at most? There certainly are an awful lot of articles. But it's hard to imagine it is much more than a couple hundred novels. There must be thousands of novels released each year
  - Freak_NL 7mo ago
    Like apples to oranges. LLMs are (apparently) massively used to get information about topics in the real world. Novels aren't going to be much help there. Journalism, particularly in written form, provides a fount of facts presented from different angles, as well as opinions, and it was all there free for the taking… Wikipedia provides the scantest summary of that, fora and social media give you banter, fake news, summaries of news, and a whole lot of shaky opinions, at best. Novels give you the foundations of language, but in terms of knowledge nothing much beyond what the novel is about.
    - olalonde 7mo ago
      LLMs can get up to date information from primary sources - no journalists required.
      - freedomben 7mo ago
        Primary sources can and often are, very biased. Journalists are (supposed to be) doing fact checks and gathering multiple sources from all sides. Modern journalism is in a terrible state, but still important. Imagine if all info about Facebook came from Facebook...
      - PopAlongKid 7mo ago
        I don't understand how LLMs can ask questions at a press conference.
        olalonde 7mo ago
        Startup idea right there.
        AnthonyMouse 7mo ago
        To begin with, your premise is that the only primary sources are press conferences and that press conferences only provide information in response to questions. But even taking it literally, isn't that one of the things LLMs could actually do? You're essentially asking how a text generator could generate text. The real question is whether the questions would be any good, but the answer isn't necessarily no.
        casey2 7mo ago
        I'm sure any competent agent would send an email, or just ask as an aside in a chat.
      - none2585 7mo ago
        I don't think an LLM can have secret human sources that provide them with confidential information anonymously. Not all news shows up on Twitter.
        miki123211 7mo ago
        You don't need the secret human sources any more. You used to need them, because journalists had the distribution and the sources didn't. In a word of printed newspapers, you couldn't get your story distributed nationally (much less worldwide) without the help of a journalist, doubly so if you wanted to stay anonymous. Nowadays, you just make a Substack and there's that. See that recent expose on the Delve fraud as just one example. No journalists were harmed in the making of that article.
- tossandthrow 7mo ago
  Have a look at this article: https://www.washingtonpost.com/technology/interactive/2023/ai-chatbot-learning/ https://www.washingtonpost.com/technology/interactive/2023/a... NY Times is 0.06% of common crawl. These news media outlets provide a drop in the ocean worth of information. Both qualitatively and quantitatively. The news / media industry is really just trying to hold on to their lifeboat before inevitably becoming entirely irrelevant. (I do find this sad, but it is like the reality - I can already now get considerably better journalism using LLMs than actual journalists - both click bait stuff and high quality stuff)
  - pimlottc 7mo ago
    That seems like a reductive way to consider it. What percent of music was created by Led Zeppelin? What percent of art was painted by Monet? What percent of films by Alfred Hitchcock? It may be a small percentage objectively but they are hugely influential.
    - tossandthrow 7mo ago
      I don't think back propagation care whose text it is back propagating.
      - NiloCK 7mo ago
        The data sets aren't naively fed into the training runs. Instead, training attempts to sample more heavily from higher quality sources, with, I'm sure, a mix of manual and heuristic labeling.
        ffsm8 7mo ago
        fwiw, no llm ive ever used generated in the writing style newspapers and -sites use - hence i honestly doubt they've been given a meaningful boost in relevancy. their idioms would leak occasionally otherwise
  - Gigachad 7mo ago
    90% of common crawl is complete junk. While the tiny bit of news articles powers almost all the ai answers in Google search.
- casey2 7mo ago
  By talking to users? I'm 100% sure Google and OpenAI know every major news story, in much greater detail, long before NYT publishes it. I'd imagine they already have a database of users such that if multiple people talk about a possibly true subject they can ask subject experts; users related to the subject for clarification and further information.
phatfish 7mo ago
Isn't the non-LLM generated text becoming more valuable for training as the web at large is flooded with slop? Preventing new human generated text from being used by AI firms (without consent) seems like a valid strategy.
- tossandthrow 7mo ago
  No. Modern LLMs are trained on a large percentage of synthetic data. This sentiment is largely legacy (even though just a couple of years old).
nicbou 7mo ago
Define quality. Many publications put information on the internet for the first time, or curate it for the first time, or research a topic deeper than ever before. Someone - a thinking, feeling human - had to get out there and try restaurants, talk to people, pore through archives, read books, use products. Each of them contribute a little to what we know about the world. I do this for a living. AI might soon put me out of work. It already more than halved my audience, using my own work. It's sickening to see people cheer for it because they have a bone to pick with certain websites. Eventually those websites will be gone, but so will the good ones that produced critical information.
xigoi 7mo ago
If excluding these sources wouldn’t make a difference, why do AI companies scrape them despite explicit requests to not be scraped?
- tossandthrow 7mo ago
  They want an as diverse data set as possible? It is not like they paid anybody else for their contribution either. It is just not more worth than anything else in the data sets.

Freak_NL 7mo ago

How do you think those models get trained? You can only get so far with Wikipedia, Reddit, and non-fiction works like books and academic papers.

RugnirViking 7mo ago
How does the entire textual corpus of say, new York times compare to all novels? Each article is a page of text, maybe two at most? There certainly are an awful lot of articles. But it's hard to imagine it is much more than a couple hundred novels. There must be thousands of novels released each year
- Freak_NL 7mo ago
  Like apples to oranges. LLMs are (apparently) massively used to get information about topics in the real world. Novels aren't going to be much help there. Journalism, particularly in written form, provides a fount of facts presented from different angles, as well as opinions, and it was all there free for the taking… Wikipedia provides the scantest summary of that, fora and social media give you banter, fake news, summaries of news, and a whole lot of shaky opinions, at best. Novels give you the foundations of language, but in terms of knowledge nothing much beyond what the novel is about.
  - olalonde 7mo ago
    LLMs can get up to date information from primary sources - no journalists required.
    - freedomben 7mo ago
      Primary sources can and often are, very biased. Journalists are (supposed to be) doing fact checks and gathering multiple sources from all sides. Modern journalism is in a terrible state, but still important. Imagine if all info about Facebook came from Facebook...
    - PopAlongKid 7mo ago
      I don't understand how LLMs can ask questions at a press conference.
      - olalonde 7mo ago
        Startup idea right there.
      - AnthonyMouse 7mo ago
        To begin with, your premise is that the only primary sources are press conferences and that press conferences only provide information in response to questions. But even taking it literally, isn't that one of the things LLMs could actually do? You're essentially asking how a text generator could generate text. The real question is whether the questions would be any good, but the answer isn't necessarily no.
      - casey2 7mo ago
        I'm sure any competent agent would send an email, or just ask as an aside in a chat.
    - none2585 7mo ago
      I don't think an LLM can have secret human sources that provide them with confidential information anonymously. Not all news shows up on Twitter.
      - miki123211 7mo ago
        You don't need the secret human sources any more. You used to need them, because journalists had the distribution and the sources didn't. In a word of printed newspapers, you couldn't get your story distributed nationally (much less worldwide) without the help of a journalist, doubly so if you wanted to stay anonymous. Nowadays, you just make a Substack and there's that. See that recent expose on the Delve fraud as just one example. No journalists were harmed in the making of that article.
tossandthrow 7mo ago
Have a look at this article: https://www.washingtonpost.com/technology/interactive/2023/ai-chatbot-learning/ https://www.washingtonpost.com/technology/interactive/2023/a... NY Times is 0.06% of common crawl. These news media outlets provide a drop in the ocean worth of information. Both qualitatively and quantitatively. The news / media industry is really just trying to hold on to their lifeboat before inevitably becoming entirely irrelevant. (I do find this sad, but it is like the reality - I can already now get considerably better journalism using LLMs than actual journalists - both click bait stuff and high quality stuff)
- pimlottc 7mo ago
  That seems like a reductive way to consider it. What percent of music was created by Led Zeppelin? What percent of art was painted by Monet? What percent of films by Alfred Hitchcock? It may be a small percentage objectively but they are hugely influential.
  - tossandthrow 7mo ago
    I don't think back propagation care whose text it is back propagating.
    - NiloCK 7mo ago
      The data sets aren't naively fed into the training runs. Instead, training attempts to sample more heavily from higher quality sources, with, I'm sure, a mix of manual and heuristic labeling.
      - ffsm8 7mo ago
        fwiw, no llm ive ever used generated in the writing style newspapers and -sites use - hence i honestly doubt they've been given a meaningful boost in relevancy. their idioms would leak occasionally otherwise
- Gigachad 7mo ago
  90% of common crawl is complete junk. While the tiny bit of news articles powers almost all the ai answers in Google search.
casey2 7mo ago
By talking to users? I'm 100% sure Google and OpenAI know every major news story, in much greater detail, long before NYT publishes it. I'd imagine they already have a database of users such that if multiple people talk about a possibly true subject they can ask subject experts; users related to the subject for clarification and further information.

phatfish 7mo ago

Isn't the non-LLM generated text becoming more valuable for training as the web at large is flooded with slop? Preventing new human generated text from being used by AI firms (without consent) seems like a valid strategy.

tossandthrow 7mo ago
No. Modern LLMs are trained on a large percentage of synthetic data. This sentiment is largely legacy (even though just a couple of years old).

nicbou 7mo ago

Define quality. Many publications put information on the internet for the first time, or curate it for the first time, or research a topic deeper than ever before. Someone - a thinking, feeling human - had to get out there and try restaurants, talk to people, pore through archives, read books, use products. Each of them contribute a little to what we know about the world. I do this for a living. AI might soon put me out of work. It already more than halved my audience, using my own work. It's sickening to see people cheer for it because they have a bone to pick with certain websites. Eventually those websites will be gone, but so will the good ones that produced critical information.

xigoi 7mo ago

If excluding these sources wouldn’t make a difference, why do AI companies scrape them despite explicit requests to not be scraped?

tossandthrow 7mo ago
They want an as diverse data set as possible? It is not like they paid anybody else for their contribution either. It is just not more worth than anything else in the data sets.