4 ms·

I just realized that the NSA has probably been able to train GPT-4 equivalents on _all the data_ for a while now. We'll probably never learn about it but that's

I just realized that the NSA has probably been able to train GPT-4 equivalents on _all the data_ for a while now. We'll probably never learn about it but that's maybe scarier than just the Snowden collection story because LLMs are so good at retrieval.

dwaltrip 3y ago
Holy shit, you are right. They probably have 10-100x the data used to train gpt-4. Decades of every text message, phone call transcript, and so on. I can’t believe I haven’t seen anyone mention that yet. People keep saying we don’t have enough data. I think there is a lot more data than we realize, even ignoring things like NSA.
- dwaltrip 3y ago
  Apparently there are roughly 2 trillion text messages sent per year in the US [1]. I did a sanity check, that’s like 40 or so a day per person, so sounds reasonable. I couldn’t find the average message length, but I would guess it’s fairly short (with a fat tail of longer messages). To make the math easy, let’s say the average length is ~10 tokens. I’d be surprised if that isn’t correct within a factor of 2 or so. So we have 20 trillion tokens per year from text messages in the US alone. And this is high-quality conversational data. The annual numbers were fairly constant in recent years (and then it drops off), so the past decade of US text messages is about 200 trillion tokens! That’s a metric fuck ton… Much larger than any dataset existing models have been trained on, I believe. I would guess phone transcripts would be an order of magnitude larger at least. Talking is a lot easier than typing on a phone. You could train an absolutely insane model with that amount of data… Damn. [1] https://www.statista.com/statistics/185879/number-of-text-messages-in-the-united-states-since-2005/ https://www.statista.com/statistics/185879/number-of-text-me...
  - rrrrrrrrrrrryan 3y ago
    Interestingly, the reason Google initially created it's Google Voice service back in the day was to gather voicemail audio to train its speech to text engines. It's mind-blowing to me that with all of Google's data, Google isn't the far and away leader in this new space. I have to believe they're paralyzed by the fear of legal repercussions.
- rvba 3y ago
  When chatGPT came out I asked it "which living politician is an unknown spy" and the question got blocked
- deleted 3y ago
  [deleted]

dwaltrip 3y ago

Holy shit, you are right. They probably have 10-100x the data used to train gpt-4. Decades of every text message, phone call transcript, and so on. I can’t believe I haven’t seen anyone mention that yet. People keep saying we don’t have enough data. I think there is a lot more data than we realize, even ignoring things like NSA.

dwaltrip 3y ago
Apparently there are roughly 2 trillion text messages sent per year in the US [1]. I did a sanity check, that’s like 40 or so a day per person, so sounds reasonable. I couldn’t find the average message length, but I would guess it’s fairly short (with a fat tail of longer messages). To make the math easy, let’s say the average length is ~10 tokens. I’d be surprised if that isn’t correct within a factor of 2 or so. So we have 20 trillion tokens per year from text messages in the US alone. And this is high-quality conversational data. The annual numbers were fairly constant in recent years (and then it drops off), so the past decade of US text messages is about 200 trillion tokens! That’s a metric fuck ton… Much larger than any dataset existing models have been trained on, I believe. I would guess phone transcripts would be an order of magnitude larger at least. Talking is a lot easier than typing on a phone. You could train an absolutely insane model with that amount of data… Damn. [1] https://www.statista.com/statistics/185879/number-of-text-messages-in-the-united-states-since-2005/ https://www.statista.com/statistics/185879/number-of-text-me...
- rrrrrrrrrrrryan 3y ago
  Interestingly, the reason Google initially created it's Google Voice service back in the day was to gather voicemail audio to train its speech to text engines. It's mind-blowing to me that with all of Google's data, Google isn't the far and away leader in this new space. I have to believe they're paralyzed by the fear of legal repercussions.
rvba 3y ago
When chatGPT came out I asked it "which living politician is an unknown spy" and the question got blocked
deleted 3y ago
[deleted]