5 ms·
Reasonably speaking, there is no way they can know how they plan to invest $500 billion dollars. The current generation of large language models basically use a
by jppope 2y ago
Reasonably speaking, there is no way they can know how they plan to invest $500 billion dollars. The current generation of large language models basically use all human text thats ever been created for the parameters... not really sure where you go after than using the same tech.
- Philpax 2y agoThat's not really true - the current generation, as in "of the last three months", uses reinforcement learning to synthesize new training data for themselves: https://huggingface.co/deepseek-ai/DeepSeek-R1-Zero https://huggingface.co/deepseek-ai/DeepSeek-R1-Zero
- XorNot 2y agoRight but that's kind of the point: there's no way forward which could benefit from "moar data". In fact it's weird we need so much data now - i.e. my son in learning to talk hardly needs to have read the complete works of Shakespeare. If it's possible to produce intelligence from just ingesting text, then current tech companies have all the data they need from their initial scrapes of the internet. They don't need more. That's different to keeping models up to date on current affairs.
- throwaway4aday 2y agoThat's essentially what R1 Zero is showing: > Notably, it is the first open research to validate that reasoning capabilities of LLMs can be incentivized purely through RL, without the need for SFT.
- YetAnotherNick 2y agoO3 high compute requires 1000s of dollars to solve one medium complexity problem like ARC.
- artificialprint 2y agoLight bulbs used to be expensive too, nails as well.
- bandrami 2y agoIt worked well for the Habsburg family; what could go wrong?
- jazzyjackson 2y agoIt seems to me you could generate a lot of fresh information from running every youtube video, every hour of TV on archive.org, every movie on the pirate bay -- do scene by scene image captioning + high quality whisper transcriptions (not whatever junk auto-transcription YouTube has applied), and use that to produce screenplays of everything anyone has ever seen. I'm not sure why I've never heard of this being done, it would be a good use of GPUs in between training runs.
- airstrike 2y agoDon't forget every hour of news broadcasting, of which we likely won't run out any time soon. Plus high quality radio
- miltonlost 2y ago> a lot of fresh information from running every youtube video EVERY youtube video?? Even the 9/11 truther videos? Sandy Hook conspiracy videos? Flat earth? Even the blatantly racist? This would be some bad training data without some pruning.
- lanstin 2y agoThe best videos would be those where you accidentally start recording and you get 2 hours of naturalistic conversation between real people in reality. Not sure how often they are uploaded to YouTube. Part of the reason that kids need less material is that the aren't just listening, they are also able to do experiments to see what works and what doesn't.
- jensvdh 2y agoThe fact that OpenAI can just scrape all of Youtube and Google isn't even taking legal action or attempting to stop it is wild to me. Is Google just asleep?
- bdangubic 2y agowhat are they going to use to sue - DMCA? OpenAI (and others) are scraping everything imaginable (MS is scraping private Github repos…) - don’t think anyone in the current government will be regulating any of this anytime soon
- cavisne 2y agoThe new scaling vector is “test time compute” ie spending more compute in inference.
- riku_iki 2y agoI think there is huge amount of corporate knowledge.
- rapjr9 2y agoThe latest hype is around "agents", everyone will have agents to do things for them. The agents will incidentally collect real-time data on everything everyone uses them for. Presto! Tons of new training data. You are the product.