5 ms·
OpenAI transcribed over a million hours of YouTube videos to train GPT-4
- testfrequency 3y agoStolenAI
- jumploops 3y agoGPT-2: Reddit [0] GPT-3.5: ? GPT-4: YouTube GPT-5: ??? [0]https://www.youtube.com/watch?v=13CZPWmke6A&t=3645s https://www.youtube.com/watch?v=13CZPWmke6A&t=3645s
- bredren 3y ago>GPT-5: ??? Perhaps capture of audio and video from places lacking legal expectation of privacy. Buying conversations from listening devices at live nation sports venues or clear channel airports. Spooky, but probable. Maybe there are troves of data yet to be unearthed, or needing extraction from legacy media formats. Orgs aren't desperate enough to do that kind of schlep, but soon enough. Like archives of terrestrial AM/FM or even HAM radio up for grabs. Going backward to go forward.
- speedylight 3y agoGPT-5: Classified Information
- LunaSea 3y agoGPT-5: Microsoft Office 360 customer data (Word, Excel, PowerPoint, SharePoint, Microsoft Dynamics)
- undersuit 3y agoTeams is slow for a reason. /s, but only slightly.
- column 3y agoMicrosoft says they don't train on Office 365 customer data. Maybe you don't believe them, but that's what they say.
- belter 3y agoAccording to Mozzila four lawyers, three privacy experts, and two campaigners looked at Microsoft's new Service Agreement, and none could tell if Microsoft plans on using your personal data... https://foundation.mozilla.org/en/campaigns/microsoft-ai/ https://foundation.mozilla.org/en/campaigns/microsoft-ai/ https://news.ycombinator.com/item?id=37347528 https://news.ycombinator.com/item?id=37347528
- MaxPock 3y agospotify and xvideos
- tailspin2019 3y agoPodcasts will be next
- nradov 3y agoYes, AI/LLM audio generation is going to be huge. A lot of free podcasts are already distributed as YouTube videos with the only "video" part being the host speaking into the microphone. Some podcasting companies like Acast are sitting on a gold mine of AI training content. https://www.acast.com/ https://www.acast.com/
- gentleman11 3y agoAbusive to think of peoples creative ip as a gold mine to exploit just because we live in a world where companies can change their tos at will, so everyone signs blank contracts to live daily life that say whatever the other party ever wants
- nradov 3y agoHold on. I never said that the content creators shouldn't get their fair share. But there is clearly a role for aggregators to play here as well.
- bicx 3y agoRelated: Arvid Kahl (who hosts the podcast The Bootstrapped Founder) recently launched https://podcast.fm https://podcast.fm , which essentially transcribes thousands of podcasts and makes them available for search.
- onel 3y agoThey already did that. The main reason why they created Whisper is to transcribe and train on podcast data
- blackeyeblitzar 3y agoAs a random citizen that isn’t involved in these companies, I have no problem with OpenAI doing this anymore than I would with Google doing it. I don’t see why Google should have exclusive rights to use content that others have made just because they’re protected from competition (for YouTube) via network effects and anti-competitive practices. I’m okay with the actual video creators exercising rights and controls over the use of their content, but I would argue that Google should itself not just assume rights to train on YouTube data through expansive or updated terms of service. And it seems like that’s exactly what they did - quietly assume rights to train their AI models on YouTube content without any additional compensation to content creators: > Google also gathered transcripts from YouTube, according to the Times’ sources. Bryant said that the company has trained its models “on some YouTube content, in accordance with our agreements with YouTube creators.” > The Times writes that Google’s legal department asked the company’s privacy team to tweak its policy language to expand what it could do with consumer data, such as its office tools like Google Docs. The new policy was reportedly intentionally released on July 1st to take advantage of the distraction of the Independence Day holiday weekend. There needs to be legislation to reign in the practices of big tech and also this increasingly common dark pattern of forcing updated terms on customers. They can get away with this due to a lack of feasible competition and their massive capital. Only regulation or taxation might resolve this issue so that control lies with content creators.
- IMTDb 3y ago> I don’t see why Google should have exclusive rights to use content that others have made Two perspectives: (1) because google is paying for the storage, processing and delivery of these videos, so it makes sense that they could do something with them that others can. And (2) because maybe you uploaded those video to YouTube specifically because you find Sundar Pichai vision extremely interesting so you definitely want to be on his platform and you find Sam Altman so obnoxious so you don't want him to use your content.
- Rastonbury 3y agoThat isn't a whole lot in the grand scheme of things, wonder how they picked the videos. YouTube gets like 700k hours of videos uploaded a day, and a million hours is probably like 10bn tokens out of the 13tn for GPT4
- huac 3y agoI concur english speaking rate: ~150 words per minute gpt tokens per english word: ~1.3 tokens per word 1M hours = ~12B tokens I'd fudge that up a bit because it's probably more tokens from non-English but comparatively seems like not much
- ChrisArchitect 3y agoActual article: https://www.nytimes.com/2024/04/06/technology/tech-giants-harvest-data-artificial-intelligence.html https://www.nytimes.com/2024/04/06/technology/tech-giants-ha...
- ChrisArchitect 3y agohttps://archive.is/nRUra https://archive.is/nRUra
- yieldcrv 3y agoI think the fair use argument levied is fine. They are talking about using Whisper to transcribe videos and then train the model on that. Both fair use and derivative works are satisfied there. That fulfills the actual law. The terms of service though? who cares? deactivate their google account then, they can still access the videos unauthenticated. Personally I’ve made so much money off of AI generated works that would not be protected by copyright at all, that I think creators and owners of copyrighted works should re-evaluate their relationship with copyright. It seems more like copyright gives a sense of pride and accomplishment while almost no work is being monetized by its copyright or license alone.
- manuelmoreale 3y agoJust curious, how much money did you make before AI with content that would be protected by copyright? Wondering if you’re someone who was on one side of the fence and it’s now on the other.
- yieldcrv 3y agohmm I’ve made a lot there too but I would say worrying about protecting that right and “doing it right” is more work than actually having a license agreement or court filing, both of which are rare It is liberating to be told “this can’t have copyright” and make enough money from entertaining the humans, who cares if it gets replicated and why didnt they just AI generate their own thing
- h_tbob 3y agoI don't think people should be using videos without permission. But that said, I think we should err on the side of leniency, because if we allow openai to use the data, we will allow their competitors to as well. If push comes to shove, openai can acquire all the data they need with microsoft money. But the little guys can't. So if we let them get away with it, we are really making sure the little guys can compete!
- deredede 3y agoIf we want "the little guys" to have our data we can give it to them, it's not as if their only option is to steal it.
- moscoe 3y agoIf it’s available on the open internet for anyone to consume, I don’t see a problem.
- fvv 3y agoImagine that the USA and Europe decide to avoid using data available on the Internet that doesn't fully comply with the usage policies of the data providers. This scenario raises significant questions about the future of artificial intelligence (AI) development on a global scale. If the USA and Europe take this route, what happens next? It's conceivable that China might not follow suit, opting instead to leverage all available online data to boost its AI development. This could lead to a scenario where China manages to develop a super AI, while Europe and the USA get caught in a complex maze of licenses and expensive, specialized micro-AIs. Faced with an advanced and accessible Chinese AI, what will users around the world do? It's plausible that despite initial concerns, many will end up using the Chinese AI, thereby giving it even more power, data, and growth capacity. This brings us to the importance of balanced regulation. The European Union, for instance, is often viewed as having hindered innovation with excessive regulations over the past decade. If the USA were to follow this path, they might risk limiting their own capacity for innovation. Excessive regulation thus emerges as a double-edged sword: on one hand, it aims to protect privacy and user rights; on the other hand, it risks stifling innovation and making one's companies and technologies less competitive globally. In this scenario, the question becomes: How can the USA and Europe balance data protection needs with the necessity to remain competitive in the global AI arena? Moreover, this discussion isn't just about licensing MP3s; it's about managing common knowledge—that which is accessible to everyone, including competitors in different jurisdictions—for the creation of an industrial revolution. Before clamping down with regulations, careful consideration is needed. Licensing mechanisms can be introduced later if necessary, but they should not provide an undue advantage to competitors. Balance is essential.
- Moldoteck 3y agoIt's a slippery slope. China developed one of the most sophisticated tracking systems to monitor each step be that digital or physical of it's citizens. Should EU/US do the same? IMO no, China is a 'masked' dictatorship and we must not follow every step China does. If they are detected to train their models on global data and we don't want this, simply pass even more laws to block inbound china traffic, apply more sanctions. You have written "The European Union, for instance, is often viewed as having hindered innovation with excessive regulations over the past decade" - but how do we know if that's true? Maybe its regulation didn't do much and the US growth is simply caused by the printing machine for the global currency combined with an extremely homogeneous market (all ppl speak the same language and globally a lot more ppl speak english compared to say german/french) and it's part of one country compared to EU that's a group of countries operating under some agreements but with still different laws/cultures. How do we know that specifically EU regulation is the root cause of 'less' EU innovation compared to the US and not the other things?
- banish-m4 3y agoWhich million hours though? The quality of training material appears to be the limiting bus factor for all LLMs. Maybe the future will be quality training materials from subject matter experts run in a DRM'ed environment. I think that's probably one of the ways people with specialized knowledge will make money in the future, but they should also get "residuals" based on end-user AI use.
- patrickhogan1 3y agoFascinating to think how good GPT4 is considering this is < 1% of available YouTube data. Lots of opportunities to improve the model. Future is bright.
- kristianp 3y agoIt's probably about curating quality content. Most of YouTube is dross. How did they they find a million plus YouTube videos?
- kristianp 3y agoFrom the NY Times article: > As long as you can get over the synthetic data event horizon, where the model is smart enough to make good synthetic data, everything will be fine,” Mr. Altman said. This seems intuitively incorrect to me. How can you advance the capabilities of a model by using the output of a model of the same level?
- Tenoke 3y agoHow can AlphaGo advance capabilities by using the output of a model of the same level? It's not as simple in this domain but there are ways.
- rightbyte 3y agoGo can be modeled with perfect accuracy. The feedbackloop is perfect. LLMs have no feedback loop with reality if it is short circuited with itself. It needs fresh training data.
- willj 3y agoI think that’s different. AlphaGo is using reinforcement learning in a context in which there is a clear evaluation function— did a strategy lead to a win or loss.
- Tenoke 3y agoI said it's not as simple but there are ways - e.g. you can generate more of your best quality of data, you can try to model the direction of quality or an objective, you can have minor human input at some points to judge which direction performs better, you can objectively verify some of your input to use as a partial objective - code, math, logic etc.