6 ms·
Why Google failed to make GPT-3
- robrenaud 3y agoClick bait title, but there is a lot of interesting stuff here.
- Solvency 3y agowhat would your title be?
- choppaface 3y ago"David Luan on his struggles to champion LLMs at Google" "We tried to interview David Luan about Adept but he ran out of Brain Credits"
- swyx 3y agoas a content creator who mixes reach and professional standards i mostly just subscribe to the veritasium school of ethics here https://m.youtube.com/watch?v=S2xHZPH5Sng https://m.youtube.com/watch?v=S2xHZPH5Sng
- NavinF 3y ago> during my year where I led the Google LM effort and I was one of the brain leads, you know, it became really clear why. At the time, there was a thing called the Brain Credit Marketplace. Everyone's assigned a credit. So if you have a credit, you get to buy end chips according to supply and demand. So if you want to go do a giant job, you had to convince like 19 or 20 of your colleagues not to do work wat Can anyone confirm this? Surely you can just bill all the expenses to your project and get it signed off by someone instead of collecting credits from individuals, right?
- swyx 3y agoi mean, he was there, and seems to be p confident any google brainer knows it
- BurningFrog 3y agoIs it maybe because it was impossible to buy as many AI chips as you wanted?
- choppaface 3y agoThere are only so many TPUs available and they're kept at 100% utilization. Google has had complex "marketplaces" around company resources for years: https://www.youtube.com/watch?v=3t6L-FlfeaI https://www.youtube.com/watch?v=3t6L-FlfeaI It's a cultural value, like playing board games.
- NavinF 3y agoMarketplaces for resources are great. I was questioning the use of per-person credits as opposed to per-project credits. The former seems ridiculous, especially if everyone gets the same amount
- choppaface 3y agoThe evidence indicates that the TPU marketplace caused a ton of friction and hampered Google Brain’s ability to innovate. They got merged with Deepmind after all. Google’s golden years / Bell Labs years came from plumbing ads monopoly proceeds into blue sky projects. Contrast with Bytedance / Tiktok, which ruthlessly out-competed Facebook etc in the marketplace for attention. Googlers don’t want to actually build a good product and compete for users, they just want what suffices to feed their personal nerdy interests.
- schainks 3y agoI think this googler-made meme answers your question: https://m.youtube.com/watch?v=3t6L-FlfeaI https://m.youtube.com/watch?v=3t6L-FlfeaI
- deleted 3y ago[deleted]
- xnx 3y agoSo far, Google's missed opportunity to create GPT-3 doesn't fee like much of a setback. They're at least neck-and-neck with GPT-4, and in some areas (e.g. million token context window) clearly ahead.
- ZiiS 3y agoSeems reasonable if Google had made GPT-3 when they had their own hardware, the biggest datasets, the most money, the best people, had invented a lot of the science then everyone else may have decided not to compete, and they wouldn't only be neck-and-neck with a young startup.
- swyx 3y agothey have definitely caught up and then some. still feel like there’s interesting organizational lessons here to be learned, that is less about google and just more about how to take research bets for any company, especially a large company, especially a company accused of being too comfortable/egalitarian
- xnx 3y agoIt is curious that Google was able to make moonshots in areas outside their direct expertise (balloons, glucose-reading contact lenses, etc.) but maybe not to the same degree in AI.
- swyx 3y agoi mean the order of magnitude of money going into those things was probably pocket change whereas in AI there were real billions being horse traded. not the same stakes.
- calderknight 3y agoTo be clear, the best available version of Gemini has barely caught up with what OpenAI released over a year ago. Google's a year behind. They haven't really caught up.
- aantix 3y ago
- colesantiago 3y agoone word. antitrust. Google already made transformers, other AI research, and other products but famously never commercialised it, plus they are already seen in a bad light. A competitor must emerge e.g. OpenAI in order for Google to compete and argue they aren't a monopoly otherwise they would be target to get broken up first. I believe Google were more than capable of making a GPT-3, most forget Google already had DeepMind but Google AI, Google Brain and DeepMind were separate (until they merged last year, unsurprisingly) So they were most likely both complacent and most of all didn't want to spark an investigation for a break up of Google / Alphabets businesses.
- treve 3y agoThe title wasn't an open question directed at you, the link answers this question with references and quotes from insiders.
- colesantiago 3y agoThrough only one person? Nope, what is said in there isn't enough and only shows through an implementation level of why they failed to do this. We already know Google had multiple research labs and the technical papers and had no direction to pull a GPT-3 off (hence why they merged last year as I said). So it's a moot point about insiders, the tech, etc, it's really more than that.
- jajko 3y agoDays of megacorporations getting broken are over for quite some time it seems. Governments are afraid to kill golden geese, markets react quickly and nobody wants to be seen as the guy who tanked literally everyone's pensions. Or maybe enough bribes at right places, who knows.
- dageshi 3y agoI assume because it would directly compete with search but at the same time kill the source of much of the knowledge it needs to train from (people stop making websites cause google ain't sending them traffic). So they never made a product. Now they can say they didn't kill the web, they can point to someone else.
- swyx 3y agosure at a high level but the podcast contains actual practical reasons why even the people who wanted to do it inside google couldnt
- deleted 3y ago[deleted]
- ilrwbwrkhv 3y agoCEOs who are product managers and were a mckinsey consultant and did a "MBA" are the worst people to take over an iconic tech startup.