12 ms·
Top 6 science guys are long gone. Open AI is run by marketing, business, software and productization people. When the next wave of new deep learning innovation
by nabla9 2y ago
Top 6 science guys are long gone. Open AI is run by marketing, business, software and productization people.
When the next wave of new deep learning innovations sweeps the world, Microsoft eats whats left of them. They make lots of money, but don't have future unless they replace what they lost.
- fnordpiglet 2y agoI don’t feel that OpenAI has a huge moat against say Anthropic. And I don’t know OpenAI needs Microsoft nearly as much as Microsoft needs OpenAI
- gunalx 2y agoI Don't know. Being able to get azure credits has payed out really well for openai as a business in constant need of computer.
- fnordpiglet 2y agoWhich is a very short term advantage. And Anthropic gets aws credits which would you rather have?
- Closi 2y agoNever discount the value of short term advantages. Being first at the start (i.e. first mover advantage) is huge.
- luma 2y agoGiven Amazon's no show in the AI space? Azure. By a mile.
- cthalupa 2y agoExcept if you're Anthropic or OpenAI you don't care about what your compute provider has done in the AI space - you care about the compute power they can give you.
- luma 2y agoThat's exactly what I'm talking about: https://i.imgur.com/sZ3tniY.jpeg https://i.imgur.com/sZ3tniY.jpeg
- cthalupa 2y agoBut how many of those are ordered specifically for OpenAI, and are on order as a result of them to begin with? Do you think if we were in a parallel universe where OpenAI ended up partnering with Google or Amazon instead, the GPU shipments would look the same? I think they would reflect wherever OpenAI ended up doing all their compute showing a pretty similar lion's share. Your claim was that people should care about compute based on what the provider has done in the AI space, but Microsoft was pretty far behind on that side until OpenAI - Google was really the only player in town. Should they have wanted GCP credits instead? Do you care about their AI results or the ex post facto GPU shipments? Or, if what you actually want to argue is that Anthropic would be able to get more GPUs with Azure than AWS or GCP then this is a different argument which is going to require different evidence than raw GPU shipments.
- luma 2y agoThe claim being implied was that Anthropic was in a better position because they had partnered with AWS versus Azure and thus they would have more access to GPU. That isn't the case, at all. All I'm stating is what the chart clearly shows - Azure has invested deeply in this technology and at a rate that far exceeds AWS.
- nabla9 2y agoThere are no moats in deep learning, everything changes so fast. They have the next iteration of GPT Sutskever helped to finalize. OpenAI lost it's future unless they find new same caliber people.
- sk11001 2y ago> They have the next iteration of GPT Sutskever helped to finalize How do you know that they have the next GPT? How do you know what Sutskever contributed? (There was talk that the most valuable contributions came from the less well known researchers not from him)
- criddell 2y agoIsn't access to massive datasets and computation the moat? If you and your very talented friends wanted to build something like GPT-4, could you? It's going to get orders of magnitude less expensive, but for now, the capital requirements feel like a pretty deep moat.
- Gud 2y agoHow do you know massive datasets are required? Just because that’s how current LLMs operate, doesn’t mean it’s necessarily the only solution.
- 2y ago
- cm2187 2y agoBut is it even clear what is the next big leap after LLM? I have the feeling many tend to extrapolate the progress of AI from the last 2 years to the next 30 years but research doesn't always work like that (though improvements in computing power did).
- benterix 2y agoExtrapolating 2 years might give you a wrong idea, but extrapolating the last year suggests making another leap that was GPT3 or GPT4 is much, much more difficult. The only considerable breakthrough I can think of is Google's huge context window which I hope will be the norm one day, but in terms of actual results they're not mind-blowing yet. We see little improvements everyday and for sure there will be some leaps, but I wouldn't count on a revolution.
- trashtester 2y agoUnlike AI in the past, there is now massive amounts of money going into AI. And the number things humans are still doing significantly better than AI is going down continously now. If something like Q* is provided organically with GPT5 (which may have a different name), and allows proper planning, error correction and direct interaction with tools, that gaps is getting really close to 0.
- varjag 2y agoAI in the past (adjusted for 1980s) was pretty well funded. It's just that fundamental scientific discovery bears little relationship to the pallets of cash.
- mark_l_watson 2y agoFunding in the 1980s was sometimes very good. My company bought me an expensive Lisp Machine in 1982 and after that, even in “AI winters” it mostly seemed that money was available. AI has a certain mystique that helps get money. In the 1980s I was on a DARPA neural network tools advisory panel, and I concurrently wrote a commercial product that included the 12 most common network architectures. That allowed me to step in when a project was failing (a bomb detector we developed for the FAA) that used a linear model, with mediocre results. It was a one day internal consult to provide software for a simple one hidden layer backprop model. During that time I was getting mediocre results using symbolic AI for NLP, but the one success provided runway internally in my company to keep going.
- bamboozled 2y agoThey seem to have a huge "money moat" now. Partnerships with Apple and MS mean they have a LOT of money to try a lot of things I guess. Before the Apple partnership, maybe it seemed like the moat was shrinking, but I'm tno sure now. Likely they have access to a LOT of data now too.
- boringg 2y agoOpenAI most definitely needs the compute from MSFT. It could certainly swap out to another service but given that microsoft invested via credits it would be problematic. They have enmeshed their future.
- ralfd 2y agoHow important are top science guys though? OpenAI has a thousand employees and almost unlimited money, and llm are better understood, I would guess continous development will beat singular genius heroes?
- mrklol 2y agoThey definitely need them to find new approaches which you won’t find normally.
- AdamN 2y agoAgreed - it's good to have some far thinking innovation but really that can be acquired as needed so you really just need a few people with their pulse on innovation which there will always be more of outside a given company than within it. Right now it's all about reducing transaction costs, small-i innovating, onboarding integrations, maintaining customer and stakeholder trust, getting content, managing stakeholders, and selling.
- benterix 2y ago> OpenAI has a thousand employees and almost unlimited money You could say the same about Google - and yet they missed the consequences of their own discovery and got behind instead of being leaders. So you need specific talent to pull this off even if in theory you can hire anybody.
- wg0 2y agoI am just curious how it happened to Google? Like who were the product managers or others who didn't see an opportunity here exactly where the whole thing was invented and they had huge amounts of data already, whole web basically and the amount of video that no one else can ever hope to have?
- andy99 2y agoData volume isn't that important, that's becoming clearer now. What OpenAI did was paid for a bunch of good labelled data. I'm convinced that's basically the differentiator. It's not a academic or fundamental thing to do which is why google didn't do it, it's a pure practical product thing.
- deleted 2y ago[deleted]
- loldomsa 2y agoI honestly think that is the best course of actions for humanity. Even less chance to see AGI anytime soon if he leaves.
- dkjaudyeqooe 2y ago> Open AI is run by marketing, business, software and productization people. AKA 'the four horsemen of enshitification'.
- chx 2y ago> When the next wave of new deep learning innovations sweeps the world, that won't happen, the next scam will be different it was crypto until FTX collapsed then the usual suspects led by a16z leaned on OpenAI to rush whatever they had on market hence the odd naming of ChatGPT 3.5. When the hype is finally realized to be just mass printing bullshit -- relevant bullshit, yes, which sometimes can be useful but not billions of dollars of useful -- there will be something else. Same old, same old. The only difference is there is no new catchy tunes. Yet? https://youtu.be/I6IQ_FOCE6I https://youtu.be/I6IQ_FOCE6I https://locusmag.com/2023/12/commentary-cory-doctorow-what-kind-of-bubble-is-ai/ https://locusmag.com/2023/12/commentary-cory-doctorow-what-k...
- trashtester 2y agoCrypto currencies has the potential to grow the world economy by about 1-3%, as banking fees go down. Add other uses of crypto may double or triple that, but that's really speculative. AI, on the other hand, has a near infinite potential. It's conceivable that it will grow the global economy by 2% OR MORE per MONTH for decades or more. AI is going to be much more impactful than the internet. Probably more than internal combustion, the steam engine and electricity combined. The question is about the timescale. It could take 2 years before it really starts to generate profits, or it could take 10 or even more.
- jedrek 2y agoAI? Yes. LLMs pretending to be AI? No.
- trashtester 2y agoWhat you call "AI" is generally named AGI. LLM's are alredy a kind of AI, not just generic enough to fully replace all humans. We don't know if full AGI can be built using just current technology (like transformers) given enough scale, or if 1 or more fundamental breakthroughs are needed beyond just the scale. My hypothesis has always been that AGI will arrive roughly when the compute power and model size matches the human brain. That means models of about 100 trillion params, which is not that far away now.
- fsloth 2y agoIf we look at history of innovation and invention it’s very typical the original discovery and final productization are done by different people. For many reasons, but a lot of them are universal I would say. E.g. Oppenheimer’s team created the bomb, then following experts finetuned the subsequent weapon systems and payload designs. Etc.
- fprog 2y agoExcept OpenAI hasn’t yet finished discovery on its true goal: AGI. I wonder if they risk plateauing at a local maximum.
- Zambyte 2y agoI'm genuinely curious: what do you expect an "AGI" system to be able to do that we can't do with today's technology?
- jagrsw 2y agoSome first ideas coming to mind: Engineering Level: Solve CO2 Levels End sickness/death Enhance cognition by integrating with willing minds. Safe and efficient interplanetary travel. Harness vastly higher levels of energy (solar, nuclear) for global benefit. Science: Uncover deeper insights into the laws of nature. Explore fundamental mysteries like the simulation hypothesis, Riemann hypothesis, multiverse theory, and the existence of white holes. Effective SETI Misc: End of violent conflicts Fair yet liberal resource allocation (if still needed), "from scarcity to abundance"
- hojleorier23423 2y agoWhat an absurd thing to say. John Schulman is still at OpenAI. As are many others.
- kinnth 2y agoAI has now evolved beyond just the science and it's biggest issue is in the productization. Finding use cases for what's already available ALONG with new models will be where success lies. ChatGPT is the number 1 brand in AI and as such needs to learn what it's selling, not how its technology works. It always sucks when mission and vision don't align with the nerds ideas, but I think it's probably the best move for both parties.
- itsoktocry 2y ago>ChatGPT is the number 1 brand in AI and as such needs to learn what it's selling, not how its technology works. I'm not as in tune as some people here so: don't they need both? With the rate at which things are moving, how can it be otherwise?
- vasco 2y agoI guess their point is you already have a lot out there to create new products, and you can still read papers you just won't be writing them.
- throwthrowuknow 2y agoThey do need both but it seems like they have enough engineering talent to keep improving. Time will tell now that Ilya is out but I expect they have enough cultural cache to attract excellent engineers even if they aren’t as famous as Ilya and Karpathy. They have a strong focus on making the existing models fast and cheap without sacrificing capability which is music to the ears of those looking to build with them.
- Zacobia 2y agoI love the way you used "cultural cache" here Lol. In any case I do hope whatever Ilya is building is some sort of AI competition to stop this OpenAI monopoly.
- godelski 2y ago> With the rate at which things are moving Things have been moving fast because we had a bunch of top notch scientists in companies paired with top notch salesmen/hype machines. But you need both in combination. Hypemen make promises that can't be kept, but get absurd amounts of funding for doing so. Scientists fill in as many of the gaps as possible, but also get crazy resources due to the aforementioned funding. Obviously this train can't go forever, but I think you might understand that one of these groups is a bit more important than the other while one of these groups is more of a catalyst (makes things happen faster) for the other.
- renegade-otter 2y ago"Productization". You mean "enshitification".
- boringg 2y agoDepends on who is controlling product.
- ookdatnog 2y ago> They make lots of money Will they though? Last I heard OpenAI isn't profitable, and I don't know if it's safe to assume they every will be. People keep saying that LLMs are an existential threat to search, but I'm not so sure. I did a quick search (didn't verify in any way if this is a feasible number) to find that Google on average makes about 30 cents in revenue per query. They make a good profit on that because processing the query costs them almost nothing. But if processing a query takes multiple seconds on a high-end GPU, is that still a profitable model? How can they increase revenue per query? A subscription model can do that, but I'd argue that a paywalled service immediately means they're not a threat to traditional ad-supported search engines.
- FuriouslyAdrift 2y agoJakub Pachocki is taking over as chief scientist. https://analyticsindiamag.com/meet-jakub-pachocki-openais-new-chief-scientist/ https://analyticsindiamag.com/meet-jakub-pachocki-openais-ne...