5 ms·
When these companies are getting literally multiple billions of dollars of funding thrown at them, I can't think of a single feature that can be a moat. If it'
by prng2021 2y ago
When these companies are getting literally multiple billions of dollars of funding thrown at them, I can't think of a single feature that can be a moat. If it's truly a feature that leapfrogs competitors, there's just no way that someone like Anthropic spending $1B on engineering resources can't replicate the same exact feature that engineers elsewhere implemented.
- Frost1x 2y agoIf it’s simply an engineering feat I agree. Very, very, very often people tend to mix up science being done in technology spaces with engineering. Just because it’s being done in software and perhaps even without rigor doesn’t mean you’re not doing something novel and quite different at a fundamental perspective than someone else. Sometimes it could just be luck of the draw in an implementation approach that gets you there but that’s not always the case. I’ve worked in scientific computing for awhile now and there are countless subtle decisions often done in the implementation phase that, from a theoretical perspective, aren’t definitively answered by the science working behind the scenes. There’s often a gap in knowledge and implementers either hit those gaps by trial and error, luck, or insight. That’s my opinion, at least. So I don’t think it’s always “just an implementation problem” some will claim, as if the science is well understood and solved. Perhaps it is, but from my experience that tends to not be the case.
- danielmarkbruce 2y agoThis 100%. And LLMs and the applications around them (even something as simple seeming as ChatGPT) have more subtle decisions than any other type of software that I've ever seen. Everyone claims there isn't a moat, I bet there is.
- oakst 2y agoReally interesting perspective
- prng2021 2y agoYep I agree with that. I will say though that people, teams, and even entire companies (like what happened at Inflection) get poached everyday so maintaining a moat that way is tough. Also, even though it could happen in the future, is OpenAI's lead due to a moat of scientists with ideas so novel that no other AI company can compete? Certainly not because even though ChatGPT took the world by storm, numerous other companies built LLMs in a very short time span that now perform at very similar levels, both subjectively and based on benchmarks.
- ffsm8 2y agoThis practice supercedes modern science and was originally called trade secrets. And these are generally "defeated" through spionage. 1 billion $ let's you do a lot, including completely legal things such as poaching someone that has a basic understanding of the secret sauce and then copying that from the abstract description
- deleted 2y ago[deleted]
- WanderPanda 2y agoAgreed, on another note I also struggle to see how no one could create a (better) CUDA implementation for e.g. $1B engineering budget
- prng2021 2y agoI purposefully used the word feature in my reply and I don't think CUDA is a feature. I see that as a massive ecosystem built on a proprietary platform. For that, it takes orders of magnitude more money and something else money can't buy.. time. Time to have the platform adopted by countless vendors and the ecosystem built up.
- sterlind 2y agoGPUs are scarce, expensive and in high demand. Devs are cheap compared to the expense of training a huge model. If Cerebras or some other hardware company develops a viable competitor to the H100, it doesn't matter if the ISA is 36-bit VLIW documented in Vedic Sanskrit, they'll have infinite demand.
- p1esk 2y agoAMD has a viable competitor to H100 but no one buys it because it doesn’t support CUDA.