4 ms·

Yeah I dont see what you’re seeing. Ignoring research, OpenAI is bringing tools to market that nobody else to quite close to yet. Yes, they are getting closer b

Yeah I dont see what you’re seeing. Ignoring research, OpenAI is bringing tools to market that nobody else to quite close to yet. Yes, they are getting closer but not as you describe it.

hn_throwaway_99 3y ago
Actually, I felt pk-protect-ai's post was spot on. That is, OpenAI does have, currently, the most impressive interfaces and models. But (as was famously commented on by a Google exec) they don't really have any unique moat. That is, while what OpenAI has built may be extraordinary, there is no "secret sauce" there that prevents others from copying them. And this isn't a knock on OpenAI at all. On the contrary, my understanding is that OpenAI was really the first to take the big risk in scaling their GPT models (i.e. spending the hundreds of millions to train their models before they knew what the outcome would be).
- mupuff1234 3y ago
  > was famously commented on by a Google exec It was just a rank and file IC.
- infecto 3y ago
  People keep saying this but after 1-2 years, nobody has gotten to their level yet.
  - pk-protect-ai 3y ago
    This isn't true. Benchmarks do not define the usefulness of the models. Mixtral is much more useful for me right now than GPT-4. Look at LLaVA, or the new one, and the very impressive LWM (the text-only version is LLaMA2-based with a 512K context, which does not require TPU to run the inference). The fine-tuned LLaMA 34B is much faster than GPT-4 Turbo and less annoying, providing pretty impressive quality of results.
    - infecto 3y ago
      Who mentioned anything about benchmarks? YMMV, it depends on exact use cases, from my extensive testing using my cases, GPT-4 Turbo still is much easier to direct than any of the others you have mentioned.
season2episode3 3y ago
I would not say that OpenAI is all that far ahead of the competition. I would not count Google out, and I would be willing to bet that OpenAI/Microsoft + Google will be the duopoly of the AI age. At this point Gemini is as good as GPT4 (and anecdotally I think it's better at many coding assistance tasks, based on my experiments on the LMSys chatbot arena). Sora is getting a lot of press for text-to-video, but Google's Lumiere has already been out for a few weeks and produces pretty good results. I have no doubt that OpenAI has been cooking things up. GPT-4 is an older model which others are only just catching up to now. They have an A+ team, a giant war chest, and a lot of momentum. But just because they have momentum does not mean that they have a moat. I'd be willing to wager that the top-of-the-line foundational models are going to converge and become indistinguishable for almost all tasks. Even the open foundation models (e.g. Mixtral) are getting really good. Foundation models are not moats. The players that have moats are Microsoft (with deep enterprise software & B2B expertise, allowing them to sell AI-powered software & ward off competition from upstarts) and Google (where their decade of investment into custom silicon allows them to train & run inference for cheaper than anyone else, by far).
- hanselot 3y ago
  I don't have a large amount of time to devote to this. But no, google is not out. They just choke slammed ChatGPT with the Gemini announcement. 10m context with that retrieval rate is such a monstrous leap. And to top it off, we got LargeWorldModel in the same week, capable of 1M token context with insane retrieval rate in the open source space. So not only is the open source world currently technically ahead of ChatGPT, so is Google. Which is why they had to announce SORA, because google's model is so far ahead of the competition. That's also why it will probably be ages before we get access to SORA. Now don't get me wrong, the average person can't afford 32 TPU's to run LWM, but we already have quants for it, which is a step towards enabling the average person (that somehow has 24-48gb of VRAM to get a taste of that power). What is also striking is the fact that the new models are all multimodal as a standard. We not only leapfrogged in context size, but also in modalities. The model seems to only benefit from having more modalities to work with. I think the statement Bill Gates made claiming that "LLM's have reached a plateau" itself indicates they don't believe they can make more money from training better/larger models. Which indicates that they already did as well as they could with their existing people, and are now "years" behind google. I never thought google could catch up, especially after their infamous "We have no moat" situation. But it seems they actually doubled down and did something about it. To a lot of people, last Thursday was a very nihilistic day for Local Models, as the goalposts shifted from 128-200k context to 10M tokens with near perfect retrieval. It's literally insanely scary. But luckily we got LWM, and that means we have only been 10xed. Now the local people will work on figuring out how to bridge the gap, before being leapfrogged again. What is really insane is that, we have had LLAMA2 for over a year now, and nobody else figured out how to get this result from it, despite it being around so long. I still believe there are modifications to the architecture of MoE that will unlock new powers that we haven't even dreamed of yet. Sorry, this was supposed to be well thought out, but it turned more into stream of consciousness, and I honestly had no intention of disagreeing with you.

Yeah I dont see what you’re seeing. Ignoring research, OpenAI is bringing tools to market that nobody else to quite close to yet. Yes, they are getting closer but not as you describe it.

hn_throwaway_99 3y ago

Actually, I felt pk-protect-ai's post was spot on. That is, OpenAI does have, currently, the most impressive interfaces and models. But (as was famously commented on by a Google exec) they don't really have any unique moat. That is, while what OpenAI has built may be extraordinary, there is no "secret sauce" there that prevents others from copying them. And this isn't a knock on OpenAI at all. On the contrary, my understanding is that OpenAI was really the first to take the big risk in scaling their GPT models (i.e. spending the hundreds of millions to train their models before they knew what the outcome would be).

mupuff1234 3y ago
> was famously commented on by a Google exec It was just a rank and file IC.
infecto 3y ago
People keep saying this but after 1-2 years, nobody has gotten to their level yet.
- pk-protect-ai 3y ago
  This isn't true. Benchmarks do not define the usefulness of the models. Mixtral is much more useful for me right now than GPT-4. Look at LLaVA, or the new one, and the very impressive LWM (the text-only version is LLaMA2-based with a 512K context, which does not require TPU to run the inference). The fine-tuned LLaMA 34B is much faster than GPT-4 Turbo and less annoying, providing pretty impressive quality of results.
  - infecto 3y ago
    Who mentioned anything about benchmarks? YMMV, it depends on exact use cases, from my extensive testing using my cases, GPT-4 Turbo still is much easier to direct than any of the others you have mentioned.

season2episode3 3y ago

I would not say that OpenAI is all that far ahead of the competition. I would not count Google out, and I would be willing to bet that OpenAI/Microsoft + Google will be the duopoly of the AI age. At this point Gemini is as good as GPT4 (and anecdotally I think it's better at many coding assistance tasks, based on my experiments on the LMSys chatbot arena). Sora is getting a lot of press for text-to-video, but Google's Lumiere has already been out for a few weeks and produces pretty good results. I have no doubt that OpenAI has been cooking things up. GPT-4 is an older model which others are only just catching up to now. They have an A+ team, a giant war chest, and a lot of momentum. But just because they have momentum does not mean that they have a moat. I'd be willing to wager that the top-of-the-line foundational models are going to converge and become indistinguishable for almost all tasks. Even the open foundation models (e.g. Mixtral) are getting really good. Foundation models are not moats. The players that have moats are Microsoft (with deep enterprise software & B2B expertise, allowing them to sell AI-powered software & ward off competition from upstarts) and Google (where their decade of investment into custom silicon allows them to train & run inference for cheaper than anyone else, by far).

hanselot 3y ago
I don't have a large amount of time to devote to this. But no, google is not out. They just choke slammed ChatGPT with the Gemini announcement. 10m context with that retrieval rate is such a monstrous leap. And to top it off, we got LargeWorldModel in the same week, capable of 1M token context with insane retrieval rate in the open source space. So not only is the open source world currently technically ahead of ChatGPT, so is Google. Which is why they had to announce SORA, because google's model is so far ahead of the competition. That's also why it will probably be ages before we get access to SORA. Now don't get me wrong, the average person can't afford 32 TPU's to run LWM, but we already have quants for it, which is a step towards enabling the average person (that somehow has 24-48gb of VRAM to get a taste of that power). What is also striking is the fact that the new models are all multimodal as a standard. We not only leapfrogged in context size, but also in modalities. The model seems to only benefit from having more modalities to work with. I think the statement Bill Gates made claiming that "LLM's have reached a plateau" itself indicates they don't believe they can make more money from training better/larger models. Which indicates that they already did as well as they could with their existing people, and are now "years" behind google. I never thought google could catch up, especially after their infamous "We have no moat" situation. But it seems they actually doubled down and did something about it. To a lot of people, last Thursday was a very nihilistic day for Local Models, as the goalposts shifted from 128-200k context to 10M tokens with near perfect retrieval. It's literally insanely scary. But luckily we got LWM, and that means we have only been 10xed. Now the local people will work on figuring out how to bridge the gap, before being leapfrogged again. What is really insane is that, we have had LLAMA2 for over a year now, and nobody else figured out how to get this result from it, despite it being around so long. I still believe there are modifications to the architecture of MoE that will unlock new powers that we haven't even dreamed of yet. Sorry, this was supposed to be well thought out, but it turned more into stream of consciousness, and I honestly had no intention of disagreeing with you.