4 ms·
If the rumors about the upcoming Strawberry and Orion models from OpenAI are true - supposedly capable of deep research, reasoning and math - they probably don’
by thorum 2y ago
If the rumors about the upcoming Strawberry and Orion models from OpenAI are true - supposedly capable of deep research, reasoning and math - they probably don’t have much to worry about. Not to mention they still have the only fully multimodal model.
- forrestthewoods 2y agoModels are only state of the art for 12 to 18 months. There’s not a model in existence today that will have any value in 5 years. They will all be obsolete. Thus far no one has any moat on their model.
- willy_k 2y agoNot on the model but most companies don’t sell models, they sell products and that use models. GPT-3 became obsolete at some point but ChatGPT hasn’t, and has some level of moat with their ecosystem. And cloud providers have moats via lock in, just for blanket products instead of any one model.
- krackers 2y agoAccording to that recent The Information report, Orion is supposed to be just a regular LLM except trained with synthetic data generated via Strawberry. Anthropic et al. have also been working on ways to generate synthetic data (as seen in the success of Sonnet 3.5) so I don't really know if that's going to be a big lead. And of course the ever-hyped strawberry is supposed to be some sort of tree-of-thought type thing I think, or maybe it's related to https://arxiv.org/abs/2203.14465 https://arxiv.org/abs/2203.14465. Either way, nothing so far has come out that it's a completely novel training technique or architecture, just a gpt-4 scale model with different post-processing.
- CuriouslyC 2y agoI'm not sure why people act so mystified about Q*, the name gives it away, it's an obvious reference to A*, the only question is what the nodes in the graph are and what they're using as a Heuristic function.
- fzzzy 2y agoIt's not. It's Quiet STaR.
- imtringued 2y agoIt is in reference to this: https://arxiv.org/abs/2203.14465 https://arxiv.org/abs/2203.14465 https://arxiv.org/abs/2403.09629 https://arxiv.org/abs/2403.09629
- Eliezer 2y agoQ* is also a term from reinforcement learning.
- llmfan 2y ago"just a regular LLM except [trained on very different data]." I'm not saying there's some big moat, anyone can read https://arxiv.org/pdf/2305.20050 https://arxiv.org/pdf/2305.20050, but not all synthetic data is created equal. Strawberry I'm sure generates beautiful, valid chain-of-thought reasoning data. Wouldn't surprise me if OpenAI is just significantly ahead of the competition.
- GaggiX 2y agoWhat is the only fully multimodal model? The GPT-4o checkpoint available to the public can see images but not generate them (it can generate prompts for Dalle 3 to use). OpenAI has an internal model with this capability, but if you don't make it a actual product, it doesn't really matter.
- HeatrayEnjoyer 2y agoIt's even more modal than that, 4o accepts text, image, audio, and video input, and produces text, image, or audio output. Video input isn't available yet and was only briefly demoed. Image and video output haven't been demonstrated publicly at all yet. Rapid productization isn't the priority of most ML devs.
- GaggiX 2y agoOpenAI has a few examples of image output being produced directly by the GPT-4o checkpoint they have. >Rapid productization isn't the priority of most ML devs. It depends if they need money or not, Google has not publish a single image generator that is not a demo.