4 ms·
First impression: Third-party benchmarks or gtfo. Personally, I've never heard of either of these companies before. We're just supposed to take their word that
by fwipsy 4mo ago
First impression: Third-party benchmarks or gtfo. Personally, I've never heard of either of these companies before. We're just supposed to take their word that they've matched the best models on the market?
Sakana describes their model as a "Orchestration Model." Does that mean that it's actually a bunch of different models glued together?
- OutOfHere 4mo agoDid Anthropic give you third-party benchmarks? Is that what you said to them? Yes, they're important, but the attitude is wrong.
- bloppe 4mo agoAnthropic always publishes 3p benchmarks every time they announce a new model
- MostlyStable 4mo agoAnd even if they didn't, they have a track record. Even if we did have benchmarks in this case I would still wait until people got there hands on it and formed a more holistic opinion.
- OutOfHere 4mo agoNo, stop right there. Anything published by Anthropic implicitly is not third party. For it to be third party, the third party has to be the one publishing it.
- bloppe 4mo agoWhen you're announcing a new model, typically, nobody else has benchmarked it yet, because it hasn't been released yet. You can still run 3p benchmarks on it and publish those results. If other parties later run the same benchmarks independently, and find major discrepancies, that would be a scandal.
- fwipsy 4mo agoFudging benchmarks is a cheap way to get attention. If the model is really that good, it will have plenty of attention soon enough.
- greenavocado 4mo agoYeah, what happened to that scam startup that alleged to have made a model context window breakthrough a few weeks ago?
- lifeformed 4mo agoIs it actually that hard to make good models or is it just about the amount of resources you have to do training? (This is an actual question, I really don't know.) I'm sure it's not trivial but does it really take world class secret knowledge to build off of the known existing techniques? I feel like there's tons of low hanging fruit still to explore, and time and resources are the limiting factor.
- MostlyStable 4mo agoThe gap between grok and Gemini to Claude and chatgpt suggests that yes it is that hard.
- arw0n 4mo agoI suspect that Grok has been ironically lobotomized by pressures to correct its political views. Similarly, I could imagine the Gemini folks working in a significantly more complex corporate climate, with different parts of Google pushing for different capability focuses. They are only lagging behind less than a year, so it isn't too large of a gap yet. That said, the fact that Anthropic is currently the top dog suggests that talent and execution is incredibly important. A year ago none of my normie friends new them, and when i suggested using Claude looked at me like when I recommend Linux.
- janalsncm 4mo agoThat shouldn’t affect Grok’ coding ability. How often are people discussing politics with Claude code? Writing decent code is just hard and it’s not just Grok.
- bwhiting2356 4mo agoIt affects their ability to hire and retain talent.
- janalsncm 4mo ago
- Ifkaluva 4mo agoTheir release post was on HN recently. The comments seemed to think that it was similar to OpenRouter, not an actual model.
- alwa 4mo agoMy impression is that the answer is yes, that it purports to dispense the glue on-the-fly in some kind of dynamic way rather than being some kind of new model-amalgam. See also contemporaneous reaction at: https://news.ycombinator.com/item?id=48624782 https://news.ycombinator.com/item?id=48624782 (6 days ago, 244 points, 133 comments)
- tough 4mo agoAlso sakana has misrepresented their findings previously i think to remember [1] 1.https://www.reddit.com/r/singularity/comments/1iwbwgu/sakana_discovered_its_ai_cuda_engineer_cheating/ https://www.reddit.com/r/singularity/comments/1iwbwgu/sakana...