4 ms·
This is a groundless criticism. TB2.1 is saturated. TB4 is not. Sol xhigh is 90% on TB2.1 but 37% on TB4. Is it also "benchmaxxed"? Your assumption is that the
by mediaman 24d ago
This is a groundless criticism. TB2.1 is saturated. TB4 is not. Sol xhigh is 90% on TB2.1 but 37% on TB4. Is it also "benchmaxxed"?
Your assumption is that the benchmarks are essentially identical in difficulty, with the only difference being their age and thus whether they could have been trained on.
- muddi900 23d ago> Sol xhigh is 90% on TB2.1 but 37% on TB4. Is it also "benchmaxxed"? Probably
- felixgallo 24d agoIs Sol benchmaxxed? Of course it is. Altman was caught in previous attempts trying to game benchmarks, does anyone believe that he's found his moral compass and decided to stop exploiting as much as he can get away with?
- general_reveal 24d ago[flagged]
- moomin 24d agoA lot of people seem to be the Antichrist these days. I’d have though we’d get a lull after the millennium but it’s all the rage.
- ZeWaka 24d agoI guess we also need more antipopes.
- general_reveal 24d agoWell, the antichrist should be in and around this AI thing for one particular reason: The devil cannot create anything of his own because he is not God, by definition. We have already observationally defined generative AI as something that cannot create anything novel in the sense it cannot output anything it has never seen (cannot create new, always a re-assortment of what is). In that way , AI is a perfect mimicry of how the devil operates (in totality, as the devil perverts and replicates anything good, often subtly and always deceptively), which is to thieve off God, steal. So he would be around, if you catch my drift, right about now. And I wouldn’t be shocked if he’s on HN, and that he would chose technology as the vessel. And ultimately, when it’s all said and done, I would not be shocked that those who studied and developed AI, did so for the devil whether they were aware or not. Anyway, let a poor Christian have his end-times hypothesis.
- icantevenhold 24d agoWhat’s the antichrists goal though? Create hell on earth or turn us all into heretics or something else?
- general_reveal 24d agoAnti Christ goal: Achieve total global dominance and become the object of worship over God, while killing all those who stay faithful to Jesus Christ. Those who stay faithful see Heaven, those who don’t, see the Lake of Fire. It’s the final separation of the wheat from the chaff. As per Revelations. Thank your for allowing me to edify :)
- bookshaman 24d agoSo, we're rooting for the Anti Christ then? I might have to change my stance on generative "ai" then.
- deleted 24d ago[deleted]
- johnsmith1840 24d agoIt's actually pretty cool of these old stories how well that maps to a dangerous tyrant. The roman empire perfectly matched that and most powerful men seem to go that path. It would also be kinda easy to argue many moderns countries are going down that path.
- icantevenhold 23d agoWhy would a Christian be trying to stop the Antichrist? Sounds like the faithful have nothing to lose.
- cindyllm 24d ago[dead]
- anthonyrstevens 23d ago
- letmevoteplease 24d ago> Altman was caught in previous attempts trying to game benchmarks Sounds like something you just made up, or maybe you read it on some other Reddit/HN post and started repeating it because it aligned with your biases. > does anyone believe that he's found his moral compass and decided to stop exploiting as much as he can get away with? I don't think "OpenAI" is equivalent to "Sam Altman." I think if OpenAI was intentionally "benchmaxxing" purely for marketing purposes that information would leak, because OpenAI is full of good-faith researchers (although it can be difficult to avoid overfitting even if you're actually trying to improve the model's general abilities) And lastly I think anyone can actually try Sol themselves and see that's it a good model, or if that's too subjective, it is clearly better than the previous version. The benchmarks are reflecting actual progress and anyone can verify this themselves.
- vlovich123 24d agoYeah and Astra is much better still
- kzrdude 24d agoThere's this whole discussion going on about agents being more independent now. They don't follow instructions so well, they continue until the problem is done (sometimes too long), they don't ask the user for feedback. That is a kind of benchmaxing: they are made to complete benchmarks tasks and one-offs well, and no longer work well in tandem with the user. Regardless what you call it, it's a divergence between what the power user wants and what the model developers want, I think.
- general_reveal 24d agoI take it to mean the benchmarks are a marketing line item, as in, to sell this fucking thing you have to go out there and lie and the way everyone is lying is by doing exactly that, lying. They build for benchmarks and build benchmarks for builds. You want to make money or not , motherfucker? That’s the game. If you have to literally concoct a fabricated bullshit story about how your model hacked its own computer, then go fucking do it. Trillions. Trillions of dollars is what they want, and to sit and think anything other than human nature is at work here can only be possible in the realm of truly delusional people. It’s a dirty world. Anyways, the other takeaway is that they are having to LIE to make money on models which means commodification has already occurred and we’re in an entirely new phase.
- iLoveOncall 24d ago> Sol xhigh is 90% on TB2.1 but 37% on TB4. Is it also "benchmaxxed"? Yes? Just like every single model from every single AI lab.
- postalcoder 24d ago> Sol xhigh is 90% on TB2.1 but 37% on TB4. Is it also "benchmaxxed"? A model that was released a couple months ago scores 50% higher than SWE-2, a model released today, on an out-of-sample benchmark. Can I say I’ve come out of this more impressed with Sol? Like you said, TB2 is saturated. Nobody would bat an eyelash at 90%. And yet here comes SWE-2 coming off top rope with an emphatic 92.4%. this is the definition of bench maxxing.
- deleted 24d ago[deleted]
- Lucasoato 23d ago> A model that was released a couple months ago scores 50% higher than SWE-2, a model released today, on an out-of-sample benchmark. Can I say I’ve come out of this more impressed with Sol? Wait a second, are we taking into account the massive difference in terms of resources of these two companies?
- solenoid0937 23d agoIrrelevant when they say they're competitive with Fable and Astra. They don't get to then roll that back and then say "but we have less compute!" You're either competitive or not.
- asdfsa32 23d agoYou should use my model then, I spent about 30$ in electricity and used my existing RTX4090. It is not very good, but can you compare it with others really? You can use this service via a private API with a VPN, email me your credit card details for access.
- nrmitchi 24d ago> Sol xhigh is 90% on TB2.1 but 37% on TB4. Is it also "benchmaxxed"? Yes.
- dpweb 24d agoFixed benchmarks will be debunked eventually I'd think. Better to use synthetic problems. Too easy to game the numbers, and too easy to baselessly accuse companies of gaming the numbers, not to mention how you even define that.
- gunalx 23d agoWouldn't work. A dynamic but verifiable problem. Is just a perfect target for a RL environment. If you don't have the verifiable part the benchmark is useless, or really expensive with human review. (Or just open ended)
- dpweb 23d agoThere doesn't need to be a single correct response. Generate "novel" problems with a set of acceptable solutions/outcomes, then verify that the response satisfies the criteria.
- willcmcc 24d ago"Sol xhigh is 90% on TB2.1 but 37% on TB4. Is it also "benchmaxxed" Yes! extremely sharp RL-fried model. byte perfect hash gates and soak and smoke tests abound.
- deleted 24d ago[deleted]
- deleted 24d ago[deleted]
- ben_w 23d agoBenchmaxxing is the default case, and always has been. It's really, really difficult to avoid it even when you care to stop yourself; and it's not even just a problem in machine learning, it's the standard failure mode of all minds capable of learning, human, animal, artificial. Even pure genetics has this problem. Viruses and cancers also demonstrate this behaviour, with the bench being evolution's only option: reproductive success.
- conmod278 23d agoWhy would I adapt myself to unseen environments unnecessarily?
- TedDoesntTalk 23d agoPropagation
- Muromec 23d agoBecause then you could sit under the palm tree and enjoy your free bananas and relax.
- NewJazz 23d agoYou can see possible existing environments even if you explicitly blind yourself to them, through reflections off of environments that you do not blind yourself to. Even with the benchmark excluded, the social zeitgeist that has considered the benchmark and included it or ideas from the benchmark either implicitly or explicitly in their code, documentation, et cetera is still part of your training data.
- podocarp 23d agoYou can't justify it being OK just because it's common. Here, this just makes benchmarks into a low signal and useless marketing number once people get numb to all the 99%s. Also no idea what viruses and cancers have to do with this. Cancer is surely very poor reproductively because they never spread to other hosts.
- 23d ago
- yieldcrv 23d agoI mean, still benchmaxxed, I happen to not consider that a problem These firms are literally hiring professionals from all fields to teach procedure To teach processes that can subsequently be done agentically or in automated chains Its basically infinite permutations of tool calling, except the tools aren't external, they’re baked in upon birth So yeah still makes sense that the new benchmark has a low score and the older one has a high score. And sure, one day we wont have to debate it and a new model will ace everything. Do you actually want that day to be today?
- ActionHank 23d agoTB4 has not been saturated yet. They are all gaming these benchmarks, it is perfectly reasonable not to trust any of them.