usaar333

8 ms·

61.
by usaar333 2y ago
Overall impressive. Though, the jump for Gaia relative to SOTA is relatively not that high. Especially given that this is o3
62.
by usaar333 2y ago
A paper that came out 15 months ago?
63.
by usaar333 2y ago
Huh? It was made in 2019
64.
by usaar333 2y ago
Even 4o mini gets that
65.
by usaar333 2y ago
I'm not that surprised. Codeforces is a relatively low area of vocabulary knowledge needed. Real software has a lot more complexity and constraints, as well as ambiguity. Claude nowhere scores so high on codeforces, but crushes o1 on w
66.
by usaar333 2y ago
Where are you reading that?
67.
by usaar333 2y ago
For non-stem perhaps. For math/coding problems, o3 mini is tied if not better than o1.
68.
by usaar333 2y ago
Overall good post, but feels like he has an axe to grind with LLMs to the point it is misleading: > Last week, DeepSeek published their new R1-Zero and R1 “reasoner” systems that is competitive with OpenAI’s o1 system on ARC-AGI-1. R1-Ze
69.
by usaar333 2y ago
No they aren't. Every arc problem is novel - that's why it resisted deep learning for so long (and still does to a degree). We just don't know how much the model seeing what an arc problem is on the first place boosts its abi
70.
by usaar333 2y ago
They were 6 months behind US frontier until deepseek r1. Now maybe 4? It's hard to say.
71.
by usaar333 2y ago
38% on osworld vs 22% for Claude. That seems like a jump
72.
by usaar333 2y ago
That's more of a leetcode bench than real world coding bench
73.
by usaar333 2y ago
My current visual reasoning test is seeing if models can figure out what train to take to get from stop A to B when not all trains go to both stops. Something like: https://www.caltrain.com/media/22502/ So far non
74.
by usaar333 2y ago
That's an old leaderboard -- has no one checked any SOTA LLM in the last 8 months?
75.
by usaar333 2y ago
For what it's worth, I'm much more impressed with the frontier math score.
76.
by usaar333 2y ago
Super human isn't beating rando mech turk. Their post has stem grad at nearly 100%
77.
by usaar333 2y ago
It looks like gemini-exp-1121 slightly upgraded. 1206 is something else.
78.
by usaar333 2y ago
Which GPT-4O are those? The 08-06 release seems to be a bit higher on numerous benchmarks than what that shows: https://github.com/openai/simple-evals?tab=readme-ov-file#be...
79.
by usaar333 2y ago
Neither Israel nor the de-facto government of Gaza they are fighting ever accepted the authority of the ICC; neither has signed the Rome Treaty. The ICC authority is being derived from the Palestinian Authority applying for membership and t
80.
by usaar333 2y ago
Wages generally kept track with inflation since before covid: https://fred.stlouisfed.org/series/LES1252881600Q I really don't understand the dynamic here.
81.
by usaar333 2y ago
America's economy probably did better than anywhere else in the rich world. I don't see how we can view this as a fail
82.
by usaar333 2y ago
Dems don't control the fed. A fast rate hike might have caused massive unemployment which would be much worse.
83.
by usaar333 2y ago
They are still counting votes. Prediction markets have turnout at about 64%, which is more like 5 million less. That's still historically high
84.
by usaar333 2y ago
I agree that maybe people need their feelings humored, but how is this gaslighting? I'm not denying that there's food inflation or restaurant inflation - I pointed out that it's a narrow way to look at even your own economic
85.
by usaar333 2y ago
I don't think that can be true in the Bay. They would have an even higher percent of expenditure to rent, which is flat. America wide looks at worse flat: https://fred.stlouisfed.org/series/LES1252881600Q (ignorin
86.
by usaar333 2y ago
Housing: not in the Bay: https://www.zillow.com/rental-manager/market-trends/mountain...
87.
by usaar333 2y ago
I'm in the Bay - am I the only person that thinks the economy is going great? My wages are up since Biden started. My rent, my biggest expense, has held the same. NW up a lot from stock market gains. There seems to be a lot of inflatio
88.
by usaar333 2y ago
They definitely considered it. Early theinformation articles talked about how high the performance of strawberry was on it.
89.
by usaar333 2y ago
FWIW, the refactor benchmark is quite mechanical - it just stresses reliability of LLMs over long context windows: Questions are variants of: Refactor the _set_csrf_cookie method in the CsrfViewMiddleware class to be a stand alone, top leve
90.
by usaar333 2y ago
It's roughly tied in benchmarks

More ›