XCSme

19 ms·

451.
by XCSme 8mo ago
It's benchmaxxed: https://aibenchy.com/model/minimax-minimax-m2-5-medium/
452.
by XCSme 8mo ago
Not so bad, it's above MiniMax M2.5 with reasoning in my tests (mostly because MiniMax m2.5 seems to reason forever): https://aibenchy.com/compare/?left=stepfun-step-3-5-flash-fr...
453.
by XCSme 8mo ago
Funnily, on my tests, 3 flash with medium reasoning does better. Seems like 3.1 pro reasoned about the correct answer, but chose to go with a different (wrong) one: https://aibenchy.com/compare/?left=google-gemini-3-fla
454.
by XCSme 8mo ago
Added one more test, which surprisingly gemini flash 3 reasoning passes, but gemini 3.1 pro not
455.
by XCSme 8mo ago
Yes. I am trying to think what's the best way to give most information about how the AI models fail, without revealing information that can help them overfit on those specific tests. I am planning to add some extra LLM calls, to summar
456.
by XCSme 8mo ago
Now I need to write more tests. It's a bit hard to trick reasoning models, because they explore a lot of the angles of a problem, and they might accidentally have an "a-ha" moment that leads them on the right path. It's
457.
by XCSme 8mo ago
Gets 10/10 on my potato benchmarks: https://aibenchy.com/model/google-gemini-3-1-pro-preview-med...
458.
by XCSme 8mo ago
I use structured format in many of live AI systems, maybe my point was not clear. For some tasks it's impossible to define a JSON schema. Let's say you want the message to end with "Thank you", in any language. Should I
459.
by XCSme 8mo ago
There's definitely many ways to improve the output of the AI, and provide it extra hints. Also, some AIs are made for a specific use-case. Maybe I should rephrase it and say that those benchmarks are more about the single-reply intelli
460.
by XCSme 8mo ago
Because the format can't also be strictly defined via structured output, and you have to write it in plain words. Imagine you also have a field within your JSON, which also needs a specific format. It's AI, you don't want to
461.
by XCSme 8mo ago
Offtopic: Dang, I'm fighting with the HackerNews formatting. Anyone has a link to the HN formatting guide?
462.
Show HN: AIBenchy – Independent AI Leaderboard(aibenchy.com)
1 points by XCSme 8mo ago|1 comments
463.
by XCSme 8mo ago
What do you mean? You can force JSON with structured output. It was just an example though, in real-world scenarios, sometimes I have to tell the AI to respond in a specific strict format, which is not JSON (e.g. asking it to end with &quot
464.
by XCSme 8mo ago
After looking more into it, Claude DOES give the correct answer, just not in the format that it's asked, it always adds more info at the end, even when asked to just give the answer...
465.
by XCSme 8mo ago
I made my own benchmarks, very basic questions, and Claude 4.6 is actually worse than the free Stepfun 3.5 version: https://aibenchy.com It is smart, but it fails at basic instruction following sometimes. I remember this is a Cl
466.
by XCSme 8mo ago
It doesn't do so well on my stupid benchmarks, lol: https://aibenchy.com Gets wrong some tests. It does answer correctly, BUT it doesn't respect the request to respond ONLY with the answer, it keeps adding extra explan
467.
by XCSme 8mo ago
I just started creating my own benchmarks (very simple questions for humans but tricky for AI, like how many r's in strawberry kind of questions, still WIP). Qwen3.5 is doing ok on my limited tests: https://aibenchy.com
468.
by XCSme 8mo ago
Let's see what Grok 4.20 looks like, not open-weight, but so far one of the high-end models at real good rates.
469.
by XCSme 8mo ago
In my case it's for analytics, so I am ok with some data loss in case of a failure. How do I set the fsync stuff? Does if have to be turned off?
470.
by XCSme 8mo ago
Do you mean fsync can lead ro poor performance?
471.
by XCSme 8mo ago
Thinking back about it, I think the biggest issue was the size, not performance. For a write-intensive app, the bin long quickly got to tens of GBs and filled the entire disk, which was a problem when running the app on smaller VPSs.
472.
by XCSme 8mo ago
I am doing many writes inserts/updates per second.
473.
by XCSme 8mo ago
I always end up disabling bin log for single-db setups, and simlly run backup jobs. Using bin log drastically reduces performance. Am I crazy?
474.
by XCSme 8mo ago
Thanks, you finally settled my dilemma of whether I should have a free version of UXWizz...
475.
by XCSme 8mo ago
That's a cool project/idea. Will I ever use it? Actually, I wanted to add trendlines to charts in UXWizz (especially for the current day, where the data is incomplete and showing the current value makes the chart look broken). Wou
476.
by XCSme 8mo ago
Interesting angle. I’ve found those human vs bot scores are useful as a first pass, but sometimes they misclassify power users or weird edge cases. When I built UXWizz[0], session recordings ended up being the clearest signal of real intent
477.
by XCSme 8mo ago
For the distinguishing part, it's hard to tell if it can be done anyway, as the agent browsers are still new and constantly changing, and it's up to them if they will correctly identify themselves or not (same as with crawler&#x2F
478.
by XCSme 8mo ago
That's true. The problem is that if the automation breaks at any point, the entire system fails. And programming automations are extremely sensitive to minor errors (i.e. a missing semicolon). AI does have an interesting feature though
479.
by XCSme 8mo ago
Yeah, been working on it for... 14 years. Wow, time flies
480.
by XCSme 8mo ago
If they are knowledgeable enough and pay attention, yes. Also, if they are given enough time for the task. But the idea of automation is to make a lot fewer mistakes than a human, not just to do things faster and worse.

More ›