SwellJoe

21 ms·

451.
by SwellJoe 4mo ago
The post was published on May 30, and written over a few days before that. Well before Fable was banned. And, before the NSA hacking thing. But, I am skeptical of the AI global catastrophe, it still feels like a mix of marketing hype and re
452.
by SwellJoe 4mo ago
But GPT 5.5 on extra high is not Pro. When I looked into it, Pro was not available for agentic use via any rolling limits plan. But, I'll look again into whether there's some reasonable way to complete the test for GPT Pro.
453.
by SwellJoe 4mo ago
I tried to prove quantization made models worse, but in my testing Qwen 3.6 27b performed statistically the same from 4 bits to 16, using the unsloth dynamic quantizations. Gemma 4 4-bit QAT seems to perform the same as the full-fat version
454.
by SwellJoe 4mo ago
Right. I noted that in the post. Some of the information out of Anthropic indicates dumb loops, sometimes, but some hint at a more sophisticated harness and process for some of the Mythos bug hunts. But, nothing specific. I've been doi
455.
by SwellJoe 4mo ago
Yeah, I'm not super happy with the chart sorting order, but trying to balance all the information is challenging. I chose not to include partials (right place, inaccurate bug description, so it smelled something funny but didn't q
456.
by SwellJoe 4mo ago
Gemma 4 beat Gemini 3.1 Pro, as well. In a later replication test I haven't published yet, it found more bugs than all other models (somewhat inconsistently) when given multiple attempts. So, it seems like they are doing real work but
457.
by SwellJoe 4mo ago
They were not pointed at the problem. You're reading the section about corpus selection and mixing it up with the benchmark rules. And, false positives are reported in the results.
458.
by SwellJoe 4mo ago
I don't know. I think it proves that if Google is baking guardrails into their models that prevent them from finding security bugs, they didn't bake those guardrails into Gemma 4, because it is very good at it. Maybe that means Go
459.
by SwellJoe 4mo ago
I'm talking about guardrails that prevent finding exploits, which is only peripherally related to writing secure code. This benchmark is about finding security bugs, not writing secure code. I don't believe the models have guardra
460.
by SwellJoe 4mo ago
Also, with regard to tools, I originally ran a batch of several models in a full-featured agent (and whatever tools the agent provides), and they didn't perform better than the basic minimal harness with just read and grep. They chewed
461.
by SwellJoe 4mo ago
At the time a GPT subscription didn't include Pro usage in the rolling limits. It was billed at API rates. Does it now? If anyone wants to fund the other five cases (~$125), I'll run them. I find that an unrealistic cost, though..
462.
by SwellJoe 4mo ago
Gemini CLI actually had an extension explicitly for security tasks: https://github.com/gemini-cli-extensions/security But, Gemini CLI is deprecated. So, I tried to use Antigravity and it simply refused. Weirdly, Gemma
463.
by SwellJoe 4mo ago
And, I said I'm still working on it also in the post.
464.
by SwellJoe 4mo ago
Early on, I had a vague suspicion that the reason some of the Chinese models, including quite small ones, perform so well on this task, especially relative to their size and cost, is because they don't have the same safety guardrails b
465.
by SwellJoe 4mo ago
Who are you talking about? I don't believe I have downplayed anything? And, I did briefly use Fable. It was excellent for general coding but it was blocked before I could benchmark it. I kinda suspect it would refuse this task, though.
466.
by SwellJoe 4mo ago
You're mixing up corpus selection and the benchmark. I possibly could have explained better. In the benchmark the models were told to look at the file and were allowed to look at the rest of the repo, with no clues about what to look f
467.
by SwellJoe 4mo ago
It's terrible at hunting security bugs (I expected it to be, but I wanted to be sure). I added it to a benchmark I made with a corpus of some Mythos-discovered bugs, and it found zero. The smallest pretty successful models remain Qwen
468.
by SwellJoe 4mo ago
Anything "for agents" needs to provide some kind of evidence it's better than what the agents already have baked into the model training data. It can't just be "easier" on some dimension, because the model has
469.
by SwellJoe 4mo ago
The kind of funding it takes to take on US tech corporations, especially in AI, will be astronomical. For an open source solution, it will take state action, and given how unpopular AI is with average folks (an entirely reasonable position
470.
by SwellJoe 4mo ago
It obviously does not. But, there is nothing preventing it. The US has given away all of our foreign scientists, if Europe wants them. All Europe has to do to take the lead in tech is ramp up research spending by an order of magnitude or tw
471.
by SwellJoe 4mo ago
Another day another "memory" system for a tool that cannot ever have memory. LLMs have context, and the more you fill that context with unrelated junk, the worse they perform.
472.
by SwellJoe 4mo ago
It isn't completely busted, unless the Trump administration has a personal interest in overriding the law. As sometimes happens when some foreign power, or just a random politician in another nation, does something he doesn't like
473.
by SwellJoe 4mo ago
And, there's no shortage of those. Do those religious folks often buy luxury products?
474.
by SwellJoe 4mo ago
I like the idea, and it has become more pressing that everyone outside the US think about tech sovereignty because the US has become an unsafe place to keep your data, but the impression I get from Apertus is that it moves at the speed of a
475.
by SwellJoe 4mo ago
Exactly that. The idea that Commodore can decide which apps go on the device, and I'm supposed to pay a premium for that, is crazy to me. Parental controls solves the problem for kids, without preventing them from having apps the paren
476.
by SwellJoe 4mo ago
Having WhatsApp but no email is the weirdest part about this thing (and there are many weird choices here). This is the most "built specifically for one person" product I've ever seen. I've never used WhatsApp. Never kno
477.
by SwellJoe 4mo ago
If I were going to be nostalgic about a phone, it'd be the Sidekick II (Danger Hiptop outside the US). That was a banger. Physical keyboard, very comfortable form factor for double thumb typing. A modern take on that would be my ideal
478.
by SwellJoe 4mo ago
WTF are you talking about? This is not at all fit for that purpose.
479.
by SwellJoe 4mo ago
Those parents can choose from the many dumb phones already on the market that cost under $100. Why on earth would someone who isn't specifically nostalgic about Commodore products spend $500 for this specific dumb phone? https:/&
480.
by SwellJoe 4mo ago
There aren't any merits to talk about. A flip phone is still a nostalgia play. If someone is nostalgic about flip phones, they're nostalgic about a Nokia, Commodore was long gone by the era of flip phones. And, you can still buy f

More ›