SwellJoe

18 ms·

421.
by SwellJoe 4mo ago
I don't know how you'd judge benchmarks beyond "did it test and measure what it says it tests and measures". And, I guess there have been instances where the benchmark failed to do that, and the models could cheat in som
422.
by SwellJoe 4mo ago
This is really clever. Seems obvious in hindsight, as I've seen this tactic used for jailbreaks: modify the chat history to add the model affirming the user has the right to do the thing because they satisfied some requirement, and the
423.
by SwellJoe 4mo ago
I added GLM 5.2 to my security bug hunting benchmark when it came out, and found it to be a good performer, but not the best open model. The benchmark tests whether models can find bugs Mythos found. The best open models in the initial benc
424.
by SwellJoe 4mo ago
DeepSeek is seemingly novel in its caching capability. In most of my usage of it, I'm seeing something like 80% cached tokens, which makes using DeepSeek cheaper than damned near any other model anywhere near its size. I don't thi
425.
by SwellJoe 4mo ago
I added this to a benchmark I've been doing of how well agents find security bugs, specifically security bugs originally found by Mythos. It performs poorly with only read/grep/ls tools, but in a follow-up test with a full sh
426.
by SwellJoe 4mo ago
I'm not going to stop. Though, I often get lazy and just use "--".
427.
Show HN: Prose or Con, can you detect AI writing?(prose-or-con.com)
2 points by SwellJoe 4mo ago|3 comments
428.
by SwellJoe 4mo ago
Every turn of a conversation with an LLM is getting the whole conversation. Caching complicates the picture, but not by a huge amount. That's why a short question at the end of a long conversation chews tokens faster than it would in a
429.
by SwellJoe 4mo ago
I wanted to be sure I wasn't lying to myself. So, I built this: https://prose-or-con.com/ I can still detect AI-written prose with pretty high confidence (I average about 85% accuracy after a bunch of rounds while test
430.
by SwellJoe 4mo ago
OK, this is a silly thing to do, but I wanted to be sure I wasn't imagining that I can tell when prose is written by AI. So I made a game. Turns out Claude Opus can fool me better than any other model, but even it can't fool me a
431.
by SwellJoe 4mo ago
I don't think I'm special. I think many people can tell the difference. Edit: Also, I'm surprised images have gotten to the point where I have a hard time detecting AI in some cases, and they got there more quickly than prose
432.
by SwellJoe 4mo ago
The Dallas Central Market is a 35 minute bike ride away. Too far. And, besides, it's not HEB. It's fine, but there's a Whole Foods 27 minutes away if I want the fancy groceries, though that's also too far for a regular g
433.
by SwellJoe 4mo ago
My mom moved out of Texas about 20 years ago, and still sometimes talks fondly about HEB. I'm currently in Dallas, and annoyingly there is no HEB within a reasonable distance, though they have finally started opening stores in the DFW
434.
by SwellJoe 4mo ago
Doubtful. I read a lot of forum posts, as I maintain a popular OSS project. And, I read a lot of books from the before times. And, I read a lot of LLM prose, because I use them every day for coding and various assistant type tasks. It'
435.
by SwellJoe 4mo ago
Trivial to simulate. Though I disagree that AI writing is impossible to differentiate from human prose, at least for now. It's still pretty obvious and still much worse than human prose (or, at least, much less interesting to read), th
436.
by SwellJoe 4mo ago
Of course. No true Scotsman would ever do such a thing.
437.
by SwellJoe 4mo ago
I'll point out that a lot of the current administration and Republican Congress that enables the current administration are Ayn Rand fans.
438.
by SwellJoe 4mo ago
"I have determined that appropriate safeguards are in place to permit certain trusted partners to access the Claude Mythos 5 Model" I assume "trusted partners" means, "companies that have bribed Trump an appropriate
439.
by SwellJoe 4mo ago
The one that pleasantly surprised me when I did some testing was DeepSeek. The self-hosted version, at least, answered honestly about a wide variety of topics that are sensitive in China. The only topic it refused to discuss was Tiananmen S
440.
by SwellJoe 4mo ago
The models have to run on something or they're useless. They can't run on future hardware today, and people want to use models today. So, if hardware is obsolete the day it ships, we're all using obsolete hardware, and there&
441.
by SwellJoe 4mo ago
When you have a few billion dollars you can hire chip people and partner with a chip company. That's not to say I expect they'll ship something competitive with Google's custom AI hardware on the first go, since Google has be
442.
by SwellJoe 4mo ago
I'd go further, and say that most of the time, "SQLite is enough". But, yes, PostgreSQL is all I ever use for anything that needs to be big. I ported a big old web app that had ScyllaDB, Elastic Search, Redis, and probably so
443.
by SwellJoe 4mo ago
> This is still easier than telling Mythos to generally look at a codebase and find any bug. Many security tools for doing security audits with LLMs are based around a "look at this file" loop, where each file gets analyzed ind
444.
by SwellJoe 4mo ago
You can see which bugs each model found in the full report. And, you have misunderstood what the benchmark does. It tells the model to audit the file, and it is allowed to look at the rest of the repo. It is not pointed at the bug. The jud
445.
by SwellJoe 4mo ago
I have done replication tests of Qwen and the Gemma models. The Qwen benchmarks are published: https://swelljoe.com/post/qwen-quantization-degradation/ . (Though, I still want to add the other three cases to that
446.
by SwellJoe 4mo ago
6 tokens per second is not fit for interactive use. I find Gemma 4 (QAT 4-bit, MTP) to be tolerable at about 30 tokens per second on my old GPUs. Anything slower than 15 is annoying. I tried DS4 on my Strix halo (1-bit quantization of DeepS
447.
by SwellJoe 4mo ago
That's an artificially inflated market. OpenAI and xAI bought everything for like two years into the future, partly to inflate the AI bubble, partly to lock-in a monopoly on the kinds of compute you need for AI, and partly to scale up
448.
by SwellJoe 4mo ago
I consider Opus 4.5 the crossover point where coding with agents got more efficient than not coding with agents. They were too stupid before that, and wasted more time than they saved for anything beyond a basic CRUD app or HTML page. Certa
449.
by SwellJoe 4mo ago
China has the benefit of vast solar power and rapidly increasing battery capacity. Yes, that's subsidized, but it pays for itself in the long run. And, even with the price increases, Z.ai and Tencent are still much cheaper than Anthrop
450.
by SwellJoe 4mo ago
Yep, if they make it to IPO, as SpaceX has, and if they manage to get into several indexes (as SpaceX is already doing, I assume it's already in the Russell indexes, and will soon be in the Nasdaq 100 index), it'll be a bunch of w

More ›