aspenmartin

7 ms·

91.
by aspenmartin 4mo ago
You don't have to imagine!
92.
by aspenmartin 4mo ago
I am in full support of custom workflow benchmarks, and choosing the best model for your use case to balance performance and expense. Thats just good operating behavior, but the problem is the foot guns and biases people have that they are
93.
by aspenmartin 4mo ago
> I don't give a crap about how good a shovel may be in a theoretical experiment when it's digging in sand, when I work with hard earth. I don't know what this means, benchmark tasks are pretty hard and pretty in domain. &
94.
by aspenmartin 4mo ago
Yet human judgement isn’t subject to side effects like fluency and persuasiveness? It’s like everyone in this thread dismisses benchmarks and then…describes a crappy benchmark. Sure you can create a personal benchmark. Who will evaluate it,
95.
by aspenmartin 4mo ago
Um kind of but not really, it’s a mix of UX and actual measurements of what tasks it can do. Also UX is virtually the same thing: scaled quantitative surveys and preference metrics. It’s again, just benchmarking, and it’s done carefully and
96.
by aspenmartin 4mo ago
> Evaluating an intelligence is a task that generally requires at least an equally capable intelligence, if not one of greater capability. How is this remotely true. You can have verifiable tasks that you can’t do. Where does this idea c
97.
by aspenmartin 4mo ago
You are literally describing a benchmark
98.
by aspenmartin 4mo ago
N isn’t small, science means what it’s always meant, statistics is a thing, and what you’re describing is just putting your trust in a very poor quality benchmark. You said you would not trust any data that indicates something that contradi
99.
by aspenmartin 4mo ago
Nobody anywhere is a good guy but I don’t think you’ve managed to pick the lesser of the evils here
100.
by aspenmartin 4mo ago
That is very depressing. Can anyone make something up that will cheer me up? An LLM hallucination would do just fine.
101.
by aspenmartin 4mo ago
If benchmarks didn’t exist we would have to invent them because “vibes” is a ridiculous idea: oh I know I’ll be super unscientific and horrendously biased and that’s far better than a team of experts carefully AND CONTINUALLY developing a v
102.
by aspenmartin 4mo ago
I agree, I would say the difference here is economies weren’t heavily resting on nuclear armament, but maybe that’s the wrong take.
103.
by aspenmartin 4mo ago
Investigating on your specific use cases, codebases, workflows and tasks is important, there is nothing wrong with this and in fact it’s more important than benchmarks if you can do it well but the point is that is very hard and easy to t
104.
by aspenmartin 4mo ago
That’s very exciting! Thank you for the reference!
105.
by aspenmartin 4mo ago
Almost every day. Meanwhile my quest pro gathers a lot of dust. The AVP as a screen is just so much more comfortable and productive for me and nothing beats it for entertainment. It also just still feels like a really great user experience
106.
by aspenmartin 4mo ago
That’s ok but at what point is this getting into conspiracy territory? You have just said there is nothing you would believe to the contrary, but then by definition that’s not exactly a very thoughtful or insightful position.
107.
by aspenmartin 4mo ago
Yea but it’s not right? You or I or the myriad of other institutions inside and outside of academia can probe these models with an evolving landscape of evaluation sets, even those unavailable to the developers. It’s just ignorance to claim
108.
by aspenmartin 4mo ago
Which value system to align to is absolutely the right question both rhetorically and otherwise. These models have a fairly western bias due to the domain of the training data. But also, these models are capable of adjusting their value sys
109.
by aspenmartin 4mo ago
No it’s: evaluating these systems are complex and there’s a reason why sociology, cognitive psychology, medicine, etc are all done in careful double blind conditions with pre registered tests. It’s not that humans are not smart enough, as I
110.
by aspenmartin 4mo ago
No it’s not selfless, but I can’t imagine a more shareholder minded CEO would not have done a slow rollout of mythos. The point is: creating ever stronger AI systems is what these companies do, it is integral to what they even are. If you t
111.
by aspenmartin 4mo ago
They are state of the art at all 3! As are other labs. Of all the labs they seem to take alignment and interpretability the most seriously to the point where they are hampering their own revenue in service of trying to not cause problems wh
112.
by aspenmartin 4mo ago
You want Anthropic to fund your healthcare or something? Also, have you seen the impact of these models on healthcare? Also most of our GDP growth this year is from AI buildouts, would you rather that be negative? And not even considering:
113.
by aspenmartin 4mo ago
If anything its slightly higher. Feel free to provide any evidence to the contrary. ECI (good aggregate measure using IRT): https://epoch.ai/eci?view=graph&tab=release-date&subset-view... METR time horizon (now topp
114.
by aspenmartin 4mo ago
You can call it a cult but it’s several thousand skilled workers who know what they’re doing, by and large, most of whom have a PhD and know how science and statistics work. Benchmarks are incredibly hard, and any PR or comms department at
115.
by aspenmartin 4mo ago
And you can say “If it can’t do the most basic of things at least as good as it used to, this is table stakes” all day long while people point you to much better evidence to the contrary too, I’d rather be on the other side of that.
116.
by aspenmartin 4mo ago
Curious what your idea would be here for a truly good actor in this space; no AI development?
117.
by aspenmartin 4mo ago
That’s your call just don’t expect anyone ever to take that seriously. It’s not like we don’t have exact evaluations like this.
118.
by aspenmartin 4mo ago
Again correct but it overstates the issue. I can say labs don’t want this. This happened arguably unintentionally in Metas llama 4 release, it went horribly, heads rolled, and like several billion dollars were paid for new talent and the or
119.
by aspenmartin 4mo ago
Yes but there’s a reason we don’t evaluate these models this way and instead do it as carefully and thoughtfully as we can at scale. Human evaluations are important but they are an absolute minefield of footguns. 4.8 is not a downgrade from
120.
by aspenmartin 4mo ago
Your observations are right but pretty insane to consider them a pure PR company lol. They are making more frequent releases so yes the release-to-release quality is smaller but we’re still ascending quality and reliability curves the same

More ›