aspenmartin

7 ms·

61.
by aspenmartin 4mo ago
Right but the issue is users have full control over context. A security-violating action by a coding agent in one context can be completely innocuous under other contexts etc, or breaking down the task into multiple tasks that in isolation
62.
by aspenmartin 4mo ago
Why are we missing the point here: LLMs are of major geopolitical significance. Autonomous drones kill children, now what? Would you rather have a home grown frontier model capability to defend yourself? Or would you prefer to rely on the g
63.
by aspenmartin 4mo ago
You are totally right, reading emails and vibe coding saas apps is all we can do. Omg why didn’t we think of that? Someone get the pentagon on the phone and cancel the drone warfare technology they’re already using and proving out in the ba
64.
by aspenmartin 4mo ago
LLMs are a “nice to have” blows my mind. LLMs power autonomous drones that kill real people in real wars. Coding agents are the primary way people code. That and we get a GPT4->5 jump every 18 months. Are we on the same planet?
65.
by aspenmartin 4mo ago
> That businesses view it as essential...is not a profitability argument. What do you mean, as in it doesn't imply profitability (and profitability of what ?) today ? > Businesses also bought dot com infrastructure, telecom fib
66.
by aspenmartin 4mo ago
You can do what frontier labs do today which is to properly license things that are copyrighted and use open source web crawls for things that don’t have copyright issues. You can then also commission new datasets (volume needed goes down w
67.
by aspenmartin 4mo ago
You can grumble about the way things are but Europe being so far behind a technological race with important geopolitical ramifications means that you guys are cutting off your nose to spite your face. Have fun with zero of European regulati
68.
by aspenmartin 4mo ago
You know what I'm surprised to find out this is far more feasible than I assumed; DiLoCo + INTELLECT models demonstrate how feasible decentralized training is already, that is very surprising to me that you can get that far with so muc
69.
by aspenmartin 4mo ago
> The US government basically has to nationalize AI and capture an outsize portion of the revenue from it in order to fix the economy, as the combination of debt burden and interest rate pressure from de-dollarization/global realign
70.
by aspenmartin 4mo ago
It's the most naive opinion that keeps getting shoveled around. You have a product that is viewed as essential by businesses, with revenue growing by 10x a year and geopolitical ramifications that have continued to rear their heads and
71.
by aspenmartin 4mo ago
It's more than the raw hardware, it's the interconnect and communication between the hardware at scale. These models are trained on hundreds of thousands of GPUs today. You _will_ start to see cross-datacenter training runs but th
72.
by aspenmartin 4mo ago
Yes but not violations of the laws of physics. You need extremely fast communications, memory bandwidth, etc; you cannot get that with distributed training. You're up against the speed of light and the interconnect that powers the inte
73.
by aspenmartin 4mo ago
This is really a feel good argument and I agree with what he’s saying in principle but it offers zero in terms of a practical strategy or stable state where this is feasible. If you want to jump on the bandwagon then let’s put our pants on
74.
by aspenmartin 4mo ago
This is really wonderful -- One thing that may be really cool if you have the data is to add a time-axis ability (unless I missed it) for a given location. This is such a delightful application of AI!
75.
by aspenmartin 4mo ago
Absolutely. All of these "but you could have done that easily" from frontend developers or backend developers or systems engineers -- like yea, if I have the time or interest in those things, sure. But I don't. I care about a
76.
by aspenmartin 4mo ago
Those are some incredible sprinkles.
77.
by aspenmartin 4mo ago
Oh ok well then you’re definitely right about that, they can tell and sometimes it really matters (I can’t remember if it was SWEBench or not but there was a major benchmark where the models were just inspecting git histories that were leak
78.
by aspenmartin 4mo ago
Yes but so what right? This is a problem for both alignment evals and actual cheating (e.g. someone forgot to delete .git history and the model was able to back out the original PR, or they can decrypt something by finding a key, etc), but
79.
by aspenmartin 4mo ago
They don't have control over measurement. Consider also it's easy to figure this out and it creates a scandal. Like I said, consider Llama 4 which a lot of people pointed out used a custom model in LMArena to inflate their scores;
80.
by aspenmartin 4mo ago
Its not a limit scenario is my point: these models are evaluated constantly, new benchmarks both public and proprietary are in constant development, benchmarks are not always static either, they can often times be living benchmarks that upd
81.
by aspenmartin 4mo ago
Yes I've seen this before, and while the critiques are fair and high quality (and unfortunately not unique to METR) we're missing the forest for the trees here. First of all, if you take the articles critiques and work out the imp
82.
by aspenmartin 4mo ago
I appreciate the data here but I don't think the read is quite right; Saying we have linear capability for super-linear cost compares an unbounded variable (dollars) to bounded instruments (because benchmarks saturate). On unbounded me
83.
by aspenmartin 4mo ago
Some people sure, but some people believe the earth is flat. What’s without a doubt is the impact AI has already had, something maybe 2 years ago was dismissed as religious fervor maybe. Statistics and robust measurements plus stable scalin
84.
by aspenmartin 4mo ago
Ooof a Yann LeCun quote from '24, a risky time period to be mining his quotes from lol. But I think broadly he's right, but the argument is: we don't see any evidence for the curve flattening, and we should plan for it _not_
85.
by aspenmartin 4mo ago
> Tech people are following a religious belief system whose utopian promise is the all-powerful computer that will end all suffering. Uh, I don't really think that's anywhere close to an accurate characterization of most people
86.
by aspenmartin 4mo ago
I think at this point the "instilling" part is not nearly as challenging and thorny as "what values should we instill"; that part is hard to imagine going away as it feels pretty fundamental to humanity that wars have be
87.
by aspenmartin 4mo ago
It's nice that people are genuinely curious about this. - All of your observations are absolutely dead on - Yet, we have very very very robust scaling laws that as Dario points out we've had and validated for over a decade. This e
88.
by aspenmartin 4mo ago
> The hype is coming to an end I'm sure you have evidence for this
89.
by aspenmartin 4mo ago
Why does this have anything to do with what I’m saying, of course the models are updated. I’m saying a new benchmark isn’t public and the model wouldn’t know they are being evaluated on a new benchmark. Not to mention: thinking that the api
90.
by aspenmartin 4mo ago
> Sure you can, just do it silently and don't tell the people hitting your API that the model is different now. Unless it's open weight, we're just taking your word for it. Even better, do a VW and try to detect which benc

More ›