2 ms·

> This is a bad comparison. How so? First, kernel writing (or ML engineering more broadly) is a highly specialised task. Not everyone can do it. It shows that

> This is a bad comparison. How so? First, kernel writing (or ML engineering more broadly) is a highly specialised task. Not everyone can do it. It shows that models are getting better and better at (easily verifiable) hard tasks. And you can "hire" that expertise much easier than you can hire the equivalent meatbags. And more importantly you can "fire" them as soon as the task is done. And then hire them 3 months later, when the new model drops. And so on. Second, 20% gains in inference today gives better end results (i.e. lower overall cost) than 1% in training 2 years ago. Today's models are improving mostly via RL. And RL is highly dependant on fast inference (you want many rollouts for each training scenario). Same for dataset filtering, environment generation, distillation, etc.

elil17 3mo ago
> How so? Without looking at the specific changes made, it is impossible to know whether this represents a real capability improvement. For example, maybe the 20% gain was due to a very obvious/easy to catch inefficiency while the 1% improvement was due to something subtle.
- NitpickLawyer 3mo ago
  I'm not convinced. Both are improvements on something implemented by highly trained/credentialed/successful teams. Something "missed" by either team is impressive either way.
  - elil17 2mo ago
    Regardless of how impressive each are on their own, it just doesn't tell you anything about their comparative capabilities.
magicalist 2mo ago
> How so? First, kernel writing (or ML engineering more broadly) is a highly specialised task. Not everyone can do it. It shows that models are getting better and better at (easily verifiable) hard tasks. No it doesn't. You have no idea how optimised the underlying kernels already were. If a kernel was already optimal, it doesn't matter if you bring a brain the size of a planet, you're not going improve it. > Second, 20% gains in inference today gives better end results (i.e. lower overall cost) than 1% in training 2 years ago Your claim (at least in implication) was some kind of trend here. It doesn't matter which one was better to optimise because it "gives better end results" if you're trying to make a trend line out of two unrelated things. I'm looking for more efficient ways to pack items into packaging. You're looking for more efficient ways to dispatch workers. I report how much faster I can get a shipping container loaded. You report how much less it costs to to get powerlines fixed during a storm. It's a bad comparison.

elil17 3mo ago

> How so? Without looking at the specific changes made, it is impossible to know whether this represents a real capability improvement. For example, maybe the 20% gain was due to a very obvious/easy to catch inefficiency while the 1% improvement was due to something subtle.

NitpickLawyer 3mo ago
I'm not convinced. Both are improvements on something implemented by highly trained/credentialed/successful teams. Something "missed" by either team is impressive either way.
- elil17 2mo ago
  Regardless of how impressive each are on their own, it just doesn't tell you anything about their comparative capabilities.

magicalist 2mo ago

> How so? First, kernel writing (or ML engineering more broadly) is a highly specialised task. Not everyone can do it. It shows that models are getting better and better at (easily verifiable) hard tasks. No it doesn't. You have no idea how optimised the underlying kernels already were. If a kernel was already optimal, it doesn't matter if you bring a brain the size of a planet, you're not going improve it. > Second, 20% gains in inference today gives better end results (i.e. lower overall cost) than 1% in training 2 years ago Your claim (at least in implication) was some kind of trend here. It doesn't matter which one was better to optimise because it "gives better end results" if you're trying to make a trend line out of two unrelated things. I'm looking for more efficient ways to pack items into packaging. You're looking for more efficient ways to dispatch workers. I report how much faster I can get a shipping container loaded. You report how much less it costs to to get powerlines fixed during a storm. It's a bad comparison.