3 ms·

On the SDK front, you tried just having an agent reimplement the model you're interested in and just use the weights? I've taken to treating off the shelf imple

On the SDK front, you tried just having an agent reimplement the model you're interested in and just use the weights? I've taken to treating off the shelf implementations as reference implementations anyway, because I can often squeeze out significantly better performance for my configuration and use case by having Codex hammer at it for a few hours.

drivebyhooting 13d ago
Could you share the prompts and workflow? I’ve tried this too, but with mixed success when it comes to creating custom tile kernels. I would really appreciate your input!
- mistercow 13d ago
  It's been pretty ad hoc, but my prompts are nothing special. Things I generally do: 1. Top end model on high/xhigh thinking (last time I did it it was Sol xhigh I think) 2. Make sure it creates some representative fixtures of different sizes and sets up a good testing, profiling and benchmarking loop that doesn't require my input. 3. Make sure it has access to reference implementation code Edit: Oh and one obvious pitfall that for some reason I still have to remind even smart models of from time to time: make sure it knows not to try to parallelize its benchmark runs. I've occasionally had an agent struggle to figure out absolutely nonsensical data because it tried to run multiple tests on the same compute hardware simultaneously.
sroussey 13d ago
Hugging face is working on something like this where well known models get fused into a single implementation.
- Melatonic 11d ago
  Good thing they're an independent org with no one owner that has a huge investment in a single hardware implementation
alfiedotwtf 12d ago
Woah! Custom to your chosen model and customer to your computer setup. Nice! Can you tell us what kind of gain you got ie. what was the speed bump
- mistercow 12d ago
  I think it was on qwen 3.5 9B or thereabouts that I saw a ~30% decode rate improvement on my M2 Mac vs MLX. That’s probably the best I’ve gotten. Keep in mind though that this is also sometimes use case dependent. Off the shelf implementations are generally pretty good overall, but can have pathological behavior on specific workload shapes you care about. So do this as a somewhat later optimization, and particularly when you see performance characteristics that don’t seem to make any sense.

drivebyhooting 13d ago

Could you share the prompts and workflow? I’ve tried this too, but with mixed success when it comes to creating custom tile kernels. I would really appreciate your input!

mistercow 13d ago
It's been pretty ad hoc, but my prompts are nothing special. Things I generally do: 1. Top end model on high/xhigh thinking (last time I did it it was Sol xhigh I think) 2. Make sure it creates some representative fixtures of different sizes and sets up a good testing, profiling and benchmarking loop that doesn't require my input. 3. Make sure it has access to reference implementation code Edit: Oh and one obvious pitfall that for some reason I still have to remind even smart models of from time to time: make sure it knows not to try to parallelize its benchmark runs. I've occasionally had an agent struggle to figure out absolutely nonsensical data because it tried to run multiple tests on the same compute hardware simultaneously.

sroussey 13d ago

Hugging face is working on something like this where well known models get fused into a single implementation.

Melatonic 11d ago
Good thing they're an independent org with no one owner that has a huge investment in a single hardware implementation

alfiedotwtf 12d ago

Woah! Custom to your chosen model and customer to your computer setup. Nice! Can you tell us what kind of gain you got ie. what was the speed bump

mistercow 12d ago
I think it was on qwen 3.5 9B or thereabouts that I saw a ~30% decode rate improvement on my M2 Mac vs MLX. That’s probably the best I’ve gotten. Keep in mind though that this is also sometimes use case dependent. Off the shelf implementations are generally pretty good overall, but can have pathological behavior on specific workload shapes you care about. So do this as a somewhat later optimization, and particularly when you see performance characteristics that don’t seem to make any sense.