4 ms·
I find that AI models are very bad at doing performance work because they keep guessing how their changes affect things or not really understanding how the prof
by saagarjha 7mo ago
I find that AI models are very bad at doing performance work because they keep guessing how their changes affect things or not really understanding how the profiler results work, leading to them going in circles and taking forever. I have noticed this effect in surprisingly few lines of code (hundreds).
- michaelmior 7mo agoOne thing I've found that I've found super helpful for this is converting profiling results to Markdown and feeding it back into the agent in a loop. I've done it with a bit of manual orchestration, but it could probably be automated pretty well. Specifically, pprof-rs[0] and pprof-to-md[1] have worked pretty well for me, YMMV. [0] https://github.com/tikv/pprof-rs https://github.com/tikv/pprof-rs [1] https://github.com/platformatic/pprof-to-md https://github.com/platformatic/pprof-to-md
- saagarjha 7mo agoYes but the problem is that the agent reads the profile and doesn't seem to really understand how to improve things. For example, it will see "cycles are spent in GC" and make up a bunch of reasons why that might be happening.
- logicallee 7mo agodo you have a solution that still includes the model in the loop or do you just look at the profiler results yourself and then direct it what to do?
- saagarjha 7mo agoWell I basically cannot use it for performance decisions at all. I make all the choices and use it for simple things like "find me the top functions in this profile" or "refactor this code to test xyz prototype".