4 ms·
From our experience, single or few line replacements generally fine, since many of the changes are many few-line changes in multiple spots accross multiple file
by kevinlu1248 3y ago
From our experience, single or few line replacements generally fine, since many of the changes are many few-line changes in multiple spots accross multiple files. We also provide surrounding context for the search-and-replace pairs, which helps with the model. Beyond 10 lines and the model also usually add the function headers in there which helps with the code generation.
I'm also curious, how are you guys evaluating the performance of your models?
- reitzensteinm 3y agoThere's no systematic evaluation yet which is the next step. It's successfully bootstrapping itself which is a fairly high bar, but quantitative performance measurements are getting more and more important as the project progresses.
- kevinlu1248 3y agoI feel the same, benchmarking in general is a pain but a good benchmark for us could go a long way.