When it comes to truly novel things, 20% of even modest success is a very high number. I worked in research heavy places (industry labs) over the last decade an

When it comes to truly novel things, 20% of even modest success is a very high number. I worked in research heavy places (industry labs) over the last decade and if 90% of things you try do not fail, your work is not ambitious enough. That is very hard thing for a SWE to live with, but such is the price of progress. The remaining 10% tend to make it worthwhile. 20% is twice that. It needs to go lower still - you’re not going to succeed by just finetuning yet another llama variant.