One thing I realized is just how much the harnesses are geared towards _not_ parsing files and take shortcuts. And even then I am very unimpressed at the speed

One thing I realized is just how much the harnesses are geared towards _not_ parsing files and take shortcuts. And even then I am very unimpressed at the speed these systems output code and the amount of tokens you consume doing fairly basic stuff is quite high. My gut feeling is that it will take at least a couple of orders of magnitude improvements before these LLMs can even hold large systems fully in their context, much less understand them holistically. And I don't see an order of magnitude improvement coming any time soon, it feels the last one was GPT 3.5.