> It’s not that they’re just learning codebases, they’re learning what working with code looks like. Working in any not-in-training-set environment very quickl

> It’s not that they’re just learning codebases, they’re learning what working with code looks like. Working in any not-in-training-set environment very quickly shows the shortcomings of this belief. For example, Cloudflare Workers is V8 but it sure ain't Node, and the local sqlite in a Durable Object has a sync API with very different guarantees than a typical client-server SQL setup. Even in a more standard setting, it's really hard to even get an LLM to use the current-stable APIs when its training data contains now-deprecated examples. Your local rules, llms.txt mentions, corrections etc slip out of the context pretty fast and it goes back to trained data. The LLM can perhaps "read any code" but it really really prefers writing only code that was in its training set.