4 ms·
This rings true for me. LLMs in my experience are great at Go, a little less good at Java, and much less good at GCL (internal config language). This is defini
by danpalmer 7mo ago
This rings true for me. LLMs in my experience are great at Go, a little less good at Java, and much less good at GCL (internal config language).
This is definitely partly training data, but if you give an LLM a simple language to use on the fly it can usually do ok. I think the real problem is complexity.
Go and Java require very little mental modelling of the problem, everything is written down on the page really quite clearly (moreso with Go, but still with Java).
In GCL however the semantics are _weird_, the scoping is unlike most languages, because it's designed for DSLs. Humans writing DSL content requires little thought, but authoring DSLs requires a fair amount of mental modelling about the structure of the data that is not present on the page. I'd wager that Lisp is similar, more of a mental model is required.
The problem is of course that LLMs don't have a mental model, or at least what they do have is far from what humans have. This is very apparent when doing non-trivial code, non-CRUD, non-React, anything that requires thinking hard about problems more than it requires monkeys at typewriters.
- eldenring 7mo agoHow many docs do you put in the context? we maintain a lot of dsl code internally, and each file has a copy of the spec + guide as a comment at the top. Its about 50 locs and the relevant models are great at writing it.
- danpalmer 7mo agoOh yeah the models are great at writing the DSLs, there are enough examples to do that very effectively. It's the building of the DSL, which is implemented in the config language, which is tricky. i.e, writing a new A/B test in the language is trivial, writing an A/B testing config DSL in the language is hard. The main problem is the dynamic scoping (as opposed to lexical scoping like most languages), and the fact that lots of things are untyped and implicitly referenced.
- miki123211 7mo agoI bet it would do much better at hcl (or Starlark, maybe even yaml, something that it has seen plenty of examples of in the wild). This is a weird moment in time where proprietary technology can hurt more than it can help, even if it's superior to what's available in public in principle.
- pjmlp 7mo agoDepends if the AI masters also own said proprietary technology.
- miki123211 7mo agoWell, GCL is (afaik) a Google technology, and they do have some kind of internal, fine-tuned models just for their stack. Who owns the tech doesn't matter, what matters is whether there's a set of diverse examples of its use spread around the internet.
- danpalmer 7mo agoYeah it's internal, and we have fine tuned models and more lines of it than you can imagine. That's the reason I think it honestly depends more on the complexity to understand and the necessity of having a mental model of the code.