3 ms·

I don't buy into it. The benefits of abstraction hold for machines, as they can spend less bandwidth when modeling, using and modifying a system, and error is m

I don't buy into it. The benefits of abstraction hold for machines, as they can spend less bandwidth when modeling, using and modifying a system, and error is minimized during long, repetitive operations. Abstraction can be thought of as a system of interfaces. The right abstractions can totally transform how a human or machine interpret and solve a problem. The most effective and elegant machines will still make use of abstraction above machine code.

aforwardslash 2y ago
Many modern compilers already do this - they generate intermediate code, and then a specialized binary is generated. Why couldn't an LLM generate intermediate code directly?
- soulofmischief 2y ago
  Again, abstraction allows for domain-specific language which shapes the way a problem is interpreted and solved. Perspective is everything, and it's hard to have perspective of the entire system on the ground floor. Getting the priors right, or getting the LLM to first consider different grammars can greatly simplify the problem and its solution. You can think of learning as decontextualization of input data into pure relational constructs. Application, then, is the recontextualization of some relational construct in order to apply generalized relations and insights to problems of different shapes and sizes. Understanding a concept thus means mastering the decontextualization and recontextualization of a set of relations. Using a DSL instead of pure machine code is an example of recontextualization, and it can totally change how a model understands and processes input and what kind of output is produced.
- chippiewill 2y ago
  Going from high-level to the intermediate representation loses a lot of the semantic information about why something has been implemented in a certain way. LLMs essentially read their own code when producing it, so if they lose the abstractions then they can lose the context around what they were doing and why they were doing it. It also makes it harder for them to modify their existing code. I could certainly believe that it's plausible that there's a low level representation that LLMs would find easier than our own languages though - but an existing IR is unlikely to be it. Also more practically speaking, LLMs struggle to write in languages that they don't have a large training dataset for.

aforwardslash 2y ago

Many modern compilers already do this - they generate intermediate code, and then a specialized binary is generated. Why couldn't an LLM generate intermediate code directly?

soulofmischief 2y ago
Again, abstraction allows for domain-specific language which shapes the way a problem is interpreted and solved. Perspective is everything, and it's hard to have perspective of the entire system on the ground floor. Getting the priors right, or getting the LLM to first consider different grammars can greatly simplify the problem and its solution. You can think of learning as decontextualization of input data into pure relational constructs. Application, then, is the recontextualization of some relational construct in order to apply generalized relations and insights to problems of different shapes and sizes. Understanding a concept thus means mastering the decontextualization and recontextualization of a set of relations. Using a DSL instead of pure machine code is an example of recontextualization, and it can totally change how a model understands and processes input and what kind of output is produced.
chippiewill 2y ago
Going from high-level to the intermediate representation loses a lot of the semantic information about why something has been implemented in a certain way. LLMs essentially read their own code when producing it, so if they lose the abstractions then they can lose the context around what they were doing and why they were doing it. It also makes it harder for them to modify their existing code. I could certainly believe that it's plausible that there's a low level representation that LLMs would find easier than our own languages though - but an existing IR is unlikely to be it. Also more practically speaking, LLMs struggle to write in languages that they don't have a large training dataset for.