4 ms·

I'm not saying this is a bad idea, but it does sound like a rather risky prospect. You're basically proposing a bet against the ability of LLMs to generalize ac

I'm not saying this is a bad idea, but it does sound like a rather risky prospect. You're basically proposing a bet against the ability of LLMs to generalize across programming languages, and to embed concepts at a deeper level than the syntax. Many people do think this, but I'm not sure many of them are running AI labs.

tough 1y ago
it feels to me most of the real usage of AI is in coding right now, so a small lab that decided to go all in into just code-gen would have at least the differentiator of a narrower field to beat the bigger incumbents doing it all? I dunno tho. Big AI labs also have their own agendas and would rather keep scaling and growing than serving a rather smaller real market ? Once you're into real usage territory, you can't no longer use make up numbers to justify future growth.
- eigenspace 1y ago
  Again though, my point was just that it's not actually clear that you can do better than these big models by taking a narrower focus. I'm saying that that the things these big LLMs are learning about other languages probably do have utility when applied even to quite niche languages. If you take some niche language and build an LLM from scratch that's hyperspecialized on that language, will that LLM actually outperform some big LLM that's trained on all the programming resources out there, and all the blogs, forum conversations, stack overflow posts on all those languages, and then learns to generalize that information and apply it to your niche language? One of the things that LLMs seem to excel at is taking information from one context, transforming it and applying it to another context.
  - tough 1y ago
    So how I envision this would be like a dual system, you let the frontier bigger LLM come up with the overall function signature, structure, and reasoning/planning around the specific code, but then have it ask the hyperspecialized fine-tuned model which can only output valid code, to create it. You get then best of both worlds at the expense of a double-round trip or x2, which for something like coding seems fine, people are OK paying 200 for ChatGPT Plus This also would solve the context window sizes problem of them getting full and the model starting to generate non-sense, if you have the bigger model using the bigger context window to orchestrate and organize the task calling smaller specialized sub-modules, that seems like it should yield better final code outptus than just one big ass LLM but we'r'e moving the goalposts from 1 model to multi-agentic system i guess so nvermind and i agree it seems all the big corps are betting for bigger more data for now
harperlee 1y ago
From my experience around less-used languages (with clojure on one hand and code aster's python on the other), LLMs may be able to generalize syntax but availability of APIs, functions, etc. is something that you can't solve by generalizing. Or more precisely, you can generalize but that means hallucinating non existing tools.
- unshavedyak 1y ago
  Would non-generalizing solve this issue for libraries though? Ie a lot of models produce reasonable code for me, but i almost always care about usage of libraries. That's where they get the wrong version, or hallucinate, or etc for me.
- JohnMakin 1y ago
  General purpose LLM's fail really hard at this in domains like terraform. There may be drastic differences in syntax and semantics between the massive matrix of terraform version + provider version(s) and they've shown to be absolutely terrible at navigating that, even if you specify versions specifically. Even worse, and probably what exacerbates it, this version matrix changes at a much faster pace than most programming languages typically introduce large changes.
  - _w1tm 1y ago
    > There may be drastic differences in syntax and semantics between the massive matrix of terraform version + provider version(s) and they've shown to be absolutely terrible at navigating that, even if you specify versions specifically. To be fair, humans have trouble with that as well.
    - JohnMakin 1y ago
      That is always the counter but that is not the promise of these tools to be as bad as humans are. Humans can also work their way through it. The context ruts these tools get into with terraform specifically they can never dig out of, it's pretty much worthless for it, at least as many times as I've tried. You will waste far more time trying to figure out where it messed up on something dead simple than if you just looked over a colleague's shoulder, who probably wouldn't run into the same kinds of basic mistakes like making up fields.
robrenaud 1y ago
Meta synthetically generated lots of PHP from Python for Llama 3 for training purposes. Meta writes a crazy amount of PHP internally. Translation tends to be way easier than unconstrained generation for LLMs. But if you can translate and filter a large amount of code, you can learn to generate. If you also translate and run the unittests, you get another layer of error checking. https://arxiv.org/abs/2407.21783 https://arxiv.org/abs/2407.21783 See figure 8.

tough 1y ago

it feels to me most of the real usage of AI is in coding right now, so a small lab that decided to go all in into just code-gen would have at least the differentiator of a narrower field to beat the bigger incumbents doing it all? I dunno tho. Big AI labs also have their own agendas and would rather keep scaling and growing than serving a rather smaller real market ? Once you're into real usage territory, you can't no longer use make up numbers to justify future growth.

eigenspace 1y ago
Again though, my point was just that it's not actually clear that you can do better than these big models by taking a narrower focus. I'm saying that that the things these big LLMs are learning about other languages probably do have utility when applied even to quite niche languages. If you take some niche language and build an LLM from scratch that's hyperspecialized on that language, will that LLM actually outperform some big LLM that's trained on all the programming resources out there, and all the blogs, forum conversations, stack overflow posts on all those languages, and then learns to generalize that information and apply it to your niche language? One of the things that LLMs seem to excel at is taking information from one context, transforming it and applying it to another context.
- tough 1y ago
  So how I envision this would be like a dual system, you let the frontier bigger LLM come up with the overall function signature, structure, and reasoning/planning around the specific code, but then have it ask the hyperspecialized fine-tuned model which can only output valid code, to create it. You get then best of both worlds at the expense of a double-round trip or x2, which for something like coding seems fine, people are OK paying 200 for ChatGPT Plus This also would solve the context window sizes problem of them getting full and the model starting to generate non-sense, if you have the bigger model using the bigger context window to orchestrate and organize the task calling smaller specialized sub-modules, that seems like it should yield better final code outptus than just one big ass LLM but we'r'e moving the goalposts from 1 model to multi-agentic system i guess so nvermind and i agree it seems all the big corps are betting for bigger more data for now

harperlee 1y ago

From my experience around less-used languages (with clojure on one hand and code aster's python on the other), LLMs may be able to generalize syntax but availability of APIs, functions, etc. is something that you can't solve by generalizing. Or more precisely, you can generalize but that means hallucinating non existing tools.

unshavedyak 1y ago
Would non-generalizing solve this issue for libraries though? Ie a lot of models produce reasonable code for me, but i almost always care about usage of libraries. That's where they get the wrong version, or hallucinate, or etc for me.
JohnMakin 1y ago
General purpose LLM's fail really hard at this in domains like terraform. There may be drastic differences in syntax and semantics between the massive matrix of terraform version + provider version(s) and they've shown to be absolutely terrible at navigating that, even if you specify versions specifically. Even worse, and probably what exacerbates it, this version matrix changes at a much faster pace than most programming languages typically introduce large changes.
- _w1tm 1y ago
  > There may be drastic differences in syntax and semantics between the massive matrix of terraform version + provider version(s) and they've shown to be absolutely terrible at navigating that, even if you specify versions specifically. To be fair, humans have trouble with that as well.
  - JohnMakin 1y ago
    That is always the counter but that is not the promise of these tools to be as bad as humans are. Humans can also work their way through it. The context ruts these tools get into with terraform specifically they can never dig out of, it's pretty much worthless for it, at least as many times as I've tried. You will waste far more time trying to figure out where it messed up on something dead simple than if you just looked over a colleague's shoulder, who probably wouldn't run into the same kinds of basic mistakes like making up fields.

robrenaud 1y ago

Meta synthetically generated lots of PHP from Python for Llama 3 for training purposes. Meta writes a crazy amount of PHP internally. Translation tends to be way easier than unconstrained generation for LLMs. But if you can translate and filter a large amount of code, you can learn to generate. If you also translate and run the unittests, you get another layer of error checking. https://arxiv.org/abs/2407.21783 https://arxiv.org/abs/2407.21783 See figure 8.