4 ms·

Code llama Python is very interesting. Specifically tuned for Python. I wonder if we could make such specific LLMs (one that is proficient in all things Rust,

Code llama Python is very interesting. Specifically tuned for Python. I wonder if we could make such specific LLMs (one that is proficient in all things Rust, another- all things Linux, all things genomics, all things physics modeling etc) and have them talk to each other to collaboratively solve problems. That would be a crazy future thing! Putting machines truly to work..

brucethemoose2 3y ago
If you can find a large body of good, permissively licensed example code, you can finetune an LLM on it! There was a similar attempt for Godot script trained a few months ago, and its reportedly pretty good: https://github.com/minosvasilias/godot-dodo https://github.com/minosvasilias/godot-dodo I think more attempts havent been made because base llama is not that great at coding in general, relative to its other strengths, and stuff like Starcoder has flown under the radar.
deleted 3y ago
[deleted]
esperent 3y ago
I think this is called "mixture of experts" and also there's a lot of speculation that it's how GPT-4 works, although probably with just a few large models rather than many small ones.
- jmiskovic 3y ago
  It's been confirmed by multiple (unofficial) sources that GPT-4 is 8 models, each 220B parameters. Another rumor is GPT-4 being 16x111B models. There's a quite fresh and active project replicating something similar with herd of llamas: https://github.com/jondurbin/airoboros https://github.com/jondurbin/airoboros
  - codercowmoo 3y ago
    Can you provide some of these sources?
    - jmiskovic 3y ago
      All of these should have close ties with OpenAI, but no one from inside confirmed it as far as I know. https://twitter.com/swyx/status/1671272883379908608 https://twitter.com/swyx/status/1671272883379908608 https://twitter.com/soumithchintala/status/1671267150101721090 https://twitter.com/soumithchintala/status/16712671501017210... https://twitter.com/MParakhin/status/1670666605427298304 https://twitter.com/MParakhin/status/1670666605427298304
bbor 3y ago
Mark my words: you’ve caught a glimpse of the near future :). Google “Society of Mind” if you’re not yet familiar
seydor 3y ago
Start with a CodeLlama for C, and start treating these systems as natural language compilers. C is low level enough and still readable for those rare moments

brucethemoose2 3y ago

If you can find a large body of good, permissively licensed example code, you can finetune an LLM on it! There was a similar attempt for Godot script trained a few months ago, and its reportedly pretty good: https://github.com/minosvasilias/godot-dodo https://github.com/minosvasilias/godot-dodo I think more attempts havent been made because base llama is not that great at coding in general, relative to its other strengths, and stuff like Starcoder has flown under the radar.

deleted 3y ago

[deleted]

esperent 3y ago

I think this is called "mixture of experts" and also there's a lot of speculation that it's how GPT-4 works, although probably with just a few large models rather than many small ones.

jmiskovic 3y ago
It's been confirmed by multiple (unofficial) sources that GPT-4 is 8 models, each 220B parameters. Another rumor is GPT-4 being 16x111B models. There's a quite fresh and active project replicating something similar with herd of llamas: https://github.com/jondurbin/airoboros https://github.com/jondurbin/airoboros
- codercowmoo 3y ago
  Can you provide some of these sources?
  - jmiskovic 3y ago
    All of these should have close ties with OpenAI, but no one from inside confirmed it as far as I know. https://twitter.com/swyx/status/1671272883379908608 https://twitter.com/swyx/status/1671272883379908608 https://twitter.com/soumithchintala/status/1671267150101721090 https://twitter.com/soumithchintala/status/16712671501017210... https://twitter.com/MParakhin/status/1670666605427298304 https://twitter.com/MParakhin/status/1670666605427298304

bbor 3y ago

Mark my words: you’ve caught a glimpse of the near future :). Google “Society of Mind” if you’re not yet familiar

seydor 3y ago

Start with a CodeLlama for C, and start treating these systems as natural language compilers. C is low level enough and still readable for those rare moments