4 ms·

What is the model architecture? I'm assuming it's far away from LLMs, but I'm curious about knowing more. Can anyone provide links that describe architectures f

What is the model architecture? I'm assuming it's far away from LLMs, but I'm curious about knowing more. Can anyone provide links that describe architectures for VLA?

KoolKat23 1y ago
Actually very close to one I'd say. It's a "visual language action" VLA model "built on the foundations of Gemini 2.0". As Gemini 2.0 has native language, audio and video support, I suspect it has been adapted to include native "action" data too, perhaps only on output fine-tuning rather than input/output at training stage (given its Gemini 2.0 foundation). Natively multimodal LLM's are basically brains.
- martythemaniak 1y ago
  OpenVLA is basically a slightly modified, fine-tuned llama2. I found the launch/intro talk by lead author to be quite accessible: https://www.youtube.com/watch?v=-0s0v3q7mBk https://www.youtube.com/watch?v=-0s0v3q7mBk
  - m00x 1y ago
    A more modern one, smolVLA is similar and uses a VLM but skips a few layers and uses an action adapter for outputs. Both are from HF and run on LeRobot. https://arxiv.org/abs/2506.01844 https://arxiv.org/abs/2506.01844 Explanation by PhosphoAI: https://www.youtube.com/watch?v=00A6j02v450 https://www.youtube.com/watch?v=00A6j02v450
  - KoolKat23 1y ago
    In the paper at the bottom of googles page, this VLA says it is built on the foundations of Gemini 2.0 (hence my quotations). They'd be using Gemini 2.0 rather than llama. https://arxiv.org/pdf/2503.20020 https://arxiv.org/pdf/2503.20020
- quantumHazer 1y ago
  > Natively multimodal LLM's are basically brains. Absolutely not.
  - KoolKat23 1y ago
    Lol keep telling yourself that. It's not a human brain nor is it necessarily a very intelligent brain, but it is a brain nonetheless.
    - quantumHazer 1y ago
      Not a useful commentary. ANN and BNN are slightly correlated. That fact that you want to believe it is a brain tells a lot about you, but it doesn’t make a model a brain. Only suggestion I have is “study more”.
      - KoolKat23 1y ago
        They're not merely slightly correlated. If it looks like a duck and quacks like a duck... Just because it is alien to you, does not mean it is not a brain, please go look up the definition of the word. And my comment is useful, a VLA implies it is processing it's input and output natively, something a brain does hence my comment.

What is the model architecture? I'm assuming it's far away from LLMs, but I'm curious about knowing more. Can anyone provide links that describe architectures for VLA?

KoolKat23 1y ago

Actually very close to one I'd say. It's a "visual language action" VLA model "built on the foundations of Gemini 2.0". As Gemini 2.0 has native language, audio and video support, I suspect it has been adapted to include native "action" data too, perhaps only on output fine-tuning rather than input/output at training stage (given its Gemini 2.0 foundation). Natively multimodal LLM's are basically brains.

martythemaniak 1y ago
OpenVLA is basically a slightly modified, fine-tuned llama2. I found the launch/intro talk by lead author to be quite accessible: https://www.youtube.com/watch?v=-0s0v3q7mBk https://www.youtube.com/watch?v=-0s0v3q7mBk
- m00x 1y ago
  A more modern one, smolVLA is similar and uses a VLM but skips a few layers and uses an action adapter for outputs. Both are from HF and run on LeRobot. https://arxiv.org/abs/2506.01844 https://arxiv.org/abs/2506.01844 Explanation by PhosphoAI: https://www.youtube.com/watch?v=00A6j02v450 https://www.youtube.com/watch?v=00A6j02v450
- KoolKat23 1y ago
  In the paper at the bottom of googles page, this VLA says it is built on the foundations of Gemini 2.0 (hence my quotations). They'd be using Gemini 2.0 rather than llama. https://arxiv.org/pdf/2503.20020 https://arxiv.org/pdf/2503.20020
quantumHazer 1y ago
> Natively multimodal LLM's are basically brains. Absolutely not.
- KoolKat23 1y ago
  Lol keep telling yourself that. It's not a human brain nor is it necessarily a very intelligent brain, but it is a brain nonetheless.
  - quantumHazer 1y ago
    Not a useful commentary. ANN and BNN are slightly correlated. That fact that you want to believe it is a brain tells a lot about you, but it doesn’t make a model a brain. Only suggestion I have is “study more”.
    - KoolKat23 1y ago
      They're not merely slightly correlated. If it looks like a duck and quacks like a duck... Just because it is alien to you, does not mean it is not a brain, please go look up the definition of the word. And my comment is useful, a VLA implies it is processing it's input and output natively, something a brain does hence my comment.