5 ms·

Did I read that its a 3B model? That means it can run on my 3060 laptop? Also, I imagine there isnt some oobabooga/automatic1111 for this?

seaal 2y ago
LM Studio is pretty nice and simple to setup. https://lmstudio.ai https://lmstudio.ai
sroussey 2y ago
Can probably run on your iPhone
- resource_waste 2y ago
  CPU isnt realistic, regardless what the marketers tell you.
  - simonw 2y ago
    iPhones have GPUs. I've successfully run Mistral on mine using this app: https://llm.mlc.ai/docs/deploy/ios.html https://llm.mlc.ai/docs/deploy/ios.html
    - resource_waste 2y ago
      Please don't say 'what is technically possible' It blurs what is realistically useful. Also 'GPUs' No. This is a major red flag man. They do not. The marketers told you this, and you believed them.
      - simonw 2y ago
        I don't get it. The iPhone does have a GPU, and it can be used to accelerate AI workloads. This isn't theoretical. You can see this for yourself by installing https://apps.apple.com/us/app/mlc-chat/id6448482937 https://apps.apple.com/us/app/mlc-chat/id6448482937 and turning off your wifi and running the quite capable Mistral 7B Instruct LLM on your phone. I've even used it to answer simple questions while I was offline and only had my phone with me. What am I missing here?
        resource_waste 2y ago
        What like 50 token responses? Buddy I'm cranking out 4k. Also LOL at the GPU thing. Its insane how well Apple marketing worked.
        simonw 2y ago
        I didn't say an iPhone was as good as a desktop with a dedicated graphics card. I said that an iPhone has a GPU and is capable of running a small but capable LLM. Most people don't know that's possible yet. Why do keep indicating that iPhones don't have a GPU?
  - regularfry 2y ago
    I can and do run phi-2/3 on cpu, which are in the same size class.
Jensson 2y ago
Image models are much smaller than language models, so they put some extra language features on an image model here and then it doesn't need to be that large.

seaal 2y ago

LM Studio is pretty nice and simple to setup. https://lmstudio.ai https://lmstudio.ai

sroussey 2y ago

Can probably run on your iPhone

resource_waste 2y ago
CPU isnt realistic, regardless what the marketers tell you.
- simonw 2y ago
  iPhones have GPUs. I've successfully run Mistral on mine using this app: https://llm.mlc.ai/docs/deploy/ios.html https://llm.mlc.ai/docs/deploy/ios.html
  - resource_waste 2y ago
    Please don't say 'what is technically possible' It blurs what is realistically useful. Also 'GPUs' No. This is a major red flag man. They do not. The marketers told you this, and you believed them.
    - simonw 2y ago
      I don't get it. The iPhone does have a GPU, and it can be used to accelerate AI workloads. This isn't theoretical. You can see this for yourself by installing https://apps.apple.com/us/app/mlc-chat/id6448482937 https://apps.apple.com/us/app/mlc-chat/id6448482937 and turning off your wifi and running the quite capable Mistral 7B Instruct LLM on your phone. I've even used it to answer simple questions while I was offline and only had my phone with me. What am I missing here?
      - resource_waste 2y ago
        What like 50 token responses? Buddy I'm cranking out 4k. Also LOL at the GPU thing. Its insane how well Apple marketing worked.
        simonw 2y ago
        I didn't say an iPhone was as good as a desktop with a dedicated graphics card. I said that an iPhone has a GPU and is capable of running a small but capable LLM. Most people don't know that's possible yet. Why do keep indicating that iPhones don't have a GPU?
- regularfry 2y ago
  I can and do run phi-2/3 on cpu, which are in the same size class.

Jensson 2y ago

Image models are much smaller than language models, so they put some extra language features on an image model here and then it doesn't need to be that large.