4 ms·

I'm using Mixtral 8x7b as a llamafile on an M1 regularly for coding help and general Q&A. It's really something wonderful to just run a single command and have

I'm using Mixtral 8x7b as a llamafile on an M1 regularly for coding help and general Q&A. It's really something wonderful to just run a single command and have this incredible offline resource.

tgma 3y ago
I concur; in my experience Mixtral is one of the best ~30G models (likely the best pro laptop-size model currently) and Gemma is quite good compared to other below 8GB models.
tchvil 3y ago
By any chance, do you have a good link to some help with the installation?
- yaantc 3y ago
  Use llamafile [1], it can be as simple as downloading a file (for mixtral, [2]), making it executable and running it. The repo README has all the info, it's simple and downloading the model is what takes the most time. In my case I got the runtime detection issue (explained in the README "gotcha" section). Solved my running "assimilate" [3] on the downloaded llamafile. [1] https://github.com/Mozilla-Ocho/llamafile/ [2] https://huggingface.co/jartine/Mixtral-8x7B-Instruct-v0.1-llamafile/resolve/main/mixtral-8x7b-instruct-v0.1.Q5_K_M.llamafile?download=true [3] https://cosmo.zip/pub/cosmos/bin/assimilate
  - tchvil 3y ago
    Thank you !
- tgma 3y ago
  Either https://lmstudio.ai https://lmstudio.ai (desktop app with nice GUI) or https://ollama.com https://ollama.com (command-like more like a docker container that you can also hook up to a web UI via https://openwebui.com https://openwebui.com) should be super straightforward to get running.
  - tchvil 3y ago
    Thank you for letting me know it was possible on an M1. I'll try all this now.
- chown 3y ago
  I am the author of Msty [1]. My goal is to make it as straightforward as possible with just one click (once you download the app). If you try it, let me know what you think. 1: https://msty.app https://msty.app
  - tchvil 3y ago
    I'll try in a week+ when I'm back to a fast connection. Thank you.
  - yunohn 3y ago
    Why is this both free and closed source? Ideally, when you advertise privacy-first, I’d like to see a GitHub link with real source code. Or I’d rather pay for it to ensure you have a financial incentive to not sell my data.
    - chown 3y ago
      It will be paid down the road, but we are not there yet. It’s all offline, data is locally saved. You own it, we don’t have it even if you ask for it.
      - yunohn 3y ago
        There’s incredible competition in this space already - I’d highly recommend outright stating your future pricing plans, instead of a bait-and-switch later.
  - omnibrain 3y ago
    Looks great. Can you recommend what GPU to get to just play with the models for a bit? (I want to have perform it fast, otherwise I lose interest too quickly). Are consumer GPUs like the RTX 4080 Super sufficient, or do I need anything else?
- firewolf34 3y ago
  Check out PrivateGPT on GitHub. Pretty much just works put of the box. I got Mistral7B running on a GTX 970 in about 30 minutes flat first try. Yep, that's the triple-digit GTX 970.

I'm using Mixtral 8x7b as a llamafile on an M1 regularly for coding help and general Q&A. It's really something wonderful to just run a single command and have this incredible offline resource.

tgma 3y ago

I concur; in my experience Mixtral is one of the best ~30G models (likely the best pro laptop-size model currently) and Gemma is quite good compared to other below 8GB models.

tchvil 3y ago

By any chance, do you have a good link to some help with the installation?

yaantc 3y ago
Use llamafile [1], it can be as simple as downloading a file (for mixtral, [2]), making it executable and running it. The repo README has all the info, it's simple and downloading the model is what takes the most time. In my case I got the runtime detection issue (explained in the README "gotcha" section). Solved my running "assimilate" [3] on the downloaded llamafile. [1] https://github.com/Mozilla-Ocho/llamafile/ [2] https://huggingface.co/jartine/Mixtral-8x7B-Instruct-v0.1-llamafile/resolve/main/mixtral-8x7b-instruct-v0.1.Q5_K_M.llamafile?download=true [3] https://cosmo.zip/pub/cosmos/bin/assimilate
- tchvil 3y ago
  Thank you !
tgma 3y ago
Either https://lmstudio.ai https://lmstudio.ai (desktop app with nice GUI) or https://ollama.com https://ollama.com (command-like more like a docker container that you can also hook up to a web UI via https://openwebui.com https://openwebui.com) should be super straightforward to get running.
- tchvil 3y ago
  Thank you for letting me know it was possible on an M1. I'll try all this now.
chown 3y ago
I am the author of Msty [1]. My goal is to make it as straightforward as possible with just one click (once you download the app). If you try it, let me know what you think. 1: https://msty.app https://msty.app
- tchvil 3y ago
  I'll try in a week+ when I'm back to a fast connection. Thank you.
- yunohn 3y ago
  Why is this both free and closed source? Ideally, when you advertise privacy-first, I’d like to see a GitHub link with real source code. Or I’d rather pay for it to ensure you have a financial incentive to not sell my data.
  - chown 3y ago
    It will be paid down the road, but we are not there yet. It’s all offline, data is locally saved. You own it, we don’t have it even if you ask for it.
    - yunohn 3y ago
      There’s incredible competition in this space already - I’d highly recommend outright stating your future pricing plans, instead of a bait-and-switch later.
- omnibrain 3y ago
  Looks great. Can you recommend what GPU to get to just play with the models for a bit? (I want to have perform it fast, otherwise I lose interest too quickly). Are consumer GPUs like the RTX 4080 Super sufficient, or do I need anything else?
firewolf34 3y ago
Check out PrivateGPT on GitHub. Pretty much just works put of the box. I got Mistral7B running on a GTX 970 in about 30 minutes flat first try. Yep, that's the triple-digit GTX 970.