12 ms·

Getting Started with Mistral-7b-Instruct-v0.1

140 points by 3Sophons 3y ago

politelemon 3y ago
Mistral have created a docker image which hosts their model in vllm. Vllm creates an openai like http API interface. https://docs.mistral.ai/quickstart/ https://docs.mistral.ai/quickstart/
- deleted 3y ago
  [deleted]
- 3Sophons 3y ago
  While Mistral's Docker image provides a practical solution, it's important to note that Docker isn't truly cross-platform. A Docker image compiled for x86 architecture won't run on ARM, which can be a limitation for diverse hardware environments. On the other hand, Rust + WebAssembly (Wasm) offers genuine cross-platform capabilities. Wasm is not only faster to start and more resource-efficient but also more secure. This makes it exceptionally suitable for handling large language models, ensuring high performance while being cost-effective. In scenarios where diverse hardware support and efficiency are crucial, Rust + Wasm emerges as a superior choice over Docker.
  - hobofan 3y ago
    Is there any data to back up the "faster to start and more resource-efficient", especially if you compare it to the native non-Python solutions that most people are using to run LLMs on local machines? I'm as big of a fan of Rust and WASM as the next person, but throwing around claims like that without benchmarks is one of the quickest ways to get your product dismissed.
    - 3Sophons 3y ago
      Faster is compared to Python. Portable, more secure and lightweight are compared with Python and other native solutions. In terms of benchmarks, Rust / C++ is 50,000x faster than Python; WasmEdge runtime + portable app is 30M compared with 4G Python and 300MB llama.cpp Docker image that is NOT portable across CPU or GPU; Wasm sandbox is more secure than native binary.
      - hobofan 3y ago
        Thanks for clarifying!
    - 3Sophons 3y ago
      Details https://www.cncf.io/blog/2023/10/30/rust-webassembly-building-infrastructure-for-large-language-model-ecosystems/ https://www.cncf.io/blog/2023/10/30/rust-webassembly-buildin...
  - smcleod 3y ago
    Usually when you’re building container images at any scale you’d use something like buildx, kaniko or buildah which allow you to easily set multiple target architectures (AMD64, ARM64) for your images. There’s hardly any / basically no overhead to running an application in a container if that’s what you mean by overhead. If you mean the image size - well you only need to add the things you need - the problem is people tend to abuse images and install a lot of packages in the final image which absolutely aren’t required. Rust or golang at the core would be nice as Python can be slow at times. I do wish more folks would give Tauri a go for GUI apps.
iAkashPaul 3y ago
llama.cpp can run the Q4 variant of the same at 30tok/s on an M1 Pro unlike the 20tok/s being quoted
- rgbrgb 3y ago
  I get similar speeds on my m1, though it depends a bit on the quantization. If anyone else is on Mac and just want to have a play, FreeChat (my app) is a simple chat wrapper for llama.cpp. Clicking the little … on a message and checking the info will tell you the generation speed. [0]: https://www.freechat.run/ https://www.freechat.run/
jesterson 3y ago
Does anyone have any feedback on using those open-source models with any language except English? Particularly non-western group of languages like korean/japanese/chinese? Will assess myself but wonder if anyone tried.
- 3Sophons 3y ago
  For Chinese, there are this Yi Model by 0.1 AI Yi 34B https://www.secondstate.io/articles/yi-34b/ https://www.secondstate.io/articles/yi-34b/ Uncencored Yi: https://www.secondstate.io/articles/dolphin-2.2-yi-34b/ https://www.secondstate.io/articles/dolphin-2.2-yi-34b/ and also Baichuan https://www.secondstate.io/articles/baichuan2-13b-chat/ https://www.secondstate.io/articles/baichuan2-13b-chat/ There is an Arabic one called Jais mentioned by Satya a few days ago on the Microsoft dev day would like to try out
  - wahnfrieden 3y ago
    Yi has weird license
    - 3Sophons 3y ago
      emmm do you mean not truly open source
      - wahnfrieden 3y ago
        You must send them an email to get commercial use permission. It does not look like a lawyer wrote their license either
        3Sophons 3y ago
        I think you have to do similar things with Baichuan too. Which is also a bit pseudo open source
- nmfisher 3y ago
  Alibaba's Qwen is excellent for Chinese.
xeckr 3y ago
Thanks for sharing this! I read somewhere that Mistral 7B had a similar performance to GPT-3, but it seems to be miles behind it unfortunately.
- vidarh 3y ago
  Remember GPT-3 != ChatGPT. I could perhaps believe it is similar to GPT-3, but it's certainly miles behind ChatGPT even with 3.5
  - xeckr 3y ago
    I was talking about GPT-3, which I have interacted with via the developer API. To be fair to the developers of Mistral, it's still amazing that I can run this on my computer, and it's certainly better than the first LLaMA.
- s369610 3y ago
  I tried this fine tune of it https://huggingface.co/teknium/OpenHermes-2.5-Mistral-7B https://huggingface.co/teknium/OpenHermes-2.5-Mistral-7B and was impressed. It seems to be one of the better performing models on the LLM leaderboard https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...
tyayers 3y ago
Impressive use of wasmedge, great to see ML projects getting away from python for efficiency's sake, thanks for sharing!
udev4096 3y ago
How is this different than ollama (https://github.com/jmorganca/ollama https://github.com/jmorganca/ollama)? I would argue it's even simpler to run LLMs locally with ollama
- 3Sophons 3y ago
  Ollama's approach with Docker is indeed user-friendly, but Rust + WebAssembly (Wasm) offers some cool perks that are worth considering. With Wasm, you get true cross-platform support, meaning it runs smoothly on various devices, automagically uses the local hardware accelerator to run at full speed. It's also super lightweight, making it faster and more resource-efficient – a big plus for running LLMs, especially on devices with limited resources. A Wasm inference application can be as small as 2MB, including all dependencies, which is significantly smaller than most Docker containers.
  - hobofan 3y ago
    Ollama doesn't use Docker by default. While it is one of the install/download options, the default mode of operation is installing it as a native application (where they provide prebuilt binaries for the main platforms/architectures) into which you can easily load models from HuggingFace repos or local GGUF files.
    - hiatus 3y ago
      I don't understand. When you run `ollama pull` (or even `ollama run` which will pull if you don't have the model), that is using docker is it not? The code even references pulling from a registry. https://github.com/jmorganca/ollama/blob/main/cmd/cmd.go#L959 https://github.com/jmorganca/ollama/blob/main/cmd/cmd.go#L95...
      - _ea1k 3y ago
        ollama maintains a registry of ollama compatible llms. I don't see any indication that they use docker for that registry, but I could have overlooked something. I also don't see any docker/podman containers running when using ollama.
android521 3y ago
The demo looks too slow for practical usage. How much it will cost if i host it in cloud to get instant response similar to the speed of openAI?
- 3Sophons 3y ago
  https://github.com/second-state/WasmEdge-WASINN-examples/tree/master/wasmedge-ggml-llama-interactive#performance https://github.com/second-state/WasmEdge-WASINN-examples/tre... These are the speed of different hardwares. You can rent GPU, which will be much faster
- 3Sophons 3y ago
  The demos are as fast as the ChatGPT, they just look slower. -First. It might seem slow at first because it's loading the model. But once that's done, it gets much quicker for any follow-up requests. Second, Turning on streaming output helps a lot, as it shows responses as they're processed. And yes, the hardware matters too. A good GPU setup in the cloud can work wonders, though it might bump up the cost a bit. So actually the demo's not slower than ChatGPT overall; it just takes a moment to warm up at the start.
MaKey 3y ago
What are Mistrals strengths and weaknesses? I tried it for infrastructure as code and it wasn't able to output more than the most basic examples, let alone modify them.
- 3abiton 3y ago
  I never heard of IaaC before.
  - smcleod 3y ago
    I assume you mean IaC? It’s how you manage infrastructure with automation. Terraform, Ansible, CDK etc… back in the day it used to be for managing pets with Pupper, Chef and friends.
- hobofan 3y ago
  Mistrals main strength is that it is a good allrounder model with a 7B size, so it runs on a lot of consumer hardware. If you are looking for specific tasks to run locally (with decent RAM and no GPU), the realistic best option in most categories is a fine-tuned 13B model. In my experience most open source models do quite well with Python and Javascript, but don't perform so well when it comes to other languages and the usage of their special characteristics (e.g. it will still be able to write simple control flow in Rust, but probably not use async/await correctly). I've tried using LLMs in general with IaC (K8s resources/Helm charts), and they all did well when asking it a very specific thing about it (e.g. "Is there an alternative way to accomplish X?"), but also never had it perform well when outputting YAML directly (either from scratch of modifying it).
- jokethrowaway 3y ago
  Excellent at creative writing Pretty good at instructions Uncensored, no RHLF bs
  - ashupadhi01 3y ago
    Tell me how can I run it locally.
smcleod 3y ago
It seems a lot slower than ollama / LM Studio. Is that likely because it’s not optimised as much or a limitation of some sort?
xrd 3y ago
I really like the post that they mention (https://www.secondstate.io/articles/fast-llm-inference/ https://www.secondstate.io/articles/fast-llm-inference/). The reasons for avoiding python all resonate with me. I'm excited to play with WASI-NN (https://github.com/WebAssembly/wasi-nn https://github.com/WebAssembly/wasi-nn) and that rust code is very readable to load up a GGUL model.
- euazOn 3y ago
  Related discussion: https://news.ycombinator.com/item?id=38246668 https://news.ycombinator.com/item?id=38246668
tormeh 3y ago
The WasmEdge README gives me the heebie-jeebies. Starry-eyed emojis, highlights use-case for today's most trendy thing even though it's general-purpose, mentions blockchain. This reeks of former cryptobros chasing the next big thing. I'd trust Wasmtime more.
- electrondood 3y ago
  You do realize HuggingFace is named after an emoji?
  - tormeh 3y ago
    The emojis themselves aren't a problem. I love silliness. They just seem to be correlated with rugpulls.

140 points by 3Sophons 3y ago

Mistral have created a docker image which hosts their model in vllm. Vllm creates an openai like http API interface. https://docs.mistral.ai/quickstart/ https://docs.mistral.ai/quickstart/

deleted 3y ago
[deleted]
3Sophons 3y ago
While Mistral's Docker image provides a practical solution, it's important to note that Docker isn't truly cross-platform. A Docker image compiled for x86 architecture won't run on ARM, which can be a limitation for diverse hardware environments. On the other hand, Rust + WebAssembly (Wasm) offers genuine cross-platform capabilities. Wasm is not only faster to start and more resource-efficient but also more secure. This makes it exceptionally suitable for handling large language models, ensuring high performance while being cost-effective. In scenarios where diverse hardware support and efficiency are crucial, Rust + Wasm emerges as a superior choice over Docker.
- hobofan 3y ago
  Is there any data to back up the "faster to start and more resource-efficient", especially if you compare it to the native non-Python solutions that most people are using to run LLMs on local machines? I'm as big of a fan of Rust and WASM as the next person, but throwing around claims like that without benchmarks is one of the quickest ways to get your product dismissed.
  - 3Sophons 3y ago
    Faster is compared to Python. Portable, more secure and lightweight are compared with Python and other native solutions. In terms of benchmarks, Rust / C++ is 50,000x faster than Python; WasmEdge runtime + portable app is 30M compared with 4G Python and 300MB llama.cpp Docker image that is NOT portable across CPU or GPU; Wasm sandbox is more secure than native binary.
    - hobofan 3y ago
      Thanks for clarifying!
  - 3Sophons 3y ago
    Details https://www.cncf.io/blog/2023/10/30/rust-webassembly-building-infrastructure-for-large-language-model-ecosystems/ https://www.cncf.io/blog/2023/10/30/rust-webassembly-buildin...
- smcleod 3y ago
  Usually when you’re building container images at any scale you’d use something like buildx, kaniko or buildah which allow you to easily set multiple target architectures (AMD64, ARM64) for your images. There’s hardly any / basically no overhead to running an application in a container if that’s what you mean by overhead. If you mean the image size - well you only need to add the things you need - the problem is people tend to abuse images and install a lot of packages in the final image which absolutely aren’t required. Rust or golang at the core would be nice as Python can be slow at times. I do wish more folks would give Tauri a go for GUI apps.

iAkashPaul 3y ago

llama.cpp can run the Q4 variant of the same at 30tok/s on an M1 Pro unlike the 20tok/s being quoted

rgbrgb 3y ago
I get similar speeds on my m1, though it depends a bit on the quantization. If anyone else is on Mac and just want to have a play, FreeChat (my app) is a simple chat wrapper for llama.cpp. Clicking the little … on a message and checking the info will tell you the generation speed. [0]: https://www.freechat.run/ https://www.freechat.run/

jesterson 3y ago

Does anyone have any feedback on using those open-source models with any language except English? Particularly non-western group of languages like korean/japanese/chinese? Will assess myself but wonder if anyone tried.

3Sophons 3y ago
For Chinese, there are this Yi Model by 0.1 AI Yi 34B https://www.secondstate.io/articles/yi-34b/ https://www.secondstate.io/articles/yi-34b/ Uncencored Yi: https://www.secondstate.io/articles/dolphin-2.2-yi-34b/ https://www.secondstate.io/articles/dolphin-2.2-yi-34b/ and also Baichuan https://www.secondstate.io/articles/baichuan2-13b-chat/ https://www.secondstate.io/articles/baichuan2-13b-chat/ There is an Arabic one called Jais mentioned by Satya a few days ago on the Microsoft dev day would like to try out
- wahnfrieden 3y ago
  Yi has weird license
  - 3Sophons 3y ago
    emmm do you mean not truly open source
    - wahnfrieden 3y ago
      You must send them an email to get commercial use permission. It does not look like a lawyer wrote their license either
      - 3Sophons 3y ago
        I think you have to do similar things with Baichuan too. Which is also a bit pseudo open source
nmfisher 3y ago
Alibaba's Qwen is excellent for Chinese.

xeckr 3y ago

Thanks for sharing this! I read somewhere that Mistral 7B had a similar performance to GPT-3, but it seems to be miles behind it unfortunately.

vidarh 3y ago
Remember GPT-3 != ChatGPT. I could perhaps believe it is similar to GPT-3, but it's certainly miles behind ChatGPT even with 3.5
- xeckr 3y ago
  I was talking about GPT-3, which I have interacted with via the developer API. To be fair to the developers of Mistral, it's still amazing that I can run this on my computer, and it's certainly better than the first LLaMA.
s369610 3y ago
I tried this fine tune of it https://huggingface.co/teknium/OpenHermes-2.5-Mistral-7B https://huggingface.co/teknium/OpenHermes-2.5-Mistral-7B and was impressed. It seems to be one of the better performing models on the LLM leaderboard https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...

tyayers 3y ago

Impressive use of wasmedge, great to see ML projects getting away from python for efficiency's sake, thanks for sharing!

udev4096 3y ago

How is this different than ollama (https://github.com/jmorganca/ollama https://github.com/jmorganca/ollama)? I would argue it's even simpler to run LLMs locally with ollama

3Sophons 3y ago
Ollama's approach with Docker is indeed user-friendly, but Rust + WebAssembly (Wasm) offers some cool perks that are worth considering. With Wasm, you get true cross-platform support, meaning it runs smoothly on various devices, automagically uses the local hardware accelerator to run at full speed. It's also super lightweight, making it faster and more resource-efficient – a big plus for running LLMs, especially on devices with limited resources. A Wasm inference application can be as small as 2MB, including all dependencies, which is significantly smaller than most Docker containers.
- hobofan 3y ago
  Ollama doesn't use Docker by default. While it is one of the install/download options, the default mode of operation is installing it as a native application (where they provide prebuilt binaries for the main platforms/architectures) into which you can easily load models from HuggingFace repos or local GGUF files.
  - hiatus 3y ago
    I don't understand. When you run `ollama pull` (or even `ollama run` which will pull if you don't have the model), that is using docker is it not? The code even references pulling from a registry. https://github.com/jmorganca/ollama/blob/main/cmd/cmd.go#L959 https://github.com/jmorganca/ollama/blob/main/cmd/cmd.go#L95...
    - _ea1k 3y ago
      ollama maintains a registry of ollama compatible llms. I don't see any indication that they use docker for that registry, but I could have overlooked something. I also don't see any docker/podman containers running when using ollama.

android521 3y ago

The demo looks too slow for practical usage. How much it will cost if i host it in cloud to get instant response similar to the speed of openAI?

3Sophons 3y ago
https://github.com/second-state/WasmEdge-WASINN-examples/tree/master/wasmedge-ggml-llama-interactive#performance https://github.com/second-state/WasmEdge-WASINN-examples/tre... These are the speed of different hardwares. You can rent GPU, which will be much faster
3Sophons 3y ago
The demos are as fast as the ChatGPT, they just look slower. -First. It might seem slow at first because it's loading the model. But once that's done, it gets much quicker for any follow-up requests. Second, Turning on streaming output helps a lot, as it shows responses as they're processed. And yes, the hardware matters too. A good GPU setup in the cloud can work wonders, though it might bump up the cost a bit. So actually the demo's not slower than ChatGPT overall; it just takes a moment to warm up at the start.

MaKey 3y ago

What are Mistrals strengths and weaknesses? I tried it for infrastructure as code and it wasn't able to output more than the most basic examples, let alone modify them.

3abiton 3y ago
I never heard of IaaC before.
- smcleod 3y ago
  I assume you mean IaC? It’s how you manage infrastructure with automation. Terraform, Ansible, CDK etc… back in the day it used to be for managing pets with Pupper, Chef and friends.
hobofan 3y ago
Mistrals main strength is that it is a good allrounder model with a 7B size, so it runs on a lot of consumer hardware. If you are looking for specific tasks to run locally (with decent RAM and no GPU), the realistic best option in most categories is a fine-tuned 13B model. In my experience most open source models do quite well with Python and Javascript, but don't perform so well when it comes to other languages and the usage of their special characteristics (e.g. it will still be able to write simple control flow in Rust, but probably not use async/await correctly). I've tried using LLMs in general with IaC (K8s resources/Helm charts), and they all did well when asking it a very specific thing about it (e.g. "Is there an alternative way to accomplish X?"), but also never had it perform well when outputting YAML directly (either from scratch of modifying it).
jokethrowaway 3y ago
Excellent at creative writing Pretty good at instructions Uncensored, no RHLF bs
- ashupadhi01 3y ago
  Tell me how can I run it locally.

smcleod 3y ago

It seems a lot slower than ollama / LM Studio. Is that likely because it’s not optimised as much or a limitation of some sort?

xrd 3y ago

I really like the post that they mention (https://www.secondstate.io/articles/fast-llm-inference/ https://www.secondstate.io/articles/fast-llm-inference/). The reasons for avoiding python all resonate with me. I'm excited to play with WASI-NN (https://github.com/WebAssembly/wasi-nn https://github.com/WebAssembly/wasi-nn) and that rust code is very readable to load up a GGUL model.

euazOn 3y ago
Related discussion: https://news.ycombinator.com/item?id=38246668 https://news.ycombinator.com/item?id=38246668

tormeh 3y ago

The WasmEdge README gives me the heebie-jeebies. Starry-eyed emojis, highlights use-case for today's most trendy thing even though it's general-purpose, mentions blockchain. This reeks of former cryptobros chasing the next big thing. I'd trust Wasmtime more.

electrondood 3y ago
You do realize HuggingFace is named after an emoji?
- tormeh 3y ago
  The emojis themselves aren't a problem. I love silliness. They just seem to be correlated with rugpulls.