4 ms·

Have you tried falcon 40b instruct? Also take into account that chatgpt likely has some preprompt and by talking to falcon or other OS models it's all in your h

Have you tried falcon 40b instruct? Also take into account that chatgpt likely has some preprompt and by talking to falcon or other OS models it's all in your hands. Furthermore, Not many people discuss the significance of proper output sampling. I myself used to just test open source models with the greedy decoding only. Who knows if they wouldn't even beat (not at all)OpenAI with some clever output sampling scheme.

schappim 3y ago
Does anyone have a link / instructions on getting Falcon 40b to install on Apple Silicon? Apparently "Hugging Face" have some internal swift code that works (but it has not been released). I'm keen to see how it performs on a maxed out Mac Studio (with all that unified memory available).
- _ea1k 3y ago
  I thought this too, but it sounds like the performance of the GPU ultimately holds it back. Maybe it is the software, but I have yet to see a test on Mac silicon that performed well.
  - schappim 3y ago
    It is not too bad, 7B is doing 432 tokens per second on a Macbook[1]. [1] Video: https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/147_falcon/falcon-7b.mp4 https://huggingface.co/datasets/huggingface/documentation-im...
moffkalast 3y ago
Idk, has anyone tried falcon yet? The support for running it remains nonexistent except for one fork of llama.cpp that isn't integrated into anything. This trend of every new model breaking compatibility really needs to stop.
- Roark66 3y ago
  I have and I am running it locally. Mostly in 7b variant which runs pretty much at chatgpt speed (when streaming) on my ryzen 3700 cpu + 32gb ram + 2xnvme in a mirror (it doesn't fit in the memory in its entirety, few gb go into the swap). Of course to run it like that I have to be running nothing else. No xorg, no chromium etc. Just a pure linux console. If you want to try falcon 40b instruct by yourself here I'd a public demo : https://huggingface.co/blog/falcon https://huggingface.co/blog/falcon Go to the bottom of the page.

schappim 3y ago

Does anyone have a link / instructions on getting Falcon 40b to install on Apple Silicon? Apparently "Hugging Face" have some internal swift code that works (but it has not been released). I'm keen to see how it performs on a maxed out Mac Studio (with all that unified memory available).

_ea1k 3y ago
I thought this too, but it sounds like the performance of the GPU ultimately holds it back. Maybe it is the software, but I have yet to see a test on Mac silicon that performed well.
- schappim 3y ago
  It is not too bad, 7B is doing 432 tokens per second on a Macbook[1]. [1] Video: https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/147_falcon/falcon-7b.mp4 https://huggingface.co/datasets/huggingface/documentation-im...

moffkalast 3y ago

Idk, has anyone tried falcon yet? The support for running it remains nonexistent except for one fork of llama.cpp that isn't integrated into anything. This trend of every new model breaking compatibility really needs to stop.

Roark66 3y ago
I have and I am running it locally. Mostly in 7b variant which runs pretty much at chatgpt speed (when streaming) on my ryzen 3700 cpu + 32gb ram + 2xnvme in a mirror (it doesn't fit in the memory in its entirety, few gb go into the swap). Of course to run it like that I have to be running nothing else. No xorg, no chromium etc. Just a pure linux console. If you want to try falcon 40b instruct by yourself here I'd a public demo : https://huggingface.co/blog/falcon https://huggingface.co/blog/falcon Go to the bottom of the page.