5 ms·

Cosmopolitan is a nice hack but I'd never use it. I really don't see the point, cross-platform software with multiple binaries is absolutely fine. I've shipped

Cosmopolitan is a nice hack but I'd never use it. I really don't see the point, cross-platform software with multiple binaries is absolutely fine. I've shipped cross-platform software and it's a one time trivial cost to prepare a multi OS pipeline. The actual problems come from dependencies and I'm sure you'll have to solve the same problems with cosmopolitan. Llamafile is another hack I'd never use - why limit myself to the models supported by llamafile when I can just install llama.cpp? (the web interface is also part of llama.cpp https://github.com/ggerganov/llama.cpp/pull/1998 https://github.com/ggerganov/llama.cpp/pull/1998) Moreover, people with GPUs will likely want to use something with better performance than llama.cpp (gptq or exllama2)

chaosharmonic 3y ago
Actually if you poke around at it a little more closely you'll find there's also a version of Llamafile that just ships llama.cpp, enabling you to supply a model of your choosing. Also the point of a tool like this to make the ecosystem more accessible to everyone, not just software developers.
- cyanydeez 3y ago
  gonna be awhile before sufficiently powerful hardware is GA for LLM
  - chaosharmonic 3y ago
    I mean, that depends somewhat on your use case. My primary development machine -- a Framework with a recent board and 32GB of RAM -- is perfectly capable of running smaller models on CPU, and while it's certainly easy to find where the ceiling is (and this obviously isn't the hardware profile of an average user), it's still more than sufficient for basic use Even if it does also chew through my battery...
csdvrx 3y ago
> I really don't see the point, cross-platform software with multiple binaries is absolutely fine. I've shipped cross-platform software and it's a one time trivial cost to prepare a multi OS pipeline. That's like people who said "I could do this dropbox thing in a weekend with just rsync, ssh and ftp" You think you can, but even if you might cover this "one time trivial cost" once, 1) it doesn't scale well so you'll end up not doing it that often 2) most other people can't. > people with GPUs likely want to use something with better performance End users like performance, but first they need to be able to use something at all. If you start with "compile this binary then download this 4Gb file then run that command in this terminal" you've lost 99% of the potential audience I think you are missing the forest for the tree if you only see a technological hack instead of the change of paradigm it allows.
- bordercases 3y ago
  > That's like people who said "I could do this dropbox thing in a weekend with just rsync, ssh and ftp" And so begins the cycle of enshitification!
- pseudosavant 3y ago
  I'm getting >50 tokens/second with an RTX 2070 Super 8GB on Mistral 7B with llamafile. More than adequate performance.
  - pseudosavant 3y ago
    For context, I'm running this on my basic mainstream laptop (6-core, IGP, Ryzen 5 5500U) with 16GB RAM. I'm getting ~6 tokens/second for mistral-7b-v0.1.Q4_K_M.

chaosharmonic 3y ago

Actually if you poke around at it a little more closely you'll find there's also a version of Llamafile that just ships llama.cpp, enabling you to supply a model of your choosing. Also the point of a tool like this to make the ecosystem more accessible to everyone, not just software developers.

cyanydeez 3y ago
gonna be awhile before sufficiently powerful hardware is GA for LLM
- chaosharmonic 3y ago
  I mean, that depends somewhat on your use case. My primary development machine -- a Framework with a recent board and 32GB of RAM -- is perfectly capable of running smaller models on CPU, and while it's certainly easy to find where the ceiling is (and this obviously isn't the hardware profile of an average user), it's still more than sufficient for basic use Even if it does also chew through my battery...

csdvrx 3y ago

> I really don't see the point, cross-platform software with multiple binaries is absolutely fine. I've shipped cross-platform software and it's a one time trivial cost to prepare a multi OS pipeline. That's like people who said "I could do this dropbox thing in a weekend with just rsync, ssh and ftp" You think you can, but even if you might cover this "one time trivial cost" once, 1) it doesn't scale well so you'll end up not doing it that often 2) most other people can't. > people with GPUs likely want to use something with better performance End users like performance, but first they need to be able to use something at all. If you start with "compile this binary then download this 4Gb file then run that command in this terminal" you've lost 99% of the potential audience I think you are missing the forest for the tree if you only see a technological hack instead of the change of paradigm it allows.

bordercases 3y ago
> That's like people who said "I could do this dropbox thing in a weekend with just rsync, ssh and ftp" And so begins the cycle of enshitification!
pseudosavant 3y ago
I'm getting >50 tokens/second with an RTX 2070 Super 8GB on Mistral 7B with llamafile. More than adequate performance.
- pseudosavant 3y ago
  For context, I'm running this on my basic mainstream laptop (6-core, IGP, Ryzen 5 5500U) with 16GB RAM. I'm getting ~6 tokens/second for mistral-7b-v0.1.Q4_K_M.