5 ms·

Perhaps another unpopular opinion that can get the comment outright down-voted, but still... While jart's work is very interesting in nature and execution, comm

by larodi 3y ago

Perhaps another unpopular opinion that can get the comment outright down-voted, but still... While jart's work is very interesting in nature and execution, commendable stuff indeed of a person with very high IQ and discipline, I still wonder whether Justine simply can't get over the fact they got kicked out of the llama.cpp project (yes, I understand jart is frequenting HN, and also let's agree llama.cpp is at least as cool as jart's projcets). No, I'm not going in details of said dismissal, as both sides seem to have had their proper arguments, but still. And of course, I can imagine where the whole cosmopolitan thing comes from,... even as manifest of sorts for the idea of systems-neutrality and potentially gender fluidity. But I really wonder whether GGUF actually needs this, since llama.cpp already compiles and runs pretty much everywhere. Why introduce one more container? Who benefits from binary distribution of this sort?

ukuina 3y ago
> Why introduce one more container? Who benefits from binary distribution of this sort? Mass adoption comes from ease-of-use, which includes ease-of-deployment. Most of the HN crowd is savvy enough to run HF models directly in their terminal, but there's a reason Ollama, LM Studio, and Faraday are so popular. Until LLM runners are part of the base OS, methods like this are extremely helpful to reduce friction.
whstl 3y ago
I read the Github repository README and the comments here and I found absolutely nothing that could suggest the need for the first two paragraphs you wrote. It seems this stems from a misconception from your side about the purpose of this project. About your question in the third paragraph: This is totally orthogonal to GGUF, and a cursory reading of the README shows that it does uses GGUF. This is not about a new universal LLM format, this is about packing it in a universal executable that runs everywhere, using Cosmopolitan. Some examples do pack the executable and GGUF weights together in a single file, but that's not dissimilar from an self-executing zip, the only difference is that this executable is not OS-specific, so you can use the same exact binary for macOS or Linux, for example.
ElectricalUnion 3y ago
> llama.cpp already compiles and runs pretty much everywhere. Well, it simplifies things when you don't need to compile things. Also, you literally can't download or compile the wrong binary by mistake, it's the same binary for all supported processor/OSes Cartesian product matrix. > Why introduce one more container? It makes stuff more convenient. `application/zip` is also a ubiquitous standard. I doubt anyone is being "introduced to it". I also appreciate the fact that tooling for handling `application/zip` is very widespread, so you don't need totally bespoke tooling to retrieve the models from inside a `llamafile`. > Who benefits from binary distribution of this sort? Anyone that doesn't have a compiler SDK on their computer.

by larodi 3y ago

ukuina 3y ago

> Why introduce one more container? Who benefits from binary distribution of this sort? Mass adoption comes from ease-of-use, which includes ease-of-deployment. Most of the HN crowd is savvy enough to run HF models directly in their terminal, but there's a reason Ollama, LM Studio, and Faraday are so popular. Until LLM runners are part of the base OS, methods like this are extremely helpful to reduce friction.

whstl 3y ago

I read the Github repository README and the comments here and I found absolutely nothing that could suggest the need for the first two paragraphs you wrote. It seems this stems from a misconception from your side about the purpose of this project. About your question in the third paragraph: This is totally orthogonal to GGUF, and a cursory reading of the README shows that it does uses GGUF. This is not about a new universal LLM format, this is about packing it in a universal executable that runs everywhere, using Cosmopolitan. Some examples do pack the executable and GGUF weights together in a single file, but that's not dissimilar from an self-executing zip, the only difference is that this executable is not OS-specific, so you can use the same exact binary for macOS or Linux, for example.

ElectricalUnion 3y ago

> llama.cpp already compiles and runs pretty much everywhere. Well, it simplifies things when you don't need to compile things. Also, you literally can't download or compile the wrong binary by mistake, it's the same binary for all supported processor/OSes Cartesian product matrix. > Why introduce one more container? It makes stuff more convenient. `application/zip` is also a ubiquitous standard. I doubt anyone is being "introduced to it". I also appreciate the fact that tooling for handling `application/zip` is very widespread, so you don't need totally bespoke tooling to retrieve the models from inside a `llamafile`. > Who benefits from binary distribution of this sort? Anyone that doesn't have a compiler SDK on their computer.