5 ms·

Very unfavorably. Mostly because the ONNX models are FP32/FP16 (so ~3-4x the RAM use), but also because llama.cpp is well optimized with many features (like pro

Very unfavorably. Mostly because the ONNX models are FP32/FP16 (so ~3-4x the RAM use), but also because llama.cpp is well optimized with many features (like prompt caching, grammar, device splitting, context extending, cfg...) MLC's Apache TVM implementation is also excellent. The autotuning in particular is like black magic.

skeletoncrew 3y ago
I tried quite a few of these and the ONNX one seems the most elegantly put together of all. I’m impressed. Speed can be improved. Quick and dirty/hype solutions, not sure. I really hope ONNX gets traction it deserves.
- brucethemoose2 3y ago
  > ONNX one seems the most elegantly put together of all. What do you mean by this? The demo UI? Code quality?
- version_five 3y ago
  > Quick and dirty/hype solutions, not sure. Curious what you mean by this
  - refulgentis 3y ago
    It's tough to hear and communicate, but TL;DR: it's very good for HN headlines to make a $X.cpp but it's not the right tool for products. There's only going to be more of these models and ONNX starts from the right place, cross-platform and from base principles rather than coupling tightly to one model structure. Most importantly, it is freakin' awesome, the comments thus far, 30 in, don't reflect what its like to use or it's technical realities.* * the main threads of discussion are "not even wrong": float16 is big compared to float4 (its trivial to quantize to your liking) and looking for an alternative that supports CoreML (ONNX is the magic that makes your model take advantage of CoreML / WebGPU / WebGL / whatever Android's marketing name for its API is etc. etc. etc.)
    - kiratp 3y ago
      When hardware is so expensive and so difficult to obtain, performance starts to trump the cost of “single model implementation” pretty quickly. It can be cheaper to deploy Llama.cpp and then foobar.cpp (6 months from now) than it is to have inference that is 2x slower. Interestingly in the LLM space all these model servers seem to be converging to using the same API as OpenAI, making it easy to swap containers to get a different model+inference server with 0 code change.
    - redox99 3y ago
      > it's very good for HN headlines to make a $X.cpp but it's not the right tool for products. Funny how it's the opposite. ONNX in this case, outside of the HN headline and saying "we did it" is almost useless. LLMs are so heavy that you can't afford running a suboptimized version. This FP16 ONNX takes 4x as much memory and is probably 5-10x slower than something hand optimized such as llama.cpp or exllama with 4 bit quants.
      - refulgentis 3y ago
        Right, and as my comment alludes to, quantize it to FP4, they got a nice CLI tool that took me about 15 seconds to do that to a 150 MB model. Here youd do that with each shard
        deleted 3y ago
        [deleted]
        brucethemoose2 3y ago
        Straight 4 bit quantization with LLMs sacrifices some quality. The currently popular frameworks quantize the models in blocks to get better results, which also makes runtime performance less straightforward. I think ONNX would need to natively support this "packed" format or otherwise quantize fp16 models from disk on the fly... Which is a problem, as the FP16 models are huge.
        ImprobableTruth 3y ago
- mrbungie 3y ago
  Unless I'm working in an academic project I couldn't care less about elegancy. Speed (and ergo costs) trumps elegancy in industrial settings imho, specially considering how expensive to run LLMs are. Such an OSS dependency can be improved or at least "wrapped" to isolate it from the rest of your codebase.
  - brucethemoose2 3y ago
    As do features. Even if llama.cpp was ~30% slower and the same size as ONNX, something like a custom grammar implementation or an extended context would be a huge deal.
    - kanwisher 3y ago
      ONNX doesn’t give you any benefits over something like llama.cpp and it’s significantly slower for zero advantage
Havoc 3y ago
Speaking of MLC - recently discovered they have a iphone app that can do lama7b locally on high end iphones at decent pace. Bit hard to find in the store given the ocean of API front end apps - called MLCchat.

skeletoncrew 3y ago

I tried quite a few of these and the ONNX one seems the most elegantly put together of all. I’m impressed. Speed can be improved. Quick and dirty/hype solutions, not sure. I really hope ONNX gets traction it deserves.

brucethemoose2 3y ago
> ONNX one seems the most elegantly put together of all. What do you mean by this? The demo UI? Code quality?
version_five 3y ago
> Quick and dirty/hype solutions, not sure. Curious what you mean by this
- refulgentis 3y ago
  It's tough to hear and communicate, but TL;DR: it's very good for HN headlines to make a $X.cpp but it's not the right tool for products. There's only going to be more of these models and ONNX starts from the right place, cross-platform and from base principles rather than coupling tightly to one model structure. Most importantly, it is freakin' awesome, the comments thus far, 30 in, don't reflect what its like to use or it's technical realities.* * the main threads of discussion are "not even wrong": float16 is big compared to float4 (its trivial to quantize to your liking) and looking for an alternative that supports CoreML (ONNX is the magic that makes your model take advantage of CoreML / WebGPU / WebGL / whatever Android's marketing name for its API is etc. etc. etc.)
  - kiratp 3y ago
    When hardware is so expensive and so difficult to obtain, performance starts to trump the cost of “single model implementation” pretty quickly. It can be cheaper to deploy Llama.cpp and then foobar.cpp (6 months from now) than it is to have inference that is 2x slower. Interestingly in the LLM space all these model servers seem to be converging to using the same API as OpenAI, making it easy to swap containers to get a different model+inference server with 0 code change.
  - redox99 3y ago
    > it's very good for HN headlines to make a $X.cpp but it's not the right tool for products. Funny how it's the opposite. ONNX in this case, outside of the HN headline and saying "we did it" is almost useless. LLMs are so heavy that you can't afford running a suboptimized version. This FP16 ONNX takes 4x as much memory and is probably 5-10x slower than something hand optimized such as llama.cpp or exllama with 4 bit quants.
    - refulgentis 3y ago
      Right, and as my comment alludes to, quantize it to FP4, they got a nice CLI tool that took me about 15 seconds to do that to a 150 MB model. Here youd do that with each shard
      - deleted 3y ago
        [deleted]
      - brucethemoose2 3y ago
        Straight 4 bit quantization with LLMs sacrifices some quality. The currently popular frameworks quantize the models in blocks to get better results, which also makes runtime performance less straightforward. I think ONNX would need to natively support this "packed" format or otherwise quantize fp16 models from disk on the fly... Which is a problem, as the FP16 models are huge.
        ImprobableTruth 3y ago
mrbungie 3y ago
Unless I'm working in an academic project I couldn't care less about elegancy. Speed (and ergo costs) trumps elegancy in industrial settings imho, specially considering how expensive to run LLMs are. Such an OSS dependency can be improved or at least "wrapped" to isolate it from the rest of your codebase.
- brucethemoose2 3y ago
  As do features. Even if llama.cpp was ~30% slower and the same size as ONNX, something like a custom grammar implementation or an extended context would be a huge deal.
  - kanwisher 3y ago
    ONNX doesn’t give you any benefits over something like llama.cpp and it’s significantly slower for zero advantage

Havoc 3y ago

Speaking of MLC - recently discovered they have a iphone app that can do lama7b locally on high end iphones at decent pace. Bit hard to find in the store given the ocean of API front end apps - called MLCchat.