5 ms·

It would be nice for the inference time to be paired with measure of output quality. I'm not well versed in how the architecture works, but I have a hard time b

• not_a_dane 2y ago
  Nothing is free in this world.
• zackangelo 2y ago
  I've only read the abstract but they don't mention quantizing the weights or otherwise trying to shrink the model in any way. They're claiming to be able to efficiently run larger models without loading the entire thing into GPU memory. If they're using the same weights, the same architecture and just using tensor parallel operations to perform the forward pass that would imply no loss in quality. I'm sure there are trade-offs but they're not clear by just looking at the abstract.
  • tgtweak 2y ago
    I read it like this too - no drop in weights or model quality just optimizing the lower boundaries of performance when you are splitting from vram to ram to disk (or network).
• freehorse 2y ago
  From what I get skimming through the article the main cost is speed of token generation (token latency). You can always run a large model by reading directly from the disk and not care much about ram; but it is very slow. They try to improve that aspect doing some optimisations, but it is still definitely slower than using ram or vram.
  • refulgentis 2y ago
    Table 3 directly refutes this* and claims 0 tradeoffs.** Below that, they indicate that a key part of the implementation is loading weights from disk before they're needed using a separate thread.*** * maybe I'm missing something though, someone please triple check :) ** ttft (time to first token) and s/token (seconds per token) are both lower than any alternative in all cases. *** "daemon thread asynchronously preloads the weights"
    • sgc 2y ago
      I want to add that their chart shows s/token per device (edit: as per the heading on table 1 - it could also be confusing grammar), so it sounds like you are getting 4x the listed s/t on their 4 laptop cluster . Their laptops are not even hardwired - they are connecting over wifi. This comes at a very interesting time for me. I have an ancient dual xeon workstation with 64gb memory that I was researching how to convert to run an llm. I can just run that with 4 instances on the same machine and see how it goes, without purchasing a better GPU, to start. It sounds like this will allow you to run very large models with minimal quants, on craigslist quality devices. If it does what they say it does (and it seems to do), it will be an absolute game changer for most users.
• deleted 2y ago
  [deleted]
• woadwarrior01 2y ago
  It's not cost-free. It comes at the cost of greatly increased latency. 29.9 seconds per token with Llama 3.1-70B. This is from Table 1 (pg 8) of the paper.
  • m3kw9 2y ago
    Ah the disk swap method
    • thelastparadise 2y ago
      Is there any predictability/patterns for neuron/layer activation? If so, would it be reasonable to have a second tiny model that specifically tries to predict activation and preemptively swap those into memory?
      • tcdent 2y ago
        Depends on the architecture, but generally you just move through the layers linearly. Simple iteration. The number of layers, and the amount of time spent in each of them, makes me think any benefit from pre-loading the layer ahead is negligible. You really need the entire model on device to consider it performant.
      • miki123211 2y ago
        This isn't how neural networks work. For vanilla models, you always use all the weights. That isn't true for mixture-of-experts, though, and in that setting, your approach has merit.
    • _ache_ 2y ago
      It's not disk swap. It's multi-devices LLM.
      • kridsdale3 2y ago
        That looked like an analogy. Back in the days of a mechanical arm moving magnetic fields around in our PCs, you could have the illusion of infinite RAM as long as you're ok with microsecond operations now taking two million times longer. This is akin.