8 ms·

After reading the results I skipped back to the comment section to ask if this was real because it looks a little too good to be true, but figured I should chec

After reading the results I skipped back to the comment section to ask if this was real because it looks a little too good to be true, but figured I should check authors and it's Microsoft research and UCAS so yeah, real. This is going to change a lot of things, obviously the edge computing applications they point out, but also this is going to bottom out the cost of providing high-performance LLMs in the cloud. I don't know what that means for the economics long term, naively way less costs maybe means new entrants without an entire cloud available can compete easier? I do wonder if something like this has already been found and implemented by either OpenAI or Google.

anon373839 3y ago
It also means the largest models can be scaled up significantly with the same inference budget.
- llm_trw 3y ago
  Depends. The only paper they cite for training: https://arxiv.org/pdf/2310.11453.pdf https://arxiv.org/pdf/2310.11453.pdf doesn't improve training costs much and most models are already training constrained. Not everyone has $200m to throw at training another model from scratch.
  - arunk47 3y ago
    Is there any scope for indie builders?
    - llm_trw 3y ago
      Not really. These are slightly better for memory during pre-training and fine turning but not enough to make a 4090 usable even for a 7b model.
aurareturn 3y ago
After playing with OpenAI's GPT4 API, I'm quite convinced that LLMs would be in everything and everywhere today if inference cost is as low as loading a website and context size is 100x higher. In other words, only inference cost is holding it back from completely changing everything. So if we have a shortcut to getting something like GPT4 to run locally on a small device, watch out.
- rvnx 3y ago
  It's coming in October with the new Apple chip
  - sigmoid10 3y ago
    I'd be very surprised if Apple can put something on the level of GPT4 on a handheld. Remember, GPT4 is estimated to be around 1.7 trillion parameters. That's 3.4TB at 16 bit and it would still be ~340GB at 1.58bits. The best we can hope for is a low-ish level few billion parameter model. Which would still be cool on a phone, but as of today these models are nowhere near GPT4.
    - jairuhme 3y ago
      They won't have something at that size because as you pointed out, it is still huge. But depending on how they are used, smaller parameter models may be better for specific on-phone tasks that start to make the size of the model not a problem. GPT4 is so large because it is very general purpose with the goal seeming to be to answer anything. You could have a smaller model focused solely on Siri or something that wouldn't require the parameter size of GPT4
      - sigmoid10 3y ago
        The thing a about GPT4 that matters so much is not just raw knowledge retention, but complex, abstract reasoning and even knowing what it doesn't know. We haven't seen that yet in smaller models and it's unclear if it is even possible. The best we could hope for right now is a better natural language interface than Siri for calling OS functions.
wongarsu 3y ago
I wouldn't be surprised if this causes hardware startups to pop up that build accelerator cards tuned for this architecture. It seems stupidly simple to do inference in hardware, and with most of the training being quantized as well you might even be able to provide speedups (and energy savings) for training with reasonable investment and on cheaper processor nodes than what Nvidia is using. Sure, Nvidia might eat their lunch in a couple of years, but bitcoin ASICs prove that you can have a niche producing specialized processors, and VCs would probably jump at the thought of disrupting Nvidia's high margin business.
- anon291 3y ago
  There's like a million startups promising analog / bit-level computation, inference-only, cheap computation. There's rain.ai, d-matrix, etc.
btbuildem 3y ago
If this dethrones Nvidia, it would be a wonderful side effect
- rafaelero 3y ago
  It's more likely that Nvidia will offer support to INT2 in the next generation and keep their dominance.
  - Klipper3 3y ago
    INT2 ternary is equivalent to INT1 + binary mask. Nvidia supprted INT1 matrix multiply in RTX20 and RTX30 generations, nobody used it, so they removed INT1 support from RTX40 generation.
    - ByThyGrace 3y ago
      What I get from your comment is now older RTX gens are going to be in high demand soon.
  - ActionHank 3y ago
    "next generation" those two words mean a whole lot. Intel and AMD could also implement support in their "next generation" and that would be huge.

anon373839 3y ago

It also means the largest models can be scaled up significantly with the same inference budget.

llm_trw 3y ago
Depends. The only paper they cite for training: https://arxiv.org/pdf/2310.11453.pdf https://arxiv.org/pdf/2310.11453.pdf doesn't improve training costs much and most models are already training constrained. Not everyone has $200m to throw at training another model from scratch.
- arunk47 3y ago
  Is there any scope for indie builders?
  - llm_trw 3y ago
    Not really. These are slightly better for memory during pre-training and fine turning but not enough to make a 4090 usable even for a 7b model.

aurareturn 3y ago

After playing with OpenAI's GPT4 API, I'm quite convinced that LLMs would be in everything and everywhere today if inference cost is as low as loading a website and context size is 100x higher. In other words, only inference cost is holding it back from completely changing everything. So if we have a shortcut to getting something like GPT4 to run locally on a small device, watch out.

rvnx 3y ago
It's coming in October with the new Apple chip
- sigmoid10 3y ago
  I'd be very surprised if Apple can put something on the level of GPT4 on a handheld. Remember, GPT4 is estimated to be around 1.7 trillion parameters. That's 3.4TB at 16 bit and it would still be ~340GB at 1.58bits. The best we can hope for is a low-ish level few billion parameter model. Which would still be cool on a phone, but as of today these models are nowhere near GPT4.
  - jairuhme 3y ago
    They won't have something at that size because as you pointed out, it is still huge. But depending on how they are used, smaller parameter models may be better for specific on-phone tasks that start to make the size of the model not a problem. GPT4 is so large because it is very general purpose with the goal seeming to be to answer anything. You could have a smaller model focused solely on Siri or something that wouldn't require the parameter size of GPT4
    - sigmoid10 3y ago
      The thing a about GPT4 that matters so much is not just raw knowledge retention, but complex, abstract reasoning and even knowing what it doesn't know. We haven't seen that yet in smaller models and it's unclear if it is even possible. The best we could hope for right now is a better natural language interface than Siri for calling OS functions.

wongarsu 3y ago

I wouldn't be surprised if this causes hardware startups to pop up that build accelerator cards tuned for this architecture. It seems stupidly simple to do inference in hardware, and with most of the training being quantized as well you might even be able to provide speedups (and energy savings) for training with reasonable investment and on cheaper processor nodes than what Nvidia is using. Sure, Nvidia might eat their lunch in a couple of years, but bitcoin ASICs prove that you can have a niche producing specialized processors, and VCs would probably jump at the thought of disrupting Nvidia's high margin business.

anon291 3y ago
There's like a million startups promising analog / bit-level computation, inference-only, cheap computation. There's rain.ai, d-matrix, etc.

btbuildem 3y ago

If this dethrones Nvidia, it would be a wonderful side effect

rafaelero 3y ago
It's more likely that Nvidia will offer support to INT2 in the next generation and keep their dominance.
- Klipper3 3y ago
  INT2 ternary is equivalent to INT1 + binary mask. Nvidia supprted INT1 matrix multiply in RTX20 and RTX30 generations, nobody used it, so they removed INT1 support from RTX40 generation.
  - ByThyGrace 3y ago
    What I get from your comment is now older RTX gens are going to be in high demand soon.
- ActionHank 3y ago
  "next generation" those two words mean a whole lot. Intel and AMD could also implement support in their "next generation" and that would be huge.