5 ms·

I don't think it's going to happen in the next few years the prices are gonna drop like hell, but ain't no way we run models meant to run on 8 nvidia A100 on o

• Analog24 3y ago
  This is the reason why they're not going to move on device anytime soon. You can use compression techniques, sure, but you're not going to get anywhere near the level of performance of GPT-4 at a size that can fit on most consumer devices
  • mkaic 3y ago
    I think we’ll see completely new architectures dominate in the near future, ousting the transformer. I am strongly suspicious that, while impressive, transformers use several orders of magnitude more compute than is “needed” for the tasks they perform—if for no other reason because the human brain performs similarly and it only draws 20 watts! And it isn’t even an engineered system, jus the product of a very, very long history of natural selection! I fully anticipate that we’ll see AI in the near future that achieves human-level performance on sub-human power budgets like the ones you’d be constrained by on a phone :)
    • Analog24 3y ago
      "neat future" is very ambiguous. At the moment there is nothing even close to transformers in terms of performance. I suspect you are right in general but I'm not sure about the "near future" part, there needs to be a pretty significant paradigm shift for that to happen (which is possible, of course, I just don't see any hints of it yet).
      • mkaic 3y ago
        RWKV is an attention-free architecture that's showing promising scaling at a similar level to Transformers right now! There's also recently been Hyena, which uses a new mechanism that's kind of a weird mix of attention, convolution, and implicit modelling all at once. It's shown promise as well. Remains to be seen if these competing methods will truly scale as well as Transformers, but I've got my fingers crossed. Only a matter of time! I agree that "near future" is quite ambiguous though. If I were to disambiguate my claims, I think I'd personally expect a Transformer-killing architecture to arise in the next 4-5 years.
• w4ffl35 3y ago
  > but ain't no way we run models meant to run on 8 nvidia A100 on our smartphones in the next 5 years m$ has been working on an AI chip since 2019 so i think we will.
  • d33 3y ago
    They still can't beat the laws of physics. It's just the kind of density that doesn't seem feasible to put in a smartphone.
    • wewtyflakes 3y ago
      Human brains seem to be of the same order of magnitude in terms of size, and seem more nuanced and capable than GPT4.
      • fbhabbed 3y ago
        It's not apples to apples comparison. There's a multitude of tasks that human brains are very bad at. GPT excellence is in raw knowledge and answering machine. You won't find a single human brain that can hold the same amount of knowledge
      • kortex 3y ago
        The human brain is also three-dimensional, heavily interconnected, and has built-in thermal management at every scale. Chips are much faster, but still operate on the essentially linear memory cells, and this limits how many matmuls you can do per second. If we can figure out true connectivity without doing tons of matmuls, then we should be able to massively cut computational demands of models.
  • what_ever 3y ago
    And? Nvidia has been working on AI chips for years.
  • throwawayadvsec 3y ago
    An A100 is about the size of a brick, there is no way we're fitting those 8 bricks in a phone in the next five years, without even thinking about heat management
• davnicwil 3y ago
  the only thing I can say to this is that Apple have seemed laser focused on tuning their silicon for ML crunching, that that focus is clearly now going to be amped up further still, and that in tandem the software itself will be tuned to Apple silicon. GPUs on the other hand are pretty general purpose. And 5 years on a focused superlinear ramp up is a long time, lots can happen. I am not saying it's 100%, or even 80% likely. It'll be super impressive if it happens, but I see it as well within the realms of reason.
  • tehsauce 3y ago
    Apple's new M2 Max has a neural engine which can do 15 trillion flops. Nvidias's A100 chip (released almost 3 years ago) can do 315 trillion flops. Apple is not going to close this 20x gap in a few years.
    • davnicwil 3y ago
      right, it's a huge challenge. I think the tuning the models to the hardware piece is important, and of course there is much more incentive to do this for Apple than nvidia because of the distribution and ecosystem advantages Apple have. But also, I don't know... let's see what the curve looks like! It's only been a couple of years of these neural engines. Let's see how many flops M3 can hit this year. And then m4 the next. Again, 5 years is a long time actually when real improvement is happening. I am optimistic.
    • moffkalast 3y ago
      > this 168x gap FTFY, remember it takes 8 of those to even load the thing. And when the average laptop has that much compute, GPT 4 will seem like Cleverbot in comparison to the state of the art.
    • sroussey 3y ago
      At some point, they will put the models in silicon. I’m curious as to when… 5yr?
      • viraptor 3y ago
• viraptor 3y ago
  > but ain't no way we run models meant to run on 8 nvidia A100 on our smartphones in the next 5 years When I leaned about neutral networks, the general advice at the time was "you'll only need one hidden layer, with somewhere between the number of your input and output neurons". While that was more than 5 years ago, my point is - both the approach and the architecture changes over time. I would not bet on what we won't have in 5 years.
• brucethemoose2 3y ago
  Vicuna (LLama) 13B is pretty good IMO. A 20B model can definitely fit in RAM on future devices.