3 ms·

This doesn’t make sense - there are literally thousands of academic AI research labs who are severely limited by compute resources. If anything could work bette

by p1esk 3y ago

This doesn’t make sense - there are literally thousands of academic AI research labs who are severely limited by compute resources. If anything could work better than transformers and require less compute they would be all over that.

behnamoh 3y ago
I guess the argument is that most AI research is supported by the big tech, and they have heavily invested in the deep learning approach. If the fundings were funneled to research groups working on alternative approaches, maybe we'd see the same amount of progress in AI only using another approach.
- kettleballroll 3y ago
  As a member of the research community: that's nonsense. Like already pointed out: academic groups (who by no means are dependent on big tech) would jump all over that. Mamba has been out long enough that you'd already see tons of papers at arxiv showing mamba dominating transformers in all sorts of applications. But that's not happening, despite the ton of hype. That doesn't mean that mamba is nonsense. Just that it isn't the immediate transformer killer. It remains to be seen if something comes from it, eventually.
  - godelski 3y ago
    As a member of the research community: that's nonsense. Publishing is an extremely noisy process in ML and is getting increasingly difficult for smaller non big tech collaborating labs. Reviewers' go to are: more datasets, scale, not novel. The easiest way to approach this is to work off of pretrained models. This is probably more obvious in the NLP world. I agree that Mamba doesn't solve everything and it still needs work. But I disagree with the logic that there isn't an issue of railroading.
    - p1esk 3y ago
      What’s the main difference between an ape’s brain and a human brain? Scale. So that’s the train we’re riding at the moment. No roadblocks yet, aside from cost.
      - godelski 3y ago
        > What’s the main difference between an ape’s brain and a human brain? Scale. This is incredibly naive with absolutely no scientific basis. There is no evidence that this is in scale of data nor scale of architecture. There are a number of animals with larger brains in terms of both mass and total number of neurons. An African Elephant has roughly 3x the number of neurons humans have. Dolphins beat humans in total surface area. Neanderthals are estimated to have had larger brains too! It isn't mass, neurons, neuron density, surface area. We aren't just scaled up chimps.
        p1esk 3y ago
        Other animals with larger brains might have other bottlenecks preventing them from reaching full potential of their intelligence. Neanderthals might have been smarter than us, but went extinct for reasons not related to intelligence. But my point stands - our brains have evolved directly from apes brains and the main difference between them and us is brain size.
        godelski 3y ago
        > Other animals with larger brains might have other bottlenecks >>> What’s the main difference between an ape’s brain and a human brain? Scale. Your argument is inconsistent. Very clearly everything isn't scale or we'd use other things besides transformers. Different architectures scale in different ways and everything has different inductive biases. No one doubts scale is important, but there's a lot more.
        p1esk 3y ago
        Scale is all we need for transformers (so far). It might also be all we need for ape brains. It’s not all we need for whatever elephant or dolphin brains evolved from. When this stops being the case for transformers, we will need something else. I’m just pointing out it’s not the case yet.
  - algo_trader 3y ago
    > Just that it isn't the immediate transformer killer. What is the best/stable-ish linear alternative for transformer right now? Especially for text generation and summarization. We have domain specific ways of over sampling and search, so we much prefer less expensive models.
- landryraccoon 3y ago
  Why would Meta, Microsoft, Amazon and Google want Nvidia to remain dominant in hardware? Are you treating “big tech” like they all have one hive mind?
  - behnamoh 3y ago
    For MSFT, AMZN, GOOG, the competitive advantage comes from having huge datasets (that Nvidia doesn't have). It's a symbiosis that benefits the data-rich and GPU-rich players.
    - pixl97 3y ago
      This still makes little sense as that scale will always matter. If you can drop the compute cost of a model by 10x it means you can increase model integrity/intelligence/speed etc beyond what your compute bound competitors have. Simply put, for the time being huge datasets are going to be needed and those with bigger (cleaner?) datasets will have a better behaving model.
    - fauigerzigerk 3y ago
      Where is the symbiosis? If data is the differentiator, how do the data owners benefit from Nvidia eating into their margins?
      - coldtea 3y ago
        It's sumbiosis based on a common factor they both appreciate: - the data/processing having to be large means the data-owners have a benefit - the data/processing having to be large means NVIDIA also has a benefit (sells more GPUs to handle all that load)
        fauigerzigerk 3y ago
        Data owners are benefitting from having access to data while others don't. High processing cost is not a benefit to them at all. It's just a cost eating into their margins.
- anon291 3y ago
  As someone who's worked at several NVIDIA competitors, including Groq, I can guarantee you that, based on my knowledge, of existing products, they would be able to make much more money should they have lower memory footprint models. Given the amount of VC capital deployed for this (on the order of 100s of millions), I don't believe this is a reasonable take. Sure, NVIDIA et al may not want that (although, again I don't see why... they too can't produce chips fast enough so being able to provide models for customers now ought to be good), but there's so much money out there that does...

by p1esk 3y ago

behnamoh 3y ago

I guess the argument is that most AI research is supported by the big tech, and they have heavily invested in the deep learning approach. If the fundings were funneled to research groups working on alternative approaches, maybe we'd see the same amount of progress in AI only using another approach.

kettleballroll 3y ago
As a member of the research community: that's nonsense. Like already pointed out: academic groups (who by no means are dependent on big tech) would jump all over that. Mamba has been out long enough that you'd already see tons of papers at arxiv showing mamba dominating transformers in all sorts of applications. But that's not happening, despite the ton of hype. That doesn't mean that mamba is nonsense. Just that it isn't the immediate transformer killer. It remains to be seen if something comes from it, eventually.
- godelski 3y ago
  As a member of the research community: that's nonsense. Publishing is an extremely noisy process in ML and is getting increasingly difficult for smaller non big tech collaborating labs. Reviewers' go to are: more datasets, scale, not novel. The easiest way to approach this is to work off of pretrained models. This is probably more obvious in the NLP world. I agree that Mamba doesn't solve everything and it still needs work. But I disagree with the logic that there isn't an issue of railroading.
  - p1esk 3y ago
    What’s the main difference between an ape’s brain and a human brain? Scale. So that’s the train we’re riding at the moment. No roadblocks yet, aside from cost.
    - godelski 3y ago
      > What’s the main difference between an ape’s brain and a human brain? Scale. This is incredibly naive with absolutely no scientific basis. There is no evidence that this is in scale of data nor scale of architecture. There are a number of animals with larger brains in terms of both mass and total number of neurons. An African Elephant has roughly 3x the number of neurons humans have. Dolphins beat humans in total surface area. Neanderthals are estimated to have had larger brains too! It isn't mass, neurons, neuron density, surface area. We aren't just scaled up chimps.
      - p1esk 3y ago
        Other animals with larger brains might have other bottlenecks preventing them from reaching full potential of their intelligence. Neanderthals might have been smarter than us, but went extinct for reasons not related to intelligence. But my point stands - our brains have evolved directly from apes brains and the main difference between them and us is brain size.
        godelski 3y ago
        > Other animals with larger brains might have other bottlenecks >>> What’s the main difference between an ape’s brain and a human brain? Scale. Your argument is inconsistent. Very clearly everything isn't scale or we'd use other things besides transformers. Different architectures scale in different ways and everything has different inductive biases. No one doubts scale is important, but there's a lot more.
        p1esk 3y ago
        Scale is all we need for transformers (so far). It might also be all we need for ape brains. It’s not all we need for whatever elephant or dolphin brains evolved from. When this stops being the case for transformers, we will need something else. I’m just pointing out it’s not the case yet.
- algo_trader 3y ago
  > Just that it isn't the immediate transformer killer. What is the best/stable-ish linear alternative for transformer right now? Especially for text generation and summarization. We have domain specific ways of over sampling and search, so we much prefer less expensive models.
landryraccoon 3y ago
Why would Meta, Microsoft, Amazon and Google want Nvidia to remain dominant in hardware? Are you treating “big tech” like they all have one hive mind?
- behnamoh 3y ago
  For MSFT, AMZN, GOOG, the competitive advantage comes from having huge datasets (that Nvidia doesn't have). It's a symbiosis that benefits the data-rich and GPU-rich players.
  - pixl97 3y ago
    This still makes little sense as that scale will always matter. If you can drop the compute cost of a model by 10x it means you can increase model integrity/intelligence/speed etc beyond what your compute bound competitors have. Simply put, for the time being huge datasets are going to be needed and those with bigger (cleaner?) datasets will have a better behaving model.
  - fauigerzigerk 3y ago
    Where is the symbiosis? If data is the differentiator, how do the data owners benefit from Nvidia eating into their margins?
    - coldtea 3y ago
      It's sumbiosis based on a common factor they both appreciate: - the data/processing having to be large means the data-owners have a benefit - the data/processing having to be large means NVIDIA also has a benefit (sells more GPUs to handle all that load)
      - fauigerzigerk 3y ago
        Data owners are benefitting from having access to data while others don't. High processing cost is not a benefit to them at all. It's just a cost eating into their margins.
anon291 3y ago
As someone who's worked at several NVIDIA competitors, including Groq, I can guarantee you that, based on my knowledge, of existing products, they would be able to make much more money should they have lower memory footprint models. Given the amount of VC capital deployed for this (on the order of 100s of millions), I don't believe this is a reasonable take. Sure, NVIDIA et al may not want that (although, again I don't see why... they too can't produce chips fast enough so being able to provide models for customers now ought to be good), but there's so much money out there that does...