4 ms·

> Where does it originate from, I wonder? I suspect someone said something like that on social media, their post went viral, and now all of the internet is reve

by saiojd 4y ago

> Where does it originate from, I wonder? I suspect someone said something like that on social media, their post went viral, and now all of the internet is reverberating with this thing. It's a meme, yes? I don't have social media outside of HN. It comes from a few observations: 1) Large models are still improving with increased parameter counts (we do not know where the ceiling is yet; it could be low but it could also be high). 2) Most current architectures train by using all model parameters to produce an output, which is vastly inefficient. While it is not clear how to improve on this in the general case yet, in the simpler problem of NERFs, sidestepping this issue has led to a ~100x improvement in training time. 3) https://mingukkang.github.io/GigaGAN/ https://mingukkang.github.io/GigaGAN/ very recently increased the parameter count of StyleGAN by selecting parameters dynamically at runtime. They improved on previous results by a very, very large margin, at somewhat comparable training times. I stand by my claim: "neural network architectures are still in their infancy, it is very likely that more efficient approaches exist". I am not claiming that AI will become sentient or anything crazy and do not understand why you are associating my point of view with other people. I just said that it is likely that a novel technology will continue to improve (has this it ever NOT been the case for any new technology?).

YeGoblynQueenne 4y ago
>> It comes from a few observations: Well, if you want to know whether neural nets are in their "infancy" you shouldn't make "observations", you should read the literature. It goes back many years. Go to the primary sources, why try to guess and risk guessing wrong, as here? >> I stand by my claim: "neural network architectures are still in their infancy, it is very likely that more efficient approaches exist". Half of your "claim" is incorrect. Don't just double down on it! There's really nothing to "claim" here, the "infancy" or not of neural nets is not a matter of claiming or guessing. Either you know what it is, or you don't.
- saiojd 4y ago
  Your tone comes off as very adversarial/angry. Intentional or not? The claim is about the likelihood of more efficient approaches existing, not about the subjective qualifier preceding it (misunderstanding?). Yes, NNs have existed for a while now, but they will also exist for a long time forward in the future (hence the perhaps poor choice of word: infancy). "if you want to know whether neural nets are in their 'infancy' you shouldn't make 'observations'" -> I think you have understood this as "neural networks did not exist before" when what I meant is "neural networks will still change a lot in the future" I was interested in discussing, specifically: will NNs continue scaling up in size in the near/long term. In answer to the statement "the training costs will exceed all the compute capacity in existence", I added "it is very likely that more efficient approaches exist.". You replied that NNs have existed for a long time, and that because of this, it is a complete fantasy that more efficient approaches exist. Do you feel like this is a fair assessment or no?
  - YeGoblynQueenne 4y ago
    My problem is that I read what people write on HN and treat it with the same seriousness and respect I want people to treat my comments, when the majority are only saying whatever comes to their mind just to make some sort of impression. Then when I point out some obvious error, people freak out and get defensive because they never expected anyone to take them seriously, they're just spouting off whatever without really thinking. And then they try to wiggle out of the conversation, just like you're doing right now, pretending that you were misunderstood. Well, my mistake then for taking you seriously. Many apologies. You can rest assured it won't happen again.
    - saiojd 4y ago
      I feel very sad reading this. You are absolutely correct that I view HN as watercooler where I make hyperbolic statements, but incorrect when you say that that I did not take you seriously. I was frustrated because I felt like you were latching on to a figure of speech just to be argumentative instead of trying to understand what I meant (I was not trying to say that neural networks are literally new, I was trying to say that we have no proof that the parameter to flop ratio is even close to optimal because architectures are still changing a lot). You are right that neural networks are not in their infancy. Best wishes.

by saiojd 4y ago

YeGoblynQueenne 4y ago

>> It comes from a few observations: Well, if you want to know whether neural nets are in their "infancy" you shouldn't make "observations", you should read the literature. It goes back many years. Go to the primary sources, why try to guess and risk guessing wrong, as here? >> I stand by my claim: "neural network architectures are still in their infancy, it is very likely that more efficient approaches exist". Half of your "claim" is incorrect. Don't just double down on it! There's really nothing to "claim" here, the "infancy" or not of neural nets is not a matter of claiming or guessing. Either you know what it is, or you don't.

saiojd 4y ago
Your tone comes off as very adversarial/angry. Intentional or not? The claim is about the likelihood of more efficient approaches existing, not about the subjective qualifier preceding it (misunderstanding?). Yes, NNs have existed for a while now, but they will also exist for a long time forward in the future (hence the perhaps poor choice of word: infancy). "if you want to know whether neural nets are in their 'infancy' you shouldn't make 'observations'" -> I think you have understood this as "neural networks did not exist before" when what I meant is "neural networks will still change a lot in the future" I was interested in discussing, specifically: will NNs continue scaling up in size in the near/long term. In answer to the statement "the training costs will exceed all the compute capacity in existence", I added "it is very likely that more efficient approaches exist.". You replied that NNs have existed for a long time, and that because of this, it is a complete fantasy that more efficient approaches exist. Do you feel like this is a fair assessment or no?
- YeGoblynQueenne 4y ago
  My problem is that I read what people write on HN and treat it with the same seriousness and respect I want people to treat my comments, when the majority are only saying whatever comes to their mind just to make some sort of impression. Then when I point out some obvious error, people freak out and get defensive because they never expected anyone to take them seriously, they're just spouting off whatever without really thinking. And then they try to wiggle out of the conversation, just like you're doing right now, pretending that you were misunderstood. Well, my mistake then for taking you seriously. Many apologies. You can rest assured it won't happen again.
  - saiojd 4y ago
    I feel very sad reading this. You are absolutely correct that I view HN as watercooler where I make hyperbolic statements, but incorrect when you say that that I did not take you seriously. I was frustrated because I felt like you were latching on to a figure of speech just to be argumentative instead of trying to understand what I meant (I was not trying to say that neural networks are literally new, I was trying to say that we have no proof that the parameter to flop ratio is even close to optimal because architectures are still changing a lot). You are right that neural networks are not in their infancy. Best wishes.