4 ms·

Great work! Hacker News still seems to have a deeply skeptical culture with regard to machine learning - not sure why. There's always someone saying it's "not n

Great work! Hacker News still seems to have a deeply skeptical culture with regard to machine learning - not sure why. There's always someone saying it's "not novel" and it's "just doing x". Overfitting is a known issues in machine learning, people. If you still think all neural networks are doing is memorizing the dataset completely in the year 2021 - you might want to revisit the topic. It is one of the first concerns anyone training a deep model will have and to assume this model is overfit _without_ providing specific examples is arguing in bad faith. Sentdex has shown his GAN is able to generalize various game logic like collision/friction with vehicles and also learns aspects of rendering such as a proper reflection of the sun on the back of the car. He also showed weak points where the model is incapable of handling some situations and even did the impossible task of "splitting a car in two" to try and solve a head-on collision. Even though this is a failure case; it should at least provide you with some intuition that the GAN isn't just spitting out frames memorized from the dataset because that never happens in the dataset. You will need to apply a little more rigor before outright dismissing these weights as merely overfit. @sentdex Have you considered a guided diffusion approach now that that's all the rage? It's all rather new still but I believe it could be applied to these concepts as well.

rasz 5y ago
One of the main problems with ML/NN is it often works like magic, aka the trick works as long as audience doesnt know the secret behind it. Its fascinating to gullible audience, mundane bordering on boring to practitioners. My Tiger repelling rock^^^^^^leopard detection model works great on all animal pictures ... until you feed it a sofa https://web.archive.org/web/20150703094328/http://rocknrollnerd.github.io/ml/2015/05/27/leopard-sofa.html https://web.archive.org/web/20150703094328/http://rocknrolln... >able to generalize various game logic like collision/friction with vehicles and also learns aspects of rendering such as a proper reflection of the sun on the back of the car id did none of that, what this model did is learn all the frames of video and their chronological order according to the input. > impossible task of "splitting a car in two" to try and solve a head-on collision. it played back both learned versions at once, like reporting confidence of round thing being 50% ball and 50% orange.
- sentdex 5y ago
  In the end, everything is boiling down to matrix math, so you can always make the argument that no neural network is impressive if you want. The model's size is ~173MB, depending on settings. That's not much space to have memorized every single possible combination of events, nor was our data enough to cover that either.
  - fartcannon 5y ago
    Your original self driving GTA5 videos are what helped me come to understand machine learning in the first place (along with some of Seth Bling's MarI/O, and a bit of Tom7's learn/play-fun magic). I used your tech to make an AI that played Donkey Kong Country in LSNES emulator shortly before Gym-Retro was released. So, thanks a bunch, Sentdex. You are rad.
    - sentdex 5y ago
      Hah, awesome! Any plans to apply GAN Theft Auto to something else? :o
      - fartcannon 5y ago
        Not offhand, but you've probably inspired a lot of creativity with this across the internet... and a lot of copy cats. I'm looking forward to seeing what gets made.
  - teruakohatu 5y ago
    > The model's size is ~173MB That is impressive! Less than twice the size of ResNet-50 weights. Surely that is within an order of magnitude of an equivalent Unity or GoDot game+models.
  - haecceity 5y ago
    Video frame resolution is pretty small...
  - YeGoblynQueenne 5y ago
sentdex 5y ago
Heh, yeah, tough crowd I guess. The full code, models, and videos are all released and people are still skeptical. I feel like 95%+ of papers don't do anything besides tell you what happened and you're just supposed to believe them. Drives me nuts. Not sure why all the hate when you could just see for yourself. I'd welcome someone who can actually prove the model just "memorized" every combo possible and didn't do any generalization. I imagine the original GameGAN researchers from NVIDIA would be interested too. Interesting @ guided diffusion, not aware of its existence til now. We've had our heads down for a while. Will look into it, thanks!
- godelski 5y ago
  > I feel like 95%+ of papers don't do anything besides tell you what happened and you're just supposed to believe them. Honestly I think there's a big problem with page limits. My team recently had a pre-print that was well over 10 pages and we still didn't get everything and then when we submitted to NeurlIPS we had to reduce it to 9! This seems to be a common problem and why you should often check different versions on ArXiv. And we had more experiments and data we needed to convey since the pre-print. This problem is growing as we have to compare more things and tables can easily take up a single page. I think this causes an exaggeration of the problem that always exists of not explaining things in detail and expecting readers to be experts. Luckily most people share source code which helps show all the tricks authors used and blogging is becoming more common which further helps. > I'd welcome someone who can actually prove the model just "memorized" every combo possible Honestly this would be impressive in of itself.
  - danuker 5y ago
    There's the Hutter Prize [1] - memorizing is useful (and arguably intelligent) if it's compressed. http://prize.hutter1.net/ http://prize.hutter1.net/
    - justinjlynn 5y ago
      Indeed. Novel, efficient program synthesis is still novel, efficient program synthesis even if it's a novel, efficient data compression codec you're synthesising.
andrepd 5y ago
> Hacker News still seems to have a deeply skeptical culture with regard to machine learning Is... that a bad thing? Skepticism is good. When it's about something as hyped as "deep learning", even more so.
- dkarras 5y ago
  >Is... that a bad thing? Yes, when it is there for no valid reason, or ridiculous reasons. Skepticism is not a default position you can take like a toddler refusing to eat their vegetables. You need some informed (and non-fallacious) intelligent reasoning behind that. "I'm skeptic about this thing using X because X is so hyped these days" is not such reasoning.
  - akiselev 5y ago
    "I'm skeptical about this thing using X to do Y because the burden of proof is on people claiming X does Y and historically they have failed to meet that burden" I don't know what skepticism has to do with ridiculous toddlers - they are almost universally incapable of grasping the nuances of epistemology.
  - andrepd 5y ago
    Well, it kind of is. Blockchain has been hyped by charlatans as the cure to all world's ills. That means when you read something about blockchain you should be especially suspicious. Similarly, I've read too many people hyping up glorified chatbots as one step below AGI (see the :o reactions to GPT3), so I'm now extra skeptical about claims about machine learning.
- bastawhiz 5y ago
  Skepticism is good when it targets bold claims with vague proof. This is not a bold claim (it's a video demo showing the process) and its proof is not vague (you can inspect the source). Skepticism over something like GPT-2 without more than sample output is good. Skepticism over GPT-2 with a workable demo and source is unhelpful.
  - 5y ago
deleted 5y ago
[deleted]

rasz 5y ago

One of the main problems with ML/NN is it often works like magic, aka the trick works as long as audience doesnt know the secret behind it. Its fascinating to gullible audience, mundane bordering on boring to practitioners. My Tiger repelling rock^^^^^^leopard detection model works great on all animal pictures ... until you feed it a sofa https://web.archive.org/web/20150703094328/http://rocknrollnerd.github.io/ml/2015/05/27/leopard-sofa.html https://web.archive.org/web/20150703094328/http://rocknrolln... >able to generalize various game logic like collision/friction with vehicles and also learns aspects of rendering such as a proper reflection of the sun on the back of the car id did none of that, what this model did is learn all the frames of video and their chronological order according to the input. > impossible task of "splitting a car in two" to try and solve a head-on collision. it played back both learned versions at once, like reporting confidence of round thing being 50% ball and 50% orange.

sentdex 5y ago
In the end, everything is boiling down to matrix math, so you can always make the argument that no neural network is impressive if you want. The model's size is ~173MB, depending on settings. That's not much space to have memorized every single possible combination of events, nor was our data enough to cover that either.
- fartcannon 5y ago
  Your original self driving GTA5 videos are what helped me come to understand machine learning in the first place (along with some of Seth Bling's MarI/O, and a bit of Tom7's learn/play-fun magic). I used your tech to make an AI that played Donkey Kong Country in LSNES emulator shortly before Gym-Retro was released. So, thanks a bunch, Sentdex. You are rad.
  - sentdex 5y ago
    Hah, awesome! Any plans to apply GAN Theft Auto to something else? :o
    - fartcannon 5y ago
      Not offhand, but you've probably inspired a lot of creativity with this across the internet... and a lot of copy cats. I'm looking forward to seeing what gets made.
- teruakohatu 5y ago
  > The model's size is ~173MB That is impressive! Less than twice the size of ResNet-50 weights. Surely that is within an order of magnitude of an equivalent Unity or GoDot game+models.
- haecceity 5y ago
  Video frame resolution is pretty small...
- YeGoblynQueenne 5y ago

sentdex 5y ago

Heh, yeah, tough crowd I guess. The full code, models, and videos are all released and people are still skeptical. I feel like 95%+ of papers don't do anything besides tell you what happened and you're just supposed to believe them. Drives me nuts. Not sure why all the hate when you could just see for yourself. I'd welcome someone who can actually prove the model just "memorized" every combo possible and didn't do any generalization. I imagine the original GameGAN researchers from NVIDIA would be interested too. Interesting @ guided diffusion, not aware of its existence til now. We've had our heads down for a while. Will look into it, thanks!

godelski 5y ago
> I feel like 95%+ of papers don't do anything besides tell you what happened and you're just supposed to believe them. Honestly I think there's a big problem with page limits. My team recently had a pre-print that was well over 10 pages and we still didn't get everything and then when we submitted to NeurlIPS we had to reduce it to 9! This seems to be a common problem and why you should often check different versions on ArXiv. And we had more experiments and data we needed to convey since the pre-print. This problem is growing as we have to compare more things and tables can easily take up a single page. I think this causes an exaggeration of the problem that always exists of not explaining things in detail and expecting readers to be experts. Luckily most people share source code which helps show all the tricks authors used and blogging is becoming more common which further helps. > I'd welcome someone who can actually prove the model just "memorized" every combo possible Honestly this would be impressive in of itself.
- danuker 5y ago
  There's the Hutter Prize [1] - memorizing is useful (and arguably intelligent) if it's compressed. http://prize.hutter1.net/ http://prize.hutter1.net/
  - justinjlynn 5y ago
    Indeed. Novel, efficient program synthesis is still novel, efficient program synthesis even if it's a novel, efficient data compression codec you're synthesising.

andrepd 5y ago

> Hacker News still seems to have a deeply skeptical culture with regard to machine learning Is... that a bad thing? Skepticism is good. When it's about something as hyped as "deep learning", even more so.

dkarras 5y ago
>Is... that a bad thing? Yes, when it is there for no valid reason, or ridiculous reasons. Skepticism is not a default position you can take like a toddler refusing to eat their vegetables. You need some informed (and non-fallacious) intelligent reasoning behind that. "I'm skeptic about this thing using X because X is so hyped these days" is not such reasoning.
- akiselev 5y ago
  "I'm skeptical about this thing using X to do Y because the burden of proof is on people claiming X does Y and historically they have failed to meet that burden" I don't know what skepticism has to do with ridiculous toddlers - they are almost universally incapable of grasping the nuances of epistemology.
- andrepd 5y ago
  Well, it kind of is. Blockchain has been hyped by charlatans as the cure to all world's ills. That means when you read something about blockchain you should be especially suspicious. Similarly, I've read too many people hyping up glorified chatbots as one step below AGI (see the :o reactions to GPT3), so I'm now extra skeptical about claims about machine learning.
bastawhiz 5y ago
Skepticism is good when it targets bold claims with vague proof. This is not a bold claim (it's a video demo showing the process) and its proof is not vague (you can inspect the source). Skepticism over something like GPT-2 without more than sample output is good. Skepticism over GPT-2 with a workable demo and source is unhelpful.
- 5y ago

deleted 5y ago

[deleted]