4 ms·

This is both a fascinating read of an obviously bright and clever fellow and also a TERRIBLE way to approach building intuition around the emergent behaviors of

by tacos 10y ago

This is both a fascinating read of an obviously bright and clever fellow and also a TERRIBLE way to approach building intuition around the emergent behaviors of neural networks. If you want to learn about them without repeating the terrible mistakes this guy is about to make if he continues down this path, pick up a book.

CardenB 10y ago
What sort of terrible mistakes are you referring to?
- tacos 10y ago
  A fundamental misunderstanding of the space, focusing on the wrong things, thinking he can visualize (or should visualize) the mechanics, hyperfocus on topologies, misunderstanding of key distance functions, sigmoids, training/overtraining, add buzzword here. Right now he's a chef who obsessively spent six months thinking about knives. But it's about the meat and the heat, not the chef knife. Just pick a problem and start playing with it! If you lean to this sort of pedantic "I MUST UNDERSTAND ALL FUNDAMENTALS" (I do!) I still recommend Matlab (available for $49-$99 in student editions) because the tools are so good and the docs are so damn readable. This post is basically a matlab doc page minus the links to the stuff that actually works.
  - eli_gottlieb 10y ago
    >Just pick a problem and start playing with it! If you lean to this sort of pedantic "I MUST UNDERSTAND ALL FUNDAMENTALS" (I do!) I still recommend Matlab (available for $49-$99 in student editions) because the tools are so good and the docs are so damn readable. Don't pay to torture yourself! Use Python instead! Numpy and scipy are the standard libraries for most scientific code nowadays, and you'll be able to use Theano, Caffe, TensorFlow, PyMC, scikit-learn, pybrain, and loads of other machine-learning and statistics libraries with Python interfaces!
    - tacos 10y ago
      +1 for python however good luck building fundamentals given the sea of chaotic products. The whole point of the Matlab recommendation is that, like Amazon S3, they haven't really touched the damn NN toolkit in a decade. While y'all were tinkering with MVC apps in 2007 some of us were researching and publishing and laying the foundation for the nine different open source learning libraries that can't agree on terminology or workflow patterns or which cloud provider they are clandestinely pushing you toward. That said, I deploy using Python and several of these libraries (and more interesting unmentioned ones). But often, Matlab for exploratory dev! I re-recommend it for getting started.
      - thatcat 10y ago
        >more interesting unmentioned ones care to share more info?
        tacos 10y ago
        NN is flavor of the month at HN but these core toolkits don't really do much. They're all running the same Fortran code and MIT FFT libraries Matlab's been using for 20+ years. It's the base that the cool experiments happen on top of. Once you dig in and try to actually do something, you stumble on the cool stuff. It's often Python throwaway or Matlab spaghetti code in a .edu directory that starts with a tilde. And it disappears when you need it most. (If it's Python, it's guaranteed to be something wonky that uses 2.7 if you're on 3.x or vice versa, or requires three additional packages that throw obscure build errors on your distro. And if it's Matlab, it inevitably requires you to shell out another $299 for some stupid toolbox just to generate white noise...) Good times. I won't kill your fun by being overly specific. Dig in!
      - eli_gottlieb 10y ago
        >While y'all were tinkering with MVC apps in 2007 some of us were researching and publishing and laying the foundation for the nine different open source learning libraries I was a kid in 2007, and I'd prefer that you didn't insinuate I developed MVC apps! Right now I'm trying to get transfer-learning to show up correctly in a Hakaru model translated from a Church model so I can do some extra information-theoretic measurements.
  - deepnet 10y ago
    Octave is a free (GPL) MATLAB alternative. Andrew Ng's introductory Machine Learning Coursera uses Octave to teach programming a simple Neural Net that recognises handwritten digits (MNIST).
    - tacos 10y ago
      The course also uses Matlab. Perhaps we can avoid the Octave vs Matlab debacle by linking to this previous discussion. https://news.ycombinator.com/item?id=10232595 https://news.ycombinator.com/item?id=10232595 Consensus (I'll use the term loosely) was that you get what you pay for. Either spend the bucks on Matlab or preferably use Python. I'll dare to provide a stronger opinion: Octave just sucks and is one of those unfortunate GNU tools that forever seems stuck 15 years in the past. Many people moved to Python, we're all eyeing Julia, the rest of us are largely stuck in Matlab.
      - deepnet 10y ago
        I really liked Octave and used it for a lot of prototyping - it was certainly sufficient for my needs and the maintainers are helpful & friendly and were unofficial TA's during the 1st run of Professor Ng's Coursera course. Given the range of finances of a MOOC class of 160,000, Octave's price (free) was vital for some to participate. Matlab looks really cool, but I feel it unfair to advocate un-Free software if my students can't take it home. I never got a chance to play with Matlab so can't really comment. As you suggest I graduated from Octave to Python, sometimes Lua & Torch7 and then the subsets of C for CUDA & OPENCL's GPU parallelism. Javascript is so universal accessible a platform that I like to give demonstrations using Karpathy's ConvnetJS & ReinforceJS as everyone can explore it straight away using the browser console as REPL. Professor Ng's MOOC students managed a good fundraiser for Octave development so those that could, paid something.
        tacos 10y ago
        If exploring neural nets in a REPL is the spark that finally lights a fire under Octave, I'm all for it! But I think they already lost the battle to Python. > Matlab looks really cool, but I feel it unfair to advocate un-Free software if my students can't take it home. Why is it your job to advocate anything? Inform them of what's available. Challenge yourself to make your lessons applicable to a broader range of tools, including ones your students might craft themselves. Let them know they can get the job done with GIMP but there's Photoshop at $9.99/month that might save them time and make them more employable. $1999 MacBook, $649 phone, but somehow $99 for a critical piece of software is unacceptable. And people wonder why students have debt but can't find jobs.
  - venomsnake 10y ago
    >Right now he's a chef who obsessively spent six months thinking about knives. But it's about the meat and the heat, not the chef knife. I can only guess you never learned about sushi. Or any other cuisine in which high precision cuts are the foundation/hallmark of excellence. Good chefs are obsessed over their knives. With good reason.
    - tacos 10y ago
      Good chefs are obsessed with technique. This transcends cultures--from French to Japanese. Moreover, they actually prepare food. In contrast, enthusiasts spend six months obsessing about different types of knives. And then write a blog post detailing their thought process. Which, like this post, is often meaningless because it's a guy writing about knives who's never cut into anything.
      - tobz 10y ago
        Meaningless to who? Professional chefs who already know which knives they need and want to use? No shit, they're professional chefs and this guy is a just an enthusiastic. They wouldn't be reading this post saying "nice, I can finally figure out which knife I need!". Reading a few of your comments in this thread, it seems like you're heavily involved in this area of technology/research, but it's depressing that you're making comments that range from passive-aggressive to condescending to dismissive. Further, you've yet to actually offer constructive criticism about the post, or even detailed criticism, one way or another.
        tacos 10y ago
        I can't fix this guy... he's off in the weeds doing his own thing. However I can provide constructive input by providing broader perspectives and alternate viewpoints. This technically weak, mis-titled post sat at the top of HN for most of a day. It's now a top hit on Google when you search "Neural Net Compression" -- ranked above classic research papers from the 80s and 90s. That's the depressing part.

by tacos 10y ago

CardenB 10y ago

What sort of terrible mistakes are you referring to?

tacos 10y ago
A fundamental misunderstanding of the space, focusing on the wrong things, thinking he can visualize (or should visualize) the mechanics, hyperfocus on topologies, misunderstanding of key distance functions, sigmoids, training/overtraining, add buzzword here. Right now he's a chef who obsessively spent six months thinking about knives. But it's about the meat and the heat, not the chef knife. Just pick a problem and start playing with it! If you lean to this sort of pedantic "I MUST UNDERSTAND ALL FUNDAMENTALS" (I do!) I still recommend Matlab (available for $49-$99 in student editions) because the tools are so good and the docs are so damn readable. This post is basically a matlab doc page minus the links to the stuff that actually works.
- eli_gottlieb 10y ago
  >Just pick a problem and start playing with it! If you lean to this sort of pedantic "I MUST UNDERSTAND ALL FUNDAMENTALS" (I do!) I still recommend Matlab (available for $49-$99 in student editions) because the tools are so good and the docs are so damn readable. Don't pay to torture yourself! Use Python instead! Numpy and scipy are the standard libraries for most scientific code nowadays, and you'll be able to use Theano, Caffe, TensorFlow, PyMC, scikit-learn, pybrain, and loads of other machine-learning and statistics libraries with Python interfaces!
  - tacos 10y ago
    +1 for python however good luck building fundamentals given the sea of chaotic products. The whole point of the Matlab recommendation is that, like Amazon S3, they haven't really touched the damn NN toolkit in a decade. While y'all were tinkering with MVC apps in 2007 some of us were researching and publishing and laying the foundation for the nine different open source learning libraries that can't agree on terminology or workflow patterns or which cloud provider they are clandestinely pushing you toward. That said, I deploy using Python and several of these libraries (and more interesting unmentioned ones). But often, Matlab for exploratory dev! I re-recommend it for getting started.
    - thatcat 10y ago
      >more interesting unmentioned ones care to share more info?
      - tacos 10y ago
        NN is flavor of the month at HN but these core toolkits don't really do much. They're all running the same Fortran code and MIT FFT libraries Matlab's been using for 20+ years. It's the base that the cool experiments happen on top of. Once you dig in and try to actually do something, you stumble on the cool stuff. It's often Python throwaway or Matlab spaghetti code in a .edu directory that starts with a tilde. And it disappears when you need it most. (If it's Python, it's guaranteed to be something wonky that uses 2.7 if you're on 3.x or vice versa, or requires three additional packages that throw obscure build errors on your distro. And if it's Matlab, it inevitably requires you to shell out another $299 for some stupid toolbox just to generate white noise...) Good times. I won't kill your fun by being overly specific. Dig in!
    - eli_gottlieb 10y ago
      >While y'all were tinkering with MVC apps in 2007 some of us were researching and publishing and laying the foundation for the nine different open source learning libraries I was a kid in 2007, and I'd prefer that you didn't insinuate I developed MVC apps! Right now I'm trying to get transfer-learning to show up correctly in a Hakaru model translated from a Church model so I can do some extra information-theoretic measurements.
- deepnet 10y ago
  Octave is a free (GPL) MATLAB alternative. Andrew Ng's introductory Machine Learning Coursera uses Octave to teach programming a simple Neural Net that recognises handwritten digits (MNIST).
  - tacos 10y ago
    The course also uses Matlab. Perhaps we can avoid the Octave vs Matlab debacle by linking to this previous discussion. https://news.ycombinator.com/item?id=10232595 https://news.ycombinator.com/item?id=10232595 Consensus (I'll use the term loosely) was that you get what you pay for. Either spend the bucks on Matlab or preferably use Python. I'll dare to provide a stronger opinion: Octave just sucks and is one of those unfortunate GNU tools that forever seems stuck 15 years in the past. Many people moved to Python, we're all eyeing Julia, the rest of us are largely stuck in Matlab.
    - deepnet 10y ago
      I really liked Octave and used it for a lot of prototyping - it was certainly sufficient for my needs and the maintainers are helpful & friendly and were unofficial TA's during the 1st run of Professor Ng's Coursera course. Given the range of finances of a MOOC class of 160,000, Octave's price (free) was vital for some to participate. Matlab looks really cool, but I feel it unfair to advocate un-Free software if my students can't take it home. I never got a chance to play with Matlab so can't really comment. As you suggest I graduated from Octave to Python, sometimes Lua & Torch7 and then the subsets of C for CUDA & OPENCL's GPU parallelism. Javascript is so universal accessible a platform that I like to give demonstrations using Karpathy's ConvnetJS & ReinforceJS as everyone can explore it straight away using the browser console as REPL. Professor Ng's MOOC students managed a good fundraiser for Octave development so those that could, paid something.
      - tacos 10y ago
        If exploring neural nets in a REPL is the spark that finally lights a fire under Octave, I'm all for it! But I think they already lost the battle to Python. > Matlab looks really cool, but I feel it unfair to advocate un-Free software if my students can't take it home. Why is it your job to advocate anything? Inform them of what's available. Challenge yourself to make your lessons applicable to a broader range of tools, including ones your students might craft themselves. Let them know they can get the job done with GIMP but there's Photoshop at $9.99/month that might save them time and make them more employable. $1999 MacBook, $649 phone, but somehow $99 for a critical piece of software is unacceptable. And people wonder why students have debt but can't find jobs.
- venomsnake 10y ago
  >Right now he's a chef who obsessively spent six months thinking about knives. But it's about the meat and the heat, not the chef knife. I can only guess you never learned about sushi. Or any other cuisine in which high precision cuts are the foundation/hallmark of excellence. Good chefs are obsessed over their knives. With good reason.
  - tacos 10y ago
    Good chefs are obsessed with technique. This transcends cultures--from French to Japanese. Moreover, they actually prepare food. In contrast, enthusiasts spend six months obsessing about different types of knives. And then write a blog post detailing their thought process. Which, like this post, is often meaningless because it's a guy writing about knives who's never cut into anything.
    - tobz 10y ago
      Meaningless to who? Professional chefs who already know which knives they need and want to use? No shit, they're professional chefs and this guy is a just an enthusiastic. They wouldn't be reading this post saying "nice, I can finally figure out which knife I need!". Reading a few of your comments in this thread, it seems like you're heavily involved in this area of technology/research, but it's depressing that you're making comments that range from passive-aggressive to condescending to dismissive. Further, you've yet to actually offer constructive criticism about the post, or even detailed criticism, one way or another.
      - tacos 10y ago
        I can't fix this guy... he's off in the weeds doing his own thing. However I can provide constructive input by providing broader perspectives and alternate viewpoints. This technically weak, mis-titled post sat at the top of HN for most of a day. It's now a top hit on Google when you search "Neural Net Compression" -- ranked above classic research papers from the 80s and 90s. That's the depressing part.