4 ms·

Completely agreed with pron's comment. Also, the process by which they designed the new architecture is adhoc. They saw a problem, and then tried a bunch of sol

Completely agreed with pron's comment. Also, the process by which they designed the new architecture is adhoc. They saw a problem, and then tried a bunch of solutions, and got something to work. Sure, their effort was guided by intuition, but that is adhoc. My point is, there was no theory they worked out saying "you should do this," or "these are the non-empirical reasons this method is superior." I'm not negative on neural nets. I actually like that the field is relatively shallow and fast-moving. I just dislike it when people on HN valorize neural nets to be more than they are.

zodiac 10y ago
You can't just say that things are either adhoc or not adhoc - it's a spectrum. On one end we will have engineers coming up with architectures by consulting a RNG and then testing out those architectures. On the other end we will have a theory, the theory will make 100% good predictions about which designs will perform well, and we can just send send those designs to a factory to fabricate into asics, no testing needed. The adhoc-ness of how we design neural net architectures, CPUs, GPUs, high-temperature suerconductors, etc all fall on this spectrum, just at different places. pron's original comment made it sound like designing NNs were on the extreme ad-hoc end of the spectrum, I was just refuting that.
- argonaut 10y ago
  pron makes no claim that neural nets are on the extreme end of ad hoc.
  - zodiac 10y ago
    > like somebody tried lots of things and one seemed to work
    - argonaut 10y ago
      Which is a perfectly accurate description of many architectural improvements in deep learning (e.g. residual networks).
      - zodiac 10y ago
        Yeah, but it sounds like the extreme end of ad-hoc.