3 ms·

Step 1 is kind of ‘rest of the owl.’ It’s about as vague as ‘write some code.’

jstx1 5y ago
What I meant more specifically is that there's a limited number of operations that go into a neural network and the justification for the best architectures is that they have the best performance. You can see it in this paper too - there isn't any motivating theory about how to come up with something like this; the entire paper is "we tried some things, here's what worked and what didn't". (This is just an observation, I'm not criticising the authors at all)
- 6gvONxR4sf7o 5y ago
  > I'm not criticising the authors at all Maybe I'm being oversensitive about my field, but it comes across as dismissive of the field itself. Especially with "how do these people stay motivated?" > there's a limited number of operations that go into a neural network and the justification for the best architectures is that they have the best performance. There are a limited number of ways to put ink on a page, but the 'rest of the owl' meme is all about how there's a lot more to it than that. And the justification of best performance is because it's an unsolved problem that is being incrementally solved. Over the last decade you can see tons of tasks go from 20% accuracy being an achievement to 24% to 30% to 31% to 32% to 38%... eventually to 90%. The motivation is that these little changes on a 'does it do the job' metric add up. It's very motivating to see things slowly move from impossible for computers to pip installable.
  - jstx1 5y ago
    > There are a limited number of ways to put ink on a page, but the 'rest of the owl' meme is all about how there's a lot more to it than that. I'm not convinced. If you want to stick to the analogy, ML research looks like being able to draw a bunch of different animal parts and trying out different configurations of them; then once the drawing is done, you check whether you got close to an owl. That's what I meant when I said that there's little underlying theory to motivate the decisions that are being made (apart from the resulting performance of course).

What I meant more specifically is that there's a limited number of operations that go into a neural network and the justification for the best architectures is that they have the best performance. You can see it in this paper too - there isn't any motivating theory about how to come up with something like this; the entire paper is "we tried some things, here's what worked and what didn't". (This is just an observation, I'm not criticising the authors at all)

6gvONxR4sf7o 5y ago
> I'm not criticising the authors at all Maybe I'm being oversensitive about my field, but it comes across as dismissive of the field itself. Especially with "how do these people stay motivated?" > there's a limited number of operations that go into a neural network and the justification for the best architectures is that they have the best performance. There are a limited number of ways to put ink on a page, but the 'rest of the owl' meme is all about how there's a lot more to it than that. And the justification of best performance is because it's an unsolved problem that is being incrementally solved. Over the last decade you can see tons of tasks go from 20% accuracy being an achievement to 24% to 30% to 31% to 32% to 38%... eventually to 90%. The motivation is that these little changes on a 'does it do the job' metric add up. It's very motivating to see things slowly move from impossible for computers to pip installable.
- jstx1 5y ago
  > There are a limited number of ways to put ink on a page, but the 'rest of the owl' meme is all about how there's a lot more to it than that. I'm not convinced. If you want to stick to the analogy, ML research looks like being able to draw a bunch of different animal parts and trying out different configurations of them; then once the drawing is done, you check whether you got close to an owl. That's what I meant when I said that there's little underlying theory to motivate the decisions that are being made (apart from the resulting performance of course).