11 ms·

How I Fail – Ian Goodfellow

153 points by harias 8y ago

nl 8y ago
Probably the failure I consider the biggest is that I spent most of my PhD trying to solve supervised learning for computer vision using unsupervised feature learning methods, and was caught totally off guard when Alex, Ilya, and Geoff won the ImageNet contest with purely supervised methods. I think that in general wasting time writing papers that turn out to be dead ends is the main way that I fail in my own eyes. Wow, what an amazing story. And worth pointing out that Goodfellows work on unsupervised feature learning is clearly seen in his invention of GANs. Edit, and: I think it’s hard to extract value from negative results in machine learning because it can be so hard to tell what caused the negative result. A negative result might point to something very fundamental wrong with an idea, but it might also just be the result of a very small software bug, the wrong idea of the hyperparameter values to try out, too small of a model, etc. So, so true. I read comments on HN on how important negative results are in ML and I look at all the things I’ve tried and someone else made work, and all I see is reporting negative results just discourages ppl from trying things. This is a great interview.
akhilcacharya 8y ago
Good interview. It really surprised me to see a Stanford undergrad fail at anything to be honest, much less admission to a similarly ranked school for a specific PhD concentration they are known for. I guess in this circumstance it actually was their loss.
- ubernostrum 8y ago
  It really surprised me to see a Stanford undergrad fail at anything to be honest Why? The impression I've gotten from a lot of time in industry is that the average Stanford student/graduate is, well, average.
  - akhilcacharya 8y ago
    Because Stanford is incredibly selective, and is one of the 4 premier CS institutions in the world? I dunno, as someone that went to a public school with a 50% accept rate it seems difficult to imagine “academic superhumans” struggle with anything, ever.
    - pritambaral 8y ago
      > it seems difficult to imagine “academic superhumans” struggle with anything, ever. I know a person who went to a comparable institution in my country, who was considered something of a prodigy by his peers (and me) both before being selected and after joining the institution, and who struggled a lot with his academic responsibilities. In fact, I know a few of them. One of them had to drop out of the institution after repeatedly trying to finish his academic programme, and after two years of extensions. People struggle with things, in general. Everybody has their struggles, everybody has their demons. Even "superhumans". What we should be aiming for is not the lack of struggles — for that is merely the absence of effort — but perseverance in the face of it and growth from going through it.
    - ubernostrum 8y ago
      I'm sure you believe that there's something special about Stanford. Many people believe that. What I'm saying is that, in my experience, the belief doesn't hold up. If it were possible to reliably identify "academic superhumans" and concentrate them in one place, we'd have absolutely indisputable evidence of it. The fact that we don't suggests that perhaps your beliefs about how Stanford's admission practices correlate with quality of graduates may be mistaken.
      - deleted 8y ago
        [deleted]
    - CuriouslyC 8y ago
      The best performing students I've ever met weren't ridiculously smart. They were slightly above average, but they excelled at finding out exactly what the teacher was going to test and drilling that ruthlessly. They also worked harder than most of their peers. They also know how to present their answers well, and I think teachers subconsciously give them more credit when they're not exactly right. As a funny anecdote, once I let a top copy my homework because he had just come home from a family emergency. He actually got a higher score on that homework than I did, despite the content of our answers being the same.
- laurieg 8y ago
  I did my undergrad at Cambridge University and I was surprised at how normal people were. Of course, there were a couple of utter geniuses, but they were the exception. I think an undergrad friend of mine put it best: "Cambridge doesn't take the top 1%, it takes a certain type of person from the top 10%"
  - tonyedgecombe 8y ago
    When my son interviewed there one of the tutors told me he thought they should abandon the interviewing and select at random from a pool of the top 10%.
  - icc97 8y ago
    How many of your friends had anything less than 3 A's (A* since 2010) at A-level? There's 424,000 students got into university [0] and Cambridge enrols 3,480 out of 17,000 applicants [1]. So they don't take even 1% and only the top 4% of students even bother applying. Edit: I suspect that 3,480 includes overseas students which makes the the percentage of UK enrolments is even less. Further [1]: > In 2006, 5,228 students who were rejected went on to get 3 A levels or more at grade A Perhaps being surrounded by the top 1% makes you feel very normal/average, but there's certainly nothing normal about the Cambridge applicants and your figure of 10% definitely isn't correct. I got accepted at Imperial with predictions of ABC (+ no extra curricular) to do Maths and Computer Science there's no way Oxford or Cambridge would have looked at me without 4 A's + a significant amount of extra curricular activities. [0]: https://www.theguardian.com/education/live/2016/aug/18/a-level-results-day-2016-uk-students-get-their-grades-live https://www.theguardian.com/education/live/2016/aug/18/a-lev... [1]: https://en.wikipedia.org/wiki/University_of_Cambridge#Admissions https://en.wikipedia.org/wiki/University_of_Cambridge#Admiss...
    - laurieg 8y ago
      > So they don't take even 1% and only the top 4% of students even bother applying. I went to the college with the highest number of state school students. After participating in outreach events to get more people from a variety of backgrounds to apply to Cambridge I honestly think the biggest bottleneck is people from state schools not applying. They hear all the stories about strange interviews or being surrounding by posh people they have nothing in common with and just don't bother. I was the only person in my (rather large) school year to get into Cambridge, and only a couple of people applied. I was told all sorts of strange things about the application process which turned out not to be true. I imagine it must be far easier to apply if you go to a school where almost everyone does and the staff are well versed in what to get ready.
deleted 8y ago
[deleted]
danieldk 8y ago
That was a nice read, thanks! In general, I don't know how I feel about these articles, they seem to be trending among my Twitter friends. I know that these interviews are meant as an encouragement. To show that even the great people of our field had to struggle, so we should not give up when we are still struggling. However, these stories can also be understood negatively - if you didn't make it, you probably did not work or try hard enough. What makes this effect stronger is the selection bias - we do not see articles of the other 99% that work their tails of and do not become big in their fields. With a society that focuses on being the best, 'making it', etc. it leaves a lot of people feel inadequate, incompetent, or even depressed, even when they are big net contributors to society. E.g. think of the 50-year old professor a moderately successful academic career. He/she is educating hundreds of students and having a big positive impact on their lives. But they are seen by others and themselves as failed academics who were not good enough to make it. (I don't know what the solution is.)
- AndrewKemendo 8y ago
  Well there isn't a solution because the metrics for "failure" and "success" are fuzzy. In my mind, someone is a "success" if they accomplish the challenge of being the person they see themselves as/want to be. It's way harder than it sounds. So while there are certainly some people who, as Pickard said "made no mistakes but still lost," there are others who didn't put in sufficient effort and failed as a result. I'd imagine that there are more in the latter camp than in the former. In your example, the professor is only a failure if their goal was something other than moderately successful academic.
  - tonyedgecombe 8y ago
    I suspect the whole goal thing is a problem, setting some specific outcome you expect to reach when you have no idea what obstacles you might face on the way is setting yourself up for failure.
    - AndrewKemendo 8y ago
      Indeed! Goal setting is one of the most complex tasks we do. Doing it successfully, defined as a measurable outcome, requires that you have a pretty accurate assessment of both the landscape, your abilities and the path to achieve the goal - with some factor considered for deviation from the charted path. Depending on the scope of goal, the confidence interval on probability of success probably decreases exponentially with respect to time.
deleted 8y ago
[deleted]
yial 8y ago
I feel the story of his professor who responded “why do you have an ‘a’ in my class?” Is so incredibly relatable. It seems that sometimes; it’s far to easy to do in some “luck”.
degrews 8y ago
I'm getting a 404 Not Found. Anyone else?
- brainfog 8y ago
  From the top level site: > This account has been suspended. > Either the domain has been overused, or the reseller ran out of resources.
- faded242 8y ago
  ... That's how he fails.
- WordSkill 8y ago
  It has been down for a couple of hours. I mentioned that in a comment here and, for reasons I don't understand, was immediately downvoted. I deleted the comment in case there is some sort of HN 404 taboo I'm not aware of.
  - montyf 8y ago
    > I deleted the comment in case there is some sort of HN 404 taboo I'm not aware of. Never do this. If someone doesn't like your comment, then you are just letting them win by removing it. That's exactly what they want: for the comment not to exist. Don't give them the satisfaction; people who engage in censorship practices don't deserve it.
  - dragonwriter 8y ago
    > I mentioned that in a comment here and, for reasons I don't understand, was immediately downvoted. I deleted the comment in case there is some sort of HN 404 taboo I'm not aware of. It's good to reflect on whether a comment that you posted that is downvoted really is problematic in some way, and if you see a real problem with that reflection to delete or edit it, but you shouldn't delete a comment when you don't understand a downvote because, frankly, lots of good posts get stray early downvotes, and that usually gets corrected over time.
- veronikach 8y ago
ericliuche 8y ago
Getting a 404, the top level site has the error "This account has been suspended. Either the domain has been overused, or the reseller ran out of resources." Here's an archive link https://web.archive.org/web/20180505190115/http://www.veronikach.com/how-i-fail/how-i-fail-ian-goodfellow-phd14-computer-science/ https://web.archive.org/web/20180505190115/http://www.veroni...
- deleted 8y ago
  [deleted]
- consp 8y ago
  The 404 is caused by the provider expecting less get requests than GB's bought. The author (sitting next to me) actually bought more traffic but the company decided to just stop after a number of GET requests.
  - quasimodem 8y ago
    Oh I thought the 404 was the joke.
  - wyclif 8y ago
    The author clearly needs a new provider.
ivan_ah 8y ago
The Deep Learning Book by Goodfellow et al. is amazing. I found the writing to be very down-to-earth and it explains concepts without too much jargon. http://www.deeplearningbook.org/ http://www.deeplearningbook.org/
colmvp 8y ago
I can only assume his answer to question 11 is in response to the very public discourse between him and Jürgen Schmidhuber.
- amai 8y ago
  see also https://stats.stackexchange.com/questions/251460/were-generative-adversarial-networks-introduced-by-j%C3%BCrgen-schmidhuber https://stats.stackexchange.com/questions/251460/were-genera...
agentofoblivion 8y ago
I think the most likely cause of his success is not related to his talent, but his focus. When I was choosing a college, I wasn’t thinking about advisors. It certainly never occurred to me to find the best guys in the field and try to get in under them. It just sounds too far fetched to be a real possibility. But it is a real possibility, of course, and the people that identify that and go for it have a remarkable outcome of accumulating advantage. It’s like compound interest. Go to the best school you can. Get under a top advisor in grad school. Publish in top journals. Work for major companies. At this point you’re surrounded by the best and have become the best. You have to know what you want early, and then optimize your ass off. I was late to the party, but the last couple years have been huge successes for me relative to previous parts of my life, and that started with saying to myself “start by applying to the very best and working down from there.”

153 points by harias 8y ago

nl 8y ago

Probably the failure I consider the biggest is that I spent most of my PhD trying to solve supervised learning for computer vision using unsupervised feature learning methods, and was caught totally off guard when Alex, Ilya, and Geoff won the ImageNet contest with purely supervised methods. I think that in general wasting time writing papers that turn out to be dead ends is the main way that I fail in my own eyes. Wow, what an amazing story. And worth pointing out that Goodfellows work on unsupervised feature learning is clearly seen in his invention of GANs. Edit, and: I think it’s hard to extract value from negative results in machine learning because it can be so hard to tell what caused the negative result. A negative result might point to something very fundamental wrong with an idea, but it might also just be the result of a very small software bug, the wrong idea of the hyperparameter values to try out, too small of a model, etc. So, so true. I read comments on HN on how important negative results are in ML and I look at all the things I’ve tried and someone else made work, and all I see is reporting negative results just discourages ppl from trying things. This is a great interview.

akhilcacharya 8y ago

Good interview. It really surprised me to see a Stanford undergrad fail at anything to be honest, much less admission to a similarly ranked school for a specific PhD concentration they are known for. I guess in this circumstance it actually was their loss.

ubernostrum 8y ago
It really surprised me to see a Stanford undergrad fail at anything to be honest Why? The impression I've gotten from a lot of time in industry is that the average Stanford student/graduate is, well, average.
- akhilcacharya 8y ago
  Because Stanford is incredibly selective, and is one of the 4 premier CS institutions in the world? I dunno, as someone that went to a public school with a 50% accept rate it seems difficult to imagine “academic superhumans” struggle with anything, ever.
  - pritambaral 8y ago
    > it seems difficult to imagine “academic superhumans” struggle with anything, ever. I know a person who went to a comparable institution in my country, who was considered something of a prodigy by his peers (and me) both before being selected and after joining the institution, and who struggled a lot with his academic responsibilities. In fact, I know a few of them. One of them had to drop out of the institution after repeatedly trying to finish his academic programme, and after two years of extensions. People struggle with things, in general. Everybody has their struggles, everybody has their demons. Even "superhumans". What we should be aiming for is not the lack of struggles — for that is merely the absence of effort — but perseverance in the face of it and growth from going through it.
  - ubernostrum 8y ago
    I'm sure you believe that there's something special about Stanford. Many people believe that. What I'm saying is that, in my experience, the belief doesn't hold up. If it were possible to reliably identify "academic superhumans" and concentrate them in one place, we'd have absolutely indisputable evidence of it. The fact that we don't suggests that perhaps your beliefs about how Stanford's admission practices correlate with quality of graduates may be mistaken.
    - deleted 8y ago
      [deleted]
  - CuriouslyC 8y ago
    The best performing students I've ever met weren't ridiculously smart. They were slightly above average, but they excelled at finding out exactly what the teacher was going to test and drilling that ruthlessly. They also worked harder than most of their peers. They also know how to present their answers well, and I think teachers subconsciously give them more credit when they're not exactly right. As a funny anecdote, once I let a top copy my homework because he had just come home from a family emergency. He actually got a higher score on that homework than I did, despite the content of our answers being the same.
laurieg 8y ago
I did my undergrad at Cambridge University and I was surprised at how normal people were. Of course, there were a couple of utter geniuses, but they were the exception. I think an undergrad friend of mine put it best: "Cambridge doesn't take the top 1%, it takes a certain type of person from the top 10%"
- tonyedgecombe 8y ago
  When my son interviewed there one of the tutors told me he thought they should abandon the interviewing and select at random from a pool of the top 10%.
- icc97 8y ago
  How many of your friends had anything less than 3 A's (A* since 2010) at A-level? There's 424,000 students got into university [0] and Cambridge enrols 3,480 out of 17,000 applicants [1]. So they don't take even 1% and only the top 4% of students even bother applying. Edit: I suspect that 3,480 includes overseas students which makes the the percentage of UK enrolments is even less. Further [1]: > In 2006, 5,228 students who were rejected went on to get 3 A levels or more at grade A Perhaps being surrounded by the top 1% makes you feel very normal/average, but there's certainly nothing normal about the Cambridge applicants and your figure of 10% definitely isn't correct. I got accepted at Imperial with predictions of ABC (+ no extra curricular) to do Maths and Computer Science there's no way Oxford or Cambridge would have looked at me without 4 A's + a significant amount of extra curricular activities. [0]: https://www.theguardian.com/education/live/2016/aug/18/a-level-results-day-2016-uk-students-get-their-grades-live https://www.theguardian.com/education/live/2016/aug/18/a-lev... [1]: https://en.wikipedia.org/wiki/University_of_Cambridge#Admissions https://en.wikipedia.org/wiki/University_of_Cambridge#Admiss...
  - laurieg 8y ago
    > So they don't take even 1% and only the top 4% of students even bother applying. I went to the college with the highest number of state school students. After participating in outreach events to get more people from a variety of backgrounds to apply to Cambridge I honestly think the biggest bottleneck is people from state schools not applying. They hear all the stories about strange interviews or being surrounding by posh people they have nothing in common with and just don't bother. I was the only person in my (rather large) school year to get into Cambridge, and only a couple of people applied. I was told all sorts of strange things about the application process which turned out not to be true. I imagine it must be far easier to apply if you go to a school where almost everyone does and the staff are well versed in what to get ready.

deleted 8y ago

[deleted]

danieldk 8y ago

That was a nice read, thanks! In general, I don't know how I feel about these articles, they seem to be trending among my Twitter friends. I know that these interviews are meant as an encouragement. To show that even the great people of our field had to struggle, so we should not give up when we are still struggling. However, these stories can also be understood negatively - if you didn't make it, you probably did not work or try hard enough. What makes this effect stronger is the selection bias - we do not see articles of the other 99% that work their tails of and do not become big in their fields. With a society that focuses on being the best, 'making it', etc. it leaves a lot of people feel inadequate, incompetent, or even depressed, even when they are big net contributors to society. E.g. think of the 50-year old professor a moderately successful academic career. He/she is educating hundreds of students and having a big positive impact on their lives. But they are seen by others and themselves as failed academics who were not good enough to make it. (I don't know what the solution is.)

AndrewKemendo 8y ago
Well there isn't a solution because the metrics for "failure" and "success" are fuzzy. In my mind, someone is a "success" if they accomplish the challenge of being the person they see themselves as/want to be. It's way harder than it sounds. So while there are certainly some people who, as Pickard said "made no mistakes but still lost," there are others who didn't put in sufficient effort and failed as a result. I'd imagine that there are more in the latter camp than in the former. In your example, the professor is only a failure if their goal was something other than moderately successful academic.
- tonyedgecombe 8y ago
  I suspect the whole goal thing is a problem, setting some specific outcome you expect to reach when you have no idea what obstacles you might face on the way is setting yourself up for failure.
  - AndrewKemendo 8y ago
    Indeed! Goal setting is one of the most complex tasks we do. Doing it successfully, defined as a measurable outcome, requires that you have a pretty accurate assessment of both the landscape, your abilities and the path to achieve the goal - with some factor considered for deviation from the charted path. Depending on the scope of goal, the confidence interval on probability of success probably decreases exponentially with respect to time.

deleted 8y ago

[deleted]

yial 8y ago

I feel the story of his professor who responded “why do you have an ‘a’ in my class?” Is so incredibly relatable. It seems that sometimes; it’s far to easy to do in some “luck”.

degrews 8y ago

I'm getting a 404 Not Found. Anyone else?

brainfog 8y ago
From the top level site: > This account has been suspended. > Either the domain has been overused, or the reseller ran out of resources.
faded242 8y ago
... That's how he fails.
WordSkill 8y ago
It has been down for a couple of hours. I mentioned that in a comment here and, for reasons I don't understand, was immediately downvoted. I deleted the comment in case there is some sort of HN 404 taboo I'm not aware of.
- montyf 8y ago
  > I deleted the comment in case there is some sort of HN 404 taboo I'm not aware of. Never do this. If someone doesn't like your comment, then you are just letting them win by removing it. That's exactly what they want: for the comment not to exist. Don't give them the satisfaction; people who engage in censorship practices don't deserve it.
- dragonwriter 8y ago
  > I mentioned that in a comment here and, for reasons I don't understand, was immediately downvoted. I deleted the comment in case there is some sort of HN 404 taboo I'm not aware of. It's good to reflect on whether a comment that you posted that is downvoted really is problematic in some way, and if you see a real problem with that reflection to delete or edit it, but you shouldn't delete a comment when you don't understand a downvote because, frankly, lots of good posts get stray early downvotes, and that usually gets corrected over time.
veronikach 8y ago

ericliuche 8y ago

Getting a 404, the top level site has the error "This account has been suspended. Either the domain has been overused, or the reseller ran out of resources." Here's an archive link https://web.archive.org/web/20180505190115/http://www.veronikach.com/how-i-fail/how-i-fail-ian-goodfellow-phd14-computer-science/ https://web.archive.org/web/20180505190115/http://www.veroni...

deleted 8y ago
[deleted]
consp 8y ago
The 404 is caused by the provider expecting less get requests than GB's bought. The author (sitting next to me) actually bought more traffic but the company decided to just stop after a number of GET requests.
- quasimodem 8y ago
  Oh I thought the 404 was the joke.
- wyclif 8y ago
  The author clearly needs a new provider.

ivan_ah 8y ago

The Deep Learning Book by Goodfellow et al. is amazing. I found the writing to be very down-to-earth and it explains concepts without too much jargon. http://www.deeplearningbook.org/ http://www.deeplearningbook.org/

colmvp 8y ago

I can only assume his answer to question 11 is in response to the very public discourse between him and Jürgen Schmidhuber.

amai 8y ago
see also https://stats.stackexchange.com/questions/251460/were-generative-adversarial-networks-introduced-by-j%C3%BCrgen-schmidhuber https://stats.stackexchange.com/questions/251460/were-genera...

agentofoblivion 8y ago

I think the most likely cause of his success is not related to his talent, but his focus. When I was choosing a college, I wasn’t thinking about advisors. It certainly never occurred to me to find the best guys in the field and try to get in under them. It just sounds too far fetched to be a real possibility. But it is a real possibility, of course, and the people that identify that and go for it have a remarkable outcome of accumulating advantage. It’s like compound interest. Go to the best school you can. Get under a top advisor in grad school. Publish in top journals. Work for major companies. At this point you’re surrounded by the best and have become the best. You have to know what you want early, and then optimize your ass off. I was late to the party, but the last couple years have been huge successes for me relative to previous parts of my life, and that started with saying to myself “start by applying to the very best and working down from there.”