5 ms·

Prof Thrun mentioned in the last office hours that there were about 1650 people with "perfect scores" and he seemed a little thunderstruck by that and said he w

Prof Thrun mentioned in the last office hours that there were about 1650 people with "perfect scores" and he seemed a little thunderstruck by that and said he would admit these folks into Stanford if he could, because they are (paraphrasing, from memory)"Stanford quality". What struck me was how much importance he gave to this metric, which isn't that hard to game on an online offering. I know a few people (who shall remain nameless) who collaborate and check each others answers and so on before submission, in direct violation of the Stanford policy (and have 100s or close to it), and so probably have received this mail, whereas more "deserving" (note quotes) people who honestly work through the course material may not because they have, say, a 85% or 90% score. That said, my key takeaway from this is that professors are very impressed by perfect scores irrespective of how you got them. There must be something magical about that row of 100s. Once you set up a grading/ranking system, it is psychologically very hard not to admire people who end up at the top. I am personally a little dubious that the people with the highest scores would make the best pool of employees, especially given that this is an online course without the programming component, but what do I know? I wrote Java code for most of the algorithms in AIMA as a side project a few years ago [1], and after I read an online post by Peter Norvig saying a few of his students had tried and failed a few times (to implement the code in Java- Common Lisp code existed and the Python version was in its infancy), I sent him the code and this became the "official" java distribution for AIMA ( though I don't maintain it anymore- the immensely talented Ciaran O'Reilly of the Stanford Research Institute does) and no one ever invited me to Stanford or offered me a cool AI job[2], sob! :-P. No I am not bitter I tell you, not even the teeniest bit :-p [3] I wonder how this signalling will play into the upcoming courses? If there are tangential real world benefits to be gained by attempting a "perfect" score, then you can expect a lot more game playing wrt scores and exams. [1] More about how Peter Norvig shredded my initial code etc here http://news.ycombinator.com/item?id=2405277 http://news.ycombinator.com/item?id=2405277 [2] though eventually, after a lot more work, it did lead to my working on good ML/robotics etc projects from Bangalore, which is a hard thing to do in the Great Outsourcing Wasteland. [3] I am really not bitter. I wrote the code for the hell of it, not to get a job. AIMA was my introduction to the fascinating field of AI. It is a great, great book and it has a lot more material than is covered in the course. I once did want to go to Stanford and learn from the great profs there, but now in a "mountain comes to Mohammed" fashion, Stanford is coming to me. I don't care about the credentialling - I just want to learn. I took the AI online course and enjoyed Peter's and Sebastian's teaching immensely. Fwiw I should have a high 90's score, (I didn't add it all up) but nowhere near a perfect score.

danielharan 15y ago
Especially considering how many of those questions ask you to guess. These people either already knew the material, cheated or were really effing lucky. The work you did strikes me as a far better measure of character and subject understanding.
jbermudes 15y ago
While I agree with your general sentiment, note that at least in ML-class, you can resubmit quizzes as many times as you want for full credit. Thus over the course of an hour you can retake the quiz enough times to brute force a perfect score without breaking the letter (and arguably the spirit) of the honor code. Another interesting aspect of this "online ethics" is that there's no technical measures preventing you from pulling up past quiz results, and even if there was, you could still keep previous sessions open in a browser and even if there were countermeasures against that (javascript erasing), JS could be disabled, etc. The Programming Projects that ML class had were slightly better metric of performance as there's more work that would have to be plagiarized, and if you're just going to go through life outsourcing all of your work then I guess that's your prerogative. However, I think that if you wanted to be very serious about actually testing for knowledge of material then the addition of some sort of interview component (phone/skype session), while time-consuming, could help.
- cshesse 15y ago
  In a way, yes, the programming projects in ML seemed like a better measure of performance in that you have to actually figure something out. However, they have two (sort of) disadvantages vs normal homework: 1) you immediately know if you got it right or wrong when you submit, so you can to a lesser extent brute force the correct answer 2) with the exception of maybe the first assignment, they are all "fill in the blank" sort of programming assignments. You basically just have to find the equations they give you in the PDF, translate them directly to Octave, and bam you're done.
Technophilis 15y ago
It is surprising that you thought of this as a "cool AI job offer". I have two remarks here. First, the email sent to the class is barely an invitation to send resumes. Something many programmers/CS Students with online presence experience on a regular basis. Probably not from a Stanford Professor but at least from major companies recruiters. It would be interesting to know how many will actually make it through the screening, phone/on-site interviews and get a job offer. Second, I registered for the Machine Learning course (I am not sure if the same applies to the AI course) and I compared it with the actual ML course at Stanford (CS229) (I mainly looked at Youtube videos of Andrew[1] as well as Assignments/Midterm[2]). The latter is by far more advanced and theoretical. The assignments tend to test more than basic comprehension of the material presented in the lectures, which is exactly what the online course reviews tend to evaluate. They require strong mathematical knowledge and obviously a minimum level of creativity/intelligence. [1] http://www.youtube.com/results?search_query=machine+learning+stanford http://www.youtube.com/results?search_query=machine+learning... [2] http://cs229.stanford.edu/ http://cs229.stanford.edu/

danielharan 15y ago

Especially considering how many of those questions ask you to guess. These people either already knew the material, cheated or were really effing lucky. The work you did strikes me as a far better measure of character and subject understanding.

jbermudes 15y ago

While I agree with your general sentiment, note that at least in ML-class, you can resubmit quizzes as many times as you want for full credit. Thus over the course of an hour you can retake the quiz enough times to brute force a perfect score without breaking the letter (and arguably the spirit) of the honor code. Another interesting aspect of this "online ethics" is that there's no technical measures preventing you from pulling up past quiz results, and even if there was, you could still keep previous sessions open in a browser and even if there were countermeasures against that (javascript erasing), JS could be disabled, etc. The Programming Projects that ML class had were slightly better metric of performance as there's more work that would have to be plagiarized, and if you're just going to go through life outsourcing all of your work then I guess that's your prerogative. However, I think that if you wanted to be very serious about actually testing for knowledge of material then the addition of some sort of interview component (phone/skype session), while time-consuming, could help.

cshesse 15y ago
In a way, yes, the programming projects in ML seemed like a better measure of performance in that you have to actually figure something out. However, they have two (sort of) disadvantages vs normal homework: 1) you immediately know if you got it right or wrong when you submit, so you can to a lesser extent brute force the correct answer 2) with the exception of maybe the first assignment, they are all "fill in the blank" sort of programming assignments. You basically just have to find the equations they give you in the PDF, translate them directly to Octave, and bam you're done.

Technophilis 15y ago

It is surprising that you thought of this as a "cool AI job offer". I have two remarks here. First, the email sent to the class is barely an invitation to send resumes. Something many programmers/CS Students with online presence experience on a regular basis. Probably not from a Stanford Professor but at least from major companies recruiters. It would be interesting to know how many will actually make it through the screening, phone/on-site interviews and get a job offer. Second, I registered for the Machine Learning course (I am not sure if the same applies to the AI course) and I compared it with the actual ML course at Stanford (CS229) (I mainly looked at Youtube videos of Andrew[1] as well as Assignments/Midterm[2]). The latter is by far more advanced and theoretical. The assignments tend to test more than basic comprehension of the material presented in the lectures, which is exactly what the online course reviews tend to evaluate. They require strong mathematical knowledge and obviously a minimum level of creativity/intelligence. [1] http://www.youtube.com/results?search_query=machine+learning+stanford http://www.youtube.com/results?search_query=machine+learning... [2] http://cs229.stanford.edu/ http://cs229.stanford.edu/