5 ms·

I spent a few years at Caltech in an academic role (not as a student). I never understood the rationale behind the unsupervised take-home exams. Could you elabo

I spent a few years at Caltech in an academic role (not as a student). I never understood the rationale behind the unsupervised take-home exams. Could you elaborate what is it that makes it a 'success', in your opinion? The shortcomings that seem obvious to me are: - Penalizes honesty - Implied notion that Caltech students are "honourable" and honest. How is this achieved in practice?

beisner 6y ago
The idea is that you create questions that are ungoogleable. Most of my upper-level CS courses had take-homes that were open-book, open-internet, and the professor basically said “good luck finding the answer to this problem online; you will not”. This is harder for professors to do, and maybe fails in intro courses, but generally is quite effective.
- 908B64B197 6y ago
  This. Else you might as well call the degree "rote" because that's what you measure. > maybe fails in intro courses If someone cheats their way through intro courses they are up for a bad surprise since the upper level courses assumes a mastery of the intro classes!
  - _8091149529 6y ago
    Interesting, thanks for the replies. I'd still point out that the system is not robust against: - Working in groups. (Bad if tests are supposed to assess individual performance.) - Asking outsiders for help. Also, your downplaying of "rote" learning feels misguided, no matter how advanced/abstract/high-level the domain in question is. Cue the Bruce Lee quote about 10,000 kicks..
    - turbinerneiter 6y ago
      Just imagine a world were solving problems in groups and asking for outside help is bad. We are so used to exams being about doing stuff that literally nobody in real life would ever even consider.
      - _8091149529 6y ago
        I was aware of the implications, hence the disclaimer "(Bad if ...)" However, I'd assume that doing either during a test would be against the honor code.
    - 908B64B197 6y ago
      > Working in groups. Students hate carrying dead weight so to speak. Sure they might be tempted to swap favors (A helps B for a certain subject and the reverse is true for an other one). If the exam is properly constructed you might be able to detect plagiarism however. Rote is absolutely necessary for a well rounded education, but I really feel the need to overcorrect in the opposite direction because testing for rote is the lazy approach. Over time a lot of testing has the tendency to shift to simply measuring rote.
- BalinKing 6y ago
  One of my Ph 1b (freshman special relativity and electrostatics) quizzes guided you through "discovering" a new method of solving that type of problem. I don't remember the specific details, but it's one of the rare tests I've taken that I actually enjoyed! Although most homeworks and exams aren't nearly that cool IME, they're virtually all about synthesis and fundamental understanding rather than rote memorization. This "first principles" approach to most everything is one of my favorite things about Caltech.
  - tzs 6y ago
    When Carl Sagan was writing "Contact" he attended a conference that was also attended by his friend Kip Thorne. Sagan knew that one of Thorne's pet peeves was science fiction that just hand-waved away the physics of FTL travel. Sagan told Thorne he was writing a science fiction novel, needed FTL travel, and asked Thorne if he could suggest something that would be reasonable. Thorne agreed to look into it. After the conference, Thorne spent a while working on it and came up with a wormhole approach and worked out the physics of it. In addition to giving it to Sagan, Thorne also put it on the Ph 236 (General Relativity) final exam. He didn't tell the students on the exam that it might imply FTL travel. He just set up the conditions and had them work out the physics. Most of his students succeeded in that, but he was a little disappointed that none of them happened to notice that it implied FTL travel. (I got the above from an unpublished book Thorne was working on in the early '80s. It was a collection of biographies of and interviews with physicists, astronomers, cosmologists, etc., written and conducted by Thorne. He had the draft chapters and the raw interview transcripts in a world readable directory on the physic's departments VAX, where they were widely read by the rest of us with accounts on that machine).
    - aschatten 6y ago
      Do you know if it's possible to get ahold of a copy?
      - tzs 6y ago
        As far as I know, he never finished it, so unless he still has and would give out a copy it is probably unobtainable.
- jackson1442 6y ago
  I think the most important part of "proctoring" exams should be that it is individual work. Let's face it- you will almost always have access to the internet. The only thing that should matter is that the assigned student can solve the problem _in some manner._ This may include the internet, notes, or really anything. The real problem is people paying others to take exams, or simply copying answers from others. I work as an independent tutor, and I get contacted far too much by people looking to have me take their exams for them.
  - asdff 6y ago
    That was certainly a problem in person too. At my undergrad, for larger exams you would just pass your ID down the row to the person in the aisle and a TA would collect them all then check everyone in. Wouldn't be hard to just hand your ringer your ID to pass on up. Not unusual to look nothing like your freshman year ID photo in person.
- dghughes 6y ago
  That reminds me of when I took and distance learning electronics technician course. You could Google (search) all you wanted to but seeing answers didn't help in knowing how to solve the equations.
- maerF0x0 6y ago
  I had a professor that intentionally asked a question which wikipedia has the wrong answer... Called out and shamed everyone who used wikipedia as a source.
BalinKing 6y ago
Regarding "penalizing honesty," I think it actually rewards honestly quite greatly—because of the Honor Code's importance in Caltech culture (i.e. one of the undergrad application essays is specifically about the Honor Code), it means that everyone gets to benefit from take-home exams. I guess what I'm trying to say is that the "implied notion that Caltech students are... honest" is, in my experience, largely true (possibly because Caltech tries to select people who think the Honor Code is good from the get-go).
- _8091149529 6y ago
  My counterarguments, in good faith: Clearly, any society collectively benefits from honesty, whereas an honest act is, at least in a strict game-theoretic view, a loss to the individual in the short term. The question "Are you honest?" does not necessarily filter out dishonest people. (Incidentally, I always felt like I would not have been admitted to Caltech as a student.)
impendia 6y ago
I was an undergrad at Rice where we had a very strong honor code. Unsupervised exams were common, and to the best of my knowledge cheating was very rare. If you were accused of cheating then you went before the Honor Council, which was run entirely by students. I considered this an enormous success. It created an atmosphere where we were treated with respect, and where we were expected to treat others in the same way. If someone did cheat, then they would certainly be too ashamed to admit to it. It was a community where I was proud to spend four years. Yes, I would say that Rice students are honorable -- without the scare quotes. How is this achieved in practice? Wish I knew. This is the sort of thing that every university, and every organization, wants to achieve. Trust is very difficult to build if it's not already there; and it's much easier to erode.
- RonanTheGrey 6y ago
  > How is this achieved in practice? You were accountable to each other, not to an authority. Authority gets you enough to do the bare minimum to not get fired or not get caught. Being accountable to each other feels very very different. It's tough to explain in an HN post but anyone who has felt the difference knows what I meant. You care what your peers think. You care about that authority only enough for them to leave you alone. Never let it be said that shame isn't an effective tool.
  - smogcutter 6y ago
    > Never let it be said that shame isn't an effective tool. This is true, but at the same time I think shame as a motivator is often really unhealthy. People can do all kinds of terrible things to avoid shame: at the extremes, witness things like family annihilators or so-called “honor killings”. I’ve been reading a book about Midway lately so this is on the brain, but a culture of shame is part of the complex that prevented the Japanese from reeling in insubordinate but “patriotic” junior officers, and from realistically assessing their own and their opponents’ capabilities. Meanwhile the modern process for evaluating air accidents (constantly applauded on HN) explicitly removes the question of blame and shame. So, I’m not really saying anything about Rice university. I’m sure they don’t have to worry about honor killings on campus. This has just been on my mind lately. Just saying that without social release valves for shame you can get some really wild consequences. PS for anyone interested in Midway, read Shattered Sword for everything you ever wanted to know about early-war Japanese flight deck operations.
an_opabinia 6y ago
Insofar as universities produce pedigrees, it doesn’t really matter if Caltech students cheat. I suppose there’s greater justice in a student cheating on an exam and getting good grades while nobody knows - the student already won the lottery, they are getting the degree and a floor of a good lifestyle. Yet they will be less likely to commit suicide, you know? Or some other act of desperation. It’s a success because it lets students quietly cheat - if they need to - without reducing the graduation rate.
mmmBacon 6y ago
I had these at my university (not Caltech). These take home exams are much, much harder than anything they can give you in a timed in-person exam. It’s not even close. This is simply because the set of problems a person can solve in 3hrs limits the scope of the questions that could be asked. In fact, my friends and I would make lists of things we were sure wouldn’t be on the exam due to time constraints. I have to say that nobody cheated. This was partly due to my physics graduating class being 12 students.

beisner 6y ago

The idea is that you create questions that are ungoogleable. Most of my upper-level CS courses had take-homes that were open-book, open-internet, and the professor basically said “good luck finding the answer to this problem online; you will not”. This is harder for professors to do, and maybe fails in intro courses, but generally is quite effective.

908B64B197 6y ago
This. Else you might as well call the degree "rote" because that's what you measure. > maybe fails in intro courses If someone cheats their way through intro courses they are up for a bad surprise since the upper level courses assumes a mastery of the intro classes!
- _8091149529 6y ago
  Interesting, thanks for the replies. I'd still point out that the system is not robust against: - Working in groups. (Bad if tests are supposed to assess individual performance.) - Asking outsiders for help. Also, your downplaying of "rote" learning feels misguided, no matter how advanced/abstract/high-level the domain in question is. Cue the Bruce Lee quote about 10,000 kicks..
  - turbinerneiter 6y ago
    Just imagine a world were solving problems in groups and asking for outside help is bad. We are so used to exams being about doing stuff that literally nobody in real life would ever even consider.
    - _8091149529 6y ago
      I was aware of the implications, hence the disclaimer "(Bad if ...)" However, I'd assume that doing either during a test would be against the honor code.
  - 908B64B197 6y ago
    > Working in groups. Students hate carrying dead weight so to speak. Sure they might be tempted to swap favors (A helps B for a certain subject and the reverse is true for an other one). If the exam is properly constructed you might be able to detect plagiarism however. Rote is absolutely necessary for a well rounded education, but I really feel the need to overcorrect in the opposite direction because testing for rote is the lazy approach. Over time a lot of testing has the tendency to shift to simply measuring rote.
BalinKing 6y ago
One of my Ph 1b (freshman special relativity and electrostatics) quizzes guided you through "discovering" a new method of solving that type of problem. I don't remember the specific details, but it's one of the rare tests I've taken that I actually enjoyed! Although most homeworks and exams aren't nearly that cool IME, they're virtually all about synthesis and fundamental understanding rather than rote memorization. This "first principles" approach to most everything is one of my favorite things about Caltech.
- tzs 6y ago
  When Carl Sagan was writing "Contact" he attended a conference that was also attended by his friend Kip Thorne. Sagan knew that one of Thorne's pet peeves was science fiction that just hand-waved away the physics of FTL travel. Sagan told Thorne he was writing a science fiction novel, needed FTL travel, and asked Thorne if he could suggest something that would be reasonable. Thorne agreed to look into it. After the conference, Thorne spent a while working on it and came up with a wormhole approach and worked out the physics of it. In addition to giving it to Sagan, Thorne also put it on the Ph 236 (General Relativity) final exam. He didn't tell the students on the exam that it might imply FTL travel. He just set up the conditions and had them work out the physics. Most of his students succeeded in that, but he was a little disappointed that none of them happened to notice that it implied FTL travel. (I got the above from an unpublished book Thorne was working on in the early '80s. It was a collection of biographies of and interviews with physicists, astronomers, cosmologists, etc., written and conducted by Thorne. He had the draft chapters and the raw interview transcripts in a world readable directory on the physic's departments VAX, where they were widely read by the rest of us with accounts on that machine).
  - aschatten 6y ago
    Do you know if it's possible to get ahold of a copy?
    - tzs 6y ago
      As far as I know, he never finished it, so unless he still has and would give out a copy it is probably unobtainable.
jackson1442 6y ago
I think the most important part of "proctoring" exams should be that it is individual work. Let's face it- you will almost always have access to the internet. The only thing that should matter is that the assigned student can solve the problem _in some manner._ This may include the internet, notes, or really anything. The real problem is people paying others to take exams, or simply copying answers from others. I work as an independent tutor, and I get contacted far too much by people looking to have me take their exams for them.
- asdff 6y ago
  That was certainly a problem in person too. At my undergrad, for larger exams you would just pass your ID down the row to the person in the aisle and a TA would collect them all then check everyone in. Wouldn't be hard to just hand your ringer your ID to pass on up. Not unusual to look nothing like your freshman year ID photo in person.
dghughes 6y ago
That reminds me of when I took and distance learning electronics technician course. You could Google (search) all you wanted to but seeing answers didn't help in knowing how to solve the equations.
maerF0x0 6y ago
I had a professor that intentionally asked a question which wikipedia has the wrong answer... Called out and shamed everyone who used wikipedia as a source.

BalinKing 6y ago

Regarding "penalizing honesty," I think it actually rewards honestly quite greatly—because of the Honor Code's importance in Caltech culture (i.e. one of the undergrad application essays is specifically about the Honor Code), it means that everyone gets to benefit from take-home exams. I guess what I'm trying to say is that the "implied notion that Caltech students are... honest" is, in my experience, largely true (possibly because Caltech tries to select people who think the Honor Code is good from the get-go).

_8091149529 6y ago
My counterarguments, in good faith: Clearly, any society collectively benefits from honesty, whereas an honest act is, at least in a strict game-theoretic view, a loss to the individual in the short term. The question "Are you honest?" does not necessarily filter out dishonest people. (Incidentally, I always felt like I would not have been admitted to Caltech as a student.)

impendia 6y ago

I was an undergrad at Rice where we had a very strong honor code. Unsupervised exams were common, and to the best of my knowledge cheating was very rare. If you were accused of cheating then you went before the Honor Council, which was run entirely by students. I considered this an enormous success. It created an atmosphere where we were treated with respect, and where we were expected to treat others in the same way. If someone did cheat, then they would certainly be too ashamed to admit to it. It was a community where I was proud to spend four years. Yes, I would say that Rice students are honorable -- without the scare quotes. How is this achieved in practice? Wish I knew. This is the sort of thing that every university, and every organization, wants to achieve. Trust is very difficult to build if it's not already there; and it's much easier to erode.

RonanTheGrey 6y ago
> How is this achieved in practice? You were accountable to each other, not to an authority. Authority gets you enough to do the bare minimum to not get fired or not get caught. Being accountable to each other feels very very different. It's tough to explain in an HN post but anyone who has felt the difference knows what I meant. You care what your peers think. You care about that authority only enough for them to leave you alone. Never let it be said that shame isn't an effective tool.
- smogcutter 6y ago
  > Never let it be said that shame isn't an effective tool. This is true, but at the same time I think shame as a motivator is often really unhealthy. People can do all kinds of terrible things to avoid shame: at the extremes, witness things like family annihilators or so-called “honor killings”. I’ve been reading a book about Midway lately so this is on the brain, but a culture of shame is part of the complex that prevented the Japanese from reeling in insubordinate but “patriotic” junior officers, and from realistically assessing their own and their opponents’ capabilities. Meanwhile the modern process for evaluating air accidents (constantly applauded on HN) explicitly removes the question of blame and shame. So, I’m not really saying anything about Rice university. I’m sure they don’t have to worry about honor killings on campus. This has just been on my mind lately. Just saying that without social release valves for shame you can get some really wild consequences. PS for anyone interested in Midway, read Shattered Sword for everything you ever wanted to know about early-war Japanese flight deck operations.

an_opabinia 6y ago

Insofar as universities produce pedigrees, it doesn’t really matter if Caltech students cheat. I suppose there’s greater justice in a student cheating on an exam and getting good grades while nobody knows - the student already won the lottery, they are getting the degree and a floor of a good lifestyle. Yet they will be less likely to commit suicide, you know? Or some other act of desperation. It’s a success because it lets students quietly cheat - if they need to - without reducing the graduation rate.

mmmBacon 6y ago

I had these at my university (not Caltech). These take home exams are much, much harder than anything they can give you in a timed in-person exam. It’s not even close. This is simply because the set of problems a person can solve in 3hrs limits the scope of the questions that could be asked. In fact, my friends and I would make lists of things we were sure wouldn’t be on the exam due to time constraints. I have to say that nobody cheated. This was partly due to my physics graduating class being 12 students.