4 ms·

I judge high school engineering projects at the Synopsys Science Fair and California State Science Fair, and have become quite skilled in BS detection. Aside fr

I judge high school engineering projects at the Synopsys Science Fair and California State Science Fair, and have become quite skilled in BS detection. Aside from the usual methods to detect BS, the most revealing question in my experience has been "what problems did you run into while doing this project?" I ask this question to every student regardless of whether I think their project was BS or not, and the responses from legitimate students are usually sincere admissions of incompetence, while the response from a BSer is some inconsequential roadblock that they easily overcame. Students who are willing to submit a project where they did not produce the actual content will be willing to pawn off any experience or skill as their own, and will vastly overstate their abilities. The students who are humble about what they have accomplished and understand the full implications of their work are usually the ones who produce original work and win science fairs. This year, I saw a project where a high school senior used deep power analysis and machine learning to break AES 128 on a microcontroller (https://instagram.com/p/0JzlxBoIaJ/ https://instagram.com/p/0JzlxBoIaJ/). The student was from a prestigious private school, and the project required an expensive high-precision oscilloscope. Naturally I assumed that he had bought the project off eBay, so I didn't hold back in my questioning. But after I began questioning him, it quickly became clear that this was just an exceptionally bright mind who was willing to put hundreds of hours into deep understanding of a topic and reading the relevant research, before embarking on some original work of his own. He iterated over many possible solutions, and had an incredible demonstration of a working one. When I asked him difficult cryptography questions, he answered them precisely. And when I questioned him on his research methodology, he was humble in acknowledging his shortcomings and truthful in citing the research that guided certain aspects of his project. It is unfair to the exceptional few students capable of producing such a project to assume that all high school students are incapable of doing graduate-level research.

timr 12y ago
...or, you were fooled, and your methodology for BS detection isn't perfect. I've seen, first-hand, kids who were coached to perfection at this sort of thing. And they could easily pass your test: one common pattern is for the kid to be handed a "project" that is pre-conceptualized and nearly complete, and have them do the last 5% of work in consultation with a senior researcher (typically a post-doc). The kid basically flails around for a while, struggles a lot, and learns to talk the talk, but doesn't do much of anything in terms of productive work. So they'll struggle like crazy and hang around brilliant, hard-working people (and therefore be humbled and able to pass your test), but that doesn't mean they did the research. Doing a research project end-to-end takes years of concerted effort. It's almost completely unrealistic to think that a high school senior has put in the time.
- primitivesuave 12y ago
  I get what you're saying. Think about it from the perspective of the dishonest student though - they're handed a project that does some amazing thing, and need to pass it off as their own work. Unless they have the domain knowledge around that amazing thing, they couldn't possibly have done the amazing thing, and science fair bullshitters are generally not willing to put in the time to get a deep understanding of the area their BS'd project is in. The student I mentioned had a deep knowledge of cryptography, and with that understanding and the knowledge of relevant scientific research, was able to build something new that did an amazing thing. But if he was handed that project, I doubt he would have the motivation to research crypto to the extent that he did, or be conversant with another person. And most importantly, he knew what he didn't know.

timr 12y ago

...or, you were fooled, and your methodology for BS detection isn't perfect. I've seen, first-hand, kids who were coached to perfection at this sort of thing. And they could easily pass your test: one common pattern is for the kid to be handed a "project" that is pre-conceptualized and nearly complete, and have them do the last 5% of work in consultation with a senior researcher (typically a post-doc). The kid basically flails around for a while, struggles a lot, and learns to talk the talk, but doesn't do much of anything in terms of productive work. So they'll struggle like crazy and hang around brilliant, hard-working people (and therefore be humbled and able to pass your test), but that doesn't mean they did the research. Doing a research project end-to-end takes years of concerted effort. It's almost completely unrealistic to think that a high school senior has put in the time.

primitivesuave 12y ago
I get what you're saying. Think about it from the perspective of the dishonest student though - they're handed a project that does some amazing thing, and need to pass it off as their own work. Unless they have the domain knowledge around that amazing thing, they couldn't possibly have done the amazing thing, and science fair bullshitters are generally not willing to put in the time to get a deep understanding of the area their BS'd project is in. The student I mentioned had a deep knowledge of cryptography, and with that understanding and the knowledge of relevant scientific research, was able to build something new that did an amazing thing. But if he was handed that project, I doubt he would have the motivation to research crypto to the extent that he did, or be conversant with another person. And most importantly, he knew what he didn't know.