3 ms·

The author argues for evaluating Ph.D. candidates based more on the oral thesis defense than on the actual thesis. By essentially the same reasoning, I’ve been

by wrs 12d ago

The author argues for evaluating Ph.D. candidates based more on the oral thesis defense than on the actual thesis. By essentially the same reasoning, I’ve been arguing for prioritizing in-person design/code reviews over code-only async PR comments. The important thing is to verify that the human has a coherent design in mind and can demonstrate that it got implemented, regardless of who or what was at the keyboard. “I dunno, I guess Claude thought this was a good idea” is not a coherent design.

getnormality 12d ago
I think upweighting the live components of academia is inevitable in the age of automatically produced writing, but I also find it depressing that people think so little of writing that they imagine it obsolete because of AI. AI writing is aggressively, aggressively mediocre. It is only good for how cheap it is. If you're indifferent to how much better good human writing is than AI writing, you should not be the one to evaluate human writing.
Frost1x 12d ago
I can see how this trend is going to go. Manager: “Why isn’t feature X available?” Person B: “key pieces are delayed due to the developer not understanding all of the LLM doesn’t and implementation.” Manager: “does it work? What are the risks?” Person B: “well yes it works for now but we’re accumulating tech debt due to a lack of understanding and potential flaws that haven’t been thought out yet” Manager: “they want feature X, ship it, we can deal with it later, I don’t care if it’s not coherent as long as it works.” How many decades at this point has these been a push for functionality over everything at all costs? And you have a mechanical snow plow now. Most businesses don’t care about later risk or any future planning beyond the quarter horizon, they’re not concerned about how it will effect their performance in 3 quarters or lead to instability or issues, those are future problems for a future person and we’re here for money now.
- AdieuToLogic 11d ago
  The simulated dialog you have provided is very much representative of what many of us have heard first-hand. However, I disagree with: Most businesses don’t care about later risk or any future planning beyond the quarter horizon, they’re not concerned about how it will effect their performance in 3 quarters or lead to instability or issues, those are future problems for a future person and we’re here for money now. Businesses care about "later risk" and what it implies. Individuals within an organization do not unless it will specifically affect their bonuses/promotions.
  - gmadsen 11d ago
    It is also true developers can have a myopic view of what should be valued and that can be incongruent with the business. You can make a perfectly rational business choice to take on tech debt to get a feature sooner.
    - 0hmygauss 11d ago
      IMO this is exactly how software business operates in practice. Immaculate codebases we can effortlessly extended are a fairy tale of consultants. The world runs on tech debt that doesn’t matter enough to fix.
- 0hmygauss 11d ago
  Another way to say this is that this has worked fine for decades— ie, software businesses have done just fine prioritizing features over code cathedrals. What I find more surprising is that many engineers still can’t seem to accept this basic fact, after decades. I suspect AI will actually change this. The opportunity presented to business in dramatically relaxing both our understanding of our codebases and our tastes and preferences is big enough to justify letting go of folks who don’t get it.
jltsiren 12d ago
An oral defense is ultimately a meeting, with the same weaknesses as other types of meetings. In particular, if new information comes up in a meeting, you cannot reasonably expect to get useful responses from the other participants. First impressions maybe, and plausible-sounding bullshit from those prone to generating that. If you want anything more, you need to provide the information in advance or schedule another meeting later. Oral defenses in academia are largely rituals. If a student fails their defense, it's almost always the supervisor's fault. The supervisor is supposed to be the primary quality control. With their regular meetings with the student over the years, they should be able to tell whether the student has achieved sufficient understanding and contributed enough to graduate. If the supervisor thinks the student is ready to defend, the reasonable expectation is that the student will pass and graduate. The defense is mostly there to let the other examiners validate the supervisor's judgment.
- smueller1234 11d ago
  Here's another example of a similar but slightly different failure mode of the live defense: The department where I got my degree had a practice around PhD defense where there would be a prof each representing the four research pillars + the dean. It was well known that certain professors disliked each other enough that they would snipe each other's candidates. You'd just hope not to work for one whose enemies got picked for the panel. If you were a superstar, you'd pass, but if you were an okay candidate or someone strong but with stage fright, you be could toast even if you did good research. The defense was only briefly about the actual thesis, then switched over to whatever research interest the committee members had, and they'd drill into their pet subjects. This was in physics, hardly politics forward normally, and at a very well respected university.
- abdullahkhalids 11d ago
  You are missing the entire point of the oral defense while almost stating it. The existence of the oral defense is what extrinsically motivates the supervisor to do their job well. If the student fails, the supervisor loses faces in front of their peers, which obviously they try to avoid. Removing the oral defense would massively reduce the quality of PhDs because supervisors will not do their job. Also in my observational experience in physics, a fair number of students (probably like 10%) get some sort of major corrections to their thesis. After writing this, I did a search, and in the UK, the number across all fields is 16% [1]. [1] https://www.lexacademic.com/blog/how-common-is-passing-with-major-corrections-from-a-phd-lex-academic-blog/ https://www.lexacademic.com/blog/how-common-is-passing-with-...
lowbloodsugar 11d ago
I do this already if there’s any kind of issue. Just talk the person if they’re next to you or zoom them if not (don’t set up a meeting). A five minute conversation can save hours of back and forth.
MarceliusK 11d ago
The tricky part is that live review is much more expensive than async review. But maybe that just means we were getting away with a cheap proxy because it happened to work reasonably well.
- sigmoid10 11d ago
  Unfortunately, from my experience, forcing people to talk about their research to an audience mostly just condemns introverts and people with social anxiety. Being able to formulate your thoughts clearly in writing and in speech are two radically different skills. I've seen world class scientists give the most horrible conference presentations to the point where everyone in the audience agreed they'd rather have just read the paper when it came out, because they had no clue what he said while the author is amazing at writing. If we want to switch academia and to purely oral skills, you'll probably need to replace a ton of people. And in the industry these fields are already dominated by fast talking bullshitters.
lolakutty 11d ago
>“I dunno, I guess Claude thought this was a good idea” is not a coherent design. I am sorry, but the writing on the wall is that this is where software industry wants to go. As a software developer, I cannot but help notice how unrealistic this is. We are entering the next level of bloated, inefficient and unreliable software....

by wrs 12d ago

getnormality 12d ago

I think upweighting the live components of academia is inevitable in the age of automatically produced writing, but I also find it depressing that people think so little of writing that they imagine it obsolete because of AI. AI writing is aggressively, aggressively mediocre. It is only good for how cheap it is. If you're indifferent to how much better good human writing is than AI writing, you should not be the one to evaluate human writing.

Frost1x 12d ago

I can see how this trend is going to go. Manager: “Why isn’t feature X available?” Person B: “key pieces are delayed due to the developer not understanding all of the LLM doesn’t and implementation.” Manager: “does it work? What are the risks?” Person B: “well yes it works for now but we’re accumulating tech debt due to a lack of understanding and potential flaws that haven’t been thought out yet” Manager: “they want feature X, ship it, we can deal with it later, I don’t care if it’s not coherent as long as it works.” How many decades at this point has these been a push for functionality over everything at all costs? And you have a mechanical snow plow now. Most businesses don’t care about later risk or any future planning beyond the quarter horizon, they’re not concerned about how it will effect their performance in 3 quarters or lead to instability or issues, those are future problems for a future person and we’re here for money now.

AdieuToLogic 11d ago
The simulated dialog you have provided is very much representative of what many of us have heard first-hand. However, I disagree with: Most businesses don’t care about later risk or any future planning beyond the quarter horizon, they’re not concerned about how it will effect their performance in 3 quarters or lead to instability or issues, those are future problems for a future person and we’re here for money now. Businesses care about "later risk" and what it implies. Individuals within an organization do not unless it will specifically affect their bonuses/promotions.
- gmadsen 11d ago
  It is also true developers can have a myopic view of what should be valued and that can be incongruent with the business. You can make a perfectly rational business choice to take on tech debt to get a feature sooner.
  - 0hmygauss 11d ago
    IMO this is exactly how software business operates in practice. Immaculate codebases we can effortlessly extended are a fairy tale of consultants. The world runs on tech debt that doesn’t matter enough to fix.
0hmygauss 11d ago
Another way to say this is that this has worked fine for decades— ie, software businesses have done just fine prioritizing features over code cathedrals. What I find more surprising is that many engineers still can’t seem to accept this basic fact, after decades. I suspect AI will actually change this. The opportunity presented to business in dramatically relaxing both our understanding of our codebases and our tastes and preferences is big enough to justify letting go of folks who don’t get it.

jltsiren 12d ago

An oral defense is ultimately a meeting, with the same weaknesses as other types of meetings. In particular, if new information comes up in a meeting, you cannot reasonably expect to get useful responses from the other participants. First impressions maybe, and plausible-sounding bullshit from those prone to generating that. If you want anything more, you need to provide the information in advance or schedule another meeting later. Oral defenses in academia are largely rituals. If a student fails their defense, it's almost always the supervisor's fault. The supervisor is supposed to be the primary quality control. With their regular meetings with the student over the years, they should be able to tell whether the student has achieved sufficient understanding and contributed enough to graduate. If the supervisor thinks the student is ready to defend, the reasonable expectation is that the student will pass and graduate. The defense is mostly there to let the other examiners validate the supervisor's judgment.

smueller1234 11d ago
Here's another example of a similar but slightly different failure mode of the live defense: The department where I got my degree had a practice around PhD defense where there would be a prof each representing the four research pillars + the dean. It was well known that certain professors disliked each other enough that they would snipe each other's candidates. You'd just hope not to work for one whose enemies got picked for the panel. If you were a superstar, you'd pass, but if you were an okay candidate or someone strong but with stage fright, you be could toast even if you did good research. The defense was only briefly about the actual thesis, then switched over to whatever research interest the committee members had, and they'd drill into their pet subjects. This was in physics, hardly politics forward normally, and at a very well respected university.
abdullahkhalids 11d ago
You are missing the entire point of the oral defense while almost stating it. The existence of the oral defense is what extrinsically motivates the supervisor to do their job well. If the student fails, the supervisor loses faces in front of their peers, which obviously they try to avoid. Removing the oral defense would massively reduce the quality of PhDs because supervisors will not do their job. Also in my observational experience in physics, a fair number of students (probably like 10%) get some sort of major corrections to their thesis. After writing this, I did a search, and in the UK, the number across all fields is 16% [1]. [1] https://www.lexacademic.com/blog/how-common-is-passing-with-major-corrections-from-a-phd-lex-academic-blog/ https://www.lexacademic.com/blog/how-common-is-passing-with-...

lowbloodsugar 11d ago

I do this already if there’s any kind of issue. Just talk the person if they’re next to you or zoom them if not (don’t set up a meeting). A five minute conversation can save hours of back and forth.

MarceliusK 11d ago

The tricky part is that live review is much more expensive than async review. But maybe that just means we were getting away with a cheap proxy because it happened to work reasonably well.

sigmoid10 11d ago
Unfortunately, from my experience, forcing people to talk about their research to an audience mostly just condemns introverts and people with social anxiety. Being able to formulate your thoughts clearly in writing and in speech are two radically different skills. I've seen world class scientists give the most horrible conference presentations to the point where everyone in the audience agreed they'd rather have just read the paper when it came out, because they had no clue what he said while the author is amazing at writing. If we want to switch academia and to purely oral skills, you'll probably need to replace a ton of people. And in the industry these fields are already dominated by fast talking bullshitters.

lolakutty 11d ago

>“I dunno, I guess Claude thought this was a good idea” is not a coherent design. I am sorry, but the writing on the wall is that this is where software industry wants to go. As a software developer, I cannot but help notice how unrealistic this is. We are entering the next level of bloated, inefficient and unreliable software....