10 ms·

It is no longer effective to solely use a written essay to measure how deeply a student comprehends a subject. AI is here to stay; new methods should be used t

by lwhi 2y ago

It is no longer effective to solely use a written essay to measure how deeply a student comprehends a subject. AI is here to stay; new methods should be used to assess student performance. I remember being told at school, that we weren't allowed to use calculators in exams. The line provided by teachers was that we could never rely on having a calculator when we need it most—obviously there's irony associated with having 'calculators' in our pockets 24/7 now. We need to accept that the world has changed; I only hope that we get to decide how society responds to that change together .. rather than have it forced upon us.

UncleMeat 2y ago
This is mostly true, but it is also important to recognize that “hey just invent a new evaluation methodology” is a rough thing to ask people to do immediately. People are trying to figure it out in a way that works.
- A4ET8a8uTh0 2y ago
  Sadly, this is not what is happening. Based on the article ( and personal experience ), it is clear that we tend to happily accept computer output as a pronouncement from the oracle itself. It is new tech, but people do not treat it as such. They are not figuring it out. Its results are already being imposed. It is sheer luck that the individual in question choose to fight back. And even then it was only a partial victory: "The grade was ultimately changed, but not before she received a strict warning: If her work was flagged again, the teacher would treat it the same way they would with plagiarism."
ClumsyPilot 2y ago
> AI is here to stay; new methods should be used to assess student performance. This is overdue - we should be using interactive technology and not boring kids to death with a whiteboards. Bureaucracy works to protect itself and protect ease of administration. Even organising hand on practical lessons is harder
delusional 2y ago
> I remember being told at school, that we weren't allowed to use calculators in exams I remember being told the same thing, but I happen to believe that it was a fantastic policy, with a lackluster explanation. The idea that you wouldn't have a calculator was obviously silly, even at the time, but underlying observation that relying on the calculator would rob you of the mental exercise the whole ordeal was supposed to be was accurate. The problem is that you can't explain to a room full of 12 year olds that math is actually beautiful and that the systems principles it imparts fundamentally shape how you view the world. The same goes for essays. I hated writing essays, and I told myself all sort of weird copes about how I would never need to write an essay. The truth, that I've observed much later, is that structured thinking is exactly what the essay forced me to do. The essay was not a tool to asses my ability in a subject. It was a tool for me to learn. Writing the essay was part of the learning. I think that's what a lot of this "kids don't need to calculate in their heads" misses. Being able to do the calculation was only ever part of the idea. Learning that you could learn how to do the calculation was at least as important.
- _fizz_buzz_ 2y ago
  Very well put. I would actually suggest to not use calculators in high school anymore. They add very little value and if it is still the same as when I was in high school, it was a lot of remembering weird key combinations on a TI calculator. Simply make the arithmetic simple enough that a calculator isn't needed.
  - skydhash 2y ago
    I don't remember exactly, but I think we were only allowed the simplest calculators in middle school (none before), and scientific calculators in high schools (mostly for the trigonometric and power functions). I got to use a TI in university, but never used it that much as I've got the basic function graphs memorized.
- lwhi 2y ago
  Great point .. I agree; education is fundamentally exercise for the brain. Without challenge, the 'muscle' can't develop. I especially agree that essay writing is hugely useful. I'd even go as far as saying, the ability to think clearly is fundamental to a happy life.
strogonoff 2y ago
The best method for assessing performance when learning is as old as the world: assess the effort, not how well the result complies with some requirements. If the level of effort made is high, but the outcome does not comply in some way, praise is due. If the outcome complies, but the level of effort is low, there is no reason for praise (what are you praising? mere compliance?) and you must have set a wrong bar. Not doing this fosters people with mental issues such as rejection anxiety, perfectionism, narcissism, defeatism, etc. If you got good grades at school with little actual effort and the constant praise for that formed your identity, you may be in for a bad time in adulthood. Teacher’s job is to determine the appropriate bar, estimate the level of effort, and to help shape the effort applied in a way that it improves the skill in question and the more general meta skill of learning. The issue of judging by the outcome is prevalent in some (or all) school systems, so we can say LLMs are mostly orthogonal to that. However, even if that issue was addressed, in a number of skills the mere availability of ML-based generative tools makes it impossible to estimate the level of actual effort and to set the appropriate bar, and I do not see how it can be worked around. It’s yet another negative consequence of making the sacred process of producing an amalgamation of other people’s work—something we all do all the time; passing it through the lens of our consciousness is perhaps one of the core activities that make us human—to become available as a service.
- concordDance 2y ago
  > The best method for assessing performance when learning is as old as the world: assess the effort, not how well the result complies with some requirements. I am really quite confused about what you think the point of education is. In general, the world (either the physical world or the employment world) does not care about effort, it cares about results. Someone laboriously filling their kettle with a teaspoon might be putting in a ton of effort, but I'd much rather someone else make the tea who can use a tap. Why do we care about grades? Because universities and employers use them to quickly assess how useful someone is likely to be. Few people love biochemistry enough that they'd spend huge sums of money and time at university if it didn't help get them a job.
  - strogonoff 2y ago
pjc50 2y ago
> I only hope that we get to decide how society responds to that change together .. rather than have it forced upon us. That basically never happens and the outcome is the result of some sort of struggle. Usually just a peaceful one in the courts and legislatures and markets, but a struggle nonetheless. > new methods should be used to assess student performance. Such as? We need an answer now because students are being assessed now. Return to the old "viva voce" exam? Still used for PhDs. But that doesn't scale at all. Perhaps we're going to have to accept that and aggressively ration higher education by the limited amount of time available for human-to-human evaluations. Personally I think all this is unpredictable and destabilizing. If the AI advocates are right, which I don't think they are, they're going to eradicate most of the white collar jobs and academic specialties for which those people are being trained and evaluated.
- A4ET8a8uTh0 2y ago
  << The grade was ultimately changed, but not before she received a strict warning: If her work was flagged again, the teacher would treat it the same way they would with plagiarism. << But that doesn't scale at all. I realize that the level of effort for oral exam is greater for both parties involved. However, the fact it does not scale is largely irrelevant in my view. Either it evaluates something well or it does not. And, since use of AI makes written exams almost impossible, this genuinely seems to be the only real test left.
  - sersi 2y ago
    > And, since use of AI makes written exams almost impossible Isn't it easy to prevent students from using an AI if they are doing the exams in a big room? I mean when I was a student, most of my exams were written with just access to notes but no computers. Not that much resources needed to control that...
    - A4ET8a8uTh0 2y ago
      Good point. I agree, but it goes back to some level of unwillingness to do this the 'old way'. That is not say there won't be cheaters ( they always are ), but that is what proctor is for. And no, I absolutely hated the online proctor version. I swore I will never touch that thing again. And this may be the answer, people need to exercise their free will a little more forcefully.
dambi0 2y ago
An essay written under examination conditions is fine. We don’t need new assessment techniques. We have known how to asses that a student and that student alone for centuries.
- moffkalast 2y ago
  Yeah we always did that in high school for essays that were actually graded, otherwise there's always the option of having someone else write it for you, human or now machine. The only thing that's changed is the convenience of it. The problem is more with teachers lazily slapping an essay on a topic as a goto homework to eat even more of the already limited students' time with busywork.
  - tonypace 2y ago
    The lazy essay assignment is 100% real. However, the driving force there is not the teacher, but parental complaints causing ass-covering administrative mandates. "Why wasn't there any homework on topic X before the exam?" "We apologize so much for that, Mrs Keen. First, we will change Precious's grade, but from now on..."
- VBprogrammer 2y ago
  My ability to write an essay under exam conditions is...poor. Thankfully there were less than a handful of essays I had to write as part of my undergraduate CS degree and I only remember one under exam conditions. I think it's probably more concerning that spitting out the most generic mathematically formulaic bullshit on a subject is likely to get a decent mark. In that case what are we actually testing for?
  - thechao 2y ago
    Conformance.
    - bigfudge 2y ago
      Amusingly, willingness and capacity to conform to a system you are paying $30k a year for is a pretty good proxy for general intelligence. So maybe it's not that bad?!
EnigmaFlare 2y ago
[dead]
elric 2y ago
Was this ever effective? There was a lot of essay copy/pasting when I was in school, and this was when essays had to be hand written (in cursive, of course, using a fountain pen!). Same with homework. If everyone has to solve the same 10 problems, divide and conquer saves everyone a lot of time. Of course, you're only screwing yourself because you'll negatively impact your learning, but that's not something you can easily convince kids of. In person oral exams (once you get over the fear factor) work best, with or without (proctored!) prep time. Maybe it doesn't scale as well, but education is important enough not to always require maximal efficiency.
- everdrive 2y ago
  >Of course, you're only screwing yourself because you'll negatively impact your learning, but that's not something you can easily convince kids of. This assumes that homework helps kids learn, or that the knowledge required to succeed in school will help kids once they graduate.
  - elric 2y ago
    Depends on the homework, of course. In my head I guess I was talking about maths problems. Maths understanding, in my experience, greatly benefits from practice, and homework exercises might be useful there. Memorising the names of rivers ... maybe not so much.
torginus 2y ago
Just because a method of assessment became easily spoofable doesn't mean we should give up on it. Imagine if in the era before HTTPS we just said that the internet won't be really viable because it's impossible to communicate securely on it. I still feel like AI detectors would work well if we have access to the exact model, output probabilities of tokens, We can just take a bit of given text, and calculate the cumulative probability that the AI would complete it exactly like that.
- nick3443 2y ago
  Probability is not an acceptable way to determine a student's future. They may have learned from the AI and remember some of the exact phrasing, and learned writing/language cues from it as well.
  - alias_neo 2y ago
    Agreed. I'm not a good writer, tending to stick to a somewhat abrupt, point-making structure almost better suited for bullet pointing. I've taken tips from other HN users on how to improve, but I have no doubt that had I been going through university these days, I'd probably be flagged too.
red_admiral 2y ago
> It is no longer effective to solely use a written essay to measure how deeply a student comprehends a subject. It never was. It's just even more ineffective now that AI exists, than before. The central example of this is college admissions statements. Some kids have the advantage both of parents who can afford to give them the experiences that look good on such an essay (educational trips to Africa, lessons in two musical instruments, one-on-one golf coaching, that kind of thing), and who can hire tutors to "support" them in writing the essay. AI just makes the tutor part accessible/affordable for a wider segment of the population. It would be naive to assume that, pre-AI, there was not a "gray" essay-coaching market as well as the "dark" essay-writing as a service market. That market still works better than AI in many cases.
- johnisgood 2y ago
  It is not so black and white though: there is a difference between having your whole essay written by a tutor, or having some things corrected by the tutor, or the tutor giving you general tips that you yourself apply.
  - cryptonym 2y ago
    Just like there is a difference between having your whole essay written by a LLM, or having some things corrected by the LLM, or the LLM giving you general tips that you yourself apply.
    - johnisgood 2y ago
      I agree.
  - red_admiral 2y ago
    Oh, I completely agree. In some cases, discussing a draft with your _university-appointed_ tutor before submitting your final essay is even part of the assignment (I believe Oxford/Cambridge humanities work this way), and a great learning experience, and a way for people who can't afford private tutors to get the same kind of coaching (how you get into this calibre of university in the first place notwithstanding).
dandanua 2y ago
On some exams in our university 20y ago, we were allowed to use any literature or lecture notes to answer the questions. The thing is, it was a high level abstract algebra. If you don't understand the subject, no amount of literature would help you to answer the questions correctly (unless you find the exact or a very similar question). I believe it's still true today, but with future AI systems even highly abstract math is under the danger.
welder 2y ago
My first semester undergrad English course, the professor graded all my papers D or worse. Had to repeat the course with a different professor. They shared assignments so I re-used the same essays with zero modifications... but this time I got an A or higher!
- lwhi 2y ago
  I have a similar memory. I wrote an essay about a poem. The poem was assigned to us, but for some reason the subject matter really chimed with me personally. I thought about it a lot, and—as a result—ended up writing a great essay. Because I did well, I was accused of cheating in front the class. Teachers are definitely fallible.
renegade-otter 2y ago
> AI is here to stay Let's not assume a lot right now. OpenAI and other companies are torching through cash like drunken socialist sailors. Will AI be here as a Big Data 2.0 B2B technology? Most likely, but a viable model where students and laypeople have access to it? To be seen. We all mooched off of dumb VC money at one point or another. I acquired a few expensive watches at Fab dot com at 80% off when they were giving money away, eh.
- Ukv 2y ago
  > [...] but a viable model where students and laypeople have access to it? To be seen. You can run GPT-4-equivalent models locally. Even if all software and hardware advancements immediately halt, models at the current level will remain available.
  - blibble 2y ago
    how useful will a 2024 era model be in 2030? 2040? 2050?
    - Ukv 2y ago
      A TI-84 Plus calculator (over 20 years old) is still useful today. In isolation, I don't think a model necessarily becomes less useful over time. It'll still be as good at summarizing articles, translating text, correcting grammar, etc. for you as it is today. If things do continue to advance and new models are released, which I think is likely, the old ones become less useful by comparison and in situations where there's competition. But then, through hardware/algorithmic improvements, better models also become feasible for universities/open-source groups/individuals - so you shouldn't be stuck with a 2024 era model.
    - ssl-3 2y ago
      How useful is it to argue about what would happen in the extraordinarily unlikely eventuality that all LLM development will cease in 2024, wherein everyone with the proclivity to use LLMs will be stuck with exactly these same models for decades to come?
gklitz 2y ago
Written assay evaluation is not and has never been an effective evaluation. It was always a cost saving measure because allocating 30min face to face time with each individual student for each class is such a gigantic cost for the institution that they cannot even imagine doing it. Think about that the next time you look at your student debt, it couldn’t even buy you 30min time per class individually with the teacher to evaluate your performance. Instead you had to waste more time on a written assignment so they could offload grading to a minimum wage assistent.
- Teever 2y ago
  There is truth to this perspective but it's also missing one of the fundamental purposes of writing essays in an educational setting. Writing essays isn't just about evaluation, it's also about teaching you how to think. The process of reading textual material, thinking about it, and then producing more textual material about the stuff you just read (and maybe connecting it to other stuff that you've read in the past) is a critical way of developing thinking skills and refining your ability to communicate to an audience. The value of that shouldn't be overlooked just like the value of basic numeracy shouldn't be overlooked because we all carry calculators. You're right that it would be better if post secondary institutions would test people's ability to think in more ways than just what they can regurgitate onto a piece of paper, if only because that can be easily cheated but that doesn't mean that there isn't personal benefit in the experience of writing an essay. I may not be the best writer but I am a better writer because I wrote essays in university, and I may not be great at math but I can reason and estimate about a variety of things because I have taken many math courses. These things have ultimately made me a better thinker and I am grateful to have had that imparted to me.
  - SketchySeaBeast 2y ago
    All excellent point, but I'd like to add that it also forces you to do your own research the correct way, by surveying the current state of academic research and then finding and incorporating scholarly sources into your own arguments. Every academic essay I ever wrote after high school started with a trip to the library and JSTOR. I had to guide my own education instead of learning from the teacher and then repeating what had been taught.
lm28469 2y ago
In France my essays were written in class, no phones, no book, just your brain, a sheet a paper and a pen. That's still 100% doable today
- junaru 2y ago
  It even came with handwriting built in as authentication mechanism! AI detectors hate this secret! On a more serious note - US removed cursive from their curriculum almost two decades ago - something i cant wrap my head around as cursive is something the rest of the world(?) uses starting in middle school and onwards through the whole adult life.
  - mikeocool 2y ago
    21 states still mandate cursive in their curriculum. There are lots of things I spent a lot of time learning school that I rarely use, but see the value in having learnt. Cursive, beyond a very basic level, is not one of those things. Though I’m no education expert, perhaps there is a subliminal value to spending all that time.
    - iteria 2y ago
      I learned cursive in school and never used it. I have written countless essays by hand in grade school and college and never felt the need to do it in cursive because to me my cursive was just way more unreadable than my printing and not particularly faster.
  - kibwen 2y ago
    I don't know what the rest of the world calls "cursive", but here in the US the cursive we get taught is strictly inferior: slower to write, less compact on the page, and harder to read (while also being strictly uglier than true calligraphy). It's a script designed for allowing you to avoid lifting a quill from the page and thereby avoiding ink blots; it's entirely obsolete.
    - lolinder 2y ago
deleted 2y ago
[deleted]
ransom1538 2y ago
IMHO. With calculators introduced, there is zero add in you learning long division. Worse than zero, you could have done something better with your time. ChatGPT is a calculator for all subjects. People have a hard time letting that sink in.
- erikerikson 2y ago
  AI is a calculator for all subjects, ChatGPT is not that advanced.
  - spacebanana7 2y ago
    I still bet someone like you could pass any university exam in any subject with access to the ChatGPT app. Without any prep time. At that point it’s good enough.
- vundercind 2y ago
  Long (or short—screw long division, with its transcription error opportunities and huge amounts of paper-space used) division is a good exercise to cement the notion of place value, that happens to also teach you how to divide by hand for when it's occasionally more convenient than finding a phone/computer/calculator.
ReptileMan 2y ago
>AI is here to stay; new methods should be used to assess student performance. Here is the brand new method - asking verbal questions in person and evaluating answers. Also allow high tech aides in the form of chalk and blackboard
- spacebanana7 2y ago
  The downside of downgrading technology like this is that tests and skills become less relevant to the real world. For all their problems, 5000 word take home assignments in Microsoft Office have a lot in common with the activities of a junior management consultant, NGO writer, lawyer or business analyst. And same with for scientists but with Latex. I’d rather hire a lawyer who could only do their job with AI than one who couldn’t use a computer to create documents or use digital tools to search case law.
darepublic 2y ago
When institutions use simple rules to respond to change and rigidly follow them without due judgement, then some will fall through the cracks, and others will grift off them
caseyy 2y ago
We blasted through the “you won’t always have AI in your pocket” phase in a blink of an eye. Local LLMs were running on smartphones before the world got to terms of LLMs being used everywhere. It’s one of many examples of exponential technological advancement.
caseyy 2y ago
The old colloquium exam format reigns supreme again. And that is fantastic. We shouldn’t reserve it for only “most important” occasions because quality education is important enough by itself.
naming_the_user 2y ago
Nonsense. You are in a room with a sheet of paper and a pen. Go. You’re acting as if 2010 was a hundred years ago.
bonoboTP 2y ago
Learning takes time. And the fully trained/educated/skilled/expert human performance is higher than AI performance. But AI performance may be higher than intermediate human performance after 1 or 2 semesters. But you need to reach intermediate performance first in order to later reach expert performance. During that time you still need a learning "slope", you need to be tested on your knowledge at that level. If you're given the AI at the outset, you will not develop the skill to surpass the AI performance. Calculators are just one analogy, there is no guarantee it will work out that way. It's just as likely that this over-technologization of the classroom will go the way of whole-language reading education.
yojo 2y ago
I won’t claim this is by design, but at the very least a side effect of writing term papers is getting practice at organizing your thoughts and drawing conclusions from them. While writing term papers is a skill that is only minimally useful in the real world (save for grant writers and post docs, pretty much), the patterns of thinking it encourages are valuable to everything that isn’t ditch digging. Maybe we can outsource this part of our cognition to AI, but I’m skeptical of the wisdom of doing so. Are we all going to break to consult ChatGPT in strategy meetings?
dogleash 2y ago
> we could never rely on having a calculator when we need it most—obviously there's irony associated with having 'calculators' in our pockets 24/7 now That was just a simple quip to shut down student bellyaching. Even before we had pocket calculators, it was never a strong answer. It just had to hold over long enough so when you realized it was bad answer you weren't that teacher's problem anymore. The actual answer was that they're complaining about a minor inconvenience designed for reinforcement, and if they really did need a calculator for the arithmetic on a test designed deliberately designed to be taken without a calculator, then they don't belong in that class.
kjkjadksj 2y ago
We used to write essays in class on blue books. That can still be done today.

by lwhi 2y ago

UncleMeat 2y ago

This is mostly true, but it is also important to recognize that “hey just invent a new evaluation methodology” is a rough thing to ask people to do immediately. People are trying to figure it out in a way that works.

A4ET8a8uTh0 2y ago
Sadly, this is not what is happening. Based on the article ( and personal experience ), it is clear that we tend to happily accept computer output as a pronouncement from the oracle itself. It is new tech, but people do not treat it as such. They are not figuring it out. Its results are already being imposed. It is sheer luck that the individual in question choose to fight back. And even then it was only a partial victory: "The grade was ultimately changed, but not before she received a strict warning: If her work was flagged again, the teacher would treat it the same way they would with plagiarism."

ClumsyPilot 2y ago

> AI is here to stay; new methods should be used to assess student performance. This is overdue - we should be using interactive technology and not boring kids to death with a whiteboards. Bureaucracy works to protect itself and protect ease of administration. Even organising hand on practical lessons is harder

delusional 2y ago

> I remember being told at school, that we weren't allowed to use calculators in exams I remember being told the same thing, but I happen to believe that it was a fantastic policy, with a lackluster explanation. The idea that you wouldn't have a calculator was obviously silly, even at the time, but underlying observation that relying on the calculator would rob you of the mental exercise the whole ordeal was supposed to be was accurate. The problem is that you can't explain to a room full of 12 year olds that math is actually beautiful and that the systems principles it imparts fundamentally shape how you view the world. The same goes for essays. I hated writing essays, and I told myself all sort of weird copes about how I would never need to write an essay. The truth, that I've observed much later, is that structured thinking is exactly what the essay forced me to do. The essay was not a tool to asses my ability in a subject. It was a tool for me to learn. Writing the essay was part of the learning. I think that's what a lot of this "kids don't need to calculate in their heads" misses. Being able to do the calculation was only ever part of the idea. Learning that you could learn how to do the calculation was at least as important.

_fizz_buzz_ 2y ago
Very well put. I would actually suggest to not use calculators in high school anymore. They add very little value and if it is still the same as when I was in high school, it was a lot of remembering weird key combinations on a TI calculator. Simply make the arithmetic simple enough that a calculator isn't needed.
- skydhash 2y ago
  I don't remember exactly, but I think we were only allowed the simplest calculators in middle school (none before), and scientific calculators in high schools (mostly for the trigonometric and power functions). I got to use a TI in university, but never used it that much as I've got the basic function graphs memorized.
lwhi 2y ago
Great point .. I agree; education is fundamentally exercise for the brain. Without challenge, the 'muscle' can't develop. I especially agree that essay writing is hugely useful. I'd even go as far as saying, the ability to think clearly is fundamental to a happy life.

strogonoff 2y ago

The best method for assessing performance when learning is as old as the world: assess the effort, not how well the result complies with some requirements. If the level of effort made is high, but the outcome does not comply in some way, praise is due. If the outcome complies, but the level of effort is low, there is no reason for praise (what are you praising? mere compliance?) and you must have set a wrong bar. Not doing this fosters people with mental issues such as rejection anxiety, perfectionism, narcissism, defeatism, etc. If you got good grades at school with little actual effort and the constant praise for that formed your identity, you may be in for a bad time in adulthood. Teacher’s job is to determine the appropriate bar, estimate the level of effort, and to help shape the effort applied in a way that it improves the skill in question and the more general meta skill of learning. The issue of judging by the outcome is prevalent in some (or all) school systems, so we can say LLMs are mostly orthogonal to that. However, even if that issue was addressed, in a number of skills the mere availability of ML-based generative tools makes it impossible to estimate the level of actual effort and to set the appropriate bar, and I do not see how it can be worked around. It’s yet another negative consequence of making the sacred process of producing an amalgamation of other people’s work—something we all do all the time; passing it through the lens of our consciousness is perhaps one of the core activities that make us human—to become available as a service.

concordDance 2y ago
> The best method for assessing performance when learning is as old as the world: assess the effort, not how well the result complies with some requirements. I am really quite confused about what you think the point of education is. In general, the world (either the physical world or the employment world) does not care about effort, it cares about results. Someone laboriously filling their kettle with a teaspoon might be putting in a ton of effort, but I'd much rather someone else make the tea who can use a tap. Why do we care about grades? Because universities and employers use them to quickly assess how useful someone is likely to be. Few people love biochemistry enough that they'd spend huge sums of money and time at university if it didn't help get them a job.
- strogonoff 2y ago

pjc50 2y ago

> I only hope that we get to decide how society responds to that change together .. rather than have it forced upon us. That basically never happens and the outcome is the result of some sort of struggle. Usually just a peaceful one in the courts and legislatures and markets, but a struggle nonetheless. > new methods should be used to assess student performance. Such as? We need an answer now because students are being assessed now. Return to the old "viva voce" exam? Still used for PhDs. But that doesn't scale at all. Perhaps we're going to have to accept that and aggressively ration higher education by the limited amount of time available for human-to-human evaluations. Personally I think all this is unpredictable and destabilizing. If the AI advocates are right, which I don't think they are, they're going to eradicate most of the white collar jobs and academic specialties for which those people are being trained and evaluated.

A4ET8a8uTh0 2y ago
<< The grade was ultimately changed, but not before she received a strict warning: If her work was flagged again, the teacher would treat it the same way they would with plagiarism. << But that doesn't scale at all. I realize that the level of effort for oral exam is greater for both parties involved. However, the fact it does not scale is largely irrelevant in my view. Either it evaluates something well or it does not. And, since use of AI makes written exams almost impossible, this genuinely seems to be the only real test left.
- sersi 2y ago
  > And, since use of AI makes written exams almost impossible Isn't it easy to prevent students from using an AI if they are doing the exams in a big room? I mean when I was a student, most of my exams were written with just access to notes but no computers. Not that much resources needed to control that...
  - A4ET8a8uTh0 2y ago
    Good point. I agree, but it goes back to some level of unwillingness to do this the 'old way'. That is not say there won't be cheaters ( they always are ), but that is what proctor is for. And no, I absolutely hated the online proctor version. I swore I will never touch that thing again. And this may be the answer, people need to exercise their free will a little more forcefully.

dambi0 2y ago

An essay written under examination conditions is fine. We don’t need new assessment techniques. We have known how to asses that a student and that student alone for centuries.

moffkalast 2y ago
Yeah we always did that in high school for essays that were actually graded, otherwise there's always the option of having someone else write it for you, human or now machine. The only thing that's changed is the convenience of it. The problem is more with teachers lazily slapping an essay on a topic as a goto homework to eat even more of the already limited students' time with busywork.
- tonypace 2y ago
  The lazy essay assignment is 100% real. However, the driving force there is not the teacher, but parental complaints causing ass-covering administrative mandates. "Why wasn't there any homework on topic X before the exam?" "We apologize so much for that, Mrs Keen. First, we will change Precious's grade, but from now on..."
VBprogrammer 2y ago
My ability to write an essay under exam conditions is...poor. Thankfully there were less than a handful of essays I had to write as part of my undergraduate CS degree and I only remember one under exam conditions. I think it's probably more concerning that spitting out the most generic mathematically formulaic bullshit on a subject is likely to get a decent mark. In that case what are we actually testing for?
- thechao 2y ago
  Conformance.
  - bigfudge 2y ago
    Amusingly, willingness and capacity to conform to a system you are paying $30k a year for is a pretty good proxy for general intelligence. So maybe it's not that bad?!

EnigmaFlare 2y ago

[dead]

elric 2y ago

Was this ever effective? There was a lot of essay copy/pasting when I was in school, and this was when essays had to be hand written (in cursive, of course, using a fountain pen!). Same with homework. If everyone has to solve the same 10 problems, divide and conquer saves everyone a lot of time. Of course, you're only screwing yourself because you'll negatively impact your learning, but that's not something you can easily convince kids of. In person oral exams (once you get over the fear factor) work best, with or without (proctored!) prep time. Maybe it doesn't scale as well, but education is important enough not to always require maximal efficiency.

everdrive 2y ago
>Of course, you're only screwing yourself because you'll negatively impact your learning, but that's not something you can easily convince kids of. This assumes that homework helps kids learn, or that the knowledge required to succeed in school will help kids once they graduate.
- elric 2y ago
  Depends on the homework, of course. In my head I guess I was talking about maths problems. Maths understanding, in my experience, greatly benefits from practice, and homework exercises might be useful there. Memorising the names of rivers ... maybe not so much.

torginus 2y ago

Just because a method of assessment became easily spoofable doesn't mean we should give up on it. Imagine if in the era before HTTPS we just said that the internet won't be really viable because it's impossible to communicate securely on it. I still feel like AI detectors would work well if we have access to the exact model, output probabilities of tokens, We can just take a bit of given text, and calculate the cumulative probability that the AI would complete it exactly like that.

nick3443 2y ago
Probability is not an acceptable way to determine a student's future. They may have learned from the AI and remember some of the exact phrasing, and learned writing/language cues from it as well.
- alias_neo 2y ago
  Agreed. I'm not a good writer, tending to stick to a somewhat abrupt, point-making structure almost better suited for bullet pointing. I've taken tips from other HN users on how to improve, but I have no doubt that had I been going through university these days, I'd probably be flagged too.

red_admiral 2y ago

> It is no longer effective to solely use a written essay to measure how deeply a student comprehends a subject. It never was. It's just even more ineffective now that AI exists, than before. The central example of this is college admissions statements. Some kids have the advantage both of parents who can afford to give them the experiences that look good on such an essay (educational trips to Africa, lessons in two musical instruments, one-on-one golf coaching, that kind of thing), and who can hire tutors to "support" them in writing the essay. AI just makes the tutor part accessible/affordable for a wider segment of the population. It would be naive to assume that, pre-AI, there was not a "gray" essay-coaching market as well as the "dark" essay-writing as a service market. That market still works better than AI in many cases.

johnisgood 2y ago
It is not so black and white though: there is a difference between having your whole essay written by a tutor, or having some things corrected by the tutor, or the tutor giving you general tips that you yourself apply.
- cryptonym 2y ago
  Just like there is a difference between having your whole essay written by a LLM, or having some things corrected by the LLM, or the LLM giving you general tips that you yourself apply.
  - johnisgood 2y ago
    I agree.
- red_admiral 2y ago
  Oh, I completely agree. In some cases, discussing a draft with your _university-appointed_ tutor before submitting your final essay is even part of the assignment (I believe Oxford/Cambridge humanities work this way), and a great learning experience, and a way for people who can't afford private tutors to get the same kind of coaching (how you get into this calibre of university in the first place notwithstanding).

dandanua 2y ago

On some exams in our university 20y ago, we were allowed to use any literature or lecture notes to answer the questions. The thing is, it was a high level abstract algebra. If you don't understand the subject, no amount of literature would help you to answer the questions correctly (unless you find the exact or a very similar question). I believe it's still true today, but with future AI systems even highly abstract math is under the danger.

welder 2y ago

My first semester undergrad English course, the professor graded all my papers D or worse. Had to repeat the course with a different professor. They shared assignments so I re-used the same essays with zero modifications... but this time I got an A or higher!

lwhi 2y ago
I have a similar memory. I wrote an essay about a poem. The poem was assigned to us, but for some reason the subject matter really chimed with me personally. I thought about it a lot, and—as a result—ended up writing a great essay. Because I did well, I was accused of cheating in front the class. Teachers are definitely fallible.

renegade-otter 2y ago

> AI is here to stay Let's not assume a lot right now. OpenAI and other companies are torching through cash like drunken socialist sailors. Will AI be here as a Big Data 2.0 B2B technology? Most likely, but a viable model where students and laypeople have access to it? To be seen. We all mooched off of dumb VC money at one point or another. I acquired a few expensive watches at Fab dot com at 80% off when they were giving money away, eh.

Ukv 2y ago
> [...] but a viable model where students and laypeople have access to it? To be seen. You can run GPT-4-equivalent models locally. Even if all software and hardware advancements immediately halt, models at the current level will remain available.
- blibble 2y ago
  how useful will a 2024 era model be in 2030? 2040? 2050?
  - Ukv 2y ago
    A TI-84 Plus calculator (over 20 years old) is still useful today. In isolation, I don't think a model necessarily becomes less useful over time. It'll still be as good at summarizing articles, translating text, correcting grammar, etc. for you as it is today. If things do continue to advance and new models are released, which I think is likely, the old ones become less useful by comparison and in situations where there's competition. But then, through hardware/algorithmic improvements, better models also become feasible for universities/open-source groups/individuals - so you shouldn't be stuck with a 2024 era model.
  - ssl-3 2y ago
    How useful is it to argue about what would happen in the extraordinarily unlikely eventuality that all LLM development will cease in 2024, wherein everyone with the proclivity to use LLMs will be stuck with exactly these same models for decades to come?

gklitz 2y ago

Written assay evaluation is not and has never been an effective evaluation. It was always a cost saving measure because allocating 30min face to face time with each individual student for each class is such a gigantic cost for the institution that they cannot even imagine doing it. Think about that the next time you look at your student debt, it couldn’t even buy you 30min time per class individually with the teacher to evaluate your performance. Instead you had to waste more time on a written assignment so they could offload grading to a minimum wage assistent.

Teever 2y ago
There is truth to this perspective but it's also missing one of the fundamental purposes of writing essays in an educational setting. Writing essays isn't just about evaluation, it's also about teaching you how to think. The process of reading textual material, thinking about it, and then producing more textual material about the stuff you just read (and maybe connecting it to other stuff that you've read in the past) is a critical way of developing thinking skills and refining your ability to communicate to an audience. The value of that shouldn't be overlooked just like the value of basic numeracy shouldn't be overlooked because we all carry calculators. You're right that it would be better if post secondary institutions would test people's ability to think in more ways than just what they can regurgitate onto a piece of paper, if only because that can be easily cheated but that doesn't mean that there isn't personal benefit in the experience of writing an essay. I may not be the best writer but I am a better writer because I wrote essays in university, and I may not be great at math but I can reason and estimate about a variety of things because I have taken many math courses. These things have ultimately made me a better thinker and I am grateful to have had that imparted to me.
- SketchySeaBeast 2y ago
  All excellent point, but I'd like to add that it also forces you to do your own research the correct way, by surveying the current state of academic research and then finding and incorporating scholarly sources into your own arguments. Every academic essay I ever wrote after high school started with a trip to the library and JSTOR. I had to guide my own education instead of learning from the teacher and then repeating what had been taught.

lm28469 2y ago

In France my essays were written in class, no phones, no book, just your brain, a sheet a paper and a pen. That's still 100% doable today

junaru 2y ago
It even came with handwriting built in as authentication mechanism! AI detectors hate this secret! On a more serious note - US removed cursive from their curriculum almost two decades ago - something i cant wrap my head around as cursive is something the rest of the world(?) uses starting in middle school and onwards through the whole adult life.
- mikeocool 2y ago
  21 states still mandate cursive in their curriculum. There are lots of things I spent a lot of time learning school that I rarely use, but see the value in having learnt. Cursive, beyond a very basic level, is not one of those things. Though I’m no education expert, perhaps there is a subliminal value to spending all that time.
  - iteria 2y ago
    I learned cursive in school and never used it. I have written countless essays by hand in grade school and college and never felt the need to do it in cursive because to me my cursive was just way more unreadable than my printing and not particularly faster.
- kibwen 2y ago
  I don't know what the rest of the world calls "cursive", but here in the US the cursive we get taught is strictly inferior: slower to write, less compact on the page, and harder to read (while also being strictly uglier than true calligraphy). It's a script designed for allowing you to avoid lifting a quill from the page and thereby avoiding ink blots; it's entirely obsolete.
  - lolinder 2y ago

deleted 2y ago

[deleted]

ransom1538 2y ago

IMHO. With calculators introduced, there is zero add in you learning long division. Worse than zero, you could have done something better with your time. ChatGPT is a calculator for all subjects. People have a hard time letting that sink in.

erikerikson 2y ago
AI is a calculator for all subjects, ChatGPT is not that advanced.
- spacebanana7 2y ago
  I still bet someone like you could pass any university exam in any subject with access to the ChatGPT app. Without any prep time. At that point it’s good enough.
vundercind 2y ago
Long (or short—screw long division, with its transcription error opportunities and huge amounts of paper-space used) division is a good exercise to cement the notion of place value, that happens to also teach you how to divide by hand for when it's occasionally more convenient than finding a phone/computer/calculator.

ReptileMan 2y ago

>AI is here to stay; new methods should be used to assess student performance. Here is the brand new method - asking verbal questions in person and evaluating answers. Also allow high tech aides in the form of chalk and blackboard

spacebanana7 2y ago
The downside of downgrading technology like this is that tests and skills become less relevant to the real world. For all their problems, 5000 word take home assignments in Microsoft Office have a lot in common with the activities of a junior management consultant, NGO writer, lawyer or business analyst. And same with for scientists but with Latex. I’d rather hire a lawyer who could only do their job with AI than one who couldn’t use a computer to create documents or use digital tools to search case law.

darepublic 2y ago

When institutions use simple rules to respond to change and rigidly follow them without due judgement, then some will fall through the cracks, and others will grift off them

caseyy 2y ago

We blasted through the “you won’t always have AI in your pocket” phase in a blink of an eye. Local LLMs were running on smartphones before the world got to terms of LLMs being used everywhere. It’s one of many examples of exponential technological advancement.

caseyy 2y ago

The old colloquium exam format reigns supreme again. And that is fantastic. We shouldn’t reserve it for only “most important” occasions because quality education is important enough by itself.

naming_the_user 2y ago

Nonsense. You are in a room with a sheet of paper and a pen. Go. You’re acting as if 2010 was a hundred years ago.

bonoboTP 2y ago

Learning takes time. And the fully trained/educated/skilled/expert human performance is higher than AI performance. But AI performance may be higher than intermediate human performance after 1 or 2 semesters. But you need to reach intermediate performance first in order to later reach expert performance. During that time you still need a learning "slope", you need to be tested on your knowledge at that level. If you're given the AI at the outset, you will not develop the skill to surpass the AI performance. Calculators are just one analogy, there is no guarantee it will work out that way. It's just as likely that this over-technologization of the classroom will go the way of whole-language reading education.

yojo 2y ago

I won’t claim this is by design, but at the very least a side effect of writing term papers is getting practice at organizing your thoughts and drawing conclusions from them. While writing term papers is a skill that is only minimally useful in the real world (save for grant writers and post docs, pretty much), the patterns of thinking it encourages are valuable to everything that isn’t ditch digging. Maybe we can outsource this part of our cognition to AI, but I’m skeptical of the wisdom of doing so. Are we all going to break to consult ChatGPT in strategy meetings?

dogleash 2y ago

> we could never rely on having a calculator when we need it most—obviously there's irony associated with having 'calculators' in our pockets 24/7 now That was just a simple quip to shut down student bellyaching. Even before we had pocket calculators, it was never a strong answer. It just had to hold over long enough so when you realized it was bad answer you weren't that teacher's problem anymore. The actual answer was that they're complaining about a minor inconvenience designed for reinforcement, and if they really did need a calculator for the arithmetic on a test designed deliberately designed to be taken without a calculator, then they don't belong in that class.

kjkjadksj 2y ago

We used to write essays in class on blue books. That can still be done today.