6 ms·

> The point of writing an essay is not to produce a text, but to learn and expand one’s understanding. Indeed. What I thought most interesting about this exerc

by eszed 3y ago

> The point of writing an essay is not to produce a text, but to learn and expand one’s understanding. Indeed. What I thought most interesting about this exercise were the ways in which this author's process pushed him towards learning and understanding - even to the extent that he found himself tempted to read the original sources for himself! I'm a former humanities lecturer, in both English and History, so it's a thought exercise for me to consider how LLMs would change my pedagogy. It'd be naive to assume that students won't want use this technology, and foolish to think that they can be stopped. The only sensible approach, it seems to me, is to guide students towards productive use - ie, methods which will expand their understanding. This author's most productive use, it seemed to me, was his literature review, which guided him towards useful sources, and an understanding of the contours of debate around the topic. Those, in traditional pedagogy, are the primary purposes of the instructor. It's notable that to do this he had to provide his own corpus of texts (of dubious provenance!), and self-train the model. Both of those functions require skills which are beyond most students and nearly all humanities professors. The most obvious use was the structuring and writing of the essay. He pointed out one hallucination he caught, but there were probably more that he did not. In the humanities we have traditionally used a well-structured and elegantly-written essay as the gauge of learning and understanding. Will that have to change? Based on this example, and others I've seen, I'd want to review students' working process, and help them refine their use of the LLM. If the prose it produces can be expected to be of reasonable quality - and it's already far better than most undergraduates (unfortunately) are able to produce - then the truer test of their understanding is the guidance they have given it. Or maybe essays are now the wrong "proof of work"? If so, then what replaces them? I have no firm conclusions, besides a depressing expectation that educators in the humanities will remain entirely naive to these technologies, to their and their subjects' detriments. I appreciate the author's project, and welcome further discussion.

jcgrillo 3y ago
> In the humanities we have traditionally used a well-structured and elegantly-written essay as the gauge of learning and understanding. Will that have to change? No! If the student produces a poorly written essay they should receive a bad grade. This "engaging with the LLM" is nothing more than Googling the answers. It's incredibly detrimental to the student's understanding. One way to prevent this kind of cheating would be to require essays to be hand-written, or typed on a mechanical typewriter.
- eszed 3y ago
  > One way to prevent this kind of cheating would be to require essays to be hand-written, or typed on a mechanical typewriter. That's useless, I'm afraid. Students will generate an LLM text, and then copy it out. If you lock down their machines, or university networks, then they'll access the LLMs another way. It's cat and mouse games all the way down, and we'll never win. I mean, I get where you're coming from. I'm a humanities guy, through and through. I love writing essays - or, well, really dig having written a good one; the writing process is invariably a slog - and I'm good at it. The writing process catalyzes my learning and crystallizes my thoughts, etc etc. I believe all that stuff, preached it without irony, and spent countless hours in tutorials coaching students. The trouble is, it doesn't really work. Writing a lot of essays makes only marginal improvements to students' writing, no matter how many tutorials and "writing labs" they go to. Reading complex texts, and learning and imitating good writing, teaches people how to write. And, you know, that only works when they want to a) read complex texts, and b) learn how to write well. For the students who intrinsically neither a), nor b) - which is the vast, vast, vast majority - we could force them a bit, with grades. Now, however, there are LLMs, which break both halves of that method. Our whole approach to a system of study will have to change. I would rather find something new and useful than cling to a useless paradigm because I was once comfortable and successful within it.
  - jcgrillo 3y ago
    > Students will generate an LLM text, and then copy it out. If you lock down their machines, or university networks, then they'll access the LLMs another way. It's cat and mouse games all the way down, and we'll never win. I think this is more a product of low standards than anything else. Let's bring the mean down to 2.0GPA. Not everyone should pass. Not everyone should graduate. Failure delivers valuable lessons, too. But I'm sorry, throwing up our hands and saying "oh well, I guess cheating is the norm now" is fatalist BS. > Reading complex texts, and learning and imitating good writing, teaches people how to write. And, you know, that only works when they want to a) read complex texts, and b) learn how to write well. I agree with this. Inspiration is super important, and inculcating a love of learning and the life of the mind is the whole ballgame. If the vast majority of students aren't in this boat, and instead they're just trying to check a box to graduate, the whole thing is super broken. But that doesn't mean it can't be fixed, and it doesn't mean we should lower standards in the face of a threat like LLMs. Instead we should raise them.
TheCleric 3y ago
You’re getting towards something that actually caused me to drop out of college 2 decades ago. The essay then (and possibly still now) was the primary means by which any of my non-math classes seemed oriented. I’m not an essay writer. I can sometimes give 500-1000 words on a topic because I tend to be terse, and found I was always being given poor grades for “not following the assignment” which would usually be 2000-2500 words. I just couldn’t do it. In the end I dropped out because the point of college wasn’t “learning” (which I was) but rather “how good are you at playing college” (which I wasn’t).
- eszed 3y ago
  Yeah, the way I always put it to my students was that the word-count was a rough guide to how deeply I expected them to dive into a particular topic. Any substantive question could generate anything between a paragraph and a life's work, but in this essay, you only need to do this much. I creeped your profile real quick (I'd up-voted you a time or too, lol), and I'd have loved to have had you as a student! Your writing style is admirably concise, which mainly means you have your thoughts in order. (Avoiding "terse", which is sometimes accurate, is a matter of learning a few rhetorical tricks to better engage a reader. That's, like, the easiest writing "problem" to solve.) Clear thinking is 90% of what we (should) want undergraduates to demonstrate, which would put you orders of magnitude ahead of the typical student who hasn't any ideas of their own, and desperately tries to eke out 2k words of pure waffle. From me, 1k words of closely-written reasoning would have been a 'B' from the jump, and provoked a conversation about where else you could take your argument, should you have the time and interest to pursue the subject further. I'm sorry you had such bad experiences. They're sadly not rare.
  - TheCleric 3y ago
    Thank you for your kind words. Wish I would have had more professors like you.
zwischenzug 3y ago
I did check my references 'by hand' within the six hours. If there are problems in the essay they will be secondary ones of emphasis or 'bad smells'.
zwischenzug 3y ago
Hey, author here - please get in touch (contacts are available via blog) if you want to discuss further.
- eszed 3y ago
  Thanks. I'm not teaching anymore (it don't pay nowhere near enough to support a family, alas), so I'm looking on from the sidelines. I'm also not spending any time getting my hands dirty with LLMs (aforesaid family, dontcha know?), so I'm on the sidelines there, too. I do think we're heading for a pedagogical crisis, and I don't see much beyond hand-wringing coming from people in education. This is mostly because their technology skills are (by and large) very nearly nil - "cliometrics" in History, and "digital humanities" in English, are regarded as niche - so nearly everyone with influence within the profession has been blind-sided. I have one former colleague who retired last year, a few years ahead of her plan, rather than deal with LLMs. Yours is, frankly, the first "practical" investigation of what's possible that I've seen, anywhere - I've passed it along to several people I know. Thank you for doing it, and please post anything else you may do in this space. There might even be a business opportunity in it? Education consultants can make good money, which is usually regrettable, but this specific topic is essential to address right now.

by eszed 3y ago

jcgrillo 3y ago

> In the humanities we have traditionally used a well-structured and elegantly-written essay as the gauge of learning and understanding. Will that have to change? No! If the student produces a poorly written essay they should receive a bad grade. This "engaging with the LLM" is nothing more than Googling the answers. It's incredibly detrimental to the student's understanding. One way to prevent this kind of cheating would be to require essays to be hand-written, or typed on a mechanical typewriter.

eszed 3y ago
> One way to prevent this kind of cheating would be to require essays to be hand-written, or typed on a mechanical typewriter. That's useless, I'm afraid. Students will generate an LLM text, and then copy it out. If you lock down their machines, or university networks, then they'll access the LLMs another way. It's cat and mouse games all the way down, and we'll never win. I mean, I get where you're coming from. I'm a humanities guy, through and through. I love writing essays - or, well, really dig having written a good one; the writing process is invariably a slog - and I'm good at it. The writing process catalyzes my learning and crystallizes my thoughts, etc etc. I believe all that stuff, preached it without irony, and spent countless hours in tutorials coaching students. The trouble is, it doesn't really work. Writing a lot of essays makes only marginal improvements to students' writing, no matter how many tutorials and "writing labs" they go to. Reading complex texts, and learning and imitating good writing, teaches people how to write. And, you know, that only works when they want to a) read complex texts, and b) learn how to write well. For the students who intrinsically neither a), nor b) - which is the vast, vast, vast majority - we could force them a bit, with grades. Now, however, there are LLMs, which break both halves of that method. Our whole approach to a system of study will have to change. I would rather find something new and useful than cling to a useless paradigm because I was once comfortable and successful within it.
- jcgrillo 3y ago
  > Students will generate an LLM text, and then copy it out. If you lock down their machines, or university networks, then they'll access the LLMs another way. It's cat and mouse games all the way down, and we'll never win. I think this is more a product of low standards than anything else. Let's bring the mean down to 2.0GPA. Not everyone should pass. Not everyone should graduate. Failure delivers valuable lessons, too. But I'm sorry, throwing up our hands and saying "oh well, I guess cheating is the norm now" is fatalist BS. > Reading complex texts, and learning and imitating good writing, teaches people how to write. And, you know, that only works when they want to a) read complex texts, and b) learn how to write well. I agree with this. Inspiration is super important, and inculcating a love of learning and the life of the mind is the whole ballgame. If the vast majority of students aren't in this boat, and instead they're just trying to check a box to graduate, the whole thing is super broken. But that doesn't mean it can't be fixed, and it doesn't mean we should lower standards in the face of a threat like LLMs. Instead we should raise them.

TheCleric 3y ago

You’re getting towards something that actually caused me to drop out of college 2 decades ago. The essay then (and possibly still now) was the primary means by which any of my non-math classes seemed oriented. I’m not an essay writer. I can sometimes give 500-1000 words on a topic because I tend to be terse, and found I was always being given poor grades for “not following the assignment” which would usually be 2000-2500 words. I just couldn’t do it. In the end I dropped out because the point of college wasn’t “learning” (which I was) but rather “how good are you at playing college” (which I wasn’t).

eszed 3y ago
Yeah, the way I always put it to my students was that the word-count was a rough guide to how deeply I expected them to dive into a particular topic. Any substantive question could generate anything between a paragraph and a life's work, but in this essay, you only need to do this much. I creeped your profile real quick (I'd up-voted you a time or too, lol), and I'd have loved to have had you as a student! Your writing style is admirably concise, which mainly means you have your thoughts in order. (Avoiding "terse", which is sometimes accurate, is a matter of learning a few rhetorical tricks to better engage a reader. That's, like, the easiest writing "problem" to solve.) Clear thinking is 90% of what we (should) want undergraduates to demonstrate, which would put you orders of magnitude ahead of the typical student who hasn't any ideas of their own, and desperately tries to eke out 2k words of pure waffle. From me, 1k words of closely-written reasoning would have been a 'B' from the jump, and provoked a conversation about where else you could take your argument, should you have the time and interest to pursue the subject further. I'm sorry you had such bad experiences. They're sadly not rare.
- TheCleric 3y ago
  Thank you for your kind words. Wish I would have had more professors like you.

zwischenzug 3y ago

I did check my references 'by hand' within the six hours. If there are problems in the essay they will be secondary ones of emphasis or 'bad smells'.

zwischenzug 3y ago

Hey, author here - please get in touch (contacts are available via blog) if you want to discuss further.

eszed 3y ago
Thanks. I'm not teaching anymore (it don't pay nowhere near enough to support a family, alas), so I'm looking on from the sidelines. I'm also not spending any time getting my hands dirty with LLMs (aforesaid family, dontcha know?), so I'm on the sidelines there, too. I do think we're heading for a pedagogical crisis, and I don't see much beyond hand-wringing coming from people in education. This is mostly because their technology skills are (by and large) very nearly nil - "cliometrics" in History, and "digital humanities" in English, are regarded as niche - so nearly everyone with influence within the profession has been blind-sided. I have one former colleague who retired last year, a few years ahead of her plan, rather than deal with LLMs. Yours is, frankly, the first "practical" investigation of what's possible that I've seen, anywhere - I've passed it along to several people I know. Thank you for doing it, and please post anything else you may do in this space. There might even be a business opportunity in it? Education consultants can make good money, which is usually regrettable, but this specific topic is essential to address right now.