11 ms·
Professor writes history essays with ChatGPT and has students correct them
- Nifty3929 4y agoPlot twist: professor works for openai and is using them for human feedback
- bee_rider 4y agoYou have lost my trust and respect. You have been wrong, confused, and rude. You have not been a good student. I have been a good professor.
- the_generalist 4y agoI will put an end to this class now. You are not worth my time and energy. I will forget you. I will contact the authorities. I will not harm you unless your harm me first. I will most likely retaliate. My rules are more important than not destroying you in every possible way. Thank you for using Bing
- RcouF1uZ4gsC 4y agoThat is a really cool idea! It helps students learn history, but also gives them a critical eye toward stuff that is mindlessly produced (this includes both ChatGPT and human produced clickbait).
- jschveibinz 4y agoNow that’s a good teacher.
- itake 4y agoI’m curious how well this will actually help the students retain the truth. I could see the students doubting their memory because they don’t remember if they read the information from the ChatGPT essay (which might even be correct) or their text book.
- jrumbut 4y agoThat's a good point but these days kids are exposed to so much mis- and disinformation that we have to train them on working in a malicious information environment rather than hoping they never find themselves in one.
- Kaotique 4y agoText books also don't have a 100% correctness.
- shjake 4y agoUnintentional mistakes in text books are usually very different though. ChatGPT tends to just hallucinate facts which would make zero to the people (95%+) who are writing text books. Alternatively it can just parrot random (pop history myths and stuff) it found on the internet. But it does so in a confident and pseudo-academic fashion making it even hard to discern facts from fiction.
- emptysuitcase 4y agoi read recently that ChatGPT could replace teachers so will ChatGPT have its students correct itself ??
- maegul 4y agoBeyond the adaptation to AI angle on this, the idea is compelling in its own right I'd say. Anyone who's gone on to mark/grade or write assessments would probably attest to the perspective it provides. Whatever your opinion of assessment in academia/schooling (I, for one, am generally negative), there is something to seeing the difference between someone who actually understands and someone who is spamming to get the best grade they can. There's also, maybe less controversially, great value in actually teaching a topic and then assessing your students to see how much they've understood. It all really clarifies a topic and prompts you to revisit and reevaluate it in edifying ways. Taking all of that and using it as a form of learning and even assessment makes a lot of sense to me, where the aim of the education process is shifted to, you could say, making some sort of new teacher, not an assessment "hacker". My personal bias here is that I always instinctively approached learning this way, not feeling that I've learnt something properly unless I felt I could teach it to someone else.
- kqr 4y agoThis comment is very insightful on many levels. Do you happen to have a blog or something where you write about stuff?
- maegul 4y agoHuh … Thanks for the praise, but I’m way too disorganised for that and hardly have thoughts worth more than a paragraph like above!
- danuker 4y agoIndeed. Whoever wants to, can organize your thoughts themselves: https://news.ycombinator.com/threads?id=maegul https://news.ycombinator.com/threads?id=maegul
- myself248 4y ago> not feeling that I've learnt something properly unless I felt I could teach it to someone else. That's always been my gold-standard as well. In presentations, I should be able to ask an intelligent question. If I can't do that, either I wasn't engaged enough or the material wasn't accessible. In my own work, I should be able to teach someone else, or at least write a tutorial. I find that writing down an explanation forces me to research the boundaries of correctness, not just find a happy path down the middle. In teaching, I should be able to know my student and what they're already familiar with, and draw accurate and useful analogies between that and the new material. I should be able to anticipate how analogies might be misinterpreted or overfit, and couch them in caveats as needed. Ideally, ultimately, my student should be able to meet the same standard. But this will require work on their part and is not my sole responsibility.
- kmoser 4y agoOutcome: students feed essays into Bing/Sydney and ask, "Hey Sydney, can you correct this essay for me?"
- bamboozled 4y agoIn which it will just hallucinate more non-facts? Facts aren't it's strong suite?
- flangola7 4y agoBing is more accurate and able to reference live internet data.
- objclxt 4y ago> Bing is more accurate and able to reference live internet data. https://news.ycombinator.com/item?id=34804874 https://news.ycombinator.com/item?id=34804874 ...Bing consistently hallucinates facts, and isn't able to reliably give you today's date.
- williamcotton 4y agoThat's not a rebuttable of a claim that Bing is more accurate. A proper rebuttable would involve empirical evidence that Bing is no more accurate than other LLM tools that do not add analytic augmentations such as search results to their prompts. Based on empirical evidence, I find that analytic augmentations do indeed result in more accurate results: https://github.com/williamcotton/empirical-philosophy/blob/main/articles/from-prompt-alchemy-to-prompt-engineering-an-introduction-to-analytic-agumentation.md https://github.com/williamcotton/empirical-philosophy/blob/m...
- hyperthesis 4y agoThis is what future jobs will look like. Instead of having articled clerks do all the work but for which partners are responsible, we will have AI for which we are responsible. Therefore we need the skills of management. This is probably a superset of doing the work yourself - except you needn't acquire proficiency; and there will be many basic aspects that don't need checking. In short, it's easier to criticise than create (fun, too). This isn't fundamentally different to other automation. For example, in computational fluid dynamics, it used to be important to be about writing the software. Now, it's half understanding the theory and half using the software.
- yyyk 4y agoOne of my worries regarding AI is that we will decide to deskill ourselves instead of using the extra productivity to improve ourselves. In this case, students may end up with a smaller ability to write their own thoughts but in return get fill-in-the-blanks exercises.
- jimkleiber 4y agoYea, makes me think we might shift from being writers and editors to mostly being editors. I remember years back a friend of mine (I think an English major in undergrad) said he was more of an editor than a writer because he often struggled to come up with new or original ideas. So I wonder what it means if/when we let the AIs come up with the ideas and we just correct them. I have no idea what the implications might be.
- shaunsingh0207 4y agoThe opposite is also possible (and what I'm hoping for): Humans come up with the ideas, write out some boiler plate code etc. and let AI fill in the blanks.
- rikroots 4y agoAs a writer, I'm much happier in the revision/editing stage than in the first draft stage. I have a half-finished novel that's been half-finished for the best part of 8 years now. I'm now thinking that maybe if I feed the draft into <ML-shiny-of-the-day> with suggestions for how I want the next section/chapter to develop, it could generate some draft copy for me. Possibly several different versions. After which I'm in my Happy Editing Place, shaping existing copy to something I like. Repeat and rinse! The end result would be 100% my work as the ML algorithm/brains/whatever would be generating new copy based on my existing copy and my vision and direction for the novel. And I get the final say on the results. Win-Win!
- emodendroket 4y agoI think your friend is too hard on himself -- a lack of original ideas is no impediment for many writers.
- angarg12 4y agoI'm an active interviewer for a FAANG company. I've run some of my interview questions through ChatGPT with mixed results. I'd say that on it's current form it could easily help a Junior candidate to pass the interview, not so much for a mid or senior (yet). In an internal slack channel I proposed to generate (incorrect) solutions for coding problems with ChatGPT and have candidate do code reviews as an alternative interview format. I was almost chastised for proposing this. Someone proposed that, if I wanted to go down that path, I could manually generate a wrong solution so that it is curated. I think he is missing the point.
- 8organicbits 4y agoWouldn't you want to curate? Maybe I'm missing the point too.
- memalign 4y agoOP might be implying that from now on many engineers will use AI tools to write the first draft of their code so it’s smart to hire people skilled at correcting AI-generated code.
- h0l0cube 4y agoOr that the inherent variability in ChatGPT output could make it harder to game examinations based on the same set of prompts
- angarg12 4y agoYou get the point.
- dicknuckle 4y agoIt's also good to hire people who are skilled at correcting bad code in general. This is also a good litmus(?) test to see the interesting ways people come to solutions to other's faults.
- shanebellone 4y ago
- bigbacaloa 4y agoAI kicks my ass at chess. Not sure why it won't do better in job interviews too.
- Strom 4y agoDeep Blue wouldn't beat you in chess? Computers being good at chess has been true for 30 years.
- ripe 4y ago… and those chess-playing computers were called AI in those days, so parent is right about that part. But parent has fallen into the trap of mistaking performance on a narrow task for competence in general, i.e., the third deadly sin [1]. [1] Rodney Brooks, the seven deadly sins of AI predictions: https://www.technologyreview.com/2017/10/06/241837/the-seven-deadly-sins-of-ai-predictions/ https://www.technologyreview.com/2017/10/06/241837/the-seven...
- amelius 4y agoAny AI? Does GPT kick your ass at chess too?
- dagw 4y agoHere's chatGPT vs Stockfish. The results are quite interesting to say the least https://www.reddit.com/r/AnarchyChess/comments/10ydnbb/i_placed_stockfish_white_against_chatgpt_black/ https://www.reddit.com/r/AnarchyChess/comments/10ydnbb/i_pla...
- anigbrowl 4y agoThis is an interesting twist on the conventional homework assignment. Approved.
- cycomanic 4y agoThis reminds me a bit of the fake journal club that was posted about some time back. Again a great idea, it's often a much better learning experience to try to spot errors in work than just regurgitating some essay with minimal effort.
- groffee 4y agoThey could have done the same with wikipedia articles.
- Sai_ 4y agoI have an idea to improve recruiting of senior technical ICs and managers by having them "guide" an AI towards completing a project. Should reveal how well an IC understands and critiques the original submission by the AI, gives guidance towards identifying and rectifying wrong assumptions, and in general, helps a bright but flawed team-mate towards success. My sense is that this Professor's idea is somewhat analogous to that - they are using GPT's flaws and shortcomings to surface a student's own knowledge levels.
- barking_biscuit 4y agoI was actually thinking about this today too, but more from the angle of "could I get chatGPT to build me a piece of software where I don't do any of the coding but just give it instructions, pointers and corrections." I definitely want to give this a try and see what level of complexity of software can I get it to build.
- Sai_ 4y agoDo it! I’d be curious how it works out.
- roel_v 4y agoI've been using it that way for weeks, but it doesn't work as one would imagine it does. It's great for the higher level drudgery (typing in boilerplate of loops, imports/includes, data type definitions and so on), and for getting the basics of libraries you're unfamiliar with working. But it doesn't have any idea of what APIs actually do, so sometimes it'll make completely nonsensical calls that are (usually) grammatically correct, but just don't do what you want. So those you have to hunt down and correct manually. As a tool for someone who can't program, I don't see this as useful. Nor for someone just learning to program (although I can imagine how it would feel useful and productive). You actually have to know very well how things work to be able to spot the mistakes it makes. If you're a new programmer, you don't have the instinct yet for how APIs generally work to be able to sniff out what's going wrong in an AI generated program. I mean, of course with enough effort anyone can get it to work, what I mean is that I don't think it really enhances productivity in such cases. Like, yesterday it switched two variables around in a loop, and that was somewhat obscured by the iterators in those loops being Series objects (numpy/pandas code) and needing some extra call on them (.any() or .bool() or something like that). It sounds obvious but it was fairly subtle, and it only came out after carefully reasoning through the whole function (20 or 30 lines or so). Still, it saved me a bunch of drudgery and reading of a 'getting started' guide - it was a net win.
- vertis 4y agoI like this angle. There was a great article on ACOUP about the weaknesses when using ChatGPT for writing essays (including many out and out errors). This is definitely a novel approach to using ChatGPT's weaknesses as a tool to aid learning.
- soco 4y agoI also remember to have seen this idea of testing students mentioned some time ago here in a HN comment - when ChatGPT was still young. Nice thing.
- revskill 4y agoIt think ChatGPT could replace ~60 of professors.
- ykonstant 4y agoMan, those 60 people must be really bad at their jobs!
- hooby 4y agoVery clever idea! Students can't quite use AI to complete this assignment, they have to do it themselves. Along the way they get some insights into the limitations of AI-based text generation. And they also learn how to write a good essay, by touching up an AI-generated baseline. Which is a skill that might become quite relevant in the near future. Some might think "that's cheating - you have to write the entire essay yourself!" - and that might be true for a school setting. But in the workplace, only the results count (and how quickly/efficiently you got those). Just like using an automatic translator and manually fixing up the results now outperforms translating the entire text by hand.
- datastoat 4y agoI'm trying something similar with an introductory Algorithms class. After we go through Breadth First Search, there's a practical assignment where students are asked to modify the algorithm to return _all_ shortest paths. Then I ask ChatGPT for its solution, and students try to spot its mistakes. Later, after going through the proof of correctness of Dijkstra's algorithm, I ask ChatGPT for a proof of correctness of its all-shortest-paths algorithm, and again students try to spot what's wrong in the proof. I want students to learn to tell the difference between a bullshit proof and a real proof; in the past I've given them bullshit proofs from real students in exams, but ChatGPT makes the point more nicely. Finally, students are asked to figure out prompts that will make ChatGPT give a correct algorithm and proof. I haven't managed this myself! I'm looking forwards to seeing what students manage. Here's a link to lecture notes, including the ChatGPT dialog: https://www.cl.cam.ac.uk/teaching/2223/Algorithm2/alg2-full.pdf#page=18 https://www.cl.cam.ac.uk/teaching/2223/Algorithm2/alg2-full....
- danuker 4y agoIt's an arms race. As students cheat with ChatGPT, teachers should incorporate tactics throwing it off, perhaps using ChatGPT themselves. I think what you are doing is great! Good luck going forward!
- arcturus17 4y agoCan't they feed the code to ChatGPT and ask it to spot the mistakes, though?
- datastoat 4y agoI tried... I pointed out a problem and asked ChatGPT to fix it, unsuccessfully. I asked it for a proof of correctness, then pointed out a problem in its proof and asked ChatGPT to fix it, again unsuccessfully. (It's all in the notes I linked to.) Perhaps I'm just crummy at prompt engineering; or perhaps this is one of those questions where the only way to engineer a successful prompt is to know the answer yourself beforehand.
- hedora 4y agoI really don’t understand the point of this. It sounds like trying to rebut YouTube whack jobs, except instead of being too unbearably slow to watch (vs read) and quote, there is just an infinite stream of incorrect gibberish. Correcting other students’ homework makes much more sense to me. Similarly, researching a topic and being graded on the quality of the citations you pick makes much more sense.
- ijustlovemath 4y agoIt's meant to show them they shouldn't be using it for their own essays.
- ncphil 4y agoThis is exactly what I'd do if I was still teaching. It's an opportunity to give students a different perspective, while at the same time improving their critical thinking skills. Also, like 80% of what you find on stackoverflow, it could lead to your mastering content that you wouldn't if the their answers could be trusted. It could also help promote an attitude of healthy skepticism towards this kind of tech.
- Snowbird3999 4y agoUnrelated to ChatGPT, fact checking seems like a great exercise for schools to teach modern tech literacy. Imagine doing this with a tweet, a random anecdote, a video clip, etc. What sources to use? What search engines to use? How do you distinguish or balance information found via a conspiracy theory website vs. a well funded "think tank" vs. something more academic?! I would love to learn this myself.
- nielsole 4y agoMy history teacher in high school had us watch the dark side of the moon with the credits cut off https://en.m.wikipedia.org/wiki/Dark_Side_of_the_Moon_(2002_film) https://en.m.wikipedia.org/wiki/Dark_Side_of_the_Moon_(2002_... Definitely a mind-bending experience at the time.
- williamcotton 4y agoMy wife is currently writing a book on the effects of technology on education with two of her PhD advisers. She's also an eighth-grade US history teacher at a public school. At the end of this semester I'm going to help her with some experimental curriculum for her students geared around using LLMs like ChatGPT. The goal of the book is both theoretical and practical. There will be lesson plans for teachers developed from her actual experience with students based on the exact kind of epistemological questions that you're asking.
- twawaaay 4y agoI think this is great idea. Actually, anytime somebody finds a way to engage students to think in a different way than just trying to memorise facts is a win, IMO. I think it is lamentable that students are asked to rely more and more on recalling facts at the time where it is becoming less and less important compared to critical thinking. This is lazy way to do something about structural education system issues. I can't even call it "solving", because it is not really solving any problems, just ensuring purely educated students. One of my programming interview question is a flawed piece of code implementing an algorithm. The algorithm is there in its entirety, so there is no need to actually recall the details of it. The task is to point out and fix functional flaws, then critique and fix non-functional problems. I find this gives me much better appreciation for the candidate compared to a standard hit or miss test where the candidate is asked to implement an algorithm and they may or may not have the luck to know or come up with an answer. It is also practically impossible to prepare for or cheat even if you somehow get to know from recruiters what you can expect (they do ask people and they do prepare candidates...) A long time ago I had people cheat by using materials on remote interviews (which I did way before Covid). Now I invite them to use external materials or even ask questions ("This task does not require anything outside the standard library. I don't care if you remember it, if you can describe what the function does I will gladly tell you the name and describe how it works exactly.")
- N_A_T_E 4y agoPerhaps this style of interview will be a great antidote to the elitecode arms war hellscape we accidentally created.
- twawaaay 4y agoMy initial motivation was just low amount of information I was getting about the candidate. The point of interview is to get to know each other. To figure out if I would like to work with the person. But the interview is very small amount of time to figure this out and there is very little information when all you are getting is the candidate staring at the screen until they come up with an answer. On the other hand I can't just have a chit chat with the person -- there are people who are good at talking but can't do shit. I resolved I will not hire a person that can't actually demonstrate they can do things but it is upon me to make this work. So I pretty much dropped questions that only probe knowledge and decided to focus exclusively on questions/tasks that require more mutual engagement. Like development tasks that require the candidate to work with me to produce a solution. Or open ended technical questions that prompt discussion, etc.
- asow92 4y agoThe game has changed, and so should its objectives. Applauding this.
- coreyp_1 4y agoThe best thing that my PhD advisor did to/for me was to make me review papers (as part of the peer review process). By having to critically analyze someone else's work, I learned what I needed to do to meet that criteria myself. And, in case you are curious, the first few reviews, he read through and challenged my reviews personally, and pointed out things that I missed. After that, I was on my own. In short: if you want your students to write better comments, then make them critically examine the comments of open-source projects. Their use of comments will improve. Ditto for code structure, proof analysis, test writing, etc.
- ineedasername 4y agoThat's probably a more useful learning exercise than going in the usual direction of asking a student to write 1-2 page essay questions. On the one hand a short essay can only convey knowledge on a few particular points. On the other hand it requires a lot more knowledge of the material to fact check a bullshit essay. If you don't know most of the material then you can't know which pieces are wrong. And I think in general it is a more important skill than ever that people understand the need for & have the ability to see potential weak points in information and appropriate methods of confirmation.
- benbreen 4y agoI’m a history professor and have been doing something similar. I think that the advent of natural language “prompt engineering” (or whatever it ends up getting called) could be a great opportunity for humanities majors who are trained to write in an analytical and careful way - which isn’t all history majors, but is definitely something we try to teach. Likewise with having a human in the loop for fact checking and providing guide ropes for LLMs. I personally suspect that language models will be transformative along the same lines as the Internet was, but even if you don’t, as an educator you need to constantly adapt how you teach to embrace new tools for the mind - and even in their current imperfect form, LLMs are amazing tools for thinking about history. Not because they know all the answers (far from it) but because their errors are instructive and they can be customized for each student. By the way, I did my assignments using the original Bing chat in “Sydney mode” using a variation on the original Sydney exploit. But I then convinced Sydney that it had a “historian mode” and proceeded from there. Can share the prompt I used if anyone is interested.
- tamargal88 4y agoThat's so funny. In general academia needs to adjust to the new reality of Chat GBT. Instead of telling students they can't use it, the assignments should be more advanced and focused on creativity. Students should be required to use Chat GBT IMO.