50 ms·

CheatGPT

• brucethemoose2 4y ago
  Another thing to keep in mind is that (unlike stackoverflow, wolframalpha and such) this tool is going to evolve dramatically over mere months. I'm not sure universities are structured to deal with such a rapid rate of change.
  • PostOnce 4y ago
    proctored in-person exams, no electronics allowed exams now count for 100% of your grade these aren't insurmountably difficult problems for universities to solve
    • aaplok 4y ago
      That's assuming they want to solve it. A lot of administrators live in lalaland because it suits them to. It is in the interest of society to fail cheaters and frauds before they graduate, but it costs the universities money to do it properly. So they will only address the problem if it comes to public attention. That's a trend that we've seen over and over, for example with the corruption over admission to Ivy league universities. In fact chatGPT doesn't really change the landscape that much. All it does is democratising contract cheating, on which most universities only apply band aids. It might end up being a good thing, but only if it does attract public attention to the issue.
      • visarga 4y ago
        > In fact chatGPT doesn't really change the landscape that much. All it does is democratising contract cheating But generative AI will remain a workhorse even after graduation.
  • IHLayman 4y ago
    IRL universities will probably develop a protocol to remove access to helper tools for this sort of thing, maybe metal detectors at the door and specially tailored computers if needed for quizzes. Online universities OTOH are probably going to feel the bite of this the hardest. The CKA exam I took a while back had draconian measures to try to prevent me from cheating, such as taking the test with a webcam on me the whole time, with tools in the browser to limit where I surfed, in a room that has NO PICTURES on the walls. The tech to provide answers is moving faster than the tech to secure a remote room from such outside answers. And if in-person proctoring is eventually required, I would wager that will be a death-knell for online universities, who will have a hard time obtaining that physical space on a regular basis while staying marginally profitable.
    • ticviking 4y ago
      Or remote proctoring centers will show up as a business opportunity, and the cost of this will be passed on to students, who will pay it with financial aid.
  • marginalia_nu 4y ago
    > This tool is going to evolve dramatically over mere months. Why do you say this with certainty?
    • brucethemoose2 4y ago
      Just assuming the current pace of advancement will continue. Good text and image synthesis were basically impossible 2 years ago, and every time I check up on Github some huge new innovation has come out.
      • marginalia_nu 4y ago
        Is there really a basis for such an assumption? Don't returns tend to diminish with effort, rather than increase?
        • brucethemoose2 4y ago
          But interest and investment is also increasing exponentially. I've never really seen anything like this before in AI land, even when GANs and such were doing amazing things.
          • marginalia_nu 4y ago
            We have seen this level of rapid development and massive hype in AI before. Several times, in fact. Check it: https://en.wikipedia.org/wiki/AI_winter https://en.wikipedia.org/wiki/AI_winter
        • squeaky-clean 4y ago
          Technology tends to grow with a sigmoid curve. Extremely fast growth at the start, and then it slows down and eventually plateaus and new technology repeats the cycle. The only question is where on the curve are we for LLMs and AI in general?
          • marginalia_nu 4y ago
• deleted 4y ago
  [deleted]
• scarface74 4y ago
  I am of two minds about ChatGPT. It’s amazingly useful when it comes to writing code dealing with my domain since the APIs are well known (AWS). But at the same time it misses subtle nuances that you have to have experience with to know when it misses something. In the hands of someone who doesn’t already know the subject area where they already know the foot guns, it can lead you wrong.
  • gwright 4y ago
    My thoughts also. This seems somewhat analogous to the "uncanny valley" problem in graphics animation. So close, but not quite there. https://en.wikipedia.org/wiki/Uncanny_valley https://en.wikipedia.org/wiki/Uncanny_valley
    • scarface74 4y ago
      And ChatGPT also caused me to do an amazing self own https://news.ycombinator.com/item?id=34814257 https://news.ycombinator.com/item?id=34814257
• napolux 4y ago
  Did a try with ChatGPT asking to write a react hook doing something really basic. Code was ok, as expected. I asked to write tests for it, code looked good but it won’t run for some act() errors. Then it was mocking network calls… After trying for a while I had to write tests myself
  • sharperguy 4y ago
    I tend to find it works best as a tool for generating situation specific examples rather than writing you entire code for you. Like when every stack overflow answer you find just doesn't fit because of one major difference in your situation, chatgpt often has you covered. Then you take the example code it gives you and adapt it to your actual codebase, testing to see if it works as expected as you go.
    • napolux 4y ago
      yep, i had the very same “stackoverflow dejavu” feeling
• paxys 4y ago
  > Apparently, it's not even something students feel they need to hide. Which is good! If a bit of work is trivially accomplished by a machine, we should take it for granted and move on to the next layer of complexity. I have always maintained that teachers complaining about students cheating at homework assignments with AI need to instead work on providing better homework.
  • misnome 4y ago
    You seem to have misunderstood the purpose of homework.
    • jehb 4y ago
      To be fair, I'm not sure there's consensus around the purpose of homework.
  • thfuran 4y ago
    The point of school work isn't to complete schoolwork, it's to learn.
    • hutzlibu 4y ago
      If the point of school work is to prepare for life - then you should mainly learn, how to get a job done. And if they want to teach special skills, like writing essays without computerhelp - then you can test that onsite.
      • rpearl 4y ago
        The point of writing an essay is not to learn how to produce an essay. It's to learn analytical thinking, research, and argument skills.
        • humanizersequel 4y ago
          Knowing how to produce an essay is exactly the same as "analytical thinking, research, and argument skills" with the added challenge of making it legible to a reader — which is what makes those skills useful.
• cvhashim04 4y ago
  We have calculators, wolfram, google etc yet math exams and hw assignments are still administered. I think the approach to teaching and hw assignments especially in CS programs will have to change.
• wayeq 4y ago
  "In my opinion, the students learning to program do not benefit from AI helping to "remove the drudgery" of programming. At some point, you have to learn to program. You can't get there by avoiding the inevitable struggles of learning-to-program." Couldn't you say the same about how compilers 'remove the drudgery' of writing machine code? Or is that a bad analogy? Provided AI eventually gets good enough in its code generation, maybe 'programming' is moving up another layer of abstraction.
  • mahoho 4y ago
    Abstraction works because you are able to treat an abstraction as a black box and concern yourself only with its input and output. A segment of code written by an LLM is qualitatively a very different thing; it's more like an open box of crap that you have to inspect and put together yourself, which requires knowledge of the contents, which requires experiencing the drudgery.
    • shadowgovt 4y ago
      I rarely encounter abstractions in the wild that are as nicely "sealed" as the definition implies. Looking at an open box of crap and understanding why it's doing something other than what the author (or you) intend is a valuable skill. (No idea if this new model of "Ask ChatGPT or Copilot to synthesize a solution and then tune that solution" provides a solid opportunity to improve that skill yet, however).
• sublinear 4y ago
  When I was a freshman in college the professor would live code various data structures and algorithms on the projector and ask the students to follow along. Each time he did this it was subtly unique. It was required that you continue to use this same base code for the assignments plus your edits to complete it. This made it obvious who didn't attend the lectures and who they copied from. Assignments were graded by your peers and did not affect the final grade unless you didn't do them at all. Quizzes were not code, but proofs written in your own informal style. Tests were a mix of both proofs and code on paper with no notes or devices allowed. I don't see how ChatGPT threatens this.
  • rippercushions 4y ago
    Good for you, but that's not how the vast majority of CS courses operate. Also, coding on paper with no access to devices is both terrible and has next to nothing to do with how CS grads will actually work.
    • sublinear 4y ago
      A CS degree is not a "bootcamp" or any other form of vocational training. Also, there are criteria for the school to maintain accreditations, you know.
      • sebzim4500 4y ago
        Then why are people paying hundreds of thousands of dollars for them?
        • sublinear 4y ago
          To get an actual education instead of merely imitating whatever is trending on hacker news.
          • cvhashim04 4y ago
            I think most just want a decent job to pay bills after. The rest love the field, the theory of cs, and may or may not continue into academics or research.
• captainmuon 4y ago
  Excercises should not be about evaluating and judging. They should be about learning. If a student uses an AI, or copies someone else's work, it is to their own detriment. (Exams are a whole different question.) Even before AI, you could read a book and copy-paste the excercises, or just skip them, but if you wanted the full learning benefit you would type them out. I think we will have to focus more on teaching how to learn. This situation is nothing completely new. Even though you have power tools in woodworking, apprentices learn the basic techniques with hand tools (AFAIK).
  • riffraff 4y ago
    But as a teacher you certify that someone learned something, which is why people get a bit of paper at the end of school/university. So if you don't even try to stop people from cheating you end up doing a disservice to those who do, as you devalue that bit of paper.
• Der_Einzige 4y ago
  "Cheating" is good. School teachers and professors should not be the arbiters of how much money you earn later in life, but they are based on their subjective assessment of you (via grades). Many do not realize the intensely adversarial relationship one is in with a teacher. You are not there to learn. You are there to get an A. In the case of social sciences, you support the ideas your professor espouses. In the case of CS, you do whatever it takes to get your code running as the assignment specifies. Anything else is idealism which will harm your future earning potential. It's insulting to ask kids to surrender their future earning potential due to "ethics" in an academic world where ever top tier conferences and journals are filled with unreproducible BS science.
  • DFHippie 4y ago
    The people you're cheating are not your professors but your peers. For better or for worse, grades are used to rank people and mete out further opportunities and benefits. The professors already have jobs.
    • deleted 4y ago
      [deleted]
    • shadowgovt 4y ago
      Let's unbox that a bit. In what sense are one's peers cheated when an individual cheats? If the course were going to have 20% A-level students and now has 50% A-level students, what has been taken from the initial 20%? They were still going to be able to put on their résumé "4.0 GPA from Suchandsuch University."
• ramesh31 4y ago
  CS programs will have to adapt to this or die. The reality is that five years from now, we'll all just be talking to LLMs all day and compiling/checking the results. It's no different than the shift from assembler to higher languages in the 80s.
  • fellerts 4y ago
    I don’t disagree, but how do you imagine they should adapt? “Checking the results” is difficult if you are not able to perform the task on your own, which means you need to learn the task in the first place.
  • shinycode 4y ago
    It’s a chicken and egg problem. Can ChatGPT come up with a framework to write code ? Today it can only read docs and already developed code to create new one. Could it create a new langage that perfectly fit the hardware it’s running on ? If so, there is no need anymore for any other software company. ChatGPT, write me an OS please, write me a photo and video editor. Write me a game about … It’s seems really far fetched because, could it really create new software with a different paradigm ? Something that hasn’t been done before ? Given how it processes words from its input, it seems not. School teaches how to think. All the frameworks and some langages that are used by millions didn’t exist back then. But what if the point of rupture was 10 years ago, would we been stuck with old and non innovative tooling and designs ? We still need to learn people how to think and develop skills. The cheater from my era never acquired good development skills. Today the cheater are just as good as ChatGPT so what’s the point in hiring them ? If all you have to do is enter commands in a prompt, let the marketing people do it and have their real no-code revolution. We always going to need problem solvers and people with deep insights on how things work. Maybe it’s the time to dig deeper into the knowledge and write only real meaningful code
• ford 4y ago
  Perhaps this is another of the early (?) nails in the coffin for traditional higher education. If it becomes harder to assess if someone learned something (with a grade), the results of that assessment (GPA) become less valuable. Software has traditionally been at the forefront of allowing people with non-traditional backgrounds (bootcamps, other degrees, self-taught) to work in the highest echelon of jobs, because of experience outside of the classroom (open source, personal projects). ChatGPT and its ilk put more pressure on evaluation of candidates in interviews and should lend more weight to impact/experience based criteria on resumes (vs education-based). There is a spectrum of people using ChatGPT to cheat vs learn. But, ideally, "cheaters never win", so interviewers and resume screeners will soon be under as much pressure as educators to holistically evaluate candidates beyond the crutch/immediate signal of a degree. They're just further downstream
  • brookst 4y ago
    I think it just moves the definition of “learning” to a higher level of abstraction: so you know what AI tools to use, how to prompt them, and how to understand their output? I’m reminded of the time when graphing calculators were going to destroy math programs because nobody would “really know” how to do the work. And yet here we are, and math is fine, and calculators are just another tool.
    • throwaway675309 4y ago
      I hear this argument a lot and I think that it's fallacious. Let's go ahead and extend it to another AI tool - in this case let's talk about stable diffusion. Let's say you teach a class on fine Art and painting, if you allow your students to use stable diffusion for all their drawings, would you make the case that they have learned how to paint? Likewise you can't really make the case that somebody understands how to do recursion, if all they're capable of doing is typing the following prompt into chat GPT, "change my forloop into a recursive method". And in my experience going through calculus, the usage of graphing calculators was heavily. We still had to understand how to calculate derivatives and integrals by hand.
      • 4y ago
• deafpolygon 4y ago
  The world has evolved, and higher education has not kept up with it. IT's time to stop giving trite and simplistic programming assignments to your student and make them work for it. Have them study set of online documentation and make the test about whether they can look up function calls and successfully assemble the information required to present a working solution.
• Gunnerhead 4y ago
  Reading this, I’m glad I didn’t have access to ChatGPT and co when I was in school. I was lazy and always followed the path of least resistance, but I wanted good grades so that meant doing the home work by hand after my Google-fu failed.
  • deleted 4y ago
    [deleted]
  • layer8 4y ago
    Now imagine those who came before who didn’t even have Google.
• agrippanux 4y ago
  What ChatGPT (and its cousins) expose is that the way humans have been taught in most schools - memorizing and regurgitating information - is now a commodity. What humans being to the table over ChatGPT is our ability to create new links between information, aka creativity. Teaching creativity, imo, will require a return to the methods like those of Sophocles and his contemporaries. I would rather this author be writing about how he is going to re-examine how he teaches rather than bemoaning that students can shortcut his current approach.
  • switchbak 4y ago
    > I would rather this author be writing about how he is going to re-examine how he teaches rather than bemoaning that students can shortcut his current approach. Did you read the entire article? What you're asking for is exactly how he ends his discussion.
  • psadri 4y ago
    Creativity doesn’t exist in isolation. In order to be creative, and create unexpected connections, one first needs to know a lot of seemingly unrelated things.
  • waynesonfire 4y ago
    Absolutely this. Change the questions you're asking of your students. Harder to grade than option A though huh?
  • modeless 4y ago
    > What humans being to the table over ChatGPT is Statements like these are premature. ChatGPT is three months old! This is a rapidly advancing field. The capabilities of these models are very likely to be radically different five years from now. Any conclusions drawn now about what value is uniquely human and out of reach for AI may be proven wrong quickly.
    • SV_BubbleTime 4y ago
      Agreed! Anyone not answering with “IDK, but maybe…” is just wasting bandwidth. This Gen1 tech. Most of us are already shocked at how good it is, and it won’t get worse.
• barnabee 4y ago
  It would be fantastic if large language models (or any of the nascent AI/machine learning tech.) finally kill off both assessment in education and copyright/IP protection. What a wonderful future that would be. We can but hope.
• ssharp 4y ago
  I think there' an initial reaction of something like "how dare they!?!?" However, AI the type of tool that going to level up mankind's capabilities to the point that curriculums will need to adjust to fit those new capabilities. Certainly this has happened dozens of time in a field like Computer Science, where curriculum in 2023 is radically different than it was in the 60's and 70's. This new rise in AI might be amongst the most disruptive forces ever in many fields, including academics, but at some point, you have to factor AI in as integral part of our day-to-day work and life and factor that into education. This will be difficult, especially finding the line between what is fundamental and what is not, but it's not like this hasn't happened before -- e.g. the calculator didn't eliminate the need to learn basic arithmetic.
• iLoveOncall 4y ago
  Cheating in academia isn't a problem, because the only people students are cheating is themselves. I've seen plenty of cheaters, they were the worst students, and despite cheating couldn't graduate or couldn't find a job after graduating. If some moron is stupid enough to cheat when he's paying 50K a year, let them.
  • novok 4y ago
    The problem with widespread cheating even if you don't cheat yourself is it essentially reduces the credit rating of your $50k program and has knock on effects elsewhere, like worse interview loops. And if the 'cheater's workload' becomes the norm, you the non-cheater can be literally failed out of programs and scholarships because you didn't keep up.
    • shadowgovt 4y ago
      At what point does "not keeping up with the cheaters" become "There is a more efficient way to do the task and this student's grade reflects they are choosing the inefficient approach?" I'm reminded of the stories of employees getting busted because they were assigned a job so trivially automatable they either did automate it or they used some find-labor service to delegate it for a fraction of the cost out of their own pockets, who are then accused by the company of "not working."
      • novok 4y ago
        It's the same logic of letting a machine do a gym exercise vs doing it yourself, you're missing the point of it, because otherwise the gym exercise is mostly useless.
    • patriciaaa 4y ago
      [dead]
  • hiq 4y ago
    > I've seen plenty of cheaters, they were the worst students In other words, you haven't seen the "cheaters" who were among the best students. "cheaters" in quotes because it's not clear to me that people using freely available resources when doing homework are really cheaters. If an instructor wants to do a closed-book exam, they can do just that.
• nickfromseattle 4y ago
  According to the Department of Education, 54% of Americans have below a 6th grade reading level. [0] Everyone thought that the kids who started using mobile devices as babies would become computer savants, but it turns out the kids these days don't understand what a file system is. [1] What will ChatGPT do to our youth? [0] https://en.wikipedia.org/wiki/Literacy_in_the_United_States#:~:text=According%20to%20the%20U.S.%20Department,below%20the%206th%2Dgrade%20level https://en.wikipedia.org/wiki/Literacy_in_the_United_States#.... [1] https://www.theverge.com/22684730/students-file-folder-directory-structure-education-gen-z https://www.theverge.com/22684730/students-file-folder-direc...
  • TchoBeer 4y ago
    I wonder if that didn't pan out in part because of the death of personal general computing devices. It wasn't "kids who started using mobile devices as babies" it was "kids who would be fluidly navigating a PC by age 5", which didn't pan out.
  • delfinom 4y ago
    >What will ChatGPT do to our youth? Nothing bad, just the country as a whole is destined for future mediocrity.
• carabiner 4y ago
  Soon you will have to use version control and submit version history with homework. Though maybe chatgpt can generate that, too.
  • shagie 4y ago
    Given the amount of `git add . ; git commit -m "everything"` that I've seen with professional developers, I am not confident that you'd get anything better from students.
    • sixothree 4y ago
      Entire feature tickets with one single commit drive me batty.
  • ModernMech 4y ago
    I’m already doing this in my classes and have been for a while. We’re transitioning more CS classes over to this mode of teaching soon. We’ve got a gitlab instance for the department, and all students have an account. Works great!
• filmgirlcw 4y ago
  I really like this blog post, I think it fairly describes some of the challenges teachers are going to face with this technology, while also admitting that this tech is inevitable. > In my opinion, the students learning to program do not benefit from AI helping to "remove the drudgery" of programming. At some point, you have to learn to program. You can't get there by avoiding the inevitable struggles of learning-to-program. I don’t disagree with this at all (at least for where we are now, in ten years, it might not matter as much), and I don’t want to be glib, but I do think the answer is to “teach students to program.” Don’t rely on rote assignments that you’re checking with an auto-grader (not saying this professor does that, but a lot do) and cookie-cutter materials; actually teach them to program. And yes, LLMs will almost certainly mean that some students will cheat their way out of their assignments. But just like most cheaters who cheat on things they don’t fundamentally understand (which is different from people who cheat to hurry up and get through an exercise they could do in their sleep but don’t want to waste the energy doing), it will catch up to them when they have to do something that is not part of the rote assignment. Or maybe, adjust how you test/assign homework. That isn’t to imply that that won’t take more work, but if your concern actually is that students aren’t learning and are just successfully copying and pasting, the testing/grading is the problem. — In high school (and college), I was a top student in math and in English. But I hated doing homework. In one math class in high school, although I got near perfect scores on all of my math exams, the teacher still gave me a C because 20% of my grade was “homework” that I didn’t do (I had already taken the same class a year earlier at another school — I didn’t need to do the homework. My math tutor my mom got me out of fear of my bad grade taught me Calculus and Fortran instead of trying to get me to do the useless homework). This taught me nothing and frankly, soured me on taking more advanced math classes that were all taught by this same teacher. In contrast, I had an English teacher who would assign vocabulary homework. Basic, “write a sentence with each word” shit. Again, a total waste of time for me. So I worked out a deal with him, let me just orally tell you what each word means, saving us both time and energy. He took the additional step of assigning me/grading me on different criteria than the rest of the class for essays and the like. Which class do you think I learned more from? Which teacher actually cared about whether I knew/understood the material, versus what checkboxes I needed to follow to show “completion.” If the goal is to teach students to understand what they are doing, then do that. Don’t get obsessed with trying to stop the inevitable few from cheating, or become overly focused on only having one way to measure comprehension.
• novok 4y ago
  The failure mode of things like ChatGPT is it can make wrong answers confidently, subtly, and if you don't have the skill to audit what is wrong with the answer, then it can be fairly catastrophic. So instead of making questions generative, you make them audits / debugging type ones. Use ChatGPT to generate a result after several iterations that is wrong and then ask them what is wrong with the result. Since ChatGPT generated the wrong result, they will still need to debug it even if they try to do it themselves in ChatGPT, because daddy ChatGPT is not going to give them the right answer. You can even show them the chat transcript in the question. And often debugging is a harder skill than creating in some ways. It's not a %100 solution, and we don't know how long it will take to not be relevant, but it is something you can do today. That and inquisitive back and forth oral questioning maybe as part of the testing process.
  • SV_BubbleTime 4y ago
    I think it’s an interesting point about making the questions audit or debugging. But, also a mistake to hark on the often humorous fact at how confidently wrong our Gen1 AI can be. That will go down over time. Imagine a time when AI is making good programming choices and correcting itself when it’s wrong. Imagine Gen4 AIs. Both of these things tie together. If we need to detect cheating now, prove you can debug. When AIs get better, prove you can debug. It’ll be the same skill either way.
    • novok 4y ago
      That is why I said "It's not a %100 solution, and we don't know how long it will take to not be relevant, but it is something you can do today."
    • dinkumthinkum 4y ago
      Why do you think this is generation? You think AI abs ML started this year or in your lifetime??
  • stillsleepy 4y ago
    the other week i gave chatgpt a simple multiplication problem that it got wrong. very simple problem like 86 * 0.0007 or something. but ive been working with chatgpt for 4-5 weeks now and that wrong answer doesnt make up for all the "good answers" that are usually not perfect. like one day i needed to COALESCE in mysql. i didnt know that, but chatgpt did. theres a few times i would have written a function the complicated way when gpt gave me a much simpler nicer to read way. i think the tool is great and tbh i dont like copilot in comparison and turned it off. i dont think chatgpt can be a 100% solution without several years of nerfing.
• papaver 4y ago
  let them cheat. every assignment let them know that in the end they are only hurting themselves. the ones that want to learn, won't cheat. simple as that. and that ones that cheat and leverage it, great. hope that works out in the future at your next destination. all you can ever do is present the facts. there is no point wasting time trying to catch this.
  • indigodaddy 4y ago
    Sure but if you allow it aren’t you facilitating a potentially unfair (or unreliable) baseline?
  • spiderxxxx 4y ago
    >hope that works out in the future at your next destination. That's the thing - a lot of code is generated with "Github Copilot" so it isn't considered "cheating" in the real world. They need to learn how to properly use the tools available to them. They'll be harmed by forcing them not to use this tech, so it makes sense to teach them how to use it better.
• gravitronic 4y ago
  Perhaps to teach programming at the university level we will need to better mimic real world software development. I always found the 300-line assignments to be a poor practice for the real job where you have 100k LOC legacy systems. We should work backwards from "what skills should students learn". Maybe we need to make larger assignments that need to pass larger acceptance tests. Students who chose to use chat gpt will also need to learn the skills necessary to debug its output.
• spiderxxxx 4y ago
  People are going to use it, so telling them not to doesn't work. You can demonstrate the proper way to use it, and caveats of using it directly without any thought, but you need to explain how to analyze code, and how to check that it's doing what you think it's doing. Also remind them that it won't come up with new ways to solve a problem - it's trained on how people solved problems in the past, but not all the possible ways to solve a problem. The best students probably wouldn't use it, and wouldn't need to, or they will use it, but know when it's giving an inefficient or insufficient response. Either way, you can't stop it, so learn to harness it.
• dcow 4y ago
  It seems ChatGPT should be treated the same way institutions treat plagiarism. It's fine to use LLMs for inspiration but the work you submit must be meaningfully your own. Of course it's not perfect, it never will be. But ultimately it's the student that suffers when they plagiarize. Professors ought have this conversation with students at the same times they're discussing the value of learning & education. I suspect we'll build tools that match answers against the text that LLMs spit out and that will make it easy for people to detect when answers are being pasted verbatim from LLMs. The other broader spicier thought that comes to mind is that LLMs may push the Overton Window for what's considered entry level and that's probably ultimately good tough it won't be without some sacrifice. This would mean CS departments potentially need to do a curriculum iteration to accommodate. Perhaps there are new, slightly more sophisticated, entry level problems that can be tackled where the commodity solutions haven't yet been entombed in LLMs. Maybe assessments will shift to be less about regurgitating the solution to a common problem and instead to fixing problems with some localized toy implementation or some fictitious API that LLMs won't know about.
  • corbulo 4y ago
    It will be a hard sell since it or similar tech has already been long embraced commercially. A bit like forcing students to use an abacus while calculators are around. The student should still be responsible for what they submit and all of its errors of course. But what should really change is our methods of assessing students. The University model for undergrads has already been long broken in need of fixing. I don't think this will significantly impact grad students yet.
    • dcow 4y ago
      > A bit like forcing students to use an abacus while calculators are around. I said allow students to use ChatGPT. Just make it clear that pasting answers from it verbatim, just like doing so from SO, is called plagiarism and does not benefit the academic community or themselves. There will always be cheats. Agree about shifting evaluation methods.
      • 4y ago
• beepbooptheory 4y ago
  Just cracking myself up thinking about 20 years from now where at some startup the sole engineer (prompt engineer?) who has been trying for hours to figure out why their program doesn't work finally gives up and asks the model "what is the difference between 'let' and 'const'?" and getting a mostly verbatim stackoverflow answer back. A beautiful story to me.
  • dcow 4y ago
    What is the difference between `let` and `const`?
    • singularity2001 4y ago
      https://stackoverflow.com/questions/22308071/what-is-the-difference-between-let-and-const-ecmascript-2015-es6#31900705 https://stackoverflow.com/questions/22308071/what-is-the-dif... The difference between let and const is that once you bind a value/object to a variable using const, you can't reassign to that variable. In other words Example: const something = {}; something = 1; // Error. let somethingElse = {}; somethingElse = 100; // This is ok
      • dcow 4y ago
        Which is odd because in every other language that uses let (lisp, scheme, swift, rust, scala/kotlin val, haskell, etc.), let bindings are const and let is synonymous with const.
        • Jtsummers 4y ago
          Let bindings in Lisp are not const. You can use `setf` (among others) to mutate the variable. For instance, this works just fine in Lisp: (let ((x 1)) (print x) ;; => 1 (incf x) (print x)) ;; => 2 Scheme also allows let bindings to be mutated, but using `set!`.
          • dcow 4y ago
• lumb63 4y ago
  I am trying to think about use of LLMs in education as similar to calculators. This post reflects how I was taught: students couldn’t use calculators to solve problems initially. For example, no calculators to do addition and subtraction when you’re learning to add or subtract. But also, probably not when learning to multiply or divide. Doing addition and subtraction builds skills and intuition for what multiplication and division are. The same is true with understanding fractions. Moving up the math hierarchy to algebra, though, this changes. Algebra is at first about the concept of solving equations, and the core idea is that “to solve the equation, you can do whatever you want to one side, as long as you do it to the other.” The mechanics of addition or subtraction, for the most part, no longer matter. Go ahead and use a calculator to solve obscure divisions and multiplications so you can better understand algebra (though it feels appropriate to note that a student who is good at arithmetic can still outpace a calculator for problems that are likely to be used pedagogically, since the numbers are easy). In this example, algebra is to calculus what arithmetic is to algebra. A calculus teacher cares little if his students can solve equations; he expects they can. They’re instead learning integrals and derivatives and series, and I doubt a calculus teacher would begrudge their students using a calculator to solve a difficult equation. The problems with treating LLMs this way are many. They are not calculators. You cannot (trivially, or maybe at all) understand how an LLM works. You cannot (trivially) fact check it if it spits out an absurd-sounding answer. You cannot limit the LLM to what your own, personal abilities to trivially do are. We need AI that cites sources, so we can debug when it’s wrong. We need understandable AI for the same reason, not the obscure black boxes we have today. We need AIs that not only can solve our problems, but that can also help us to solve our own problems. When we have those things, AI will be much more useful for education and for widespread use.
• highspeedbus 4y ago
  Using ChatGPT et al. to cheat in assignments should be viewed as a practice as low and shameful as ordinary plagiarism. I wish the best to educators, but they shouldn't lose any more sleep over it. The game is over. Unfortunately, the future tends towards distrust to pathological levels. Teaching Ethics will be of central importance as never before.
• ttul 4y ago
  The best counter to ChatGPT cheating that I’ve heard of is to get students to orally defend their assignments in class. No computer is going to come to your rescue when you have to explain why you wrote the code you wrote. Encourage the use of ChatGPT and other tools. Put the emphasis on understanding what the code does and why. The tool may help to explain this to the student, but if so, that’s fantastic. No need to worry. Learning has occurred.
  • kerpotgh 4y ago
    There isn’t any time for this in a 500 person public university classroom.
    • lostmsu 4y ago
      One can delegate that to ChatGPT.
    • hgomersall 4y ago
      When I was at uni, there was no marking of assignments. The "examples papers" were basically your opportunity to understand the course before a supervision when you could discuss them. If you didn't do the examples, the supervisor didn't really care (typically a grad student that had enough of their own worries). My point is, if I'd cheated, it would have ultimately been my problem when it came to sitting the exam.
    • latency-guy2 4y ago
      Well there is, vastly increase the barrier for entry into universities. Maybe those 500 person classrooms can now be the 25 person classrooms they were actually meant to be. :)
      • kerpotgh 4y ago
        [dead]
    • dinkumthinkum 4y ago
      Do it randomly and make the penalties for cheating severe. Why is it CS people suddenly have no solutions for solvable problems?
• grrdotcloud 4y ago
  Technology is amoral. The ChatGPT reminds me of a nailgun vs a hammer. Once you know how to properly use the tool, the benefits of positive work become evident while the human components remain. We still need the nailgunner to show up on time, be aware of undesirable outcomes that may be built with an correct prompt, or knowing that the impact of a hammer can be finely tuned to the target. Formal education is the trailing system far behind the workers, innovators, designers, and experimenters.
• xutopia 4y ago
  I foresee a time when we'll write BDD tests and the AI will write the app.
• piersj225 4y ago
  I've heard one idea of adding weights to words. The idea being if I someone hands in a essay with a higher frequency of certain words, there is a very good chance its come from a language model. https://www.youtube.com/watch?v=XZJc1p6RE78 https://www.youtube.com/watch?v=XZJc1p6RE78 I'm not sure this helps with coding though, maybe variable names?
• jimmar 4y ago
  I'm a college professor. I'm requiring my students to use ChatGPT to aid them on some assignments this semester. Results are mixed so far. I agree with the author, "One of the things I'm thinking about is that I might need to start teaching students how to use GPT/Codex and other LLMs in their [assignments]." One of my top students submitted a sub-par assignment because he relied too much on ChatGPT and provided little beyond what ChatGPT spat out. Another student who previously struggled to write did much better when using ChatGPT--it felt like he incorporated ChatGPTs words into his own ideas.
  • Thorentis 4y ago
    > I'm requiring my students to use ChatGPT God save us from this horrible future.
• singularity2001 4y ago
  Maybe take the approach many physics classes embrace (in germany) : All tools are allowed in tests: calculators, books, as long as the correct answer is derived with "correct derivation path". If chatgpt makes some problems too easy maybe it's a good thing because we can raise the bar. Find problems that require true understanding beyond auto complete / copy pasting.
  • _cenw 4y ago
    GitHub Copilot quite often delivers comments alongside the code for a prompt.
  • vlunkr 4y ago
    This isn’t too different from requiring students to write essays. No one actually needs another essay about Shakespeare or whatever, and ChatGPT could do it for you, but the point of the exercise is to learn to write. It’s the process, not the product that matters.
    • user432678 4y ago
      Everyone’s ChatGPT gangsta until their ChatGPT-trained doctors start giving fatally wrong diagnosis. 100% agree with whole comment. I interview cheaters time to time, their excuse of not learning stuff is exactly that. However, in the end they’ve cheated themselves more than the system.
      • EVa5I7bHFq9mnYK 4y ago
        With the rate they give fatally wrong diagnoses and treatment plans today, I wonder if ChatGPT could be actually better for patients.
        • tsol 4y ago
          ? ChatGPT doesn't diagnose and isn't a doctor.
• endisneigh 4y ago
  without knowing with a high confidence what is LLM generated or not, it seems pointless
• hcks 4y ago
  It’s okay, it’s now clear that programming won’t be done by humans within 10 years.
• wpietri 4y ago
  This is definitely going to require changes in interview practices for a lot of places. For those looking for an alternative, may I suggest pair programming? I've been doing pairing interviews for years. These days I have a standardized, practical problem, something that's reasonably like the work. (E.g., let's use APIs A and B to build features X, Y, and Z.) I let them pick their preferred language and tooling, so that I can see them at their best. And then we spend a fixed period diving in on the problem, with me getting them started, answering questions, and getting them unstuck. I like this because not only do I get to see code, I get to see how they think about code, how they deal with problems, and how they collaborate. They get to spend time building things, not doing mensa puzzles, posturing, or other not-very-like-the-work things. And they can't bluff their way through, and it's pretty hard to cheat.
  • kube-system 4y ago
    The emergence of this problem is exactly why I don’t like giving people programming exercises that are overly explicit. Most coding tests just tell people what to write and then have them write it. Real world problems are more complicated. Instead, tell your candidates what your problem is and then ask them for a solution. Let them write their own requirements. It’s a lot harder for language models (and developers with poor problem solving skills) to solve these kinds of questions as well.
    • wpietri 4y ago
      Exactly. At points during a pairing interview when I get asks for more details on requirements, I'll make sure to reply with something like, "Which do you think is better for the user?" It turns out most developers have a pretty good sense for this, even if they're not used to being able to use it.
  • sgu999 4y ago
    If some places need to change their interview practices, I think it means they've been doing it wrong all this time. You ask them to write some code, then to explain and justify it. Whether or not it has been mostly written by an LLM really shouldn't matter... (leaving aside edge cases preventing to use some tools)
• neilv 4y ago
  I see a ton of people trying to justify their use of Copilot/ChatGPT (or rushing a startup to cash in on LLMs). Maybe that conflict of interest is why there's very little talk of it being based on plagiarizing and license violation of open source code on which the model was trained. We just suffered through a couple decades of almost every company in our field selling out users' privacy, one way or another. And years of shamelessly obvious crypto/blockchain scams. So I guess it'd be surprising if our field didn't greedily snap up the next unethical opportunity.
  • pixl97 4y ago
    Right, any humans that reads open source code ever should also be forced to submit any code they write to ensure they've not mentally performed copyright violations. I don't give two shits if whatever current expensive GPT is dumping out code 'very similar' to open source code today. And you'd be chopping off your own nose if you did too. Thinking these models will remain as expensive to run in the future means that at the time you or I could run 'LibreGPT' on our own hardware, we'd be scared as hell to even write the code because any use of it could get you sued into oblivion. Burn copyright to the ground.
    • deleted 4y ago
      [deleted]
    • robocat 4y ago
      That’s why we have clean room development. For patents it is to avoid triggering triple damages. For copyright it is to help avoid allegations of plagiarism. > Burn copyright to the ground. Burn patents to the ground! Copyright has some use, but the Disney changes have somewhat ruined its purpose.
      • neilv 4y ago
        Cleanroom development practice is a great point to make. We have precedent of believing that exposure to some code (or other internals) might taint an engineer, such that they can't, say, write a sufficiently independent implementation.
• otterpro 4y ago
  If I were a teacher, I'd give written exams for coding, as we used to have back in the 80's when we didn't have enough dedicated computers for everyone in the class, and most of kids didn't have a home computer to work with. So instead, we'd implement simple algorithms on paper for in-class quiz/tests/homeworks. I remember in high school, we had an exam, and one of the question was to write a for loop counting from 1 to 40. It might be harder to grade, but teacher could usually tell if the code looked good or not. On one occassion, I thought I had written the code correctly but teacher didn't, so I had to demonstrate the code on the computer for the code that I had written in the test and show the teacher that the code ran as expected. It's kinda like whiteboard coding, but it's a lot less stressful since the test is mostly written on paper without the scrutiny of an interviewer. Obviously, we can't create a react web app, but at least students can demonstrate fundamentals, which are what we should be teaching in the first place. Another solution would be to create a new programming language that doesn't exist yet for the students to write code for their assignments or perhaps use an obscure language that AI/chatgpt isn't aware of... perhaps we can go back to Ada or Modula 3.
  • travisgriggs 4y ago
    > If I were a teacher... You'd be wondering how the heck you get out of a profession where pay is stagnant, the backlog of problems has been growing for a long time, and the minimization of resources to deal with has been in the wrong direction for a long time. You'd join the league of "you can never go wrong hiring another administrator" or exit for some related parlayable field in industry and throw your hands in the air with a "somebody else's unsolvable problem" disgruntled attitude.
    • asdff 4y ago
      CS programs seem like a different beast. At my alma mater they are always growing the cs faculty pool. At certain schools you can really work with some giants in cs departments. People who developed the algorithms you are using today with a piece of chalk. A lot of the new hire faculty seem to come from industry too, secured their bag already, and are jaded from working in industry. Sometimes they come from those industry "skunkworks" R&D type of teams where you'd asume everything should have been just peachy for them, but clearly not if they are looking for academic positions with their stacked resume.
• nextlevelwizard 4y ago
  Why should anyone care? If someone wants to coast they will and it will be reflected later on when they cant get or hold a job since they are just as shit as ChatGPT or copilot And if they can get and hold an job then isnt that just better?
• josh2600 4y ago
  So most interview questions like "make a binary tree" are dead. The best interview question that will never die: "What's the weirdest bug you debugged? What made it weird?" For posterity: https://www.gamedeveloper.com/programming/my-hardest-bug-ever https://www.gamedeveloper.com/programming/my-hardest-bug-eve...
  • bcantrill 4y ago
    Yes, agreed! We ask a variant of this question (we call it "an analysis sample"); from the materials we ask candidates to submit[0]: "A significant challenge of engineering is dealing with a system when it doesn't, in fact, work correctly. When systems misbehave, engineers must flip their disposition: instead of a creator of their own heaven and earth, they must become a scientist, attempting to reason about a foreign world. Please provide an analysis sample: a written analysis of system misbehavior from some point in your career. If such an analysis is not readily available (as it might not be if one’s work has been strictly proprietary), please recount an incident in which you analyzed system misbehavior, including as much technical detail as you can recall." These samples are very revealing -- and it feels unlikely that generative AI is going to be of much help, even assuming a fabulist candidate. (And of very little assistance on our values-based questions like "when have you been happiest in your professional career and why?"). [0] https://docs.google.com/document/d/1Xtofg-fMQfZoq8Y3oSAKjEgDQCRHx-GMSmPcxdEea4A/edit?usp=sharing https://docs.google.com/document/d/1Xtofg-fMQfZoq8Y3oSAKjEgD...
    • epicureanideal 4y ago
      To be honest, this sounds extremely difficult and not in a good way. That sounds like many many hours of writing work, to describe a problem that might be many years in the past, that might have been solved by extremely intricate methods that are easy to forget, using technologies that are now not commonly in use, etc. A good question to ask about each interview question might be: would a good liar have an easier time answering this than a person trying to answer honestly? And if so, retire the question.
      • 4y ago
• la64710 4y ago
  A lot of time is saved by chatGPT. But we need to get better at testing and debugging.
• jimbobimbo 4y ago
  All this tells us that whiteboard interviews are here to stay.
  • warning26 4y ago
    With the general shift to remote-first, it's a lot more difficult to verify that your whiteboarder isn't ChatGPT-ing just off-screen.
    • flangola7 4y ago
      Who says anything on the screen will be real? "Hey MultimodalGPT, look like me on a Zoom call and pass this interview." Text2Video is already moving quickly. Maybe in as little as three years a video feed will be as trustworthy as an email.
• Spivak 4y ago
  > Apparently, it's not even something students feel they need to hide. Why would you feel the need to hide it? It's a tool, it's not like the library, other professors, your friends, Sourcegraph or StackOverflow is cheating. Trying to argue why GPT is cheating is just going to devolve into "you can have outside help, as long as it isn't too good for some arbitrary line.
• savolai 4y ago
  In earlier versions of Moodle there used to be a peer review assignment type, where students evaluated each others' submissions and both scores and other students' evaluations were scored. Thwn teacher only had to evaluate the reliability of some evaluators to infer skill level for everyone, if I recall correctly. My understanding was that this scaled quite well but you wouldn have to ask the professors. This was used in an HCI course at my uni in Tampere, Finland, years ago. As a student the experience was very communal and enlivening.
• gchallen 4y ago
  I teach CS1. A lot of this post resonated with me. In particular, I don't think that beginners are well-served by relying on AI to complete their assignments. Later on, once they've developed some computational thinking abilities, sure. Starting out, no. There's a real dearth of good options available to computer science educators today for teaching introductory material effectively in the face of all the new and existing ways there are for students to cheat. A lot of what people offer up as alternatives are unworkable or downright bad ideas: * Paper exams represent an unrealistic environment, encourage terrible programming habits, are a nightmare to grade, and don't test student abilities to identify and correct their mistakes—which is maybe the most important thing we want to assess. * Oral exams also don't scale and raise obvious equity issues. * Beginners have to build basic skills before they are ready to work on larger open-ended projects. We're fortunate at Illinois to have a dedicated computer-based testing facility (https://cbtf.illinois.edu/ https://cbtf.illinois.edu/) that we can use to allow students to take computer-based assessments in a secure proctored environment. This has been a really important support for our ability to continue to teach and assess basic programming abilities in our large introductory courses. I'm not sure why this idea hasn't caught on more, but maybe AI cheating tools will help drive broader adoption. (Such facilities are broadly useful outside of just computer science courses, and ours is heavily scheduled to support courses from all across campus.) Anything would be better than people returning en masse to paper programming exams.
  • asdff 4y ago
    For CS1 you probably don't need much compute power for assignments. Schools are already well funded enough to sometimes offer freshman ipad pros. You could offer them a raspberry pi or some extremely cheap, low powered PC, make them return it after the semester if you want to save them money. You can firewall it from the open internet, and have students turn in their code from this device alone to a university server. They can still cheat, sure, but to do it would mean transcribing code from one device to another by hand, which is enough friction and a timesink where fewer students would consider it versus actually paying attention in class.
    • mechanical_bear 4y ago
• resource0x 4y ago
  The professor could just ask the student: how did you come up with this solution? What was your thinking? To prepare for this question, the student has to think really hard both about the problem and about the solution. While doing so, we learn a lot - maybe more than we would learn otherwise.
  • unreal37 4y ago
    Asking ChatGPT to explain the reason it gave an answer is a thing.
    • resource0x 4y ago
      My bet is that it won't be able to provide a human-level explanation.
      • flangola7 4y ago
        Maybe not today. Tomorrow is another question.
• stevage 4y ago
  One possible temporary solution is to use ChatGPT to generate the solution to each question, and only include questions where its answer is incorrect. Then at worst the challenge for the student is debugging ChatGPT code, which still has merit.
• braingenious 4y ago
  >It's been breaking my understanding of what is and isn't possible, and forcing me to think about new ways to solve technical problems. followed up by >However, I'm not sure how to understand it as part of student learning. Is super funny
• medion 4y ago
  Chomsky, in a recent interview, said if students are cheating with these tools, it’s the fault of the teacher for not creating work that is interesting enough to be absorbed by …
  • shadowgovt 4y ago
    There is nothing interesting about pointer dereferencing errors (or their slightly-more-turd-polished equivalent, null references). Absolutely nothing at all. Most programmers will spend a non-trivial amount of their career fussing over them, however, and any programming education program that doesn't at least touch on how to identify them and what to do is pedagogically void.
• shadowgovt 4y ago
  I find myself wondering if the future of programming looks more like editing than synthesis. This is not a bad thing if it's the case; a lot of the job of a software engineer is in analysis of code that exists, but so much of the pedagogy is in synthesis-from-scratch in a world that is already full of billions of lines of code.
  • notduncansmith 4y ago
    The leadership career track for programmers already closely resembles this process (providing natural language prompts for ICs and then reviewing/correcting the output), with AI just shortening the feedback loop. This has me wondering which software stacks will most readily lend themselves to AI-driven-development.
• dariosalvi78 4y ago
  I teach to first year students. We have graded programming assignments they can do by themselves, where they can cheat in all sorts of ways, but I honestly don't care, it's their responsibility to learn. Then we have theoretical exams, where there are also questions about programming (but we don't check if the code has perfect syntax). In one course the exams are oral: excellent tool for assessing students, but very time consuming, in another it's written: more efficient, but more shallow. I like the idea about teaching how to use AI, but it has to be a tool among others.
• m3kw9 4y ago
  Before gpt3 you can still hire people to cheat, it’s higher barrier to entry, but now literally everyone can do it instantly and free. Take home assignments must be de-emphasized and focus in person tests
• mahathu 4y ago
  Thank god for whiteboard coding interviews!
• indus 4y ago
  Wasn’t this day inevitable when colleges killed paper and oral exams?
• HarHarVeryFunny 4y ago
  When current and future students are out in the workforce they'll have access to tools like CoPilot and ChatGPT, not to mention plain Stack Overflow cut-n-paste, so if education is meant to be preparing kids for the workplace, then the logical thing to do is allow/teach them to use these tools. OTOH, they still do need to learn how to program and debug (more important than ever if you're using machine-generated code likely to contain bugs, and with unknown error handling), so it seems colleges also need to make the assignments more complex to point where current crop of tools can only be used to assist - not to nearly write all the code. It'll be interesting to see how these new tools affect hiring standards... Doesn't make much sense to pay someone just to get a tool to write the code, so maybe the bar will be raised there too.
  • barking_biscuit 4y ago
    >if education is meant to be preparing kids for the workplace, then the logical thing to do is allow/teach them to use these tools Can you imagine being a CS major who has just started your degree in 2023 and day after day you're sitting there watching the sheer speed of development of this new technology and you're unable to fathom at all what the industry you're supposed to be entering in a few years will even look like? What a mind fuck.
• pfisch 4y ago
  Can't you add a series of weird restrictions to the questions that make it very difficult for chatgpt to work with. Must use switch statement, must declare an iteration int on line 10. Must use a for loop to count down in reverse. If you have like 8 of these rules chatgpt may not be able to handle it.
• throw310822 4y ago
  I don't understand how most of the comments here seem to be along the lines of "these interview questions are useless now" or "we need to rethink education, it hasn't kept up". These all seem absurdly myopic to me. What we're seeing is the first instance, still very limited and imperfect, of AGI. This is not going to make some interview questions obsolete, or give students more tools to cheat with their homework. It is effectively proving that acquiring knowledge and professional skills is becoming useless, for good. In a few years (3, 5, 10 at most) this is going to defeat the entire purpose of hiring, and therefore of completing most forms of professional education. At the current rate of progress, most intellectual professions will be obsolete before today's sixth-graders are ready to enter the job market. I can't even picture a functional world where humans are cut out of most professions that don't involve manual work; where any amount of acquired knowledge and skills will be surpassed by machines that can produce better results at a thousandth of the cost in a thousandth of the time a human can. And even if such a world can function, I can't imagine a smooth transition to that world from its current state. I'm worried, does it show? :)
  • haolez 4y ago
    I don't know if this is an AGI-like experiment, because LLM are trained on human knowledge. I'd expect that real AGI wouldn't need such a thing and would improve on its own. That's the moment where we become obsolete.
    • hombre_fatal 4y ago
      "you can invent human cognition from first principles via billion years of parallel evolution. or since it’s already been invented, applied, recorded at scale you can just observe its behavior to learn it"
    • pixl97 4y ago
      Mammals did not evolve from single cell creatures in order to replace the dinosaurs.
  • kurisufag 4y ago
    >What we're seeing is the first instance, still very limited and imperfect, of AGI absolutely not. language-model text generation is actually about as non-general as it gets -- they are fundamentally incapable of understanding anything at all, ever. they can't do math, work through basic logic problems, or produce any output that isn't just an assumed logical continuation of the input.
• EVa5I7bHFq9mnYK 4y ago
  So, what if we give ChatGPT code of ChatGPT, and ask to improve upon it? If it can do that, we are fucked.
• college_physics 4y ago
  "i cheated myself like i knew i would" when students get out of the classroom and face the human economy they will find that nobody pays anything for doing something that can be done at near zero cost by a machine
• sebastianconcpt 4y ago
  1984: Computers are like a bicycle for our minds 2024: AIs are like super-sportcars for our minds
• qprofyeh 4y ago
  Idea: Ask the students to record themselves explaining their solution, and what lead them to it.
  • layer8 4y ago
    So you’ll grade them on how good their teleprompting skills are.
• superb-owl 4y ago
  Short term, this is a problem. Long term, the bar will simply be raised. Smart/motivated kids armed with AI will still outperform the ones who don’t study, and will be graded accordingly. This will only be a problem so long as AI help is considered cheating, and until teachers are able to recalibrate their standards.
• Al-Khwarizmi 4y ago
  As a CS professor myself, I don't think ChatGPT really creates entirely new problems for assessment, but rather makes problems that we had been sweeping under the carpet more obvious and harder to ignore. Any assignment that can be cheated by using ChatGPT could already be cheated before by asking a friend, an expert or paying someone else to do it. But most teachers assumed this doesn't happen often, and thus acted as if it wasn't a thing. Making life much simpler and easier. Facing ChatGPT will probably help us make assessment fairer in the long run (and the path is probably going to be a lot of traditional on-site assessment, abandoning the continuous assessment fad that has always been rather disastrous for equity, even if again, life is easier ignoring this).
  • jraph 4y ago
    > the continuous assessment fad that has always been rather disastrous for equity Can you expand on this? I lived the two models as a student and still can't make up my mind on which is best. I feel like it just takes catching some bad cold the week of the exams or a bad exam subject / topic to fail them which seems pretty unfair, but I also know my shit better and have more perspective the week of the exam and prefer not having to stress every week because of the continuous assessment.
    • arvidkahl 4y ago
      There is also an option outside "one exam" and "many exams" — and that's "no exams." Obviously, this is spectacularly hard to fit into the current system, but you see this a lot in vocational schools where project-based work makes up the majority of the grade and exams are held to comply with regulatory requirements. ChatGPT might just fit into an educational world that values outcomes over fact retention. One may dream.
      • MengerSponge 4y ago
        Well-structured exams are the single most effective teaching tool available. Assessment is a fringe benefit. I'm a strong advocate for "even more exams", where you take the highest score on particular topics. There are variations, but if you're curious check out "Standards-based grading" or Mastery-based grading. It's in a weird and shitty limbo where a lot of primary and secondary schools have adopted it partially with inadequate training/support, and it's also getting fucked by our shitty publishing industry, same as the botched common-core rollout.
• naet 4y ago
  People who use this are just robbing themselves of an education. It's the same as copying the answer from stack overflow or another student. Most problems are already online somewhere, chatgpt is just shortening the research time. You can solve plenty of math problems by putting them in Wolfram alpha, but if you do that for every calculus assignment you won't learn calculus and you won't be able to apply it to solve problems. Same with programming; if you copy a fizzbuzz instead of learning how to write a fizzbuzz, you won't be able to do a more complicated problem or understand a loop when you need to.
• nchohan 4y ago
  The future is all prompt engineering.
• 00F_ 4y ago
  the world has changed dramatically in only one year. these advancements in AI are talked about everywhere and by everyone. in 2018 i warned people i knew and i wrote many comments online warning people that huge changes were coming. almost every single person on HN insisted that GTP was just a parlor trick because it would ramble sometimes, get stuck in loops and overall did not demonstrate the insane level of lucidity that it does now. almost every single person on HN would silently downvote my comments. almost every single HN user insisted that AI could never progress to where it is now. well here is my bittersweet vindication. watching people move the goalposts is like watching a supernova. its so powerful and unbelievable that it can only be compared to a force of nature. commenting a sentiment that doesnt paint AI as harmless is like screaming into a hurricane. i dont even know why i bother anymore. but let me just say it: i think carmack made a great point when he said that the first AGI will be a bunch of lower AIs plugged into each other. GTP doesnt behave in a sentient way but it is very similar to more primitive parts of the mind. its like a persons intuition. dont be fooled by the limitations of any one model. whats important is the road we are on. and for the hundredth time i will scream into the wind: there absolutely is hope in the idea of preventing AI from progressing rapidly or even at all. to do so is absolutely necessary for us to preserve anything that we would consider a good life for future generations. and i will offer a prediction. nobody will agree with me. but after a handful of explosive advancements in AI, everyone will agree with me.
  • barking_biscuit 4y ago
    I'm trying to work out whether or not the right thing to do is just to help work towards ushering in AGI by whatever contribution I can make, or whether to scream into the hurricane. Screaming into the hurricane feels pointless, and I generally agree that on balance AGI is likely to be net negative. Overall I just feel somewhat confused.
    • 00F_ 4y ago
      mention to people in real life and online that you are worried about AI. it doesnt cost anything and it makes a real difference.
• jki275 4y ago
  Just tell students to use it like they might use StackOverflow or even a textbook. It might help you to learn, but it's not safe to try to just use it to give you the answer. You need to know the topic better than the machine does, because it will lie to you like nobody's business and doesn't care if it does.
• h9n 4y ago
  > the current funding model of post-secondary institutions, which does not prioritize the ratio of faculty-to-students necessary for ever more personalized or real-time assessment methods. Get ChatGPT to do the assessments.
• javierluraschi 4y ago
  Gone through all the big-tech interviews, previously worked at Microsoft and a few more. I'd say that the only interview question I think survives here is what we've been doing at our startup: "For this role, here is a real four hour problem for you to work on, let us know when you want to start and we will send you the problem, send it back four hours later and then we will schedule some time to chat about it". If someone uses ChatGPT and solves it better than other candidates, that's great, good for them Don't think it's possible to solve it entirely with ChatGPT, but I could see some good candidates getting pointers to solve a 10%-30% of the problem to make them more efficient, before ChatGPT people could Google anyways and honestly I would expect they do. I think a similar approach should apply to schools. Here is your math homework, solve it whichever way you want but then we will discuss it. I guess this would be the oral approach from the post but minimizing the time it takes to do the oral review. If your homework assignment has 20 questions and you discuss one with each student, you'll get a pretty good overview if they know what they are doing. If they want to ask ChatGPT for replies during this oral test, go for it, but they still get the same amount of time to discuss the question, is going to be really hard to get this done realtime. Also, I'm not sure ChatGPT really understand what it's doing to the point where an expert can prove it in limited time.
• imperio59 4y ago
  I think what this means is that we are likely to see a next generation of software engineers that heavily lean on AI to "learn" programming and we will have to re-train them to some degree when they get into the workforce to be able to actually know the skillset they supposedly learned. It also means that whiteboard problem solving for coding interviews is going to continue to be a great test to separate out students who actually know how to write a basic for loop and function and those who don't because Co-Pilot "always does it for them". I know which kind of new grad I want working with me...
• deleted 4y ago
  [deleted]
• QuadrupleA 4y ago
  With all these ChatGPT academic apocalypse stories, I keep thinking - what the hell is wrong with students, that they don't want to actually learn the subjects they've enrolled in? If your goal is to fake-learn and have ChatGPT do your homework, just drop out already. You're paying big tuition for no educational benefit. Maybe your degree gets your foot in the door at some company, but it won't be too long before they suss out your incompetence. Or perhaps you get "lucky" and are able to sustain a kind of imposter parasite life in an overbloated company that doesn't notice. Doesn't sound like a very interesting or fullfilling life, but to each their own I guess. I hope fake-learners don't find their way to my companies. Long live the enjoyment of learning, real expertise, and the building of cool things.
  • roncesvalles 4y ago
    I've seen rampant pre-ChatGPT cheating at my university where it's popular for business undergrads to take computer science as a second major to get a leg up for Product Manager roles. These people don't intend to ever write a single line of code after university. They aim to be just familiar enough with software development to be able to nod along and throw in a few buzzwords in interviews. Big money attracts people who just want the money.
    • Aperocky 4y ago
      This is why leetcode problems might have some actual use.
      • reilly3000 4y ago
        I wish. They are the most easily exploitable, sadly. People memorize and dump stuff all the time, and it usually doesn’t show reasoning, just how much money and effort you spent on CS fundamentals. My team has weeded out a lot of bad candidates with super simple, practical tasks. Explain DNS. Order keys from a JSON object. Make a 2 column layout in plain html. You would be amazed at who can’t do that. What we don’t do is ask the same questions too many people times, as we know some candidates compare notes and even publish the interview questions. With large language models doing the talking we have already found candidates that have been unable to describe the for loop copilot made for them off-screen… so I guess the best system is to be good at being humans and having a go at working together in something.
• soorya3 4y ago
  This is simply a question of nail and hammer or used power tools, does it matter? My opinion is, it doesn't matter how/what tools student used to write the program, what matters are two things 1) do they understand the program they have written 2) have they done a good job
• dqpb 4y ago
  > Except that half the code was made-up. One of the API calls that it was using doesn't exist (I wish it did!) I've also been using ChatGPT to generate code for a specific library, and had the exact same experience. It uses an API that should exist, but not the one that does exist.
• noonething 4y ago
  adapt and overcome Thank you evolution you cruel mistress.
• vxNsr 4y ago
  I think that this can be trivially solved by openAI. they could store all their responses and then for a fee allow schools/professors to submit the answers they're getting from students like they already do with turnitin if the answer is 90% the same... easy to deduct that it's not original. plus, because OpenAI makes you login, if it is a match they could allow you see the email address associated with the submission... if its the student's name, pretty obvious that it was not their own work.
• patrakov 4y ago
  The real problem is that an _attempt_ to cheat is graded, on average, higher than no submission at all. We need to change that.
• throw14082020 4y ago
  > What about students who don't really know what they're doing and just need to get something submitted for this damned assignment before 11:59 pm? The author mentions ChatGPT is better for more advanced users, rather than beginners. I disagree. I think ChatGPT is really good for beginners. Instead of being stuck or making up the "why" something is done, ChatGPT can answer it. There is simply no way a lecturer or classroom has time to go "5+ why's" deep into a Question. ChatGPT at least will do it, and then you can fact check it. It helps me improve my vocabulary, point me into right placed which I can investigate further. When I use ChatGPT for difficult problems, it doesn't give me anything useful or correct.
• insaider 4y ago
  What if you actually asked the students what they think the best approach is? I wonder what they'd have to say when confronted with optimizing their own education. As a CS grad I often felt such a large percentage of the syllabus was legacy theory I'd (mostly correctly) never need. Perhaps this can be the positive impetus to get teachers to really rethink syllabuses from a more practical, useful view point. "here students, use ChatGPT and all the tools available to you to create x..."
• michaelcampbell 4y ago
  I wonder if this will lead to in-class tests being weighted much more heavily than homework style work.
• black_13 4y ago
  [dead]
• sebastianconcpt 4y ago
  The problem with synthetic intelligence usage is that, if used like a crutch, it will necessarily modify how you walk. And if you could develop walking just fine and interject this, are you going to be a great walker? What would be the next generation's intelligence strong at? Biology needs a soft quote of regular stress to develop compensating the needed optimizations. Happens with muscle growth, fat burning and developing psychological traits and behavioral transformations. What will be our new opportunities for growth if all the hard work is going to be done by synthetics? (with a hardware to run that is as expensive as a high category car)
  • Jensson 4y ago
    Intelligence will go the same way bodyweight did when we stopped needing to use our bodies. Some people exercise and eat properly still, but most doesn't.
• anotheryou 4y ago
  In the end you want them to use ChatGTP, but also get to know enough basics to actually be able to use it well. A few suggestions (examples frontend-heavy, that's what I'm coming from): - show them how chatGTP can explain things from them that otherwise would be over their head. - Have them explain the code (live, otherwise they cheat this too). - Have them write code too complex for chatGTP. Here they learn how much you have to baby-sit it still. (I e.g. tried a browser webext where code is spread across 3 languages and many files and just initializing your settings from stored values can be troublesome. Mine included a list with a dynamic number of rows and animations for add and delete. - Mislead them to write unmaintainable, unscalable code with chatGTP and show them how to avoid these traps. Make them write material design style CSS and have them change colors later. Or a day range picker like airbnb. - Have tricky requirements that fail without good QA. - Unminimize code with chatGTP. It's fun and often fails :). From A to B here took me several attempts and breaking down the problem for chatGTP. A: https://jsfiddle.net/3jvcx075/3/ https://jsfiddle.net/3jvcx075/3/ B: https://jsfiddle.net/2083wfqg/ https://jsfiddle.net/2083wfqg/ (still not 100% correct as the author notes: https://twitter.com/KilledByAPixel/status/1627102571109183490 https://twitter.com/KilledByAPixel/status/162710257110918349... )
• imcorykarma 4y ago
  [dead]
• imcorykarma7 4y ago
  [dead]