8 ms·

There has been a lot of "well, we had a good run" comments on ChatGPT threads from engineers lately. I get where this sentiment is coming from, but I also think

There has been a lot of "well, we had a good run" comments on ChatGPT threads from engineers lately. I get where this sentiment is coming from, but I also think that the article paints a good picture of how we must "pivot" as a species to avoid faceplanting completely. Zero Trust Homework sounds like a strategy we will be forced to implement: the system will frequently give the wrong answers (and not just on accident — wrong answers will be often pushed out on purpose); the real skill in the homework assignment will be in verifying the answers the system churns out — learning how to be a verifier and an editor, instead of a regurgitator. If done well, I believe this can prepare the next generation well for a future we cannot even imagine. The next 10 years will be interesting to say the least.

soco 4y ago
I can imagine somebody knowing the answer being able to properly verify the output. But a student which is supposed to learn something form that homework? We're missing the entire point of homework here.
- appletrotter 4y ago
  > But a student which is supposed to learn something form that homework? We're missing the entire point of homework here. They have to learn to verify it. I think you missed point of OPs idea.
- mavhc 4y ago
  The homework is originally: research this topic, write an essay about what you found. Now it's: here's an essay, research this topic to critique it
- 314 4y ago
  Well, they would need to know something about the topic. The easiest way for them to do this would be to read about it. When picking what to read they would need to be able to estimate if it contained valid information. The meta-skills that would be developed would be: * Understanding the quality of sources * Understanding the different between primary and secondary sources. * Establishing chains of evidence and tracing them through large datasets. * Integrating information from different sources and using correlation to establish validity. Basically, it would be a crash course in how to do research, and the best possible preparation for living in a world where we drown in uncertain information and still have to make the best choices that we can. For bonus points, at the end of the process they would have a better understanding of the subject matter than somebody who had only read about it.
  - ghaff 4y ago
    Someone was also demonstrating over the weekend that literally cut and pasted ChatGPT text output is easily detected as LLM output with near certainty. https://huggingface.co/openai-detector/ https://huggingface.co/openai-detector/ That said, adult me could probably use ChatGPT to stitch together a 1000 word high school paper pretty easily (and probably evade plagiarism/LLM detectors). But, as you say, I could probably get a lot of the way there by other means as well and putting words on the page is mostly not the hard part anyway.
    - adamsmith143 4y ago
      Ironically inputting: "This is human generated text." generated an 86% probability of being fake text. But it seems kind of dubious to begin with, I'm not sure why LLM output would be so easily detected.
      - ghaff 4y ago
        One short sentence isn't enough. If you play with it a bit, there are definitely patterns. Here's one I did over the weekend: 99.98% fake. The Battle of Agincourt was a significant event in English and French history, as it was a major victory for the English army against the French during the Hundred Years' War. The battle took place on October 25, 1415, near the town of Agincourt in northern France. The English army, led by King Henry V, was vastly outnumbered by the French forces, which were estimated to be around 30,000 soldiers compared to the English army of just 6,000. Despite the odds, the English were able to emerge victorious thanks to their superior tactics and weaponry. The English army was composed of mostly longbowmen, who were able to shoot arrows with great accuracy and power. The French, on the other hand, relied heavily on heavily-armored knights, who were less effective in the muddy and wet conditions of the battlefield. The English were also able to take advantage of the narrow front of the battlefield, which allowed them to focus their firepower and overwhelm the French forces. Whereas the first couple paragraphs from a book I'm working on came out 0.02% fake.
        adamsmith143 4y ago
- HammadB 4y ago
  I think what’s worth mentioning is that for people who grew up with google, schools already were having a bit of a panic with resources like wolfram alpha and Wikipedia. For this cohort of users, myself included, the idea of fact checking ”found” material and weaving it into a proper answer is very familiar. With this view, I think chatGPT is less of a novel UX shift than an accelerant on an existing one. I think it proves the age old belief that as technology reaches maturity it loses its fancy monikers and we will probably just look at this as “very good, very specific, google” one day
  - ghaff 4y ago
    Of course, even back in the day, I'm sure many a school report cribbed heavily from Encyclopedia Brittanica etc.
- constantcrying 4y ago
  >But a student which is supposed to learn something form that homework? The student still had access to primary sources, textbooks, google and AI answers. In general I am a fan of homework which forces critical thinking. Rephrasing someone elses work is a far less valuable skill than the critical thinking which is demanded for that type of work. Unrelated of any AI, students should learn how to read a text and be able to investigate its contents.
- indigochill 4y ago
  Actually this is interesting framing given the misinformation pandemic on the internet today. Suppose students are taught material and then given a test in the form of a mix of correct and subtly incorrect assertions that they need to filter through. Might help them learn to distinguish information from misinformation as well?
- yamtaddle 4y ago
  I'm expecting this whole thing to deepen educational class divides. Public schools will struggle to teach kids anything, and to distinguish between A grades that were earned and those achieved largely through cheating, while parents will resist efforts to switch to grading schemes that can't be trivially gamed (just try switching to only-in-person-tests-count grading and witness the rage that follows when Little Johnny's grades plummet). Kids who lean on the motivation from graded homework to get themselves to do the work (and so, learn the material) will struggle, as well, even if that is allowed to happen. This will enhance the reputations of schools that can buck the trend, applying more resources (mostly more teachers, for smaller class sizes, to make it easier to keep an eye on everything and really gauge & guide learning progress for each student) and an "if you don't like it, fuck off to somewhere else" attitude to parent complaints about grading practices. Which means, elite private schools will benefit. I expect that a 4.0 from a public school will mean almost nothing, while at the same time coming in under a 4.0 will look terrible ("everyone cheats and you still couldn't get a 4.0? How dumb are you!?") which will just worsen the cheating problem. Maybe parents will surprise me and not throw shit-fits over solutions to this, and public schools will be able to implement them. But I doubt it.
thundergolfer 4y ago
Over the weekend Twitter was 30% ChatGPT screenshots and 20% engineers frothing about how ChatGPT would replace programming, article writing, even Google Search. Such engineers must be juniors, or have persisted in their life with a totally wrong idea of how engineering is done and engineering is built. I can't imagine thinking that one could build software systems or essay arguments by trusting an AI system you don't understand to provide answers you don't understand. Like you say, we very well could faceplant society with this misconception that regurgitation plausible code and prose is what education and understanding looks like.
- rednerrus 4y ago
  Couldn't this help a 10x engineer become a 100x engineer? The secret is still going to be in understanding what to ask for and ensuring that what gets spit out works. It's just going to be so much faster. Is there anyplace in the world for a .5x developer or even a 1x developer in a chatgpt world?
  - ghaff 4y ago
    But this sort of change is happening all the time in all sorts of different domains. There probably isn't much of a role for a web developer who just knows some basic HTML like there was in the late 90s. And you can apply this to a lot of basic wring, transcriptions, etc.--at least at western pay scales.
  - flatline 4y ago
    I do not think it could. What I see GPT doing here is generating a lot of plausible boilerplate. We already have this via things like code snippets. I find them to be useless, like autocorrect on my phone. It gets in the way of my reasoning and does not really enhance it in any way. Sometimes I make mistakes typing but I’d rather them be my honest mistakes than the computer censoring/rewriting my thoughts. Good engineering requires good reasoning skills and GPT has exactly zero reasoning. It cannot do the thing that humans do and it cannot do what a calculator can do. I think it is neat and fun, but that is all, a novelty. I’ve used auto-routers for PCB layout and they will do a 90% job that takes just as much work to redo to get the last 10% as it would to have done it right by hand from the start. There may be a future for operator-in-the-loop type guided AI generative models but I don’t see a lot of effort devoted to making real systems like that. Watson seemed to have this potential and failed even after a brilliant display of ingenuity on Jeopardy. I see these models headed the same way.
thaeli 4y ago
I'm more concerned that we would see these essays being _graded_ by systems that make these sorts of errors.
- nonrandomstring 4y ago
  I'm more concerned that we would see these essays _not_ being graded by systems. I can't imagine teachers reading through essay after essay, each a tedious cookie-cutter nugget of banal pseudo-perfection. The teachers will give up in no time. There would be no point giving feedback, since it is not the student to who you are giving it. Likewise, the students would have little interest in feedback, since they have no personal ego investment in their output. Besides the best essays will be generated by those with access to the better generative software. At this point there is simply nothing to grade. What would be the point? Once essay writing and grading bots are locked in combat it's only a matter of time until that market exhausts itself and disappears in a puff of logic. Maybe this is all a good thing. The process of writing and grading essays as a way of "measuring" knowledge has been a sham for too long. It's a throwback to school from the turn of the last century.
arcturus17 4y ago
I agree with the general optimism in your take, however, I am unsure whether people without any training in ex-novo production (ie, writing from scratch) will be able to operate competently, at least in the realm of code. I am blown away by Copilot's capabilities (haven't tried ChatGPT yet, it's down right now), but I can only operate it because I have written tons of code beforehand. I am getting the best results from it by writing prompts in a language that sometimes includes domain-specific language ("accounts", etc.) but still heavily leans on programming constructs ("write a map with a tuple of integers as a key, and a string as a value") Maybe there's a way of learning how to code without ever writing a genuine line by yourself. You learn some rudiments, read the language docs, start writing prompts and create a testing loop based on expected outputs. Maybe. But it's uncharted territory. Same question goes for good ole writing - can you produce new Hemingways with the process you describe? I know there is a certain vision where the AI will be the new Hemingway but you are talking about a shorter term (~10 years) where it's unlikely will be the case... Will kids be able to properly write if they only train to be editors? As an addendum, I was at GitHub conf and saw Prof. Malan of Harvard CS50 fame claiming that at present his course will not allow the use of tools like Copilot. I have taken the course and they have plagiarism control systems but I don't know how they are going to control this potential explosion.
- fellerts 4y ago
  Taking a step back: Isn't it kind of daft that we humans must go through years of training to learn esoteric languages and ways of bending our brains backwards in order to tell a computer how to perform arguably simple tasks? In the ideal world, you can tell the computer what you want to happen and... it happens. If not, then you did not explain all your edge cases, and the conversation continues. I think it can be argued that we have, seemingly overnight, made vast progress towards that world. You specifically mention code, but that concept is irrelevant in this world. Indeed, this is uncharted territory!
  - arcturus17 4y ago
    > Isn't it kind of daft that we humans must go through years of training to learn esoteric languages I mean, what do you propose, that we skip all the steps in computer science history and just start at the end? Hardly "daft, it's simply the best way we have come up to provide machines with instructions until now. And it's not like people have not tried other paradigms (ex: graphical programming, "low-code", etc.). Also, compared to programming in assembly or binary, programming in Python or other high-level languages is a huge advance in itself. Python, at the end of the day, is nothing but a bridge between natural language and machine code. > You specifically mention code, that concept is irrelevant in this world Current computer systems run on trillions upon trillions of lines of code. GPT-3 or Copilot don't change that fact, in fact they will continue to pile up lines of code. They are systems that map natural language to code, by drawing from an infinitely massive corpus of code. They bring the idea of expressing systems entirely in natural language one step closer, but it's still very far away - almost a pipe dream. The output of these innovations is still code. So tell me again how code has become irrelevant, or how people who do not understand code at all will be able to leverage these advances?
fleddr 4y ago
Zero Trust is based on the idea that us super smart humans will fix dumb AI, manually. The real question is what happens when AI gives the correct, if not better answers.
plastiquebeech 4y ago
Not necessarily a new idea, but it's a good one. One of my schoolteachers edited a historical figure's Wikipedia page the day before a major essay about them was due. The need to check your sources is an important lesson to learn early, AI or no.
qualudeheart 4y ago
Coexistence with AI can go very well. If ir doesn’t…

soco 4y ago

I can imagine somebody knowing the answer being able to properly verify the output. But a student which is supposed to learn something form that homework? We're missing the entire point of homework here.

appletrotter 4y ago
> But a student which is supposed to learn something form that homework? We're missing the entire point of homework here. They have to learn to verify it. I think you missed point of OPs idea.
mavhc 4y ago
The homework is originally: research this topic, write an essay about what you found. Now it's: here's an essay, research this topic to critique it
314 4y ago
Well, they would need to know something about the topic. The easiest way for them to do this would be to read about it. When picking what to read they would need to be able to estimate if it contained valid information. The meta-skills that would be developed would be: * Understanding the quality of sources * Understanding the different between primary and secondary sources. * Establishing chains of evidence and tracing them through large datasets. * Integrating information from different sources and using correlation to establish validity. Basically, it would be a crash course in how to do research, and the best possible preparation for living in a world where we drown in uncertain information and still have to make the best choices that we can. For bonus points, at the end of the process they would have a better understanding of the subject matter than somebody who had only read about it.
- ghaff 4y ago
  Someone was also demonstrating over the weekend that literally cut and pasted ChatGPT text output is easily detected as LLM output with near certainty. https://huggingface.co/openai-detector/ https://huggingface.co/openai-detector/ That said, adult me could probably use ChatGPT to stitch together a 1000 word high school paper pretty easily (and probably evade plagiarism/LLM detectors). But, as you say, I could probably get a lot of the way there by other means as well and putting words on the page is mostly not the hard part anyway.
  - adamsmith143 4y ago
    Ironically inputting: "This is human generated text." generated an 86% probability of being fake text. But it seems kind of dubious to begin with, I'm not sure why LLM output would be so easily detected.
    - ghaff 4y ago
      One short sentence isn't enough. If you play with it a bit, there are definitely patterns. Here's one I did over the weekend: 99.98% fake. The Battle of Agincourt was a significant event in English and French history, as it was a major victory for the English army against the French during the Hundred Years' War. The battle took place on October 25, 1415, near the town of Agincourt in northern France. The English army, led by King Henry V, was vastly outnumbered by the French forces, which were estimated to be around 30,000 soldiers compared to the English army of just 6,000. Despite the odds, the English were able to emerge victorious thanks to their superior tactics and weaponry. The English army was composed of mostly longbowmen, who were able to shoot arrows with great accuracy and power. The French, on the other hand, relied heavily on heavily-armored knights, who were less effective in the muddy and wet conditions of the battlefield. The English were also able to take advantage of the narrow front of the battlefield, which allowed them to focus their firepower and overwhelm the French forces. Whereas the first couple paragraphs from a book I'm working on came out 0.02% fake.
      - adamsmith143 4y ago
HammadB 4y ago
I think what’s worth mentioning is that for people who grew up with google, schools already were having a bit of a panic with resources like wolfram alpha and Wikipedia. For this cohort of users, myself included, the idea of fact checking ”found” material and weaving it into a proper answer is very familiar. With this view, I think chatGPT is less of a novel UX shift than an accelerant on an existing one. I think it proves the age old belief that as technology reaches maturity it loses its fancy monikers and we will probably just look at this as “very good, very specific, google” one day
- ghaff 4y ago
  Of course, even back in the day, I'm sure many a school report cribbed heavily from Encyclopedia Brittanica etc.
constantcrying 4y ago
>But a student which is supposed to learn something form that homework? The student still had access to primary sources, textbooks, google and AI answers. In general I am a fan of homework which forces critical thinking. Rephrasing someone elses work is a far less valuable skill than the critical thinking which is demanded for that type of work. Unrelated of any AI, students should learn how to read a text and be able to investigate its contents.
indigochill 4y ago
Actually this is interesting framing given the misinformation pandemic on the internet today. Suppose students are taught material and then given a test in the form of a mix of correct and subtly incorrect assertions that they need to filter through. Might help them learn to distinguish information from misinformation as well?
yamtaddle 4y ago
I'm expecting this whole thing to deepen educational class divides. Public schools will struggle to teach kids anything, and to distinguish between A grades that were earned and those achieved largely through cheating, while parents will resist efforts to switch to grading schemes that can't be trivially gamed (just try switching to only-in-person-tests-count grading and witness the rage that follows when Little Johnny's grades plummet). Kids who lean on the motivation from graded homework to get themselves to do the work (and so, learn the material) will struggle, as well, even if that is allowed to happen. This will enhance the reputations of schools that can buck the trend, applying more resources (mostly more teachers, for smaller class sizes, to make it easier to keep an eye on everything and really gauge & guide learning progress for each student) and an "if you don't like it, fuck off to somewhere else" attitude to parent complaints about grading practices. Which means, elite private schools will benefit. I expect that a 4.0 from a public school will mean almost nothing, while at the same time coming in under a 4.0 will look terrible ("everyone cheats and you still couldn't get a 4.0? How dumb are you!?") which will just worsen the cheating problem. Maybe parents will surprise me and not throw shit-fits over solutions to this, and public schools will be able to implement them. But I doubt it.

thundergolfer 4y ago

Over the weekend Twitter was 30% ChatGPT screenshots and 20% engineers frothing about how ChatGPT would replace programming, article writing, even Google Search. Such engineers must be juniors, or have persisted in their life with a totally wrong idea of how engineering is done and engineering is built. I can't imagine thinking that one could build software systems or essay arguments by trusting an AI system you don't understand to provide answers you don't understand. Like you say, we very well could faceplant society with this misconception that regurgitation plausible code and prose is what education and understanding looks like.

rednerrus 4y ago
Couldn't this help a 10x engineer become a 100x engineer? The secret is still going to be in understanding what to ask for and ensuring that what gets spit out works. It's just going to be so much faster. Is there anyplace in the world for a .5x developer or even a 1x developer in a chatgpt world?
- ghaff 4y ago
  But this sort of change is happening all the time in all sorts of different domains. There probably isn't much of a role for a web developer who just knows some basic HTML like there was in the late 90s. And you can apply this to a lot of basic wring, transcriptions, etc.--at least at western pay scales.
- flatline 4y ago
  I do not think it could. What I see GPT doing here is generating a lot of plausible boilerplate. We already have this via things like code snippets. I find them to be useless, like autocorrect on my phone. It gets in the way of my reasoning and does not really enhance it in any way. Sometimes I make mistakes typing but I’d rather them be my honest mistakes than the computer censoring/rewriting my thoughts. Good engineering requires good reasoning skills and GPT has exactly zero reasoning. It cannot do the thing that humans do and it cannot do what a calculator can do. I think it is neat and fun, but that is all, a novelty. I’ve used auto-routers for PCB layout and they will do a 90% job that takes just as much work to redo to get the last 10% as it would to have done it right by hand from the start. There may be a future for operator-in-the-loop type guided AI generative models but I don’t see a lot of effort devoted to making real systems like that. Watson seemed to have this potential and failed even after a brilliant display of ingenuity on Jeopardy. I see these models headed the same way.

thaeli 4y ago

I'm more concerned that we would see these essays being _graded_ by systems that make these sorts of errors.

nonrandomstring 4y ago
I'm more concerned that we would see these essays _not_ being graded by systems. I can't imagine teachers reading through essay after essay, each a tedious cookie-cutter nugget of banal pseudo-perfection. The teachers will give up in no time. There would be no point giving feedback, since it is not the student to who you are giving it. Likewise, the students would have little interest in feedback, since they have no personal ego investment in their output. Besides the best essays will be generated by those with access to the better generative software. At this point there is simply nothing to grade. What would be the point? Once essay writing and grading bots are locked in combat it's only a matter of time until that market exhausts itself and disappears in a puff of logic. Maybe this is all a good thing. The process of writing and grading essays as a way of "measuring" knowledge has been a sham for too long. It's a throwback to school from the turn of the last century.

arcturus17 4y ago

I agree with the general optimism in your take, however, I am unsure whether people without any training in ex-novo production (ie, writing from scratch) will be able to operate competently, at least in the realm of code. I am blown away by Copilot's capabilities (haven't tried ChatGPT yet, it's down right now), but I can only operate it because I have written tons of code beforehand. I am getting the best results from it by writing prompts in a language that sometimes includes domain-specific language ("accounts", etc.) but still heavily leans on programming constructs ("write a map with a tuple of integers as a key, and a string as a value") Maybe there's a way of learning how to code without ever writing a genuine line by yourself. You learn some rudiments, read the language docs, start writing prompts and create a testing loop based on expected outputs. Maybe. But it's uncharted territory. Same question goes for good ole writing - can you produce new Hemingways with the process you describe? I know there is a certain vision where the AI will be the new Hemingway but you are talking about a shorter term (~10 years) where it's unlikely will be the case... Will kids be able to properly write if they only train to be editors? As an addendum, I was at GitHub conf and saw Prof. Malan of Harvard CS50 fame claiming that at present his course will not allow the use of tools like Copilot. I have taken the course and they have plagiarism control systems but I don't know how they are going to control this potential explosion.

fellerts 4y ago
Taking a step back: Isn't it kind of daft that we humans must go through years of training to learn esoteric languages and ways of bending our brains backwards in order to tell a computer how to perform arguably simple tasks? In the ideal world, you can tell the computer what you want to happen and... it happens. If not, then you did not explain all your edge cases, and the conversation continues. I think it can be argued that we have, seemingly overnight, made vast progress towards that world. You specifically mention code, but that concept is irrelevant in this world. Indeed, this is uncharted territory!
- arcturus17 4y ago
  > Isn't it kind of daft that we humans must go through years of training to learn esoteric languages I mean, what do you propose, that we skip all the steps in computer science history and just start at the end? Hardly "daft, it's simply the best way we have come up to provide machines with instructions until now. And it's not like people have not tried other paradigms (ex: graphical programming, "low-code", etc.). Also, compared to programming in assembly or binary, programming in Python or other high-level languages is a huge advance in itself. Python, at the end of the day, is nothing but a bridge between natural language and machine code. > You specifically mention code, that concept is irrelevant in this world Current computer systems run on trillions upon trillions of lines of code. GPT-3 or Copilot don't change that fact, in fact they will continue to pile up lines of code. They are systems that map natural language to code, by drawing from an infinitely massive corpus of code. They bring the idea of expressing systems entirely in natural language one step closer, but it's still very far away - almost a pipe dream. The output of these innovations is still code. So tell me again how code has become irrelevant, or how people who do not understand code at all will be able to leverage these advances?

fleddr 4y ago

Zero Trust is based on the idea that us super smart humans will fix dumb AI, manually. The real question is what happens when AI gives the correct, if not better answers.

plastiquebeech 4y ago

Not necessarily a new idea, but it's a good one. One of my schoolteachers edited a historical figure's Wikipedia page the day before a major essay about them was due. The need to check your sources is an important lesson to learn early, AI or no.

qualudeheart 4y ago

Coexistence with AI can go very well. If ir doesn’t…