4 ms·
It takes only a one token error to have code that doesn't compile, execute, throws exceptions, or to be wrong in some serious way. These use cases are amazing n
by doctorpangloss 3y ago
It takes only a one token error to have code that doesn't compile, execute, throws exceptions, or to be wrong in some serious way. These use cases are amazing nonetheless.
Can the same be said for facts? Hard to generalize. Like lets say I asked GPTV the year a painted was painted. Way back when, I was right, I will never need to know the exact year of a painting - whatever that means - after I finished my modern art history exam. Does GPTV? No. I am happy with ranges in my answers about general knowledge, because at the end of the day, there isn't some error-intolerant process using those outputs anyway. Even order of magnitude errors may be fine, getting the zero wrong in the token sometimes just doesn't matter. It always matters when programming though.
Writers increasingly have the same opinion. As more people use GPT4 and learn how stilted and uncreative it is, and that a one token error can make a plot stop making sense, it isn't clear how it's going to deliver on bulk writing. Spam and content farming sure, but even short prose is not in the realm of reality for actual, real breathing human consumption.
How do you categorize this? I don't know, I'm not in the field. There's some huge difference between one token error means 0% correct, versus one token error means 99% correct. Maybe transformers can solve this. Maybe they never will.
- leesec 3y agoYou can just ask it about the error, and it will suggest fixes. I don't know why it has to be 100% the first time. Humans almost never are.
- bo0tzz 3y agoThat works for a straightforward compiler error, but not for plot or logic errors in prose, which cannot be detected automatically.
- famouswaffles 3y agoIt works for all kinds of errors to varying degrees. Even "logic" errors can be generated in a go and detected by another instance. Like a "this is what i wanted but you did this" works sometimes too.
- huytersd 3y agoMost enterprise software just follows basic logic like the workflow above. So worst case scenario chatGPT can build 90% of the code we need.
- crooked-v 3y agoYou can also ask it about "errors" that aren't errors and it will suggest "fixes".
- MeImCounting 3y agoYou can also do that with humans
- pxx 3y agoNot really. It's actually a fairly good interview practice to see if somebody will defend their solution if probed on it.
- DaiPlusPlus 3y agoHumans tend to put up a bit of a fight if you accuse them of producing incorrect program code; you know you're in good company if they pull-out Z3, slam out a few lines in their terminal keyboard, and show you rigid mathematical proof that their code is correct. LLMs don't do that. ...yet.
- lostmsu 3y agoYou can skip this above comment. GPT4 will absolutely double down when it knows it's right. The parent is clueless.
- BoorishBears 3y agoLLMs can do that with chain of thought in a way that generalized to multiple tasks
- DaiPlusPlus 3y agoOnly in a vague way. Even with train-of-thought, feedback loops, and other neat tricks, I've never seen an LLM produce valid theorems for Z3 (beyond trivial examples).
- huytersd 3y ago
- dclowd9901 3y agoBecause why are we considering supplanting humans for this labor if it provides no additional value (apart from sacrificing humans at the altar of capitalism)?
- dotancohen 3y agoCheaper always wins. But besides that, the LLM is less likely to demand unscheduled time off - especially as a fraction of the hours it can put in. If I have a family emergency once per year, and I need my eight hour day off, I've just removed 1/200 of my yearly output potential. The LLM would need to be down for over 400 hours per year to get to that type of output reduction. Realistically, that is unlikely to happen.
- nicolapede 3y agoThe curious thing about this, though, is that of course you can start by replacing software engineers with it, but how far away are you going to be from being able to replace a CEO with it? I would say not that far away.
- root_axis 3y agoI've attempted to use this iterative method with GPT4 to build an application, and things just get clumsier and more error-prone as the program grows in complexity. Eventually I get to the point where asking it to make revisions becomes a dice roll with respect to keeping the code behaving as expected or having it arbitrarily omit random portions of the application logic in the rewrite. It's certainly a great way to brainstorm or to quickly produce snippets of logic but it fails for anything beyond toy apps.
- famouswaffles 3y agoThis looks promising when Microsoft release the code for it. https://arxiv.org/abs/2309.12499 https://arxiv.org/abs/2309.12499
- giarc 3y ago"It takes only a one token error to have code that doesn't compile, execute, throws exceptions, or to be wrong in some serious way." So does my code though. You can take this code, put it into xcode and then ask GPT about any errors. Reiterate and repeat. I did the same while building some Chrome plugins.
- satisfice 3y agoAnd you have no real idea if that worked. You aren’t taking any responsibility. You are just acting as a spokesmodel for the tool.
- huytersd 3y agoOf course it worked. It did what you wanted it to do.
- flir 3y agoIt's ok, I asked it to write some unit tests too. Look, seriously, "it hasn't crashed lately" is the standard by which 99.9% of human-written software is judged.
- giarc 3y agoThe trope is software is to Google a problem, find a SO thread and copy the code from there. How is that any different?
- gpderetta 3y agoYou could have a feedback loop, feeding back compilation errors to the model untill it converges to no errors. Potentially you could do the same for fact-related questions, by fact checking the result against the (properly indexed) training set and feeding the results back. I wonder if it would work.
- abi 3y agoYeah, I've been exploring this generate->error->fix feedback loop along with some test cases on my app builder, and it's quite good but not perfect. It just goes in loops on some errors still.
- api 3y agoYou can always just take the result and polish it up.
- huytersd 3y agoJust ask about the error and it suggests ways to fix it. Alternatively in this development paradigm you only need one senior software dev to work through non compiling and non functioning code to make this whole thing work. Either ways we’re screwed.
- JumpCrisscross 3y ago> only a one token error to have code that doesn't compile, execute, throws exceptions, or to be wrong in some serious way This was also true in the early days of cars, planes and—for that matter—computers. Most coding, even today, doesn’t require a full-fledged engineer.
- ssdspoimdsjvv 3y agoSoon, your compiler will interact with LLMs to fix the mistake automatically. In fact, maybe the LLM becomes the compiler.
- lostmsu 3y agoThis is actually a genius idea. The thing is: it probably already exists, just not packaged nicely yet.