9 ms·

How to tackle unreliability of coding assistants

159 points by ingve 3y ago

DonaldPShimoda 3y ago
I'm just waiting for people to catch on that using an inherently unreliable tool that cannot gauge its own reliability to generate code or answer fact-based queries is a fool's errand. But I expect we'll spend a lot of effort on "verifying" the results before we just give up entirely.
- onethought 3y ago
  Don’t humans fit that definition? We’ve managed okay for 1000s of years under those conditions.
  - bena 3y ago
    They do not. A human can verify its own reliability.
    - ryanklee 3y ago
      A human cannot do this self-sufficiently. This is why we work so hard to implement risk mitigation measures and checks and balances on changes.
      - bena 3y ago
        Yes, not all humans at all tasks all the time. And some things are important enough to implement checks regardless. However, there's a lot we can just throw humans at and trust that the thing will get complete and be correct. And even with the checks and balances, we can have the human perform those checks and balances. A human is a pretty autonomous unit on average. So far, AI can't really say "let me double check that for you" for instance. You ask it a thing, it does a thing, and that's it. If it's wrong, you have to tell it to do the thing again, but differently. In all the rush to paint these LLMs as "pretty much human", we've instead taken to severely downplaying just how adaptable and clever most sentient beings can be.
        ryanklee 3y ago
        AIs can double check. Agent to agent confirmation and validation is a thing. In any case, the point is that we have learned techniques to compensate for human fallibility. We will learn techniques to compensate for gen AI fallibility, as well. The objection that AIs can be wrong is far less a barrier to the rise of their utility than is often supposed.
        bena 3y ago
        You see how that's worse, right? The original argument put forth was that "an inherently unreliable tool cannot gauge its own reliability". Someone responded that humans fit that description as well. I said we don't. We can and do verify our own reliability. We can essentially test our assumptions against the real world and fix them. You then claimed we couldn't do that "self-sufficiently". I responded that while that is true for some tasks, for a lot of tasks, we can. That an AI can't check itself and won't even try. And now you're telling me that they can check against each other. But if you can't trust the originals, asking them if they trust each other is kind of pointless. You're not really doing anything more than adding another layer. For example: If I put my pants on backwards, I correct that myself. Without the need to check and/or balance against any other person. I am self-correcting to a large degree. The AI would not even know to check its pants until someone told it it was wrong. The objection isn't that "AIs can be wrong", the objection is that AIs can't really tell the difference between correct and incorrect. So everything has to be checked, often with as much effort as it would take to do the thing in the first place.
        ryanklee 3y ago
        This is a very narrow view of how LLMs can interact to improve inference accuracy. Different LLMs have different capabilities. They can be used in coordination to improve results. Your objections seem to rely on a restricted view that says "we can't do better" but with no evidence. Whereas we have plenty of evidence of massive, continual improvement in the very areas you are holding up as problematic.
    - cornel_io 3y ago
      If that was even close to true then I would have had to fire far fewer people over the years.
  - ryanklee 3y ago
    They do fit this definition. One result of the rise of generative AI is exposing just how severely and commonly people misperceive their own capabilities and the functioning of their cognitive powers.
  - trefoiled 3y ago
    I hear this argument applied often when people bring up the deficiencies of AI, and I don't find it convincing. Compare an AI coding assistant to reaching out to another engineer on my team as an example. If I know this engineer, I will likely have an idea of their relative skill level, their familiarity with the problem at hand, their propensity to suggest one type of solution over another, etc. People are pretty good at developing this kind of sense because we work with other people constantly. The AI assistant, on the other hand, is very much not like a human. I have a limited capacity to understand its "thought process," and I consider myself far more technical than the average person. This makes a verification step troublesome, because I don't know what to expect. This difference is even more stark when it comes to driving assistants. Video compilations of Teslas with FSD behaving erratically and most importantly, unpredictably, are all over the place. Experienced Tesla drivers seem to have some limited ability to predict the weaknesses of the FSD package, but the issue is that the driving assistant is so unlike a human. I've seen multiple examples of people saying "well, humans cause car crashes too," but the key difference is that I have to sit behind the wheel and deal with the fact that my driving assistant may or may not suddenly swerve into oncoming traffic. The reasons for it doing so are likely obscure to me, and this is a real problem.
  - abeppu 3y ago
    Humans are unreliable, but we are also under normal circumstances thoroughly and continually grounded in an external world whose mechanics we interact with, make predictions about, and correct our beliefs about. The specific way we're training coding assistants for next-token-prediction would also be an incredibly difficult context for humans to produce code. Suppose you were dropped off in an society of aliens whose perceptual, cultural and cognitive universe is meaningfully different from our own; you don't have a grounding in concepts of what they're trying to _do_ with their programs. You receive a giant dump of reams and reams of source code, in their unfamiliar script, where none of the names initially mean anything to you. In the pile of training material handed to you, you might find some documentation about their programming language, but it's written in their (foreign, weird to you) natural language, and is mixed with everythign else. You never get a teacher who can answer questions, never get access to a IDE/repl/interpreter/debugger/compiler, never get to _run_ a program on different inputs to see its outputs, never get to add a log line to peek at the program's internal state, etc. After a _lot_ of training, you can often predict the next symbol in a program text. But shouldn't we _expect_ you to be "unreliable"? You don't have the ability to run checks against the code you produce! You don't get a warning if you use a variable that doesn't exist! You just produce _tokens_, and get no feedback. To the degree humans are reliable at coding, it's because we can simulate what program execution will do, with a level of abstraction which we vary in a task dependent way. You can mentally step through every line in a program carefully if you need to. But you can also mentally choose to trust some abstraction and skip steps which you infer cannot be related to some attribute or condition of interest if that abstraction is upheld. The most important parts of your attention are on _what the program does_. This is fully hidden in the next-token-prediction scenario, which is totally focused on _what tokens are used to write the program_.
- kazinator 3y ago
  I've written a few pieces of code with the help of AI. The trick is that I don't need the help. I could see where the bugs are. The AI could do certain things much faster than I would be able to by hand. For instance, in a certain hash table containing structures, it turned out that the deletion algorithm which moves elements couldn't be used because other code retains (reference counted) pointers to the structures, so the addresses of the object have to be stable. AI easily and correctly rewrote the code from "array of struct" to "array of pointer to struct" representation: the declaration of the thing and all the code working on it was correctly adjusted. I can do that myself, but not in two seconds.
- throwuwu 3y ago
  Just wait a few months. Verifiability is the technique du jour and will make its way into a variety of models soon enough.
  - westoncb 3y ago
    Seems like there are certain fundamental limits to what can be done here though. Much of the advantage to using these models is being able to come up with a vague/informal spec and most of the time have it get what you mean and come up with something serviceable with very little effort. If the spec you have in mind to begin with is fuzzy and informal, what do you use to perform verification? After all, whether a result is correct or not depends on whether it matches the user's desire, so verification criteria must come from that source too. Sure there are certain types of relatively objective correctness that most of the time will line up with a user's desires, but this kind of verification can never be complete afaict.
    - throwuwu 3y ago
      It’s a multi turn conversation, just workshop the idea and make changes along the way
- coffeecantcode 3y ago
  .
  - agentdrek 3y ago
    is that a vi repeat? lol
    - kevindamm 3y ago
      yyp
- willsmith72 3y ago
  I could not disagree more. Having something generate code which is 90-100% correct is extremely valuable. E.g. creating a new page in an app. Feed an LLM the design, the language/framework/component library you're using, and get a page which is 90% of the way there. Some tweaks and you're good. Far, far quicker than going by hand, and often better quality code than a junior developer. Now I would never deploy that code without reading it, understanding it, and testing it, but I would always do that anyway. GPT4 is close enough to a good software engineer, that to resist is it to disadvantage your business. Now if you're coding for pleasure and the fun of creation and creativity, then ditch the LLMs, they take some of that fun away for sure. But most of the time, I'm focused on achieving business outcomes quickly. That's way more productive with a modern LLM
  - bigstrat2003 3y ago
    I strongly disagree. Having something which is 90% reliable doesn't save me any effort over just doing it myself. If I have to spend the time to check everything the tool generates (which I do, because LLMs are unreliable) then I may as well have written it myself to start with. I firmly believe that the worst kind of help is unreliable help. If your help never does its job, then you know you have to do everything yourself. If your help always does its job, you know you can trust it. If your help only sometimes does its job, you get the worst deal of all because you never know what you'll get and have to check every single time.
    - willsmith72 3y ago
      > If I have to spend the time to check everything the tool generates (which I do, because LLMs are unreliable) then I may as well have written it myself to start with. Do you feel the same way about a PR from a coworker?
      - diputsmonro 3y ago
        Depends on the coworker. I would say that I trust ChatGPT code much less than the average coworker, because coworkers usually don't do something like use imaginary variables or check-in code that doesn't compile.
kazinator 3y ago
The way you tackle the unreliability of coding assistants is to know what you're doing so that you don't need a coding assistant, so that you're only using it to save time and effort. Roughly speaking, if you stick to using AI for writing code you could have written yourself, you're okay.
- NateEag 3y ago
  Though regular use of the assistant will degrade your ability to program without it.
  - majormajor 3y ago
    I think in the same way that autocomplete does - I may not have typed out `.length` or whatever it happens to be in the language I used last year enough to remember a year later if it's len or length or size and if it's a property or a method after a year goes by without me touching that language... but then it's a simple search away to refresh my memory, and overall the autocomplete saves a LOT of time that makes up for this. Yeah, if you never knew what the code that got generated did in the first place, that's not gonna apply, but if you're using it as basically just a code expander for things you could do the pseudocode for in your sleep, you're probably gonna be ok.
- digging 3y ago
  I've only used a very small amount of AI assistance (mostly Anthropic's Claude), and always to learn, not to do. That is, I will ask it what's happening, why are things breaking, etc. It doesn't have to have the right answers, it just needs to unblock me. I hear it is also quite useful for doing things which you know extremely well but are tedious to do. Anything in between is certainly a danger zone.
cleandreams 3y ago
I have several issues with coding assistants. Over time, will less skilled programmers produce more critical code? I think so. At some point a jet will fall out of the sky because the coding assistant wasn't correct and the whole profession will have a black eye. The programmers will be less skilled because the (up until recently) lack of coding assistants provides a more rigorous and thorough training environment. With coding assistants the profession will be less intellectually selective because a wider range of people can do it. I don't think this is good given how critical some code is. There is another related issue. Studies have shown that use of google maps has resulted in a deterioration of people's ability to navigate. Human mapping and navigation ability needs training which it doesn't get when google maps are used. This sort of thing will be an issue with coding.
- SoftTalker 3y ago
  This will be an issue with all knowledge work. The machines will have the knowledge and more and more we will just trust them because we don't know ourselves. Google Maps is a great example.
  - hamburga 3y ago
    I want to build a "Google Maps that doesn't make you dumber." For local navigation, first and foremost. The goal is to teach you how to navigate your locale, so you use it less and less. You still will want to ask it for traffic updates, but you talk to it like you would between locals who know all the roads. As a model for how to do AI in a way that enhances your thinking rather than softens/replaces it.
    - mejutoco 3y ago
      I thought about something like this as well. Maybe it can suggest new routes so you can know new streets. It could work by vibration (1 vibration right, 2 left) like some running apps do so you don't even have to look at your phone and can focus on what is in front of you.
- bena 3y ago
the8472 3y ago
Are there tools yet that put the compiler in the loop? Maybe even a check-compile-test loop that bails on the first failure and then tries to refine it in the background based on what failed?
- bee_rider 3y ago
  Then maybe the model could be fine-tuned on that loop, haha. Could be a fun thing to try at least.
  - dontupvoteme 3y ago
    It's a bit of fools-errand because they train on information which is no longer valid and will get stuck if you don't inform them. For instance GPT cannot write a connection to an openai endpoint because the API was upgraded to 1.0 and broke compatibility with all the code it learned from
    - bloopernova 3y ago
      Is it possible for an LLM to have knowledge/input marked as "deprecated/obsolete"? Either by a user or some sort of "retraining" process?
- simonw 3y ago
  Yes. ChatGPT Code Interpreter mode does exactly this - it writes code, runs it through a Python interpreter, then if there are any errors it rewrites the code and runs it until it works. ... and you can use it to run a C compiler too, if you know what you are doing! https://simonwillison.net/2023/Oct/17/open-questions/#open-questions.037.jpeg https://simonwillison.net/2023/Oct/17/open-questions/#open-q...
  - the8472 3y ago
    Does that compile my local projects?
    - 3y ago
righthand 3y ago
Honestly I’d rather just RTFM and code it myself than invent a game to deal with these issues.
meindnoch 3y ago
I'm happy that more and more people embrace these tools for more and more critical software. It keeps me employed, and even increases my rate quite a bit.
- bee_rider 3y ago
  Are you an EMT?
abeppu 3y ago
I think this is "how to think about coding assistants and your task" but none of this is "tackling" their unreliability. While coding assistants seem to do well in a range of situations, I continue to believe that for coding specifically, merely training on next-token-prediction is leaving too much on the table. Yes, source code is represented as text, but computer programs are an area where there's available information which is _so much richer_. We can know not only the text of the program but the type of every expression, which variables are in scope at any point, what is the signature of a method we're trying to call, etc. These assistants should be able to make predictions about program _traces_, not just program source text. A step further would be to guess potential loop invariants, pre/post conditions, etc, confirm which are upheld by existing tests, and down-weight recommending changes which introduce violations to those inferred conditions. ChatGPT and tab-completion assistants have both given me things that are not even valid programs (e.g. will not compile, use a variable that isn't actually in scope, etc). ChatGPT even told me that an example it generated wasn't compiling for me b/c I wasn't using a new enough version of the language, and then referenced a language version which does not yet exist. All of this is possible in part b/c these tools are engaging only at the level of text, and are structurally isolated from the rich information available inside an interpreter or debugger. "Tackling" unreliability should start with reframing tasks in a way which lets tools better see the causes of their failures.
- jkaptur 3y ago
  I absolutely agree, but I find the situation incredibly funny. There are three characters here: me, the LLM, and the compiler. Two of them are robots, but they refuse to talk to each other - it's up to me to tell the LLM the bad news from the compiler.
  - sa-code 3y ago
    This sounds like the worst game of telephone
  - fragmede 3y ago
    that's a frontend issue though. If you use the python interpreter in ChatGPT, you can tell it to run the code until it works, at which point it'll do a couple of iterations before giving you code.
wrs 3y ago
Several times when I’ve asked ChatGPT for an approach to something, it has spit out code that uses an API that looks perfect for my use case, but doesn’t actually exist. So I’m thinking someone should be building an automated product manager!
- jwells89 3y ago
  Have had the same experience several times. “Yes ChatGPT, it makes perfect sense for that to exist and would be wonderful if it did, but unfortunately it does not.”
- gnulinux 3y ago
  This is what ChatGPT does to me whenever I ask anything non-trivial. I find it funny people think it'll take over our jobs, it simply can't even do the most basic things beyond the beaten path.
  - airstrike 3y ago
    are you using GPT-4? because it's been incredibly helpful to me. major refactoring gets done in a matter of seconds, and when it doesn't work, I can just paste whatever compiler error I'm getting and it usually fixes the problem
    - ozim 3y ago
      Can you give an example of what you consider “major refactoring”?
  - droopyEyelids 3y ago
    [flagged]
- eru 3y ago
  Perhaps that means you should implement that API.
anigbrowl 3y ago
In practice I find a bigger problem is perversity - AI assistant doing OK with incremental prompting, but sometimes decides to just remove all comments from the code, or if asked to focus its effort on one section, deletes all other code. Code assistants need to be much integrated with IDEs and I think you probably need 2 or 3 running in parallel, maybe more.
- sevagh 3y ago
  Yes, this really irritates me. Me: <prompt 1: modify this function> AI assistant (either ChatGPT or Copilot-X): <attempt 1> Me: <feedback> AI assistant: <attempt 2>, fixed with feedback, but deleted something crucial from attempt 1, for no reason at all
  - all2 3y ago
    To avoid stuff like this I've tried to prompt for only the necessary changes. I haven't found a good prompt that does this repeatably. It might be worth some experimentation.
    - anigbrowl 3y ago
      One can usually get around it by modifying the prompt, but one really shouldn't have to include things like 'do not remove comments' over and over. My IDE (shoutout to Jetbrains) lets me make custom prompts, but the real problems here are the transformer model's inability to maintain context or ask questions of its own, more readily, and a lack of visual grammar for 'what are we both talking about, and how'. Pair programming has gone some way toward improving that in recent years, but realistically transformers are getting to where they can spit out code faster than most people can compose or type it (not you of course). Looking at live diffs is not ideal, especially late in a coding session when fatigue is setting in. Some sort of shared context map for functions and variables might be a better approach. As things are I find myself using a clumsy mix of save-as and commits and often throwing away hours of work when an assistant is unable to backtrack.
    - CGamesPlay 3y ago
darkteflon 3y ago
I feel like the work on using CFGs with LLMs should be low-hanging fruit for improving code assistants, but perhaps someone more knowledgeable could chime in[1], [2], [3]. At lot of the confabulations we see today - non-existent variables, imports and APIs, out-of-scope variables, etc - would seem (to me) to be meaningfully addressable with these techniques. Relatedly, I have gotten surprisingly great mileage out of treating confabulations, in the first instance, as a user (ie, my own) failure to adequately contextualise, rather than error. In a sense, CFGs give you sharper tools to do that contextualisation. I wonder how far the “sharper tools” approach will carry us. It seems, to this interested layman, consistent with Shannon’s work in statistical language modelling.[4] The term “prompt engineering” implies, to me, a focus on the instructive aspects of prompting, which is a bit unfortunate but also wholly consistent with the way I see colleagues and friends trying to interact with LLMs. Perhaps we should instead call it “context composition” or something, to emphasise the constructive nature. [1] https://github.com/outlines-dev/outlines https://github.com/outlines-dev/outlines [2] https://github.com/ggerganov/llama.cpp/pull/1773 https://github.com/ggerganov/llama.cpp/pull/1773 [3] https://www.reddit.com/r/LocalLLaMA/comments/156gu8c/d_constraining_llama_models_with_context_free/ https://www.reddit.com/r/LocalLLaMA/comments/156gu8c/d_const... [4] https://hedgehogreview.com/issues/markets-and-the-good/articles/language-machinery https://hedgehogreview.com/issues/markets-and-the-good/artic...
firebot 3y ago
Maybe learn to code?
- sciolist 3y ago
  My good-faith interpretation of your comment is: "Maybe learn to code (without external resources or tools)?" LLMs are another tool and resource, just like StackOverflow, linters, autocomplete, Google, etc. None of these tools are infallible, but they provide value. Just like all other tools, you don't need to use LLMs because of their issues - but we want them to be as useful as possible - what the author is trying to do.
slalomskiing 3y ago
Sounds like an exercise in frustration
vishnumenon 3y ago
My current hypothesis here is that the way to make coding assistants as reliable as possible is to shift the balance towards making their output rely on context provided in-prompt rather than information stored in LLM weights. As all the major providers shift towards larger context-windows, it seems increasingly viable to give the LLM the necessary docs for whatever libraries are being used in the current file. I've been working on an experiment in this space[0], and while it's obviously bottle-necked by the size of the documentation index, even a couple-hundred documentation sources seems to help a ton when working with less-used languages/libraries. [0]: https://indexical.dev/ https://indexical.dev/
- mr_toad 3y ago
  I like that your solution is basically telling the LLM to RTFM.
  - vishnumenon 3y ago
    Yeah I've been using it with prompts that ask it to cite sources as well, honestly I think the best results are when I'm still interacting w/ the docs directly in addition to having the LLM look at em - still can't quite replace needing to RTFM!
- spadufed 3y ago
  This is the way forward imo. Particularly as we've started to flesh out the relationship between model size and true context reliability. We've found that raw context-window size is not representative of what the model can actually consistently recall, but we've also found the recall is consistently reliable out to a point. I suspect more robust theoretical models around superposition will move us a long way towards understanding the limits of context reliability rather than what would currently be an experimental approach.
tomjakubowski 3y ago
fairly off-topic, sorry: Midjourney gets the cartoon donkey's snout wrong, so wrong. It looks more like a cartoon dog. For some reason I'm really bothered by it.
irrational 3y ago
I just started using copilot last week. I was blown away. I would barely start typing and it would show me anywhere from a single line to 15 lines of exactly what I had been intending to write. It was mind blowing. In one instance, it found reference to data in a totally different part of the page and correctly used it in the code it was creating. I still can't figure out how it did that. I was beyond impressed. Sometimes it was 100% spot on, other times it was 90% of the way there, but it was crazy how much time it saved. Though, I have been programming for many decades, so I was able to tell pretty easily if what it was creating was good or not. I think this could lead a new coder astray.
- chatmasta 3y ago
  Yeah, you really need to experience it first hand to get to that "wow" moment. For me, that moment was when I was writing some code in my over-engineered codebase, with some prematurely generalized abstractions that even I can barely understand. I'd pause for a second to think "wtf am I doing again?" and Copilot would suggest a whole block of code that aligned perfectly with my intent. This happened repeatedly and it felt like I was being guided to a solution. The incredible thing was that I hadn't even thought about it yet. It's like Copilot was thinking a step ahead of me. I found this effect to be most pronounced when writing tests. I think Copilot shines in codebases with static typing, clear interfaces, doctstrings and unit tests. That's really about the densest, most richly annotated context you could give to an LLM. And that's before adding capabilities for more well-defined reasoning about static languages, types, etc. - there is potential for it to get even better at this.
  - redhale 3y ago
    Writing unit tests is the most effective use of copilot hands-down. For pure functions, it is 100% correct and complete essentially 100% of the time, allowing me to write descriptive test names and nothing more. Even when mocks or spies have to come into the picture, it is usually 95% accurate. The key is that you should always have the file you are testing open in an editor tab, as well as another test class that demonstrates the testing style you want it to emulate. Forget everything else about Copilot, it's worth the cost for this alone. Time writing tests reduced ~80%. They could remove all other functionality and rebrand it as Test Copilot, and they would still get my money.
PH95VuimJjqBqy 3y ago
I basically refuse to use any sort of AI assisted autocomplete for this very reason. I want consistency more than accuracy. consistency enables power and reliability, if it changes every 3rd time on the same input it slows me down.
bmitc 3y ago
Not use them?
simonw 3y ago
If you're going to be productive with LLM coding assistants, the skill you most need to develop is a strong QA process. You need to be really good at running code, trying edge-cases and quickly getting to a point where you have proven to yourself that the code works. Having good habits around automated testing helps with this a lot. If you're already good at this stuff, you'll find the risk of coding assistants getting things wrong is pretty minimal for you. If you have bad habits where you frequently write and commit code without first executing it and trying to poke holes in it, you'll find AI assisted coding full of traps.
Obscurity4340 3y ago
[]
- eru 3y ago
  Why? And why allow autocomplete? Why not ban anything including compilers, especially optimizing compilers? How do you draw the line?
eetus 3y ago
It's confusing that you had to scroll up to realize the article isn't by Martin Fowler.
calamari4065 3y ago
I view these LLM tools as just another code generator. I have no idea how many bespoke single-use code generators I've written, but I know that I've spent a nontrivial amount of time writing code that writes code. If an LLM can do it with a natural language interface, it'll save me a lot of time. But also, I would never trust a script I threw together in 15 minutes to actually produce real code and solve the problem. All it does is generate the text I tell it to. It can't understand the system or how it works, it just procedurally spits out text. In the same way that a dozen lines of Python cannot understand your program, an LLM is also fundamentally incapable of understanding it. That's the crucial part of programming. An LLM will give you text all day, but it can't write your program. Sure an LLM can produce trivial scripts, but in my experience so far, it can only reliably generate trivial programs that I could write blindfolded in the same amount of time. If we just treat these tools like the text generators they are instead of insisting they're code generators, we'll all be better off.
jillesvangurp 3y ago
I've been using chat gpt plus quite a bit for supporting me in programming tasks. It's too flaky to trust blindly and you need to narrowly scope what it does. It can also be a bit hand wavy with some things and I've had issues getting it to generate working code for some more complicated things. But where it really shines is in documenting things, explaining things, and suggesting problem areas in code and possible solutions. And you can do fun things like point it at an openapi schema and asking it some questions about that API. I've given it screenshots of websites and asked it to criticize the design or document what is visible. It's amazingly good at supporting localization work. I was working with some geospatial code for an algorithm that generates UTM coordinates from GPS coordinates a few weeks ago. I needed a Kotlin implementation that I could use on multiple of its platforms (js and jvm, i.e. no java dependencies allowed). It was kind of useless writing the code for me for this (that's the first thing I tried obviously) but it unblocked me a couple of times and helped me figure out some details. I ultimately found several old Java implementations that contained a lot of undocumented magical numbers and I just asked it the meaning of those numbers (about a dozen) and it came back with good explanations (basically things like wgs84 ellipsoid parameters, radius of the earth, etc.). The code wasn't quite working (it failed a test I wrote) so I asked it to identify possible causes for that and it came up with some uncovered edge cases that I was able to cross confirm with other implementations. In the end I was able to piece together a working implementation by combining different elements from several implementations. Each of them individually had issues. A lot of this code is ancient and there is apparently a lot of copy paste reuse in this space.
russfink 3y ago
I use it like a man page on steroids. “Write an example bash program that implements a set data structure, and demonstrate it with examples.”
mikerg87 3y ago
I see lots of people saying the code is suspect, but I never hear about using LLMs for writing tests to validate their own code or LLM code. My success has been in generating test. Often faster and simpler than I would write by myself. What I test I can trust, especially if it comes out of an LLM.
fuzzfactor 3y ago
>How to tackle unreliability of coding assistants One of the well-established techniques was always to give more worthwhile incentives that are especially meaningful to the hardworking assistants.
CodeWriter23 3y ago
I dunno, maybe hammer out a couple dozen lines of code sourced from your own brain?

159 points by ingve 3y ago

I'm just waiting for people to catch on that using an inherently unreliable tool that cannot gauge its own reliability to generate code or answer fact-based queries is a fool's errand. But I expect we'll spend a lot of effort on "verifying" the results before we just give up entirely.

onethought 3y ago
Don’t humans fit that definition? We’ve managed okay for 1000s of years under those conditions.
- bena 3y ago
  They do not. A human can verify its own reliability.
  - ryanklee 3y ago
    A human cannot do this self-sufficiently. This is why we work so hard to implement risk mitigation measures and checks and balances on changes.
    - bena 3y ago
      Yes, not all humans at all tasks all the time. And some things are important enough to implement checks regardless. However, there's a lot we can just throw humans at and trust that the thing will get complete and be correct. And even with the checks and balances, we can have the human perform those checks and balances. A human is a pretty autonomous unit on average. So far, AI can't really say "let me double check that for you" for instance. You ask it a thing, it does a thing, and that's it. If it's wrong, you have to tell it to do the thing again, but differently. In all the rush to paint these LLMs as "pretty much human", we've instead taken to severely downplaying just how adaptable and clever most sentient beings can be.
      - ryanklee 3y ago
        AIs can double check. Agent to agent confirmation and validation is a thing. In any case, the point is that we have learned techniques to compensate for human fallibility. We will learn techniques to compensate for gen AI fallibility, as well. The objection that AIs can be wrong is far less a barrier to the rise of their utility than is often supposed.
        bena 3y ago
        You see how that's worse, right? The original argument put forth was that "an inherently unreliable tool cannot gauge its own reliability". Someone responded that humans fit that description as well. I said we don't. We can and do verify our own reliability. We can essentially test our assumptions against the real world and fix them. You then claimed we couldn't do that "self-sufficiently". I responded that while that is true for some tasks, for a lot of tasks, we can. That an AI can't check itself and won't even try. And now you're telling me that they can check against each other. But if you can't trust the originals, asking them if they trust each other is kind of pointless. You're not really doing anything more than adding another layer. For example: If I put my pants on backwards, I correct that myself. Without the need to check and/or balance against any other person. I am self-correcting to a large degree. The AI would not even know to check its pants until someone told it it was wrong. The objection isn't that "AIs can be wrong", the objection is that AIs can't really tell the difference between correct and incorrect. So everything has to be checked, often with as much effort as it would take to do the thing in the first place.
        ryanklee 3y ago
        This is a very narrow view of how LLMs can interact to improve inference accuracy. Different LLMs have different capabilities. They can be used in coordination to improve results. Your objections seem to rely on a restricted view that says "we can't do better" but with no evidence. Whereas we have plenty of evidence of massive, continual improvement in the very areas you are holding up as problematic.
  - cornel_io 3y ago
    If that was even close to true then I would have had to fire far fewer people over the years.
- ryanklee 3y ago
  They do fit this definition. One result of the rise of generative AI is exposing just how severely and commonly people misperceive their own capabilities and the functioning of their cognitive powers.
- trefoiled 3y ago
  I hear this argument applied often when people bring up the deficiencies of AI, and I don't find it convincing. Compare an AI coding assistant to reaching out to another engineer on my team as an example. If I know this engineer, I will likely have an idea of their relative skill level, their familiarity with the problem at hand, their propensity to suggest one type of solution over another, etc. People are pretty good at developing this kind of sense because we work with other people constantly. The AI assistant, on the other hand, is very much not like a human. I have a limited capacity to understand its "thought process," and I consider myself far more technical than the average person. This makes a verification step troublesome, because I don't know what to expect. This difference is even more stark when it comes to driving assistants. Video compilations of Teslas with FSD behaving erratically and most importantly, unpredictably, are all over the place. Experienced Tesla drivers seem to have some limited ability to predict the weaknesses of the FSD package, but the issue is that the driving assistant is so unlike a human. I've seen multiple examples of people saying "well, humans cause car crashes too," but the key difference is that I have to sit behind the wheel and deal with the fact that my driving assistant may or may not suddenly swerve into oncoming traffic. The reasons for it doing so are likely obscure to me, and this is a real problem.
- abeppu 3y ago
  Humans are unreliable, but we are also under normal circumstances thoroughly and continually grounded in an external world whose mechanics we interact with, make predictions about, and correct our beliefs about. The specific way we're training coding assistants for next-token-prediction would also be an incredibly difficult context for humans to produce code. Suppose you were dropped off in an society of aliens whose perceptual, cultural and cognitive universe is meaningfully different from our own; you don't have a grounding in concepts of what they're trying to _do_ with their programs. You receive a giant dump of reams and reams of source code, in their unfamiliar script, where none of the names initially mean anything to you. In the pile of training material handed to you, you might find some documentation about their programming language, but it's written in their (foreign, weird to you) natural language, and is mixed with everythign else. You never get a teacher who can answer questions, never get access to a IDE/repl/interpreter/debugger/compiler, never get to _run_ a program on different inputs to see its outputs, never get to add a log line to peek at the program's internal state, etc. After a _lot_ of training, you can often predict the next symbol in a program text. But shouldn't we _expect_ you to be "unreliable"? You don't have the ability to run checks against the code you produce! You don't get a warning if you use a variable that doesn't exist! You just produce _tokens_, and get no feedback. To the degree humans are reliable at coding, it's because we can simulate what program execution will do, with a level of abstraction which we vary in a task dependent way. You can mentally step through every line in a program carefully if you need to. But you can also mentally choose to trust some abstraction and skip steps which you infer cannot be related to some attribute or condition of interest if that abstraction is upheld. The most important parts of your attention are on _what the program does_. This is fully hidden in the next-token-prediction scenario, which is totally focused on _what tokens are used to write the program_.
kazinator 3y ago
I've written a few pieces of code with the help of AI. The trick is that I don't need the help. I could see where the bugs are. The AI could do certain things much faster than I would be able to by hand. For instance, in a certain hash table containing structures, it turned out that the deletion algorithm which moves elements couldn't be used because other code retains (reference counted) pointers to the structures, so the addresses of the object have to be stable. AI easily and correctly rewrote the code from "array of struct" to "array of pointer to struct" representation: the declaration of the thing and all the code working on it was correctly adjusted. I can do that myself, but not in two seconds.
throwuwu 3y ago
Just wait a few months. Verifiability is the technique du jour and will make its way into a variety of models soon enough.
- westoncb 3y ago
  Seems like there are certain fundamental limits to what can be done here though. Much of the advantage to using these models is being able to come up with a vague/informal spec and most of the time have it get what you mean and come up with something serviceable with very little effort. If the spec you have in mind to begin with is fuzzy and informal, what do you use to perform verification? After all, whether a result is correct or not depends on whether it matches the user's desire, so verification criteria must come from that source too. Sure there are certain types of relatively objective correctness that most of the time will line up with a user's desires, but this kind of verification can never be complete afaict.
  - throwuwu 3y ago
    It’s a multi turn conversation, just workshop the idea and make changes along the way
coffeecantcode 3y ago
.
- agentdrek 3y ago
  is that a vi repeat? lol
  - kevindamm 3y ago
    yyp
willsmith72 3y ago
I could not disagree more. Having something generate code which is 90-100% correct is extremely valuable. E.g. creating a new page in an app. Feed an LLM the design, the language/framework/component library you're using, and get a page which is 90% of the way there. Some tweaks and you're good. Far, far quicker than going by hand, and often better quality code than a junior developer. Now I would never deploy that code without reading it, understanding it, and testing it, but I would always do that anyway. GPT4 is close enough to a good software engineer, that to resist is it to disadvantage your business. Now if you're coding for pleasure and the fun of creation and creativity, then ditch the LLMs, they take some of that fun away for sure. But most of the time, I'm focused on achieving business outcomes quickly. That's way more productive with a modern LLM
- bigstrat2003 3y ago
  I strongly disagree. Having something which is 90% reliable doesn't save me any effort over just doing it myself. If I have to spend the time to check everything the tool generates (which I do, because LLMs are unreliable) then I may as well have written it myself to start with. I firmly believe that the worst kind of help is unreliable help. If your help never does its job, then you know you have to do everything yourself. If your help always does its job, you know you can trust it. If your help only sometimes does its job, you get the worst deal of all because you never know what you'll get and have to check every single time.
  - willsmith72 3y ago
    > If I have to spend the time to check everything the tool generates (which I do, because LLMs are unreliable) then I may as well have written it myself to start with. Do you feel the same way about a PR from a coworker?
    - diputsmonro 3y ago
      Depends on the coworker. I would say that I trust ChatGPT code much less than the average coworker, because coworkers usually don't do something like use imaginary variables or check-in code that doesn't compile.

kazinator 3y ago

The way you tackle the unreliability of coding assistants is to know what you're doing so that you don't need a coding assistant, so that you're only using it to save time and effort. Roughly speaking, if you stick to using AI for writing code you could have written yourself, you're okay.

NateEag 3y ago
Though regular use of the assistant will degrade your ability to program without it.
- majormajor 3y ago
  I think in the same way that autocomplete does - I may not have typed out `.length` or whatever it happens to be in the language I used last year enough to remember a year later if it's len or length or size and if it's a property or a method after a year goes by without me touching that language... but then it's a simple search away to refresh my memory, and overall the autocomplete saves a LOT of time that makes up for this. Yeah, if you never knew what the code that got generated did in the first place, that's not gonna apply, but if you're using it as basically just a code expander for things you could do the pseudocode for in your sleep, you're probably gonna be ok.
digging 3y ago
I've only used a very small amount of AI assistance (mostly Anthropic's Claude), and always to learn, not to do. That is, I will ask it what's happening, why are things breaking, etc. It doesn't have to have the right answers, it just needs to unblock me. I hear it is also quite useful for doing things which you know extremely well but are tedious to do. Anything in between is certainly a danger zone.

cleandreams 3y ago

I have several issues with coding assistants. Over time, will less skilled programmers produce more critical code? I think so. At some point a jet will fall out of the sky because the coding assistant wasn't correct and the whole profession will have a black eye. The programmers will be less skilled because the (up until recently) lack of coding assistants provides a more rigorous and thorough training environment. With coding assistants the profession will be less intellectually selective because a wider range of people can do it. I don't think this is good given how critical some code is. There is another related issue. Studies have shown that use of google maps has resulted in a deterioration of people's ability to navigate. Human mapping and navigation ability needs training which it doesn't get when google maps are used. This sort of thing will be an issue with coding.

SoftTalker 3y ago
This will be an issue with all knowledge work. The machines will have the knowledge and more and more we will just trust them because we don't know ourselves. Google Maps is a great example.
- hamburga 3y ago
  I want to build a "Google Maps that doesn't make you dumber." For local navigation, first and foremost. The goal is to teach you how to navigate your locale, so you use it less and less. You still will want to ask it for traffic updates, but you talk to it like you would between locals who know all the roads. As a model for how to do AI in a way that enhances your thinking rather than softens/replaces it.
  - mejutoco 3y ago
    I thought about something like this as well. Maybe it can suggest new routes so you can know new streets. It could work by vibration (1 vibration right, 2 left) like some running apps do so you don't even have to look at your phone and can focus on what is in front of you.
bena 3y ago

the8472 3y ago

Are there tools yet that put the compiler in the loop? Maybe even a check-compile-test loop that bails on the first failure and then tries to refine it in the background based on what failed?

bee_rider 3y ago
Then maybe the model could be fine-tuned on that loop, haha. Could be a fun thing to try at least.
- dontupvoteme 3y ago
  It's a bit of fools-errand because they train on information which is no longer valid and will get stuck if you don't inform them. For instance GPT cannot write a connection to an openai endpoint because the API was upgraded to 1.0 and broke compatibility with all the code it learned from
  - bloopernova 3y ago
    Is it possible for an LLM to have knowledge/input marked as "deprecated/obsolete"? Either by a user or some sort of "retraining" process?
simonw 3y ago
Yes. ChatGPT Code Interpreter mode does exactly this - it writes code, runs it through a Python interpreter, then if there are any errors it rewrites the code and runs it until it works. ... and you can use it to run a C compiler too, if you know what you are doing! https://simonwillison.net/2023/Oct/17/open-questions/#open-questions.037.jpeg https://simonwillison.net/2023/Oct/17/open-questions/#open-q...
- the8472 3y ago
  Does that compile my local projects?
  - 3y ago

righthand 3y ago

Honestly I’d rather just RTFM and code it myself than invent a game to deal with these issues.

meindnoch 3y ago

I'm happy that more and more people embrace these tools for more and more critical software. It keeps me employed, and even increases my rate quite a bit.

bee_rider 3y ago
Are you an EMT?

abeppu 3y ago

I think this is "how to think about coding assistants and your task" but none of this is "tackling" their unreliability. While coding assistants seem to do well in a range of situations, I continue to believe that for coding specifically, merely training on next-token-prediction is leaving too much on the table. Yes, source code is represented as text, but computer programs are an area where there's available information which is _so much richer_. We can know not only the text of the program but the type of every expression, which variables are in scope at any point, what is the signature of a method we're trying to call, etc. These assistants should be able to make predictions about program _traces_, not just program source text. A step further would be to guess potential loop invariants, pre/post conditions, etc, confirm which are upheld by existing tests, and down-weight recommending changes which introduce violations to those inferred conditions. ChatGPT and tab-completion assistants have both given me things that are not even valid programs (e.g. will not compile, use a variable that isn't actually in scope, etc). ChatGPT even told me that an example it generated wasn't compiling for me b/c I wasn't using a new enough version of the language, and then referenced a language version which does not yet exist. All of this is possible in part b/c these tools are engaging only at the level of text, and are structurally isolated from the rich information available inside an interpreter or debugger. "Tackling" unreliability should start with reframing tasks in a way which lets tools better see the causes of their failures.

jkaptur 3y ago
I absolutely agree, but I find the situation incredibly funny. There are three characters here: me, the LLM, and the compiler. Two of them are robots, but they refuse to talk to each other - it's up to me to tell the LLM the bad news from the compiler.
- sa-code 3y ago
  This sounds like the worst game of telephone
- fragmede 3y ago
  that's a frontend issue though. If you use the python interpreter in ChatGPT, you can tell it to run the code until it works, at which point it'll do a couple of iterations before giving you code.

wrs 3y ago

Several times when I’ve asked ChatGPT for an approach to something, it has spit out code that uses an API that looks perfect for my use case, but doesn’t actually exist. So I’m thinking someone should be building an automated product manager!

jwells89 3y ago
Have had the same experience several times. “Yes ChatGPT, it makes perfect sense for that to exist and would be wonderful if it did, but unfortunately it does not.”
gnulinux 3y ago
This is what ChatGPT does to me whenever I ask anything non-trivial. I find it funny people think it'll take over our jobs, it simply can't even do the most basic things beyond the beaten path.
- airstrike 3y ago
  are you using GPT-4? because it's been incredibly helpful to me. major refactoring gets done in a matter of seconds, and when it doesn't work, I can just paste whatever compiler error I'm getting and it usually fixes the problem
  - ozim 3y ago
    Can you give an example of what you consider “major refactoring”?
- droopyEyelids 3y ago
  [flagged]
eru 3y ago
Perhaps that means you should implement that API.

anigbrowl 3y ago

In practice I find a bigger problem is perversity - AI assistant doing OK with incremental prompting, but sometimes decides to just remove all comments from the code, or if asked to focus its effort on one section, deletes all other code. Code assistants need to be much integrated with IDEs and I think you probably need 2 or 3 running in parallel, maybe more.

sevagh 3y ago
Yes, this really irritates me. Me: <prompt 1: modify this function> AI assistant (either ChatGPT or Copilot-X): <attempt 1> Me: <feedback> AI assistant: <attempt 2>, fixed with feedback, but deleted something crucial from attempt 1, for no reason at all
- all2 3y ago
  To avoid stuff like this I've tried to prompt for only the necessary changes. I haven't found a good prompt that does this repeatably. It might be worth some experimentation.
  - anigbrowl 3y ago
    One can usually get around it by modifying the prompt, but one really shouldn't have to include things like 'do not remove comments' over and over. My IDE (shoutout to Jetbrains) lets me make custom prompts, but the real problems here are the transformer model's inability to maintain context or ask questions of its own, more readily, and a lack of visual grammar for 'what are we both talking about, and how'. Pair programming has gone some way toward improving that in recent years, but realistically transformers are getting to where they can spit out code faster than most people can compose or type it (not you of course). Looking at live diffs is not ideal, especially late in a coding session when fatigue is setting in. Some sort of shared context map for functions and variables might be a better approach. As things are I find myself using a clumsy mix of save-as and commits and often throwing away hours of work when an assistant is unable to backtrack.
  - CGamesPlay 3y ago

darkteflon 3y ago

I feel like the work on using CFGs with LLMs should be low-hanging fruit for improving code assistants, but perhaps someone more knowledgeable could chime in[1], [2], [3]. At lot of the confabulations we see today - non-existent variables, imports and APIs, out-of-scope variables, etc - would seem (to me) to be meaningfully addressable with these techniques. Relatedly, I have gotten surprisingly great mileage out of treating confabulations, in the first instance, as a user (ie, my own) failure to adequately contextualise, rather than error. In a sense, CFGs give you sharper tools to do that contextualisation. I wonder how far the “sharper tools” approach will carry us. It seems, to this interested layman, consistent with Shannon’s work in statistical language modelling.[4] The term “prompt engineering” implies, to me, a focus on the instructive aspects of prompting, which is a bit unfortunate but also wholly consistent with the way I see colleagues and friends trying to interact with LLMs. Perhaps we should instead call it “context composition” or something, to emphasise the constructive nature. [1] https://github.com/outlines-dev/outlines https://github.com/outlines-dev/outlines [2] https://github.com/ggerganov/llama.cpp/pull/1773 https://github.com/ggerganov/llama.cpp/pull/1773 [3] https://www.reddit.com/r/LocalLLaMA/comments/156gu8c/d_constraining_llama_models_with_context_free/ https://www.reddit.com/r/LocalLLaMA/comments/156gu8c/d_const... [4] https://hedgehogreview.com/issues/markets-and-the-good/articles/language-machinery https://hedgehogreview.com/issues/markets-and-the-good/artic...

firebot 3y ago

Maybe learn to code?

sciolist 3y ago
My good-faith interpretation of your comment is: "Maybe learn to code (without external resources or tools)?" LLMs are another tool and resource, just like StackOverflow, linters, autocomplete, Google, etc. None of these tools are infallible, but they provide value. Just like all other tools, you don't need to use LLMs because of their issues - but we want them to be as useful as possible - what the author is trying to do.

slalomskiing 3y ago

Sounds like an exercise in frustration

vishnumenon 3y ago

My current hypothesis here is that the way to make coding assistants as reliable as possible is to shift the balance towards making their output rely on context provided in-prompt rather than information stored in LLM weights. As all the major providers shift towards larger context-windows, it seems increasingly viable to give the LLM the necessary docs for whatever libraries are being used in the current file. I've been working on an experiment in this space[0], and while it's obviously bottle-necked by the size of the documentation index, even a couple-hundred documentation sources seems to help a ton when working with less-used languages/libraries. [0]: https://indexical.dev/ https://indexical.dev/

mr_toad 3y ago
I like that your solution is basically telling the LLM to RTFM.
- vishnumenon 3y ago
  Yeah I've been using it with prompts that ask it to cite sources as well, honestly I think the best results are when I'm still interacting w/ the docs directly in addition to having the LLM look at em - still can't quite replace needing to RTFM!
spadufed 3y ago
This is the way forward imo. Particularly as we've started to flesh out the relationship between model size and true context reliability. We've found that raw context-window size is not representative of what the model can actually consistently recall, but we've also found the recall is consistently reliable out to a point. I suspect more robust theoretical models around superposition will move us a long way towards understanding the limits of context reliability rather than what would currently be an experimental approach.

tomjakubowski 3y ago

fairly off-topic, sorry: Midjourney gets the cartoon donkey's snout wrong, so wrong. It looks more like a cartoon dog. For some reason I'm really bothered by it.

irrational 3y ago

I just started using copilot last week. I was blown away. I would barely start typing and it would show me anywhere from a single line to 15 lines of exactly what I had been intending to write. It was mind blowing. In one instance, it found reference to data in a totally different part of the page and correctly used it in the code it was creating. I still can't figure out how it did that. I was beyond impressed. Sometimes it was 100% spot on, other times it was 90% of the way there, but it was crazy how much time it saved. Though, I have been programming for many decades, so I was able to tell pretty easily if what it was creating was good or not. I think this could lead a new coder astray.

chatmasta 3y ago
Yeah, you really need to experience it first hand to get to that "wow" moment. For me, that moment was when I was writing some code in my over-engineered codebase, with some prematurely generalized abstractions that even I can barely understand. I'd pause for a second to think "wtf am I doing again?" and Copilot would suggest a whole block of code that aligned perfectly with my intent. This happened repeatedly and it felt like I was being guided to a solution. The incredible thing was that I hadn't even thought about it yet. It's like Copilot was thinking a step ahead of me. I found this effect to be most pronounced when writing tests. I think Copilot shines in codebases with static typing, clear interfaces, doctstrings and unit tests. That's really about the densest, most richly annotated context you could give to an LLM. And that's before adding capabilities for more well-defined reasoning about static languages, types, etc. - there is potential for it to get even better at this.
- redhale 3y ago
  Writing unit tests is the most effective use of copilot hands-down. For pure functions, it is 100% correct and complete essentially 100% of the time, allowing me to write descriptive test names and nothing more. Even when mocks or spies have to come into the picture, it is usually 95% accurate. The key is that you should always have the file you are testing open in an editor tab, as well as another test class that demonstrates the testing style you want it to emulate. Forget everything else about Copilot, it's worth the cost for this alone. Time writing tests reduced ~80%. They could remove all other functionality and rebrand it as Test Copilot, and they would still get my money.

PH95VuimJjqBqy 3y ago

I basically refuse to use any sort of AI assisted autocomplete for this very reason. I want consistency more than accuracy. consistency enables power and reliability, if it changes every 3rd time on the same input it slows me down.

bmitc 3y ago

Not use them?

simonw 3y ago

If you're going to be productive with LLM coding assistants, the skill you most need to develop is a strong QA process. You need to be really good at running code, trying edge-cases and quickly getting to a point where you have proven to yourself that the code works. Having good habits around automated testing helps with this a lot. If you're already good at this stuff, you'll find the risk of coding assistants getting things wrong is pretty minimal for you. If you have bad habits where you frequently write and commit code without first executing it and trying to poke holes in it, you'll find AI assisted coding full of traps.

Obscurity4340 3y ago

[]

eru 3y ago
Why? And why allow autocomplete? Why not ban anything including compilers, especially optimizing compilers? How do you draw the line?

eetus 3y ago

It's confusing that you had to scroll up to realize the article isn't by Martin Fowler.

calamari4065 3y ago

I view these LLM tools as just another code generator. I have no idea how many bespoke single-use code generators I've written, but I know that I've spent a nontrivial amount of time writing code that writes code. If an LLM can do it with a natural language interface, it'll save me a lot of time. But also, I would never trust a script I threw together in 15 minutes to actually produce real code and solve the problem. All it does is generate the text I tell it to. It can't understand the system or how it works, it just procedurally spits out text. In the same way that a dozen lines of Python cannot understand your program, an LLM is also fundamentally incapable of understanding it. That's the crucial part of programming. An LLM will give you text all day, but it can't write your program. Sure an LLM can produce trivial scripts, but in my experience so far, it can only reliably generate trivial programs that I could write blindfolded in the same amount of time. If we just treat these tools like the text generators they are instead of insisting they're code generators, we'll all be better off.

jillesvangurp 3y ago

I've been using chat gpt plus quite a bit for supporting me in programming tasks. It's too flaky to trust blindly and you need to narrowly scope what it does. It can also be a bit hand wavy with some things and I've had issues getting it to generate working code for some more complicated things. But where it really shines is in documenting things, explaining things, and suggesting problem areas in code and possible solutions. And you can do fun things like point it at an openapi schema and asking it some questions about that API. I've given it screenshots of websites and asked it to criticize the design or document what is visible. It's amazingly good at supporting localization work. I was working with some geospatial code for an algorithm that generates UTM coordinates from GPS coordinates a few weeks ago. I needed a Kotlin implementation that I could use on multiple of its platforms (js and jvm, i.e. no java dependencies allowed). It was kind of useless writing the code for me for this (that's the first thing I tried obviously) but it unblocked me a couple of times and helped me figure out some details. I ultimately found several old Java implementations that contained a lot of undocumented magical numbers and I just asked it the meaning of those numbers (about a dozen) and it came back with good explanations (basically things like wgs84 ellipsoid parameters, radius of the earth, etc.). The code wasn't quite working (it failed a test I wrote) so I asked it to identify possible causes for that and it came up with some uncovered edge cases that I was able to cross confirm with other implementations. In the end I was able to piece together a working implementation by combining different elements from several implementations. Each of them individually had issues. A lot of this code is ancient and there is apparently a lot of copy paste reuse in this space.

russfink 3y ago

I use it like a man page on steroids. “Write an example bash program that implements a set data structure, and demonstrate it with examples.”

mikerg87 3y ago

I see lots of people saying the code is suspect, but I never hear about using LLMs for writing tests to validate their own code or LLM code. My success has been in generating test. Often faster and simpler than I would write by myself. What I test I can trust, especially if it comes out of an LLM.

fuzzfactor 3y ago

>How to tackle unreliability of coding assistants One of the well-established techniques was always to give more worthwhile incentives that are especially meaningful to the hardworking assistants.

CodeWriter23 3y ago

I dunno, maybe hammer out a couple dozen lines of code sourced from your own brain?