6 ms·
I think this is "how to think about coding assistants and your task" but none of this is "tackling" their unreliability. While coding assistants seem to do wel
by abeppu 3y ago
I think this is "how to think about coding assistants and your task" but none of this is "tackling" their unreliability.
While coding assistants seem to do well in a range of situations, I continue to believe that for coding specifically, merely training on next-token-prediction is leaving too much on the table. Yes, source code is represented as text, but computer programs are an area where there's available information which is _so much richer_. We can know not only the text of the program but the type of every expression, which variables are in scope at any point, what is the signature of a method we're trying to call, etc. These assistants should be able to make predictions about program _traces_, not just program source text. A step further would be to guess potential loop invariants, pre/post conditions, etc, confirm which are upheld by existing tests, and down-weight recommending changes which introduce violations to those inferred conditions.
ChatGPT and tab-completion assistants have both given me things that are not even valid programs (e.g. will not compile, use a variable that isn't actually in scope, etc). ChatGPT even told me that an example it generated wasn't compiling for me b/c I wasn't using a new enough version of the language, and then referenced a language version which does not yet exist. All of this is possible in part b/c these tools are engaging only at the level of text, and are structurally isolated from the rich information available inside an interpreter or debugger. "Tackling" unreliability should start with reframing tasks in a way which lets tools better see the causes of their failures.
- jkaptur 3y agoI absolutely agree, but I find the situation incredibly funny. There are three characters here: me, the LLM, and the compiler. Two of them are robots, but they refuse to talk to each other - it's up to me to tell the LLM the bad news from the compiler.
- sa-code 3y agoThis sounds like the worst game of telephone
- fragmede 3y agothat's a frontend issue though. If you use the python interpreter in ChatGPT, you can tell it to run the code until it works, at which point it'll do a couple of iterations before giving you code.
- deleted 3y ago[deleted]
- jgalt212 3y agoHow many people are putting service credentials into a ChatGPT Python interpreter to see if their generated code "works"?
- notjoemama 3y agoOne of our very large customers was hit with ransomware recently. So…best practice is don’t do things like that.
- danielmarkbruce 3y agoYou can just use your own.... There is still the risk GPT-4 gives you ~ "delete all the things" and you blindly run it on your infra, but.. it's better than what. you describe.
- convolvatron 3y agodont you already cut and paste your error message to google like everyone else?
- stavros 3y agoI decided to cut out the middleman: https://github.com/skorokithakis/sysaidmin/ https://github.com/skorokithakis/sysaidmin/
- actionfromafar 3y agoHm, that’s true. I guess someone could have hooked up pattern matching and google search results and iterated that directly into some IDE.
- pjerem 3y agoIt will come. In fact, GPT is really good at correcting its code given the compiler output. We just need to automate that. I think that this is not done yet because LLMs costs are O(n+1) so you really don’t want it to be stuck in some loop.
- swatcoder 3y agoIndeed. These are basically tech demos at this point. A marvel to see, and sometimes useful, but still extremely crude. There's a lot of headroom for sophistication as a few more research insights are made and an experienced tooling team commits a few years to making rich multimodal/ensemble code assistants that can perform smart analyses, transformations, boilerplating, etc on your project instead of just adding some text to your file. But it'll take a years of insight and labor to build that system up, adapt it to different industries/uses, and prove it out as engineering-ready. People get caught up in the novelty of Copilot and ChatGPT and then imagine that the revolution arrives when some new paper comes out tomorrow (or that none will arrive because of today's limitations), but the far more likely reality is that the revolution paces as something more like the internet's -- real and profound, but unfolding gradually over the span of decades as people work hard to make new things with it.
- renegade-otter 3y ago"Any sufficiently advanced technology is indistinguishable from magic." -Arthur C. Clarke
- TerrifiedMouse 3y agoFrankly, we may be barking up the wrong tree with LLMs. Sure they deliver novel and very marketable results, hence the insane funding, but I can’t help but feel it’s really just a parlor trick and there is a yet undiscovered algorithm that can actually deliver the AGI that we seek - AGI that reliable and precise like fictional AIs.
- jrockway 3y agoI think a lot of progress in the field of AI comes from random breakthroughs rather than evolution of existing ideas. For that reason, I think AGI will probably be another random breakthrough, and not just an LLM with more parameters. Good if you're NVidia, bad if you're OpenAI. The random breakthrough could happen at Google, or it could happen in some professor's basement. There is no way to know, and no way to throw money at the problem and guarantee a return.
- Night_Thastus 3y agoIn order to really solve this, you couldn't use an LLM, at least not one in any shape that we have now. You'd need something that actually understands the language. What is a lifetime, what is scope, what are types, functions, variables, etc. Something that can contextually look at correct, incomplete, or broken programs and reason about what they're doing by knowing the rules that the language operates on and following them to a conclusion. It would also need to understand high-level design patterns and structure to not just know what's happening in the literal sense "this variable is being incremented" but also in a more abstract sense "this variable keeps track of references because this is mixed C/Python code that needs that to handle deallocation". Something that recognizes patterns both within and outside of the code, with appropriately fuzzy matching where needed. And I think importantly you'd need to be able to query it at a variety of levels. What's happening on a line-by-line basis and what's happening at the high level at a given point of code. One OR the other isn't sufficient. That is not a simple ask. We're a long way away from something that smart existing.
- LeifCarrotson 3y agoWhat LLM text generation has shown is that you don't actually have to understand English to generate pretty decent English. You just have to have enough examples. This is where the massive corpus of source code available on the Internet can help generate a "LSM" (large software model) if you can expose the tokens as the lexer understands them in the training set. If your LSM sees a trillion examples of correct usage of lifetime and scope and types and so on, then in the same way that an LLM trained on English grammar will emit text with correct grammar as if it understands English, your LSM will generate software with correct syntax as if it understands the software. Whatever the definition of "understands" is in the context of an LLM.
- fragmede 3y agoright. we're already abstracting from English words and characters into tokens, piping code through half a compiler so the LSM is given the AST to train on doesn't seem all that far fetched.
- 3y ago
- dxbydt 3y agoGenerating invalid code is a big hassle. I have used ChatGPT for generating R code & sometimes it refers to functions that don't exist. The standard deviation of [1,2,1] is 0.577, given by sd(c(1,2,1)). Sometimes I get stdev(c(1,2,3)) - there is no stdev in R. Why not have a slower mode where you run the code thru the R interpreter first, & only emit valid code that passes that step ?
- fragmede 3y agoThe Python one that currently exists can use Pandas, but then you're using Python and not R. Other language support must be on their roadmap, only question is how far down the list it is.
- blharr 3y agoThis can be done pretty easily with their api if you are willing to spend some time on it
- walt_grata 3y agoYou know I've been thinking about this for a while, I honestly don't think LLMs will ever be the right choice for a coding assistant. Where I've had a strange idea they could help, although it's completely unproven yet, is replacing what we traditionally think of as a developer and code. I'm envisioning, for simple application, a completely non-technical person have a conversation with a coding assistant and it something directly executable, think the tokens it outputs are JVM bytecode or something similar. I'm sure there's innumerable flaws with my thinking/approach but so far, it's been fun to think and code around.
- CSMastermind 3y agoWould be interesting to see them trained on actual syntax trees from text. Then maybe have a separate model trained on going from syntax tree to source code.
- CaptainFever 3y ago> Then maybe have a separate model trained on going from syntax tree to source code. I don't see why you need a model for this. But yes, this is a very cool idea.
- tonyedgecombe 3y agoPresumably so you don’t need to write a code generator for every language you support.
- delta_p_delta_x 3y ago> these tools are engaging only at the level of text Couldn't we say the same thing about almost all UNIX/Linux coreutils? There's no way to get a strongly-typed array/tree of folders, files, and metadata from `ls`; it has to be awk-ed and sed-ed into compliance. There's no way to get a strongly-typed dictionary of command-line options and what they do for pretty much every coreutil; you have to invoke `something --help` and then awk/sed that output again. These coreutils are at their core, only stringly-typed, which makes life more difficult than it strictly needs to be. Everything is a bag of bytes, or a stream to be manipulated. This philosophy simply leaked into LLMs.
- PoignardAzur 3y agoI think GP's point is we could get much better results if we fed the LLMs with more than the input text. It could still be text, or at least bytestreams: program traces, OpenTelemetry logs, objdump of the compiled binaries, LLVM IR dumps, compiler errors, syntax highlighting markers, etc.
- sanderjd 3y agoThis is, indeed, also a weakness of those tools.
- PoignardAzur 3y agoGood point about program traces. Another area I wish Copilot understood: where my cursor will go next. Right now it can only feed me new lines, but I bet the underlying LLM is already powerful enough that with little fine-tuning, it could guess that after I edit a call to foobar() to add an argument the next thing I'll do is probably edit the definition of foobar. It feels like there's a trove of UX improvements in there; we've barely scratched the surface.
- actionfromafar 3y agoIDEs already has this built in. The problem is more an effect of us deciding that source code is text instead of something integrated, like Smalltalk.
- PoignardAzur 3y agoWell yeah, but IDEs have code completion built-in as well. My point is, I think Copilot-style tools could do more that insert text. They could recommend lines of code to remove, give you a quick keyboard shortcut to "place cursor in next file / line you're likely to want to edit", etc. IDEs can do some rigid versions of that (eg "Jump to next diagnostic", "Refactor", "Jump to implementation", etc) but Copilot could be the more general, more powerful version of these features.
- sanderjd 3y agoVery well put. The current generation of these tools are incredibly useful but clearly leaving a huge amount on the table. Just makes me more excited for future iterations!
- disconcision 3y agoI've been working on some basics in this direction... taking the expected type at the cursor and using this to inform exactly what type & function definitions to splice into the prompt: https://andrewblinn.com/papers/2023-MWPLS-Type-directed-Prompt-Construction-for-LLM-powered-Programming-Assistants.pdf https://andrewblinn.com/papers/2023-MWPLS-Type-directed-Prom... We're currently working on two forks off this... one, using the expected type information in conjunction with existing grammar constrained generation to enforce per-token type-correct generations, and two, using some program synthesis unevaluation techniques to also provided runtime trace data relevant to the current program hole we're trying to generate a completion for. But yeah in general there is so much existing work on static and dynamic program analysis which can be applied here... I think a lot of the interesting challenges are going to be UI/UX ones... interactive processes to help more precisely specify intent and iteratively valid generations as more and more code is written autonomously.
- pedrovhb 3y agoThat's interesting. In my experience, although synctatic errors do happen, they're not nearly as common as, say, hallucinating methods or tripping over the exact API of a library. One thought I've had but haven't experimented with yet is that we could leverage a lot of the existing tooling that was made for humans - e.g. tab-complete providers like Jedi, which do some type inference behind the scenes. It's able to provide suggestions of valid members for a given cursor position, and so logits could be warped to prefer tokens which match the suggestions (so if the output at a given time is `math.sq`, `math.sqrt` would be much preferred over `math.square_root` which doesn't exist). You'd have to be a bit smart around this though because in since situations such as when using an identifier for the first time it's not yet in scope and you don't want the LLM to never create variables. Maybe some beam search shenanigans and heuristics could be enough to get useful output, but at that point it no longer seems like a quick thing to just try out :)
- disconcision 3y agoit's been done, in a limited way: https://arxiv.org/abs/2306.10763 https://arxiv.org/abs/2306.10763; we're working on extended this, but yeah, there are a lot of details to work out
- mr_toad 3y ago> then referenced a language version which does not yet exist It must have gaming forums in its training data. Gamers know that the solution to all problems is to upgrade (game/drivers/OS) to the latest version.
- eru 3y ago> These assistants should be able to make predictions about program _traces_, not just program source text. Are you sure 'traces' is the right word? Not something more like ASTs? Btw, the predict-next-token approach has the benefit of also being able to deal with broken code. You might also want to compare http://www.incompleteideas.net/IncIdeas/BitterLesson.html http://www.incompleteideas.net/IncIdeas/BitterLesson.html and https://gwern.net/scaling-hypothesis https://gwern.net/scaling-hypothesis with your idea of adding more domain specific knowledge.
- Belomolo 3y agoAt least chatgpt can already use a python interpreter and can also write unit tests for it. Alone with copilot the amount of money flowing into this area is much more now than just a year ago. And based on a Google research blog article, they use it internally with over 50% of suggestions being accepted. The race is on and no one can afford not to play the game.