6 ms·
> The advent and rise of AI-based code assistants are already impacting the data that populates RedMonk’s language rankings. Not only that, they are impacting
by cle 3y ago
> The advent and rise of AI-based code assistants are already impacting the data that populates RedMonk’s language rankings.
Not only that, they are impacting the actual languages people use. Why use some new / esoteric language that an LLM doesn't know much about, when you can get the same job done much faster using a language that the LLM knows well and can debug?
It's going to get increasingly difficult to bootstrap new languages unless they are available in LLMs.
- echelon 3y ago> Why use some new / esoteric language that an LLM doesn't know much about [...] Rust is wildly easy to generate (idiomatically so). I've been building in our Rust monorepo without LLM tools for years and just tried them out yesterday -- I was utterly blown away. Rust might become more popular with LLM accessibility.
- stavros 3y agoI've been wanting to learn Rust for years but the learning curve was too steep, I think LLMs are what will finally do it for me.
- lolinder 3y agoCopilot was key for me for learning Rust. The syntax is verbose and unique enough that it was hard to get into from nothing without constant syntax errors. Copilot eliminated that barrier: I could describe what I wanted in a comment and immediately see a (relatively) idiomatic way to do it in Rust.
- NooneAtAll3 3y ago...why did you decide to start talking about rust out of the blue? do you need help? are you getting paid?
- evilduck 3y agoBecause it was an anecdote directly relevant to the comment it replied to, which happened to be someone using Rust? Would you have replied the same way had they mentioned the ability to use LLMs to generate Python? Would you equally imply an accusation of a Python developer of being a shill for mentioning it?
- throwup238 3y agoDid you not get your monthly check from big Oxide? You have to make sure your DeepState Connect account is up to date.
- pharmakom 3y agoI deeply disagree with this. With Good Old Fashioned Dev Tools like Zero TurnAround, ReSharper, PyCharm, Clang tools, etc. the tool was specific to the language stack. Esoteric languages would simply not have many tools. All the LLM-based tool needs is to hoover up some code on GitHub, or even just your other project files as context, and it will start generating useful stuff. On top of that, translating between languages just became a whole lot easier.
- cle 3y agoWhat other project files or GitHub code? It’s a new language. And what does “hoover up” mean? Use the context window for language basics? RAG? Retrain the model? Also the LLM won’t have been trained on the vast corpus of debugging sessions, blog posts, etc of existing languages. I’d expect the quality to be very poor compared to the current top players. (It’s effectively the same phenomenon as folks sticking to languages with a big hiring pool.)
- travisgriggs 3y agoI've seen this. I can get quite a bit of decent responses from LLMs about Python. But using it for Elixir produces more shady/dated answers. And Jetpack Compose, it just completely gets weird (and mostly worthless, being either too synthetically approximated or dated). It really illustrates (to me) that LLMs are "Plausibility Machines". They produce plausible answers. Plausibility is informed by repeated observability over large sets. We asses something as plausible when it matches the majority of our observations. That's how the parlor trick of seeming to "tell the truth" is pulled off. The venn overlap of "plausible" and "true" is pretty high. But when "truth" is in flux, you start to see the feet of the wizard behind the curtain.
- School-Cotton 3y agoWhich LLMs have you tried?
- sho 3y agoThat's weird. Which ones are you trying? I get great elixir help out of CGPT4. Sure, it needs close supervision - you can't just trust it and you need to actually understand everything it suggests, and it loves to "forget" things you told it just a moment ago about how you can't use a guard clause there, or whatever - but in terms of just getting a decent starting point, man, it's been a game-changer. The $20 I send OpenAI each month is the most no-brainer money I spend. It was definitely pretty bad with liveview for a while, while that was under heavy development and API changes, but it's had a data update that mostly fixes that, and if you specify the version it's helpful there too.
- AlchemistCamp 3y agoI’ve found that ChatGTP’s usefulness for Elixir improved dramatically after the updates around dev day. I’d say it’s still uniquely good with Python, but I find it roughly as useful with Elixir as it is with some other very popular languages.
- ideamotor 3y agoNot just that. When the general public uses LLMs more, this will calcify markets for Everything. Try bootstrapping any new product, store, idea, art - you name it.
- mattgreenrocks 3y agoIs there evidence of this happening yet? It makes sense in theory, but this is the type of thing I want to see data on before I consider it a possible problem, even. No sense in worrying about things that you can't control, can't prepare for and have not occurred yet, after all. :)
- com2kid 3y ago> Is there evidence of this happening yet? Right now, today, when choosing a library, top of my mind is "has this been around long enough to be in GPT4's training set?" Because to give up the 5-10x productivity boost GPT allows for, a library better be really good.
- ideamotor 3y agoYup
- tester457 3y agoThose are my first thoughts too. The calcification is real.
- datadrivenangel 3y agoIt'll be a moot question when OpenAI starts building a RAG index of the internet to provide up to date answers... almost like a page rank...
- com2kid 3y agoIt is not just a pure RAG index, it is having enough source material and tutorials to draw upon such that "best practices" for a given library or toolset get encoded into the model. I've seen a couple companies offer a RAG enabled ChatGPT for their SDK, but without a large number of stack overflow and medium posts detailing how to use the SDK in different scenarios, well, there isn't much help an LLM can offer that I cannot just get from whatever is on a company's site. Back in the day Microsoft used to pay boat loads of money to have absurdly complex sample applications shipped alongside new libraries and SDKs. The sample apps were often fully featured end to end solutions that had been gone over by testers as well. The idea being, since there wasn't any really good online help back then (aside from news groups and such, which were useful if you knew about them), you had to rely on sample code or on books, books which Microsoft also commissioned to be written. Maybe someday we'll get back to that point, companies paying money for large volumes of sample code to be written, but this time with the goal of having it ingested by LLMs.
- zamadatix 3y agoUsing something like e.g. Zig right now is already slower because examples are limited, a barrage of common questions over the last decade don't exist on a Stack Overflow type site, few people are experts in it, and the tooling/ecosystem is immature. Is replacing part of this with an LLM really impacting anything or just continuing what was already true before?
- tryptophan 3y agoI wonder if this can be turned into an advantage however. Say the development team invests time into writing really good documentation and examples and then feeds it into LLMs and posts a link to a chat bot on their documentation. Much easier and faster for that bot to answer the inevitable 1000s of easy questions than waiting around for the community to do so.
- stusmall 3y ago> Not only that, they are impacting the actual languages people use. Why use some new / esoteric language that an LLM doesn't know much about, when you can get the same job done much faster using a language that the LLM knows well and can debug? I don't see how this is anything new. Similar tension has always existed slowing adoption of new technology. I've resisted picking up new languages because I could implement something faster/better in languages I already know. I've been skeptical about some because the ecosystem just isn't there yet. You said "a language that the LLM knows well and can debug" when sometimes the resistance is that the new language does have as good of tooling/debugging support. There is nothing new here and if anything LLMs could help. I've noticed the community seems pretty understanding/accepting of low quality outputs of some of these LLMs. The small corpus of a new/esoteric would mean even lower quality, but I don't think that'd be a deal breaker for some folks.
- vintermann 3y agoFrom my intuition (and experience) with LLMs, they make more of what is in the context. If you use a language that is often used for business CRUD software, the assistant will be great it giving you business CRUD software, of typical business CRUD software quality. Using copilot for work, my experience is that it will happily offer ugly but context-plausible solutions (that often don't even work), but if you stop and ask "maybe there's a better way to do this", it often will suggest good simplifications. Especially if you give it a hint ("Maybe it's worth trying...") I haven't really used copilot with, say, Rust or Haskell, but if the average quality of code is better in those languages, maybe it will take less coaxing to actually suggest good code? Either way, LLMs are great at generalizing between human languages, so I expect they transfer a lot between programming languages too. I think your language has to be pretty weird for copilot to not get a grip on it with enough context.
- bluGill 3y agoI had someone submit a C++ code review where AI was used to write the code. It looked like C++98: new and delete all over, even though modern c++ is 12 years old by now. There are a lot of other things that scream I learned to program C++ in 1997 and see no reason to update my skills to anything newer. Modern C++ is really a different language and yet there is a lot of not modern code out there that AIs don't know is bad.
- vintermann 3y agoIt's not that they don't know it's bad, it's that fundamentally, they're not trying to help you. They're trying to match a probability distribution. And if there's lot of 1997-quality C++ code out there (there is!), that's highly likely what you're going to get. If you provide high quality context, it's more likely to produce high quality code too - but it can also spiral into creating bad code if it accidentally starts off on the wrong foot. The more bad code you have, the more bad code it suggests. That's why I think it's very important to clean up the code it suggests, even if it's just cosmetically.
- bsder 3y agoUnsurprising as StackOverflow has exactly the same problem. There are lots of answers that are flat out wrong 5+ years after they were answered. There is no way to go back and "unmark" that answer. > Modern C++ is really a different language and yet there is a lot of not modern code out there that AIs don't know is bad. This is a human problem and why I won't use C++ anymore. Instead of "modern C++" we needed a new language to replace C++ and leave the old nastiness behind. We'll see if any of the new replacements like Carbon get any traction.
- lmm 3y ago> Not only that, they are impacting the actual languages people use. Why use some new / esoteric language that an LLM doesn't know much about, when you can get the same job done much faster using a language that the LLM knows well and can debug? Hmm, I can see it going the opposite way - maybe the availability of LLMs means that having an active user community in your timezone is less important, and makes semi-obscure languages more competitive.
- janalsncm 3y agoIt depends. Usually there’s other things like thorough documentation, bug fixes, and comprehensive libraries that accompany popular languages. And LLM support is basically only as good as the documentation and online resources. For me, GPT4 means Rust is approachable, something I would never have considered a couple years ago. There’s tons of documentation online, but it’s great to be able to get the “vocabulary” of my question from GPT4 before looking it up online. Rust compiler errors are pretty good, but a LLM really smooths over the rough spots.
- lolinder 3y agoI've actually been building toy languages for fun (nothing I'm ready to share) with GitHub Copilot as an aid, and it's able to pick up on my homegrown language just fine. It helps that I opted for C-style syntax, but it's able to pick up on semantic concepts like effects that don't exist in the mainstream, as well as specific syntactic choices like `:` vs `=` for record constructors. Where a new language will be severely disadvantaged is in the blank page problem. With a well-known language you can start a new project with just a comment and let Copilot get you over the initial barrier of just having something down on the page. A new language won't work because it needs a fair amount of context before it's able to produce correct code.
- ljm 3y agoCopilot stuff has made some things harder for me to review, funnily enough. I don't blame it on the model, but the programmer using it with zero scrutiny. If people doggedly continue down that path they're going to stagnate for as long as they assume the AI is always right. Just because an LLM can can cobble together some output doesn't mean that the output is instantly trustworthy. Not to mention languages often optimise for different things. If you want to default to JS for everything because the AI can auto-complete that more easily, you're locking into a very specific operating model.
- irrational 3y agoI would think ColdFusion (CFML) would be esoteric, but recently I was tasked to work on an ancient CFML code base for a few days and Copilot saved my bacon. I was frankly astonished at how well it knew CFML. It was like a mind reader. I'd write <cfset some_var =... and it would fill in about 15 lines of code that were exactly what I needed to have happen. How did it do so well at such an esoteric language?
- Buttons840 3y ago> It's going to get increasingly difficult to bootstrap new languages unless they are available in LLMs. Or maybe you'll teach your new language to the LLM and it will make porting and binding to everything else easy.
- mynameisnoone 3y agoLLMs can barely generate legal code, so they're certainly unable to debug them or write the test cases completely without expert guidance of someone who knows what the hell they're doing. It will take 10 to 20 years before there is "Star Trek computer"-level of self-programming, agent-based systems.
- anacrolix 3y agoThose are rookie numbers. You need to get those timeframes up.
- mynameisnoone 3y agoPrognostication is an inexact, bullshit guestimate that you can't hope to improve upon except to claim you have special or greater Dunning-Kruger "knowledge".
- uticus 3y agoAt what point does the language become a useless abstraction, and AI becomes the new “language”? I’m a software developer yet still amazed at how many layers of abstraction are tolerated in user space. Take web development. You’d think at some point the stack of a half dozen frameworks would be abandoned in favor of vanilla JavaScript. But the house of cards continues to grow. So maybe languages will still stick around when they are no longer the human interface.
- sensanaty 3y agoWhat is HN's obsession with JS frameworks? There's 4 big ones (React, Vue, Angular, Ember (arguable if Ember still counts)) that have been in use for the better part of 2 decades at this point. Sure there's plenty of smaller ones like Svelte cropping up, but most places just use one of these 4 for the most part. Anyone who's ever tried to build anything big with vanilla JS knows why these frameworks exist in the first place. The ones like Vue or Svelte aren't even really that complex to grok anyways, despite all the hyperbole about them.
- uticus 3y agoSorry, yes I was picking too hard on JS. But hopefully my point still stands. A coder writes instructions for the computer; it amazes me the tolerance for having those instructions go through multiple layers of transformations (sometimes even multiple layers of languages) before hitting the OS. If the point of a scripting language is to provide a user experience not attainable by a lower-level compiled language (which in turn is not attainable by machine code), wouldn't AI nullify the benefits of both the scripting and the lower-level languages?