6 ms·
> Is there some expectation that these things won't improve? I definitely expect them to improve. But I also think the point at which they can actually replace
by IshKebab 2y ago
> Is there some expectation that these things won't improve?
I definitely expect them to improve. But I also think the point at which they can actually replace a senior programmer is pretty much the exact point at which they can replace any knowledge worker, at which point western society (possibly all society) is in way deeper shit than just me being out of a job.
> This is ego speaking.
It definitely isn't. LLMs are useful for coding now, but they can't really do the whole job without help - at least not for anything non-trivial.
- dingnuts 2y ago> LLMs are useful for coding now *sort of, sometimes, with simple enough problems with sufficiently little context, for code that can be easily tested, and for which sufficient examples exist in the training data. I mean hey, two years after being promised AGI was literally here, LLMs are almost as useful as traditional static analysis tools! I guess you could have them generate comments for you based on the code as long as you're happy to proofread and correct them when they're wrong. Remember when CPUs were obsolete after three years? GPT has shown zero improvement in its ability to generate novel content since it was first released as GPT2 almost ten years ago! I would know because I spent countless hours playing with that model.
- margalabargala 2y agoFirstly, GPT-2 was released in 2019. Five years is not "almost ten years". Secondly, LLMs are objectively useful for coding now. That's not the same thing as saying they are replacements for SWEs. They're a tool, like syntax highlighting or real-time compiler error visibility or even context-aware keyword autocompletion. Some individuals don't find those things useful, and prefer to develop in a plain text editor that does not have those features, and that's fine. But all of those features, and LLMs are now on that list, are broadly useful in the sense that they generally improve productivity across the industry. They already right now save enormous amounts of developer time, and to ignore that because you are not one of the people whose time is currently being saved, indicates that you may not be keeping up with understanding the technology of your field. There's an important difference between a tool being useful for generating novel content, and a tool being useful. I can think of a lot of useful tools that are not useful for generating novel content.
- bawolff 2y ago> are broadly useful in the sense that they generally improve productivity across the industry. They already right now save enormous amounts of developer time, But is that actually a true statement. Are there actual studies to back that up? AI is hyped to the moon right now. It is really difficult to separate the hype from reality. There are ancedotal reports of ai helping with coding, but there are also ancedotal reports that they get things almost right but not quite, which often leads to bugs which wouldn't otherwise happen. I think its unclear if that is a net win for productivity in software engineering. It would be interesting if there was a robust study about it.
- margalabargala 2y ago> Are there actual studies to back that up? I am aware of an equal number of studies about the time saved overall by use of LLMs, and time saved overall by use of syntax highlighting. In fact, here's a study claiming syntax highlighting in IDEs does not help code comprehension: https://link.springer.com/article/10.1007/s10664-017-9579-0 https://link.springer.com/article/10.1007/s10664-017-9579-0 Shall we therefore conclude that syntax highlighting is not useful, that developers who use syntax highlighting are just part of the IDE hype train, and that anecdotal reports of syntax highlighting being helpful are counterbalanced by anecdotal reports of $IDE having incorrect syntax highlighitng on $Esoteric_file_format? Most of the failures of LLMs with coding that I have seen has been a result of asking too much of the LLM. Writing a hundred context-aware unit tests is something that an LLM is excellent at, and would have taken a developer a long time previously. Asking an LLM to write a novel algorithm to speed up image processing of the output of your electron microscope will go less well.
- bawolff 2y ago> Shall we therefore conclude that syntax highlighting is not useful, that developers who use syntax highlighting are just part of the IDE hype train, and that anecdotal reports of syntax highlighting being helpful are counterbalanced by anecdotal reports of $IDE having incorrect syntax highlighitng on $Esoteric_file_format? Yes. We should conclude that syntax highlighting is not useful in languages that the syntax highlighter does not support. I think basically everyone would agree with this statement. Similarly an llm that worked 100% of the time and could solve any problem would be pretty useful. (Or at least worked correctly as often as syntax highlighting in situations where it is actually used does) However that's not the world we live in. Its a reasonable question to ask if llm is good enough yet where the productivity gain outweighs the productivity lost.
- munk-a 2y agoIntellisense style systems were a huge feature leap when they gained wider language support and reliability. LLMs are yet another step forward for intellisense and the effort of comprehending the code you're altering. I don't think I will ever benefit from code generation in a serious setting (it's excellent for prototyping) simply due to the fact that it's solving the easy problem (write some code) while creating a larger problem (figure out of the code that was generated is correct). As another senior developer I won't say it's impossible that I'll ever benefit from code generation but I just think it's a terrible space to try and build a solution - we don't need a solution here - I can already type faster than I can think. I am keenly interested in seeing if someone can leverage AI for query performance tuning or, within the RDBMS, query planning. That feels like an excellent (if highly specific) domain for an LLM.
- LouisSayers 2y ago> I am keenly interested in seeing if someone can leverage AI for query performance tuning or, within the RDBMS, query planning. That feels like an excellent (if highly specific) domain for an LLM. Pay the $20 for Claude, copy the table DDL's in along with a query you'd like to tune. Copy in any similar tuned queries you have and tell it you'd like to tune your query in a similar manner. Once you've explained what you'd like it to do and provided context hit enter. I'd be very surprised if having done this you can't find value in what it generates.
- tlarkworthy 2y ago> I can already type faster than I can think. But can you write tickets faster than you can implement them? I certainly can.
- vinnymac 2y agoDepends on the ticket. If it's "Get us to the moon", it's gonna take me years to write that ticket. If it was "Make the CTA on the homepage red", it is up for debate whether I needed a ticket at all.
- 2y ago
- rybosworld 2y ago> LLM’s never provide code that pass my sniff test If that statement isn't coming from ego, then where is it coming from? It's provably true that LLM's can generate working code. They've been trained on billions of examples. Developers seem to focus on the set of cases that LLM's produce code that doesn't work, and use that as evidence that these tools are "useless".
- selfmodruntime 2y ago> If that statement isn't coming from ego, then where is it coming from? It's provably true that LLM's can generate working code. They've been trained on billions of examples. I am pro AI and I'm probably even overvaluing the value AI brings. However, for me, this doesn't work in more "esoteric" programming languages or those with stricter rulesets like Rust. LLMS provide fine JS code, since there's no compiler to satisfy, but CPP without undefined behaviour or compiling Rust code is rare. There's also no chance of LLMS providing compiling code if you're using a library version with a newer API than the one in the training set.
- hmillison 2y agothere's a lot more involved in senior dev work beyond producing code that works. if the stakeholders knew how to do what they needed to build and how, then they could use LLMs, but translating complex requirements into code is something that these tools are not even close to cracking.
- rybosworld 2y ago> there's a lot more involved in senior dev work beyond producing code that works. Completely agree. What I don't agree with is statements like these: > LLM’s never provide code that pass my sniff test To me, these (false) absolutions about chat bot capabilities, are being rehashed so frequently, that it derails every conversation about using LLM's for dev work. You'll find similar statements in nearly every thread about LLM's for coding tasks. It's provably true that LLM's can produce working code. It's also true, that some increasingly large portion of coding is being offloaded to LLM's. In my opinion, developers need to grow out of this attitude that they are John Henry and they'll outpace the mechanical drilling machine. It's a tired conversation.
- JKCalhoun 2y agoI imagine though they might replace 3 out of 4 senior programmers (keep one around to sanity check the AI).
- munk-a 2y agoThat's the same figuring a lot of business folks had when considering off-shoring in the early 2000s - those companies ended up hiring twice as many senior programmers to sanity check and correct the code they got back. The same story can be heard from companies that fired their expensive seniors to hire twice as many juniors at a quarter the price. I think that software development is just an extremely poor market segment for these kinds of tools - we've already got mountains of productivity tools that minimize how much time we need to spend doing the silly rote programming stuff - most of software development is problem solving.
- mrbungie 2y agoOof, the times I've heard something like that with X tech.
- tonyarkles 2y agoHeh, UML is going to save us! The business people can just write the requirements and the code will write itself! /s Given the growth-oriented capitalist society we live in in the west, I'm not all that worried about senior and super-senior engineers being fired. I think a much more likely outcome is that if a business does figure out a good way to turn an LLM into a force-multiplier for senior engineers, they're going to use that to grow faster. There is a large untapped niche too that this could potentially unlock: projects that aren't currently economically viable due to the current cost of development. I've done a few of these on a volunteer basis for non-profits but can't do it all the time due to time/financial constraints. If LLM tech actually makes me 5x more productive on simple stuff (most of these projects are simple) then it could get viable to start knocking those out more often.
- PeterisP 2y agoIf a LLM (or any other tool) makes so that team of 8 can get the same results in the same time as it used to take a team of 10 to do, then I would count that as "replaced 2 programmers" - even if there's no particular person for which the whole job has been replaced, that's not a meaningful practical difference, replacing a significant fraction of every programmer's job has the same outcomes and impacts as replacing a significant fraction of programmers.
- RangerScience 2y agoFav anecdote from ages ago: When hand-held power tools became a thing, the Hollywood set builder’s union was afraid of this exact same thing - people would be replaced by the tools. Instead, productions built bigger sets (the ceiling was raised) and smaller productions could get in on things (the floor was lowered). I always took that to mean “people aren’t going to spend less to do the job - they’ll just do a bigger job.”
- sigmarule 2y agoThis could very well prove to be the case in software engineering, but also could very well not; what is the equivalent of "larger sets" in our domain, and is that something that is even preferable to begin with? Should we build larger codebases just because we _can_? I'd say likely not, while it does make sense to build larger/more elaborate movie sets because they could. Also, a piece missing from this comparison is a set of people who don't believe the new tool will actually have a measurable impact on their domain. I assume few-to-none could argue that power tools would have no impact on their profession.
- ska 2y ago> Should we build larger codebases just because we _can_? The history of software production as a profession (as against computer science) is essentially a series of incremental increases in the size and complexity of systems (and teams) that don't fall apart under their own weight. There isn't much evidence we have approached the limit here, so it's a pretty good bet for at least the medium term. But focusing on system size is perhaps a red herring. There is an almost unfathomably vast pool of potential software systems (or customization of systems) that aren't realized today because they aren't cost effective...
- rozap 2y ago> I definitely expect them to improve. But I also think the point at which they can actually replace a senior programmer is pretty much the exact point at which they can replace any knowledge worker, at which point western society (possibly all society) is in way deeper shit than just me being out of a job. Agree with this take. I think the probability that this happens within my next 20 years of work is very low, but are non-zero. I do cultivate skills that are a hedge against this, and if time moves on and the probability of this scenario seems to get larger and larger, I'll work harder on those skills. Things like fixing cars, welding, fabrication, growing potatoes, etc (which I already enjoy as hobbies). As you said, skills that are helpful if shit were to really hit the fan. I think there are other "knowledge workers" that will get replaced before that point though, and society will go through some sort of upheaval as this happens. My guess is that capital will get even more consolidated, which is sort of unpleasant to think about.
- boogieknite 2y agoim also expecting this outcome. my thoughts are that once devs are completely replaced by ml then were going to finally have to adapt society to raise the floor because we can no longer outcompete automation. im ready to embrace this, but it seems like there is no plan for mass retirement and were going to need it. if something as complicated as app and system dev is automated, pretty much anything can be. earnest question because i still consider this a vague, distant future: how did you come up with 20 years?
- risyachka 2y agoThis. When this moment becomes reality - the world economy will change a lot, all jobs and markets will shift. And there won’t be any way to future proof your skills and they all will be irrelevant. Right now many like to say “learn how to work with ai, it will be valuable “ . No, it won’t. Because even now it is absolutely easy to work with it, any developer can pick up ai in a week, and it will become easier and easier. A better time spent is developing evergreen skills.