10 ms·

> 99% of the code in this PR [for llama.cpp] is written by DeekSeek-R1 I hope we can put to rest the argument that LLMs are only marginally useful in coding -

• mclau156 2y ago
  Its cope
• Waterluvian 2y ago
  I want this to be true. Actually writing the code is the least creative, least interesting part of my job. But I think it’s still much too early for any form of “can we all just call it settled now? In this case, as we all know, lines of code is not a useful metric. How many person hours were spent doing anything associated with this PR’s generation and how does that compare to not using AI tools, and how does the result compare in terms of the various forms of quality? That’s the rubric I’d like to see us use in a more consistent manner.
• cjbgkagh 2y ago
  Who would the new applications be for? I figure that it’ll be far easier to build apps for use by LLMs than building apps for people to use. I don’t think there will be this large increase of induced demand, the whole world just got a lot more efficient and that’s probably a bad thing for the average person.
  • fragmede 2y ago
    take some process that you, or someone you know does right now that involves spreadsheets and copy-pasting between various apps. hiring a software engineer to build an app so it's just a [do-it] button previously didn't make sense because software engineer time was too expensive. Now, that app can be made, so the HR or whatever person doesn't need to waste their time on automatable tasks.
    • cjbgkagh 2y ago
      I'm pretty sure a LLM can be taught to press a button
    • 6510 2y ago
      The thing that has me most inspired is that one will finally get to ask the questions that seemed strange to ask before. Like, 1:40 times, when I press the button nothing happens for 10 seconds and I don't know if I've pressed the button properly.
    • yobbo 2y ago
      If their time was actually wasted, paying a few or several thousand dollars for a tool would have been profitable a long time ago. Usually, there are hidden payoffs that motivate things that seem like waste.
      • simonw 2y ago
        The people who are spending time manually doing a task that could be handled by a program are usually the exact same people who don't have the experience (or authority) to be able to say "this is a thing that could be automated with a tool if we paid a few thousand dollars to develop it". Hiring someone to remodel a bathroom is hard enough, now try hiring a contract software engineer, especially when you don't have budget authority! That said, I heard about a fire chief last year who had to spend two days manually copying and pasting from one CRM to another. I wish I could help people like that know when to pay someone to write a script! I imagine even in that role figuring out how to hire someone so solve a problem would still take longer than manually crunching through that themselves.
  • 6510 2y ago
    Oh right, we will have a B2B B2C B2L L2B L2C and ultimately the L2L market.
• rozap 2y ago
  Broadly agree. Whether or not it is useful isn't really an interesting discussion, because it so clearly is useful. The more interesting question is what it does to supply and demand. If the past is any indication, I think we've seen that lowering to barrier to getting software shipped and out the door (whether it's higher level languages, better tooling) has only made demand greater. Maybe this time it's different because it's such a leap vs an incremental gain? I don't know. The cynical part of me thinks that software always begets more software, and systems just become ever more complex. That would suggest that our jobs are safe. But again, I don't say that with confidence.
  • oorza 2y ago
    I don't think it's necessarily any larger of a leap than any of the other big breakthroughs in the space. Does writing safe C++ with an LLM matter more than choosing Rust? Does writing a jQuery-style gMail with an LLM matter more than choosing a declarative UI tool? Does adding an LLM to Java 6 matter more than letting the devs switch to Kotlin? Individual developer productivity will be expected to rise. Timelines will shorten. I don't think we've reached Peak Software where the limiting factor on software being written is demand for software, I think the bottlenecks are expense and time. AI tools can decrease both of those, which _should_ increase demand. You might be expected to spend a month outputting a project that would previously have taken four people that month, but I think we'll have more than enough demand increase to cover the difference. How many business models in the last twenty years that weren't viable would've been if the engineering department could have floated the company to series B with only a half dozen employees? What IS larger than before, IMO, is the talent gap we're creating at the top of the industry funnel. Fewer juniors are getting hired than ever before, so as seniors leave the industry due to standard attrition reasons, there are going to be fewer candidates to replace them. If you're currently a software engineer with 10+ YoE, I don't think there's much to worry about - in fact, I'd be surprised if "was a successful Software Engineer before the AI revolution" doesn't become a key resume bullet point in the next several years. I also think that if you're in a position of leadership and have the creativity and leadership to make it work, juniors and mid-level engineers are going to be incredibly cost effective because most middle managers won't have those things. And companies will absolutely succeed or fail on that in the coming years.
• nine_zeros 2y ago
  > Our trade has changed forever, and there's no going back. When companies claim that AI will replace developers, it isn't entirely bluster. Jobs are going to be lost unless there's somehow a demand for more applications This is a key insight - the trade has changed. For a long time, hoarding talent - who could conceive and implement such PRs - was a competitive advantage. It no longer is because companies can hire and get similar outcomes, with fewer and mediocre devs. But at the same time, these companies have lost their technological moat. The people were the biggest moat. The hoarding of people were the reason why SV could stay ahead of other concentrated geographies. This is why SV companies grew larger and larger. But now, anyone anywhere can produce anything and literally demolish any competitive advantage of large companies. As an example, literally a single Deepseek release yesterday destroyed large market cap companies. It means that the future world is likely to have a large number of geographically distributed developers, always competing, and the large companies will have to shed market cap because their customers will be distributed among this competition. It's not going to be pleasant. Life and work will change but it is not merely loss of jobs but it is going to be loss of the large corporation paradigm.
  • happyopossum 2y ago
    > literally a single Deepseek release yesterday destroyed large market cap companies Nobody was “destroyed” - a handful of companies had their stock price drop, a couple had big drops, but most of those stocks are up today, showing that the market is reactionary.
    • nine_zeros 2y ago
      Alright, if this is more palatable - let's just say market caps will decline because of small code updates made anywhere in the world. The point still is: Software/engineering is no longer the moat creator.
      • pertymcpert 2y ago
        You completely misunderstood the reason for the stock price drop. It was because of the DeepSeek MoE model's compute efficiency which vastly reduced the compute requirements needed to achieve a certain level of performance. Notice how Apple and Meta stocks went up last 2 days?
• simonw 2y ago
  "Jobs are going to be lost unless there's somehow a demand for more applications." That's why I'm not worried. There is already SO MUCH more demand for code than we're able to keep up with. Show me a company that doesn't have a backlog a mile long where most of the internal conversations are about how to prioritize what to build next. I think LLM assistance makes programmers significantly more productive, which makes us MORE valuable because we can deliver more business value in the same amount of time. Companies that would never have considered building custom software because they'd need a team of 6 working for 12 months may now hire developers if they only need 2 working for 3 months to get something useful.
  • sitkack 2y ago
    We have already entered a new paradigm of software development, where small teams build software for themselves to solve their own problems rather than making software to sell to people. I think selling software will get harder in the future unless it comes with special affordances.
    • LeFantome 2y ago
      I think some of the CEOs have it right on this one. What is going to get harder is selling “applications” that are really just user friendly ways of getting data in and out of databases. Honestly, most enterprise software is just this. AI agents will do the same job. What will still matter is software that constrains what kind of data ends up in the database and ensures that data means what it is supposed to. That software will be created by local teams that know the business and the data. They will use AI to write the software and test it. Will those teams be “developers”? It is probably semantics or a matter of degree. Half the people writing advanced Excel spreadsheets today should probably be considered developers really.
      • butlike 2y ago
        Maybe, but it's the same argument trickling down. You'll need the CRUD-apps because you hired Cindy to press the button, and if shit goes pear-shaped, you can point to Cindy in the post-mortem. If it's some AI agent pressing the button to egress data from the database, and there's an anomaly, then it's a systemic failure at a macro level at that company, which is harder to write a press release about.
• lenerdenator 2y ago
  My greatest problem is duplicating the secret sauce of GHCP: it has access to your project and can use it as context. Admittedly, I haven't looked too hard, but how could I do that with a model from, say, Ollama and run exclusively on my machine?
  • simonw 2y ago
    There are a bunch of tools that might be able to do that. I'd start by exploring https://aider.chat/ https://aider.chat/
    • lenerdenator 2y ago
      I'll take a look, thank you!
    • realo 2y ago
      Neat. Can I use aider with a local model running in LMStudio (or ollama)? After a very quick reading of their pages it does not seem so. Hopefully I am wrong...
      • simonw 2y ago
        Yes you can: https://aider.chat/docs/llms/ollama.html https://aider.chat/docs/llms/ollama.html
  • nkozyra 2y ago
    Couldn't you load the whole thing into a database or memory and use it as a RAG source? Not sure if that would fully scratch the itch.
• TheBigSalad 2y ago
  LLMs are only marginally useful for coding. You have simply chosen to dismiss or or 'give up' on that fact. You've chosen what you want to believe in contrast to the reality that we are all experiencing.
  • simonw 2y ago
    LLMs are incredibly useful for coding, if you learn how to apply them effectively. You have simply chosen to dismiss or 'give up' on that fact.
    • TheBigSalad 2y ago
      They are a useful tool, but not 'incredibly useful'. The simple, repetitive code in this example is what they are good at. It's like 1% of what I do working on products. Writing code isn't even that impressive, the whole job is figuring out exactly what people want.
      • fragmede 2y ago
        given that there's no standardized scale of usefulness, is the distinction between "useful" for one person vs "incredibly useful", when nothing concrete has been specified; is that distinction really the important thing here? both of you find it useful. I might go off on a long tangent about how I love my hammer, it's the best, and you'll think I'm ridiculous because it's just a hammer, but at the end of the day, we can both agree that the hammer is doing the job of driving in nails.
    • astrobe_ 2y ago
      I think the reality is that these AI output the "average" of what was in their training set, and people receive it differently depending on if they are below or above this average. It's a bit like what happens with "illusion of knowledge" or "illusion of understanding". When one knows the topic, one can correct the output of AI. When one doesn't, one tends to forget it can be inaccurate or plain wrong.
• jasonthorsness 2y ago
  I think quality is going to go up - I have so much code I wish I could go back and optimize for better performance, or add more comprehensive tests for, and LLMs are getting great at both of those as they work really well off of things that already exist. There has never been enough time/resources to apply towards even the current software demand, let alone future needs.
• dfilppi 2y ago
  [dead]
• mjr00 2y ago
  > I hope we can put to rest the argument that LLMs are only marginally useful in coding - which are often among the top comments on many threads. I suppose these arguments arise from (a) having used only GH copilot which is the worst tool, or (b) not having spent enough time with the tool/llm, or (c) apprehension. I've given up responding to these. Look at the code that was changed[0]. It's a single file. From what I can tell, it's almost purely functional with clearly specified inputs and outputs. There's no need to implement half the code, realize the requirements weren't specified properly, and go back and have a conversation with the PM about it. Which is, you know, what developers actually do. This is the kind of stuff LLMs are great at, but it's not representative of a typical change request by Java Developer #1753 at Fortune 500 Enterprise Company #271. [0] https://github.com/ggerganov/llama.cpp/pull/11453/files https://github.com/ggerganov/llama.cpp/pull/11453/files
  • simonw 2y ago
    "Yeah, but LLMs can't handle millions of lines of crufty old Java" is a guaranteed reply any time this topic comes up. (That's not to say it isn't a valid argument.) Short answer: LLMs are amazingly useful on large codebases, but they are useful in different ways. They aren't going to bang out a new feature perfectly first time, but in the right hands they can dramatically accelerate all sorts of important activities, such as: - Understanding code. If code has no documentation, dumping it into an LLM can help a lot. - Writing individual functions, classes and modules. You have to be good at software architecture and good at prompting to use them in this way - you take on the role of picking out the tasks that can be done independently of the rest of the code. - Writing tests - again, if you have the skill and experience to prompt them in the right way.
    • mjr00 2y ago
      Yes, LLMs are very useful, when used properly. But the linked change request is not a good example of how they would be used by a typical software developer. The linked pull request is essentially output from a compiler that's been hardcoded. > Writing individual functions, classes and modules. You have to be good at software architecture and good at prompting to use them in this way - you take on the role of picking out the tasks that can be done independently of the rest of the code. If you have enough skill and understanding to do this, it means you already have enough general software development experience and domain-specific experience and experience with a specific, existing codebase to be in rarefied air. It's like saying, oh yeah a wrench makes plumbing easy. You just need to turn the wrench, and 25 years of plumbing knowledge to know where to turn it. > Writing tests - again, if you have the skill and experience to prompt them in the right way. This is very true and more accessible to most developers, though my big fear is it encourages people to crap out low-value unit tests. Not that they don't love to do that already.
• 20k 2y ago
  Eh it performed a 1:1 conversion of ARM NEON to wasm SIMD, which with the greatest will in the world is pretty trivial work. Its something that ML is good at, because its the same problem area as "translate this from english to french", but more mechanistic This is a task that would likely have taken as long to write by hand as the AI took to do it, given how long the actual task took to execute. 98% of the work is find and replace Don't get me wrong - this kind of thing is useful and cool, but you're mixing up the easy coding donkey work with the stuff that takes up time If you look at the actual prompt engineering part, its clear that this prompting produced extensively wrong results as well, which is tricky. Because it wasn't produced by a human, it requires extensive edge case testing and review, to make sure that the AI didn't screw anything up. If you have the knowledge to validate the output, it would have been quicker to write it by hand instead of reverse engineering the logic by hand. Its bumping the work off from writing it by hand, to the reviewers who now have to check your ML code because you didn't want to put in the work by hand So overall - while its extremely cool that it was able to do this, it has strong downsides for practical projects as well
  • WhitneyLand 2y ago
    Every time AI achieves something new/productive/interesting, cue the apologists who chime in to say “well yeah but that really just decomposes into this stuff so it doesn’t mean much”. I don’t get why people don’t understand that everything decomposes into other things. You can draw the line for when AI will truly blow your mind anywhere you want, the point is the dominoes keep falling relentlessly and there’s no end in sight.
    • 20k 2y ago
      The thing is, that's not true at all. AI is great for some tasks, and poor for other tasks. That's the reason to break it down like this, because people are trying to explain where AI will and won't revolutionise things, instead of following along with the already-popping AI bubble uncritically For example: AI's smash translation. They won't ever beat out humans, but as an automated solution? They rock. Natural language processing in general is great. If you want to smush in a large amount of text, and smush out a large amount of other text that's 98% equivalent but in a different structure, that's what AI is good for. Same for audio, or picture manipulation. It works because it has tonnes of training data to match your input against What AI cannot do, and will never be able to do, is take in a small amount of text (ie a prompt), and generate a large novel output with 100% accuracy. It simply doesn't have the training data to do this. AI excels in tasks where it is given large amounts of context and asked to perform a mechanistic operation, because its a tool which is designed to extract context and perform conversions based on that context due to its large amounts of training data. This is why in this article the author was able to get this to work: they could paste in a bunch of examples of similar mechanical conversions, and ask the AI to repeat the same process. It has trained on these kinds of conversions, so it works reasonably well Its great at this, because its not a novel problem, and you're giving it its exact high quality use case: take a large amount of text in, and perform some kind of structural conversion on it Where AI fails is when being asked to invent whole cloth solutions to new problems. This is where its very bad. So for example, if you ask an AI tool to solve your business problem via code, its going to suck. Because unless your business problem is something where there are literally 1000s examples of how to solve it, the AI simply lacks the training data to do what you ask it, it'll make gibberish It isn't the nature of the power of the AI, its that its inherently good for solving certain kinds of problems, vs other kinds of problems. It can't be solved with more training. The OPs problem is a decent use case for it. Most coding problems aren't. That's not that it isn't useful - people have already been successfully using them for tonnes of stuff - but its important to point out that its only done so well because of the specific nature of the use case Its become clear that AI requires someone of equivalent skill as the original use case to manage its output if 100% accuracy is required, which means that it can only ever function as an assistant for coders. Again, that's not to say it isn't wildly cool, its just acknowledging what its actually useful for instead of 'waiting to have my mind blown'
• chrisguilbeau 2y ago
  I'm a developer that primarily uses gh copilot for python dev. I find it pretty useful as an intelligent auto-completer that understands our project's style, and unusual decorators we use. What tools would you tell a copilot dev to try? For example, I have a $20/mo ChatGPT account and asking it to write code or even fix things hasn't worked very well. What am I missing?
  • byteknight 2y ago
    While I dont know your scenario as an avid user of both gpt and claude, I would recommend move away from Google style search queries, and begin conversing. The more you give the LLM the more you'll get close to what you want.
• cynicalpeace 2y ago
  It's making programming more boring and more of an admin task- which is sure to attract different types of people to the field.
  • woah 2y ago
    Seems like the exact opposite. The very example you are replying to is the mechanistic translation of one low level language to another, maybe one of the most boring tasks imaginable.
    • cynicalpeace 2y ago
      It goes both ways- manually manipulating bits is boring. But so is reviewing and rubber stamping code. It's like the transition from hand-crafted furniture to assembly line mass produced furniture. The assembly line brought its own excitement, but that excitement was not to be found on the actual assembly line.
  • mythrwy 2y ago
    You are getting downvoted but I agree. For whatever reason a good part of the joy of day to day coding for me was solving many trivial problems I knew how to solve. Sort of like putting a puzzle together. Now I think higher level and am more productive but it's not as much fun because the little easy problems aren't worth my time anymore.
• reitzensteinm 2y ago
  When GPT-4 came out, I worked on a project called Duopoly [1], which was a coding bot that aimed to develop itself as much as possible. The first commit was half a page of code that read itself in, asked the user what change they'd like to make, sent that to GPT-4, and overwrote itself with the result. The second commit was GPT-4 adding docstrings and type hints. Over 80% of the code was written by AI in this manner, and at some point, I pulled the plug on humans, and the last couple hundred commits were entirely written by AI. It was a huge pain to develop with how slow and expensive and flaky the GPT-4 API was at the time. There was a lot of dancing around the tiny 8k context window. After spending thousands in GPT-4 credits, I decided to mark it as proof of concept complete and move on developing other tech with LLMs. Today, with Sonnet and R1, I don't think it would be difficult or expensive to bootstrap the thing entirely with AI, never writing a line of code. Aider, a fantastic similar tool written by HN user anotherpaulg, wasn't writing large amounts of its own code in the GPT-4 days. But today it's above 80% in some releases [2]. Even if the models froze to what we have today, I don't think we've scratched the surface on what sophisticated tooling could get out of them. [1]: https://github.com/reitzensteinm/duopoly https://github.com/reitzensteinm/duopoly [2]: https://aider.chat/HISTORY.html https://aider.chat/HISTORY.html
• woah 2y ago
  LLMs excel at tasks with very clear instructions and parameters. Porting from one language to another is something that is one step away from being done by a compiler. Another place that I've used them is for initial scaffolding of React components.
• headcanon 2y ago
  Agreed, though to your point I think we'll end up seeing more induced demand long-term - This will enable more software to be built and maintained by same or fewer people (initially). Things that we wouldn't previously bother to do are now possible. - More software means more problems (not just LLM-generated bugs which can be handled by test suites and canary deploys, but overall features and domains of what software does) - This means skilled SWEs will still be in demand, but we need to figure out how to leverage them better. - Many codebases will be managed almost entirely by agents, effectively turning it into the new "build target". This means we need to build more tooling to manage these agents and keep them aligned on the goal, which will be a related but new discipline. SWEs would need to evolve skillsets but wasn't that always the deal?
• matsemann 2y ago
  I read that Meta is tasking all engineers with figuring out how they got owned by deepseek. Couldn't they just have asked an llm instead? After their claim of replacing all of us... I'm not too worried. If anything we're the last generation that knows how to debug and work through issues.
  • nkozyra 2y ago
    > If anything we're the last generation that knows how to debug and work through issues. I suspect that comment might soon feel like saying "not too worried about assembly line robots, we're the only ones who know how to screw on the lug nuts when they pop off"
    • lukan 2y ago
      Not before AGI and I still see no signs of it.
    • matsemann 2y ago
      Heh, yeah. But the llm in this instance only wrote 99% after the author guided it and prompted over and over again and even guided it how to start certain lines. I can do that. But can a beginner ever get to that level when not having that underlying knowledge?
    • Barrin92 2y ago
      I don't even see the irony in the comparison to be honest, being the assembly line robot controller and repairman is quite literally a better job than doing what the robot does by hand. If you're working in a modern manufacturing business the fact that you do your work with the aid of robots is hardly a sign of despair
      • nkozyra 2y ago
        I don't claim it's a sign of despair. Rather, it's a boots-dug-in belief that one does is special and cannot be done autonomously. I think it's wholly natural. Work, time, education ... these operate like sunk costs in our brains. I think what we're all learning in real-time is that human technology is perpetually aimed at replacing itself and we may soon see the largest such example of human utility displacement.
• JKCalhoun 2y ago
  > When companies claim that AI will replace developers, it isn't entirely bluster. I'm not so sure there isn't a bit of bluster in there. Imagine when you hand-coded in either machine code or assembly and then high level languages became a thing. I assume there was some handwringing then as well.
• jvanderbot 2y ago
  This is great. Really! Buuut... How do you get these tools to not fall over completely when relying on an existing non-public codebase that isn't visible in just the current file? Or, how do you get them to use a recent API that doesn't dominate their training data? Combining the both, I just cannot for the life of me get them to be useful beyond the most basic boilerplate. Arguably, SIMD intrinsics are a one-to-one translation boilerplate, and in the case of this PR, is a leetcode style, well-defined problem with a correct answer, and an extremely well-known api to use. This is not a dig on LLMs for coding. I'm an adopter - I want them to take my work away. But this is maybe 5% of my use case for an LLM. The other 95% is "Crawl this existing codebase and use my APIs that are not in this file to build a feature that does X". This has never materialized for me -- what tool should I be using?
  • simonw 2y ago
    "Or, how do you get them to use a recent API that doesn't dominate their training data?" Paste in the documentation or some examples. I do this all the time - "teaching" an LLM about an API it doesn't know yet is trivially easy if you take advantage of the longer context inputs to models these days.
    • jvanderbot 2y ago
      I've tried this. I've scraped example pages directly from github, and given them a 200 line file with the instructions "just insert this type of thing", and it will invariably use bad APIs. I'd be happy to share the example with you.
      • simonw 2y ago
        Go for it - can you share it in a Gist? I use this technique all the time. Here's one written-up example: https://simonwillison.net/2024/Mar/30/ocr-pdfs-images/ https://simonwillison.net/2024/Mar/30/ocr-pdfs-images/ - transcript here: https://gist.github.com/simonw/6a9f077bf8db616e44893a24ae1d36eb https://gist.github.com/simonw/6a9f077bf8db616e44893a24ae1d3...
• thefourthchime 2y ago
  There is a near-infinite demand for more applications. They simply become more specific and more niche. You can think to a point where everyone has their own set of applications custom for the exact workflow that they like. Just look at the options dialogue for Microsoft Word at least back in the day. It was pretty much everyone's pet feature over the last 10 years.
• lukan 2y ago
  "I hope we can put to rest the argument that LLMs are only marginally useful in coding" I more often heard the argument, they are not useful for them. I agree. If a LLM would be trained on my codebase and the exact libaries and APIs I use - I would use them daily I guess. But currently they still make too many misstake and mess up different APIs for example, so not useful to me, except for small experiments. But if I could train deepseek on my codebase for a reasonable amount(and they seemed to have improved on the training?), running it locally on my workstation: then I am likely in as well.
  • rane 2y ago
    The idea is that you give the libraries and APIs as context with your prompt.
    • lukan 2y ago
      Thank you. But that doesn't work for me. If you mean just the name of the version in the prompt? No way. If you mean all the libary and my code in the contextwindow? Way too small.
      • rane 2y ago
        Not _all_ the code. Just the relevant parts.
      • kragen 2y ago
        Maybe the LLM could issue queries to fetch parts of your codebase as it needs to look at them, using something like GDB or cscope.
      • smallerfish 2y ago
        Play around with projects in Claude for an hour. You'll see.
      • simonw 2y ago
        Give it examples of the library being used in the way you need. Here's an example transcript where I did that: https://gist.github.com/simonw/6a9f077bf8db616e44893a24ae1d36eb https://gist.github.com/simonw/6a9f077bf8db616e44893a24ae1d3...
• nkozyra 2y ago
  > it isn't entirely bluster "Development" is effectively translating abstractions of an intended operation to machine language. What I find kind of funny about the current state is we're using large language models to, like, spit out React or Python code. This use case is obviously an optimization to WASM, so a little closer to the metal, but at what point to programs (effectively suites of operations) just cut out the middleman entirely?
  • dboreham 2y ago
    I've wondered about this too. The LLM could just write machine code. But now a human can't easily review it. But perhaps TDD makes that ok. But now the tests need to be written in a human readable language so they can be checked. Or do they? And if the LLM is always right why does the code need to be tested?
    • nkozyra 2y ago
      At a certain point I don't see why a human needs to be in the loop at all. But I suppose that's the most dystopian part of it all.
      • kragen 2y ago
        Maybe the human has the money.
    • kragen 2y ago
      The LLM might be terrible at writing machine code directly. The kinds of mistakes I see GPT-4 making in Python, PostScript, or JS would be a much bigger problem in machine code. It "gets confused" and "makes mistakes" in ways very similar to humans. I haven't had a chance to try DeepSeek R1 yet.
• thrance 2y ago
  Did you even look at the generated code? DeepSeek simply rewrote part of the inference code making use of SIMD instructions on wasm. It literally boils down to inserting `if defined __wasm_simd128__` at some places then rewritting the loops to do floating point operations two by two instead of one after the other (which is where the 2X claim comes from). This is very standard and mostly boilerplate. Useful, sure, in that it saved some time in this particular case. But most of the AI-generated code I interact with is a hot unmaintainable mess of very verbose code, which I'd argue actually hurts the project in the long term.
  • simonw 2y ago
    "But most of the AI-generated code I interact with is a hot unmaintainable mess of very verbose code" That sounds like you're working with unskilled developers who are landing bad code.
• myrmi 2y ago
  I feel uncomfortably called out by all three points. What tools should I be trying to see what you are?
  • jeswin 2y ago
    I use my own tools and scripts, and those aren't for everyone - so I'm just gonna make some general suggestions. 1. You should try Aider. Even if you don't end up using it, you'll learn a lot from it. 2. Conversations are useful and important. You need to figure out a way to include (efficiently, with a few clicks) the necessary files into the context, and then start a conversation. Refine the output as a part of the conversation - by continuously making suggestions and corrections. 3. Conversational editing as a workflow is important. A better auto-complete is almost useless. 4. Github copilot has several issues - interface is just one of them. Conversational style was bolted on to it later, and it shows. It's easier to chat on Claude/Librechat/etc and copy files back manually. Or use a tool like Aider. 5. While you can apply LLMs to solve a particular lower level detail, it's equally effective (perhaps more effective) to have a higher level conversation. Start your project by having a conversation around features. And then refine the structure/scaffold and drill-down to the details. 6. Gradually, you'll know how to better organize a project and how to use better prompts. If you are familiar with best practices/design patterns, they're immediately useful for two reasons. (1) LLMs are also familar with those, and will help with prompt clarity; (2) Modular code is easier to extend. 7. Keep an eye on better performing models. I haven't used GPT-4o is a while, Claude works much, much better. And sometimes you might want to reach for o1 models. Other lower-end models might not offer any time savings; so stick to top tier models you can afford. Deepseek models have brought down the API cost, so it's now affordable to even more people. 8. Finally, it takes time. Just as any other tool.
    • Myrmornis 2y ago
      I agree with your overall point, and your despair at software engineers who are still refusing to acknowledge the value of these tools during the process of writing code. However > A better auto-complete is almost useless. That's not true. I agree that Copilot seemed unhelpful when I last tried it, but Cursor's autocomplete is extremely useful.
• simlevesque 2y ago
  I got incredible results in asking AIs for sql queries. I just enter my data and what I want the output to look like. Then I ask it to provide 10 different versions that might be faster. I test them all and tell it which is faster and then I ask it to make variations on this path. Then I ask it to add comments to the code which is the fastest. I verify the query, do some more test, and I'm good to go. I understand SQL pretty well but trying to make 10 different versions of one code would've took me at least an hour.
• gmt2027 2y ago
  If AI increases the productivity of a single engineer between 10-100x over the next decade, there will be a seismic shift in the industry and the tech giants will not walk away unscathed. There are coordination costs to organising large amounts of labour. Costs that scale non-linearly as massive inefficiencies are introduced. This ability to scale, provide capital and defer profitability is a moat for big tech and the silicon valley model. If a team of 10 engineers become as productive as a team of 100-1000 today, they will get serious leverage to build products and start companies in domains and niches that are not currently profitable because the middle managers, C-Suite, offices and lawyers are expensive coordination overhead. It is also easier to assemble a team of 10 exceptional and motivated partners than 1000 employees and managers. Another way to think about it is what happens when every engineer can marshal the AI equivalent of $10-100m dollars of labour? My optimistic take is that the profession will reach maturity when we become aware of the shift in the balance of power. There will be more solo engineers and we will see the emergence of software practices like the ones doctors, lawyers and accountants operate.
  • darkwater 2y ago
    This is a really interesting take that I don't see often in the wild. Actually, it's the first time I read someone saying this. But I think you are definitely onto something, especially if costs of AI are going to lower faster than expected even a few weeks ago.
  • WXLCKNO 2y ago
    Like darkwater's comment, this is my first time seeing this take and I like it a lot. I hate the idea of building a business to hundreds/thousands of employees, I love startups and small but highly profitable businesses. Having productivity be unleashed in this way with a small team of people I trust would be amazing.
  • AznHisoka 2y ago
    To play devils advocate, the main obstacle in launching a product doesn't involve the actual development/coding. Unless you're building something in hard-tech, it's relatively easy to build the run of the mill software. The obstacles are in marketing, selling it, building a brand/reputation, integrating it with lots of 3rd party vendors, and supporting it. So yes, you can build your own Salesforce, or your own Adobe Photoshop with a one-man crew much faster and easier. But that doesn't mean you, as an engineer can now build your own business selling it to companies who don't know anything about you.
• nh2 2y ago
  Challenge: I would really like somebody that has experience in LLM based coding tools to try and fix gnome-terminal: https://news.ycombinator.com/item?id=42814509 https://news.ycombinator.com/item?id=42814509
  • smokel 2y ago
    I really like this idea. However, it also highlights a key problem that LLMs don’t solve: while they’re great at generating code, that’s only a small part of real-world software development. Setting up a GitHub account, establishing credibility within a community, and handling PR feedback all require significant effort. In my view, lowering the barriers to open-source participation could have a bigger impact than these AI models alone. Some software already gathers telemetry and allows sharing bug reports, but why not allow the system to drop down to a debugger in an IDE? And why can’t code be shared as easily as in Google Docs, rather than relying on text-based files and Git? Even if someone has the skills to fix bugs, the learning curve for compilers, build tools, and Git often dilutes their motivation to contribute anything.
• fauigerzigerk 2y ago
  A long time ago, I held the grandiose title of software architect. My job was to describe in a mix of diagrams, natural language and method signatures what developers were supposed to do. The back and forth was agonising. They were all competent software engineers but communicating with them was often far more work than just writing the damn code myself. So yes I do believe that our trade has changed forever. But the fact that some of our coworkers will be AIs doesn't mean that communicating with them is suddenly free. Communcation comes with costs (and I don't mean tokens). That won't change. If you know your stuff really well, i.e. you work on a familiar codebase using a familiar toolset, the shortest path from your intentions to finished code will often not include anyone else - no humans and no AI either. In my opinion, "LLMs are only marginally useful in coding" is not true in general, but it could well be true for a specific person and a specific coding task.
• attractivechaos 2y ago
  I wonder what prompt they use. Before asking DeekSeek – is there a good post/video that walks through this procedure?
• sarasasa28 2y ago
  I mean, I don't know when do you retire in your countries. Here, it's at 65 years old (ridiculous) I am 30 and even before AI, I NEVER thought for a moment I would get to keep coding until I am f*king 65, lol
  • esafak 2y ago
    Why, ageism?
• plainOldText 2y ago
  Indeed, our trade has changed forever, and more specifically, we might have to alter our operational workflows in the entire industry as well. There are so many potential trajectories going forward for things to turn sour, I don't even know where to start the analysis. The level of sophistication an AI can achieve has no upper bound. I think we've had a good run so far. We've been able to produce software in the open with contributions from any human on the planet, trusting it was them who wrote the code, and with the expectation that they also understand it. But now things will change. Any developer, irrespective of skill and understanding of the problem and technical domains can generate sophisticated looking code. Unfortunately, we've reached a level of operational complexity in the software industry, that thanks to AI, could be exploited in a myriad ways going forward. So perhaps we're going to have to aggressively re-adjust our ways.
  • herval 2y ago
    I don't think trusting that someone wrote the code was ever a good assurance of anything, and I don't see how that changes with AI. There will always be certain _individuals_ who are more reliable than others, not because they handcraft code, but because they follow through with it (make sure it works, fix bugs after release, keep an eye to make sure it worked, etc). Yes, AI will enable exponentially more people to write code, but that's not a new phenomenon - bootcamps enabled an order of magnitude more people to become developers. So did higher level languages, IDEs, frameworks, etc. The march of technology has always been about doing more while having to understand less - higher and higher levels of abstraction. Isn't that a good thing?
    • plainOldText 2y ago
      Until now, the march of technology has taken place through a realm which was somewhat limited or slowed down only by our advancements in the physical and cognitive realities. This has given us ample time to catch up, to adjust. The cognitive reality of AI, and more specifically of AI+Humans in the context of a social and globally connected world, is on a higher level of sophistication and can unfold much faster, which in turn might generate entirely unexpected trajectories.
• Jerrrry 2y ago
  Maybe those software engineers should "lrn2code", just as the journalists, artists, and trucker drivers had to.
• casenmgreen 2y ago
  I may be wrong, but I think right now, from reading stories of people looking at use AI and having poor experiences, AI is useful and effective for some tasks and not for others, and this is an intrinsic property - it won't get better with bigger models. You need a task which fits well with what AI can do, which is basically auto-complete. If you have a task which does not fit well, it's not going to fly.
  • simonw 2y ago
    Right: LLMs have a "jagged frontier". They are really good at some things and terrible at other things, but figuring out WHAT those things are is extremely unintuitive. You have to spend a lot of time experimenting with them to develop good intuitions for where they make sense to apply. I expect the people who think LLMs are useless are people who haven't invested that time yet. This happens a lot, because the AI vendors themselves don't exactly advertise their systems as "they're great at some stuff and terrible at other stuff and here's how to figure that out".
• chefandy 2y ago
  GH copilot code completion is really the only one I’ve found to be consistently more of a benefit than a time sync. Even with the spiffy code generators using Claude or whatever, I often find myself spending as much time figuring out where the logical problem is than if I had just coded it myself, and you still need to know exactly what needs to be done. I’d be interested in seeing how much time they spent debugging the generated code and and how long they spent constructing and reconstructing the prompts. I’m not a software developer anymore as my primary career, so if the entire lower-half of the software development market went away catering wages as it did, it wouldn’t directly affect my professional life. (And with the kind of conceited, gleeful techno-libertarian shit I’ve gotten from the software world at large over the past couple of years as a type of specialized commercial artist, it would be tough to turn that schadenfreude into empathy. But we honestly need to figure out a way to stick together or else we’re speeding towards a less mechanical version of Metropolis.)
• kikimora 2y ago
  I don’t understand. When I asked DeepSeek how to find AWS IoT Thing creation time it suggested me to use “version” field and treat it as a Unix timestamp. This is obvious nonsense. How can this tool generate anything useful other than summaries of pre-existing text? My knowledge of theory behind LLMs also suggests this is all they can do reasonably well. When I see claims like this I suspect that either people around me somehow 10x better at promoting or they use different models.
  • simonw 2y ago
    You're making the mistake of treating an LLM like a search engine, and expecting it to be able to answer questions directly from its training data. Sometimes this works! But it's not guaranteed - this isn't their core strength, especially once you get into really deep knowledge of complex APIs. They are MUCH more useful when you use them for transformation tasks: feed in examples of the APIs you need to work with, then have them write new code based on that. Working effectively with LLMs for writing code is an extremely deep topic. Most people who think they aren't useful for code have been mislead into believing that the LLMs will just work - and that they don't first need to learn a whole bunch of unintuitive stuff in order to take advantage of the technology.
    • tejinderss 2y ago
      > Working effectively with LLMs for writing code is an extremely deep topic. There is a space for learning materials here. I would love to see books/trainings/courses on how to use AI effectively. I am more and more interested in this instead of learning new programming language of the week.
      • pertymcpert 2y ago
        At the moment the space is moving so fast that anyone who tries to write a book will be outdated by the time it's published. The only option is to dive in yourself or give up and wait for things to settle down and plateau.
    • swfsql 2y ago
• karmasimida 2y ago
  I am mixed on this point. I 100% agree with you our trade is changed forever. On the other hand, I am writing like 1000+ LOC daily, without much compromise on quality and my mental health, and thought of writing some code that is necessary but feels like a chore is not longer the case. The boost in output is incredible.
• submeta 2y ago
  The dev jobs won‘t go away, but they will change. Devs will be more and more like requirements engineers who need to understand the problem to then write prompts with the peoper context so that the llm can produce valuable and working code. And the next level will be to prompt llms to generate prompts for llms to produce code and solutions. But already I hire less and less developers for smaller tasks. The things that I‘d assign to a dev in Ukraine to explore an idea, do a data transformation, make a UI for the internal company tool. I can do these things quicker with llm than trying to find a dev and explain the task.
  • WXLCKNO 2y ago
    I think what you're describing is going to be a very short transitional period. Once current AI gets good enough, the people micromanaging parts of it will do more to hinder the process than to help it. One person setting the objectives and the AI handling literally everything else including brainstorming issues etc, is going to be all that's needed.
    • yoyohello13 2y ago
      > One person setting the objectives and the AI handling literally everything else including brainstorming issues etc, is going to be all that's needed. A person just setting the prompt and letting the AI do all the work is not adding any additional value. Any other person can come in and perform the exact same task. The only way to actually provide differentiation in this scenario is to either build your own models, or micromanage the outputs.
    • esafak 2y ago
      What are you going to do when the output is wrong? You're not expecting it to always be right, are you?
      • rybosworld 2y ago
        I said this in another comment but look at the leading chess engines. They are already so far above human level of play that having a human override the engines choice will nearly always lead to a worse position. > You're not expecting it to always be right, are you? I think another thing that gets lost in these conversations is that humans already produce things that are "wrong". That's what bugs are. AI will also sometimes create things that have bugs and that's fine so long as they do so at a rate lower than human software developers. We already don't expect humans to write absolutely perfect software so it's unreasonable to expect that AI will do so.
• xd 2y ago
  The thing with programming, to do it well, you need to fully understand the problem and then you implement the solution expressing it in code. AI will be used to create code based on a deficit of clear understanding and we will end up with a hell of a lot of garbage code. I foresee the industry demand for programmers sky rocketing in the future, as companies scramble to unfuck the mountains of shit code they lash up over the coming years. It's just a new age of copy paste coders.
• Myrmornis 2y ago
  In my experience a lot of it is (d) defaulting to criticizing new things, especially things that are "trendy" or "hot" and (e) not liking to admit that one's own work can partially be done by such a trendy or hot thing.
• unshavedyak 2y ago
  I’m still just looking for a good workflow where I can stay in my editor and largely focus on code, rather than trying to explain what I want to an LLM. I want to stay in Helix and find a workflow that “just works”. Not sure even what that looks like yet
  • WXLCKNO 2y ago
    Just to clarify, something like Cursor doesn't fit your needs right?
    • unshavedyak 2y ago
      I've not tried tbh. Most of the workflows i've seen (i know i looked at Cursor, but it's been a while) appear to be to write lengthy descriptions of what you want it to do. As well as struggling with the amount of context you need to give it because context windows are way too small. I feel like i want a more intuitive, natural process. Purely for illustration -- because i have no idea what the ideal workflow is -- I'd want something that could allow for large autocomplete without changing much. Maybe a process by which i write a function, args, docstring on the func and then as i write the body autocomplete becomes multiline and very good. Something like this could be an extension of the normal autocomplete that most of us know and love. A lack of talking to an AI, and more about just tweaking how you write code to be very metadata rich so AIs have a rich understanding of intent. I know there are LLM LSPs which sort of do this. They can make shorter autocompletes that are logical to what you're typing, but i think i'm talking about something larger than that. So yea.. i don't know, but i just know i have hated talking to the LLM. Usually it felt like "get out of the way, i can do it faster" sort of thing. I want something to improve how we write code, not an intern that we manage. If that makes sense.
      • stnmtn 2y ago
        I'd recommend trying out Cursor, I personally find it does more or less what you want out of the box.
        • 2y ago
• kemiller 2y ago
  My observation in my years running a dev shop was that there are two classes of applications that could get built. One was the high-end, full-bore model requiring a team of engineers and hundreds of thousands of dollars to get to a basic MVP, which thus required an economic opportunity in at least the tends of millions. The other, very niche or geographically local businesses that can get their needs met with a self-service tool, max budget maybe $5k or so. Could stretch that to $25k if you use offshore team to customize. But 9/10 incoming leads had budgets between $25k and $100k. We just had to turn them away. There's nothing meaningful you can do with that range of budget. I haven't seen anything particularly change that. Self-service tools get gradually better, but not enough to make a huge difference. The high end if anything has receded even faster as dev salaries have soared. AI coding, for all its flaws now, is the first thing that takes a chunk out of this, and there is a HUGE backlog of good-but-not-great ideas that are now viable. That said, this particular story is bogus. He "just wrote the tests" but that's a spec — implementing from a quality executable spec is much more straightforward. Deepseek isn't doing the design, he is. Still a massive accelerant.
• spease 2y ago
  The nature of this PR looks like it’s very LLM-friendly - it’s essentially translating existing code into SIMD. LLMs seem to do well at any kind of mapping / translating task, but they seem to have a harder time when you give them either a broader or less deterministic task, or when they don’t have the knowledge to complete the task and start hallucinating. It’s not a great metric to benchmark their ability to write typical code.
  • kridsdale3 2y ago
    Sure, but let's still appreciate how awesome it is that this very difficult (for a human) PR is now essentially self-serve. How much hardware efficiency have we left on the the table all these years because people don't like to think about optimal use of cache lines, array alignment, SIMD, etc. I bet we could double or triple the speeds of all our computers.
    • spease 2y ago
      Hopefully this results in some big improvements with compilation.
• qrsjutsu 2y ago
  [dead]
• redcobra762 2y ago
  When tools increase a worker's efficiency, it's rare that the job is lost. It's much more common that the demand for that job changes to take advantage of the productivity growth. This is why the concerns from Keynes and Russel about people having nothing to do as machines automated away more work ended up being unfounded. We fill the time... with more work. And workers that can't use these tools to increase their productivity will need to be retrained or moved out of the field. That is a genuine concern, but this friction is literally called the "natural rate of unemployment" and happens all the time. The only surprise is we expected knowledge work to be more inoculated from this than it turns out to be.
• Vegenoid 2y ago
  > Our trade has changed forever, and there's no going back Forever? Hell, it hasn't even existed for a lifetime yet.
  • kragen 2y ago
    01945 to 02025 is 80 years, longer than human life expectancy at birth. What's your definition of a "lifetime"?
    • Vegenoid 2y ago
      The point was not to be very accurate, it was to make the point that it has existed for a very short amount of time on the scale of humanity. Quibbling over whether software engineering started in the 40s or the 50s and whether that is greater or less than an average life expectancy is beside the point.
      • kragen 2y ago
        People posting comments without caring whether they are true or false undermines the presumption of good faith that underlies rational discourse. Please stop posting such comments on this site. Instead, only post comments that you have some reason to believe are true.
        • Vegenoid 2y ago
          It is not about not caring if the statement is true or false. The statement neither absolutely true nor absolutely false, because there is no absolute definition of when software engineering started, or what "a lifetime" is. It is about making a statement that communicates information. However, if I must prove that my statement can reasonably considered "true" in order to prove that it does communicate the short span of time for which software engineering has existed: - Avg. life expectancy in USA: 77.5 years - 2025 - 77.5 = 1947.5 - In 1945, Turing published "Proposed Electronic Calculator" - The first stored-program computer was built in 1948 - The term "software engineering" wasn't used until the 1960s If you want to define "software engineering" such that it is more than 77.5 years old, that's fine. But saying that software engineering is less than 77.5 years old is clearly a reasonable stance. Please stop berating me for a perfectly harmless and reasonably accurate statement. If you're going to berate me for anything, it should be for its brevity and lack of discussion-worthy content. But those are posted all the time.
• kragen 2y ago
  It's possible that the previous tools just weren't good enough yet. I play with GPT-4 programming a lot, and it usually takes more work than it would take to write the code myself. I keep playing with it because it's so amazing, but it isn't to the point where it's useful to me in practice for that purpose. (If I were an even worse coder than I am, it would be.) DeepSeek looks like it is.
• nuancebydefault 2y ago
  AI will only ever be able to develop what it is asked/prompted for. The question is often ill formed, resulting in an app that does not do what you want. So the prompt needs to be updated, the result needs to be evaluated and tweaks need to be done to the code with or without help of AI. In fact, from a distance seen, the software development pattern in AI times stays the same as it was pre-AI, pre-SO, pre-IDE as well as pre-internet. Just to say, sw developers will still be sw developers.
• fennecfoxy 2y ago
  I'm not entirely worried, yet. I don't think that LLMs produce code/architecture that is trustworthy enough for them to operate independently. Of course suits & ties will believe it and kill many dev jobs eventually, but as per usual wtf do they know about how things work on the ground. Executives thrive on NYT articles and hearsay from LinkedIn.