5 ms·

I believe that they work particularly well for CRUD in known frameworks like Rails. OTOH I tried building a native Windows Application using Direct2D in Rust a

by kubb 1y ago

I believe that they work particularly well for CRUD in known frameworks like Rails. OTOH I tried building a native Windows Application using Direct2D in Rust and it was a disaster. I wish people could be a bit more open about what they build.

quantumHazer 1y ago
yeah, tipically they are building a to do list and organizer app and have not found that github is flooded with college students' project of their revolutionary to-do apps
- kubb 1y ago
  I don’t want to dismiss or disrespect anyone’s work. But I never see precise descriptions of categories of tasks that work well, it’s all based on vibes.
sdesol 1y ago
> I wish people could be a bit more open about what they build. I would say for the last 6 months, 95% of the code for my chat app (https://github.com/gitsense/chat https://github.com/gitsense/chat) was AI generated (98% human architected). I believe what I created in the last 6 months was far from trivial. One of the features that AI helped a lot with, was the AI Search Assistant feature. You can learn more about it here https://github.com/gitsense/chat/blob/main/packages/chat/widgets/app/components/chat-builder/trees/help/documentation/understanding-your-personalized-ai-search-assistant/1.md https://github.com/gitsense/chat/blob/main/packages/chat/wid... As a debugging partner, LLMs are invaluable. I could easily load all the backend search code into context and have it trace a query and create a context bundle with just the affected files. Once I had that, I would use my tool to filter the context to just those files and then chat with the LLM to figure out what went wrong or why the search was slow. I very much agree with the author of the blog post about why LLMs can't really build software. AI is an industry game changer as it can truly 3x to 4x senior developers in my opinion. I should also note that I spend about $2 a day on LLM API calls (99% to Gemini 2.5 Flash) and I probably have to read 200+ LLM generated messages a day and reply back in great detail about 5 times a day (think of an email instead of chat message). Note: The demo on that I have in the README hasn't been setup, as I am still in the process of finalizing things for release but the NPM install instructions should work.
- QuadmasterXLII 1y ago
  What happens when you tell the AI to set up the demo in the README?
  - sdesol 1y ago
    It summarized the instructions required to install and setup. It (Gemini and Sonnet) did fail to mention that I need to setup a server and create a DNS entry for the sub domain.
- leptons 1y ago
  > probably have to read 200+ LLM generated messages a day and reply back in great detail about 5 times a day (think of an email instead of chat message). I can think of nothing more tiresome than having to read 200 emails a day, or LLM chat messages. And then respond in detail 5 of those times. It wouldn't lead to "3x to 4x" performance gain after tallying up all the time reading messages and replying. I'm not sure people that use LLMs this way are really tracking their time enough to say with any confidence that "3x to 4x" is anywhere close to reality.
  - sdesol 1y ago
    A lot of the messages are revisions so it is not as tedious as it may seem. As for the "3x to 4x", this is my own experience. It is possible that I am an outlier, but 80% of the generated AI code that I have are one-shot. I spend an hour or two (usually spread over days thinking about the problem) to accomplish something that would have taken a week or more for me to do. I'm going to start producing metrics regarding how much code is AI generated along with some complexity metrics. I am obviously bias, but this definitely feels like a paradigm shift and if people do not fully learn to adapt to it, it might be too late. I am not sure if you have ever watched Gattaca, but this sort of feels like it...the astronaut part, that is. The profession that I have known for decades is starting to feel very different, in the same way that while watching Gattaca, my perception of astronauts changed. It was strange, but plausible and that is what I see for the software industry. Those that can articulate the problem I believe will become more valuable than the silent genius.
andrewmutz 1y ago
I agree that it is probably easier for an LLM to write good code in any framework (like Rails) that has a lot of well-documented opinions about how things should be done. If there is a "right" place to put things, or a "right" way to model problems in a framework, its more likely that the model's opinions are going to line up with the human engineer's opinions.
- alkonaut 1y ago
  Also - that's easy for everyone. It's basically a framework so rigid/simple (Those are adjacent concepts for frameworks) that the business logic is almost boilerplate. That is, so long as you stay inside the guard rails. Ask it to make something in a rails app that's slightly beyond the CRUD scope and it will suffer - much like most humans would. So it's not that it's bad to let bots do boilerplate. But using very qualified humans for that to begin with was a waste to begin with. Hopefully in a few years none of us will need to do ANY part of CRUD work and we can do only the fun parts of software development.-
- tripzilch 1y ago
  But isn't it crazy that while it's been impressively great at translating between human languages from the start, it's incapable of translating these well-documented best-ways-to-do-it things across domains or even programming languages.
Aeolun 1y ago
I thought Claude got significantly smarter when I started using Rust. The big problem there is that I don’t understand the rust myself :P
- klabb3 1y ago
  It’s the style. Responses are always eloquent and well structured. When you look at output for a domain you don’t know well, you give it benefit of the doubt because it sounds like a highly competent human, so you react similarly. When you use it with something you know very deeply, you naturally look more for substance rather than form, and thus spot the mistakes much easier. This breaks most illusions of amazing reasoning abilities etc. My ChatGPT is amazingly competent at gardening! Well, that’s how it feels anyway. Is it correct? I have no idea. It sounds right. Fortunately, it’s just a new hobby for me and the stakes are low. But generally I think it’s much better to be paranoid than gullible when it comes to confident sounding ramblings, whether it’s from an LLM or a marketing guru.
stingraycharles 1y ago
I recently built a data streaming connector in Go with all kinds of bells and whistles attached (yaml based data parsers, circuit breakers, e2e stress testing frameworks, etc). Worked like a charm, I estimate it made two months of work about two weeks. But you need to get your workflow right.
wg0 1y ago
The author isn't wrong that LLMs don't work like an engineer and often fail miserably. Here's what works however: Mostly CRUD apps or REST API in Rails, Django or other Microframeworks such as FastAPI etc. Or with React. In that too, focus on small components and small steps or else you'll fail to get the results.

by kubb 1y ago

quantumHazer 1y ago

yeah, tipically they are building a to do list and organizer app and have not found that github is flooded with college students' project of their revolutionary to-do apps

kubb 1y ago
I don’t want to dismiss or disrespect anyone’s work. But I never see precise descriptions of categories of tasks that work well, it’s all based on vibes.

sdesol 1y ago

> I wish people could be a bit more open about what they build. I would say for the last 6 months, 95% of the code for my chat app (https://github.com/gitsense/chat https://github.com/gitsense/chat) was AI generated (98% human architected). I believe what I created in the last 6 months was far from trivial. One of the features that AI helped a lot with, was the AI Search Assistant feature. You can learn more about it here https://github.com/gitsense/chat/blob/main/packages/chat/widgets/app/components/chat-builder/trees/help/documentation/understanding-your-personalized-ai-search-assistant/1.md https://github.com/gitsense/chat/blob/main/packages/chat/wid... As a debugging partner, LLMs are invaluable. I could easily load all the backend search code into context and have it trace a query and create a context bundle with just the affected files. Once I had that, I would use my tool to filter the context to just those files and then chat with the LLM to figure out what went wrong or why the search was slow. I very much agree with the author of the blog post about why LLMs can't really build software. AI is an industry game changer as it can truly 3x to 4x senior developers in my opinion. I should also note that I spend about $2 a day on LLM API calls (99% to Gemini 2.5 Flash) and I probably have to read 200+ LLM generated messages a day and reply back in great detail about 5 times a day (think of an email instead of chat message). Note: The demo on that I have in the README hasn't been setup, as I am still in the process of finalizing things for release but the NPM install instructions should work.

QuadmasterXLII 1y ago
What happens when you tell the AI to set up the demo in the README?
- sdesol 1y ago
  It summarized the instructions required to install and setup. It (Gemini and Sonnet) did fail to mention that I need to setup a server and create a DNS entry for the sub domain.
leptons 1y ago
> probably have to read 200+ LLM generated messages a day and reply back in great detail about 5 times a day (think of an email instead of chat message). I can think of nothing more tiresome than having to read 200 emails a day, or LLM chat messages. And then respond in detail 5 of those times. It wouldn't lead to "3x to 4x" performance gain after tallying up all the time reading messages and replying. I'm not sure people that use LLMs this way are really tracking their time enough to say with any confidence that "3x to 4x" is anywhere close to reality.
- sdesol 1y ago
  A lot of the messages are revisions so it is not as tedious as it may seem. As for the "3x to 4x", this is my own experience. It is possible that I am an outlier, but 80% of the generated AI code that I have are one-shot. I spend an hour or two (usually spread over days thinking about the problem) to accomplish something that would have taken a week or more for me to do. I'm going to start producing metrics regarding how much code is AI generated along with some complexity metrics. I am obviously bias, but this definitely feels like a paradigm shift and if people do not fully learn to adapt to it, it might be too late. I am not sure if you have ever watched Gattaca, but this sort of feels like it...the astronaut part, that is. The profession that I have known for decades is starting to feel very different, in the same way that while watching Gattaca, my perception of astronauts changed. It was strange, but plausible and that is what I see for the software industry. Those that can articulate the problem I believe will become more valuable than the silent genius.

andrewmutz 1y ago

I agree that it is probably easier for an LLM to write good code in any framework (like Rails) that has a lot of well-documented opinions about how things should be done. If there is a "right" place to put things, or a "right" way to model problems in a framework, its more likely that the model's opinions are going to line up with the human engineer's opinions.

alkonaut 1y ago
Also - that's easy for everyone. It's basically a framework so rigid/simple (Those are adjacent concepts for frameworks) that the business logic is almost boilerplate. That is, so long as you stay inside the guard rails. Ask it to make something in a rails app that's slightly beyond the CRUD scope and it will suffer - much like most humans would. So it's not that it's bad to let bots do boilerplate. But using very qualified humans for that to begin with was a waste to begin with. Hopefully in a few years none of us will need to do ANY part of CRUD work and we can do only the fun parts of software development.-
tripzilch 1y ago
But isn't it crazy that while it's been impressively great at translating between human languages from the start, it's incapable of translating these well-documented best-ways-to-do-it things across domains or even programming languages.

Aeolun 1y ago

I thought Claude got significantly smarter when I started using Rust. The big problem there is that I don’t understand the rust myself :P

klabb3 1y ago
It’s the style. Responses are always eloquent and well structured. When you look at output for a domain you don’t know well, you give it benefit of the doubt because it sounds like a highly competent human, so you react similarly. When you use it with something you know very deeply, you naturally look more for substance rather than form, and thus spot the mistakes much easier. This breaks most illusions of amazing reasoning abilities etc. My ChatGPT is amazingly competent at gardening! Well, that’s how it feels anyway. Is it correct? I have no idea. It sounds right. Fortunately, it’s just a new hobby for me and the stakes are low. But generally I think it’s much better to be paranoid than gullible when it comes to confident sounding ramblings, whether it’s from an LLM or a marketing guru.

stingraycharles 1y ago

I recently built a data streaming connector in Go with all kinds of bells and whistles attached (yaml based data parsers, circuit breakers, e2e stress testing frameworks, etc). Worked like a charm, I estimate it made two months of work about two weeks. But you need to get your workflow right.

wg0 1y ago

The author isn't wrong that LLMs don't work like an engineer and often fail miserably. Here's what works however: Mostly CRUD apps or REST API in Rails, Django or other Microframeworks such as FastAPI etc. Or with React. In that too, focus on small components and small steps or else you'll fail to get the results.