4 ms·

Where are the LLM leaderboards for software estimation accuracy? I have been using Claude Code and Aider and I do think they provide incredibly exciting potent

by xrd 2y ago

Where are the LLM leaderboards for software estimation accuracy? I have been using Claude Code and Aider and I do think they provide incredibly exciting potential. I can spin up new projects with mind boggling results. And, I can start projects in domains where I previously had almost no experience. It is truly exciting. AND... The thing I worry most about is that now non-technical managers can go into Claude and say "build me XYZ!" and the AI will build a passable first version. But, any experienced software person knows that the true cost of software is in the maintenance. That's 90% of the cost. Reducing that starting point to zero cost only reduces the total cost of software by 10%, but people are making it seem like you no longer need a software engineer that understands complex systems. Or, maybe that is just my fears vocalized. I have seen LLMs dig into old and complex codebases and fix things that I was not expecting them to handle. I tend to write a lot of tests in my code, so I can see that the tests pass and the code compiles. The barbarians have come over the walls, truly. But, IMHO, there is still a space where we cannot ask any of the AI coding tools to review a spec and say "Will you get this done in 2 months?" I don't think we are there, yet. I don't see that the context window is big enough to jam entire codebases inside it, and I don't yet see that these tools can anticipate a project growing into hundreds of files and managing the interdepedencies between them. They are good at writing tests, and I am impressed by that, so there is a pathway. I'm excited to understand more about how aider creates a map of the repository and effectively compresses the information in ways similar to how I keep track of high level ideas. But, it still feels very early and there are gaps. And, I think we are in for a rude awakening when people start thinking this pushes the cost and complexity of software projects to zero. I love the potential for AI coding, but it feels like it is dangerously marketing and sales driven right now and full of hype.

someothherguyy 2y ago
> I can spin up new projects with mind boggling results Boggle a skeptical mind
- xrd 2y ago
  Meaning, give you an example? This morning I created a new project. I provided a postgres database URL to a remote service (with a non-standard connection string, includes a parameter "?sslmode=require"). Then, I said: * "Write me a fastapi project to connect to a postgres database using a database url." * "Retrieve the schema from the remote database." It used psql to connect, retrieves the schema. That was unexpected, it figured out not only a coding task, but an external tool to connect to a database and did it without anything more than me providing the DATABASE_URL. Actually, I should say, I told it to look inside the .env file, and it did that. I had that URL wrong initially, so I told it to reload once I corrected it. It never got confused by my disorganization. * It automatically added sqlalchemy models and uses pydantic once it figured out the schema. * "Create a webpage that lets me review one table." * "Rewrite to use tailwindcss." It adds the correct tailwindcss CDN imports. * It automatically adds a modal dialog when I click on one of the records. * It categorized fields in the database into groupings inside the modal, groupings that do indeed make sense. I know the devil is in the details, or in the future. I'm sure there are gaping security holes. But, this saved me a lot of time and it works.
  - xrd 2y ago
    And, the update is that in the last hour claude somehow removed the rendering of the actual items. It is clever in troubleshooting: it created mocks if the data could not be loaded, it added error messages with the number of items retrieved. But, it does not have access to my browser console nor the DOM, and therein lies the problem. It is slow to copy and paste back and forth from the browser into the terminal. But, this feels like a great opportunity for a browser extension. But, my takeaway is that I could have fixed this exact problem in five minutes if I had written the JS code. But, I have not looked at anything other than glancing at the diffs that fly by in the session.
- Workaccount2 2y ago
  I used claude to create a piece of software that can render gerber files (essentially vector files used in electronics manufacturing), overlay another gerber file on top of it with exact alignment, and then provide a GUI for manually highlighting components. The program then calculates the centroid, the rotation, and prompts for a component designator. This all then gets stored in a properly formatted Place file, which is used for automating assembly. The day before this I was quote $1000/yr/user for software that could do this for us.
- xur17 2y ago
  I used Claude + aider to create a webhook forwarding service (receives webhooks, forwards them to multiple destinations, and handles retries, errors, etc). Included a web interface for creating new endpoints and monitoring errors, and made it fully dockerized and ready to be deployed into a kubernetes cluster. Took me like 3 hours and a few dollars in api credits to build something that would have taken me multiple days on my own. Since it's just an internal tool for our dev environments that already does what we need, I don't care that much about maintainability (especially if it takes 3 hours to build from scratch). That said, the code is reasonably usable (there was one small thing it got stuck on at the end that I assisted with).
  - xrd 2y ago
    Claude Code and Aider, or aider using claude sonnet as the model? If you are using both claude code and aider I would love to know why.
    - xur17 2y ago
      Aider + Claude sonnet as the model.
- brulard 2y ago
  I wanted a whiteboard app - simple, infinite, zoomable, scrollable svg-based area with custom shapes and elements. It just needed prompts like "add zoom", "add a rect shape", "add control points for resize", "add text shape", "make position attributes able to reference another objects properties, + basic arithmetic" (for example to make line connect 2 objects, or size of one mirror size of another, etc.). It made all of these with so little effort from me. I would never undertake such a project without an LLM.
mentalgear 2y ago
The "coding benchmarks" like "SWE-verified" are actually of very low quality and the answer riddled with problems. Good Explainer: "The Disturbing Reality of AI Coding" https://www.youtube.com/watch?v=QnOc_kKKuac https://www.youtube.com/watch?v=QnOc_kKKuac
rurp 2y ago
This largely fits with a pattern I've been seeing with LLM coding. The models are often helpful, sometimes extremely so, when it comes to creating prototypes or other small greenfield projects. They can also be great at producing a snippet of code in an unfamiliar framework or language. But when it comes to modifying a large, messy, complicated code base they are much less helpful. Some people find them a useful as a beefed up autocomplete, while others don't see enough gains to offset the time/attention to use them. I think a lot of arguments about LLM coding ability stem from people using them for the former or the latter and having very different experiences.
aaalll 2y ago
There is a tool cool lovable that is basically targeted at that exact thing having designers and product managers get something that kinda works
inerte 2y ago
I've been programming for 20 years, just so y'all know I have at least some level of competence. I tried Vibe Coding for the first time on Friday and was blown away. It was awesome. I met some people (all programmers) at a party on Friday and excitedly told them about it. One of them tried, he loved. Then yesterday I read a LinkedIn (Lunatic?) post about "Vibe Design", where PMs will just tell the computer what to do using some sort of visual language, where you create UI elements and drop them on a canvas, and AI makes your vision come true, etc, etc... And my first thought was: "Wait a minute, I've seen this movie before. Back on late 90s / early 2000s it was the 4th generation programming language, and Visual Basic would allow anyone to code any system"... And while it's true Visual Basic did allow a BUNCH of people to make money building and selling systems to video rental shops and hair saloons, programmers never went away. I welcome anyone building more software. The tools will only get better. And programmers will adapt them, and it will make us better too, and we will still be needed.
axkdev 2y ago
> I'm excited to understand more about how aider creates a map of the repository and effectively compresses the information in ways similar to how I keep track of high level ideas. Run /map. It's nothing fancy, really.
- xrd 2y ago
  Ok, I meant to say, I know about /map, but there is so much to explore here with an open source coding tool and LSP. It feels like you could build some very powerful ways to represent connections using a lot of cool tools that are in use inside our editors. More structured and more relevant that the poor representations I can keep inside my crappy head.

by xrd 2y ago

someothherguyy 2y ago

> I can spin up new projects with mind boggling results Boggle a skeptical mind

xrd 2y ago
Meaning, give you an example? This morning I created a new project. I provided a postgres database URL to a remote service (with a non-standard connection string, includes a parameter "?sslmode=require"). Then, I said: * "Write me a fastapi project to connect to a postgres database using a database url." * "Retrieve the schema from the remote database." It used psql to connect, retrieves the schema. That was unexpected, it figured out not only a coding task, but an external tool to connect to a database and did it without anything more than me providing the DATABASE_URL. Actually, I should say, I told it to look inside the .env file, and it did that. I had that URL wrong initially, so I told it to reload once I corrected it. It never got confused by my disorganization. * It automatically added sqlalchemy models and uses pydantic once it figured out the schema. * "Create a webpage that lets me review one table." * "Rewrite to use tailwindcss." It adds the correct tailwindcss CDN imports. * It automatically adds a modal dialog when I click on one of the records. * It categorized fields in the database into groupings inside the modal, groupings that do indeed make sense. I know the devil is in the details, or in the future. I'm sure there are gaping security holes. But, this saved me a lot of time and it works.
- xrd 2y ago
  And, the update is that in the last hour claude somehow removed the rendering of the actual items. It is clever in troubleshooting: it created mocks if the data could not be loaded, it added error messages with the number of items retrieved. But, it does not have access to my browser console nor the DOM, and therein lies the problem. It is slow to copy and paste back and forth from the browser into the terminal. But, this feels like a great opportunity for a browser extension. But, my takeaway is that I could have fixed this exact problem in five minutes if I had written the JS code. But, I have not looked at anything other than glancing at the diffs that fly by in the session.
Workaccount2 2y ago
I used claude to create a piece of software that can render gerber files (essentially vector files used in electronics manufacturing), overlay another gerber file on top of it with exact alignment, and then provide a GUI for manually highlighting components. The program then calculates the centroid, the rotation, and prompts for a component designator. This all then gets stored in a properly formatted Place file, which is used for automating assembly. The day before this I was quote $1000/yr/user for software that could do this for us.
xur17 2y ago
I used Claude + aider to create a webhook forwarding service (receives webhooks, forwards them to multiple destinations, and handles retries, errors, etc). Included a web interface for creating new endpoints and monitoring errors, and made it fully dockerized and ready to be deployed into a kubernetes cluster. Took me like 3 hours and a few dollars in api credits to build something that would have taken me multiple days on my own. Since it's just an internal tool for our dev environments that already does what we need, I don't care that much about maintainability (especially if it takes 3 hours to build from scratch). That said, the code is reasonably usable (there was one small thing it got stuck on at the end that I assisted with).
- xrd 2y ago
  Claude Code and Aider, or aider using claude sonnet as the model? If you are using both claude code and aider I would love to know why.
  - xur17 2y ago
    Aider + Claude sonnet as the model.
brulard 2y ago
I wanted a whiteboard app - simple, infinite, zoomable, scrollable svg-based area with custom shapes and elements. It just needed prompts like "add zoom", "add a rect shape", "add control points for resize", "add text shape", "make position attributes able to reference another objects properties, + basic arithmetic" (for example to make line connect 2 objects, or size of one mirror size of another, etc.). It made all of these with so little effort from me. I would never undertake such a project without an LLM.

mentalgear 2y ago

The "coding benchmarks" like "SWE-verified" are actually of very low quality and the answer riddled with problems. Good Explainer: "The Disturbing Reality of AI Coding" https://www.youtube.com/watch?v=QnOc_kKKuac https://www.youtube.com/watch?v=QnOc_kKKuac

rurp 2y ago

This largely fits with a pattern I've been seeing with LLM coding. The models are often helpful, sometimes extremely so, when it comes to creating prototypes or other small greenfield projects. They can also be great at producing a snippet of code in an unfamiliar framework or language. But when it comes to modifying a large, messy, complicated code base they are much less helpful. Some people find them a useful as a beefed up autocomplete, while others don't see enough gains to offset the time/attention to use them. I think a lot of arguments about LLM coding ability stem from people using them for the former or the latter and having very different experiences.

aaalll 2y ago

There is a tool cool lovable that is basically targeted at that exact thing having designers and product managers get something that kinda works

inerte 2y ago

I've been programming for 20 years, just so y'all know I have at least some level of competence. I tried Vibe Coding for the first time on Friday and was blown away. It was awesome. I met some people (all programmers) at a party on Friday and excitedly told them about it. One of them tried, he loved. Then yesterday I read a LinkedIn (Lunatic?) post about "Vibe Design", where PMs will just tell the computer what to do using some sort of visual language, where you create UI elements and drop them on a canvas, and AI makes your vision come true, etc, etc... And my first thought was: "Wait a minute, I've seen this movie before. Back on late 90s / early 2000s it was the 4th generation programming language, and Visual Basic would allow anyone to code any system"... And while it's true Visual Basic did allow a BUNCH of people to make money building and selling systems to video rental shops and hair saloons, programmers never went away. I welcome anyone building more software. The tools will only get better. And programmers will adapt them, and it will make us better too, and we will still be needed.

axkdev 2y ago

> I'm excited to understand more about how aider creates a map of the repository and effectively compresses the information in ways similar to how I keep track of high level ideas. Run /map. It's nothing fancy, really.

xrd 2y ago
Ok, I meant to say, I know about /map, but there is so much to explore here with an open source coding tool and LSP. It feels like you could build some very powerful ways to represent connections using a lot of cool tools that are in use inside our editors. More structured and more relevant that the poor representations I can keep inside my crappy head.