4 ms·

Can you maybe give an example you’ve encountered of an algorithm or a data structure that LLMs cannot handle well? In my experience implementing algorithms fro

Can you maybe give an example you’ve encountered of an algorithm or a data structure that LLMs cannot handle well? In my experience implementing algorithms from a good comprehensive description and keeping track of data models is where they shine the most.

manwe150 1y ago
Converting an algorithm implementation from recursive to iterative: it got the concept broadly right, but was quite bad at making the logic actually match up, often refusing to fix mistakes or reverting fixes two edits later. Still a positive experience though, since it was fixable issues and reduced the amount of tedious copies I had to type
- cadamsdotcom 1y ago
  Did you have it write tests and give it the ability to iterate & validate its implementation without you in the loop? Anything less is setting it up for failure...
  - manwe150 1y ago
    Yes, but it got 99% of those then got stuck on why the others made no sense to it
    - cadamsdotcom 1y ago
      It’s important to understand the tests it’s written yourself. If you’d like some help I’d be glad to, just drop me an email. My email’s in my profile.
prmph 1y ago
Example (expanding on [1]): I want to design a strongly typed fluent API interface to some role/permissions based authorization engine functionality. Even knowing how to shape the fluent interface so that is powerful but intuitive, as strongly typed as possible but also and maintainable, is a deep art. One reason I know LLM can't come close to my design is this: I've written something that works (that a typical senior engineer might write), but this not enough. I have evaluated it critically (drawing on my experience with long lived software), rewritten it again to better meet the targets above, and repeated this process several times. I don't know what would make an LLM go: now that kind of works, but is this the most intuitive, well typed, and maintainable design that there could be? 1. https://news.ycombinator.com/item?id=45728183 https://news.ycombinator.com/item?id=45728183
- simonw 1y ago
  Funny you should use role/permissions as an example here, I spent the weekend using Claude Code to rewrite my own permissions engine to a new design that uses SQL queries to solve the problem "list all of the resources that this actor can perform this action on". My previous design required looping through all known resources asking "can actor X action Y on this?". The new design gets to generate a very complex by thoroughly tested SQL query instead. Applying that new design and updating the hundred of related tests would have taken me weeks. I got it done in two days. Here's a diff that captures most of the work: https://github.com/simonw/datasette/compare/e951f7e81f038e43d34bacf9890683ee446bd327...b3721eaf507c6d8919b6e72779c1263f297d8c88 https://github.com/simonw/datasette/compare/e951f7e81f038e43...
- YZF 1y ago
  What % of the total amount of software (lessay lines of code or time invested) in the world is like that?
- realusername 1y ago
  Working on permissions for a large saas app, I can also confirm that the best LLM of the market have maybe a 10% success rate writing code in this area.
NicoJuicy 1y ago
There are severe edge cases. Here are some of the last days. Eg. Just updating bootstrap to angular bootstrap. It didn't transfer how I placed the dropdowns ( basically using dropdown-end). So everything was out of view in desktop and mobile. It forgot the transloco I used everywhere and just used default English ( happens a lot). Suggested code that fixed 1 bug ( expression property recursion), but now linq to SQL was broken. Upgrade to angular 17 in a asp.net core app. I knew it used vite now. But it also required a browser folder to deploy. 20 changes down the road, I noticed something on my ui wasn't updated in dev ( fast commits for my side project, I don't build locally), it didn't deploy anything related to angular no more... I had 2 files named ApplicationDbContext and it took the one from wrong monolith module. It adds files in the wrong directory sometimes. Eg. Some modules were made with feature folders. It sometimes forgets to update my ocelot gateway or updates the compressed version. ... Note: I documented my architecture in eg. cline. But I use multiple agents to experiment with. Tldr: it's an expert beginner programmer.
- simonw 1y ago
  Do you have any automated tests for that project? I'm bringing to suspect a lot of my great experiences with coding agents come from the fact that they can run tests to confirm they haven't broken anything.
  - NicoJuicy 1y ago
    Too little. Things moved as fast as possible to migrate from .net framework to .net core 8, angular 8 to 18 and bootstrap 4.5 to 5.x
  - solumunus 1y ago
    The test loop is integral. It’s kind of annoying hearing all this skepticism from people putting in the least effort into optimally using the tool. There is a learning curve. Every month I’ve gotten better results than the last because I’m constantly context building and refining, understanding how, what and when to prompt. It’s like hearing someone say database suck but they haven’t bothered to learn about or use indexes or foreign keys.
herbst 1y ago
Claude added a self re-calling timeout to my Typescript game loop to track time. Manually by adding 1000ms every time it's called. I removed it and it later just added it again. It's this small weird things where it can mess up a lot of code.

manwe150 1y ago

Converting an algorithm implementation from recursive to iterative: it got the concept broadly right, but was quite bad at making the logic actually match up, often refusing to fix mistakes or reverting fixes two edits later. Still a positive experience though, since it was fixable issues and reduced the amount of tedious copies I had to type

cadamsdotcom 1y ago
Did you have it write tests and give it the ability to iterate & validate its implementation without you in the loop? Anything less is setting it up for failure...
- manwe150 1y ago
  Yes, but it got 99% of those then got stuck on why the others made no sense to it
  - cadamsdotcom 1y ago
    It’s important to understand the tests it’s written yourself. If you’d like some help I’d be glad to, just drop me an email. My email’s in my profile.

prmph 1y ago

Example (expanding on [1]): I want to design a strongly typed fluent API interface to some role/permissions based authorization engine functionality. Even knowing how to shape the fluent interface so that is powerful but intuitive, as strongly typed as possible but also and maintainable, is a deep art. One reason I know LLM can't come close to my design is this: I've written something that works (that a typical senior engineer might write), but this not enough. I have evaluated it critically (drawing on my experience with long lived software), rewritten it again to better meet the targets above, and repeated this process several times. I don't know what would make an LLM go: now that kind of works, but is this the most intuitive, well typed, and maintainable design that there could be? 1. https://news.ycombinator.com/item?id=45728183 https://news.ycombinator.com/item?id=45728183

simonw 1y ago
Funny you should use role/permissions as an example here, I spent the weekend using Claude Code to rewrite my own permissions engine to a new design that uses SQL queries to solve the problem "list all of the resources that this actor can perform this action on". My previous design required looping through all known resources asking "can actor X action Y on this?". The new design gets to generate a very complex by thoroughly tested SQL query instead. Applying that new design and updating the hundred of related tests would have taken me weeks. I got it done in two days. Here's a diff that captures most of the work: https://github.com/simonw/datasette/compare/e951f7e81f038e43d34bacf9890683ee446bd327...b3721eaf507c6d8919b6e72779c1263f297d8c88 https://github.com/simonw/datasette/compare/e951f7e81f038e43...
YZF 1y ago
What % of the total amount of software (lessay lines of code or time invested) in the world is like that?
realusername 1y ago
Working on permissions for a large saas app, I can also confirm that the best LLM of the market have maybe a 10% success rate writing code in this area.

NicoJuicy 1y ago

There are severe edge cases. Here are some of the last days. Eg. Just updating bootstrap to angular bootstrap. It didn't transfer how I placed the dropdowns ( basically using dropdown-end). So everything was out of view in desktop and mobile. It forgot the transloco I used everywhere and just used default English ( happens a lot). Suggested code that fixed 1 bug ( expression property recursion), but now linq to SQL was broken. Upgrade to angular 17 in a asp.net core app. I knew it used vite now. But it also required a browser folder to deploy. 20 changes down the road, I noticed something on my ui wasn't updated in dev ( fast commits for my side project, I don't build locally), it didn't deploy anything related to angular no more... I had 2 files named ApplicationDbContext and it took the one from wrong monolith module. It adds files in the wrong directory sometimes. Eg. Some modules were made with feature folders. It sometimes forgets to update my ocelot gateway or updates the compressed version. ... Note: I documented my architecture in eg. cline. But I use multiple agents to experiment with. Tldr: it's an expert beginner programmer.

simonw 1y ago
Do you have any automated tests for that project? I'm bringing to suspect a lot of my great experiences with coding agents come from the fact that they can run tests to confirm they haven't broken anything.
- NicoJuicy 1y ago
  Too little. Things moved as fast as possible to migrate from .net framework to .net core 8, angular 8 to 18 and bootstrap 4.5 to 5.x
- solumunus 1y ago
  The test loop is integral. It’s kind of annoying hearing all this skepticism from people putting in the least effort into optimally using the tool. There is a learning curve. Every month I’ve gotten better results than the last because I’m constantly context building and refining, understanding how, what and when to prompt. It’s like hearing someone say database suck but they haven’t bothered to learn about or use indexes or foreign keys.

herbst 1y ago

Claude added a self re-calling timeout to my Typescript game loop to track time. Manually by adding 1000ms every time it's called. I removed it and it later just added it again. It's this small weird things where it can mess up a lot of code.