7 ms·

> I focus on the high-level code, and let the model focus on the lower level code. Tbh the reason I don't use LLM assistants is because they suck at the "low l

> I focus on the high-level code, and let the model focus on the lower level code. Tbh the reason I don't use LLM assistants is because they suck at the "low level". They are okay at mid level and better at high level. I find it's actual coding very mediocre and fraught with errors. I've yet to see any model understand nuance or detail. This is especially apparent in image models. Sure, it can do hands but they still don't get 3D space nor temporal movements. It's great for scrolling through Twitter but the longer you look the more surreal they get. This even includes the new ByteDance model also on the front page. But with coding models they ignore context of the codebase and the results feel more like patchwork. They feel like what you'd be annoyed at with a junior dev for writing because not only do you have to go through 10 PRs to make it pass the test cases but the lack of context just builds a lot of tech debt. How they'll build unit tests that technically work but don't capture the actual issues and usually can be highly condensed while having greater coverage. It feels very gluey, like copy pasting from stack overflow when hyper focused on the immediate outcome instead of understanding the goal. It is too "solution" oriented, not understanding the underlying heuristics and is more frustrating than dealing with the human equivalent who says something "works" as evidenced by the output. This is like trying to say a math proof is correct by looking at just the last line. Ironically, I think in part this is why chat interface sucks too. A lot of our job is to do a lot of inference in figuring out what our managers are even asking us to make. And you can't even know the answer until you're part way in.

lucasmullens 2y ago
> But with coding models they ignore context of the codebase and the results feel more like patchwork. Have you tried Cursor? It has a great feature that grabs context from the codebase, I use it all the time.
- pc86 2y ago
  I can't get the prompt because I'm on my work computer but I have about a three-quarter-page instruction set in the settings of cursor, it asks clarifying questions a LOT now, and is pretty liberal with adding in commented pseudo-code for stuff it isn't sure about. You can still trip it up if you try, but it's a lot better than stock. This is with Sonnet 3.5 agent chats (composer I think it's called?) I actually cancelled by Anthropic subscription when I started using cursor because I only ever used Claude for code generation anyway so now I just do it within the IDE.
  - slig 2y ago
    I'm very interested in your prompt and could you be so kind to paste it somewhere and link in your comment, please?
  - justneedaname 2y ago
    Also interested to see this
- godelski 2y ago
  I have not. But I also can't get the general model to work well in even toy problems. Here's a simple example with GPT-4o: https://0x0.st/8K3z.png https://0x0.st/8K3z.png It probably isn't obvious in a quick read, but there are mistakes here. Maybe the most obvious is that how `replacements` is made we need to intelligently order. This could be fixed by sorting. But is this the right data structure? Not to mention that the algorithm itself is quite... odd To give a more complicated example I passed the same prompt from this famous code golf problem[0]. Here's the results, I'll save you the time, the output is wrong https://0x0.st/8K3M.txt https://0x0.st/8K3M.txt (note, I started command likes with "$" and added some notes for you) Just for the heck of it, here's the same thing but with o1-preview Initial problem: https://0x0.st/8K3t.txt https://0x0.st/8K3t.txt Codegolf one: https://0x0.st/8K3y.txt https://0x0.st/8K3y.txt As you can see, o1 is a bit better on the initial problem but still fails at the code golf one. It really isn't beating the baseline naive solution. It does 170 MiB/s compared to 160 MiB/s (baseline with -O3). This is something I'd hope it could do really well on given that this problem is rather famous and so many occurrences of it should show up. There's tons of variations out there and It is common to see parallel fizzbuzz in a class on parallelization as well as it can teach important concepts like keeping the output in the right order. But hey, at least o1 has the correct output... It's just that that's not all that matters. I stand by this: evaluating code based on output alone is akin to evaluating a mathematical proof based on the result. And I hope these examples make the point why that matters, why checking output is insufficient. [0] https://codegolf.stackexchange.com/questions/215216/high-throughput-fizz-buzz https://codegolf.stackexchange.com/questions/215216/high-thr... Edit: I want to add that there's also an important factor here. The LLM might get you a "result" faster, but you are much more likely to miss the learning process that comes with struggling. Because that makes you much faster (and more flexible) not just next time but in many situations where even a subset is similar. Which yeah, totally fine to glue shit together when you don't care and just need something, but there's a lot of missed value if you need to revisit any of that. I do have concerns that people will be plateaued at junior levels. I hope it doesn't cause seniors to revert to juniors, which I've seen happen without LLMs. If you stop working on these types of problems, you lose the skills. There's already an issue where we rush to get output and it has clear effects on the stagnation of devs. We have far more programmers than ever but I'm not confident we have a significant number more wizards (the percentage of wizards is decreasing). There's fewer people writing programs just for fun. But "for fun" is one of our greatest learning tools as humans. Play is a common trait you see in animals and it exists for a reason.
- troupo 2y ago
  > It has a great feature that grabs context from the codebase, I use it all the time. If only this feature worked consistently, or reliably even half of the time. It will casually forget or ignore any and all context and any and all files in your codebase at random times, and you never know what set of files and docs it's working with at any point in time
wiremine 2y ago
> Tbh the reason I don't use LLM assistants is because they suck at the "low level". They are okay at mid level and better at high level. I find it's actual coding very mediocre and fraught with errors. That's interesting. I found assistants like Copilot fairly good at low level code, assuming you direct it well.
- godelski 2y ago
  I have a response to a sibling comment showing where GPT 4o and o1-preview do not yield good results. > assuming you direct it well. But hey, I admit I might not be good at this. But honestly, I've found greater value in my time reading the docs than spending trying to prompt engineer my way through. And I've given a fair amount of time to trying to get good at prompting. I just can't get it to work. I do think that when I'm coding with an LLM it _feels_ faster, but when I've timed myself, it doesn't seem that way. It just seems to be less effort (I don't mind the effort, especially because the compounding rewards).
yarekt 2y ago
> A lot of our job is to do a lot of inference in figuring out what our managers are even asking us to make This is why I think LLMs can't really replace developers. 80% of my job is already trying to figure out what's actually needed, despite being given lots of text detail, maybe even spec, or prototype code. Building the wrong thing fast is about as useful as not building anything at all. (And before someone says "at least you now know what not to do"? For any problem there are infinite number of wrong solutions, but only a handful of ones that yield success, why waste time trying all the wrong ones?)
- TeMPOraL 2y ago
  > For any problem there are infinite number of wrong solutions, but only a handful of ones that yield success, why waste time trying all the wrong ones? Devil's advocate: because unless you're working in heavily dysfunctional organization, or are doing a live coding interview, you're not playing "guess the password" with your management. Most of the time, they have even less of a clue about how the right solution looks like! "Building the wrong thing" lets them diff something concrete against what they imagined and felt like it would be, forcing them to clarify their expectations and give you more accurate directions (which, being a diff against a concrete things, are less likely to be then misunderstood by you!). And, the faster you can build that wrong thing, the less money and time is burned to buy that extra clarity.
  - bcoates 2y ago
    It's just incredibly inefficient if there's any other alternative. Doing 4 sprints over 2 months to make a prototype in order to save 3 60 minute meetings over a week where you do a few requirements analysis/proposal review cycles.
    - TeMPOraL 2y ago
      Yeah, that would be stupid. I was thinking one order of magnitude less in terms of effort. If you can make a prototype in a day, it might deliver way more value than 3x 60 minute meetings. If you can make it in a week, where the proper implementation would take more than a month, that could still be a huge win. I see this not as opposed, but as part of requirements analysis/review - working in the abstract, with imagination and prose and diagrams, it's too easy to make invalid assumptions without anyone realizing it.

lucasmullens 2y ago

> But with coding models they ignore context of the codebase and the results feel more like patchwork. Have you tried Cursor? It has a great feature that grabs context from the codebase, I use it all the time.

pc86 2y ago
I can't get the prompt because I'm on my work computer but I have about a three-quarter-page instruction set in the settings of cursor, it asks clarifying questions a LOT now, and is pretty liberal with adding in commented pseudo-code for stuff it isn't sure about. You can still trip it up if you try, but it's a lot better than stock. This is with Sonnet 3.5 agent chats (composer I think it's called?) I actually cancelled by Anthropic subscription when I started using cursor because I only ever used Claude for code generation anyway so now I just do it within the IDE.
- slig 2y ago
  I'm very interested in your prompt and could you be so kind to paste it somewhere and link in your comment, please?
- justneedaname 2y ago
  Also interested to see this
godelski 2y ago
I have not. But I also can't get the general model to work well in even toy problems. Here's a simple example with GPT-4o: https://0x0.st/8K3z.png https://0x0.st/8K3z.png It probably isn't obvious in a quick read, but there are mistakes here. Maybe the most obvious is that how `replacements` is made we need to intelligently order. This could be fixed by sorting. But is this the right data structure? Not to mention that the algorithm itself is quite... odd To give a more complicated example I passed the same prompt from this famous code golf problem[0]. Here's the results, I'll save you the time, the output is wrong https://0x0.st/8K3M.txt https://0x0.st/8K3M.txt (note, I started command likes with "$" and added some notes for you) Just for the heck of it, here's the same thing but with o1-preview Initial problem: https://0x0.st/8K3t.txt https://0x0.st/8K3t.txt Codegolf one: https://0x0.st/8K3y.txt https://0x0.st/8K3y.txt As you can see, o1 is a bit better on the initial problem but still fails at the code golf one. It really isn't beating the baseline naive solution. It does 170 MiB/s compared to 160 MiB/s (baseline with -O3). This is something I'd hope it could do really well on given that this problem is rather famous and so many occurrences of it should show up. There's tons of variations out there and It is common to see parallel fizzbuzz in a class on parallelization as well as it can teach important concepts like keeping the output in the right order. But hey, at least o1 has the correct output... It's just that that's not all that matters. I stand by this: evaluating code based on output alone is akin to evaluating a mathematical proof based on the result. And I hope these examples make the point why that matters, why checking output is insufficient. [0] https://codegolf.stackexchange.com/questions/215216/high-throughput-fizz-buzz https://codegolf.stackexchange.com/questions/215216/high-thr... Edit: I want to add that there's also an important factor here. The LLM might get you a "result" faster, but you are much more likely to miss the learning process that comes with struggling. Because that makes you much faster (and more flexible) not just next time but in many situations where even a subset is similar. Which yeah, totally fine to glue shit together when you don't care and just need something, but there's a lot of missed value if you need to revisit any of that. I do have concerns that people will be plateaued at junior levels. I hope it doesn't cause seniors to revert to juniors, which I've seen happen without LLMs. If you stop working on these types of problems, you lose the skills. There's already an issue where we rush to get output and it has clear effects on the stagnation of devs. We have far more programmers than ever but I'm not confident we have a significant number more wizards (the percentage of wizards is decreasing). There's fewer people writing programs just for fun. But "for fun" is one of our greatest learning tools as humans. Play is a common trait you see in animals and it exists for a reason.
troupo 2y ago
> It has a great feature that grabs context from the codebase, I use it all the time. If only this feature worked consistently, or reliably even half of the time. It will casually forget or ignore any and all context and any and all files in your codebase at random times, and you never know what set of files and docs it's working with at any point in time

wiremine 2y ago

> Tbh the reason I don't use LLM assistants is because they suck at the "low level". They are okay at mid level and better at high level. I find it's actual coding very mediocre and fraught with errors. That's interesting. I found assistants like Copilot fairly good at low level code, assuming you direct it well.

godelski 2y ago
I have a response to a sibling comment showing where GPT 4o and o1-preview do not yield good results. > assuming you direct it well. But hey, I admit I might not be good at this. But honestly, I've found greater value in my time reading the docs than spending trying to prompt engineer my way through. And I've given a fair amount of time to trying to get good at prompting. I just can't get it to work. I do think that when I'm coding with an LLM it _feels_ faster, but when I've timed myself, it doesn't seem that way. It just seems to be less effort (I don't mind the effort, especially because the compounding rewards).

yarekt 2y ago

> A lot of our job is to do a lot of inference in figuring out what our managers are even asking us to make This is why I think LLMs can't really replace developers. 80% of my job is already trying to figure out what's actually needed, despite being given lots of text detail, maybe even spec, or prototype code. Building the wrong thing fast is about as useful as not building anything at all. (And before someone says "at least you now know what not to do"? For any problem there are infinite number of wrong solutions, but only a handful of ones that yield success, why waste time trying all the wrong ones?)

TeMPOraL 2y ago
> For any problem there are infinite number of wrong solutions, but only a handful of ones that yield success, why waste time trying all the wrong ones? Devil's advocate: because unless you're working in heavily dysfunctional organization, or are doing a live coding interview, you're not playing "guess the password" with your management. Most of the time, they have even less of a clue about how the right solution looks like! "Building the wrong thing" lets them diff something concrete against what they imagined and felt like it would be, forcing them to clarify their expectations and give you more accurate directions (which, being a diff against a concrete things, are less likely to be then misunderstood by you!). And, the faster you can build that wrong thing, the less money and time is burned to buy that extra clarity.
- bcoates 2y ago
  It's just incredibly inefficient if there's any other alternative. Doing 4 sprints over 2 months to make a prototype in order to save 3 60 minute meetings over a week where you do a few requirements analysis/proposal review cycles.
  - TeMPOraL 2y ago
    Yeah, that would be stupid. I was thinking one order of magnitude less in terms of effort. If you can make a prototype in a day, it might deliver way more value than 3x 60 minute meetings. If you can make it in a week, where the proper implementation would take more than a month, that could still be a huge win. I see this not as opposed, but as part of requirements analysis/review - working in the abstract, with imagination and prose and diagrams, it's too easy to make invalid assumptions without anyone realizing it.