8 ms·

That's...ridiculously fast. I still feel like the best uses of models we've seen to date is for brand new code and quick prototyping. I'm less convinced of the

That's...ridiculously fast. I still feel like the best uses of models we've seen to date is for brand new code and quick prototyping. I'm less convinced of the strength of their capabilities for improving on large preexisting content over which someone has repeatedly iterated. Part of that is because, by definition, models cannot know what is not in a codebase and there is meaningful signal in that negative space. Encoding what isn't there seems like a hard problem, so even as models get smarter, they will continue to be handicapped by that lack of institutional knowledge, so to speak. Imagine giving a large codebase to an incredibly talented developer and asking them to zero-shot a particular problem in one go, with only moments to read it and no opportunity to ask questions. More often than not, a less talented developer who is very familiar with that codebase will be able to add more value with the same amount of effort when tackling that same problem.

ec109685 1y ago
If you make models fast enough, you can onboard that expert developer instantly and let them reason their way to a solution, especially when giving access to a RAG to. Over time, I models will add more memory and institutional knowledge capture rather than starting from a blank slate each time.
- airstrike 1y ago
  I thought of that as I wrote my comment, but I think the infrastructure and glue to make that possible in a consistent, fast and scalable way is still a few years out.
  - lucasacosta_ 1y ago
    Definitely. For now the "frontier-level" papers (working with repository-level coding maintenance) need to necessarily depend on previously (and statically) generated Code Knowledge Graphs or Snippet-Retrieval systems, which makes the scalable and fast aspects complicated, as any change in the code would represent a change in the graph, hence requiring a rebuild. But given the context limit, you need to rely on Graph queries to give relevant parts and then at the end of the day it just reads snippets instead of the full code, which makes the consistent an issue, as it can't learn from the entirety of the code. Papers I'm referring to (just some as example, as there're more): - CodexGraph [https://arxiv.org/abs/2408.03910 https://arxiv.org/abs/2408.03910] - Graph - Agentless [https://arxiv.org/abs/2407.01489 https://arxiv.org/abs/2407.01489] - Snippet-Retrieval
    - airstrike 1y ago
      Thanks for these links. I really appreciate it.
8n4vidtmkvmk 1y ago
That's not been my experience so far. LLMs are good at mimicking existing good, it doesn't usually bring in new things when not asked. Sometimes I have to go out of my way to point to other bits of code in the project to copy from because it hasn't ingested enough of the codebase. That said, a negative prompt like we have in stable diffusion would still be very cool.
- Incipient 1y ago
  I'm in the camp of 'no good for existing'. I try to get ~1000 line files refactored to use different libraries, design paradigms, etc and it usually outputs garbage - pulling db logic into the UI, grabbing unrelated api/function calls, to entirely just corrupting the output. I'm sure there is a way to correctly use this tool, so I'm feeling like I'm "just holding it wrong".
  - fragmede 1y ago
    Which LLM are you using? what LLM tool are you using? What's your tech stack that you're generating code for? Without sharing anything you can't, what prompts are you using?
    - Incipient 1y ago
      Was more of a general comment - I'm surprised there is significant variation between any of the frontier models? However, vscode with various python frameworks/libraries; dash, fastapi, pandas, etc. Typically passing the 4-5 relevant files in as context. Developing via docker so I haven't found a nice way for agents to work.
      - danielbln 1y ago
        I would suggest using an agentic system like Cline, so that the LLM can wander through the codebase by itself and do research and build a "mental model" and then set up an implementation plan. The you iterate in that and hand it off for implementation. This flow works significantly better than what you're describing.
manmal 1y ago
They could read the whole git history and have all issue tracker tickets in the context, and maybe even recordings from meetings. It remains to be seen though if such large context will yield usable results.
- ttoinou 1y ago
  Do we already have tools to do thar automagically ?
  - manmal 1y ago
    Yes there are MCPs for git and Jira. I‘m not sure about the utility with the current context sizes.
- eMPee584 1y ago
  This. Git ( / tig!) blame and log -p --stat -S SEARCHSTR are extremely powerful for understanding the what why and when about code..
- Cthulhu_ 1y ago
  I find most meetings I'm in nowadays are mostly noise; there's no clear "signal" that "this is the outcome", which I think is what an AI should be able to filter out. Of course, it'd be even better if people communicated more clearly and succinctly.
  - manmal 1y ago
    Maybe time to find an employer with a better culture? I rarely have meetings that I would be comfortable skipping.
- internet_points 1y ago
  That also leads to more noise and opportunities to get lost in the woods.
ManuelKiessling 1y ago
"...what is not in a codebase, and there is meaningful signal in that negative space." Man, I'm writing software for money for decades now, but this fundamental truth never occured to me, at least not consciously and with such clarity. So, thank you!
- airstrike 1y ago
  My pleasure ;-) I borrowed the term from art: https://www.michaelalfano.com/tag/negative-space/?id=400 https://www.michaelalfano.com/tag/negative-space/?id=400
  - shahar2k 1y ago
    I'm an artist who works on pre-production fast turnaround animations for films, and yeah that hits the nail on the head, knowing what NOT to do which elements not to focus on is a majority of the power that comes with experience. I'm fast because I know which corners can be cut best and how to illustrate what I need to
- woctordho 1y ago
  Then document it. Whenever you choose one algorithm/library/tech stack but not another, write your consideration in the documents.
  - ManuelKiessling 1y ago
    The funny thing is that I have at least a dozen comments in my current codebase where I explain in detail why certain things are not put in place or are not served via other-solution-that-might-seem-obvious.
  - deleted 1y ago
    [deleted]
- FieryTransition 1y ago
  There's a reason why less is called less, and not more.
Flemlo 1y ago
But plenty of companies already do this for a decade and more Having old shitty code base and not retaining the people who built it. I have done that too despite the creator sitting only 100km away. Code was shit as hell tons of c&p different logic in different endpoints for logging in. Finally it's worth it to have adrs and similar things.
deleted 1y ago
[deleted]
nopinsight 1y ago
...which is why top LLM providers' web apps like ChatGPT, Claude.ai, Gemini try to nudge you to connect with Google Drive, and where appropriate, GitHub Repos. They also allow the user/dev to provide feedback to revise the results. All the training and interaction data will help make them formidable.
westoncb 1y ago
The trick to this is you've got to talk to them and share this information in the same way. I can give an example. These days my main workflow is as follows: if I have some big feature/refactor/whatever I'm going to work on I'll just start talking to o3 about it essentially as if it was a coworker and (somewhat painstakingly) paste in relevant source files it needs for context. We'll have a high-level discussion about what it is we're trying to build and how it relates to the existing code until I get the sense o3 has a clear and nuanced understanding (these discussions tend to sharpen my own understanding as well). Then, I'll ask o3 to generate an implementation plan that describes what needs to happen across the codebase in order for whatever it is to be realized. I'll then take that and hand it off to Codex, which might spend 10min executing shell commands to read source, edit files, test, etc. and then I've got a PR ready, which sometimes takes a bit more manual editing, and other times is perfectly ready to merge. What you're saying is true RE them needing rich context, too—but this isn't a fundamental limitation, it's just an aspect of what it takes to work with them effectively. There's definitely a learning curve but once you've got it down it's not only very powerful but, for me anyway, a more enjoyable headspace to occupy than lots of lower level manual editing.
- jacob019 1y ago
  I find myself using a similar workflow with Aider. I'll use chat mode to plan, adjust context, enable edits, and let it go. I'll give it a broad objective and tell it to ask me questions until the requirements are clear, then a planning summary. Flipping the script is especially helpful when I'm unsure what I actually want.
- dimitri-vs 1y ago
  Interesting approach, I'm definitely going to steal your wording for "generate an implementation plan that...". I do something similar but entirely within Cursor: 1. create a `docs/feature_name_spec.md`, use voice-to-text to brain dump what I am trying to do 2. open up a the AI chat panel in "Ask" mode while referencing that spec file, ask (paste) a boilerplate snippet like: "1) Ask clarifying questions about intent, domain, restrictions, ambiguity or missing details 2) Briefly identify any missing documents, data, or background information that would help you complete the task thoroughly" 3. move that list of questions into the spec doc and answer them there, attach the files it asked for and just rerun the above request (optionally, switching to a different model, like gemini-2.5-pro -> o3, for different perspective) 4. ask it to make an execution plan and at that point i have a fully spec'd out feature and documented business logic, I either use the Edit mode on each step or Agent mode That's for more complex features touching many files or refactors, but I essentially do a simplified version of that within the same chat by editing my original chat prompt until I'm confident I explained myself well
mejutoco 1y ago
> Part of that is because, by definition, models cannot know what is not in a codebase and there is meaningful signal in that negative space I wonder if git history would be enough to cover this. It has alternatives tried and code that was removed at the very least.
Flemlo 1y ago
A LLM could easily use its own knowledge to create a list of things to check inside the code base and generate a fact sheet and use best practices and similar knowledge to extend on it. Just because one query might not be able to do so doesn't mean there are no ways around it
scotty79 1y ago
> they will continue to be handicapped by that lack of institutional knowledge, so to speak Until we give them access to all Jira tickets instead of just one so they know what's missing.
- campers 1y ago
  I've been thinking about adding in an agent to our Codex/Jules like platform which goes through the git history for the main files being changed, extracts the Jira ticket ID's, look through them for additional context, along with the analyzing the changes to other files in commits.
aposm 1y ago
A human working on an existing codebase does not have any special signal about what is _not_ in a codebase. Instead, a (good) human engineer can look at how a problem is handled and consider why it might have been done that way vs other options, then make an educated decision about whether that alternative would be an improvement. To me this seems like yet another piece of evidence that these models are not doing any "reasoning" or problem-solving.

ec109685 1y ago

If you make models fast enough, you can onboard that expert developer instantly and let them reason their way to a solution, especially when giving access to a RAG to. Over time, I models will add more memory and institutional knowledge capture rather than starting from a blank slate each time.

airstrike 1y ago
I thought of that as I wrote my comment, but I think the infrastructure and glue to make that possible in a consistent, fast and scalable way is still a few years out.
- lucasacosta_ 1y ago
  Definitely. For now the "frontier-level" papers (working with repository-level coding maintenance) need to necessarily depend on previously (and statically) generated Code Knowledge Graphs or Snippet-Retrieval systems, which makes the scalable and fast aspects complicated, as any change in the code would represent a change in the graph, hence requiring a rebuild. But given the context limit, you need to rely on Graph queries to give relevant parts and then at the end of the day it just reads snippets instead of the full code, which makes the consistent an issue, as it can't learn from the entirety of the code. Papers I'm referring to (just some as example, as there're more): - CodexGraph [https://arxiv.org/abs/2408.03910 https://arxiv.org/abs/2408.03910] - Graph - Agentless [https://arxiv.org/abs/2407.01489 https://arxiv.org/abs/2407.01489] - Snippet-Retrieval
  - airstrike 1y ago
    Thanks for these links. I really appreciate it.

8n4vidtmkvmk 1y ago

That's not been my experience so far. LLMs are good at mimicking existing good, it doesn't usually bring in new things when not asked. Sometimes I have to go out of my way to point to other bits of code in the project to copy from because it hasn't ingested enough of the codebase. That said, a negative prompt like we have in stable diffusion would still be very cool.

Incipient 1y ago
I'm in the camp of 'no good for existing'. I try to get ~1000 line files refactored to use different libraries, design paradigms, etc and it usually outputs garbage - pulling db logic into the UI, grabbing unrelated api/function calls, to entirely just corrupting the output. I'm sure there is a way to correctly use this tool, so I'm feeling like I'm "just holding it wrong".
- fragmede 1y ago
  Which LLM are you using? what LLM tool are you using? What's your tech stack that you're generating code for? Without sharing anything you can't, what prompts are you using?
  - Incipient 1y ago
    Was more of a general comment - I'm surprised there is significant variation between any of the frontier models? However, vscode with various python frameworks/libraries; dash, fastapi, pandas, etc. Typically passing the 4-5 relevant files in as context. Developing via docker so I haven't found a nice way for agents to work.
    - danielbln 1y ago
      I would suggest using an agentic system like Cline, so that the LLM can wander through the codebase by itself and do research and build a "mental model" and then set up an implementation plan. The you iterate in that and hand it off for implementation. This flow works significantly better than what you're describing.

manmal 1y ago

They could read the whole git history and have all issue tracker tickets in the context, and maybe even recordings from meetings. It remains to be seen though if such large context will yield usable results.

ttoinou 1y ago
Do we already have tools to do thar automagically ?
- manmal 1y ago
  Yes there are MCPs for git and Jira. I‘m not sure about the utility with the current context sizes.
eMPee584 1y ago
This. Git ( / tig!) blame and log -p --stat -S SEARCHSTR are extremely powerful for understanding the what why and when about code..
Cthulhu_ 1y ago
I find most meetings I'm in nowadays are mostly noise; there's no clear "signal" that "this is the outcome", which I think is what an AI should be able to filter out. Of course, it'd be even better if people communicated more clearly and succinctly.
- manmal 1y ago
  Maybe time to find an employer with a better culture? I rarely have meetings that I would be comfortable skipping.
internet_points 1y ago
That also leads to more noise and opportunities to get lost in the woods.

ManuelKiessling 1y ago

"...what is not in a codebase, and there is meaningful signal in that negative space." Man, I'm writing software for money for decades now, but this fundamental truth never occured to me, at least not consciously and with such clarity. So, thank you!

airstrike 1y ago
My pleasure ;-) I borrowed the term from art: https://www.michaelalfano.com/tag/negative-space/?id=400 https://www.michaelalfano.com/tag/negative-space/?id=400
- shahar2k 1y ago
  I'm an artist who works on pre-production fast turnaround animations for films, and yeah that hits the nail on the head, knowing what NOT to do which elements not to focus on is a majority of the power that comes with experience. I'm fast because I know which corners can be cut best and how to illustrate what I need to
woctordho 1y ago
Then document it. Whenever you choose one algorithm/library/tech stack but not another, write your consideration in the documents.
- ManuelKiessling 1y ago
  The funny thing is that I have at least a dozen comments in my current codebase where I explain in detail why certain things are not put in place or are not served via other-solution-that-might-seem-obvious.
- deleted 1y ago
  [deleted]
FieryTransition 1y ago
There's a reason why less is called less, and not more.

Flemlo 1y ago

But plenty of companies already do this for a decade and more Having old shitty code base and not retaining the people who built it. I have done that too despite the creator sitting only 100km away. Code was shit as hell tons of c&p different logic in different endpoints for logging in. Finally it's worth it to have adrs and similar things.

deleted 1y ago

[deleted]

nopinsight 1y ago

...which is why top LLM providers' web apps like ChatGPT, Claude.ai, Gemini try to nudge you to connect with Google Drive, and where appropriate, GitHub Repos. They also allow the user/dev to provide feedback to revise the results. All the training and interaction data will help make them formidable.

westoncb 1y ago

The trick to this is you've got to talk to them and share this information in the same way. I can give an example. These days my main workflow is as follows: if I have some big feature/refactor/whatever I'm going to work on I'll just start talking to o3 about it essentially as if it was a coworker and (somewhat painstakingly) paste in relevant source files it needs for context. We'll have a high-level discussion about what it is we're trying to build and how it relates to the existing code until I get the sense o3 has a clear and nuanced understanding (these discussions tend to sharpen my own understanding as well). Then, I'll ask o3 to generate an implementation plan that describes what needs to happen across the codebase in order for whatever it is to be realized. I'll then take that and hand it off to Codex, which might spend 10min executing shell commands to read source, edit files, test, etc. and then I've got a PR ready, which sometimes takes a bit more manual editing, and other times is perfectly ready to merge. What you're saying is true RE them needing rich context, too—but this isn't a fundamental limitation, it's just an aspect of what it takes to work with them effectively. There's definitely a learning curve but once you've got it down it's not only very powerful but, for me anyway, a more enjoyable headspace to occupy than lots of lower level manual editing.

jacob019 1y ago
I find myself using a similar workflow with Aider. I'll use chat mode to plan, adjust context, enable edits, and let it go. I'll give it a broad objective and tell it to ask me questions until the requirements are clear, then a planning summary. Flipping the script is especially helpful when I'm unsure what I actually want.
dimitri-vs 1y ago
Interesting approach, I'm definitely going to steal your wording for "generate an implementation plan that...". I do something similar but entirely within Cursor: 1. create a `docs/feature_name_spec.md`, use voice-to-text to brain dump what I am trying to do 2. open up a the AI chat panel in "Ask" mode while referencing that spec file, ask (paste) a boilerplate snippet like: "1) Ask clarifying questions about intent, domain, restrictions, ambiguity or missing details 2) Briefly identify any missing documents, data, or background information that would help you complete the task thoroughly" 3. move that list of questions into the spec doc and answer them there, attach the files it asked for and just rerun the above request (optionally, switching to a different model, like gemini-2.5-pro -> o3, for different perspective) 4. ask it to make an execution plan and at that point i have a fully spec'd out feature and documented business logic, I either use the Edit mode on each step or Agent mode That's for more complex features touching many files or refactors, but I essentially do a simplified version of that within the same chat by editing my original chat prompt until I'm confident I explained myself well

mejutoco 1y ago

> Part of that is because, by definition, models cannot know what is not in a codebase and there is meaningful signal in that negative space I wonder if git history would be enough to cover this. It has alternatives tried and code that was removed at the very least.

Flemlo 1y ago

A LLM could easily use its own knowledge to create a list of things to check inside the code base and generate a fact sheet and use best practices and similar knowledge to extend on it. Just because one query might not be able to do so doesn't mean there are no ways around it

scotty79 1y ago

> they will continue to be handicapped by that lack of institutional knowledge, so to speak Until we give them access to all Jira tickets instead of just one so they know what's missing.

campers 1y ago
I've been thinking about adding in an agent to our Codex/Jules like platform which goes through the git history for the main files being changed, extracts the Jira ticket ID's, look through them for additional context, along with the analyzing the changes to other files in commits.

aposm 1y ago

A human working on an existing codebase does not have any special signal about what is _not_ in a codebase. Instead, a (good) human engineer can look at how a problem is handled and consider why it might have been done that way vs other options, then make an educated decision about whether that alternative would be an improvement. To me this seems like yet another piece of evidence that these models are not doing any "reasoning" or problem-solving.