4 ms·

If someone asked you about the usages of a particular element in a codebase, you would probably give a more accurate answer if you were able to use a code searc

If someone asked you about the usages of a particular element in a codebase, you would probably give a more accurate answer if you were able to use a code search tool rather than reading every source file from top to bottom. For that kind of tasks (and there are many of those!), I don't see why you would expect something fundamentally different in the case of LLMs.

skydhash 1y ago
But why not provide the search tool instead of being an imperfect interface between it and the person asking? The only reason for the latter is that you have more applied knowledge in the context and can use the tool better. For any other case, the answer should be “use this tool”.
- __loam 1y ago
  The uninformed would rather have a natural language interface rather than learn how to actually use the tools.
  - skydhash 1y ago
    The reason for the expert in this case (an uninformed that wants to solve a problem) is that the expert can use metaphors as a bridge for understanding. Just like in most companies, there's the business world (which is heterogeneous) and the software engineering world. A huge part of software engineer's time is spent translating concepts across the two. And the most difficult part of that is asking questions and knowing which question to ask as natural language is so ambiguous.
- gpm 1y ago
  Because the LLM is faster at typing the input, and faster at reading the output, than I am... the amount of input I have to give the LLM is less than what I have to give the search tool invocations, and the amount of output I have to read from the LLM is less than the amount of output from the search tool invocations. To be fair it's also more likely to mess up than I am, but for reading search results to get an idea of what the code base looks like the speed/accuracy tradeoff is often worth it. And if it was just a search tool this would be barely worth it, but the effects compound as you chain more tools together. For example: reading and running searches + reading and running compiler output is worth more than double just reading and running searches. It's definitely an art to figure out when it's better to use an LLM, and when it's just going to be an impediment, though. (Which isn't to agree that "context engineering" is anything other than "prompt engineering" rebranded, or has any staying power)
  - skydhash 1y ago
    So instead of building better tool, we're patching the last one with another tool that is not even reliable, just using it faster. That reminds me of the first chapter in "The Programmer Brain" by Felienne Hermans. There's an explanation there that confusion when reading code is caused by three things: - Lack of knowledge: When you don't have the faintest idea of the notation or symbol being used, aka the WHAT. - Lack of information: When you know the WHAT, but you can't figure out the WHY. - Lack of processing power: When you have an idea of the WHY, but can't grasp the HOW. We already have methods and tooling for all the above and they work fine without having to do shamanic rituals.
manishsharan 1y ago
I provided 'grep' as a tool to LLM (deepseek) and it does a better job of finding usages. This is especially true if the code is obfuscated JavaScript.
bostik 1y ago
In my previous job I repeatedly told people that "git grep is a superpower". Especially in a monorepo, but works well in any big repository, really. To this day I think the same. With the addition that knowing about "git log -S" grants you necromancy in addition to the regular superpowers. Ability to do rapid code search, and especially code history search, make you look like a wizard without the funny hat.

skydhash 1y ago

But why not provide the search tool instead of being an imperfect interface between it and the person asking? The only reason for the latter is that you have more applied knowledge in the context and can use the tool better. For any other case, the answer should be “use this tool”.

__loam 1y ago
The uninformed would rather have a natural language interface rather than learn how to actually use the tools.
- skydhash 1y ago
  The reason for the expert in this case (an uninformed that wants to solve a problem) is that the expert can use metaphors as a bridge for understanding. Just like in most companies, there's the business world (which is heterogeneous) and the software engineering world. A huge part of software engineer's time is spent translating concepts across the two. And the most difficult part of that is asking questions and knowing which question to ask as natural language is so ambiguous.
gpm 1y ago
Because the LLM is faster at typing the input, and faster at reading the output, than I am... the amount of input I have to give the LLM is less than what I have to give the search tool invocations, and the amount of output I have to read from the LLM is less than the amount of output from the search tool invocations. To be fair it's also more likely to mess up than I am, but for reading search results to get an idea of what the code base looks like the speed/accuracy tradeoff is often worth it. And if it was just a search tool this would be barely worth it, but the effects compound as you chain more tools together. For example: reading and running searches + reading and running compiler output is worth more than double just reading and running searches. It's definitely an art to figure out when it's better to use an LLM, and when it's just going to be an impediment, though. (Which isn't to agree that "context engineering" is anything other than "prompt engineering" rebranded, or has any staying power)
- skydhash 1y ago
  So instead of building better tool, we're patching the last one with another tool that is not even reliable, just using it faster. That reminds me of the first chapter in "The Programmer Brain" by Felienne Hermans. There's an explanation there that confusion when reading code is caused by three things: - Lack of knowledge: When you don't have the faintest idea of the notation or symbol being used, aka the WHAT. - Lack of information: When you know the WHAT, but you can't figure out the WHY. - Lack of processing power: When you have an idea of the WHY, but can't grasp the HOW. We already have methods and tooling for all the above and they work fine without having to do shamanic rituals.

manishsharan 1y ago

I provided 'grep' as a tool to LLM (deepseek) and it does a better job of finding usages. This is especially true if the code is obfuscated JavaScript.

bostik 1y ago

In my previous job I repeatedly told people that "git grep is a superpower". Especially in a monorepo, but works well in any big repository, really. To this day I think the same. With the addition that knowing about "git log -S" grants you necromancy in addition to the regular superpowers. Ability to do rapid code search, and especially code history search, make you look like a wizard without the funny hat.