4 ms·

> I really desperately need LLMs to maintain extremely effective context I actually built this. I'm still not ready to say "use the tool yet" but you can learn

by sdesol 1y ago

> I really desperately need LLMs to maintain extremely effective context I actually built this. I'm still not ready to say "use the tool yet" but you can learn more about it at https://github.com/gitsense/chat https://github.com/gitsense/chat. The demo link is not up yet as I need to finalize an admin tool but you should be able to follow the npm instructions to play around with. The basic idea is, you should be able to load your entire repo or repos and use the context builder to help you refine it. Or you can can create custom analyzers that you can do 'AI Assisted' searches with like execute `!ask find all frontend code that does [this]` and the because the analyzer knows how to extract the correct metadata to support that query, you'll be able to easily build the context using it.

kvirani 1y ago
Wait that's not how Cursor etc work? (I made assumptions)
- trenchpilgrim 1y ago
  Dunno about Cursor but this is exactly how I use Zed to navigate groups of projects
- sdesol 1y ago
  I don't use Cursor so I can't say, but based on what I've read, they optimize for smaller context to reduce cost and probably for performance. The issue is, I think this is severely flawed as LLMs are insanely context sensitive and forgetting to include a reference file can lead to undesirable code. I am obviously biased, but I still think to get the best results, the context needs to be human curated to ensure everything the LLM needs will be present. LLMs are probabilistic, so the more relevant context, the greater the chances the final output is the most desired.
hirako2000 1y ago
Not clear how it gets around what is, ultimately, a context limit. I've been fiddling with some process too, would be good if you shared the how. The readme looks like yet another full fledged app.
- sdesol 1y ago
  Yes there is a context window limit, but I've found for most frontier models, you can generate very effective code if the context window is under 75,000 tokens provided the context is consistent. You have to think of everything from a probability point of view and the more logical the context, the greater the chances of better code. For example, if the frontend doesn't need to know the backend code (other than the interface) not including the backend code to solve a frontend one to solve a specific problem can reduce context size and improve the chances of expected output. You just need to ensure you include the necessary interface documenation. As for the full fledged app, I think you raised a good point and I should add a 'No lock in' section for why to use it. The app has a message tool that lets you pick and choose what messages to copy. Once you've copied the context (including any conversation messages that can help the LLM), you can use the context where ever you want. My strategy with the app is to be the first place you goto to start a conversation before you even generate code, so my focus is helping you construct contexts (the smaller the better) to feed into LLMs.
handfuloflight 1y ago
Doesn't Claude Code do all of this automatically?
- sdesol 1y ago
  I haven't looked at Claud Code, so I don't know if they have analyzers or not that understands how to extract any type of data other than specific coding data that it is trained on. Based on the runtime for some tasks, I would not be surprised if it is going through all the files and asking "is this relevant" My tool is mainly targeted at massive code bases and enterprise as I still believe the most efficient way to build accurate context is by domain experts. Right now, I would say 95% of my code is AI generated (98% human architectured) and I am spending about $2 a day on LLM costs and the code generation part usually never runs more than 30 seconds for most tasks.
  - handfuloflight 1y ago
    Well you should look at it, because it's not going through all files. I looked at your product and the workflow is essentially asking me to do manually what Claude Code does auto. Granted, manually selecting the context will probably lead to lower costs in any case because Claude Code invokes tool calls like grep to do its search, so I do see merit in your product in that respect.
    - sdesol 1y ago
      Looking at the code, it does have some sort of automatic discovery. I also don't know how scalable Claude Code is. I've spent over a decade thinking about code search, so I know what the limitations are for enterprise code. One of the neat tricks that I've developed is, I would load all my backend code for my search component and then I would ask the LLM to trace a query and create a context bundle for only the files that are affected. Once the LLM has finished, I just need to do a few clicks to refine a 80,000 token size window down to about 20,000 tokens. I would not be surprised if this is one of the tricks that it does as it is highly effective. Also, yes my tool is manual, but I treat conversations as durable asset so in the future, you should be able to say, last week I did this, load the same files and LLM will know what files to bring into context.
msikora 1y ago
Why not build this as an MCP so that people can plug it into their favorite platform?
- sdesol 1y ago
  An MCP is definitely on the roadmap. My objective is to become the context engine for LLMs so having a MCP is required. However, there will be things from a UX perspective that you'll lose out on if you just use the MCP.

by sdesol 1y ago

kvirani 1y ago

Wait that's not how Cursor etc work? (I made assumptions)

trenchpilgrim 1y ago
Dunno about Cursor but this is exactly how I use Zed to navigate groups of projects
sdesol 1y ago
I don't use Cursor so I can't say, but based on what I've read, they optimize for smaller context to reduce cost and probably for performance. The issue is, I think this is severely flawed as LLMs are insanely context sensitive and forgetting to include a reference file can lead to undesirable code. I am obviously biased, but I still think to get the best results, the context needs to be human curated to ensure everything the LLM needs will be present. LLMs are probabilistic, so the more relevant context, the greater the chances the final output is the most desired.

hirako2000 1y ago

Not clear how it gets around what is, ultimately, a context limit. I've been fiddling with some process too, would be good if you shared the how. The readme looks like yet another full fledged app.

sdesol 1y ago
Yes there is a context window limit, but I've found for most frontier models, you can generate very effective code if the context window is under 75,000 tokens provided the context is consistent. You have to think of everything from a probability point of view and the more logical the context, the greater the chances of better code. For example, if the frontend doesn't need to know the backend code (other than the interface) not including the backend code to solve a frontend one to solve a specific problem can reduce context size and improve the chances of expected output. You just need to ensure you include the necessary interface documenation. As for the full fledged app, I think you raised a good point and I should add a 'No lock in' section for why to use it. The app has a message tool that lets you pick and choose what messages to copy. Once you've copied the context (including any conversation messages that can help the LLM), you can use the context where ever you want. My strategy with the app is to be the first place you goto to start a conversation before you even generate code, so my focus is helping you construct contexts (the smaller the better) to feed into LLMs.

handfuloflight 1y ago

Doesn't Claude Code do all of this automatically?

sdesol 1y ago
I haven't looked at Claud Code, so I don't know if they have analyzers or not that understands how to extract any type of data other than specific coding data that it is trained on. Based on the runtime for some tasks, I would not be surprised if it is going through all the files and asking "is this relevant" My tool is mainly targeted at massive code bases and enterprise as I still believe the most efficient way to build accurate context is by domain experts. Right now, I would say 95% of my code is AI generated (98% human architectured) and I am spending about $2 a day on LLM costs and the code generation part usually never runs more than 30 seconds for most tasks.
- handfuloflight 1y ago
  Well you should look at it, because it's not going through all files. I looked at your product and the workflow is essentially asking me to do manually what Claude Code does auto. Granted, manually selecting the context will probably lead to lower costs in any case because Claude Code invokes tool calls like grep to do its search, so I do see merit in your product in that respect.
  - sdesol 1y ago
    Looking at the code, it does have some sort of automatic discovery. I also don't know how scalable Claude Code is. I've spent over a decade thinking about code search, so I know what the limitations are for enterprise code. One of the neat tricks that I've developed is, I would load all my backend code for my search component and then I would ask the LLM to trace a query and create a context bundle for only the files that are affected. Once the LLM has finished, I just need to do a few clicks to refine a 80,000 token size window down to about 20,000 tokens. I would not be surprised if this is one of the tricks that it does as it is highly effective. Also, yes my tool is manual, but I treat conversations as durable asset so in the future, you should be able to say, last week I did this, load the same files and LLM will know what files to bring into context.

msikora 1y ago

Why not build this as an MCP so that people can plug it into their favorite platform?

sdesol 1y ago
An MCP is definitely on the roadmap. My objective is to become the context engine for LLMs so having a MCP is required. However, there will be things from a UX perspective that you'll lose out on if you just use the MCP.