6 ms·

> I really desperately need LLMs to maintain extremely effective context The context is in the repo. An LLM will never have the context you need to solve all

> I really desperately need LLMs to maintain extremely effective context The context is in the repo. An LLM will never have the context you need to solve all problems. Large enough repos don't fit on a single machine. There's a tradeoff just like in humans where getting a specific task done requires removing distractions. A context window that contains everything makes focus harder. For a long time context windows were too small, and they probably still are. But they have to get better at understanding the repo by asking the right questions.

stuartjohnson12 1y ago
> An LLM will never have the context you need to solve all problems. How often do you need more than 10 million tokens to answer your query?
- ants_everywhere 1y ago
  I exhaust the 1 million context windows on multiple models multiple times per day. I haven't used the Llama 4 10 million context window so I don't know how it performs in practice compared to the major non-open-source offerings that have smaller context windows. But there is an induced demand effect where as the context window increases it opens up more possibilities, and those possibilities can get bottlenecked on requiring an even bigger context window size. For example, consider the idea of storing all Hollywood films on your computer. In the 1980s this was impossible. If you store them in DVD or Bluray quality you could probably do it in a few terabytes. If you store them in full quality you may be talking about petabytes. We recently struggled to get a full file into a context window. Now a lot of people feel a bit like "just take the whole repo, it's only a few MB".
  - brulard 1y ago
    I think you misunderstand how context in current LLMs works. To get the best results you have to be very careful to provide what is needed for immediate task progression, and postpone context thats needed later in the process. If you give all the context at once, you will likely get quite degraded output quality. Thats like if you want to give a junior developer his first task, you likely won't teach him every corner of your app. You would give him context he needs. It is similar with these models. Those that provided 1M or 2M of context (Gemini etc.) were getting less and less useful after cca 200k tokens in the context. Maybe models would get better in picking up relevant information from large context, but AFAIK it is not the case today.
    - remexre 1y ago
      That's a really anthropomorphizing description; a more mechanical one might be, The attention mechanism that transformers use to find information in the context is, in its simplest form, O(n^2); for each token position, the model considers whether relevant information has been produced at the position of every other token. To preserve performance when really long contexts are used, current-generation LLMs use various ways to consider fewer positions in the context; for example, they might only consider the 4096 "most likely" places to matter (de-emphasizing large numbers of "subtle hints" that something isn't correct), or they might have some way of combining multiple tokens worth of information into a single value (losing some fine detail).
    - ants_everywhere 1y ago
      > I think you misunderstand how context in current LLMs works. Thanks but I don't and I'm not sure why you're jumping to this conclusion. EDIT: Oh I think you're talking about the last bit of the comment! If you read the one before I say that feeding it the entire repo isn't a great idea. But great idea or not, people want to do it, and it illustrates that as context window increases it creates demand for even larger context windows.
      - brulard 1y ago
        I said that based on you saying you exhaust a million token context windows easily. I'm no expert on that, but I think the current state of LLMs works best if you are not approaching that 1M token limit, because large context (reportedly) deteriorates response quality quickly. I think state of the art usage is managing context in tens or low hundreds thousands tokens at most and taking advantage of splitting tasks across subtasks in time, or splitting context across multiple "expert" agents (see sub-agents in claude code).
    - jimbokun 1y ago
      It seems like LLM need to become experts at managing their OWN context. Selectively gripping and searching the code to pull into context only those parts relevant to the task at hand.
- rocqua 1y ago
  It doesn't take me 10000000 tokens to have the context "this was the general idea of the code, these were unimportant implementation details, and this is where lifetimes were tricky." And that context is the valuable bit for quickly getting back up to speed on a codebase.
onion2k 1y ago
Large enough repos don't fit on a single machine. I don't believe any human can understand a problem if they need to fit the entire problem blem domain in their head, and the scope of a domain that doesn't fit on a computer. You have to break it down into a manageable amount of information to tackle it in chunks. If a person can do that, so can an LLM prompted to do that by a person.
- wraptile 1y ago
  Right, the LLM doesn't need to know all of the code under utils.parse_id to know that this call will parse the ID. The best LLM results I get is when I manually define the the relative code graph of my problem similar how I'd imagine it my head which seems to provide optimal context. So bigger isn't really better.
  - rocqua 1y ago
    I wonder why we can't have one LLM generate this understanding for another? Perhaps this is where teaming of LLMs gets its value. In managing high and low level context in different context windows.
    - mixedCase 1y ago
      This is a thing and doesn't require a separate model. You can set up custom prompts that will, based on another prompt describing the task to achieve, generate information about the codebase and a set of TODOs to accomplish the task, generating markdown files with a summarized version of the relevant knowledge and prompting you again to refine that summary if needed. You can then use these files to let the agent take over without going on a wild goose chase.
- ehnto 1y ago
  I disagree, I may not have the whole codebase in my head in one moment but I have had all of it in my head at some point, and it is still there, that is not true of an LLM. I use LLMs and am impressed by them, but they just do not approximate a human in this particular area. My ability to break a problem down does not start from listing the files out and reading a few. I have a high level understanding of the whole project at all times, and a deep understanding of the whole project stored, and I can recall that when required, this is not true of an LLM at any point. We know this is a limitation and it's why we have various tools attempting to approximate memory and augment training on the fly, but they are approximations and they are in my opinion, not even close to real human memory and depth of understanding for data it was not trained on. Even for mutations of scenarios it was trained on, which code is a great example of that. It is trained on billions of lines of code, yet still fails to understand my codebase intuitively. I have definitely not read billions of lines of code.
sdesol 1y ago
> But they have to get better at understanding the repo by asking the right questions. How I am tackling this problem is making it dead simple for users to create analyzers that are designed to enriched text data. You can read more about how it would be used in a search at https://github.com/gitsense/chat/blob/main/packages/chat/widgets/app/components/chat-builder/trees/help/documentation/understanding-your-personalized-ai-search-assistant/1.md https://github.com/gitsense/chat/blob/main/packages/chat/wid... The basic idea is, users would construct analyzers with the help of LLMs to extract the proper metadata that can be semantically searched. So when the user does an AI Assisted search with my tool, I would load all the analyzers (description and schema) into the system prompt and the LLM can determine which analyzers can be used to answer the question. A very simplistic analyzer would be to make it easy to identify backend and frontend code so you can just use the command `!ask find all frontend files` and the LLM will construct a deterministic search that knows to match for frontend files.
- mrits 1y ago
  How is that better than just writing a line in the md?
  - sdesol 1y ago
    I am not sure I follow what you are saying. What would the line be and how would it become deterministically searchable?
    - mrits 1y ago
      frontend path: /src/frontend/* backend path: /src/* I suppose the problem you have might be unique to nextJS ?
      - sdesol 1y ago
        The issue is frontend can be a loaded question, especially if you are dealing with legacy stuff, different frameworks, etc. You also can't tell what the frontend code does by looking at that single line. Now imagine as part of your analyzer, you have the following instructions for the llm: --- For all files in `src/frontend/` treat them as frontend code. For all files in 'src/' excluding `src/frontend` treat as backend. Create a metadata called `scope` which can be 'frontend', 'backend' or 'mix' where mix means the code can be used for both front and backend like utilities. Now for each file, create a `keywords` metadata that includes up to 10 unique keywords that describes the core functionality for the file. --- So with this you can say - `!ask find all frontend files` - `!ask find all mix use files` - `!ask find all frontend files that does [this]` and so forth. The whole point of analyzers is to make it easy for the LLM to map your natural language query to a deterministic search. If the code base is straightforward and follows a well known framework, asking for frontend or backend wouldn't even need an entry as you can just include in the instructions that I use framework X and the LLM would know what to consider.
mock-possum 1y ago
> The context is in the repo Agreed but that’s a bit different from “the context is the repo” It’s been my experience that usually just picking a couple files out to add to the context is enough - Claude seems capable of following imports and finding what it needs, in most cases. I’m sure it depends on the task, and the structure of the codebase.
manmal 1y ago
> The context is in the repo No it’s in the problem at hand. I need to load all related files, documentation, and style guides into the context. This works really well for smaller modules, but currently falls apart after a certain size.
alvis 1y ago
Everything in context hurts focus. It's like some people suffering from hyperthymesia. They are easily get distracted when the recall something
injidup 1y ago
All the more reason for good software engineering. Folders of files managing one concept. Files tightly focussed on sub problems of that concept. Keep your code so that you can solve problems in self contained context windows at the right level of abstraction
- Sharlin 1y ago
  I fear that LLM-optimal code structure is different from human-optimal code structure, and people are starting to optimize for the former rather than the latter.

stuartjohnson12 1y ago

> An LLM will never have the context you need to solve all problems. How often do you need more than 10 million tokens to answer your query?

ants_everywhere 1y ago
I exhaust the 1 million context windows on multiple models multiple times per day. I haven't used the Llama 4 10 million context window so I don't know how it performs in practice compared to the major non-open-source offerings that have smaller context windows. But there is an induced demand effect where as the context window increases it opens up more possibilities, and those possibilities can get bottlenecked on requiring an even bigger context window size. For example, consider the idea of storing all Hollywood films on your computer. In the 1980s this was impossible. If you store them in DVD or Bluray quality you could probably do it in a few terabytes. If you store them in full quality you may be talking about petabytes. We recently struggled to get a full file into a context window. Now a lot of people feel a bit like "just take the whole repo, it's only a few MB".
- brulard 1y ago
  I think you misunderstand how context in current LLMs works. To get the best results you have to be very careful to provide what is needed for immediate task progression, and postpone context thats needed later in the process. If you give all the context at once, you will likely get quite degraded output quality. Thats like if you want to give a junior developer his first task, you likely won't teach him every corner of your app. You would give him context he needs. It is similar with these models. Those that provided 1M or 2M of context (Gemini etc.) were getting less and less useful after cca 200k tokens in the context. Maybe models would get better in picking up relevant information from large context, but AFAIK it is not the case today.
  - remexre 1y ago
    That's a really anthropomorphizing description; a more mechanical one might be, The attention mechanism that transformers use to find information in the context is, in its simplest form, O(n^2); for each token position, the model considers whether relevant information has been produced at the position of every other token. To preserve performance when really long contexts are used, current-generation LLMs use various ways to consider fewer positions in the context; for example, they might only consider the 4096 "most likely" places to matter (de-emphasizing large numbers of "subtle hints" that something isn't correct), or they might have some way of combining multiple tokens worth of information into a single value (losing some fine detail).
  - ants_everywhere 1y ago
    > I think you misunderstand how context in current LLMs works. Thanks but I don't and I'm not sure why you're jumping to this conclusion. EDIT: Oh I think you're talking about the last bit of the comment! If you read the one before I say that feeding it the entire repo isn't a great idea. But great idea or not, people want to do it, and it illustrates that as context window increases it creates demand for even larger context windows.
    - brulard 1y ago
      I said that based on you saying you exhaust a million token context windows easily. I'm no expert on that, but I think the current state of LLMs works best if you are not approaching that 1M token limit, because large context (reportedly) deteriorates response quality quickly. I think state of the art usage is managing context in tens or low hundreds thousands tokens at most and taking advantage of splitting tasks across subtasks in time, or splitting context across multiple "expert" agents (see sub-agents in claude code).
  - jimbokun 1y ago
    It seems like LLM need to become experts at managing their OWN context. Selectively gripping and searching the code to pull into context only those parts relevant to the task at hand.
rocqua 1y ago
It doesn't take me 10000000 tokens to have the context "this was the general idea of the code, these were unimportant implementation details, and this is where lifetimes were tricky." And that context is the valuable bit for quickly getting back up to speed on a codebase.

onion2k 1y ago

Large enough repos don't fit on a single machine. I don't believe any human can understand a problem if they need to fit the entire problem blem domain in their head, and the scope of a domain that doesn't fit on a computer. You have to break it down into a manageable amount of information to tackle it in chunks. If a person can do that, so can an LLM prompted to do that by a person.

wraptile 1y ago
Right, the LLM doesn't need to know all of the code under utils.parse_id to know that this call will parse the ID. The best LLM results I get is when I manually define the the relative code graph of my problem similar how I'd imagine it my head which seems to provide optimal context. So bigger isn't really better.
- rocqua 1y ago
  I wonder why we can't have one LLM generate this understanding for another? Perhaps this is where teaming of LLMs gets its value. In managing high and low level context in different context windows.
  - mixedCase 1y ago
    This is a thing and doesn't require a separate model. You can set up custom prompts that will, based on another prompt describing the task to achieve, generate information about the codebase and a set of TODOs to accomplish the task, generating markdown files with a summarized version of the relevant knowledge and prompting you again to refine that summary if needed. You can then use these files to let the agent take over without going on a wild goose chase.
ehnto 1y ago
I disagree, I may not have the whole codebase in my head in one moment but I have had all of it in my head at some point, and it is still there, that is not true of an LLM. I use LLMs and am impressed by them, but they just do not approximate a human in this particular area. My ability to break a problem down does not start from listing the files out and reading a few. I have a high level understanding of the whole project at all times, and a deep understanding of the whole project stored, and I can recall that when required, this is not true of an LLM at any point. We know this is a limitation and it's why we have various tools attempting to approximate memory and augment training on the fly, but they are approximations and they are in my opinion, not even close to real human memory and depth of understanding for data it was not trained on. Even for mutations of scenarios it was trained on, which code is a great example of that. It is trained on billions of lines of code, yet still fails to understand my codebase intuitively. I have definitely not read billions of lines of code.

sdesol 1y ago

> But they have to get better at understanding the repo by asking the right questions. How I am tackling this problem is making it dead simple for users to create analyzers that are designed to enriched text data. You can read more about how it would be used in a search at https://github.com/gitsense/chat/blob/main/packages/chat/widgets/app/components/chat-builder/trees/help/documentation/understanding-your-personalized-ai-search-assistant/1.md https://github.com/gitsense/chat/blob/main/packages/chat/wid... The basic idea is, users would construct analyzers with the help of LLMs to extract the proper metadata that can be semantically searched. So when the user does an AI Assisted search with my tool, I would load all the analyzers (description and schema) into the system prompt and the LLM can determine which analyzers can be used to answer the question. A very simplistic analyzer would be to make it easy to identify backend and frontend code so you can just use the command `!ask find all frontend files` and the LLM will construct a deterministic search that knows to match for frontend files.

mrits 1y ago
How is that better than just writing a line in the md?
- sdesol 1y ago
  I am not sure I follow what you are saying. What would the line be and how would it become deterministically searchable?
  - mrits 1y ago
    frontend path: /src/frontend/* backend path: /src/* I suppose the problem you have might be unique to nextJS ?
    - sdesol 1y ago
      The issue is frontend can be a loaded question, especially if you are dealing with legacy stuff, different frameworks, etc. You also can't tell what the frontend code does by looking at that single line. Now imagine as part of your analyzer, you have the following instructions for the llm: --- For all files in `src/frontend/` treat them as frontend code. For all files in 'src/' excluding `src/frontend` treat as backend. Create a metadata called `scope` which can be 'frontend', 'backend' or 'mix' where mix means the code can be used for both front and backend like utilities. Now for each file, create a `keywords` metadata that includes up to 10 unique keywords that describes the core functionality for the file. --- So with this you can say - `!ask find all frontend files` - `!ask find all mix use files` - `!ask find all frontend files that does [this]` and so forth. The whole point of analyzers is to make it easy for the LLM to map your natural language query to a deterministic search. If the code base is straightforward and follows a well known framework, asking for frontend or backend wouldn't even need an entry as you can just include in the instructions that I use framework X and the LLM would know what to consider.

mock-possum 1y ago

> The context is in the repo Agreed but that’s a bit different from “the context is the repo” It’s been my experience that usually just picking a couple files out to add to the context is enough - Claude seems capable of following imports and finding what it needs, in most cases. I’m sure it depends on the task, and the structure of the codebase.

manmal 1y ago

> The context is in the repo No it’s in the problem at hand. I need to load all related files, documentation, and style guides into the context. This works really well for smaller modules, but currently falls apart after a certain size.

alvis 1y ago

Everything in context hurts focus. It's like some people suffering from hyperthymesia. They are easily get distracted when the recall something

injidup 1y ago

All the more reason for good software engineering. Folders of files managing one concept. Files tightly focussed on sub problems of that concept. Keep your code so that you can solve problems in self contained context windows at the right level of abstraction

Sharlin 1y ago
I fear that LLM-optimal code structure is different from human-optimal code structure, and people are starting to optimize for the former rather than the latter.