3 ms·

Claude Projects which allow attaching a bunch of files to fill up the 200k context. I wrote up a script to dump a bunch of code and documentation files to markd

Claude Projects which allow attaching a bunch of files to fill up the 200k context. I wrote up a script to dump a bunch of code and documentation files to markdown as context and I add them to a bunch of Claude projects on a per topic basis. For example, I'm currently working on a Rust/Qt desktop app so I have a project with the whole Qt6 book attached to ask questions about Qt, a project with my SQL schema and ORM/Sqlite docs to ask questions about the app's data and generate models without dealing with hallucinations, a project with all my QML files and Rust QML element code, a project with a bunch of Rust crate docs, and so on and on. GPTs allow attaching files too but Claude Projects dump the entire contents of the files into the context rather than trying to do some hacky RAG that never works like I want it to.

funnygiraffe 2y ago
I was under the impression that with LLMs, in order to get high-quality answers, it's always best to keep context short. Is that not the case anymore? Does Claude under this usage paradigm not struggle with very long contexts in ways as for example described in the "lost in the middle" paper (https://arxiv.org/abs/2307.03172 https://arxiv.org/abs/2307.03172)?
- throwup238 2y ago
  I don't have the time to evaluate the effects of context length on my use cases so I have no idea. There might be some degradation when I attach the Qt book which is probably already in Claude's training data but when using it against my private code base, it's not like I have any other choice. The UX of drag and dropping a few monolithic markdown files to include entire chunks of a large project outweighs the downsides of including irrelevant context in my experience.
- inciampati 2y ago
  No, you need to provide as much information in context as possible. Otherwise you are sampling from the mode. "Write me an essay about cows" = garbage boring and probably 200 words. "here are twenty papers about cow evolution, write me an overview of findings" = yes
- azeirah 2y ago
  The conclusion you walked away with is the opposite of what usually works in practice. The more context you give the llm, the better. The key takeaway from that paper is to keep your instructions/questions/direction in the beginning or at the end of the context. Any information can go anywhere. Not to be too dismissive, it's a good paper, but we're one year further and in practice this issue seems to have been tackled by training on better data. This can differ a lot depending on what model you're using, but in the case of claude sonnet 3.5, more relevant context is generally better for anything except for speed. It does remain true that you need to keep your most important instructions at the beginning or at the end however.
- rvnx 2y ago
  At the beginning it was true, the longer the context, the more the LLM was lost, but now, the new models can retrieve information anywhere in the context c.f. https://pbs.twimg.com/media/GH2NJMxbYAAcRL3?format=jpg&name=medium https://pbs.twimg.com/media/GH2NJMxbYAAcRL3?format=jpg&name=...

funnygiraffe 2y ago

I was under the impression that with LLMs, in order to get high-quality answers, it's always best to keep context short. Is that not the case anymore? Does Claude under this usage paradigm not struggle with very long contexts in ways as for example described in the "lost in the middle" paper (https://arxiv.org/abs/2307.03172 https://arxiv.org/abs/2307.03172)?

throwup238 2y ago
I don't have the time to evaluate the effects of context length on my use cases so I have no idea. There might be some degradation when I attach the Qt book which is probably already in Claude's training data but when using it against my private code base, it's not like I have any other choice. The UX of drag and dropping a few monolithic markdown files to include entire chunks of a large project outweighs the downsides of including irrelevant context in my experience.
inciampati 2y ago
No, you need to provide as much information in context as possible. Otherwise you are sampling from the mode. "Write me an essay about cows" = garbage boring and probably 200 words. "here are twenty papers about cow evolution, write me an overview of findings" = yes
azeirah 2y ago
The conclusion you walked away with is the opposite of what usually works in practice. The more context you give the llm, the better. The key takeaway from that paper is to keep your instructions/questions/direction in the beginning or at the end of the context. Any information can go anywhere. Not to be too dismissive, it's a good paper, but we're one year further and in practice this issue seems to have been tackled by training on better data. This can differ a lot depending on what model you're using, but in the case of claude sonnet 3.5, more relevant context is generally better for anything except for speed. It does remain true that you need to keep your most important instructions at the beginning or at the end however.
rvnx 2y ago
At the beginning it was true, the longer the context, the more the LLM was lost, but now, the new models can retrieve information anywhere in the context c.f. https://pbs.twimg.com/media/GH2NJMxbYAAcRL3?format=jpg&name=medium https://pbs.twimg.com/media/GH2NJMxbYAAcRL3?format=jpg&name=...