2 ms·

I just learned something about LLMs: start a new chat as often as possible. I knew that this was a best practice, but I didn't know how quickly the situation be

I just learned something about LLMs: start a new chat as often as possible. I knew that this was a best practice, but I didn't know how quickly the situation becomes dire. > We evaluate 12 popular LLMs that claim to support contexts of at least 128K tokens. While they perform well in short contexts (<1K), performance degrades significantly as context length increases. At 32K, for instance, 10 models drop below 50% of their strong short-length baselines. Even GPT-4o, one of the top-performing exceptions, experiences a reduction from an almost-perfect baseline of 99.3% to 69.7%. [0] I copied all the code that I could from TFA and pasted it into OpenAI's tokenizer. It counted ~15k tokens. Many other tokens were generated in TFA's chat, some of which are not visible to the user. I think it's fair to assume that the entire chat was at least 25k tokens, right? Therefore, I believe that by the end of that chat, 4o's performance was significantly degraded. I think a major skill to develop for LLM supported coding is to compress a chat after just few thousand tokens into something like a step1.md file. Then start a new chat with "read step1.md" as the first prompt, and so on.. Is my logic sound here? [0] https://arxiv.org/abs/2502.05167 https://arxiv.org/abs/2502.05167

ghuntley 2y ago
It is sound! The real context window is indeed smaller than the advertised marketing number. This gets interesting in verbose languages such as Java, where I've seen +16k tokens wasted on the classpath/import stanzas within a class. Yick. The current series of prompts I'm using is somewhat 'manually manage/allocate memory' of the LLM context window. It starts with a PROMPT.md with this content. ``` create or update implementation_status.md with the implementation status study /specs and implement what has not been implemented yet in /src create property-based tests inline in the source file that is being implemented run "make" to verify the implementation after each change ``` If it goes off the rails (and it does) - restart the chat and use this. ``` @prompt continue ``` None of this works without applying /stdlib to control the technical outcomes/patterns to steer the LLM. Otherwise you just get slop.
- consumer451 2y ago
  > None of this works without applying /stdlib to control the technical outcomes/patterns to steer the LLM. Otherwise you just get slop. The LLM coding tool I have the most experience with is Windsurf IDE + Sonnet 3.5. In Windsurf you can define both global and project rules in .md files. I have found that managing those files very closely is key to success. They just tried to automate that with auto-generated "Memories" but they are generally slop, and I delete them. Managing the project rule file will save you so much pain. That is where I define the frameworks and APIs to use.
  - ghuntley 2y ago
    <3 you get it. It's a new skillset to be learned and when it clicks, you get incredible outcomes!
    - consumer451 2y ago
      Getting Supabase MCP running recently was another "oh shit!" moment for me. Prompting to check the schema is great, but prompting against live data in the DB is amazing! For example: > When logged in as user me@domain.com, I can't see the Edit button on src/pages/projects/show.tsx. Please use MCP to check if I have the correct permissions in organization_roles. If I do, look at the UI code to see if we are checking it correctly." Wowzers!

ghuntley 2y ago

It is sound! The real context window is indeed smaller than the advertised marketing number. This gets interesting in verbose languages such as Java, where I've seen +16k tokens wasted on the classpath/import stanzas within a class. Yick. The current series of prompts I'm using is somewhat 'manually manage/allocate memory' of the LLM context window. It starts with a PROMPT.md with this content. ``` create or update implementation_status.md with the implementation status study /specs and implement what has not been implemented yet in /src create property-based tests inline in the source file that is being implemented run "make" to verify the implementation after each change ``` If it goes off the rails (and it does) - restart the chat and use this. ``` @prompt continue ``` None of this works without applying /stdlib to control the technical outcomes/patterns to steer the LLM. Otherwise you just get slop.

consumer451 2y ago
> None of this works without applying /stdlib to control the technical outcomes/patterns to steer the LLM. Otherwise you just get slop. The LLM coding tool I have the most experience with is Windsurf IDE + Sonnet 3.5. In Windsurf you can define both global and project rules in .md files. I have found that managing those files very closely is key to success. They just tried to automate that with auto-generated "Memories" but they are generally slop, and I delete them. Managing the project rule file will save you so much pain. That is where I define the frameworks and APIs to use.
- ghuntley 2y ago
  <3 you get it. It's a new skillset to be learned and when it clicks, you get incredible outcomes!
  - consumer451 2y ago
    Getting Supabase MCP running recently was another "oh shit!" moment for me. Prompting to check the schema is great, but prompting against live data in the DB is amazing! For example: > When logged in as user me@domain.com, I can't see the Edit button on src/pages/projects/show.tsx. Please use MCP to check if I have the correct permissions in organization_roles. If I do, look at the UI code to see if we are checking it correctly." Wowzers!