3 ms·

> annoying enough having to switch from one LLM ecosystem to another all the time due to vague usage restrictions I use KiloCode and what I find amazing is tha

> annoying enough having to switch from one LLM ecosystem to another all the time due to vague usage restrictions I use KiloCode and what I find amazing is that it'll be working on a problem and then a message will come up about needing to topup the money in my account to continue (or switch to a free model), so I switch to a free model (currently their Code Supernova 1million context) and it doesn't miss a beat and continues working on the problem. I don't know how they do this. It went from using a Claude Sonnet model to this Code Supernova model without missing a beat. Not sure if this is a Kilocode thing or if others do this as well. How does that even work? And this wasn't a trivial problem, it was adding a microcode debugger to a microcoded state machine system (coding in C++).

qsort 1y ago
Models are stateless, why would that not work?
- UncleOxidant 1y ago
  OK I understand what those words mean, but how exactly does that work? How does the new model 'know' what's being worked on when the old model was in the middle of working on a task and then a new model is switched to? (and where the task might be modifying a C++ file)
  - riwsky 1y ago
    the underlying LLM service provider APIs require sending the entire history for every request anyway; the state is entirely in your local (or kilocode or whatever), not in some "session" on the API side. (There are some APIs that will optionally handle that state for you, like OpenAI's more recent stuff — but those are the exception, not the rule).
  - handfuloflight 1y ago
    Here's a hint. What goes inside the inference engine is an array. You control that array every time you call for inference.
  - simonw 1y ago
    Every time you send a prompt to a model you actually send the entire previous conversation along with it, in an array that looks like this: curl https://api.anthropic.com/v1/messages \ -H "content-type: application/json" \ -H "x-api-key: $(llm keys get anthropic)" \ -H "anthropic-version: 2023-06-01" \ -d '{ "model": "claude-haiku-4-5-20251001", "max_tokens": 1024, "messages": [ { "role": "user", "content": "What is the capital of France?" }, { "role": "assistant", "content": "The capital of France is Paris." }, { "role": "user", "content": "Germany?" }, { "role": "assistant", "content": "The capital of Germany is Berlin." }, { "role": "user", "content": "Belgium?" } ] }' You can see this yourself if you use their APIs.
    - behnamoh 1y ago
      that is true unless you use the Response API endpoint...
      - simonw 1y ago
        That's true, the signature feature of that API is that OpenAI can now manage your conversation state server-side for you. You still have the option to send the full conversation JSON every time if you want to. You can send "store": false to turn off the feature where it persists your conversation server-side for you.
  - flawn 1y ago
    Probably context, logs or some sort of state passed in as context by your editor/extension
  - basket_horse 1y ago
    Generally speaking, agents send the entire previous conversation to the model on every message. That’s why you have to do things like context compaction. So if you switch models mid way, you are still sending the entire previous chat history to the new model
  - nothrabannosir 1y ago
    In addition to sibling comments you can play with this yourself by sending raw api requests with fake history to gaslight the model into believing it said things which it didn’t. I use this sometimes to coerce it into specific behavior, feeling like maybe it will listen to itself more than to my prompt (though I never benchmarked it): - do <fake task> and be succinct - <fake curt reply> - I love how succinct that was. Perfect. Now please do <real prompt> The models don’t have state so they don’t know they never said it. You’re just asking “given this conversation , what is the most likely next token?”
  - sandos 1y ago
    Wow, not knowing that models have 0 working memory is.. wild.

qsort 1y ago

Models are stateless, why would that not work?

UncleOxidant 1y ago
OK I understand what those words mean, but how exactly does that work? How does the new model 'know' what's being worked on when the old model was in the middle of working on a task and then a new model is switched to? (and where the task might be modifying a C++ file)
- riwsky 1y ago
  the underlying LLM service provider APIs require sending the entire history for every request anyway; the state is entirely in your local (or kilocode or whatever), not in some "session" on the API side. (There are some APIs that will optionally handle that state for you, like OpenAI's more recent stuff — but those are the exception, not the rule).
- handfuloflight 1y ago
  Here's a hint. What goes inside the inference engine is an array. You control that array every time you call for inference.
- simonw 1y ago
  Every time you send a prompt to a model you actually send the entire previous conversation along with it, in an array that looks like this: curl https://api.anthropic.com/v1/messages \ -H "content-type: application/json" \ -H "x-api-key: $(llm keys get anthropic)" \ -H "anthropic-version: 2023-06-01" \ -d '{ "model": "claude-haiku-4-5-20251001", "max_tokens": 1024, "messages": [ { "role": "user", "content": "What is the capital of France?" }, { "role": "assistant", "content": "The capital of France is Paris." }, { "role": "user", "content": "Germany?" }, { "role": "assistant", "content": "The capital of Germany is Berlin." }, { "role": "user", "content": "Belgium?" } ] }' You can see this yourself if you use their APIs.
  - behnamoh 1y ago
    that is true unless you use the Response API endpoint...
    - simonw 1y ago
      That's true, the signature feature of that API is that OpenAI can now manage your conversation state server-side for you. You still have the option to send the full conversation JSON every time if you want to. You can send "store": false to turn off the feature where it persists your conversation server-side for you.
- flawn 1y ago
  Probably context, logs or some sort of state passed in as context by your editor/extension
- basket_horse 1y ago
  Generally speaking, agents send the entire previous conversation to the model on every message. That’s why you have to do things like context compaction. So if you switch models mid way, you are still sending the entire previous chat history to the new model
- nothrabannosir 1y ago
  In addition to sibling comments you can play with this yourself by sending raw api requests with fake history to gaslight the model into believing it said things which it didn’t. I use this sometimes to coerce it into specific behavior, feeling like maybe it will listen to itself more than to my prompt (though I never benchmarked it): - do <fake task> and be succinct - <fake curt reply> - I love how succinct that was. Perfect. Now please do <real prompt> The models don’t have state so they don’t know they never said it. You’re just asking “given this conversation , what is the most likely next token?”
- sandos 1y ago
  Wow, not knowing that models have 0 working memory is.. wild.