Interesting. I definitely saw models act strangely when I would swap between models in my harness between rounds. I have a claw where I allowed for each round t

Interesting. I definitely saw models act strangely when I would swap between models in my harness between rounds. I have a claw where I allowed for each round the model to be probabilistically selected and the results were somewhat worse than when I picked a single model and stuck to it. I blackboxed the whole thing but I should have looked through and seen what the reasoning looked like. In the end the experiment ended because it doesn’t benefit as much from caching and on-prem inference latency and effective throughput depends a lot on that. Very cool idea, man. Thanks for sharing.