2 ms·
So, how are you handling read/write caching? I mean, if I keep routing the next prompt based on the task weights? How about if I'm sending every 5th query to op
by pradeep1177 3mo ago
So, how are you handling read/write caching? I mean, if I keep routing the next prompt based on the task weights? How about if I'm sending every 5th query to opus, which do expensive write cache?
- adchurch 3mo agoWe consider the cost of missing the cache when making each routing decision after the initial one. Discussed in a bit more depth here: https://news.ycombinator.com/item?id=48689448 https://news.ycombinator.com/item?id=48689448