4 ms·
Is it because of caching? If the context changes arbitrarily every turn then you would have to throw away the cache.
by esperent 8mo ago
Is it because of caching? If the context changes arbitrarily every turn then you would have to throw away the cache.
- FuckButtons 8mo agoSo use a block based cache and tune the block size to maximize the hit rate? This isn’t rocket science.
- wonnage 8mo agoThis seems misguided, you have to cache a prefix due to attention.