4 ms·
This is a good rule of thumb. I would also include that in most cases, RAM use exponentially increases with context window size.
by kittikitti 8mo ago
This is a good rule of thumb. I would also include that in most cases, RAM use exponentially increases with context window size.
- namibj 8mo agoThere's zero exponential scaling involved. There is quadratic compute and reasonably log-linear storage, though.