8 ms·
Is there anything experimenting with that?
by pyinstallwoes 3y ago
Is there anything experimenting with that?
- Roark66 3y agoI would be very surprised if (not at all)OpenAI didn't. I've noticed it with both gpt 3.5 and Gpt4 that if you ask it a difficult question the latency increases a lot, especially at the beginning of the answer. I wouldn't ve surprised if they did exactly what was described.
- pyinstallwoes 3y agoRight I’ve noticed that too. It’s like it takes a pause and reevals. Definitely feels like a cache or memoization of token paths. Interesting.