3 ms·
Thanks for the idea, the cost wouldn't be a lot smaller because the question summary doesn't consume so many tokens compared to the context, however the latency
by tudorg 4y ago
Thanks for the idea, the cost wouldn't be a lot smaller because the question summary doesn't consume so many tokens compared to the context, however the latency would be improved significantly if we avoid one round-trip.