3 ms·

How is it great that it takes 1 minute for initial prompt processing?

• cheema33 1y ago
  That time is just for the very first prompt. It is basically the startup time for the model. Once it is loaded, it is much much faster in responding to your queries. Depending on your hardware of course.
• zackify 1y ago
  Haha great as in surprisingly good at some simple things that nothing has been able to do locally for me. The 1 minute first token sucks and has me dreaming for the day of 3-4x the bandwidth