4 ms·
Interesting, I got very different results depending on how I ran the model, will definitely give this a try! edit: Actually could you share how long it took to
by veryrealsid 2y ago
Interesting, I got very different results depending on how I ran the model, will definitely give this a try!
edit: Actually could you share how long it took to make a query? One of our issues is we need it to respond in a fast time frame
- jezzarax 2y agoI checked some logs from my past experiments, the decoding went for about 400 tps over a ~3k token query, so about 7 seconds to process it, and then the generation speed was about 28 tokens.