jezzarax

3 ms·

1. 1.▲
   by jezzarax 2y ago

   I checked some logs from my past experiments, the decoding went for about 400 tps over a ~3k token query, so about 7 seconds to process it, and then the generation speed was about 28 tokens.
2. 2.▲
   by jezzarax 2y ago

   llama.cpp + llama-3-8b in Q8 run great on a single T4 machine. Cannot remember the TPS I got there, but it was much above 6 mentioned in the article.
3. 3.▲
   by jezzarax 4y ago

   Munich airport T2 has water fountains for at least the last 2 years
4. 4.▲
   by jezzarax 4y ago

   In theory it’s easier/possible with some types of models, harder/impossible with others, but only if the model and the data processing around it is disclosed. The bigger issue here is that some seemingly unrelated factors and thei