4 ms·

I found 1053 t/s in cell G18 in the throughput tab of the full results sheet they linked to: https://docs.google.com/spreadsheets/u/0/d/1hHd9riWh-hQJnNOzEI_ZZN1

• qeternity 2y ago
  Hmm the post now links to this doc where I got my figures from: https://docs.google.com/spreadsheets/d/1hHd9riWh-hQJnNOzEI_ZZN1LpdY_bBSbvDOVPdfYslM/edit?gid=1853523197#gid=1853523197 https://docs.google.com/spreadsheets/d/1hHd9riWh-hQJnNOzEI_Z... This sheet only lists the response latency, but it does look more recent than the one you listed. In any event, the sheet you link to shows generation speed and throughput (generation speed is cell E18 which shows 527 t/s). G18 is perhaps prefill + gen but even that doesn't add up.
  • fancyfredbot 2y ago
    I noticed column G is always 2x column E. So I think they are counting both input and output tokens in the throughput for G but not for E. There's only one Google doc and you are linking to the same one I mentioned and the same one in the article... Not sure why you only see latency - there's definitely a throughput tab too.
    • qeternity 2y ago
      It's not though, it's close, but it's not. E18 is 527.75. 2 * 527.75 = 1055.5. G18 is 1053.16. So yeah, I have no idea.