3 ms·

Wondering how difficult this would be to get running on a m1 max?

• ComplexSystems 4y ago
  I got one token every 8 minutes or so.
  • 2Gkashmiri 4y ago
    Is that good? Not good?
    • Tepix 4y ago
      Usually you want tokens per second, not seconds per token. So it's a bad sign.
    • gorbypark 4y ago
      A token is approximately 4 characters. So, four characters per 8 minutes is pretty slow. This comment would take 1224 minutes to generate, if I was an AI.
  • popol12 4y ago
    Using which model ? On a pretty mid range i5 11th gen I'm getting 0.35 token/s, using the 7B model. Haven't tried the bigger models.
• deleted 4y ago
  [deleted]
• swyx 4y ago
  another commenter posted a fork that does it https://news.ycombinator.com/item?id=35067469 https://news.ycombinator.com/item?id=35067469 per the readme it looks like there a few bugs to figure out in case anyone here is a pytorch expert