3 ms·

All of this is true only while no software is utilizing parallel inference of multiple LLM queries. The Macs will hit the wall.

• deleted 2y ago
  [deleted]
• ryao 2y ago
  People interested in running multiple LLM queries in parallel are not people who would consider buying Apple Silicon.
  • int_19h 2y ago
    There are other ways to parallelize even a single query for faster output, e.g. speculative decoding with small draft models.