3 ms·

I've been running various models on a Mac Pro 2013 (8 cores, 32 GB RAM) at about 8 to 10 t/s for months. It's not fast, but it's more than enough for many actua

by wazoox 5mo ago

I've been running various models on a Mac Pro 2013 (8 cores, 32 GB RAM) at about 8 to 10 t/s for months. It's not fast, but it's more than enough for many actual tasks, in particular background tasks. An iMac pro will do just as well I suppose.

fooker 5mo ago
What are the tasks that do well with 8-10 t/s ?
- wazoox 5mo ago
  The sort of task you don't expect to end immediately. If extracting data from a bunch of PDFs takes 1 hour or the whole night, that doesn't make much difference to me. It's not fast enough for auto completion and slightly too slow for chat (but bearable IMO).
  - fooker 5mo ago
    Running a local llm at 10 t/s overnight to extract data from a few PDFs will burn more in electricity than paying cents for the hosted kimi models. You can (sometimes) break even if you have a workstation GPU.
    - wazoox 5mo ago
      Sometimes data privacy is paramount.
neverartful 5mo ago
I have and use a Mac Pro 2013 too. Mine is 8 cores with 64 GB RAM. I haven't used mine for any LLM workloads, but it does just fine for most stuff. My biggest concern with it is the OS. I'm still running macOS (the latest supported version) but it's getting continually further out-of-date security wise all the time.