4 ms·
I would happily jump ship for any competitor that offers solid AI inference benchmarks at a competitive power efficiency, but as far as I can tell Apple owns th
by throwaway894345 5mo ago
I would happily jump ship for any competitor that offers solid AI inference benchmarks at a competitive power efficiency, but as far as I can tell Apple owns that market by a pretty big margin. I’m sure someone will point out if I’m wrong.
- pyrophane 5mo agoI just have a desktop at home that I run inference off of. It is a great setup and I don't find myself wanting to inference models directly on my laptop.
- throwaway894345 5mo agoThat’s what I would do too, but I haven’t found a desktop build that can rival a Mac Mini or Mac Studio on performance per watt. I haven’t looked super hard, but it seems like Mac is in a different ballpark.
- zhivota 5mo agoI mean, once it's a desktop, watts are pretty cheap, so it's a bit strange to optimize on that factor for the desktop form factor. For laptops it makes a ton of sense.
- throwaway894345 5mo agoIf I was optimizing solely for cost I wouldn't use a desktop in the first place. Watts are cheap, but they result in pollution and noise (e.g., fan) or you need some costly alternative. Also I just value efficiency.
- topaz0 5mo agoHumans own inference power efficiency by a much bigger margin
- throwaway894345 5mo agoOwning a human is much more problematic, and anyway a human’s peak efficiency might be higher, but humans have to sleep and take breaks so overall probably not.
- rbanffy 5mo agoHumans also tend to hallucinate a lot, but it’s not polite to use that word for them. With humans we say they were wrong, because offending a human reduces performance and you can’t reset their contexts.
- nl 5mo agoRyzen AI are competitive with Macs in absolute and power efficiency terms. There is a Framework 13 with them I believe.