You need ~24-26GB for basic setup (17-19GB model + 128K 8 bit context), so you can, but not much memory would be left for doing anything else on that machine. A

You need ~24-26GB for basic setup (17-19GB model + 128K 8 bit context), so you can, but not much memory would be left for doing anything else on that machine. And even then it would run at like 5-10 t/s (due to relatively low memory bandwidth of M3 Pro) and slow prompt processing (couple hundreds of t/s?) If they end up releasing updated 35B-A3B variant, then it would be much more interesting in generation speed (~50 t/s) For inference engine/server you have two (main) choices: llama.cpp for platform-agnostic, MLX for Apple-only. They will spin up OpenAI-compatible local server, and you point your agent harness to it. For llama.cpp this should be reasonable (maybe shrink context to 128K) starting point: https://x.com/ggerganov/status/2088312671196082312 https://x.com/ggerganov/status/2088312671196082312