4 ms·
What is the benefits of on device routing? How do you decide if the task can be run on device?
by oypass 4mo ago
What is the benefits of on device routing? How do you decide if the task can be run on device?
- davidvgilmore 4mo agoFor those that have capable enough hardware, it's effectively free to run subtasks on-device. (just the marginal cost of additional electricity). With Google's most recent 12b param Gemma model, even Mac users with just 16gb of unified memory can offload some tasks on-device.