2 ms·
It seems like a stronger story for robotics, since smaller models can always react to the environment faster than large models at a given hardware budget. Also
by philipkglass 3mo ago
It seems like a stronger story for robotics, since smaller models can always react to the environment faster than large models at a given hardware budget. Also because robots that keep their models local for latency or reliability aren't going to be carrying many kilowatts of inference capacity.
- lumost 3mo agoremote inference should be sufficient for most robotics applications with potentially a small model for safety critical actions running locally. Unless you are in military robotics or automotive of course :)
- AlotOfReading 3mo agoThere are many, many factories that still don't have internet access on the floor, and commercial inference generally has response latencies measured in seconds. I struggle to imagine a factory spending hundreds of thousands for the local compute to run a large model either, given how cheap they are about expenses. I'm also skeptical that you can cleanly differentiate between "safety critical actions" and "actions", though this is less of a practical concern given how laissez-faire some manufacturers are. For context, I work on safety critical robotics (in automotive).