2 ms·
Some kinds of inference are expensive, yes, not going to dispute that. But 99.95% of it is actually surprisingly inexpensive. Hell, a lot of useful workloads ca
by m0zg 7y ago
Some kinds of inference are expensive, yes, not going to dispute that. But 99.95% of it is actually surprisingly inexpensive. Hell, a lot of useful workloads can be deployed on a cell phone nowadays, and that fraction will increase over time, further reducing inference costs or eliminating them outright (or rather moving them to the consumer).
For the vast majority of people the main expense is creating the combination of a dataset and model that works for their practical problem, with the dataset being the harder (and sometimes more expensive) problem of the two.
The dataset is also their "moat", even though most of them don't realize it, and don't put enough care into that part of the pipeline.
- fxtentacle 7y agoThe algorithms that run on cell phones tend to be specially optimized and quality-reduced neural networks. For example, https://arxiv.org/abs/1704.04861 https://arxiv.org/abs/1704.04861
- jiofih 7y agoApple does a pretty good job at that, with no compromise in quality.
- fxtentacle 7y agoI believe that just due to memory constraints, running any high-quality neural network on phones is currently impossible. State of the art optical flow tracking needs about 10 GB of GPU memory to execute on full HD frames. I wouldn't know of any mainstream phone with that much RAM. That, BTW, is also the reason why autonomous drones usually downsample the images before AI tracking, which has the nasty side effect of making thin branches, fences, telephone wires, etc. invisible.