4 ms·
Most of the effort is in fine tuning the existing methods, which is important, but also leads to reinforcing the status quo in terms of algorithms. In other wor
by mirker 5y ago
Most of the effort is in fine tuning the existing methods, which is important, but also leads to reinforcing the status quo in terms of algorithms. In other words, transistor scaling gets a sizable fraction of optimization, and the viable algorithms are ones most similar to dense matrix multiplication. If you unfix the algorithms (e.g., analog or very sparse models), you’ll fall off this optimization efforts and your model will eat dust.
- varelse 5y agoWhich is why you need a killer demo. New ideas are great, pure research rocks. But new ways to solve real world problems derived from that research is the killer product IMO. See an unnamed AI ASIC company proudly announcing they have achieved better perf/$ without achieving better perf for how not to do this.
- mirker 5y agoIt’s a Catch 22. New algorithms stink because conventional hardware stinks at running them and vice versa. Before AlexNet and GPU-optimized kernels, deep learning took too long to train large models, so large models were not used, and other ML methods seemed more promising. I agree a killer product is what you want, but I think the incentives are wrong for a startup to get both algorithms/hardware right before funding runs out. Just getting a working compiler is out of reach of many of these startups.
- varelse 5y agoWe agree that the incentives for startups are completely broken right now. But until that environment changes, the bar is pretty high for disrupting current methods.