4 ms·
deep learning uses a lot lot of sparse linear systems
by motoboi 6y ago
deep learning uses a lot lot of sparse linear systems
- jtsiskin 6y agoFor some easy to grok examples: imagine MNIST, where all the pixels besides the writing are 0; or a recommendation system, where column i has the users rating for movie i, or number of times they listened to song i; or any one-got encoding scheme
- eru 6y ago> or any one-got encoding scheme Typo: should be one-hot encoding scheme. Sparsity of input isn't often relatively easy to achieve, as you suggest. Sparsity of intermediate layers requires more work. (But is often a good regularization technique.)
- jtsiskin 6y agoHumorously “one-got” is almost a apt description :)
- johncolanduoni 6y agoIt does, but the problem this algorithm handles is dealing with demanding convergence for all input matrices without adding an additional “n” factor to handle increased precision. Meanwhile deep learning is moving in the direction of 16-bit floating point and 8-bit integer scalars.