4 ms·
Fully connected layers are matrix-vector multiplication (which turns into matrix-matrix if you do an entire batch at once). Convolutional layers are lots of tin
by jules 8y ago
Fully connected layers are matrix-vector multiplication (which turns into matrix-matrix if you do an entire batch at once). Convolutional layers are lots of tiny dot products. You can rewrite that as matrix-matrix multiplication.
- taeric 8y agoThis was my thought. Am I correct in saying the batch size influences the size of the matrix? Or is there a more natural limiting factor?
- jules 8y agoThat's right. For a fully connected layer the matrix is of dimension (batch size) x (number of neurons in layer k) which gets multiplied by a matrix of dimension (number of neurons in layer k) x (number of neurons in layer k+1). People do this because going through the network with a batch is more efficient than going through the network separately for each element in the batch, mainly because multiplying two matrices is more efficient than doing n matrix-vector products. The batch size is limited because you get diminishing speedup out of increasing your batch size. Stochastic gradient descent also gets less improvement out of each data point per pass through the data set as you increase the batch size, so you need more passes. If you take the whole data set to be your batch size then you get ordinary gradient descent, which needs a lot more passes through the data set than stochastic gradient descent.