cafaxo

4 ms·

1.
by cafaxo 2y ago
Yes, sorry -- I did not realize that for some reason. I removed my comment.
2.
by cafaxo 2y ago
[Comment removed by author]
3.
by cafaxo 2y ago
Does gradient descent really do well for deep learning when the gradient is computed with respect to the whole dataset? I assumed that the noise in SGD played an important role for escaping local minima.
4.
by cafaxo 3y ago
I did a similar thing for Julia: Llama2.jl contains vanilla Julia code [1] for training small Llama2-style models on the CPU. [1] https://github.com/cafaxo/Llama2.jl/tree/master/src/training
5.
by cafaxo 3y ago
To me, the clearest way to think about signed vs. unsigned integers is that different representatives of the integers modulo n are chosen. For example, for 8-bit signed integers we choose the representatives -128, -127, ..., 127 of the resi
6.
by cafaxo 3y ago
Yes, exactly.
7.
by cafaxo 3y ago
Of course. Theoretically, the determinant answers the binary question "singular" or "nonsingular". Numerically, such a binary answer is pretty useless. Here, we need a measure of how singular/nonsingular a matrix is
8.
by cafaxo 3y ago
Edit: Sorry, I completely messed up my original answer here. A better version: Let's say we are in a setting where we only work with integers. A matrix is invertible iff its determinant is invertible in the underlying ring. The only in
9.
by cafaxo 3y ago
They define a generic "is_singular" function and test it with a 2x2 matrix. The problem with the determinant is not about performance. It is just useless for determining if a matrix is singular. The thing that gives it away is tha
10.
by cafaxo 3y ago
From page 6, when they describe their synthetic textbook dataset: "Consider the matrix A = np.array([[1, 2], [2, 4]]). We can check if this matrix is singular or nonsingular using the determinant function. [...]" No. The determina
11.
by cafaxo 3y ago
GPT-4's explanation of its optimization does not make sense to me. It writes "Instead of moving it to P, we can directly use S in the following comparisons, saving one instruction." but then proceeds to use P as if that mov h