3 ms·
I don’t know AI, but, weight matrices aren’t square in general, right? My first guess for something like this would be to take the SVD instead, since you can al
by bee_rider 4mo ago
I don’t know AI, but, weight matrices aren’t square in general, right? My first guess for something like this would be to take the SVD instead, since you can always do that, but I’m sure that’s been tried already.
- phkahler 4mo agoBut orthogonal matrices are square.
- bee_rider 4mo agoAh, good point. I assumed they were coming up with some orthogonal basis vectors for a (potentially) non-square matrix. Edit: actually I’m not sure, this Newton-Schulz process seems to work for non-square matrices as well. Generally I see “orthogonalization” refer to the process of coming up with those orthogonal basis vectors but it could be a domain-specific lingo thing.