5 ms·

Large-Scale Matrix Factorization on TPUs

75 points by RafelMri 5y ago

danuker 5y ago
All these "papers" are just large companies flexing their hardware resources. "Join our vendor-locked implementation" is all I hear. "Trust us by investing into writing software for hardware you can never own".
- luma 5y ago
  They do occassionally sell TPU-enabled systems. My backorder from Mouser is still sitting there for several months but some day I too may have a TPU to hack on.
  - dekhn 5y ago
    that is a totally different product which is only called TPU because the team who makes it wants to sound cool
    - alias_neo 5y ago
      Could you elaborate? I guess the person you're replying to is talking about the Coral stuff such as the "USB Accelerator" et al, dubbed "Edge TPUs"; I too am after one of those for my home-security camera system but I've never really looked at what they _are_.
      - dekhn 5y ago
        Yes, Coral (edge TPU) are just custom designed ASICs mainly for inference (not training). No shared silicon with Real TPUs that Google runs in the data centers. They're just drafting off the marketing.
        alias_neo 5y ago
        Thanks, gives me something to read about!
- Athas 5y ago
  I feel the same way. But beyond these concerns, I must admit I also feel a sense of sadness due to the inevitable rise of application-specific hardware. When I hear about these super-fast specialised chips, all I can think about is whether I'd have enjoyed sitting in my bedroom as a teenager hacking on them. And the answer is probably no. They can do very specific things, but they don't have the kind of freedom of programming I enjoy. Maybe I just need to look at them differently, or maybe my perception of their capabilities is just wrong. I do like GPUs, for example. While they are ornery and limited in many ways, they are still flexible enough that you can do enjoyable general-purpose programming on them.
  - eru 5y ago
    A type-writer or a piano is an application specific piece of hardware, yet people can still do an amazing variety of things with them. In the last few decades we got really lucky that general purpose CPUs kept getting faster and faster. So many of us could get away with ignoring application specific computer hardware. Btw, as far as I can tell, TPUs are really just a variant of GPUs with an emphasis on lower precision? Have a look also at the kind of special purpose hardware people make to mine crypto-currencies. I think the challenge for your teenage hacker is not so much to make them do interesting things, but to design them? (Assuming the tools and simulators become cheap enough?) FPGAs seem like they are cheap enough for a teenager?
    - rrss 5y ago
      > Btw, as far as I can tell, TPUs are really just a variant of GPUs with an emphasis on lower precision? GPUs and TPUs have substantially different architectures, it’s not accurate to consider a TPU an a variant of a GPU. See https://dl.acm.org/doi/pdf/10.1145/3360307 https://dl.acm.org/doi/pdf/10.1145/3360307, particularly section “Contrasting GPU and TPU Architectures” looking only at the matrix multiply units (and ignoring the hardware multithreading in GPUs that was deliberately not part of the TPU architecture): TPUv3: 2 cores, 2 128x128 matrix multiply units per core V100: 80 cores, 8 4x4 matrix multiply units per core
      - 5y ago
- ArtWomb 5y ago
  Google TPUs are inspiring everyone. Dieter Fox confirmed at Stanford HAI, NVidia's investing big in cloud accelerators too ;) http://nvdla.org/primer.html http://nvdla.org/primer.html
- jeffbee 5y ago
  Owning hardware is hugely overrated. Renting it is clearly superior.
  - danuker 5y ago
    Not at 10x the price, even including maintenance. There's only so much of the Internet you can route through a small number of points, before it starts to fail.
ChrisRackauckas 5y ago
Looking at the paper was pretty disappointing. "Comparison of training time per epoch (in seconds), of plausible alternatives of linear system solver on TPU" lists CG, Cholesky, LU, and QR. It sounds like this method is a viable alternative to LU or QR factorizations from a cursory read? No, and what started tipping me off is that those methods are not comparable because CG and Cholesky are only for positive semi-definite matrices, so while it's not mentioned anywhere else, the fact that this method works with these factorization techniques means the matrices that they are looking at fall into a very small subset of special matrices. Even then, the subset is even smaller because CG works sufficiently well here without preconditioning, so the method is for not too ill-conditioned positive semi-definite matrices? And for such papers in matrix factorization, there's always some kind of discussion about numerical stability: what pivoting strategies are used, how the pivoting decreases the numerical error growth rate, with empirical studies based on the conditioning of the matrix. Because none of this is done, and in fact the exact opposite is done by emphasizing the BFloat16 precision, gives me no indication that you can have eigenvalues 1e-15 apart and have this work. Using low precision arithmetic doesn't necessarily mean you cannot get sufficiently good results, there's a whole body of work in mixed precision algorithms. I'd point to Nick Higham's fantastic work on mixed precision matrix logarithms as a very nice example and read: https://www.maths.manchester.ac.uk/~higham/talks/essam18.pdf https://www.maths.manchester.ac.uk/~higham/talks/essam18.pdf https://epubs.siam.org/doi/10.1137/17M1129866 https://epubs.siam.org/doi/10.1137/17M1129866. But this paper discusses nothing about precision or error, shrugs it off, and doesn't even empirically measure the error in the 365mil case? "Large-Scale Factorization of Well-Conditioned Positive Semi-Definite Matrices with TPUs" would be a much clearer title then. All of this doesn't mean it's not useful, it just means it's useful for a very specific class of problems and presenting it as a viable alternative to LU or QR factorization is pretty... out there.
- dchftcs 5y ago
  If solving systems of linear equations is a stronger priority they'd produce hardware with more precision. Unfortunately this is just an offshoot from a machine-learning driven project. But at least the timings could tell you how much value people can gain by implementing their own hardware solution.
minsight 5y ago
I can't resist checking out the details of anything that Google might describe using the phrase "Large-Scale". As much as I'm interested in the nuts and bolts of their solutions, I'd also be very interested in the problems that become solvable with quick factorization of matrices with billions of rows.
Khelavaster 5y ago
How quickly do they arbitrarily-nearly approximate matrix factorization of huge matrices, though? https://keg.cs.tsinghua.edu.cn/jietang/publications/WSDM18-Qiu-et-al-NetMF-network-embedding.pdf https://keg.cs.tsinghua.edu.cn/jietang/publications/WSDM18-Q... https://www.microsoft.com/en-us/research/publication/netsmf-large-scale-network-embedding-as-sparse-matrix-factorization/ https://www.microsoft.com/en-us/research/publication/netsmf-...

75 points by RafelMri 5y ago

All these "papers" are just large companies flexing their hardware resources. "Join our vendor-locked implementation" is all I hear. "Trust us by investing into writing software for hardware you can never own".

luma 5y ago
They do occassionally sell TPU-enabled systems. My backorder from Mouser is still sitting there for several months but some day I too may have a TPU to hack on.
- dekhn 5y ago
  that is a totally different product which is only called TPU because the team who makes it wants to sound cool
  - alias_neo 5y ago
    Could you elaborate? I guess the person you're replying to is talking about the Coral stuff such as the "USB Accelerator" et al, dubbed "Edge TPUs"; I too am after one of those for my home-security camera system but I've never really looked at what they _are_.
    - dekhn 5y ago
      Yes, Coral (edge TPU) are just custom designed ASICs mainly for inference (not training). No shared silicon with Real TPUs that Google runs in the data centers. They're just drafting off the marketing.
      - alias_neo 5y ago
        Thanks, gives me something to read about!
Athas 5y ago
I feel the same way. But beyond these concerns, I must admit I also feel a sense of sadness due to the inevitable rise of application-specific hardware. When I hear about these super-fast specialised chips, all I can think about is whether I'd have enjoyed sitting in my bedroom as a teenager hacking on them. And the answer is probably no. They can do very specific things, but they don't have the kind of freedom of programming I enjoy. Maybe I just need to look at them differently, or maybe my perception of their capabilities is just wrong. I do like GPUs, for example. While they are ornery and limited in many ways, they are still flexible enough that you can do enjoyable general-purpose programming on them.
- eru 5y ago
  A type-writer or a piano is an application specific piece of hardware, yet people can still do an amazing variety of things with them. In the last few decades we got really lucky that general purpose CPUs kept getting faster and faster. So many of us could get away with ignoring application specific computer hardware. Btw, as far as I can tell, TPUs are really just a variant of GPUs with an emphasis on lower precision? Have a look also at the kind of special purpose hardware people make to mine crypto-currencies. I think the challenge for your teenage hacker is not so much to make them do interesting things, but to design them? (Assuming the tools and simulators become cheap enough?) FPGAs seem like they are cheap enough for a teenager?
  - rrss 5y ago
    > Btw, as far as I can tell, TPUs are really just a variant of GPUs with an emphasis on lower precision? GPUs and TPUs have substantially different architectures, it’s not accurate to consider a TPU an a variant of a GPU. See https://dl.acm.org/doi/pdf/10.1145/3360307 https://dl.acm.org/doi/pdf/10.1145/3360307, particularly section “Contrasting GPU and TPU Architectures” looking only at the matrix multiply units (and ignoring the hardware multithreading in GPUs that was deliberately not part of the TPU architecture): TPUv3: 2 cores, 2 128x128 matrix multiply units per core V100: 80 cores, 8 4x4 matrix multiply units per core
    - 5y ago
ArtWomb 5y ago
Google TPUs are inspiring everyone. Dieter Fox confirmed at Stanford HAI, NVidia's investing big in cloud accelerators too ;) http://nvdla.org/primer.html http://nvdla.org/primer.html
jeffbee 5y ago
Owning hardware is hugely overrated. Renting it is clearly superior.
- danuker 5y ago
  Not at 10x the price, even including maintenance. There's only so much of the Internet you can route through a small number of points, before it starts to fail.

ChrisRackauckas 5y ago

Looking at the paper was pretty disappointing. "Comparison of training time per epoch (in seconds), of plausible alternatives of linear system solver on TPU" lists CG, Cholesky, LU, and QR. It sounds like this method is a viable alternative to LU or QR factorizations from a cursory read? No, and what started tipping me off is that those methods are not comparable because CG and Cholesky are only for positive semi-definite matrices, so while it's not mentioned anywhere else, the fact that this method works with these factorization techniques means the matrices that they are looking at fall into a very small subset of special matrices. Even then, the subset is even smaller because CG works sufficiently well here without preconditioning, so the method is for not too ill-conditioned positive semi-definite matrices? And for such papers in matrix factorization, there's always some kind of discussion about numerical stability: what pivoting strategies are used, how the pivoting decreases the numerical error growth rate, with empirical studies based on the conditioning of the matrix. Because none of this is done, and in fact the exact opposite is done by emphasizing the BFloat16 precision, gives me no indication that you can have eigenvalues 1e-15 apart and have this work. Using low precision arithmetic doesn't necessarily mean you cannot get sufficiently good results, there's a whole body of work in mixed precision algorithms. I'd point to Nick Higham's fantastic work on mixed precision matrix logarithms as a very nice example and read: https://www.maths.manchester.ac.uk/~higham/talks/essam18.pdf https://www.maths.manchester.ac.uk/~higham/talks/essam18.pdf https://epubs.siam.org/doi/10.1137/17M1129866 https://epubs.siam.org/doi/10.1137/17M1129866. But this paper discusses nothing about precision or error, shrugs it off, and doesn't even empirically measure the error in the 365mil case? "Large-Scale Factorization of Well-Conditioned Positive Semi-Definite Matrices with TPUs" would be a much clearer title then. All of this doesn't mean it's not useful, it just means it's useful for a very specific class of problems and presenting it as a viable alternative to LU or QR factorization is pretty... out there.

dchftcs 5y ago
If solving systems of linear equations is a stronger priority they'd produce hardware with more precision. Unfortunately this is just an offshoot from a machine-learning driven project. But at least the timings could tell you how much value people can gain by implementing their own hardware solution.

minsight 5y ago

I can't resist checking out the details of anything that Google might describe using the phrase "Large-Scale". As much as I'm interested in the nuts and bolts of their solutions, I'd also be very interested in the problems that become solvable with quick factorization of matrices with billions of rows.

Khelavaster 5y ago

How quickly do they arbitrarily-nearly approximate matrix factorization of huge matrices, though? https://keg.cs.tsinghua.edu.cn/jietang/publications/WSDM18-Qiu-et-al-NetMF-network-embedding.pdf https://keg.cs.tsinghua.edu.cn/jietang/publications/WSDM18-Q... https://www.microsoft.com/en-us/research/publication/netsmf-large-scale-network-embedding-as-sparse-matrix-factorization/ https://www.microsoft.com/en-us/research/publication/netsmf-...