5 ms·
But you can't support Pytorch without a proper foundation in place. They don't need to support zillion _end_ libraries, sure, but they do need to have at least
by shihab 3y ago
But you can't support Pytorch without a proper foundation in place. They don't need to support zillion _end_ libraries, sure, but they do need to have at least a very good set of standard libraries, equivalent of Cublas, Curand etc.
And they don't. My work recently had me working with rocRAND (Rocm's answer to Curand). It was frankly pretty bad- the design, performance (50% slower in places that don't make any sense because generating random numbers is not exact that complicated), and documentation (God it was awful).
Now, that's a small slice of the larger pie. But imagine if this trend continues for other libraries.
- lumost 3y agoFolks also underestimate how complex these libraries are. There are dozens of projects to make BLAS alternatives which give up after ~3-6 months when they realize that this project will take years to be successful.
- eternityforest 3y agoHow does that work? Why not pick up where the previous team left off instead of everyone starting new ones? Or are they all targeting different backends and hardware?
- haltist 3y agoIt's a compiler problem and there is no money in compilers [1]. If someone made an intermediate representation for AI graphs and then wrote a compiler from that intermediate format into whatever backend was the deployment target then they might be able to charge money for support and bug fixes but that would be it. It's not the kind of business anyone wants to be in so there is no good intermediate format and compiler that is platform agnostic. 1: https://tinygrad.org/ https://tinygrad.org/
- whatshisface 3y agoJAX is a compiler.
- deleted 3y ago[deleted]
- haltist 3y agoSo are TensorFlow and PyTorch. All AI/ML frameworks have to translate high-level tensor programs into executable artifacts for the given hardware and they're all given away for free because there is no way to make money with them. It's all open source and free. So the big tech companies subsidize the compilers because they want hardware to be the moat. It's why the running joke is that I need $80B to build AGI. The software is cheap/free, the hardware costs money.
- singhrac 3y agoGenerating random numbers is a bit complicated! I wrote some of the samplers in Pytorch (probably replaced by now) and some of the underlying pseudo-random algorithms that work correctly in parallel are not exactly easy... running the same PRNG with the same seed on all your cores will produce the same result, which is probably NOT what you want from your API. But, to be honest, it's not that hard either. I'm surprised their API is 2x slower, Philox is 10 years old now and I don't think there's a licensing fee?
- eternityforest 3y agoI wonder if the next generation chips are going to just have a dedicated hardware RNG per-core if that's an issue?
- lelanthran 3y agoWhy bother? It's not the generation that matters so much, it's the gathering of entropy, which comes from peripherals and not possible to generate on-die. If you don't need cryptographically secure randomness, you still want the entropy for generating the seeds per thread/die/chip.
- eternityforest 3y agoIt absolutely is possible to generate entropy on-die, assuming you actually want entropy and not just a unique value that gets XORed with the seed, so you can still have repeatable seeds. Pretty much every chip has an RNG which can be as simple as just a single free running oscillator you sample
- lelanthran 3y ago> Pretty much every chip has an RNG which can be as simple as just a single free running oscillator you sample Every chip may have some sort of noise to sample, but they are nowhere near good sources of entropy. Entropy is not a binary thing (you either have it or don't), it's a spectrum and entropy gathered on-die is poor entropy. Look, I concede that my knowledge on this subject is a bit dated, but the last time I checked there were no good sources of entropy on-die for any chip in wide use. All cryptographically secure RNGs depend on a peripheral to grab noise from the environment to mix into the entropy pool. A free-running oscillator is a very poor source of entropy.
- nradov 3y agoInstead of generating pseudorandom numbers you can just download files of them. https://archive.random.org/ https://archive.random.org/
- hcrean 3y agoOr you could just re-use the same number; no one can prove it is not random. https://xkcd.com/221/ https://xkcd.com/221/
- slavik81 3y agoIf you haven't already, please consider filing issues on the rocrand GitHub repo for the problems you encountered. The rocrand library is being actively developed and your feedback would be valuable for guiding improvements.
- shihab 3y agoAppreciate it, will do.
- jart 3y agoI honestly don't see why it's so hard. On my project we wrote our own gemm kernels from scratch so llama.cpp didn't need to depend on cublas anymore. Only took a few days and a few hundred lines of code. We had to trade away 5% performance.
- soulbadguy 3y agoFor a given set kernels, and a limited set of architectures, the problem is relatively easy. But covering all the important kernels acros all the crazy architecture out there and with relatively good performance and numerical accuracy ... Much harder
- rightbyte 3y ago> generating random numbers You can't bench implementations of random numbers against each other purely on execution speed. A better algorithm (better statistical properties) will be slower.
- codetrotter 3y agoI have the fastest random number generator in the world. And it works in parallel too! https://i.stack.imgur.com/gFZCK.jpg https://i.stack.imgur.com/gFZCK.jpg
- shihab 3y agoYeah. In this instance, I was talking about the same algorithm (phillox), the difference is purely in implementation.