7 ms·
MatX: Efficient C++17 GPU numerical computing library with Python-like syntax
- 26fingies 3y agovery nice but i assume it’s going to be limited to cuda as a backend because nvidia
- __loam 3y agoI don't actually know a lot about massively parallel libraries like CUDA. Does AMD have an equivalent technology associate with their GPUs? It feels like it should be fairly straightforward to create some kind of high level library that just uses CUDA or whatever AMD has on the back end.
- cmovq 3y agoTraditionally OpenCL was the alternative to CUDA. Recently AMD has been pushing their ROCm platform.
- pjmlp 3y agoOthers are free to provide proper C++ support on their GPUs.
- SpaceNoodled 3y agoThey're also free to write math libraries that don't depend on GPUs.
- cburdick13 3y agoHi, MatX currently has partial support for CPUs too. Please see this comment: https://news.ycombinator.com/item?id=37758635 https://news.ycombinator.com/item?id=37758635
- spicybright 3y agoThe performance boost you get from using a GPU is incredible. You might as well say they're free to make monitors that don't rely on color.
- Mr_P 3y agoGPU performance per dollar is only competitive for specific workloads. For extremely large scale compute, getting enough data center GPUs can also be challenging.
- cburdick13 3y agoCurrently we support both CUDA and CPU to some extent. CPU is done through standard C++ (and soon stdpar). Obviously standard C++ is problematic since it doesn't include everything we support (FFTs, matrix multiplies, etc). One option is to use open-source libraries that do these, but then it ends up being a lot of dependencies that are hard to manage. We have plans to improve CPU support soon, so stay tuned.
- rossy 3y agoVulkan Compute would be nice.
- nologic01 3y agoI would'nt touch nvidia "goodies" with a barge pole. Perfectly good 5-year GPU deprecated upon upgrade to ubuntu 22.04 because of cuda/drivers. The lock-in trap of the century.
- Foobar8568 3y ago>Please use CUDA 11.4 Which should be compatible with any Kepler late architecture, as in 6xx models from 10years ago+?
- cburdick13 3y agoThat's right, we've tested down to pascal, but this should work on Kepler too since CUDA and the underlying libraries support it.
- downrightmike 3y agoI have a spare Titan Z that could be useful if so.
- choppaface 3y agoWill you publish benchmarks for e.g. K80? Or provide a way for users to contribute? It's really handy to know, e.g. comparable to what is Resnet50 inference on a bunch of architectures.
- cburdick13 3y agoHi, what specifically are you looking to benchmark on the K80? Users are free to contribute and we've had many external PRs. Contribution guide is here: https://github.com/NVIDIA/MatX/blob/main/CONTRIBUTING.md https://github.com/NVIDIA/MatX/blob/main/CONTRIBUTING.md
- choppaface 3y agowell the root README.md has some "benchmarks" at the very top. maybe there's no existing benchmarks doc with more details? or even a repo of what's in the README? it's like if numpy is 5 sec on CPU, but a K80 is only 100ms, then the K80 cost might be worth not going for the A100 at 3ms. Similar argument for jetson.
- cburdick13 3y agoHi all, I'm one of the maintainers of MatX. I didn't expect it to hit HN this soon, but happy to answer any questions.
- iamlemec 3y agoThanks, looks really interesting. Do functions like matmul support inputs of differing type, like say A=int8 and B=float? Would be nice if you could get memory efficient quantized matmul with operator fusion.
- cburdick13 3y agoWe typically support whatever the underlying library supports. For int8 the support would come from cuBLASLt currently. I don't believe that or Cutlass supports mixed precision inputs, but I can check.
- gregjm 3y agoCUTLASS, which is NVIDIA’s C++ template library for writing matrix multiply and convolution kernels parametrized over input/output types, operators, and algorithm block sizes, theoretically supports this. But, each input of the (k, n) shaped matrix B will be read from global memory ceil(n / block dimension) times in an algorithm that computes one (block dimension, block dimension) submatrix of the output matrix D per thread block. It will probably be more efficient to cast your B matrix to FP16 or INT8 lower precision in a preprocessing kernel to reduce memory traffic in the matrix multiply kernel. On newer GPUs, though, we have this huge L2 cache which makes the calculus a little different if your working set fits into it. e.g. Ampere A100 has 40MB L2$.
- KeplerBoy 3y agoActually just started looking into MatX yesterday to accelerate our radar pipeline. Really interesting to see that this use-case is heavily featured in the documentation. Is the UCLA/Nvidia/Raytheon collaboration (as presented in a recent GTC talk) a major force behind the development of MatX?
- 3y ago
- trostaft 3y agoIt's nice to have easier to use C++ GPU numerical computing libraries, but it seems like there's a good bit of work to go in terms of feature set. Taking CuPy as an example, they've nearly entirely mirrored the enormous featureset of both NumPy and SciPy. Although I'm not expecting on that scale here, but what's the planned future work here? E.g. matrix free operators and operations, sparse matrix support, or symmetry aware eigenvalue / singular value decompositions? There are official cuda libraries that support these (e.g. cuSPARSE). Otherwise this looks good! Documentation is decent too.
- cburdick13 3y agoHi, having feature parity with cuPy is a daunting task, especially for a C++ library. At this point we feel we have a good foundation for all kinds of basic and advanced tensor manipulations, and have a growing number of library-based functions on top of that. The low-hanging fruit was wrapping the CUDA math libraries, so that has the most progress. Since MatX was originally intended for streaming/real-time processing the focus has been on making C++ for CUDA easier to use for those kinds of applications. There are also a lot of things in cuPy and sciPy that don't make a lot of sense to do on the GPU, like offline tasks such as filter design in signal processing. Also, since C++ users typically are writing in that language for maximum performance, we've put a large focus on making sure we are as close to writing optimized CUDA as possible. In general, most workloads we've tested are about 3-4x faster than the cuPy counterpart due to better fusion and language overhead. We have discussed supporting cusparse as well, but cusparse typically requires a different input type like a CSR matrix. This is not something you'd typically want to detect/convert, so we're still discussing ways of getting this integrated cleanly. We do have several versions of SVD, including one that calls cuSolver. Feedback is always appreciated!
- frozenport 3y agoI've preferred using libtorch for translating python to C++. I feel this effort is poorly differentiated compared to previous efforts.
- cburdick13 3y agoHi, besides an (subjectively) easier syntax, the performance should be higher compared to libtorch. Every operator expression (think of it as an arithmetic expression) is evaluated at compile-time and is often fused into a single GPU kernel. This also removes the need for JITing. If there's a specific workflow you're curious about comparing libtorch vs MatX please let us know and we can try it out.
- engfan 3y agoWhy can’t you keep the calling interfaces of functions as close to the Py libraries as possible, simplifying the transition for everyone? Will that really destroy the performance increase? Common calling interfaces make everything much simpler. Even in this simple example, the calls differ significantly.
- cburdick13 3y agoWe've tried our best to match python as well as we can, or falling back to matlab-style if Python doesn't have it. Many of our unit tests are verified against python, so the conversion is typically very easy. The one thing that python has that makes this much easier is keyword arguments. We've tried to use overloads to mimic this as best as we could. That being said, in the example on the home page there are notable differences: 1) we have the run() method. The reason is that the expression before the run is lazily evaluated for performance and does not execute anything. Having the run() method allows you to run the same line of code on either a CPU or GPU by changing the argument to run() 2) in MatX memory allocation is explicit. Python does it as-needed, but this causes a performance penalty with allocations and deallocations that are not under your control. Specifically in the FFT example, numPy will allocate an ndarray prior to calling it, but on the same line. In MatX the allocation is (typically) done before the operation so you can control the performance of the hot path of code. If you have any specific suggestions, we would love to hear it
- waynecochran 3y agoI’d be interested in a comparison with Eigen for Linear Algebra workflows. I would love to perform eigen analysis or svd on large matrices using CUDA.
- cburdick13 3y agoHi, what matrix sizes and types are you working with and how many batches? In general it sounds be similar to eigen, but with GPU support. We have several svd methods for different scenarios, so if you give us the info above we can let you know the performance.
- gyrovagueGeist 3y agoVery nice! I think the landing README should really be a comparison with CuPy and not NumPy though
- cburdick13 3y agoGood point, and agreed the landing page is a bit sensational. I mentioned it elsewhere but between MatX and cuPy we see a 3-4x performance difference on average. The gap tends to widen with more complex workflows where compile-time kernel fusion gives more improvements compared to something like a single GEMM.
- LeroyRaz 3y agoWhat are the advantages of MatX over Jax?
- cburdick13 3y agoThe main difference is Jax is for python primarily, while MatX is c++. This might seem like a poor answer, but in many domains (quasi-real time, signal processing, etc) the language is important to give certainty guarantees on performance. MatX has been used in several projects with hundreds of microsecond deadlines, which is not usually something you'd choose Python for out of the box. We are instead targeting users who already have python or high-level code that they need to port to c++ for whatever reason, and want to do it in the easiest way possible. With c++17 we're able to provide a simple syntax without compromising on performance compared to most native code.
- temren 3y agowould matx work with gnuradio?
- cburdick13 3y agoI have to admit I'm only tangentially familiar with gnuradio, but matx should be able to integrate with any C++17 codebase. We have several examples of integration with our streaming sensor pipeline called holoscan. see this radar pipeline example for one: https://github.com/nvidia-holoscan/holohub/tree/main/applications/simple_radar_pipeline/cpp https://github.com/nvidia-holoscan/holohub/tree/main/applica...
- alfalfasprout 3y agoHow does this compare to xtensor other than being GPU specific?
- cburdick13 3y agoThe main difference is the GPU part. This is a large difference because the same lazy evaluated template type can be run on the CPU or GPU through what we call an executor. On the CPU it's likely very similar to how xtensor is already using expression trees, but on the GPU it's quite a bit different because if the libraries backing it and optimized kernels. The syntax of MatX also allows us to do more kernel fusion in the future to improve performance without any changes.
- elashri 3y ago> Python/Numpy: 5360ms (Xeon(R) CPU E5-2698 v4 @ 2.20GHz) CuPy: 10.6ms (A100) MatX: 2.54ms (A100) > Are they even comparing apples to apples to claim that they see these improvements over NumPy? > While the code complexity and length are roughly the same, the MatX version shows a 2100x over the Numpy version, and over 4x faster than the CuPy version on the same GPU. NumPy doesn't use GPU by default unless you use something like Jax [1] to compile NumPy code to run on GPUs. I think more honest comparison will mainly compare MatX running on same CPU like NumPy and for GPU comparison focus to compare vs CuPy. [1] https://github.com/google/jax https://github.com/google/jax
- cburdick13 3y agoI agree we should update it given that it's also a very old comparison (2 years now). The cuPy comparison should be fair since it was the same GPU. I addressed it more here: https://news.ycombinator.com/item?id=37760120 https://news.ycombinator.com/item?id=37760120
- KeplerBoy 3y agoIt's very much an apples to oranges comparison, but nevertheless relevant, since NumPy is the defacto baseline for anything numeric in Python.
- albertzeyer 3y agoNot really, for GPU computation, PyTorch, TensorFlow or JAX are the baselines (they are not just for ML stuff). Even for CPU, that's a more relevant comparison.
- KeplerBoy 3y agoI agree in principle, but my experience is that researchers writing code just default to NumPy.
- zzbn00 3y agoThis seems like an FFT based benchmark.... I believe NumPy for licensing reasons includes a port of old FFTPACK which is only single threaded. (Py)FFTW or others are surely much faster. So the test seems to compare one thread of a CPU launched in 2016 running a port of 40year old fortran code to a optimised FFT library on a GPU launched in 2020.
- cxx 3y agoThis looks great! I'm definitely going to try this, I've been hoping for something like this for a long time. Eigen is great of course but GPU acceleration is a bit clunky to use, this looks much closer to the code written by the researchers.
- touisteur 3y agoIf I'm reading the examples correctly, this is calling parameterized kernels in sequence, with nice-to-write matrix formats and memory movement? You're still ping-ponging from global (on device) to l2/shared memory each time? Cool approach for quick prototyping and replacing/porting some Eigen-stuff. For those of us in-kernel day-laborers I can't wait for cuBLASDx and cuSolverDx releases, and for my brain to wrap around cutlass and cute. Please don't forget us!