5 ms·

I'm no GPU programmer, but seems easy to use even for someone like me. I pulled together a quick demo of using the GPU vs the CPU, based on what I could find (h

by diggan 2y ago

I'm no GPU programmer, but seems easy to use even for someone like me. I pulled together a quick demo of using the GPU vs the CPU, based on what I could find (https://gist.github.com/victorb/452a55dbcf59b3cbf84efd8c3097a855 https://gist.github.com/victorb/452a55dbcf59b3cbf84efd8c3097...) which gave these results (after downloading 2.6GB of dependencies of course): Creating 100 random matrices of size 5000x5000 on CPU... Adding matrices using CPU... CPU matrix addition completed in 0.6541 seconds CPU result matrix shape: (5000, 5000) Creating 100 random matrices of size 5000x5000 on GPU... Adding matrices using GPU... GPU matrix addition completed in 0.1480 seconds GPU result matrix shape: (5000, 5000) Definitely worth digging into more, as the API is really simple to use, at least for basic things like these. CUDA programming seems like a big chore without something higher level like this.

deleted 2y ago
[deleted]
rahimnathwani 2y ago
Thank you. I scrolled up and down the article hoping they included a code sample.
- diggan 2y ago
  Yeah, I figured I wasn't alone in doing just that :)
- rahimnathwani 2y ago
  EDIT: Just realized the code doesn't seem to be using the GPU for the addition.
wiredfool 2y ago
Curious what the timing would be if it included the memory transfer time, e.g. matricies = [np.random(...) for _ in range] time_start = time.time() cp_matricies = [cp.array(m) for m in matrices] add_(cp_matricies) sync time_end = time.time()
- hnuser123456 2y ago
  I think it does?: (the comment is in the original source) print("Adding matrices using GPU...") start_time = time.time() gpu_result = add_matrices(gpu_matrices) cp.cuda.get_current_stream().synchronize() # Not 100% sure what this does elapsed_time = time.time() - start_time I was going to ask, any CUDA professionals who want to give a crash course on what us python guys will need to know?
  - apbytes 2y ago
    When you call a cuda method, it is launched asynchronously. That is the function queues it up for execution on gpu and returns. So if you need to wait for an op to finish, you need to `synchronize` as shown above. `get_current_stream` because the queue mentioned above is actually called stream in cuda. If you want to run many independent ops concurrently, you can use several streams. Benchmarking is one use case for synchronize. Another would be if you let's say run two independent ops in different streams and need to combine their results. Btw, if you work with pytorch, when ops are run on gpu, they are launched in background. If you want to bench torch models on gpu, they also provide a sync api.
    - hnuser123456 2y ago
      Thank you kindly!
    - claytonjy 2y ago
      I’ve always thought it was weird GPU stuff in python doesn’t use asyncio, and mostly assumed it was because python-on-GPU predates asyncio. But I was hoping a new lib like this might right that wrong, but it doesn’t. Maybe for interop reasons? Do other languages surface the asynchronous nature of GPUs in language-level async, avoiding silly stuff like synchronize?
      - apbytes 2y ago
        Might have to look at specific lib implementations, but I'd guess that mostly gpu calls from python are actually happening in c++ land. And internally a lib might be using synchronize calls where needed.
ashvardanian 2y ago
CuPy has been available for years and has always worked great. The article is about the next wave of Python-oriented JIT toolchains, that will allow writing actual GPU kernels in a Pythonic-style instead of calling an existing precompiled GEMM implementation in CuPy (like in that snippet) or even JIT-ing CUDA C++ kernels from a Python source, that has also been available for years: https://docs.cupy.dev/en/stable/user_guide/kernel.html#raw-kernels https://docs.cupy.dev/en/stable/user_guide/kernel.html#raw-k...
- almostgotcaught 2y ago
  it's funny - people around here really do not have a clue about the GPU ecosystem even though everyone is always talking about AI: > The article is about the next wave of Python-oriented JIT toolchains the article is content marketing (for whatever) but the actual product has literally has nothing to do with kernels or jitting or anything https://github.com/NVIDIA/cuda-python https://github.com/NVIDIA/cuda-python literally just cython bindings to CUDA runtime and CUB. for once CUDA is aping ROCm: https://github.com/ROCm/hip-python https://github.com/ROCm/hip-python
  - dragonwriter 2y ago
    The mistake you seem to be making is confusing the existing product (which has been available for many years) with the upcoming new features for that product just announced at GTC, which are not addressed at all on the page for the existing product, but are addressed in the article about the GTC announcement.
    - almostgotcaught 2y ago
      > The mistake you seem to be making is confusing the existing product i'm not making any such mistake - i'm just able to actually read and comprehend what i'm reading rather than perform hype: > Over the last year, NVIDIA made CUDA Core, which Jones said is a “Pythonic reimagining of the CUDA runtime to be naturally and natively Python.” so the article is about cuda-core, not whatever you think it's about - so i'm responding directly to what the article is about. > CUDA Core has the execution flow of Python, which is fully in process and leans heavily into JIT compilation. this is bullshit/hype about Python's new JIT which womp womp womp isn't all that great (yet). this has absolutely nothing to do with any other JIT e.g., the cutile kernel driver JIT (which also has absolutely nothing to do with what you think it does).
moffkalast 2y ago
Only 4x speed seems rather low for GPU acceleration, does numpy already use AVX2 or anything SIMD? For comparison, doing something similar with torch on CPU and torch on GPU will get you like 100x speed difference.
- diggan 2y ago
  It's a microbenchmark (if even that), take it with a grain of salt. You'd probably see a bigger difference with bigger/more/more complicated tasks,

by diggan 2y ago

deleted 2y ago

[deleted]

rahimnathwani 2y ago

Thank you. I scrolled up and down the article hoping they included a code sample.

diggan 2y ago
Yeah, I figured I wasn't alone in doing just that :)
rahimnathwani 2y ago
EDIT: Just realized the code doesn't seem to be using the GPU for the addition.

wiredfool 2y ago

Curious what the timing would be if it included the memory transfer time, e.g. matricies = [np.random(...) for _ in range] time_start = time.time() cp_matricies = [cp.array(m) for m in matrices] add_(cp_matricies) sync time_end = time.time()

hnuser123456 2y ago
I think it does?: (the comment is in the original source) print("Adding matrices using GPU...") start_time = time.time() gpu_result = add_matrices(gpu_matrices) cp.cuda.get_current_stream().synchronize() # Not 100% sure what this does elapsed_time = time.time() - start_time I was going to ask, any CUDA professionals who want to give a crash course on what us python guys will need to know?
- apbytes 2y ago
  When you call a cuda method, it is launched asynchronously. That is the function queues it up for execution on gpu and returns. So if you need to wait for an op to finish, you need to `synchronize` as shown above. `get_current_stream` because the queue mentioned above is actually called stream in cuda. If you want to run many independent ops concurrently, you can use several streams. Benchmarking is one use case for synchronize. Another would be if you let's say run two independent ops in different streams and need to combine their results. Btw, if you work with pytorch, when ops are run on gpu, they are launched in background. If you want to bench torch models on gpu, they also provide a sync api.
  - hnuser123456 2y ago
    Thank you kindly!
  - claytonjy 2y ago
    I’ve always thought it was weird GPU stuff in python doesn’t use asyncio, and mostly assumed it was because python-on-GPU predates asyncio. But I was hoping a new lib like this might right that wrong, but it doesn’t. Maybe for interop reasons? Do other languages surface the asynchronous nature of GPUs in language-level async, avoiding silly stuff like synchronize?
    - apbytes 2y ago
      Might have to look at specific lib implementations, but I'd guess that mostly gpu calls from python are actually happening in c++ land. And internally a lib might be using synchronize calls where needed.

ashvardanian 2y ago

CuPy has been available for years and has always worked great. The article is about the next wave of Python-oriented JIT toolchains, that will allow writing actual GPU kernels in a Pythonic-style instead of calling an existing precompiled GEMM implementation in CuPy (like in that snippet) or even JIT-ing CUDA C++ kernels from a Python source, that has also been available for years: https://docs.cupy.dev/en/stable/user_guide/kernel.html#raw-kernels https://docs.cupy.dev/en/stable/user_guide/kernel.html#raw-k...

almostgotcaught 2y ago
it's funny - people around here really do not have a clue about the GPU ecosystem even though everyone is always talking about AI: > The article is about the next wave of Python-oriented JIT toolchains the article is content marketing (for whatever) but the actual product has literally has nothing to do with kernels or jitting or anything https://github.com/NVIDIA/cuda-python https://github.com/NVIDIA/cuda-python literally just cython bindings to CUDA runtime and CUB. for once CUDA is aping ROCm: https://github.com/ROCm/hip-python https://github.com/ROCm/hip-python
- dragonwriter 2y ago
  The mistake you seem to be making is confusing the existing product (which has been available for many years) with the upcoming new features for that product just announced at GTC, which are not addressed at all on the page for the existing product, but are addressed in the article about the GTC announcement.
  - almostgotcaught 2y ago
    > The mistake you seem to be making is confusing the existing product i'm not making any such mistake - i'm just able to actually read and comprehend what i'm reading rather than perform hype: > Over the last year, NVIDIA made CUDA Core, which Jones said is a “Pythonic reimagining of the CUDA runtime to be naturally and natively Python.” so the article is about cuda-core, not whatever you think it's about - so i'm responding directly to what the article is about. > CUDA Core has the execution flow of Python, which is fully in process and leans heavily into JIT compilation. this is bullshit/hype about Python's new JIT which womp womp womp isn't all that great (yet). this has absolutely nothing to do with any other JIT e.g., the cutile kernel driver JIT (which also has absolutely nothing to do with what you think it does).

moffkalast 2y ago

Only 4x speed seems rather low for GPU acceleration, does numpy already use AVX2 or anything SIMD? For comparison, doing something similar with torch on CPU and torch on GPU will get you like 100x speed difference.

diggan 2y ago
It's a microbenchmark (if even that), take it with a grain of salt. You'd probably see a bigger difference with bigger/more/more complicated tasks,