3 ms·
I wish NVidia would focus on integrating their software (like this DMA support) into more widely adopted frameworks like Tensorflow, Pandas, or Pytorch. Like T
by choppaface 7y ago
I wish NVidia would focus on integrating their software (like this DMA support) into more widely adopted frameworks like Tensorflow, Pandas, or Pytorch. Like TensorRT or nvidia-docker, they always have to release their own library that either breaks in weird ways or has utterly awful support (TensorRT).
Note that the 16-Volta contraption is basically only for use with NVidia’s own software. The two Xeons it has only have 80 PCI-e lanes, and 16 Voltas will consume 256 lanes, which is why each Volta gets its own flash. (Careful to shuffle your data before putting it on disk!). For that money, you’re better off building your own rack with a balance for your real application.
One way to avoid the PCI-e bottleneck for vision is to use NVidia’s own JPEG decompressor. Then only the file bytes have to cross the link, and the image tensor (which is probably 10x the memory) never leaves GPU RAM.
- ericd 7y agoWhoa, awesome, thanks for the tip about nvJPEG. Do you know if there are any good tools for data augmentation (rotation/contrast/flip/zoom/blur/crop/letterbox) for those JPEGs once they're on-gpu tensors? edit: Looks like DALI might handle this. Now to see how it integrates with pytorch...
- IfOnlyYouKnew 7y agoAll those operations are fairly standard vector math once you're working with bitmaps.
- ericd 7y agoYep, I know, but I’d rather not reimplement them if I don’t have to.
- baruch 7y agoThe lane issue is why there is this push for GPUDirect, the bulk memory transfer goes through the nic that is closest to the GPU and straight to the GPU DRAM without the need to cross through the CPUs limited lanes. In my experience, this works rather nicely.
- Nokinside 7y agoPCI-E bottleneck is why Nvidia bought Mellanox Technologies this year. It's likely that they replace PCI-E with InfiniBand derivative in the future.
- mbell 7y agoHow would Infiniband fix this?
- zetazzed 7y ago(NVIDIAn here but speaking for myself) - We still have a ways to go, but we're working on it! Really trying to integrate with open data/ML standards, though sometimes an existing codebase doesn't exactly match the GPU approach, so the best thing we can do is to match their APIs... TensorRT is now integrated with Tensorflow: https://github.com/tensorflow/tensorrt https://github.com/tensorflow/tensorrt and it speaks ONNX, which is PyTorch's and MXNet's inference format. cuDF (open source) supports the Pandas API on GPUs, using the Arrow standard for interop https://github.com/rapidsai/cudf https://github.com/rapidsai/cudf and cuML (also open source) matches the scikit-learn interface: https://github.com/rapidsai/cuml https://github.com/rapidsai/cuml PyTorch and a bunch of other DL libraries can interop with cuDF/cuML (as in, do your ETL in those libs and get zero-copy, on-GPU transfer to the DL framework) via the dlpack standard. This is a general standard from DMLC that any similar ML library can adopt. Also, if you're doing nvJpeg decompression on GPU, also check out Dali, which implements super-fast data loading and augmentation steps all on GPU. For this one, there wasn't an existing standard, so we had to build our own approach, but it integrates with PyTorch, TF, and MXNet pipelines nicely: https://docs.nvidia.com/deeplearning/sdk/dali-developer-guide/docs/index.html https://docs.nvidia.com/deeplearning/sdk/dali-developer-guid... Hopefully it becomes like the cuDNN of data prep - integrated into all the frameworks well enough that you don't even have to think about it.