3 ms·
Way too slow on CPU unfortunately But this does make me wonder if there's any way to allow a graphics card to use regular RAM in a fast way? AFAIK built-in GPU
by Aardwolf 4y ago
Way too slow on CPU unfortunately
But this does make me wonder if there's any way to allow a graphics card to use regular RAM in a fast way? AFAIK built-in GPU's inside CPU's can but those GPU's are not powerful enough
- easytiger 4y agoI thought cuda had a unified memory system? Maybe I misunderstood
- Cu3PO42 4y agoUnified memory exists, but it's not a magic bullet. If a page is accessed that doesn't reside on device memory (i.e. on the GPU), a memcpy is issued to fetch the page from main RAM. While the programming model is nicer, it doesn't fundamentally change the fact that you need to constantly swap data out to main RAM and while not as bad as loading it from the SSD or HDD, that's still quite slow. Integrated GPUs that use a portion of system memory are an exception to this and do not require memcpys when using unified memory. However, I'm not aware of any powerful iGPUs from Nvidia these days.
- easytiger 4y agoSure. Makes sense. So I guess for discrete GPUs the unified memory stuff provides a universal address space but merely abstracts the copying/streaming of the data. There does seem to be a zero copy concept as well and I've certainly used direct memory access over pcie before on other proprietary devices. https://docs.nvidia.com/cuda/cuda-c-best-practices-guide/index.html#zero-copy https://docs.nvidia.com/cuda/cuda-c-best-practices-guide/ind...
- julienfr112 4y agoSlow, but is it still practical, like taking minutes to generate few words ca still be useful for testing or on certain low usage use-cases ?
- yarandex 4y agoAssuming running on CPU is memory-bandwidth limited, not CPU-limited, it should take about 200GB / (50GB/sec) = 4 seconds per character. Not too bad.
- lostmsu 4y agoThat's per token. And you can generate quite a few per pass.