4 ms·

This is more or less what I was hoping for when I saw the submission title. Was disappointed to see that the submission wasn't actually a useful generic tool bu

by bhaney 2y ago

This is more or less what I was hoping for when I saw the submission title. Was disappointed to see that the submission wasn't actually a useful generic tool but instead a paid cloud service. Of course the real content is in the comments. As an aside, are there any uses for GPU-over-network other than video encoding? The increased latency seems like it would prohibit anything machine learning related or graphics intensive.

trws 2y ago
Some computation tasks can tolerate the latency if they’re written with enough overlap and can keep enough of the data resident, but they usually need more performant networking than this. See older efforts like rcuda for remote cuda over infiniband as an example. It’s not ideal, but sometimes worth it. Usually the win is in taking a multi-GPU app and giving it 16 or 32 of them rather than a single remote GPU though.
tommsy64 2y ago
There is a GPU-over-network software called Juice [1]. I've used it on AWS for running CPU-intensive workloads that also happen to need some GPU without needing to use a huge GPU instance. I was able to use a small GPU instance, which had just 4 CPU cores, and stream its GPU to one with 128 CPU cores. I found Juice to work decently for graphical applications too (e.g., games, CAD software). Latency was about what you'd expect for video encode + decode + network: 5-20ms on a LAN if I recall correctly. [1] - https://github.com/Juice-Labs/Juice-Labs https://github.com/Juice-Labs/Juice-Labs
Fnoord 2y ago
I mean, anything you use a GPU/TPU for could benefit. IPMI and such could use it. Like, for example, Proxmox could use it. Machine learning tasks (like Frigate) and hashcat could also use such. All in theory, of course. Many tasks use VNC right now, or SPICE. The ability to extract your GPU in the Unix way over TCP/IP is powerful. Though Node.js would not be the way I'd want such to go.
lostmsu 2y ago
How do you use it for video encoding/decoding? Won't the uncompressed video (input for encoding or output of decoding) be too large to transmit over network practically?
- bhaney 2y ago
  Well, the ffmpeg-over-ip tool in the GP does it by just not sending uncompressed video. It's more of an ffmpeg server where the server is implicitly expected to have access to a GPU that the client doesn't have, and only compressed video is being sent back and forth in the form of video streams that would normally be the input and output of ffmpeg. It's not a generic GPU server that tries to push a whole PCI bus over the network, which I personally think is a bit of a fool's errand and doomed to never be particularly useful to existing generic workloads. It would work if you very carefully redesign the workload to not take advantage of a GPU's typical high bandwidth and low latency, but if you have to do that then what's the point of trying to abstract over the device layer? Better to work at a higher level of abstraction where you can optimize for your particular application, rather than a lower level that you can't possibly implement well and then have to completely redo the higher levels anyway to work with it.
  - lostmsu 2y ago
    Ah, you mean transcoding scenarios. Like it can't encode my screen capture.
johnisgood 2y ago
I am increasingly growing tired of these "cloud" services, paid or not. :/
- adwn 2y ago
  Well, feel free to spend your own time on writing such a tool and releasing it as Open Source. That would be a really cool project! Until then, don't complain that others aren't willing to donate a significant amount of their work to the public.
  - deleted 2y ago
    [deleted]
  - deleted 2y ago
    [deleted]
  - rowanG077 2y ago
    There is a vast gap between walled garden cloud service rent seeking and giving away software as open source. In the olden days you could buy software licenses to run it wherever you wanted.
- mhuffman 2y ago
  I agree. Paying by the month for the rest of your life or they cut you off is not something I am a fan of. I feel sorry for people too young to remember that you could actually buy an app, get free bug updates and get a discount if they made some big changes on a new version that you might (or might not) want. But it was up to you when and where you ran it and it was yours forever. I have heard the arguments for why people enjoy this monthly subscription model, but my counter argument is that people did just fine before without them, so what is so different now? And I mean, in general, not that you need to use a GPU for 1 hour but don't want to buy one. I mean, for example, how Adobe products run on your computer but you rent them forever.

by bhaney 2y ago

trws 2y ago

Some computation tasks can tolerate the latency if they’re written with enough overlap and can keep enough of the data resident, but they usually need more performant networking than this. See older efforts like rcuda for remote cuda over infiniband as an example. It’s not ideal, but sometimes worth it. Usually the win is in taking a multi-GPU app and giving it 16 or 32 of them rather than a single remote GPU though.

tommsy64 2y ago

There is a GPU-over-network software called Juice [1]. I've used it on AWS for running CPU-intensive workloads that also happen to need some GPU without needing to use a huge GPU instance. I was able to use a small GPU instance, which had just 4 CPU cores, and stream its GPU to one with 128 CPU cores. I found Juice to work decently for graphical applications too (e.g., games, CAD software). Latency was about what you'd expect for video encode + decode + network: 5-20ms on a LAN if I recall correctly. [1] - https://github.com/Juice-Labs/Juice-Labs https://github.com/Juice-Labs/Juice-Labs

Fnoord 2y ago

I mean, anything you use a GPU/TPU for could benefit. IPMI and such could use it. Like, for example, Proxmox could use it. Machine learning tasks (like Frigate) and hashcat could also use such. All in theory, of course. Many tasks use VNC right now, or SPICE. The ability to extract your GPU in the Unix way over TCP/IP is powerful. Though Node.js would not be the way I'd want such to go.

lostmsu 2y ago

How do you use it for video encoding/decoding? Won't the uncompressed video (input for encoding or output of decoding) be too large to transmit over network practically?

bhaney 2y ago
Well, the ffmpeg-over-ip tool in the GP does it by just not sending uncompressed video. It's more of an ffmpeg server where the server is implicitly expected to have access to a GPU that the client doesn't have, and only compressed video is being sent back and forth in the form of video streams that would normally be the input and output of ffmpeg. It's not a generic GPU server that tries to push a whole PCI bus over the network, which I personally think is a bit of a fool's errand and doomed to never be particularly useful to existing generic workloads. It would work if you very carefully redesign the workload to not take advantage of a GPU's typical high bandwidth and low latency, but if you have to do that then what's the point of trying to abstract over the device layer? Better to work at a higher level of abstraction where you can optimize for your particular application, rather than a lower level that you can't possibly implement well and then have to completely redo the higher levels anyway to work with it.
- lostmsu 2y ago
  Ah, you mean transcoding scenarios. Like it can't encode my screen capture.

johnisgood 2y ago

I am increasingly growing tired of these "cloud" services, paid or not. :/

adwn 2y ago
Well, feel free to spend your own time on writing such a tool and releasing it as Open Source. That would be a really cool project! Until then, don't complain that others aren't willing to donate a significant amount of their work to the public.
- deleted 2y ago
  [deleted]
- deleted 2y ago
  [deleted]
- rowanG077 2y ago
  There is a vast gap between walled garden cloud service rent seeking and giving away software as open source. In the olden days you could buy software licenses to run it wherever you wanted.
mhuffman 2y ago
I agree. Paying by the month for the rest of your life or they cut you off is not something I am a fan of. I feel sorry for people too young to remember that you could actually buy an app, get free bug updates and get a discount if they made some big changes on a new version that you might (or might not) want. But it was up to you when and where you ran it and it was yours forever. I have heard the arguments for why people enjoy this monthly subscription model, but my counter argument is that people did just fine before without them, so what is so different now? And I mean, in general, not that you need to use a GPU for 1 hour but don't want to buy one. I mean, for example, how Adobe products run on your computer but you rent them forever.