13 ms·
Show HN: Attaching to a virtual GPU over TCP
We developed a tool to trick your computer into thinking it’s attached to a GPU which actually sits across a network. This allows you to switch the number or type of GPUs you’re using with a single command.
- talldayo 2y ago> Access serverless GPUs through a simple CLI to run your existing code on the cloud while being billed precisely for usage Hmm... well I just watched you run nvidia-smi in a Mac terminal, which is a platform it's explicitly not supported on. My instant assumption is that your tool copies my code into a private server instance and communicates back and forth to run the commands. Does this platform expose eGPU capabilities if my host machine supports it? Can I run raster workloads or network it with my own CUDA hardware? The actual way your tool and service connects isn't very clear to me and I assume other developers will be confused too.
- bmodel 2y agoGreat questions! To clarify the demo, we were ssh'd into a linux machine with no GPU. Going into more details for how this works, we intercept communication between the CPU and the GPU so only GPU code and commands are sent across the network to a GPU that we are hosting. This way we are able to virtualize a remote GPU and make your computer think it's directly attached to that GPU. We are not copying your CPU code and running it on our machines. The CPU code runs entirely on your instance (meaning no files need to be copied over or packages installed on the GPU machine). One of the benefits of this approach is that you can easily scale to a more / less powerful GPU without needing to setup a new server.
- billconan 2y agodoes this mean you have a customized/dummy kernel gpu driver? will that cause system instability, say, if the network suddenly dropped?
- bmodel 2y agoWe are not writing any kernel drivers, this runs entirely in userspace (this won't result in a crowdstrike level crash haha). Given that, if the network suddenly dropped then only the process using the GPU would fail.
- ZeroCool2u 2y agoHow do you do that exactly? Are you using eBPF or something else? Also, for my ML workloads the most common bottleneck is GPU VRAM <-> RAM copies. Doesn't this dramatically increase latency? Or is it more like it increases latency on first data transfer, but as long as you dump everything into VRAM all at once at the beginning you're fine? I'd expect this wouldn't play super well with stuff like PyTorch data loaders, but would be curious to hear how you've faired when testing.
- bmodel 2y agoWe intercept api calls and use our own implementation to forward them to a remote machine. No eBPF (which I believe need to run in the kernel). As for latency, we've done a lot of work to minimize that as much as possible. You can see the performance we get running inference on BERT from huggingface here: https://youtu.be/qsOBFQZtsFM?t=64 https://youtu.be/qsOBFQZtsFM?t=64. It's still slower than local (mainly for training workloads) but not by as much as you'd expect. We're aiming to reach near parity in the next few months!
- samstave 2y agoWhen you release a self-host version, what would be really neat would be to see it across HFT focused NICs that have huge TCP buffers... https://www.arista.com/assets/data/pdf/HFT/HFTTradingNetworkPrimer.pdf https://www.arista.com/assets/data/pdf/HFT/HFTTradingNetwork... Basically taking into account the large buffers and super-time-sensitive nature of HFT networking optimizations, I wonder if your TCP<-->GPU might benefit from both the HW and the learnings of NFT stylings?
- ZeroCool2u 2y ago
- billconan 2y agois this a remote nvapi? this is awesome. can it do 3d rendering (vulkan/opengl)
- czbond 2y agoI am not in this "space", but I second the "this is cool to see", more stuff like this needed on HN.
- cpeterson42 2y agoAppreciate the praise!
- bmodel 2y agoThank you! > is this a remote nvapi Essentially yes! Just to be clear, this covers the entire GPU not just the NVAPI (i.e. all of cuda). This functions like you have the physical card directly plugged into the machine. Right now we don't support vulkan or opengl since we're mostly focusing on AI workloads, however we plan to support these in the future (especially if there is interest!)
- billconan 2y agosorry, I didn't mean nvapi, I meant rmapi. I bet you saw this https://github.com/mikex86/LibreCuda https://github.com/mikex86/LibreCuda they implemented the cuda driver by calling into rmapi. My understanding is if there is a remote rmapi, other user mode drivers should work out of the box?
- doctorpangloss 2y agoI don't get it. Why would I start an instance in ECS, to use your GPUs in ECS, when I could start an instance for the GPUs I want in ECS? Separately, why would I want half of Nitro, instead of real Nitro?
- billconan 2y agoit's more transparent to your system, for example, if you have a gui application that needs gpu acceleration on a thin client (Matlab, solidworks, blender), you can do so without setting up ECS. you can develop without any gpu, but suddenly have one when you need to run simulation. this will be way cheaper than AWS. I think essentially this is solving the same problem Ray (https://www.ray.io/ https://www.ray.io/) is solving, but in a more generic way. it potentially can have finer grained gpu sharing, like a half-gpu. I'm very excited about this.
- bmodel 2y agoExactly! The finer grain sharing is one of the key things on our radar right now
- goku-goku 2y agowww.juicelabs.co does all this today, including the GPU sharing and fractionalization.
- ranger_danger 2y agothe free community version has been discontinued, and also doesn't support a linux client with non-CUDA graphics, regardless of the server OS, which is a non-starter for me
- bmodel 2y agoGreat point, there are a few benefits: 1. If you're actively developing and need a GPU then you typically would be paying the entire time the instance is running. Using Thunder means you only pay for the GPU while actively using it. Essentially, if you are running CPU only code you would not be paying for any GPU time. The alterative for this is to manually turn the instance on and off which can be annoying. 2. This allows you to easily scale the type and number of GPUs you're using. For example, say you want to do development on a cheap T4 instance and run a full DL training job on a set of 8 A100. Instead of needing to swap instances and setup everything again, you can just run a command and then start running on the more powerful GPUs.
- steelbrain 2y agoAh this is quite interesting! I had a usecase where I needed a GPU-over-IP but only for transcoding videos. I had a not-so-powerful AMD GPU in my homelab server that somehow kept crashing the kernel any time I tried to encode videos with it and also an NVIDIA RTX 3080 in a gaming machine. So I wrote https://github.com/steelbrain/ffmpeg-over-ip https://github.com/steelbrain/ffmpeg-over-ip and had the server running in the windows machine and the client in the media server (could be plex, emby, jellyfin etc) and it worked flawlessly.
- crishoj 2y agoInteresting. Do you know if your tool supports conversions resulting in multiple files, such as HLS and its myriad of timeslice files?
- steelbrain 2y agoSince it’s sharing the underlying file system and just running ffmpeg remotely, it should support any variation of outputs
- bhaney 2y agoThis is more or less what I was hoping for when I saw the submission title. Was disappointed to see that the submission wasn't actually a useful generic tool but instead a paid cloud service. Of course the real content is in the comments. As an aside, are there any uses for GPU-over-network other than video encoding? The increased latency seems like it would prohibit anything machine learning related or graphics intensive.
- trws 2y agoSome computation tasks can tolerate the latency if they’re written with enough overlap and can keep enough of the data resident, but they usually need more performant networking than this. See older efforts like rcuda for remote cuda over infiniband as an example. It’s not ideal, but sometimes worth it. Usually the win is in taking a multi-GPU app and giving it 16 or 32 of them rather than a single remote GPU though.
- cpeterson42 2y agoGiven the interest here we decided to open up T4 instances for free. Would love for y'all to try it and let us know your thoughts!
- dheera 2y agoWhat is your A100 and H100 pricing?
- cpeterson42 2y agoWe are super early stage and don't have A100s or H100s live yet. Exact pricing TBD but expect it to be low. If you want to use them today, reach out directly and we can set them up :)
- tptacek 2y agoThis is neat. Were you able to get MIG or vGPUs working with it?
- bmodel 2y agoWe haven't tested with MIG or vGPU, but I think it would work since it's essentially physically partitioning the GPU. One of our main goals for the near future is to allow GPU sharing. This would be better than MIG or vGPU since we'd allow users to use the entire GPU memory instead of restricting them to a fraction.
- tptacek 2y agoWe had a hell of a time dealing with the licensing issues and ultimately just gave up and give people whole GPUs. What are you doing to reset the GPU to clean state after a run? It's surprisingly complicated to do this securely (we're writing up a back-to-back sequence of audits we did with Atredis and Tetrel; should be publishing in a month or two).
- kawsper 2y agoCool idea, nice product page! Does anyone know if this is possible with USB? I have a Davinci Resolve license USB-dongle I'd like to not plugging into my laptop.
- kevmo314 2y agoYou can do that with USB/IP: https://usbip.sourceforge.net/ https://usbip.sourceforge.net/
- orsorna 2y agoSo what exactly is the pricing model? Do I need a quote? Because otherwise I don't see how to determine it without creating an account which is needlessly gatekeeping.
- bmodel 2y agoWe're still in our beta so it's entirely free for now (we can't promise a bug-free experience)! You have to make an account but it won't require payment details. Down the line we want to move to a pay-as-you-go model.
- Cieric 2y agoThis is interesting, but I'm more interested in self-hosting. I already have a lot of GPUs (some running some not.) Does this have a self-hosting option so I can use the GPUs I already have?
- cpeterson42 2y agoWe don't support self hosting yet but the same technology should work well here. Many of the same benefits apply in a self-hosted setting, namely efficient workload scheduling, GPU-sharing, and ease-of-use. Definitely open to this possibility in the future!
- goku-goku 2y ago[flagged]
- Cieric 2y agoThanks, but I've already evaluated JuiceLabs and it does not handle what I need it to. Plus with the whole project going commercial and the community edition being neglected I no longer have any interest in trying to support the project either.
- smcleod 2y agoIs it open source? The website seems to be one of those closed "join a wait list" sites
- covi 2y agoIf you want to use your own GPUs or cloud accounts but with a great dev experience, see SkyPilot.
- ellis0n 2y agoYou can rent out your GPUs in the cloud with services like Akash Network and rent GPUs at thundercompute.com.. manager's path, almost like self-hosting :)
- donnygreenberg 2y agoIf you want a PyTorch-like experience on your own GPUs (either static or cloud), see https://github.com/run-house/runhouse https://github.com/run-house/runhouse
- cpeterson42 2y agoWe created a discord for the latest updates, bug reports, feature suggestions, and memes. We will try to respond to any issues and suggestions as quickly as we can! Feel free to join here: https://discord.gg/nwuETS9jJK https://discord.gg/nwuETS9jJK
- throwaway888abc 2y agoDoes it work for gaming on windows ? or even linux ?
- cpeterson42 2y agoIn theory yes. In practice, however, latency between the CPU and remote GPU makes this impractical
- boxerbk 2y agoYou could use a remote streaming protocol, like Parsec, for that. You'd need your own cloud account and connect directly to a GPU-enabled cloud machine. Otherwise, it would work to let you game.
- rubatuga 2y agoWhat ML packages do you support? In the comments below it says you do not support Vulkan or OpenGL. Does this support AMD GPUs as well?
- bmodel 2y agoWe have tested this with pytorch and huggingface and it is mostly stable (we know there are issues with pycuda and jax). In theory this should work with any libraries, however we're still actively developing this so bugs will show up
- the_reader 2y agoWould be possible to mix it with Blender?
- bmodel 2y agoAt the moment out tech is linux-only so it would not work with Blender. Down the line, we could see this being used for batched render jobs (i.e. to replace a render farm).
- comex 2y agoBlender can run on Linux…
- bmodel 2y agoOh nice, I didn't know that! In that case it might work, you could try running `tnr run ./blender` (replace the ./blender with how you'd launch blender from the CLI) to see what happens. We haven't tested it so I can't make promises about performance or stability :)
- chmod775 2y agoDisclaimer: I only have a passing familiarity with Blender, so I might be wrong on some counts. I think you'd want to run the blender GUI locally and only call out to a headless rendering server ("render farm") that uses your service under the hood to get the actual render. This separation is already something blender supports, and you could for instance use Blender on Windows despite your render farm using Linux servers. Cloud rendering is adjacent to what you're offering, and it should be trivial for you to expand into that space by just figuring out the setup and preparing a guide for users wishing to do that with your service.
- teaearlgraycold 2y agoThis could be perfect for us. We need very limited bandwidth but have high compute needs.
- bkitano19 2y agothis is nuts
- cpeterson42 2y agoWe think so too, big things coming :)
- goku-goku 2y agowww.juicelabs.co
- dishsoap 2y agoFor anyone curious about how this actually works, it looks like a library is injected into your process to hook these functions [1] in order to forward them to the service. [1] https://pastebin.com/raw/kCYmXr5A https://pastebin.com/raw/kCYmXr5A
- almostgotcaught 2y agoHow did you figure out these were hooked? I'm assuming some flag that tells ld/ldd to tell you when some symbol is rebound? Also I thought a symbol has to be a weak symbol to be rebound and assuming nvidia doesn't expose weak symbols (why would they) the implication is that their thing is basically LD_PRELOADed?
- yarpen_z 2y agoYes. While I don't know what they do internally, API remoting has been used for GPUs since at least rCUDA - that's over 10 years ago. LD_PRELOAD trick allows you to intercept and virtualize calls to the CUDA runtime.
- the8472 2y agoAh, I assumed/hoped they had some magic that would manage to forward a whole PCIe device.
- Zambyte 2y agoReminds me of Plan9 :)
- K0IN 2y agocan you elaborate a bit on why? (noob here)
- Zambyte 2y agoIn Plan 9 everything is a file (for real this time). Remote file systems are accessible through the 9P protocol (still used in modern systems! I know it's used in QEMU and WSL). Every process has its own view of the filesystem called a namespace. The implication of these three features is that remote resources can be transparently accessed as local resources by applications.
- radarsat1 2y agoI'm confused, if this operates at the CPU/GPU boundary doesn't it create a massive I/O bottleneck for any dataset that doesn't fit into VRAM? I'm probably misunderstanding how it works but if it intercepts GPU i/o then it must stream your entire dataset on every epoch to a remote machine, which sounds wasteful, probably I'm not getting this right.
- deleted 2y ago[deleted]
- bmodel 2y agoThat understanding of the system is correct. To make it practical we've implemented a bunch of optimizations to minimize I/O cost. You can see how it performs on inference with BERT here: https://youtu.be/qsOBFQZtsFM?t=69 https://youtu.be/qsOBFQZtsFM?t=69. The overheads are larger for training compared to inference, and we are implementing more optimizations to approach native performance.
- radarsat1 2y agoAah ok thanks, that was my basic misunderstanding, my mind just jumped straight to my current training needs but for inference it makes a lot of sense. Thanks for the clarification.
- semitones 2y ago> to approach native performance. The same way one "approaches the sun" when they take the stairs?
- the8472 2y agoI guess there's non-negligible optimization potential, e.g. by doing hash-based caching. If the same data gets uploaded twice they can have the blob already sitting somewhere closer to the machine.
- ruined 2y agoyes. my building has stairs and i find them useful because usually i don't need to go to the sun
- winecamera 2y agoI saw that in the tnr CLI, there are hints of an option to self-host a GPU. Is this going to be a released feature?
- cpeterson42 2y agoWe don't support self-hosting yet but are considering adding it in the future. We're a small team working as hard as we can :) Curious where you see this in the CLI, may be an oversight on our part. If you can join the Discord and point us to this bug we would really appreciate it!
- deleted 2y ago[deleted]
- test20240809 2y agopocl (Portable Computing Language) [1] provides a remote backend [2] that allows for serialization and forwarding of OpenCL commands over a network. Another solution is qCUDA [3] which is more specialized towards CUDA. In addition to these solutions, various virtualization solutions today provide some sort of serialization mechanism for GPU commands, so they can be transferred to another host (or process). [4] One example is the QEMU-based Android Emulator. It is using special translator libraries and a "QEMU Pipe" to efficiently communicate GPU commands from the virtualized Android OS to the host OS [5]. The new Cuttlefish Android emulator [6] uses Gallium3D for transport and the virglrenderer library [7]. I'd expect that the current virtio-gpu implementation in QEMU [8] might make this job even easier, because it includes the Android's gfxstream [9] (formerly called "Vulkan Cereal") that should already support communication over network sockets out of the box. [1] https://github.com/pocl/pocl https://github.com/pocl/pocl [2] https://portablecl.org/docs/html/remote.html https://portablecl.org/docs/html/remote.html [3] https://github.com/coldfunction/qCUDA https://github.com/coldfunction/qCUDA [4] https://www.linaro.org/blog/a-closer-look-at-virtio-and-gpu-virtualisation/ https://www.linaro.org/blog/a-closer-look-at-virtio-and-gpu-... [5] https://android.googlesource.com/platform/external/qemu/+/emu-master-dev/android/android-emugl/DESIGN https://android.googlesource.com/platform/external/qemu/+/em... [6] https://source.android.com/docs/devices/cuttlefish/gpu https://source.android.com/docs/devices/cuttlefish/gpu [7] https://cs.android.com/android/platform/superproject/main/+/main:device/generic/opengl-transport/host/libs/virglrenderer/README.md https://cs.android.com/android/platform/superproject/main/+/... [8] https://www.qemu.org/docs/master/system/devices/virtio-gpu.html https://www.qemu.org/docs/master/system/devices/virtio-gpu.h... [9] https://android.googlesource.com/platform/hardware/google/gfxstream/ https://android.googlesource.com/platform/hardware/google/gf...
- fpoling 2y agoZscaler uses a similar approach in their remote browser. WebGL in the local browser exposed as a GPU to a Chromium instance in the cloud.
- mmsc 2y agoWhat's it like to actually use this for any meaningful throughput? Can this be used for hash cracking? Every time I think about virtual GPUs over a network, I think about botnets. Specifically from https://www.hpcwire.com/2012/12/06/gpu_monster_shreds_password_hashes/ https://www.hpcwire.com/2012/12/06/gpu_monster_shreds_passwo... "Gosney first had to convince Mosix co-creator Professor Amnon Barak that he was not going to “turn the world into a giant botnet.”"
- cpeterson42 2y agoThis is definitely an interesting thought experiment, however in practice our system is closer to AWS than a botnet, as the GPUs are not distributed. This technology does lend itself to some interesting applications with creating very flexible clusters within data centers that we are exploring.
- m3kw9 2y agoSo won’t that make the network the prohibitive bottle neck? Your memory bandwidth is 1gbps max
- teaearlgraycold 2y agoCloud hosts will offer 10Gb/s. Anyway, in my experience with training LoRAs and running DINOv2 inference you don’t need much bandwidth. We are usually sitting at around 10-30MB/s per GPU.
- userbinator 2y agoIt's impressive that this is even possible, but I wonder what happens if the network connection goes down or is anything but 100% stable? In my experience drivers react badly to even a local GPU that isn't behaving.
- tamimio 2y agoI’m more interested in using tools like hashcat, any benchmark on these? As the docs link returns error.
- bmodel 2y agoWe haven't tested it with hashcat yet but plan on doing so. If you get to it before us please let us know how it works!
- deleted 2y ago[deleted]
- delijati 2y agoEven a directly attached eGPU via thunderbold 4 was after some time too slow for machine learning aka training. As i work now fully remote i just have a beefy midi tower. Some context about eGPU [1]. But hey i'm happy to be proofed wrong ;) [1] https://news.ycombinator.com/item?id=38890182#38905888 https://news.ycombinator.com/item?id=38890182#38905888
- deleted 2y ago[deleted]
- xyst 2y agoExciting. But would definitely like to see a self hosted option.
- ellis0n 2y agoIn 2008, I had a powerful server with XEON CPU, but the motherboard had no slots for a graphics card. I also had a computer with a powerful graphics card but a weak Core 2 Duo. I had the idea of passing the graphics card over the network using Linux drivers. This concept has now been realized in this project. Good job!
- somat 2y agoWhat makes me sad is that the original sgi engineers who developed glx were very careful to use x11 mechanisms for the gpu transport, so it was fairly trivial to send the gl stream over the network to render on your graphics card. "run on the supercomputer down the hall, render on your workstation". More recent driver development has not shown such care and this is usually no longer possible. I am not sure how useful it was in reality(usually if you had a nice graphics card you also had a nice cpu) but I had fun playing around with it. There was something fascinating about getting accelerated graphics on a program running in the machine room. I was able to get glquake running like this once.