4 ms·

I've been toying with SD for a while, and I do want to make a nice and clean business out of it. It's more of a side-projecty thing so to speak. Our "cluster"

I've been toying with SD for a while, and I do want to make a nice and clean business out of it. It's more of a side-projecty thing so to speak. Our "cluster" is running on a ASUS ROG 2080Ti external GPU in the razer core-x housing, and that actually works just fine in my flat. We went through several iterations of how this could work at scale. The initial premise was basically the google homepage, but for images. That's when we realised that scaling this to serve the planet was going go be a hell of a lot more work. But not really, conceptualising the concurrent compute requirements as well as the ever-changing landscape and pace of innovation in this absolutely necessary. The quick fix is to use a message queue (we're using Bull) and make everything asynchronous. So essentially, we solved the scaling factor using just one GPU. You'll get your requested image, but it's in a queue, we'll let you know when it's done. With that compute model in place, we can just add more GPUs, and tickets will take less time to serve if the scale engineering is proper. I'm no expert on GPU/Machine learning/GAN stuff but Stable Diffusion actually prompted me to imagine how to build and scale such a service, and I did so. It is not live yet, but when it does become so the name reserved is dreamcreator dot ai, and I can't say when it will be animated. Hopefully this year.

throwaway675309 4y ago
A ticketing scheduler system is how 99% of systems that require long running CPU/GPU intensive that cannot be run in parallel are implemented. It's how I built up my stable diffusion discord bot which is backed by a single RTX 2060. I'm glad you have this working but I wouldn't exactly call this "solving the scaling problem", you're just running it in a blocking "serial fashion". With enough concurrent users it could still take somebody until the heat death of the universe for their image to finally be generated.
- karim79 4y ago
  Agreed. The scaling model/queueing system was implemented as a POC which can be scaled by plugging in more cards/hosts. I hope we can find the time to animate this soon enough.
DrScientist 4y ago
Queues are everywhere - at all levels - in the end a single transistor is either on or off - doing one thing at once. Your queue decouples demand from supply - though you now have another problem - if demand far exceeds supply will your queue overflow? In that scenario you might need to push the queue back to the requester - ie refuse the job and tell them to resubmit later.

throwaway675309 4y ago

A ticketing scheduler system is how 99% of systems that require long running CPU/GPU intensive that cannot be run in parallel are implemented. It's how I built up my stable diffusion discord bot which is backed by a single RTX 2060. I'm glad you have this working but I wouldn't exactly call this "solving the scaling problem", you're just running it in a blocking "serial fashion". With enough concurrent users it could still take somebody until the heat death of the universe for their image to finally be generated.

karim79 4y ago
Agreed. The scaling model/queueing system was implemented as a POC which can be scaled by plugging in more cards/hosts. I hope we can find the time to animate this soon enough.

DrScientist 4y ago

Queues are everywhere - at all levels - in the end a single transistor is either on or off - doing one thing at once. Your queue decouples demand from supply - though you now have another problem - if demand far exceeds supply will your queue overflow? In that scenario you might need to push the queue back to the requester - ie refuse the job and tell them to resubmit later.