14 ms·
StreamDiffusion: A pipeline-level solution for real-time interactive generation
- _lvbh 3y agoThis feels unreal. It feels like a decade passed within a year.
- amelius 3y agoThese softwares develop faster than I can apt-get install them.
- throwup238 3y agoNot even ArchLinux’s rolling community repositories can keep up. I’ve had to git clone everything like a package manager-less serf. Where even is my filesystem?!
- jimmyl02 3y agothe entire open source AI space feels like this right now. basically every day there is some new advancement that either makes something deemed impossible achievable and it's actually really hard to keep up with all the changes.
- legel 3y ago100% agreed. I've been developing deep neural networks for over 10 years and this is just surreal. On the bright side, one source of "sanity" that I'm finding is to review a collection of daily "hot" publications in AI/ML curated here: https://huggingface.co/papers https://huggingface.co/papers
- kwerk 3y agoThanks for sharing.
- kossTKR 3y agoTrue, the fact we now also have LLM models at 2.7B parameters that are "actually coherent and useful for niche work" like PHI-2 on my Macbook Air running super fast at 25 t/s or 12 t/s for Mistral_0.2 as an _somewhat_ knowledgable assistant / coder - is crazy! Coherent models required dozens of billions of parameters just a year ago and something like Mistral was closer to 70B and reserved for server farms or crazy setups. Locallama on Reddit has outdated info in from just a few months ago.
- gigel82 3y agoReminds me of an incremental game ( https://www.reddit.com/r/incremental_games/ https://www.reddit.com/r/incremental_games/ ) - BTW, don't start playing one of those or you'll ruin your holidays... :)
- knlam 3y agoNow as a frontend developer I understand how folks complain the frontend landscape change so fast that it is impossible to keep up
- mclightning 3y agoas a frontend developer, you understood how folks complain about the changing frontend landscape, through....changes in ML, a completely different landscape to that of your own? Are you an LLM?
- nazka 3y agoAt least this is innovation and always moving forward. For the frontend space sometimes it’s step backs, reinventing the wheel, or just because a new tool looks shinier…
- mattigames 3y agoI can't wait until it can do my job and then I will just run it on my PC and connect it to slack so my employeer will receive similar results of when I did it manually and I will be payed without spending any time actually working, I will be able to focus on my hobbies for once. This is how this all will play out in the end right?
- sroussey 3y agoYour employer can just replace you then and save the money
- waitthi 3y ago[dead]
- godelski 3y ago> I will be payed without spending any time actually working, > This is how this all will play out in the end right? Somebody is gonna tell him right? I don't want to be the one to crush such innocence.
- mattigames 3y agoIt was sarcasm.
- godelski 3y agoAhh sorry, I missed that. My bad. Have run into too many people that legitimately believe this :/
- ChatGTP 3y agoYou should try using AI to gauge sentiment.
- holoduke 3y agoIf you are the first. Yes dor a time. Make sure you duplicate your work among 500 other jobs and become a millionaire. Because it will last not so long when everyone finds out
- Roritharr 3y agoThis is the only thing that has me curious if that's how the progress curve feels like in the opening act of the Singularity.
- PartiallyTyped 3y agoWe are on the lower bottom of the S curve.
- ChatGTP 3y agoGenerating images on high end hardware and then...?
- Flux159 3y agoArxiv paper here https://arxiv.org/abs/2312.12491 https://arxiv.org/abs/2312.12491 I think that it's possible to get faster than their default timings for a 4090 (I have been able to get 10fps without optimizations with SDXL Turbo and 1 iteration step), but their other improvements like using a Stochastic Similarity Filter to prevent unnecessary generations are good for getting fast results w/out having to pin your GPU at 100% all the time.
- ilaksh 3y agoHow does the demo with the girl moving in and out of frame work? Is it ControlNet?
- woleium 3y agoIts video input. from tfa: Stochastic Similarity Filter reduces processing during video input by minimizing conversion operations when there is little change from the previous frame, thereby alleviating GPU processing load, as shown by the red frame in the above GIF
- Flux159 3y agoI think it’s just img2img with a prompt & rcfg scale and no controlnet since theres a GitHub issue about adding controlnet support open at the moment.
- ec109685 3y agoSo left is the source image and right is the resultant image?
- washadjeffmad 3y agoyes. compare to animatediff.
- brcmthrowaway 3y agoWhat is the fps on Apple Silicon?
- gaogao 3y agoAt least an 1/8 or so, but yeah, getting it running on Apple at at least 24fps would be huge. Some degree of interpolation might do it. You maybe could get away with 12fps esp. with an anime aesthetic since that's basically animating on 2's.
- washadjeffmad 3y ago0 because there's no MPS support. However, a Studio with an M1 Max 64GB is ~13x slower at generative AI with SD1.5 and SDXL than an RTX 4090 24GB at the same cost (~$1,800, refurb) right now.
- Terretta 3y ago> 0 because there's no MPS support. ... Studio with an M1 Max 64GB is ~13x slower at generative AI with SD1.5 and SDXL than an RTX 4090 24GB at the same cost (~$1,800, refurb) Does the 4090 have a computer attached to it? It seems like with no computer, the speed would also be 0.
- teaearlgraycold 3y agoIf we’re being snarky “Apple Silicon” won’t work without a motherboard and power supply either.
- givinguflac 3y agoI get what you’re saying but I don’t think there was snark. Just the fact that a 4090 without a computer attached won’t work. It’s not like you can buy apple silicon without a Mac attached.
- jbverschoor 3y ago
- badloginagain 3y agoYo I just heard about MidJourney this year. And this appears to be a local runtime stable diffusion streaming library? Bruh.
- Keyframe 3y agoSingularity is real, but it's people. Amazing fast-paced progress.
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- modeless 3y agoDoes 100fps mean I can provide a new input every 10 ms and get a new output every 10ms? Or do inputs need to be batched together to get that average throughput?
- stale2002 3y agoI haven't tried it, but just taking an educated guess, which is that I don't think batching should be required. The slow part for models is loading the model up. But once the model is up, you can send it whatever input that you want. Parsing and sending the image data just doesn't pass my gut check as to what would be the bottleneck here.
- modeless 3y agoThat's not the issue, the issue is GPU utilization. Batching enables higher utilization and higher images per second throughput, but doesn't improve latency.
- Kubuxu 3y agoIn essence batching allows for more efficient usage of memory bandwidth. Without batching for every generation you need to transfer the whole model from GPU memory to GPU core once for every image which sets an upper bound on speed. With batching the bottlenecks start showing up elsewhere. For SD1.5 4090 is able to do ~17it/s without batching and ~90-100it/s with batching. Although these numbers might be old at this point, I looked at it ~3mo ago.
- godelski 3y agoEveryone does warmup before you measure. But measuring isn't always done right because we actually measure the GPU time only but some people naively use CPU time which is problematic because the process is asynchrenous. They have a few timing scripts though and I'm away from my GPU. There are some interesting things but they look like they know how to time. But it can also get confusing because is it considering batches or not. Some works do batch some do single. Only problem is when it isn't communicated correctly or left ambiguous. Their paper is ambiguous unfortunately. Abstract, intro, and conclusion suggests single image by motivating with sequential generation (specifically mentioning metaverse). Experiment section says > We note that we evaluate the throughput mainly via the average inference time per image through processing 100 images. That implies batch along with their name Stream Batch... Looking at the code I'm a bit confused. I'm away from my GPU so can't run. Maybe someone can let me know? This block[0] measures correctly but is using a downloaded image? Then just opens the image in the preprocess? (multi looks identical) This block[1] is using CPU? But running CPU. (there's another like this) So I'm quite a bit confused tbh. [0] https://github.com/cumulo-autumn/StreamDiffusion/blob/03e2a7ff7a82f7967a211419c231704db3cd5aa2/examples/benchmark/single.py#L115-L128 https://github.com/cumulo-autumn/StreamDiffusion/blob/03e2a7... [1] https://github.com/cumulo-autumn/StreamDiffusion/blob/03e2a7ff7a82f7967a211419c231704db3cd5aa2/examples/optimal-performance/single.py#L54-L62 https://github.com/cumulo-autumn/StreamDiffusion/blob/03e2a7...
- kristopolous 3y agoThis more or less just worked as documented. Most of these demos tend to blow up and give really wonky deep errors. Good job. Give it a try. Look into the server.py of realtime-txt2img to change the model if you want to generate something other than anime. Pointing it to say https://huggingface.co/runwayml/stable-diffusion-v1-5 https://huggingface.co/runwayml/stable-diffusion-v1-5 works fine. The results are genuinely fast. Not great, but fast. If you change to the SDXL via LCM-LoRA https://huggingface.co/latent-consistency https://huggingface.co/latent-consistency you may get better stuff but that's when it's going to get difficult and you'll start to run into those mysterious crashes I talked about that require, you know, actual work. my setup: 4090/3990x/CUDA 12.2/debian sid. ymmv.
- deleted 3y ago[deleted]
- programjames 3y agoThis paper is horribly written. It's like the authors are trying to sell me on them as researchers, instead of helping me understand their research (y'know, the entire reason journals got started??). An entire section for "stream batching" was just too much, and none of their ideas were innovative or unique. It was incredibly dense, simply because it's obfuscated, which makes me believe the authors themselves don't really understand what they're doing. The results aren't even very good. They claim 60x speedup, but compared to what? HuggingFace's Diffusers Autopipeline... a company notorious for buggy code and inefficient pipelines. And that's for naively running the pipeline on every image. Give me a break.
- kristopolous 3y agoSomehow just hacking together code to create something is considered publishable these days. The code works but it really is just pasted together stuff from the last few weeks of research.
- pro9 3y agoThe parameters and algorithms can be inferred from the code. Perhaps what’s unnecessary is tradition to wrap physical reality in human language semantics. The complexity is in the hardware. Programming has only ever been templating desired machine state. Programmers fell into a religious like state of seeing their more ornate efforts as essential to making a purpose built counting machine count.
- deleted 3y ago[deleted]
- godelski 3y agoI don't think I have a problem with this tbh. Though this specifically looks more engineering and product oriented. What I do have a problem with is comparing papers across vastly different TRLs and comparing works done with 100 GPU years of compute to works with 1 GPU year (or less). Just completely different class of works and comparing is void of context, you know? The reason I don't have a problem is I see papers as how we researchers communicate with other researchers. But I feel that's not how everyone sees them and there's the aspect that this is how we're judged so incentives get misaligned with actual goal. Idk if the reward hacking is ironic or makes sense because our job is to optimize. But don't let anyone try to convince you that reward (or any cost function) is enough.
- deleted 3y ago[deleted]
- smusamashah 3y agoI just tried the realtime-text2img demo (uses npm for frontend which i think is too much for this). Modified it to produce only 1 image instead of 16. Works well on a laptop with RTX-3080. It's probably 2 images / sec. EDIT: The `examples\screen` demo almost feels realtime. Says 4 fps on the window but don't what it represents. EDIT: Denoising in img2img is very low though which means thee returned image is only slightly different from base image.
- godelski 3y agoHow's the actual quality, diversity, and alignment though. I'm away from my GPU for a few days. It's always hard to judge generative papers without getting hands on because you write to the reviewers which means you gotta cherry pick (I think this is bad, but it's where we're at). They're using tiny autoencoder? Artspew did that too and was getting higher FPS (but weren't using TensorRT but were using triton) but the quality was garbage (still cool). Regardless, these are impressive even if the quality isn't anywhere near whats shown, but it's hard to tell.
- smusamashah 3y agoNot good which is very understandable. You are not supposed to use the output as final image. Find a good propt/seed by iterating quickly. Then go for higher step count to render a higher quality image.
- foxhop 3y agoI use automatic1111 hires fix with SDXL-turbo on a RTX 4090 and it looks the best by far for high quality images once you find a good prompt/seed (but heres the thing turning it on makes all the prompts/seeds that much better...)
- godelski 3y agoI mean there's nothing wrong with low diversity of outputs, it just has different context and its uses are different. The problem is just having all of this properly communicated so that we can evaluate properly.
- timexironman 3y agoIs there a video of it I can view anywhere?
- ChatGTP 3y agotry clicking on the link ?
- _joel 3y agoMaybe we're all living in a simulation^H^H^H^H^H pipeline-level solution for real-time interactive generation.
- ChatGTP 3y agoMaybe we're not?