6 ms·

Show HN: Realtime LCM tool

12 points by dvrp 3y ago

dvrp 3y ago
will write an essay about it later, but servers exploding even with experimental alpha users. I'm excited about what H200s or B100s will mean for technology like this one.
- ilaksh 3y ago
  Is it running on H100s or A100s now?
  - dvrp 3y ago
    right now A100s, we are still benchmarking 80gb and 40gbs (of VRAM) ones to see what makes more sense to scale this.
vmatsiiako 3y ago
Can you use custom images?
- dvrp 3y ago
  yea! there was another video i put on my twitter account but it's already 24 hours old; the UI has changed a bit. here's the tweet: https://twitter.com/asciidiego/status/1723804366434627611 https://twitter.com/asciidiego/status/1723804366434627611 also, how's infisical doing?
erwannmillon 3y ago
i worked on the gpu/infra side of this, so feel free to AMA Ultimately the LCM is just a SD Unet trained with a new objective, so a lot of SD optimizations are transferable to LCMs
- dvrp 3y ago
  i also worked a bit on the infrastructure! there's a very cool consistent hashing and probability stuff going on for the GPU routing logic but again, i'll write it after this fire ends similar to a post-mortem.
brucethemoose2 3y ago
Are y'all running the diffusion in PyTorch eager mode? AITemplate, stable-fast, or even a torch.compile could get that down to 60ms, I bet. Though I'm not sure the example implementations would work on a non SD architecture.
- erwannmillon 3y ago
  LCMs are actually exactly SD architecture. LCM is initialized from a regular SD unet and finetuned on a new objective. We are already compiling to get to these times. A lot of other people getting sub-100ms times are using fewer inference steps than we do, at a quality tradeoff.
  - brucethemoose2 3y ago
    > already compiling Hmm, well if you mean torch.compile, y'all should still check out stable-fast, which is claiming ~16ms/iter on a 4090, twice that of torch.compile: https://github.com/chengzeyi/stable-fast#rtx-4090-512x512-batch-size-1-fp16-tcmalloc-enabled https://github.com/chengzeyi/stable-fast#rtx-4090-512x512-ba...
- dvrp 3y ago
  some people even claim they get ~50ms on (powerful) consumer hardware (e.g. RTX 4090). we still have not been able to replicate those results; also because we want to do it in a distributed way
  - brucethemoose2 3y ago
    They might be "cheating" with methods that significantly degrade quality.

12 points by dvrp 3y ago

dvrp 3y ago

will write an essay about it later, but servers exploding even with experimental alpha users. I'm excited about what H200s or B100s will mean for technology like this one.

ilaksh 3y ago
Is it running on H100s or A100s now?
- dvrp 3y ago
  right now A100s, we are still benchmarking 80gb and 40gbs (of VRAM) ones to see what makes more sense to scale this.

vmatsiiako 3y ago

Can you use custom images?

dvrp 3y ago
yea! there was another video i put on my twitter account but it's already 24 hours old; the UI has changed a bit. here's the tweet: https://twitter.com/asciidiego/status/1723804366434627611 https://twitter.com/asciidiego/status/1723804366434627611 also, how's infisical doing?

erwannmillon 3y ago

i worked on the gpu/infra side of this, so feel free to AMA Ultimately the LCM is just a SD Unet trained with a new objective, so a lot of SD optimizations are transferable to LCMs

dvrp 3y ago
i also worked a bit on the infrastructure! there's a very cool consistent hashing and probability stuff going on for the GPU routing logic but again, i'll write it after this fire ends similar to a post-mortem.

brucethemoose2 3y ago

Are y'all running the diffusion in PyTorch eager mode? AITemplate, stable-fast, or even a torch.compile could get that down to 60ms, I bet. Though I'm not sure the example implementations would work on a non SD architecture.

erwannmillon 3y ago
LCMs are actually exactly SD architecture. LCM is initialized from a regular SD unet and finetuned on a new objective. We are already compiling to get to these times. A lot of other people getting sub-100ms times are using fewer inference steps than we do, at a quality tradeoff.
- brucethemoose2 3y ago
  > already compiling Hmm, well if you mean torch.compile, y'all should still check out stable-fast, which is claiming ~16ms/iter on a 4090, twice that of torch.compile: https://github.com/chengzeyi/stable-fast#rtx-4090-512x512-batch-size-1-fp16-tcmalloc-enabled https://github.com/chengzeyi/stable-fast#rtx-4090-512x512-ba...
dvrp 3y ago
some people even claim they get ~50ms on (powerful) consumer hardware (e.g. RTX 4090). we still have not been able to replicate those results; also because we want to do it in a distributed way
- brucethemoose2 3y ago
  They might be "cheating" with methods that significantly degrade quality.