3 ms·

Thanks. I guess this means that fewer people will be able to use it on their own computer, but the improved efficiency makes it cheaper to run on servers with e

Thanks. I guess this means that fewer people will be able to use it on their own computer, but the improved efficiency makes it cheaper to run on servers with enough VRAM. Maybe running stage C first, unloading it from VRAM, and then do B and A would make it fit in 12 or even 8 GB, but I wonder if the memory transfers would negate any time saving. Might still be worth it if it produces better images though.

adventured 3y ago
If it worked I imagine large batching could make it worth the load/unload time cost.
- weebull 3y ago
  Shouldn't be a reason you couldn't do a ton of Layer C work on different images, and then swap in Layer B.
Filligree 3y ago
Sequential model offloading isn’t too bad. It adds about a second or less to inference, assuming it still fits in main memory.
- sorenjan 3y ago
  Sometimes I forget how fast modern computers are. PCIe v4 x16 has a transfer speed of 31.5 GB/s, so theoretically it should take less than 100 ms to transfer stage B and A. Maybe it's not so bad after all, it will be interesting to see what happens.
whywhywhywhy 3y ago
If you're serious about doing image gen locally you should be running a 24GB card anyway because honestly Nvidia's current generation 24GB is the sweet spot price to performance. 3080 ram is laughably the same as the 6 year old 1080Ti and 4080 ram is only slightly more at 16 and costs about 1.5 times the 3090 second hand. Any speed benefits of the 4080 are gonna be worthless the second it has to cycle a model in and out of ram anyway vs the 3090 in image gen.
- weebull 3y ago
  > because honestly Nvidia's current generation 24GB is the sweet spot price to performance How is the halo product of a range the "sweet spot"? I think nVidia are extremely exposed on this front. The RX 7900XTX is also 24GB and under half the price (In UK at least - £800 vs £1,700 for the 4090). It's difficult to get a performance comparison on compute tasks, but I think it's around 70-80% of the 4090 given what I can find. Even a 3090, if you can find one, is £1,500. The software isn't as stable on AMD hardware, but it does work. I'm running a RX7600 - 8GB myself, and happily doing SDXL. The main problem is that exhausting VRAM causes instability. Exceed it by a lot, and everything is handled fine, but if it's marginal... problems ensue. The AMD engineers are actively making the experience better, and it may not be long before it's a practical alternative. If/When that happens nVidia will need to slash their prices to sell anything in this sphere, which I can't really see themselves doing.
  - zargon 3y ago
    > If/When that happens nVidia will need to slash their prices to sell anything in this sphere It's just as likely that AMD will raise prices to compensate.
    - weebull 3y ago
      You think they're going to say "Hey, compute became competitive but nothing else changed performance therefore... PRICE HIKE!"? They don't have the reputation to burn in this domain for that IMHO. Granted you could see a supply/demand related increase from retailers if demand spiked, but that's the retailers capitalising.
      - zargon 3y ago
        The price hike would come proportionally with each new hardware generation as their software stack becomes competitive with CUDA. Since it will most likely take them multiple generations to catch up. It doesn't make sense for them to sell hardware with comparable capability at a deep discount. Plenty of companies will want to diversify hardware vendors just to have a bargaining chip against Nvidia.

adventured 3y ago

If it worked I imagine large batching could make it worth the load/unload time cost.

weebull 3y ago
Shouldn't be a reason you couldn't do a ton of Layer C work on different images, and then swap in Layer B.

Filligree 3y ago

Sequential model offloading isn’t too bad. It adds about a second or less to inference, assuming it still fits in main memory.

sorenjan 3y ago
Sometimes I forget how fast modern computers are. PCIe v4 x16 has a transfer speed of 31.5 GB/s, so theoretically it should take less than 100 ms to transfer stage B and A. Maybe it's not so bad after all, it will be interesting to see what happens.

whywhywhywhy 3y ago

If you're serious about doing image gen locally you should be running a 24GB card anyway because honestly Nvidia's current generation 24GB is the sweet spot price to performance. 3080 ram is laughably the same as the 6 year old 1080Ti and 4080 ram is only slightly more at 16 and costs about 1.5 times the 3090 second hand. Any speed benefits of the 4080 are gonna be worthless the second it has to cycle a model in and out of ram anyway vs the 3090 in image gen.

weebull 3y ago
> because honestly Nvidia's current generation 24GB is the sweet spot price to performance How is the halo product of a range the "sweet spot"? I think nVidia are extremely exposed on this front. The RX 7900XTX is also 24GB and under half the price (In UK at least - £800 vs £1,700 for the 4090). It's difficult to get a performance comparison on compute tasks, but I think it's around 70-80% of the 4090 given what I can find. Even a 3090, if you can find one, is £1,500. The software isn't as stable on AMD hardware, but it does work. I'm running a RX7600 - 8GB myself, and happily doing SDXL. The main problem is that exhausting VRAM causes instability. Exceed it by a lot, and everything is handled fine, but if it's marginal... problems ensue. The AMD engineers are actively making the experience better, and it may not be long before it's a practical alternative. If/When that happens nVidia will need to slash their prices to sell anything in this sphere, which I can't really see themselves doing.
- zargon 3y ago
  > If/When that happens nVidia will need to slash their prices to sell anything in this sphere It's just as likely that AMD will raise prices to compensate.
  - weebull 3y ago
    You think they're going to say "Hey, compute became competitive but nothing else changed performance therefore... PRICE HIKE!"? They don't have the reputation to burn in this domain for that IMHO. Granted you could see a supply/demand related increase from retailers if demand spiked, but that's the retailers capitalising.
    - zargon 3y ago
      The price hike would come proportionally with each new hardware generation as their software stack becomes competitive with CUDA. Since it will most likely take them multiple generations to catch up. It doesn't make sense for them to sell hardware with comparable capability at a deep discount. Plenty of companies will want to diversify hardware vendors just to have a bargaining chip against Nvidia.