4 ms·

This is incredibly easy to setup, just tried it for first time. How fast is it supposed to go? Just tried on linux with `cmake .. -DGGML_OPENBLAS=ON` on a AMD

by kpw94 3y ago

This is incredibly easy to setup, just tried it for first time. How fast is it supposed to go? Just tried on linux with `cmake .. -DGGML_OPENBLAS=ON` on a AMD Ryzen 7 5700g (no discrete GPU, only integrated graphics) ./bin/sd -m ../models/sd-v1-4-ggml-model-f32.bin -p "a lovely cat" [INFO] stable-diffusion.cpp:2525 - loading model from '../models/sd-v1-4-ggml-model-f32.bin' ... [INFO] stable-diffusion.cpp:3375 - start sampling [INFO] stable-diffusion.cpp:3067 - step 1 sampling completed, taking 12.25s [INFO] stable-diffusion.cpp:3067 - step 2 sampling completed, taking 12.22s [INFO] stable-diffusion.cpp:3067 - step 3 sampling completed, taking 12.56s ... sampling completed, taking 246.40s Is that expected performance? (EDIT: Don't have open Blas installed, so that flag is no-op)

brucethemoose2 3y ago
You should quantize the model, but 12s/iter seems about right.
- kpw94 3y ago
  Nice. Tried the fp32, q8_0, and q4_0, and for some reason they all take ~12s/iter. Must have something wrong with my setup, but no big deal, for my minimal usage of it, and the amount of time spent, fp32@12s/iter is fine
  - brucethemoose2 3y ago
    Hmm, theoretically FP16 might be the fastest, if thats an option in the implementation now.
    - Lockal 3y ago
      I did a quick run under profiler and on my AVX2-laptop the slowest part (>50%) was matrix multiplication (sgemm). In current version of GGML if OpenBLAS is enabled, they convert matrices to FP32 before running sgemm. If OpenBLAS is disabled, on AVX2 plaftorm they convert FP16 to FP32 on every FMA operation, which even worse (due to repetition). After that, both ggml_vec_dot_f16 and ggml_vec_dot_f32 took first place in profiler. Source: https://github.com/ggerganov/ggml/blob/master/src/ggml.c#L10820 https://github.com/ggerganov/ggml/blob/master/src/ggml.c#L10... But I agree, that in theory, and only with AVX512 BF16 (not exactly FP16, but similar) will be fast with VDPBF16PS instruction. Implementation is not there yet.
      - brucethemoose2 3y ago
        Interesting. I saw some discussion on llama.cpp that, theoretically, implementing matmul for each quantization should be much faster since it can skip the conversion. But practically, its actually quite difficult since the various BLAS libraries are so good.
patrakov 3y ago
CPU-only, 8-bit quant, Intel Core i7 4770S, 16 GB DDR3 RAM, 10 year old fanless PC: 32 seconds per sampling step, correct output.
badsectoracula 3y ago
This is nice, it basically does what i asked a year ago[0] and at the time pretty much every solution wanted a litany of Python dependencies that i ended up failing to install because it took ages... and then i ran out of disk space. No, really, this replaces literal gigabytes of disk space with just a 799KB binary. And as a bonus using the Q8_0 format (the one that seems to be the fastest) it also saves ~2.3GB of data too. That said, it seems to be buggy with anything other than the default 512x512 image size. Some sizes (e.g. 544x544) tend to cause assert fails, sizes smaller than 512x512 (which i tried as 512x512 is quite slow on my PC) sometimes generate garbage (anything smaller than 384x384 seems to always do that). [0] https://news.ycombinator.com/item?id=32555608 https://news.ycombinator.com/item?id=32555608
- kpw94 3y ago
  Also got segfault core dump with different sizes. 512w * 768h worked
- gsharma 3y ago
  > sizes smaller than 512x512 (which i tried as 512x512 is quite slow on my PC) sometimes generate garbage (anything smaller than 384x384 seems to always do that). Not sure about the speed, but the garbage output might be due to the model instead of the library. I’ve always got garbage (using other tools) when I tried 256x256.
- Sharlin 3y ago
  SD 1.x does not work well at resolutions less than 512x512, it's only really trained to output 512x512.
  - dragonwriter 3y ago
    IME, SD 1.5 base model, IIRC (I don’t use the base model much) works tolerably well with the larger dimension at 512 and the smaller dimension between 384-512, and quite a lot of SD 1.5 based checkpoints work pretty well up to a larger dimension of 768 (some with smaller dimension up to that, too.) But SD also has hard requirement that the output dimensions be divisible by 8 (I believe as a consequence of the pixel space to latent space ratio being 8:1 in each dimension) so its not surprising that 544x544 fails hard.

by kpw94 3y ago

brucethemoose2 3y ago

You should quantize the model, but 12s/iter seems about right.

kpw94 3y ago
Nice. Tried the fp32, q8_0, and q4_0, and for some reason they all take ~12s/iter. Must have something wrong with my setup, but no big deal, for my minimal usage of it, and the amount of time spent, fp32@12s/iter is fine
- brucethemoose2 3y ago
  Hmm, theoretically FP16 might be the fastest, if thats an option in the implementation now.
  - Lockal 3y ago
    I did a quick run under profiler and on my AVX2-laptop the slowest part (>50%) was matrix multiplication (sgemm). In current version of GGML if OpenBLAS is enabled, they convert matrices to FP32 before running sgemm. If OpenBLAS is disabled, on AVX2 plaftorm they convert FP16 to FP32 on every FMA operation, which even worse (due to repetition). After that, both ggml_vec_dot_f16 and ggml_vec_dot_f32 took first place in profiler. Source: https://github.com/ggerganov/ggml/blob/master/src/ggml.c#L10820 https://github.com/ggerganov/ggml/blob/master/src/ggml.c#L10... But I agree, that in theory, and only with AVX512 BF16 (not exactly FP16, but similar) will be fast with VDPBF16PS instruction. Implementation is not there yet.
    - brucethemoose2 3y ago
      Interesting. I saw some discussion on llama.cpp that, theoretically, implementing matmul for each quantization should be much faster since it can skip the conversion. But practically, its actually quite difficult since the various BLAS libraries are so good.

patrakov 3y ago

CPU-only, 8-bit quant, Intel Core i7 4770S, 16 GB DDR3 RAM, 10 year old fanless PC: 32 seconds per sampling step, correct output.

badsectoracula 3y ago

This is nice, it basically does what i asked a year ago[0] and at the time pretty much every solution wanted a litany of Python dependencies that i ended up failing to install because it took ages... and then i ran out of disk space. No, really, this replaces literal gigabytes of disk space with just a 799KB binary. And as a bonus using the Q8_0 format (the one that seems to be the fastest) it also saves ~2.3GB of data too. That said, it seems to be buggy with anything other than the default 512x512 image size. Some sizes (e.g. 544x544) tend to cause assert fails, sizes smaller than 512x512 (which i tried as 512x512 is quite slow on my PC) sometimes generate garbage (anything smaller than 384x384 seems to always do that). [0] https://news.ycombinator.com/item?id=32555608 https://news.ycombinator.com/item?id=32555608

kpw94 3y ago
Also got segfault core dump with different sizes. 512w * 768h worked
gsharma 3y ago
> sizes smaller than 512x512 (which i tried as 512x512 is quite slow on my PC) sometimes generate garbage (anything smaller than 384x384 seems to always do that). Not sure about the speed, but the garbage output might be due to the model instead of the library. I’ve always got garbage (using other tools) when I tried 256x256.
Sharlin 3y ago
SD 1.x does not work well at resolutions less than 512x512, it's only really trained to output 512x512.
- dragonwriter 3y ago
  IME, SD 1.5 base model, IIRC (I don’t use the base model much) works tolerably well with the larger dimension at 512 and the smaller dimension between 384-512, and quite a lot of SD 1.5 based checkpoints work pretty well up to a larger dimension of 768 (some with smaller dimension up to that, too.) But SD also has hard requirement that the output dimensions be divisible by 8 (I believe as a consequence of the pixel space to latent space ratio being 8:1 in each dimension) so its not surprising that 544x544 fails hard.