4 ms·

I would be very greatfull to see how one can leverage it not for LLMs but for Stable Diffusion models

• m00x 3y ago
  Why couldn't it be applied to SD?
  • m00x 3y ago
    It looks like it's already at thing https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob/f865d3e11647dfd6c7b2cdf90dde24680e58acd8/modules/cmd_args.py#L54 https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob...
    • zaptrem 3y ago
      That's Flash Attention 1, but yeah there's nothing stopping someone from submitting a PR.
    • hunkins 3y ago
      This enables V1. V2 is still yet to be integrated into xformers. The team replied saying it should happen this week. See the relevant Github issue here: https://github.com/facebookresearch/xformers/issues/795 https://github.com/facebookresearch/xformers/issues/795
  • mcyc 3y ago
    SD1 uses a head dimension that is not supported by the old version of Flash Attention but is supported by this version.