3 ms·

Seems I'm missing something. Does this model support other inputs? Image outputs are supported, videos I'm not sure but I don't think that's an output, just a

Seems I'm missing something. Does this model support other inputs? Image outputs are supported, videos I'm not sure but I don't think that's an output, just a preview of the equirectangular example, so, same question here, what does this model outputs that isn't supported?

deleted 6d ago
[deleted]
exe34 6d ago
It's a diffusion model, completely different from autoregressive attention models.
- utopiah 6d ago
  like https://github.com/ggml-org/llama.cpp/tree/master/examples/diffusion https://github.com/ggml-org/llama.cpp/tree/master/examples/d... ? At the risk of stating the obvious llama.cpp isn't just about LLaMa as https://github.com/ggml-org/llama.cpp/blob/master/src/llama-arch.h#L13 https://github.com/ggml-org/llama.cpp/blob/master/src/llama-... someone else pointed out.
  - exe34 6d ago
    Aha I was wrong. Thanks for sharing that!
    - utopiah 6d ago
      no worries, I was wrong too, it is multimodal but only for inputs apparently, so for now there seem to only be text as output but no image as output
mdp2021 6d ago
Not all architectures are supported by llama.cpp . The GGUF format encodes the NN in a standardized way, but then you need code that can use that NN structure. I understand that llama.cpp could only output text, last time I checked (I do not know how to find a good source for that though). See https://github.com/ggml-org/llama.cpp/blob/master/src/llama-arch.h https://github.com/ggml-org/llama.cpp/blob/master/src/llama-... , the enum llm_arch { ...
- utopiah 6d ago
  I haven't used llama.cpp for image generation either but I recall an issue about it. Unfortunately I can't pinpoint it now and there is the older closed issue https://github.com/ggml-org/llama.cpp/issues/4408 https://github.com/ggml-org/llama.cpp/issues/4408 so unless mtmd supports also multimodal outputs out of the box safe to assume output is still limited to text.

deleted 6d ago

[deleted]

exe34 6d ago

It's a diffusion model, completely different from autoregressive attention models.

utopiah 6d ago
like https://github.com/ggml-org/llama.cpp/tree/master/examples/diffusion https://github.com/ggml-org/llama.cpp/tree/master/examples/d... ? At the risk of stating the obvious llama.cpp isn't just about LLaMa as https://github.com/ggml-org/llama.cpp/blob/master/src/llama-arch.h#L13 https://github.com/ggml-org/llama.cpp/blob/master/src/llama-... someone else pointed out.
- exe34 6d ago
  Aha I was wrong. Thanks for sharing that!
  - utopiah 6d ago
    no worries, I was wrong too, it is multimodal but only for inputs apparently, so for now there seem to only be text as output but no image as output

mdp2021 6d ago

Not all architectures are supported by llama.cpp . The GGUF format encodes the NN in a standardized way, but then you need code that can use that NN structure. I understand that llama.cpp could only output text, last time I checked (I do not know how to find a good source for that though). See https://github.com/ggml-org/llama.cpp/blob/master/src/llama-arch.h https://github.com/ggml-org/llama.cpp/blob/master/src/llama-... , the enum llm_arch { ...

utopiah 6d ago
I haven't used llama.cpp for image generation either but I recall an issue about it. Unfortunately I can't pinpoint it now and there is the older closed issue https://github.com/ggml-org/llama.cpp/issues/4408 https://github.com/ggml-org/llama.cpp/issues/4408 so unless mtmd supports also multimodal outputs out of the box safe to assume output is still limited to text.