4 ms·

Ah true, missed that! Still a bit cumbersome & lazy imo, I'm a fan of just shipping with that capability out-of-the-box (Huggingface's Candle is fantastic for d

by dvt 8mo ago

Ah true, missed that! Still a bit cumbersome & lazy imo, I'm a fan of just shipping with that capability out-of-the-box (Huggingface's Candle is fantastic for downloading/syncing/running models locally).

embedding-shape 8mo ago
Ah come on, lazy? As long as it works with the runtime you wanna use, instead of hardcoding their own solution, should work fine. If you want to use Candle and have to implement new architectures with it to be able to use it, you still can, just expose it over HTTP.
- dvt 8mo ago
  I think one of the major problems with the current incarnation of AI solutions is that they're extremely brittle and hacked-together. It's a fun exciting time, especially for us technical people, but normies just want stuff to "work." Even copy-pasting an API key is probably too much of a hurdle for regular folks, let alone running a local ollama server in a Docker container.
  - Sharlin 8mo ago
    Unlike in image/video gen, at least with LLMs the "best" solution available isn’t a graph/node-based interface with an ecosystem of hundreds of hacky undocumented custom nodes that break every few days and way too complex workflows made up of a spaghetti of two dozen nodes with numerous parameters each, half of which have no discernible effect on output quality and tweaking the rest is entirely trial and error.
    - dragonwriter 8mo ago
      That's not the best solution for image or video (or audio, or 3D) any more than it is for LLMs (which it also supports.) OTOH, its the most flexible and likely to have some support for what you are doing for a lot of those, and especially if yoj are combining multiple of them in the same process.
      - Sharlin 8mo ago
        Yes, "best" is subjective and that’s why I put it in quotes. But in the community it’s definitely seen as something users should and do "upgrade" to from less intimidating but less flexible tools if they want the most power, and most importantly, support for bleeding-edge models. I rarely use Comfy myself, FWIW.
  - embedding-shape 8mo ago
    > but normies just want stuff to "work." Where in the world are you getting that this project is for "normies"? Installation steps are terminal instructions and it's a CLI, clearly meant for technical people already. If you think copying-pasting an API key is too much, don't you think cloning a git repository, installing the Rust compiler and compiling the project might be too much and hit those normies in the face sooner than the API key?
mirekrusin 8mo ago
In local setup you still usually want to split machine that runs inference from client that uses it, there are often non trivial resources used like chromium, compilation, databases etc involved that you don’t want to pollute inference machine with.

by dvt 8mo ago

embedding-shape 8mo ago

Ah come on, lazy? As long as it works with the runtime you wanna use, instead of hardcoding their own solution, should work fine. If you want to use Candle and have to implement new architectures with it to be able to use it, you still can, just expose it over HTTP.

dvt 8mo ago
I think one of the major problems with the current incarnation of AI solutions is that they're extremely brittle and hacked-together. It's a fun exciting time, especially for us technical people, but normies just want stuff to "work." Even copy-pasting an API key is probably too much of a hurdle for regular folks, let alone running a local ollama server in a Docker container.
- Sharlin 8mo ago
  Unlike in image/video gen, at least with LLMs the "best" solution available isn’t a graph/node-based interface with an ecosystem of hundreds of hacky undocumented custom nodes that break every few days and way too complex workflows made up of a spaghetti of two dozen nodes with numerous parameters each, half of which have no discernible effect on output quality and tweaking the rest is entirely trial and error.
  - dragonwriter 8mo ago
    That's not the best solution for image or video (or audio, or 3D) any more than it is for LLMs (which it also supports.) OTOH, its the most flexible and likely to have some support for what you are doing for a lot of those, and especially if yoj are combining multiple of them in the same process.
    - Sharlin 8mo ago
      Yes, "best" is subjective and that’s why I put it in quotes. But in the community it’s definitely seen as something users should and do "upgrade" to from less intimidating but less flexible tools if they want the most power, and most importantly, support for bleeding-edge models. I rarely use Comfy myself, FWIW.
- embedding-shape 8mo ago
  > but normies just want stuff to "work." Where in the world are you getting that this project is for "normies"? Installation steps are terminal instructions and it's a CLI, clearly meant for technical people already. If you think copying-pasting an API key is too much, don't you think cloning a git repository, installing the Rust compiler and compiling the project might be too much and hit those normies in the face sooner than the API key?

mirekrusin 8mo ago

In local setup you still usually want to split machine that runs inference from client that uses it, there are often non trivial resources used like chromium, compilation, databases etc involved that you don’t want to pollute inference machine with.