4 ms·

But that's not something you'd use an LLM for. There have been computer vision systems sorting bad peas for more than a decade[0], of course there are plenty of

But that's not something you'd use an LLM for. There have been computer vision systems sorting bad peas for more than a decade[0], of course there are plenty of use cases for very fast inspection systems. But when would you use an LLM for anything like that? [0] https://www.youtube.com/watch?v=eLDxXPziztw https://www.youtube.com/watch?v=eLDxXPziztw

0xbadcafebee 7mo ago
You would use a VLM (vision language model). The model analyzes the image and outputs text, along with general context, that can drive intelligent decisions. https://tryolabs.com/blog/llms-leveraging-computer-vision https://tryolabs.com/blog/llms-leveraging-computer-vision
arcanemachiner 7mo ago
Nobody said you would use an LLM for that. It's an example of a process where "industrial inspection, in particular, [would] benefit from lower latency in exchange for accuracy". The point of their comment isn't that you would use an LLM to sort fruit. It was just an illustrative example.
- sorenjan 7mo ago
  The discussion was about fine-tuned Qwen models, not industrial inspection in general. I would also find it interesting to learn about what kind of edge AI industrial inspection task you could do with fine-tuned llms, not some handwavy answer about how sometimes latency is important in real time systems. Of course it is, so generally you don't use models with several billion parameters unless you need to.
  - arcanemachiner 7mo ago
    The thread you're in broke away from the main discussion topic. Again: Nobody is using LLMs to (for example) sort fruit. But there are some industrial processes that prioritize latency over reliability.
    - nl 7mo ago
      No, we are literally trying to find a use case where using a lower accuracy LLM makes sense for a vision task. But fine - what are these industrial processes where that prioritize latency over reliability and using a LLM - as mentioned by the OP - makes sense?
      - IanCal 7mo ago
        > No, we are literally trying to find a use case where using a lower accuracy LLM makes sense for a vision task. They're reconfigurable on the fly with little technical expertise and without training data, that's really useful. Personally in projects for people I've found models have fewer unusual edge cases than traditional models, are less sensitive to minor changes in input and are easier to debug by asking them what they can see.
        sorenjan 7mo ago
        Seems like a way to use a sledgehammer to hammer in screws, and inviting nondeterminism in important systems. Besides being way larger and more complex than what most specialized industrial processes need, they are also vulnerable to adversarial attacks. https://www.lakera.ai/blog/visual-prompt-injections https://www.lakera.ai/blog/visual-prompt-injections https://www.theverge.com/2021/3/8/22319173/openai-machine-vision-adversarial-typographic-attacka-clip-multimodal-neuron https://www.theverge.com/2021/3/8/22319173/openai-machine-vi...
        IanCal 7mo ago
        > Seems like a way to use a sledgehammer to hammer in screws The lazy analogy the other way is that developing a custom system to do these jobs is like hiring a team of experts to spend 2 years designing the perfect crosshead screwdriver that fits exactly one screw (and doesn't work if the screw starts slightly rotated) when you have a flathead one right next to you that'll work and it'll work right now. > and inviting nondeterminism in important systems. Traditional ML is just as non-deterministic. > they are also vulnerable to adversarial attacks. Typically not relevant in these kinds of cases but also this is easily a problem in many traditional ML algos. Have you worked on things like this?

0xbadcafebee 7mo ago

You would use a VLM (vision language model). The model analyzes the image and outputs text, along with general context, that can drive intelligent decisions. https://tryolabs.com/blog/llms-leveraging-computer-vision https://tryolabs.com/blog/llms-leveraging-computer-vision

arcanemachiner 7mo ago

Nobody said you would use an LLM for that. It's an example of a process where "industrial inspection, in particular, [would] benefit from lower latency in exchange for accuracy". The point of their comment isn't that you would use an LLM to sort fruit. It was just an illustrative example.

sorenjan 7mo ago
The discussion was about fine-tuned Qwen models, not industrial inspection in general. I would also find it interesting to learn about what kind of edge AI industrial inspection task you could do with fine-tuned llms, not some handwavy answer about how sometimes latency is important in real time systems. Of course it is, so generally you don't use models with several billion parameters unless you need to.
- arcanemachiner 7mo ago
  The thread you're in broke away from the main discussion topic. Again: Nobody is using LLMs to (for example) sort fruit. But there are some industrial processes that prioritize latency over reliability.
  - nl 7mo ago
    No, we are literally trying to find a use case where using a lower accuracy LLM makes sense for a vision task. But fine - what are these industrial processes where that prioritize latency over reliability and using a LLM - as mentioned by the OP - makes sense?
    - IanCal 7mo ago
      > No, we are literally trying to find a use case where using a lower accuracy LLM makes sense for a vision task. They're reconfigurable on the fly with little technical expertise and without training data, that's really useful. Personally in projects for people I've found models have fewer unusual edge cases than traditional models, are less sensitive to minor changes in input and are easier to debug by asking them what they can see.
      - sorenjan 7mo ago
        Seems like a way to use a sledgehammer to hammer in screws, and inviting nondeterminism in important systems. Besides being way larger and more complex than what most specialized industrial processes need, they are also vulnerable to adversarial attacks. https://www.lakera.ai/blog/visual-prompt-injections https://www.lakera.ai/blog/visual-prompt-injections https://www.theverge.com/2021/3/8/22319173/openai-machine-vision-adversarial-typographic-attacka-clip-multimodal-neuron https://www.theverge.com/2021/3/8/22319173/openai-machine-vi...
        IanCal 7mo ago
        > Seems like a way to use a sledgehammer to hammer in screws The lazy analogy the other way is that developing a custom system to do these jobs is like hiring a team of experts to spend 2 years designing the perfect crosshead screwdriver that fits exactly one screw (and doesn't work if the screw starts slightly rotated) when you have a flathead one right next to you that'll work and it'll work right now. > and inviting nondeterminism in important systems. Traditional ML is just as non-deterministic. > they are also vulnerable to adversarial attacks. Typically not relevant in these kinds of cases but also this is easily a problem in many traditional ML algos. Have you worked on things like this?