4 ms·

Sometimes TPS doesn't matter. I've generated textual descriptions for 100K or so images in my photo archive, some of which I have absolutely no interest in uplo

by ein0p 1y ago

Sometimes TPS doesn't matter. I've generated textual descriptions for 100K or so images in my photo archive, some of which I have absolutely no interest in uploading to someone else's computer. This works pretty well with Gemma. I use local LLMs all the time for things where privacy is even remotely important. I estimate this constitutes easily a quarter of my LLM usage.

lodovic 1y ago
This is a really cool idea. Do you pretrain the model so it can tag people? I have so many photo's that it seems impossible to ever categorize them,using a workflow like yours might help a lot
- ein0p 1y ago
  No, tagging of people is already handled by another model. Gemma just describes what's in the image, and produces a comma separated list of keywords. No additional training is required besides a few tweaks to the prompt so that it outputs just the description, without any "fluff". E.g. it normally prepends such outputs with "Here's a description of the image:" unless you really insist that it should output only the description. I suppose I could use constrained decoding into JSON or something to achieve the same, but I didn't mess with that. On some images where Gemma3 struggles Mistral Small produces better descriptions, BTW. But it seems harder to make it follow my instructions exactly. I'm looking forward to the day when I can also do this with videos, a lot of which I also have no interest in uploading to someone else's computer.
  - fer 1y ago
    How do you use the keywords after? I have Immich running which does some analysis, but the querying is a bit of a hit and miss.
    - ein0p 1y ago
      Search is indeed hit and miss. Immich, for instance, currently does absolutely nothing with the EXIF "description" field, so I store textual descriptions on the side as well. I have found Immich's search by image embeddings to be pretty weak at recall, and even weaker at ranking. IIRC Lightroom Classic (which I also use, but haven't found a way to automate this for without writing an extension) does search that field, but ranking is a bit of a dumpster fire, so your best bet is searching uncommon terms or constraining search by metadata (e.g. not just "black kitten" but "black kitten AND 2025"). I expect this to improve significantly over time - it's a fairly obvious thing to add given the available tech.
  - ethersteeds 1y ago
    > No, tagging of people is already handled by another model. As an aside, what model/tools do you prefer for tagging people?
  - mentalgear 1y ago
    Since you already seem to have done some impressive work on this for your personal use, would you mind open sourcing it?
starik36 1y ago
I was thinking of doing the same, but I would like to include people's name. in the description. For example "Jennifer looking out in the desert sky.". As it stands, Gemma will just say "Woman looking out in the desert sky."
- ein0p 1y ago
  Most search rankers do not consider word order, so if you could also append the person's name at the end of text description, it'd probably work well enough for retrieval and ranking at least. If you want natural language to resolve the names, that'd at a minimum require bounding boxes of the faces and their corresponding names. It'd also require either preprocessing, or specialized training, or both. To my knowledge no locally-hostable model as of today has that. I don't know if any proprietary models can do this either, but it's certainly worth a try - they might just do it. The vast majority of the things they can do is emergent, meaning they were never specifically trained to do them.

by ein0p 1y ago

lodovic 1y ago

This is a really cool idea. Do you pretrain the model so it can tag people? I have so many photo's that it seems impossible to ever categorize them,using a workflow like yours might help a lot

ein0p 1y ago
No, tagging of people is already handled by another model. Gemma just describes what's in the image, and produces a comma separated list of keywords. No additional training is required besides a few tweaks to the prompt so that it outputs just the description, without any "fluff". E.g. it normally prepends such outputs with "Here's a description of the image:" unless you really insist that it should output only the description. I suppose I could use constrained decoding into JSON or something to achieve the same, but I didn't mess with that. On some images where Gemma3 struggles Mistral Small produces better descriptions, BTW. But it seems harder to make it follow my instructions exactly. I'm looking forward to the day when I can also do this with videos, a lot of which I also have no interest in uploading to someone else's computer.
- fer 1y ago
  How do you use the keywords after? I have Immich running which does some analysis, but the querying is a bit of a hit and miss.
  - ein0p 1y ago
    Search is indeed hit and miss. Immich, for instance, currently does absolutely nothing with the EXIF "description" field, so I store textual descriptions on the side as well. I have found Immich's search by image embeddings to be pretty weak at recall, and even weaker at ranking. IIRC Lightroom Classic (which I also use, but haven't found a way to automate this for without writing an extension) does search that field, but ranking is a bit of a dumpster fire, so your best bet is searching uncommon terms or constraining search by metadata (e.g. not just "black kitten" but "black kitten AND 2025"). I expect this to improve significantly over time - it's a fairly obvious thing to add given the available tech.
- ethersteeds 1y ago
  > No, tagging of people is already handled by another model. As an aside, what model/tools do you prefer for tagging people?
- mentalgear 1y ago
  Since you already seem to have done some impressive work on this for your personal use, would you mind open sourcing it?

starik36 1y ago

I was thinking of doing the same, but I would like to include people's name. in the description. For example "Jennifer looking out in the desert sky.". As it stands, Gemma will just say "Woman looking out in the desert sky."

ein0p 1y ago
Most search rankers do not consider word order, so if you could also append the person's name at the end of text description, it'd probably work well enough for retrieval and ranking at least. If you want natural language to resolve the names, that'd at a minimum require bounding boxes of the faces and their corresponding names. It'd also require either preprocessing, or specialized training, or both. To my knowledge no locally-hostable model as of today has that. I don't know if any proprietary models can do this either, but it's certainly worth a try - they might just do it. The vast majority of the things they can do is emergent, meaning they were never specifically trained to do them.