4 ms·

After just spending 15 minutes trying to get something useful accomplished, anything useful at all, with latest beta Apple Intelligence with a M1 iPad Pro (16G

After just spending 15 minutes trying to get something useful accomplished, anything useful at all, with latest beta Apple Intelligence with a M1 iPad Pro (16G RAM), this article appealed to me! I have been running the 32B parameters qwen2.5-coder model on my 32G M2 Mac and and it is a huge help with coding. The llama3.3-vision model does a great job processing screen shots. Small models like smollm2:latest can process a lot of text locally, very fast. Open source front ends like Open WebUI are improving rapidly. All the tools are lining up for do it yourself local AI. The only commercial vendor right now that I think is doing a fairly good job at an integrated AI workflow is Google. Last month I had all my email directed to my gmail account, and the Gemini Advanced web app did a really good job integrating email, calendar, and google docs. Job well done. That said, I am back to using ProtonMail and trying to build local AIs for my workflows. I am writing a book on the topic of local, personal, and private AIs.

mark_l_watson 2y ago
Another thought: OpenAI has done a good enough job productizing ChatGPT with advanced voice mode and now also integrated web search. I don’t know if I would trust OpenAI with access to my Apple iCloud data, Google data, my private GitHub repositories, etc., but given their history of effective productization, they could be a multi-OS/platform contender. Still, I would really prefer everything running under my own control.
- alexander2002 2y ago
  Who can trust a company whose name contradicts its presence.
  - mark_l_watson 2y ago
    I don’t disagree with you!
  - arcanemachiner 2y ago
    Not me. I learned that lesson after I tried to take a bite out of my Apple Macintosh.
    - blharr 2y ago
      Well, clearly the apple already has a bite taken out of it, so that's user error
  - sumedh 2y ago
    You are getting access to some of the best AI tools for free, by that definition isnt that open?
zerop 2y ago
Have you tried RAG on Open WebUI. How does it do in asking questions from source docs?
- mark_l_watson 2y ago
  Not yet. It has ‘Knowledge sources’ that you can set up, and I think that supplies data for built in RAG - but I am not sure until I try it.
tracerbulletx 2y ago
I wrote a script to queue and manage running llama vision on all my images and writing the results to an sqlite db used by my Media Viewer, and now I can do text or vector search on it. It's cool to not have to rely on Apple or Google to index my images and obfuscate how they're doing it from me. Next I'm going to work on a pipeline for doing more complex things like multiple frames in a video, doing multiple passes with llama vision or other models to separate out the OCR, description, and object, people recognition. Eventually I want to feed all of this in here https://lowkeyviewer.com/ https://lowkeyviewer.com/ and have the ability to manually curate the automated classifications and text.
- Eisenstein 2y ago
  I'm curious why you find descriptions of images useful for searching. I developed a similar flow and ended up embedding keywords into the image metadata instead. It makes them easily searchable and not tied to any databases, and it is faster (dealing with tens of thousands of images personally). * https://github.com/jabberjabberjabber/LLavaImageTagger https://github.com/jabberjabberjabber/LLavaImageTagger
  - tracerbulletx 2y ago
    It's not as good as tags but it does pretty ok for now especially since searching for specific text in an image is something I want to do a lot. I'm trying to work on getting llama to output according to a user defined tagging vocabulary/taxonomy and ideally learn from manual classifications. Kind of a work in progress there. This is the prompt I've been using. "Create a structured list of all of the people and things in the image and their main properties. Include a section transcribing any text. Include a section describing if the image is a photo, comic, art, or screenshot. Do not try to interpret, infer, or give subjective opinions. Only give direct, literal, objective descriptions of what you see."
    - Eisenstein 2y ago
      > I'm trying to work on getting llama to output according to a user defined tagging vocabulary/taxonomy and ideally learn from manual classifications. Kind of a work in progress there. Good luck with that. The only thing that I found that works is using gbnf to force it, which slows inference down considerably.
bboygravity 2y ago
can llama 3.3 vision do things like "there's a textbox/form field at location 1000, 800 with label "address"" ? I did a quick and dirty prototype with Claud for this, but it returned everything with an offset and/or scaled. Would be a killer app to be able to auto-fill any form using OCR.
- MaxLeiter 2y ago
  Were you using claude’s computer mode? It can do this
  - bboygravity 2y ago
    No, I used the regular Claude which can also (somewhat) do this and uses the same image processing backend as "computer use" as far as I know (source: Antropic CEO interview with Lex Friedman recently). Computer use is also not very good at it (often mis-clicking for example). I'm guessing this will work flawlessly within 6 months to a year or so, but it doesn't seem ready yet.
honestAbe22 2y ago
Open source frontend??? Wtf. Show your code. This post is bs. AI coding has not advanced software in any meaningful way and will make coders dumber

mark_l_watson 2y ago

Another thought: OpenAI has done a good enough job productizing ChatGPT with advanced voice mode and now also integrated web search. I don’t know if I would trust OpenAI with access to my Apple iCloud data, Google data, my private GitHub repositories, etc., but given their history of effective productization, they could be a multi-OS/platform contender. Still, I would really prefer everything running under my own control.

alexander2002 2y ago
Who can trust a company whose name contradicts its presence.
- mark_l_watson 2y ago
  I don’t disagree with you!
- arcanemachiner 2y ago
  Not me. I learned that lesson after I tried to take a bite out of my Apple Macintosh.
  - blharr 2y ago
    Well, clearly the apple already has a bite taken out of it, so that's user error
- sumedh 2y ago
  You are getting access to some of the best AI tools for free, by that definition isnt that open?

zerop 2y ago

Have you tried RAG on Open WebUI. How does it do in asking questions from source docs?

mark_l_watson 2y ago
Not yet. It has ‘Knowledge sources’ that you can set up, and I think that supplies data for built in RAG - but I am not sure until I try it.

tracerbulletx 2y ago

I wrote a script to queue and manage running llama vision on all my images and writing the results to an sqlite db used by my Media Viewer, and now I can do text or vector search on it. It's cool to not have to rely on Apple or Google to index my images and obfuscate how they're doing it from me. Next I'm going to work on a pipeline for doing more complex things like multiple frames in a video, doing multiple passes with llama vision or other models to separate out the OCR, description, and object, people recognition. Eventually I want to feed all of this in here https://lowkeyviewer.com/ https://lowkeyviewer.com/ and have the ability to manually curate the automated classifications and text.

Eisenstein 2y ago
I'm curious why you find descriptions of images useful for searching. I developed a similar flow and ended up embedding keywords into the image metadata instead. It makes them easily searchable and not tied to any databases, and it is faster (dealing with tens of thousands of images personally). * https://github.com/jabberjabberjabber/LLavaImageTagger https://github.com/jabberjabberjabber/LLavaImageTagger
- tracerbulletx 2y ago
  It's not as good as tags but it does pretty ok for now especially since searching for specific text in an image is something I want to do a lot. I'm trying to work on getting llama to output according to a user defined tagging vocabulary/taxonomy and ideally learn from manual classifications. Kind of a work in progress there. This is the prompt I've been using. "Create a structured list of all of the people and things in the image and their main properties. Include a section transcribing any text. Include a section describing if the image is a photo, comic, art, or screenshot. Do not try to interpret, infer, or give subjective opinions. Only give direct, literal, objective descriptions of what you see."
  - Eisenstein 2y ago
    > I'm trying to work on getting llama to output according to a user defined tagging vocabulary/taxonomy and ideally learn from manual classifications. Kind of a work in progress there. Good luck with that. The only thing that I found that works is using gbnf to force it, which slows inference down considerably.

bboygravity 2y ago

can llama 3.3 vision do things like "there's a textbox/form field at location 1000, 800 with label "address"" ? I did a quick and dirty prototype with Claud for this, but it returned everything with an offset and/or scaled. Would be a killer app to be able to auto-fill any form using OCR.

MaxLeiter 2y ago
Were you using claude’s computer mode? It can do this
- bboygravity 2y ago
  No, I used the regular Claude which can also (somewhat) do this and uses the same image processing backend as "computer use" as far as I know (source: Antropic CEO interview with Lex Friedman recently). Computer use is also not very good at it (often mis-clicking for example). I'm guessing this will work flawlessly within 6 months to a year or so, but it doesn't seem ready yet.

honestAbe22 2y ago

Open source frontend??? Wtf. Show your code. This post is bs. AI coding has not advanced software in any meaningful way and will make coders dumber