5 ms·
Llama 3-V: Matching GPT4-V with a 100x smaller model and 500 dollars
- doctorpangloss 2y agoShouldn't CogAgent be in this comparison?
- m00x 2y agoCogVLM should be, not sure how CogAgent plays into this. This isn't an agent.
- doctorpangloss 2y agoYou would use CogAgent in VQA mode. Why would someone downvote suggesting to test one of the most powerful multimodal LLMs? Because it doesn't have "V" in its name? CogAgent is improved on many tasks compared to CogVLM.
- m00x 2y agoI didn't downvote, only replied. CogAgent is also CogVLM modified to handle documents and larger images. CogVLM is better for VQA.
- lanceflt 2y ago- Llava is not the SOTA open VLM, InternVL-1.5 is https://huggingface.co/spaces/opencompass/open_vlm_leaderboard https://huggingface.co/spaces/opencompass/open_vlm_leaderboa... You need to compare the evals to strong open VLMs including this and CogVLM - This is not "first-ever multimodal model built on top of Llama3", there's already a Llava on Llama3-8b https://huggingface.co/lmms-lab https://huggingface.co/lmms-lab
- gigel82 2y agoLike InternVL, no llama.cpp support severely limits its applications. Close to GPT4v performance level and runnable locally on any machine (no need for a GPU) would be huge for the accessibility community.
- valine 2y agoVery curious how it performs on OCR tasks compared to InternVL. To be competitive at reading text you need tiling support, and InternVL does tiles exceptionally well.
- hovering_nox 2y agoI think CogVLM2 is even better than Intern at OCR (my usecase is extracting information from an invoice)
- mkesper 2y agoAfter some superficial testing I with bad quality scans you can find on kaggle I can not confirm that. CogVLM2 refuses to handle scans that InternVL-V1.5 still can comprehend.
- yieldcrv 2y agoI’m going to be saying First Ever AI something for the next 15 years for clout and capital, not going to be listening to anybody’s complicated ten step funnel if they’re not doing the obvious
- abrichr 2y agoThank you for the link! Our initial testing suggests MiniCPM outperforms InternVL for GUI understanding: https://github.com/OpenAdaptAI/OpenAdapt/issues/637#issuecomment-2136403745 https://github.com/OpenAdaptAI/OpenAdapt/issues/637#issuecom... (InternVL appears to hallucinate more.)
- behnamoh 2y agoThis "matching gpt-4" catchy phrase has lost its meaning to me. Everytime an article like this pops up, I see marketing buzz and unrealistic results in practice.
- mpalmer 2y agoFor me it's become a signal the person making the claim is unserious.
- moffkalast 2y agoIf "beats GPT 4" is in the title it's almost a guarantee that it's a bold faced lie that includes benchmark overfitting. The first time a model that actually matched GPT 4 launched (i.e. Command-R+) there was no mention of it at all. If your results speak for themselves, there's no need to shout.
- Mo3 2y agoOf course, it's nothing else. Who could possibly believe that OpenAI and others would dump billions into development and training and aren't smart enough to figure out they could also do it with $500.
- whimsicalism 2y agoit would have been a lot cheaper for oai if they had access to llama3 in 2018
- KorematsuFredt 2y agoYou have clearly not read the article. $500 is the cost of fine tuning.
- selcuka 2y agoFair enough. Is it now safe to say that OpenAI could have done with a 8B model + $500 of fine tuning instead of running a (much) larger model on their GPU cluster?
- KTibow 2y agoIs there a reason Phi Vision is omitted?
- cadence- 2y agoIs there any place that currently hosts phi3 Vision and provides API access to it? I cannot run it on my local machine, unfortunately.
- ai_what 2y agoNvidia has a 1000 free credits API for phi3 vision. You only need an email address. https://build.nvidia.com/microsoft/phi-3-vision-128k-instruct https://build.nvidia.com/microsoft/phi-3-vision-128k-instruc...
- cadence- 2y agoBeautiful. Thank you.
- trog 2y agoWas also looking for something like this - I can't find pricing listed anywhere for their API usage, only the free 1,000 credits - or am I completely misunderstanding how this works?
- cpursley 2y agoI can’t find the pricing either. I’m interested, the demo worked well.
- KTibow 2y agoAzure has a playground, I haven't tried to use it with an API though. https://ai.azure.com/explore/models/Phi-3-vision-128k-instruct/version/1/registry/azureml https://ai.azure.com/explore/models/Phi-3-vision-128k-instru...
- yeldarb 2y agoDon't see a license listed in the repo; presumably needs to be the same as Meta's Llama 3 license?
- arnaudsm 2y agoTangential question : did anyone ever use GPT4-V in production in visual tasks? It's never consistent enough for me to be useful
- zacmps 2y agoNope, I tried it for graph and diagram understanding and it wasn't good enough. Planning to repeat the evaluation with 4o when I have time.
- SparkyMcUnicorn 2y agoI'm using 4o to convert visual diagrams into mermaid, and it's been almost perfectly accurate in my experience.
- cuu508 2y agoThis is the out of the box thinking I love about HN. What do you do with the mermaid?
- SparkyMcUnicorn 2y agoThe resulting mermaid is used for... more LLM processing. Converting to mermaid first is more cost-effective, consistent, and accurate for my purposes.
- serjester 2y agoDon’t use it for anything OCR related that needs perfect accuracy. Stuff where some errors are ok, we’ve had great success. Depending on your budget, you can also run it multiple times to catch errors.
- toomuchtodo 2y agoHow does it compare to Tesseract? Edit: Thank you!
- Havoc 2y agoOh wow. I was expecting it to be the 70B one as base given those stats
- vikrantrathore 2y agoHow does it compare with MiniCPM-Llama3-V 2.5 [0]? Based on what I see it seems much better than Llama 3-V on the benchmarks. Also it can directly be tried on Huggingface Spaces to check the performance [1]. It has the dataset, code and fine-tuning details with screenshots of it running on Xiaomi 14 pro. It has strong OCR performance and supports 30+ languages. [0] https://github.com/OpenBMB/MiniCPM-V https://github.com/OpenBMB/MiniCPM-V [1] https://huggingface.co/spaces/openbmb/MiniCPM-Llama3-V-2_5 https://huggingface.co/spaces/openbmb/MiniCPM-Llama3-V-2_5
- cpursley 2y agoWoah, this actually did quite well on table data extraction. I wonder how this could be used for long documents. Maybe paired with some kind of hybrid rag approach.
- wy35 2y agoThis aged well... https://github.com/OpenBMB/MiniCPM-V/issues/196 https://github.com/OpenBMB/MiniCPM-V/issues/196
- dcreater 2y agoIf I had a nickel for every outrageous "matches/beats GPT-x" claim, I'd have more money than the capital these projects raise from VC. This absolutely is not the first Llama3 vision model. They even quote it's performance compared to Llava. Hard to take anything they say seriously with such obviously false claims
- qeternity 2y ago> This absolutely is not the first Llama3 vision model. They even quote it's performance compared to Llava. Although this is true, there have been earlier Llama3 based vision releases, none of the latest Llava releases are Llama3 based.
- abbaselmas 2y agohttps://ollama.com/library/llava-llama3 https://ollama.com/library/llava-llama3 llava-llama3
- CGamesPlay 2y agoThis appears to be a Llava model which was then fine-tuned using outputs from Llama 3. If I understand correctly, that would make it Llama-2-based.
- GaggiX 2y ago>fine-tuned using outputs from Llama 3. Llama 3 outputs text and can only see text, this is a vision model. >that would make it Llama-2-based. It's based on Llama 3, Llama 2 has nothing to do with it. They took Llama 3 Instruct and CLIP-ViT-Large-patch14-336, train the projection layer first and then later finetuned the Llama 3 checkpoint and train a LoRA for the ViT.
- qeternity 2y agoThat is someone else who has just used the Llava name. It is not by the original group who have published a series of models under the Llava name.
- deleted 2y ago[deleted]
- 2Gkashmiri 2y agoIs there a local small llm that can OCR images or haabdwritten invoices ? Traditional OCR do not handle multiple invoice formats or handwritten ones. I would like to train one locally with as many invoices it wants
- itships 2y agoIf you find one, I’d greatly appreciate it if you’d share that info.
- anais9 2y agoWould love to see Ollama support for this - seems promising given my experience with LLaVA so far and would love to get some hands on head to head experience
- deleted 2y ago[deleted]