8 ms·
Dragonfly: A large vision-language model with multi-resolution zoom
- ilaksh 2y agoI have been testing out LLMs with the together.ai API, but I can't figure out how to use the multimodal models with the API. I don't see any in their model list.
- cateye 2y agoMe too. I use now https://fireworks.ai/ https://fireworks.ai/ for vision models to test.
- GaggiX 2y agoIs there a demo or API to test the model? There are so many vision language models these days, it's hard to say which one is better, they also use in many cases different benchmarks.
- alexey-salmin 2y agoHuggingface links are in the article https://huggingface.co/togethercomputer/Llama-3-8B-Dragonfly-v1 https://huggingface.co/togethercomputer/Llama-3-8B-Dragonfly...
- GaggiX 2y agoI wasn't asking about running the model locally, also for that I have to wait for someone to quantize the model.
- vessenes 2y agotogether.ai started as an API-based inference provider, so I bet you can target these models directly with your API credits there.
- ilaksh 2y agoHow? I don't see any vision-language model in their model list for the API.
- achristmascarl 2y agoFor the model fine-tuned on biomedical image data, does anyone with domain knowledge know how the model's answers compare to the "Gold" answers?
- TechDebtDevin 2y agoI don't have exact domain knowledge but I'm fairly certain this type of tech has already been employed to do some of the heavy lifting for radiologists reviewing imaging results.
- rrsp 2y agoBoth the ‘gold’ answer and the model reference a PA and AP view respectively as well as a lateral chest radiograph. The picture only contains a lateral radiograph though.
- quantumwoke 2y agoRanges from weird phrasing to outright incorrect. Oh well, early days
- TechDebtDevin 2y agoIve been sorta following together.ai for a while. Cool company. Is this available to be used by anyone atm? Could I potentially use the model to look at my own chest xrays (I've had a lot)?
- imjonse 2y agoThey have released the models https://huggingface.co/togethercomputer/Llama-3-8B-Dragonfly-Med-v1 https://huggingface.co/togethercomputer/Llama-3-8B-Dragonfly...
- darby_nine 2y agoI can't speak for others obviously, but this sort of caption is nauseous: > In the heart of a vibrant skatepark, a skateboarder is caught in a moment of pure exhilaration. The skateboarder, dressed in a black t-shirt adorned with a yellow graphic and black pants, is suspended in mid-air, performing an impressive trick on a concrete ramp. The skateboarder's arms are outstretched, adding balance to the daring stunt. The skatepark itself is a concrete playground, with the skateboarder's ramp being the main focus. In the background, palm trees sway gently, adding a touch of nature to the urban setting. A few spectators can be seen in the distance, their attention riveted on the airborne skateboarder. The image captures not just a moment, but a story of skill, courage, and the joy of skateboarding. This seems a lot more like a puff piece from a local publisher trying to fill space, or description of a stock photo to an advertiser, than a description I'd describe as accurate from a human to another human.
- deleted 2y ago[deleted]
- esafak 2y agoThat's the price of getting an image that accurately represents what you had in mind. Otherwise you could just prompt it with "skateboarder in a skatepark".
- Jackson__ 2y ago>In the heart of a vibrant skatepark, Vibrant? It's made of white concrete. >a skateboarder is caught in a moment of pure exhilaration. Oh, great we're mind-reading from pictures now? His face isn't even visible. > adding balance to the daring stunt. What incredible commentary. >palm trees sway gently, adding a touch of nature to the urban setting. The trees aren't swaying. But because sometimes trees sway, I guess it'll just say it. > A few spectators can be seen in the distance, their attention riveted on the airborne skateboarder The spectators look more like they are just glancing over, nowhere near "riveted". > The image captures not just a moment, but a story of skill, courage, and the joy of skateboarding. This adds literally nothing.
- davidhyde 2y ago> “ Question: Write a detailed radiology note based on the chest X-ray. Gold Answer: AP upright and lateral views of the chest were provided. Left chest wall pacer pack is again seen with leads extending into the right heart. ” The bit about a “wall pacer pack is again seen…” leads me to believe this was based on another doctors note about a similar looking X-ray which was probably paired with other information like another scan at the time. That would be problematic imo.
- BriggyDwiggs42 2y agoProblematic for the functionality? If it works well enough, I’m pretty fine with them stealing data to create a useful medical tool.
- davidhyde 2y agoIt’s problematic because the LLM is describing another person’s scan and not the one presented to it. It should at least present the other scan as workings and the percentage difference between the two. Finding a similar looking scan is very useful no doubt but if the result is hallucinated that it is less so. Dangerous even. There is no confidence percentage and there should be.
- planb 2y agoThe Gold Answer is not the output of the model but the expected answer in the benchmark. Probably the benachmark contains multiple consecutive images of the same patient.
- davidhyde 2y agoThanks, I probably shouldn’t be commenting on an LLM article if I don’t even know what a gold answer is! Oh well.
- esafak 2y agoIs there a comparable service for audio analysis?
- jerpint 2y agoNot yet, some models in that direction are LTU-AS
- cateye 2y agoIt is strange that this model is not available on Together.ai to try it out after reading the blog artcile.
- stainablesteel 2y agothis looks quite impressive if image generation gets to be near perfect then it might have a larger impact on communication than gpt does, no paragraph beats a good diagram but drawing is always hard
- deleted 2y ago[deleted]