22 ms·
On a technical level, they're doing something really simple -- take BLIP2's ViT-L+Q-former, connect it to Vicuna-13B with a linear layer, and train just the ti
by 152334H 4y ago
On a technical level, they're doing something really simple -- take BLIP2's ViT-L+Q-former, connect it to Vicuna-13B with a
linear layer, and train just the tiny layer on some datasets of image-text pairs.
But the results are pretty amazing. It completely knocks Openflamingo && even the original blip2 models out of the park. And best of all, it arrived before OpenAI's GPT-4 Image Modality did. Real win for Open Source AI.
The repo's default inference code is kind of bad -- vicuna is loaded in fp16 so it can't fit on any consumer hardware. I created a PR on the repo to load it with int8, so hopefully by tomorrow it'll be runnable by 3090/4090 users.
I also developed a toy discord bot (https://github.com/152334H/MiniGPT-4-discord-bot https://github.com/152334H/MiniGPT-4-discord-bot) to show the model to some people, but inference is very slow so I doubt I'll be hosting it publicly.
- sho 4y ago> they're doing something really simple -- take BLIP2's ViT-L+Q-former, connect it to Vicuna-13B with a linear layer, and train just the tiny layer on some datasets of image-text pairs Oh yes. Simple! Jesus, this ML stuff makes a humble web dev like myself feel like a dog trying to read Tolstoy.
- mjburgess 4y agoIn practice, it's a lot more like web dev than you might imagine. The above means that the approach is web-dev like gluing, almost literally just, from existingliba import someop from existinglibb import anotherop from someaifw import glue a = someop(X) b = glue(a) Y = anotherop(b)
- marban 4y agoAnd repeat that ~4 times to make it look like LangChain
- guax 4y agoAnd just like webdev, each of those were done in a different platform and require arcane incantations and 5h of doc perusing to make it work on your system.
- KaoruAoiShiho 4y agoBuddy this ain't 2022 anymore, ask chatgpt (with a plugin that can read docs).
- MattPalmer1086 4y agoYou can just ask GPT how to do it. Much like a lot of web dev!
- Gravityloss 4y agoat some point someone makes a service where you can let AI take over your computer directly. Easier that way! Curling straight to shell taken to next level.
- vorticalbox 4y agoFound my next hobby project
- wrayjustin 4y agoSo...AutoGPT? Now with command-line access! Have fun :) https://github.com/Significant-Gravitas/Auto-GPT/ https://github.com/Significant-Gravitas/Auto-GPT/
- kerkeslager 4y agoAnd the code GPT gives you won't work, much like a lot of web dev? ;P
- pc86 4y agoMaybe it's because of how I use it, but the code ChatGPT gives me has always been super helpful and 99% correct. But, we have a policy at work not to use it for work product so I have to spend time changing enough of it where it's different, and I'm never copy/pasting anything. Enough changes to the structure and variables to make it sufficiently different that it can't be considered pasting company data into GPT, ask my question(s), see what comes back out, refactor/type manually into my IDE, test. I'd say one out of every 8-9 times I get something objectively wrong - a method that doesn't exist, something not compiling, etc. But it's faster than using google/DDG, especially with some prompting so that it just spits back code and not 5th-grade level explanatory paragraphs before and after. And well over half the time it does exactly what I need or sufficiently close that my initial refactoring step gets me the rest of the way.
- amelius 4y agoIt's more like gardening: 1. plant seed 2. ...wait a very long time... 3. observe completely unexpected but cool result The unexpected part of step 3 is what makes this very different from any kind of engineering, even webdev. Of course, there is a lot of engineering involved in good ML, but that is more comparable to agricultural engineering in the sense that it's just a lot of dumb plumbing that any engineer can do without knowledge of the actual application.
- tudorw 4y agocount me in :)
- kerkeslager 4y agoI mean, for me, the unexpected part of 3 is what got me into programming in general. The first time you type a mysterious incantation into an editor and a few more mysterious incantations into the console and the console prints "Hello, world" like it was supposed to, it's unexpected because it's hard to believe that any of this mysterious incantation stuff actually works at all. As you get better at programming you have to take on harder problems to create the surprise of something working, because you gain confidence, and as you gain confidence, you start expecting your code to work. It's only when you've compiled the thing 6 times with small corrections and gotten segfaults each time and the 7th time you finally find the place you weren't updating the pointer and you correct it, but this is the 7th error you've corrected without the segfault going away, so you don't really expect it to fix the problem, but then you run it and it's fixed! And then you get a job and the reality is that most of the jobs you're just writing CRUD apps and for a little while you can get some surprise out of learning the frameworks, but eventually you actually get really, really knowledgeable about the Postrgres/Django/React stack and nothing surprises you any more, but because nothing surprises you any more, you're really effective and you start being able to bill the big bucks but only for work on that stack because it takes time to struggle enough to get surprised, and the time that takes means your time is worth less to your clients. Money ruins everything. And if you don't do anything non-billable, it's easy to forget what programming felt like when you didn't know how your tools all worked inside and out. Not everyone takes this path but it's certainly the easiest path to take. I think for a lot of folks who have been doing this for a long time, the reason ML is so exciting is it's getting them back out of their comfort zone, and into a space where they can experience surprise again. But that surprise has always been available if you continue to find areas of programming that push you out of your comfort zone. For me it's been writing compilers/interpreters for programming languages. Crafting Interpreters was awesome: for the first time I benchmarked a program written in my language against a Python program, and my program was faster: I never expected I'd be able to do that! More recently, I wrote a generational GC. It's... way too memory-intensive to be used in my language which uses one-GC-per-thread for potentially millions of threads, but it certainly was a surprise when that worked. Personally, I'm keeping track of ML enough to know broad strokes of things but I'm not getting my hands dirty with code until there are some giants to stand on the shoulders of. Those may already exist but it's not clear who they are yet. And I've got very little interest in plugging together opaque API components; I know how to make an API call. I want to write the model code and train it myself.
- teruakohatu 4y agoThere is a little more to it than that. Abstractions in ML are very leaky.
- MrGilbert 4y agoJust get rid of all the abbreviations in your mind - they seem to be very intimidating. I really liked the explanation that Stephen Wolfram did on ChatGPT: https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-doing-and-why-does-it-work/ https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-... Maybe someone has resources to understand machine-learning on an ELI5 level.
- wwalexander 4y agoWow, he waits until halfway through the article to mention A New Kind of Science. Usually he works it into the first couple of paragraphs!
- alchemist1e9 4y agoI known it’s hard to believe but I sense LLMs have slightly knocked his ego down and injected a small dose of humility. https://youtu.be/z5WZhCBRDpU https://youtu.be/z5WZhCBRDpU I pick that up in above video and also in the post above. Definitely healthy for him which just to be clear I’m a huge Wolfram fan and the ego doesn’t really bother me, it’s just part of who he is, however I do find it nice that LLMs are having him self reflect more than typical.
- HarHarVeryFunny 4y agoNot a big Wolfram fan myself. I gave him the benefit of the doubt and bought "A New Kind of Science" (freakin' expensive when it first came out), and read the whole 1280 pages cover to cover ... Would have been better presented as a short blog post. I find it funny how despite being completely uninvolved in ChatGPT he felt the need to inject himself into the conversation and write a book about it. I guess it's the sort of important stuff that he felt an important person like himself should be educating the plebes on. Predictably he had no insight into it and will have left the plebes thinking it's something related to MNIST and cat-detection.
- MrGilbert 4y agoI just happen to read this article of him, which I found easy to understand. I'm neither a huge proponent nor opponent of the likes of his work. Or, bluntly speaking: I don’t know much else about his reputation in the community.
- selfhoster11 4y agoSeriously, ChatGPT was the thing that gave me a foothold into the AI/machine learning world... because it gave me hope that a mere mortal can achieve something reasonable with this tech without a crazy amount of work and educational background.
- 152334H 4y agoI don't have much experience myself. I only started ~10 months ago -- just a month or two before Stable Diffusion. You just have to do it every day. It's fun!
- jack_riminton 4y agoCan you recommend what kind of small daily activities would help a web dev get into it?
- 152334H 4y agoRegardless of what you want to learn, "small daily activities" is a bit hard. You can learn some stuff by osmosis, following the feeds of AI devs && AI channels, but the bulk of what I learn comes from starting projects & digging into code & reading papers. If you can hold attention span over several days (I can't), work on a project bit-by-bit. Just make sure it uses modern AI stuff, and that you have smart people to talk around with.
- anonzzzies 4y agoThere are really great resources now from eli5 about all of this tech to books like ‘the little learner’ which any programmer can get into. Yes, it takes effort but it is a great time for it.
- relativeadv 4y ago> like a dog trying to read Tolstoy this got a chuckle out loud from me. great visual.
- egeozcan 4y agoThis could be a great prompt to test the limits of txt2img models. The astronaut riding a horse got boring already :)
- mewpmewp2 4y agoOnly because of big complicated sounding terms, that also exist in web dev.
- blurbleblurble 4y agoArf!
- pansa2 4y agoWeb stuff probably makes ML devs feel the same way. ML is just a different field, using a different set of technologies from those you’re familiar with.
- tourgen 4y ago[dead]
- spaceman_2020 4y agoThe best ML PhDs can’t do what frontend devs can: understand CSS :D
- ikurei 4y ago> This ML stuff makes a humble web dev like myself feel like a dog trying to read Tolstoy. Just like any discussion between advanced web devs would make any humble woodworker feel? And just like any discussion between advanced woodworkers would make a humble web dev feel? "It's really simple, they're just using a No. 7 jointer plane with a high-angle frog and a PM-V11 blade to flatten those curly birch boards, then a No. 4 smoother plane with a Norris-type adjuster and a toothed blade for the final pass." Whut? "You could use Webpack to bundle your HTML, CSS and Babel-transpiled TypeScript 5 down to shim-included Ecmascript 4", "They're just using OAuth2 authentication with Passport.js and JWT tokens, which easily gets you CSRF protection", "Our e-learning platform uses LMS.js and xAPI.js, plus SCORM for course packaging and Moodle as the LMS backend.", ... There was a time you didn't know what any of that meant. Just because you don't know what the words mean shouldn't make it sound difficult. Not saying AI is easy, just that the jargon is not a good indication of difficulty and we should know better than to be so easily mystified.
- birdyrooster 4y agoI love your analysis.
- SpaceL10n 4y agoWeb devs have become blue collar!? =P Great idea, actually. I do hope for a curriculum that enables kids on the trade school path to learn more about programming. Why not Master/Journeyman/Apprentice style learning for web dev??
- joaogui1 4y agoFWIW I work in LLMs and I consistently fail to do simple webdev stuff
- EntrePrescott 3y agoMaybe you're just holding it wrong: You're not supposed to let your LLM rest or chat idly while you do the webdev stuff yourself, but to make your LLM do the webdev stuff for you ;P
- pc86 4y agoBig "a monad is just a monoid in the category of endofunctors" vibes from this one.
- nullsense 3y agoBut that's literally what it is...
- craigching 4y agoI was where you're at about ... oh wow, it's been almost ten years since I jumped into machine learning. Mind you, I've been learning on the side most of this time other than a theoretical class at the University of Minnesota. But, that aside, and depending on where you're at in your understanding, this is a great resource for catching up if you're really interested: https://karpathy.ai/zero-to-hero.html https://karpathy.ai/zero-to-hero.html it was posted on HN a couple of weeks ago and I have to say it's a really good introduction and Andrej Karpathy is a passionate and excellent teacher. You may want to brush up on some intro Calculus, but it's very understandable.
- thewarrior 3y agoI've only been reading ML stuff for a few months and I kind of understand what it's saying. This stuff isn't as complex as its made out to be. It's just a bunch of black boxes AKA "pure functions". BLIP2's ViT-L+Q-former AKA //I give you a picture of a plate of lobster it will say "A plate of lobster". getTextFromImage(image) -> Text Vicuna-13B AKA //I give you a prompt and you return completion ChatGPT style getCompletionFromPrompt(text) -> Text We want to take the output of the first one and then feed in a prompt to the LLM (Vicuna) that will help answer a question about the image. However the datatypes don't match. Lets add in a mapper. getAnswerToQuestion(image, question) -> answer text = getTextFromImage(image) prompt = mapTextToPrompt(text) return getCompletionForPrompt(prompt) Now where did this mapTextToPrompt come from ? This is the magic of ML. We can just "learn" this function from data. And they plugged in a "simple" layer and learned it from a few examples of (image , question) -> answer. This is what frameworks like Keras, Pytorch allow you to do. You can wire up these black boxes with some intermediate layers and pass in a bunch of data and voila you have a new model. This is called differentiable programming. The thing is you don't need to convert to text and then map back into numbers to feed into the LLM. You skip that and use the numbers it outputs and multiply directly with an intermediate matrix. getAnswerToQuestion(image, question) -> answer text = getEmbeddingFromImage(image) embedding = mapEmbeddingToInputEmbeddingForLLM(text) return getCompletionForEmbedding(embedding) Congratulations you now understood that sentence.
- eternauta3k 3y agoInteresting, so the LLM is "just" getting your question plus a normal text description of the image (as vectors)?
- thewarrior 3y agoAt a high level yes. More precisely - It gets the question After irs passed through a matrix that transforms the text description of the image so the LLM can “understand” it. It maps from the space of one ML model to the other.
- artificial 3y ago
- visarga 3y ago> take BLIP2's ViT-L+Q-former This thing takes an image and creates a representation matrix. > connect it to Vicuna-13B with a linear layer Vicuna is an open LLM, pretty good quality, not as good as GPT3.5 though. This is the beautiful part - a mere multiplication is enough to convert the image tensor to text tensor. One freaking line of code, and a simple one. > and train just the tiny layer on some datasets of image-text pairs You then get a shitload of image-text pairs and train the model to describe the images in text. But keep both the image and text model frozen. Is that hard? No, just flip a flag. So this "linear projection layer" (a matrix multiplication) is the only learned part. That means it takes less time to train, needs fewer examples and requires less memory. Training the image and text models was much more difficult. But here we don't train these models, they use them as ready-made parts. It's a hack on top of two unrelated models, so it is cheap. In the end the finishing touches - they label 3500 high quality image-text pairs, and fine-tune on them. Now the model becomes truly amazing. It has broad visual intelligence, and scooped OpenAI who didn't release Image GPT-4 in the APIs yet. The important lesson to take is that unrelated models can be composed together with a bit of extra training for the glue model. And that open AI is just as powerful as "Open"AI sometimes. It's breathing down their necks, just one step behind. This model is also significant for applications - it can power many automations in a flexible way.
- rafaelero 3y ago> This is the beautiful part - a mere multiplication is enough to convert the image tensor to text tensor. One freaking line of code, and a simple one. I thought they were creating image tokens based on the queries during finetuning and appending them to the language model. They are not text tokens.
- kramerger 4y agoCan any of this realistically run on CPU at some point? (Not training obviously)
- GaggiX 4y agoI've run LLaMa models on my CPU before, ViT-L and the Q-former are two transformer models as well, so I can't see why they wouldn't run on a CPU.
- Ambix 4y agoI'm developing framework [1] in Golang with this goal in mind :) It successfully runs relatively big LLM right now, and diffusion models will be the next step [1] https://github.com/gotzmann/llama.go/ https://github.com/gotzmann/llama.go/
- anonzzzies 4y agoIt will do, probably quite soon. Many people are trying.
- muyuu 4y agoquantized Vicuna runs ok-ish in my 16GB i7 laptop (onboard graphics) and the output is usable see this comparison: https://old.reddit.com/r/LocalLLaMA/comments/12ezcly/comparing_models_gpt4xalpaca_vicuna_and_oasst/ https://old.reddit.com/r/LocalLLaMA/comments/12ezcly/compari... these models quantised to 4bit should run in CPU set ups with 16GB of RAM + 16GB of swap (Linux) and perhaps other setups run similarly
- tantony 4y agoSomeone is probably going to port it to llama.cpp soon.
- lhl 4y agoYes, you can run inference at decent speeds on CPU with llama.cpp. A token is about 0.75/words, so you can see lots of people getting 4-8 words/s on their CPUs: https://github.com/ggerganov/llama.cpp/issues/34 https://github.com/ggerganov/llama.cpp/issues/34 There a lot of optimizations that can be done. Here's one w/ potentially a 15X AVX speedup for example: https://github.com/ggerganov/llama.cpp/pull/996 https://github.com/ggerganov/llama.cpp/pull/996
- HopenHeyHi 4y agoThanks for a useful comment. Do you reckon the 4bit quantized Vicuna just won't do here? https://huggingface.co/anon8231489123/vicuna-13b-GPTQ-4bit-128g https://huggingface.co/anon8231489123/vicuna-13b-GPTQ-4bit-1... I think with this everything OpenAI demonstrated ~5 weeks ago has been recreated by actually-open AI. Even if it runs much much slower on prosumer hardware and with less good results at least it is de-magicked.
- p1esk 4y agoThe magic is in the quality of GPT-4 output. That hasn’t been recreated yet.
- 152334H 4y agoIt'll work! I just haven't touched any of the 4bit stuff myself, so I don't personally know how to add it. Great low-hanging fruit for anyone else to take on.
- kolinko 3y agoOpen AI still hasn't exactly reached the level of gpt3.5. GPT-4 is way ahead of anything.
- chenzhekl 4y agoIt sounds like a BLIP2 with an extra linear layer for finetuning (or aligning the Q-former with a new LLM?). What makes it more powerful than BLIP2?
- 152334H 4y agoIt's better because 1. it's using vicuna as a base. 2. It has a pretty high quality fine-tuning dataset. I initially missed this, and it's a very important advantage. 3. (speculatively) it doesn't collapse to extremely short responses (which BLIP2 and other models trained on image-text caption pairs) because of how small/simple the adapter is. I was interested in training a BLIP2-LLaMA model before this, and I might still do it just to test (3).
- collaborative 4y agoWith fp8, would 4GB be enough or is 6GB more like it?
- eurekin 4y ago> I created a PR on the repo to load it with int8, so hopefully by tomorrow it'll be runnable by 3090/4090 users. How about 2x3090? Can it be run on multiple gpus?
- cs702 4y agoIndeed, really simple. And yes, the results are shockingly good. But what I find most remarkable about this is that the ViT-L+Q-former's hidden states are related by only a linear projection (plus bias) to the Vicuna-13B's token embeddings: emb_in_vicuna_space = emb_in_qformer_space @ W + B These two models are trained independently of each other, on very different data (RGB images vs integer token ids representing subwords), and yet somehow they learn to embed different data in feature vectors that are so... similar. WHY should that be the case? It suggests to me there may be something universal about the embedding layers and hidden states of all trained deep learning models.
- AbrahamParangi 4y agoI think it’s just that affine transforms in high dimensions are surprisingly expressive. Since the functions are sparsely defined they’re much less constrained compared to the low dimensional affine transformations we usually think of.
- cs702 4y agoGood point. Didn't think of that. It's a plausible explanation here, because the dimensionality of the spaces is so different, 5120 vs 768. Not surprisingly, the trained weight matrix has rank 768: it's using every feature in the lower-dimensional space. Still, it's kind of shocking that it works so well! I'd be curious to see if the learned weight matrix ends up being full-rank (or close to full-rank) if both spaces have the same dimensionality.
- visarga 3y agoThey would have full-rank because all the embedding space is used. There are no unused large pockets.
- cs702 3y agoThe weight matrix's rank would decrease for each feature in the target space that cannot be expressed as as a linear combination of features in the input space (plus a bias). For example, if the target space has a feature representing a non-visual quality like "smelliness," it would not be expressible as a linear combination of features representing visual attributes like "redness," "blueness," and "greenness," etc. in the input space. If both spaces have the same dimensionality, the learned weight matrix would be full-rank only if every feature in the target space is expressible as a linear combination of features in the input space (plus a bias). Which brings me back to my original question: WHY would that be the case when the two models are trained independently on data that is so different?
- seydor 4y agoThen it's an impressive demonstration of how modular neural networks can be. Maybe we don't even need to train monoliths
- artificial 3y agoMaybe a distributed trainer? AI@Home?
- matthewfcarlson 3y ago>so hopefully by tomorrow it'll be runnable by 3090/4090 users. Taking a step back, this is just a wild statement. I know there's some doom and gloom out there, but in certain aspects, it's an awesome time to be alive.
- jxy 3y agoThis opens a huge possibilities. It's likely we could simply plug in stable diffusion using a linear layer. As well as whisper and some TTS. Getting a back to back mixed image/sound/text engine running on a laptop. I wonder if there's powerful enough ViT model that does OCR.