10 ms·
Microsoft Kosmos-1: A Multimodal Large Language Model
- solarist 4y agoPaper: https://arXiv.org/abs/2302.14045 https://arXiv.org/abs/2302.14045 Examples: https://twitter.com/alphasignalai/status/1630651280019292161 https://twitter.com/alphasignalai/status/1630651280019292161
- ducktective 4y agoIt can even solve IQ tests...I mean, how much further are we moving the goal post? Is there a model that can solve differential equations symbolically and numerically? Most of modern engineering just boils down to diff.eqs whether ordinary or partial. It's our current best method to reason about stuff and control them.
- scotty79 4y agoYeah, that could be good. I think LLMs will start to be really useful when they start to do math at human level. When this happens, the sky is the limit.
- Y_Y 4y agoIt's pretty big by any standards, but you may find the work of Gradshteyn and Ryzhik solves this problem nicely.
- didntreadarticl 4y agoIts crap at the visual Raven IQ test though, it scores 22% vs an algorithm that takes random guesses scoring 17%.
- kalium-xyz 4y agoBet you 5 bucks I can train one that gets 100%. Just gotta train it on the ravens answer key.
- thenaturalist 4y agoI'd be cautiuous with such general statements given the rapid pace of development in this area. Benchmark shelf lives aren't that long. You ommitted the fact that tuning bumped it to 26% vs random. Sure, questionable what effort is involved in that step, but at the same time, that hints to me that tuning will be the new baseline within the next 12-24 months.
- didntreadarticl 4y agoSure I would expect it to improve. But it was a bit fishy how 'it took an IQ test!' is in all the highlights but then they mumble quietly about the score that it actually got and hope no-one is listening to that bit. Its notable that it was able to attempt it at all I suppose.
- nmarinov 4y agoSemi related, there's a (pretty good) course at OMSCS where the main project is building an agent to solve RPM problems: https://lucylabs.gatech.edu/kbai/spring-2023/project-overview/ https://lucylabs.gatech.edu/kbai/spring-2023/project-overvie... And quite a lot of papers about that: https://scholar.google.com/scholar?q=%22raven%27s+progressive+matrices%22+%22artificial+intelligence%22&hl=en&as_sdt=0,11 https://scholar.google.com/scholar?q=%22raven%27s+progressiv...
- seydor 4y agoI prefer not focusing on games and benchmarks. Hopefully we ll get to robotics soon
- moffkalast 4y agoRoger roger.
- brookst 4y agoArnold was great in that documentary!
- mhh__ 4y agoWriting down the equations is the task for the AI.
- bootsmann 4y ago> It can even solve IQ tests...I mean, how much further are we moving the goal post? The problem with test like this is that when trained on the existing big datasets (commoncrawl etc.), chances are the test is already in the input so the validation is not proper. Its the same thing with all the "AI beats SAT" headlines. The exercises for those very tests exist all over the internet already.
- nayroclade 4y agoIt's well documented that these models can solve variations of questions that are not found anywhere in their training set, and even entirely novel problems invented by prompters. Not with 100% success, but they can do it far with a rate far better than chance, so the idea that they're pulling responses from their training data is simply not correct.
- 331c8c71 4y agoWell, my experience was that ChatGPT produced utter nonsense including literally "1/19+1/19=2/19=1" when asked a simple probability question.
- shaunsingh0207 4y agoIQ test, the sat, and exact mathematics are very different fields though. ChatGPT etc. are very good at "emulating" by sheer force and size logic, exact math doesn't work with "emulation"
- nayroclade 4y agoTranscript of a conversation I just had with ChatGPT: Prompt: I'm talking with a user named "331c8c71" on Hacker News. According to statistics I found by Googling, the number of Hacker News readers living in the USA is 36%. Assuming this is correct, what is the probability that 331c8c71 does not reside in the USA? Please explain your reasoning. Response: If we assume that the statistic you found is accurate, then we know that 36% of Hacker News readers live in the USA. Therefore, the probability that any given Hacker News reader does not live in the USA is 100% - 36% = 64%. However, we do not know if 331c8c71 is a typical Hacker News reader. They could be part of a demographic that is more or less likely to live in the USA. Without more information, we cannot say for certain what the probability is that 331c8c71 does not reside in the USA. So, the answer is 64% assuming that 331c8c71 is a typical Hacker News reader, but it may not be an accurate estimate depending on additional information about the user. --- Like I said, these models are far from 100% accurate, and there is lots they get wrong, but they clearly are capable of some kind of reasoning that goes beyond simple text substitution of training data.
- RivieraKid 4y agoWhat goalpost specifically?
- ducktective 4y agoSolving IQ tests which measure quantitative reasoning.
- RivieraKid 4y agoThere was no movement of this goalpost.
- espadrine 4y ago> I mean, how much further are we moving the goal post? Look at it this way: humans don’t have BPE-encoded text as input to their brain. It is ALL visual input. For AGI, you would at least need to add audio input as well. And be driven by action and reward. The learning capabilities of the brain are currently beyond the processing capabilities of current architectures. Just the notion of a model receiving only pixel data that contains a question and being able to output voice data that produces a correct answer, using no partial model trained on another corpus, is probably not tractable without significant improvements. But the models can be very useful without being AGI!
- pmontra 4y ago> It is ALL visual input. And sound, taste, smell, touch.
- konfusinomicon 4y agodon't forget humor
- KineticLensman 4y agophlegmatic, choleric, sanguine or melancholic?
- lupire 4y agoand doom, awe, timing, a word, absurd, in, horse, common, wonder, beauty, honor, duty, doubt, danger, fear, style, worth, direction, dread, non. https://m.youtube.com/watch?v=SKnnF5qA31I https://m.youtube.com/watch?v=SKnnF5qA31I
- konfusinomicon 4y agoTIL horse sense. I was thinking what on earth is the sense of horse?
- sillysaurusx 4y ago
- Tepix 4y agoIt wasn't very good at the IQ test. But yes, it is promising. "Although there is still a large performance gap between the current model and the average level of adults, KOSMOS-1 demonstrates the potential of MLLMs to perform zero-shot nonverbal reasoning by aligning perception with language models."
- Jevon23 4y ago>how much further are we moving the goal post? My goalpost for AGI is when Microsoft can fire their entire engineering staff, replace them with AI, and not notice any decrease in productivity or quality of output. This test is empirically verifiable (in principle). No need to argue over whether the AI scoring X% on Y assessment task is “truly” impressive or not.
- staticautomatic 4y agoYou mean that isn’t the Teams origin story?
- akiselev 4y agoTeams is the projection of a fourth dimension ancient Eldritch horror onto the mortal plane. AI had nothing to do with it.
- gfodor 4y agoYou’re confusing the goal - the goal here isn’t about the finish line but the point where people all concede that the finish line is actually reachable without any major, presently unthinkable advances.
- Jevon23 4y agoSurely anyone familiar with software engineering knows that the finish line is reached when the finish line is reached. And no sooner. Physicists at the beginning of the 20th century also thought that the finish line of physics was in sight and all that remained was tightening a few constants. Look how that turned out.
- gfodor 4y agoThis perspective is too reductionist - we make predictions of success all the time based on first principles reasoning. It’s perfectly sane to try to predict, and make good arguments, if AGI is possible to achieve without new breakthroughs.
- lumost 4y agoChatGPT does a great job on symbolic manipulation. You have to prompt it to show derivations however vs. discussing the topic at a high level.
- trobertson 4y agoThere's no goalpost to move. Psychologists have been saying that IQ tests are of limited scope and utility for decades. Specifically, it is widely agreed that an IQ test is not a valid way "to assess intelligence in a broader sense". https://en.wikipedia.org/wiki/Intelligence_quotient#Validity_as_a_measure_of_intelligence https://en.wikipedia.org/wiki/Intelligence_quotient#Validity...
- pillarofkiller 4y agoyeah there is a model that can do differential equations
- tomp 4y agoIs there a better page to link to? I cannot even see "Kosmos" on this page! Edit: Ah, looks like this is the link to the paper: https://arxiv.org/abs/2302.14045 https://arxiv.org/abs/2302.14045 It was discussed yesterday: https://news.ycombinator.com/item?id=34965326 https://news.ycombinator.com/item?id=34965326
- thenaturalist 4y agoBetter link would have been the tweet, it includes the paper & GH repo: https://twitter.com/alphasignalai/status/1630651280019292161 https://twitter.com/alphasignalai/status/1630651280019292161
- aegistudio 4y agoHmm... LLMs / MLLMs might be truly a unified input / output interface of a would-be AGI, I think.
- Tepix 4y agoYeah, check out the Lex Friedman podcast episode #333 around minute 52 where Andrew Karpathy talks about the OpenAI project "World of Bits" that did this. https://youtu.be/cdiD-9MMpb0?t=3013 https://youtu.be/cdiD-9MMpb0?t=3013
- mkmk3 4y agoAnd the work he's talking about: https://paperswithcode.com/paper/world-of-bits-an-open-domain-platform-for-web https://paperswithcode.com/paper/world-of-bits-an-open-domai...
- josalhor 4y agoThe examples in the paper are pretty impressive. There is an example of a windows 11 dialog image. The computer can figure out which button to press given the desired outcome of the user. If one where to take this model and scale it, I can see an advanced bot in <5 years navigating the web and doing work based on a text input of a human purely by visual means. Interesting times.
- IanCal 4y agoYou might be interested in this: https://www.adept.ai/blog/act-1 https://www.adept.ai/blog/act-1
- freakynit 4y agoHoly tckin' cow!!! Is this real? Captchas be damned now. Beating AI with AI. What a time to be alive.
- moritonal 4y agoWe can't stop it, but giving an AI unbridled access to the Internet is a terrible idea. Whether it's a misphrased question or an clever prompt hack; entire sites will be crushed by the sheer superhuman performance of it. Hackernews will be just robots chatting to each other nudging towards the latest product-hunt.
- flangola7 4y agohttps://www.youtube.com/watch?v=efPrtcLdcdM https://www.youtube.com/watch?v=efPrtcLdcdM The internet will be saturated with fake people soon. Someone already did a PoC of this on 4chan as a joke with just a small GPT2 model. In a few years you won't be able to tell if you're talking to a human unless they're physically in front of you.
- deleted 4y ago[deleted]
- comboy 4y ago
- RcouF1uZ4gsC 4y agoI don’t trust any report of model performance from papers, unless there is a publicly accessible demo. It is way too easy to test things the model has trained on and for the model to then completely fall flat when used by people in the real world.
- nwoli 4y agoThe fb galactica model is a good example of this. Sounded really promising, impressive paper, lots of weights. But when you actually tried it it mostly produced garbage
- nl 4y agoIt's worth noting that this is a comparatively small model (1.6B params from memory). It'll be interesting what capabilities emerge as they grow that model capacity.
- kenjackson 4y agoThat’sa good point. There’s a paper that talks about the non-linear nature of these models. That is at some very large size they seem to show a leap in ability.
- naasking 4y agoAnother one that looks even more compelling: Multimodal Chain-of-Thought Reasoning in Language Models, https://arxiv.org/abs/2302.00923 https://arxiv.org/abs/2302.00923 By building in chain of thought and multimodal learning, this 1B parameter model beats GPT-3.5's 170B parameter model.
- Karellen 4y agoDid anyone else initially read that as `Kosmos~1`, and wonder what the full name of the project was?
- drKarl 4y agoAt Microsoft: Hey why don't we call our new LLM Cosmos? That's taken by the Azure Cosmos DB guys Damn it... how about Kosmos-1 ?
- Bajeezus 4y ago"Fun" fact: There is another common internal service at Microsoft called Cosmos, and it is also a database. So now there is Cosmos, Cosmos DB, and Kosmos.
- outside1234 4y agoIsn't there also a batch processing system named... you guessed it... Cosmos?
- 5- 4y agohttps://en.wikipedia.org/wiki/Kosmos_1 https://en.wikipedia.org/wiki/Kosmos_1
- xfalcox 4y agoAnyone know if this will be an openly available model?
- PaulHoule 4y agoI like this feature they are working on https://arxiv.org/abs/2212.10554 https://arxiv.org/abs/2212.10554 as I'd say the most obvious limitation of today's transformers is the limited attention window. If you want ChatGPT to do a good job of summarizing a topic based on the literature the obvious thing is to feed a bunch of articles into it and ask it to summarize (how can you cite a paper you didn't read?) and that requires looking at maybe 400,000 - 4,000,000 tokens. Similarly there is a place for a word embedding, a sentence embedding, a paragraph embedding, a chapter embedding, a book embedding, etc. but these have to be scalable and obviously the book embedding is bigger but I ought to be able to turn a query into a sentence embedding and somehow match it against larger document embeddings.
- p1esk 4y agofeed a bunch of articles into it and ask it to summarize A better way (that's how humans do it) is to first summarize each article, then feed the summaries to get an overview of the topic. This way there's no need to expand the attention window.
- PaulHoule 4y agoI've thought about that one for a long time. A long time ago I was reading proceedings of TREC trying to understand why Google was so much better than the search engines I knew how to build. TREC is pretty depressing because you find that 95% of the things you might think would improve search rankings do not. Particularly before BM25 was developed people tried indexing sub documents and consolidating them and consistently struck out. Since BERT came out there is a considerable literature of people struggling mightily to combine transformer representations of document parts into a whole that convinces me that one could spend a few lifetimes pushing a bubble around underneath that rug. I think the best argument for your case is that people seem to get along just fine with a limited short term memory. I'd temper that with the observation that a person writing a summary is actually doing a multiple stage process in which their short term memory is attending to part of what they are writing, part of what they are reading, and they are building long term memory structures at the same time. So there is a lot going on. In the sense that abstracts work well for information retrieval and that many of them would fit in the GPT attention window or only be a little bigger you could make the case that a fixed-size structure could be highly useful for IR. On the other hand, many documents, such as scientific papers, are considerably bigger than the current attention window and direct summarization of a single document via transformer will still need a bigger window, more like 40,000 tokens. A lot of things in the literature are complex, muddy, contradictory or all of the above. (Try a question like "What did Freud think about narcissism?" or "What is the clinical relevance of Bleuler's concept of ambivalence?" or "Tell me about cosmic inflation" or "What is the dark matter particle?") Hard cases really do require matching up parts of document A with parts of document B and certainly having them in the same attention window would help an LLM do that in a natural way. It might be completely impractical, not just because of computational scalability but possibly more fundamental scalability limits. (I'm not sure a person with a 10x bigger short term memory would really be able to solve problems better than the average person... There are transformers with a 500,000 token attention window today and they suck.) There could be some procedure where you cut documents up into pieces in various ways, extract critical context from documents A and B and other literature and also put in the parts you want to critique against each other, or even match up different parts of the same document to do the same. Maybe a small attention window could still be used to decompose documents into knowledge graphs but it is by no means trivial to reason over a KG once you have it. What I do know today is that I have documents >4096 tokens that I want to retrieve, cluster and classify right now and transformers were not up to the task in Feb 2023, and I am hoping for some progress soon that will help.