17 ms·

MiniGPT-4

949 points by GaggiX 4y ago

mdrzn 4y ago
Interesting, will try it out. The queue is about 100 at the moment, with 700s of waiting.
- mritchie712 4y ago
  1200s now...
  - punkspider 4y ago
    Also some links don't seem to open in a new tab and if you click around while you wait you might lose your progress. I clicked on the Video button while waiting, assuming that it would open in a new tab, and lost my place in queue.
    - zapdrive 4y ago
      Also once you get your turn and ask your question, it puts you back in the queue.
golergka 4y ago
I don't know what unholy abomination they did to the scrolling, but after I increased the page font size, it just sticks to the bottom and you can't scroll up.
- arjonagelhout 4y ago
  I’m experiencing the same thing on iOS Safari, it rapidly jumps back and forth between two positions.
- deleted 4y ago
  [deleted]
- detrites 4y ago
  Right off the bat I lost all scrolling inertia here, it's like being in the dark ages.
tysam_and 4y ago
I was not expecting this to be good due to the hype wave, but at least looking at the examples, they seem very impressive and worthy of the (potential) hype! I can't wait to see what new waves of these models keep coming out. I for one will probably still stay near the fundamental side of things as all of this thrash is simply far too much for me, but it is quite nice and beautiful for me to get to see all of the variety and beauty of what is being built here. <3 <3 <3 <3 Now, to figure out a good way to integrate these seamlessly into my development workflow....I go about as painfully low-tech as one could. I might as well be using an electronic typewriter with only Ctrl+F search, lolzers....
wsgeorge 4y ago
Giving this a quick spin and I like what I'm seeing. I gave it a picture of Paolo Veronese's Lament over the Dead Christ [0] and asked what was going on in the background. > The painting depicts the scene of the crucifixion of Jesus Christ. In the foreground, there is a group of people standing around the cross, including Mary, the mother of Jesus, and the two thieves who were crucified with him. In the background, there is a hill with a group of people standing on it, watching the crucifixion. The sky is dark and stormy, with lightning illuminating the scene. The overall mood of the painting is one of sadness and tragedy. Then I asked for a sonnet on it: "In this painting, we see the scene of the cross Where Jesus is crucified, with Mary at his side And the thieves who were crucified with him In the background, a hill with people standing Watching the tragic event unfold The sky is dark and stormy, with lightning flashing Illuminating the scene of the crucifixion The mood is one of sadness and tragedy As we contemplate the sacrifice of Jesus But even in death, he shows us the way To love and forgiveness, to live another day And so we remember the crucifixion And the love that Jesus showed us that day." So...not quite what I was expecting, but it's progress! [0] https://en.wikipedia.org/wiki/Lament_over_the_Dead_Christ_(Veronese) https://en.wikipedia.org/wiki/Lament_over_the_Dead_Christ_(V...
- omeid2 4y ago
  What did you expect?
  - causi 4y ago
    I never know what to expect anymore. We live in a world where computers can describe paintings and write sonnets about them but a half-trillion dollar car company can't figure out how to parallel park with eight cameras.
    - niemandhier 4y ago
      That's because the decided they do not need lidar.
      - 4y ago
152334H 4y ago
On a technical level, they're doing something really simple -- take BLIP2's ViT-L+Q-former, connect it to Vicuna-13B with a linear layer, and train just the tiny layer on some datasets of image-text pairs. But the results are pretty amazing. It completely knocks Openflamingo && even the original blip2 models out of the park. And best of all, it arrived before OpenAI's GPT-4 Image Modality did. Real win for Open Source AI. The repo's default inference code is kind of bad -- vicuna is loaded in fp16 so it can't fit on any consumer hardware. I created a PR on the repo to load it with int8, so hopefully by tomorrow it'll be runnable by 3090/4090 users. I also developed a toy discord bot (https://github.com/152334H/MiniGPT-4-discord-bot https://github.com/152334H/MiniGPT-4-discord-bot) to show the model to some people, but inference is very slow so I doubt I'll be hosting it publicly.
- sho 4y ago
  > they're doing something really simple -- take BLIP2's ViT-L+Q-former, connect it to Vicuna-13B with a linear layer, and train just the tiny layer on some datasets of image-text pairs Oh yes. Simple! Jesus, this ML stuff makes a humble web dev like myself feel like a dog trying to read Tolstoy.
  - mjburgess 4y ago
    In practice, it's a lot more like web dev than you might imagine. The above means that the approach is web-dev like gluing, almost literally just, from existingliba import someop from existinglibb import anotherop from someaifw import glue a = someop(X) b = glue(a) Y = anotherop(b)
    - marban 4y ago
      And repeat that ~4 times to make it look like LangChain
    - guax 4y ago
      And just like webdev, each of those were done in a different platform and require arcane incantations and 5h of doc perusing to make it work on your system.
Tepix 4y ago
I think it's poor form that they are taking the GPT-4 name for an unrelated project. After all, the underlying Vicuna is merely a fine-tuned LLaMA. Plus they use the smaller 13B version. The results look interesting, however. Here's hoping that they'll add GTPQ 4bit quantizing so the 65B version of the model can be run on 2x 3090.
- collaborative 4y ago
  Adding GPT4 to anything now increases marketing x4. So many AI news coming out lately that not adding it risks drowning in a sea of info.. even in the case of a good project
  - Uehreka 4y ago
    The word for this is “trademark infringement”. You are specifically not allowed to capitalize on the marketing of another entity’s product to bolster yours by implying through your name that you are somehow related. This is why “DALL-E Mini” had to change their name to craiyon.
    - HarHarVeryFunny 4y ago
      It's also just (deliberately) misleading. It's based on the 13B Vicuna/Llama model, not 175B GPT-3 or 1T GPT-4. There is zero justification for calling it MiniGPT-4. A more honest name would be Visual-Vicuna or Son-of-BLIP.
      - sebzim4500 4y ago
        I don't see how it's misleading. MiniGPT-4 makes it sound like a smaller alternative to GPT-4, if it was based on GPT-4 there would be nothing 'mini' about it.
        HarHarVeryFunny 4y ago
        It has more in common with GPT-3 than GPT-4 in terms of size, but in reality it's based on Vicuna/Llama which is 10x smaller than either, so as far as the LLM part of it goes its not mini-anything - it's just straight-up Vicuna 13B. The model as a whole is just BLIP-2 with a larger linear layer, and using Vicuna as the LLM. If you look at their code it's literally using the entire BLIP-2 encoder (Salesforce code). https://arxiv.org/pdf/2301.12597.pdf https://arxiv.org/pdf/2301.12597.pdf
tarruda 4y ago
Looking forward to the next generation of cheap GPUs with enough VRAM to run models like Vicuna-13 locally.
- valine 4y ago
  Cheap is relative I suppose. I’m running Vicuna 13b 16f locally and it needs 26GB of VRAM, which won’t even fit on a single RTX 4090. The next gen RTX Titan might have enough vram but that won’t come cheap. I’m expecting a price point above $2500.
  - 22c 4y ago
    I'm not sure if it's the point GP is trying to make, but I would like to see GPUs with extra VRAM that don't have the extra compute. eg. similar performance of a 4070Ti but with 24GB or 32GB of VRAM. I don't see a really good reason why OEMs couldn't do that now, in the past there have been OEM cards that have more VRAM than the reference design. I'm sure there's an appetite for cards like that for people who don't want to refinance their home loan to get 2 x RTX 4090 cards.
    - Tepix 4y ago
      I'm fairly sure that NVIDIA is making sure that consumer cards are no good alternative to their $10000 80GB VRAM A100 cards.
  - MacsHeadroom 4y ago
    13B in GPTQ 4bit has practically no quality loss and runs in 8GB. I get 8 tokens/second on my laptop CPU and 4 tokens/second on my phone CPU. Even 33B only needs 20GB of VRAM in GPTQ 4bit. 8bit has zero perplexity loss, so there's really no reason to run in 16bit. Even a $200 P40 24GB is enough to run 33B at extremely high speeds in GPTQ 4bit.
- adam_arthur 4y ago
  A lot of people are running fairly powerful models directly on the CPU these days... seems like inference will not be a GPU exclusive activity going forward. Given that RAM is the main bottleneck at this point, running on CPU seems more practical for most end users See: https://news.ycombinator.com/item?id=35602234 https://news.ycombinator.com/item?id=35602234
m00dy 4y ago
Hi CHEN, Invoker here, I would like to have a chat or send me an email @ community@invoker.network
- endorphine 4y ago
  DotA?
  - m00dy 4y ago
    I'm way better than him ;)
xnzakg 4y ago
The ramen example is kind of hilarious. Wonder if it would make more sense with a bigger model.
- m348e912 4y ago
  Ok I'll bite. What makes the Raman example so funny?
  - Thomashuet 4y ago
    The recipe it suggests is completely off. Just search for "ramen recipe" and compare. For instance it tells you to add soy sauce, sesame oil and green onions to the broth (so far so good) but then tells you to throw all of it away ("once the noodles are cooked, drain them"). The problem here is that you should cook the noodles separately in plain water, not a broth with soy sauce.
  - burkaman 4y ago
    The recipe says to make a broth, cook the noodles in the broth, then throw the broth away and add the noodles to a pan with meat. You'll end up with burned crispy noodles and meat in a dry bowl.
atgctg 4y ago
It's hard to distinguish non-Google projects with Google Sans in their templates from actual Google Research papers, as the font is meant to be exclusively used by Google[1]. [1] https://developers.google.com/fonts/faq#how_can_i_get_a_license_for_google_sans https://developers.google.com/fonts/faq#how_can_i_get_a_lice...
- sebzim4500 4y ago
  Surely most people would read the authors list to determine provenance rather than the font?
  - pxeger1 4y ago
    I didn't think about it consciously but I think I did implicitly assume it was a Google project because of the font
dom96 4y ago
Can this decode CAPTCHAs? Are we finally in a place where CAPTCHAs are useless?
- alex_duf 4y ago
  would it be a good things for captchas to be useless? Because in my career I've been a few times on the receiving end of spam attacks and various nastiness and it's nice to be able to integrate such a thing - as unpleasant as it is for the user - and solve the problem at hand. if this disappears then spammers and the various botnets will have the upper hand again.
  - dom96 4y ago
    It wouldn't be good, did I imply it would? I'm interested to know if we are now in a place where they do become useless
- dalys 4y ago
  I was also thinking about this the other day. Idea one: Captchas are to become pretty useless as a "is this a human" tactic soon. Maybe it already is, I don't know. What other things could we think off to prove someone is human? I was watching Lex Fridman and Max Tegmark and they were remarking on how Twitter using payment as a differentiator between human and bot is actually really good. And maybe the only way we can reliably determine if someone is a human or not right now. Just by the virtue that having thousands of bots doing something, that suddenly costs $5 per event will deter most attacks. Integrating online identification systems from various countries could be one tactic (such as https://en.wikipedia.org/wiki/BankID https://en.wikipedia.org/wiki/BankID that we use in Sweden to log in to basically any online service). New startup: Un-botable authentication as a service. Idea two: Since captchas are useless, we'll be able to do bots that can do almost everything on the web. No need for writing automation scripts, headless browsers, regexp etc. Just feed real visual data from browser to GPT-4 (or MiniGPT-4 or similar). Give instructions like "You need to accomplish this task: Go to facebook.com and create a user account and be friends with 100 people and act like a human. Follow the instructions on the website.". Then let the bot figure out where to move the mouse and send click events, keyboard events etc. Obviously much slower than a scripted bot, but would be very hard to detect as a bot. Good for E2E testing also? New startup: WebGPT: Bots that can do any task on any website. TestGPT: E2E testing service. Just write test instructions like "Make sure the login works on /login" and you're done! And you thought flaky tests were a thing of the past... Would be kind of cool for UX testing also. "You are a 43 year old truck driver in Spain, please go to news.ycombinator.com and see how easy it is for you to register an account, log in, go to any news story and make a new comment. Report anything you think is confusing."
devinprater 4y ago
Holy crap! If this has image to text, and can run on a CPU, even with 16 GB RAM, one could hook up an NVDA addon, straight Python so it shouldn't be too hard, and have AI read whatever the user wants. Pictures, video frames, UI's, and if it can break the UI down into something the screen reader can understand, boom!
felipesabino 4y ago
the lost me when the demo bot chose php for the website builder demo /s
thatsadude 4y ago
How is this a paper, they simply fine-tune the fully-connected layer of BLIP-2 https://arxiv.org/pdf/2301.12597.pdf https://arxiv.org/pdf/2301.12597.pdf
macrolime 4y ago
A modified BLIP2 that works with diffusion models rather than vision transformers would be pretty cool. Using Vicuna-13B or another large language model as the language model of Stable Diffusion, instead of the text encoder of CLIP, would be a game changer, it would completely change how you prompt Stable Diffusion from something kinda like a tag list to it being able to follow instructions in English instead.
simonw 4y ago
Hate to be the person complaining about the name, but we already saw how this plays out with DALL-E mini: if you name your project directly after something else like this, no matter how much extra explanatory text you attach to it a large number of people will assume it's an "official" variant of the thing it was named after. Eventually you'll have to rename it, either to resolve the confusion or because OpenAI pressure you to do so, or both. So better to pick a less confusing name from the start. (This one is even more confusing because it's about image inputs, but GPT4 with image inputs had not actually been released to anyone yet - similar in fact to how DALL-E mini got massive attention because DALL-E itself was still in closed preview)
- xiphias2 4y ago
  ,,So better to pick a less confusing name from the start.'' I'm not sure if that's better from a marketing standpoint though....it works, you still remember DALL-E mini
vlovic 4y ago
Do I understand this correctly: they just took Blip2 and replaced the LLM with Vicuna, and to do that they just added a single linear layer to translate between frozen vision encoder and (frozen) Vicuna? Additionally, and importantly, they manually create a high quality dataset for finetuning their model. If that is the case, then this is really a very, very simple paper. But I guess simple things can lead to great improvements, and indeed their results seem very impressive. Goes to show how much low hanging fruit there must be in deep learning these days by leveraging the amazing, and amazingly general, capabilities of LLMs.
- visarga 4y ago
  Yes, model composability magic.
aabajian 4y ago
From a radiology world this is fascinating. I'm not worried about job security as I'm an interventionalist. What I'm wondering is about go-to-market strategies for diagnostics. I do some diagnostic reads and I would love to have something like this pre-draft reports (especially for X-Rays). There are tons of "AI in rads" companies right now, none of which have models that come anywhere close to GPT-4 or even this. Personally, I would not start such a company because the barrier to entry is now so low. I think there needs to be an open-source project that just subtrains these models on rads images and is available to all radiologists.
- euroderf 4y ago
  Your profession and... a few hundred others ?
jiwidi 4y ago
a bit nasty to call something minigpt-4 when it comes from llama
lxe 4y ago
Why Vicuna over something like oasst-pythia? Would love to see a table comparing all the new models side by side.
- GaggiX 4y ago
  All Pythia models were trained on 300B tokens, LLaMa models were trained on 1/1.4T tokens.
- lhl 4y ago
  Fabrice Bellard has run a standard set of benchmarks w/ lm-eval on a big chunk of open models here: https://bellard.org/ts_server/ https://bellard.org/ts_server/ - Flan T5 XXL and GPT-NeoX 20B both outperform Pythia 12B on average (LLaMA 13B+ tops the charts).
  - deleted 4y ago
    [deleted]
m3kw9 4y ago
This is basically Image to Text, then from the text, start a GPT type chat. No need to overthink anything
jokoon 4y ago
Are there more university/research centric resource of communities regarding AI, I mean things on a more epistemic and higher level, that disregards current ML techniques and implementations? I have no problem when there is a lot of money spent to implement things that can be used and tested, but I wish I could read about the next step of AI, something that involves science and not software engineering. I've yet to read anything related to the analysis of trained data or trained neural networks, the structure of the brains of small insects, psychology on general intelligence, AI and emotions, the real ability of an AI to think and write meaningful code or understand what is a problem/solution etc, I mean anything that is related to AI that can be treated by computer science. I have no problems with algorithms and implementations, but don't university/professors also try to go beyond, with epistemology, to connect the dots with biology and neurology? Unless I see science cover several fields of research, it's not real, worthy research in my view, which is why the reason why I'm so stubbornly skeptical of chatGPT.
- thewarrior 4y ago
  We’ve tried but we understand even less about how “real” intelligence works. Atleast in a way that we can replicate it. The people doing so called alchemy are making far better systems that anyone who’s trying to actually understand things. And we’ve been trying for decades now.
- visarga 4y ago
  If the engineering part is of secondary importance for you, then at least remember about the dataset. It is in the composition and quality of the training data that all the skills find their origin. Most of our discussions are about model size, but few about dataset. Yet all the scaling laws hint at the great usefulness of more data. Sometimes even little data can have a great impact in the fine-tuning phase. In the end it is the training data that transforms a random init into the model.
ggregoire 4y ago
Could you use something like this to summarize the content of a video? Or would you first need to extract the images from the video (like 1 fps, with ffmpeg) and tell the AI to analyze all the images and summarize them like if it was a video?
- all2 4y ago
  I'd do this using both streams of information: audio and video. I'd segment the audio semantically based on the topic of discussion, and I'd segment the video based on editing, subjects in scene, etc. We could start simply and just have a "timestamp": [ subjects, in, frame] key-value. It'd take some fiddling to sort how to mesh these two streams of data back together. The first thing I'd try is segment by time chunks (the resolution of which would depend on min/max segment lengths in video and audio streams) and then clump the time chunks together based on audio+video content.
nobrains 4y ago
I think the real news is where this research is coming from. It is from King Abdullah University of Science and Technology in Saudi. The arab world has recently been quite behind in tech research, but this is good news overall for the arab world, which will inspire other universities and researchers in the region to one-up their colleagues, which is overall good for the entire world.
- sterlind 4y ago
  I would be more thrilled if it were literally any other country in the Arab world, but maybe that will come.
23459872345 4y ago
Has anyone gotten this running in Colab?
Imnimo 4y ago
I'm pretty surprised that the frozen encoder has enough information in its representations to do the "design the website from a hand-written whiteboard mock-up" task, and that that information can be effectively accessed by the language model with just a small projection layer. I really would have guessed that you needed to be training the image encoder simultaneously.
deeplstm 4y ago
Here's the video summary https://youtu.be/uPV9Gk3IC-g https://youtu.be/uPV9Gk3IC-g
djangovm 4y ago
If I have to learn how to be able to read this code and understand what it and its dependencies are doing, where do I start? Is reading their paper an effective strategy? https://github.com/Vision-CAIR/MiniGPT-4/blob/main/MiniGPT_4.pdf https://github.com/Vision-CAIR/MiniGPT-4/blob/main/MiniGPT_4...
HansHamster 4y ago
Just tried it with a random schematic: > The image shows a block diagram of a power amplifier circuit with a power supply and a load resistor. The power supply is connected to the input of the amplifier, which is then connected to the output. The load resistor is connected to the output of the amplifier and the power supply. The amplifier is designed to amplify the input signal to a higher level, which is then sent to the load resistor to power the load. The power supply provides the necessary voltage to operate the amplifier. On the one hand I'm impressed that it recognises an amplifier, but just about everything in the description is (slightly) wrong. It is not a power amplifier, the power supply is not connected to the input (why should it be?), there is no load resistor...
- mftb 4y ago
  I also got middling results. I gave it a novel NASA picture of the moon passing in front of the earth, taken by the Deep Space Climate Observatory Satellite, so an unusual vantage point. First it was just wrong, and said that it was a picture of a crescent moon taken from a mountain top, or something like that. Not terrible at least it recognized the moon, but wrong. I tried to give it a chance and corrected it with the additional info that the picture was taken from space. It then did better, but on the last line it threw in that it was not a photograph, but a rendering of some sort, which again, is just wrong. More and more I get the impression that people are so taken with the chat interface that they are grading these models more favorably than is warranted.
  - HansHamster 4y ago
    And after pointing out all the mistakes in the description, it took a very ... interesting approach to address my remarks: > I apologize for the confusion. The image shows a block diagram of a basic amplifier circuit with a power supply, but it does not show any input stages, output stages, or load resistors. [...] The input stage is made up of an operational amplifier, which is used to amplify the input signal. The output of the input stage is then connected to the power supply, which provides the necessary voltage to operate the amplifier. The amplifier is designed to amplify the input signal to a higher level, which is then sent to the load resistor to power the load. So it acknowledges the remarks, describes everything that is _not_ in the image and then puts the stuff back into the description anyway :)
morelight18 4y ago
[dead]
cloudking 4y ago
It doesn't seem to recognize text from a screenshot very well, I gave detailed error messages from a Windows screenshot: https://filestore.community.support.microsoft.com/api/images/485f9156-e955-4acc-a399-60d16cc64521 https://filestore.community.support.microsoft.com/api/images... and prompted "Describe any issues found in this screenshot and steps to resolve them" while it correctly identified it as a screenshot from a computer, it gave a very generic response and didn't identify the error messages correctly: "The screenshot shows a computer screen with a warning message that reads, “The system is currently unavailable. Please try again later.” This indicates that there is an issue with the computer system that needs to be resolved. To resolve this issue, you can try the following steps: 1. Check the system status: Check the system status to see if there are any errors or issues that need to be resolved. 2. Restart the computer: Try restarting the computer to see if that resolves the issue. 3. Check for updates: Check for any updates that may have been released for the computer system and apply them if necessary. 4. Check for viruses: Run a virus scan on the computer to check for any viruses that may be causing the issue. 5. Check for hardware issues: Check for any hardware issues that may be causing the issue. This could include issues with the power supply, hard drive, or other components. 6. Contact technical support: If none of the above steps resolve the issue, contact technical support for further assistance. They may be able to provide additional troubleshooting steps or help you resolve the issue."
- MacsHeadroom 4y ago
  I can barely read the text in that screenshot, and I'm a human (I think). It is both low resolution and filled with jpeg artifacts.
raylad 4y ago
I gave it a picture of a bottle, which it did a fairly good job of identifying (although it misread some of the text, mistaking a 1 for a 4). Then I asked it what are the likely ingredients of the product. It still hadn't replied after 2274s so I gave up on it.
nyanpasu64 4y ago
I hope that someday language models will be trained to accurately reflect their training data and real-world text, and not be hard-coded to spit out disclaimers like "As an artificial intelligence, I do not have personal feelings or emotions" in response to any corporate-unfriendly inputs.
no_time 4y ago
https://imgur.com/a/jEQzRzT https://imgur.com/a/jEQzRzT Great job at OCR-ing the text and recognizing the context, but it still can't do doom and gloom like us eastern Europeans.
smashfizzle7 4y ago
[dead]
monsoon11 4y ago
[dead]
blooms112 4y ago
[dead]
floufrouu11 4y ago
[dead]
morelight19 4y ago
[dead]
sharinggenes1 4y ago
[dead]
indigosparkle1 4y ago
[dead]
selfcare111 4y ago
[dead]
yourblanks17 4y ago
[dead]
butterflyeffec 4y ago
[dead]
lifebutterfly 4y ago
[dead]
gobblers1 4y ago
[dead]
livincool87 4y ago
[dead]
selfcare11 4y ago
[dead]
reading87 4y ago
[dead]
flightsnot81 4y ago
[dead]
doodles675 4y ago
[dead]
indigosparkle7 4y ago
[dead]
morelight876 4y ago
[dead]
monsoon654 4y ago
[dead]
indigosparkle6 4y ago
[dead]
fabulous13 4y ago
[dead]
champagne7 4y ago
[dead]
flightsnot45 4y ago
[dead]
styleslover123 4y ago
[dead]
isonfire11 4y ago
[dead]
shop13213 4y ago
[dead]
aholicdiva 4y ago
[dead]
workofgod132 4y ago
[dead]
pink1321 4y ago
[dead]
lusttforlife567 4y ago
[dead]
thatsassy127 4y ago
[dead]
lwithnojob65 4y ago
[dead]
inkandpaper1 4y ago
[dead]
gaming56 4y ago
[dead]
withnojob0 4y ago
[dead]
kong45 4y ago
[dead]
goddess09 4y ago
[dead]
gypsy645 4y ago
[dead]
wanderlust56 4y ago
[dead]
chicnext673 4y ago
[dead]
door7544 4y ago
[dead]
foodie564 4y ago
[dead]
lyjules464 4y ago
[dead]
nitch65854 4y ago
[dead]
avocados564 4y ago
[dead]
diet7865 4y ago
[dead]

MiniGPT-4

949 points by GaggiX 4y ago

mdrzn 4y ago

Interesting, will try it out. The queue is about 100 at the moment, with 700s of waiting.

mritchie712 4y ago
1200s now...
- punkspider 4y ago
  Also some links don't seem to open in a new tab and if you click around while you wait you might lose your progress. I clicked on the Video button while waiting, assuming that it would open in a new tab, and lost my place in queue.
  - zapdrive 4y ago
    Also once you get your turn and ask your question, it puts you back in the queue.

golergka 4y ago

I don't know what unholy abomination they did to the scrolling, but after I increased the page font size, it just sticks to the bottom and you can't scroll up.

arjonagelhout 4y ago
I’m experiencing the same thing on iOS Safari, it rapidly jumps back and forth between two positions.
deleted 4y ago
[deleted]
detrites 4y ago
Right off the bat I lost all scrolling inertia here, it's like being in the dark ages.

tysam_and 4y ago

I was not expecting this to be good due to the hype wave, but at least looking at the examples, they seem very impressive and worthy of the (potential) hype! I can't wait to see what new waves of these models keep coming out. I for one will probably still stay near the fundamental side of things as all of this thrash is simply far too much for me, but it is quite nice and beautiful for me to get to see all of the variety and beauty of what is being built here. <3 <3 <3 <3 Now, to figure out a good way to integrate these seamlessly into my development workflow....I go about as painfully low-tech as one could. I might as well be using an electronic typewriter with only Ctrl+F search, lolzers....

wsgeorge 4y ago

Giving this a quick spin and I like what I'm seeing. I gave it a picture of Paolo Veronese's Lament over the Dead Christ [0] and asked what was going on in the background. > The painting depicts the scene of the crucifixion of Jesus Christ. In the foreground, there is a group of people standing around the cross, including Mary, the mother of Jesus, and the two thieves who were crucified with him. In the background, there is a hill with a group of people standing on it, watching the crucifixion. The sky is dark and stormy, with lightning illuminating the scene. The overall mood of the painting is one of sadness and tragedy. Then I asked for a sonnet on it: "In this painting, we see the scene of the cross Where Jesus is crucified, with Mary at his side And the thieves who were crucified with him In the background, a hill with people standing Watching the tragic event unfold The sky is dark and stormy, with lightning flashing Illuminating the scene of the crucifixion The mood is one of sadness and tragedy As we contemplate the sacrifice of Jesus But even in death, he shows us the way To love and forgiveness, to live another day And so we remember the crucifixion And the love that Jesus showed us that day." So...not quite what I was expecting, but it's progress! [0] https://en.wikipedia.org/wiki/Lament_over_the_Dead_Christ_(Veronese) https://en.wikipedia.org/wiki/Lament_over_the_Dead_Christ_(V...

omeid2 4y ago
What did you expect?
- causi 4y ago
  I never know what to expect anymore. We live in a world where computers can describe paintings and write sonnets about them but a half-trillion dollar car company can't figure out how to parallel park with eight cameras.
  - niemandhier 4y ago
    That's because the decided they do not need lidar.
    - 4y ago

152334H 4y ago

On a technical level, they're doing something really simple -- take BLIP2's ViT-L+Q-former, connect it to Vicuna-13B with a linear layer, and train just the tiny layer on some datasets of image-text pairs. But the results are pretty amazing. It completely knocks Openflamingo && even the original blip2 models out of the park. And best of all, it arrived before OpenAI's GPT-4 Image Modality did. Real win for Open Source AI. The repo's default inference code is kind of bad -- vicuna is loaded in fp16 so it can't fit on any consumer hardware. I created a PR on the repo to load it with int8, so hopefully by tomorrow it'll be runnable by 3090/4090 users. I also developed a toy discord bot (https://github.com/152334H/MiniGPT-4-discord-bot https://github.com/152334H/MiniGPT-4-discord-bot) to show the model to some people, but inference is very slow so I doubt I'll be hosting it publicly.

sho 4y ago
> they're doing something really simple -- take BLIP2's ViT-L+Q-former, connect it to Vicuna-13B with a linear layer, and train just the tiny layer on some datasets of image-text pairs Oh yes. Simple! Jesus, this ML stuff makes a humble web dev like myself feel like a dog trying to read Tolstoy.
- mjburgess 4y ago
  In practice, it's a lot more like web dev than you might imagine. The above means that the approach is web-dev like gluing, almost literally just, from existingliba import someop from existinglibb import anotherop from someaifw import glue a = someop(X) b = glue(a) Y = anotherop(b)
  - marban 4y ago
    And repeat that ~4 times to make it look like LangChain
  - guax 4y ago
    And just like webdev, each of those were done in a different platform and require arcane incantations and 5h of doc perusing to make it work on your system.

Tepix 4y ago

I think it's poor form that they are taking the GPT-4 name for an unrelated project. After all, the underlying Vicuna is merely a fine-tuned LLaMA. Plus they use the smaller 13B version. The results look interesting, however. Here's hoping that they'll add GTPQ 4bit quantizing so the 65B version of the model can be run on 2x 3090.

collaborative 4y ago
Adding GPT4 to anything now increases marketing x4. So many AI news coming out lately that not adding it risks drowning in a sea of info.. even in the case of a good project
- Uehreka 4y ago
  The word for this is “trademark infringement”. You are specifically not allowed to capitalize on the marketing of another entity’s product to bolster yours by implying through your name that you are somehow related. This is why “DALL-E Mini” had to change their name to craiyon.
  - HarHarVeryFunny 4y ago
    It's also just (deliberately) misleading. It's based on the 13B Vicuna/Llama model, not 175B GPT-3 or 1T GPT-4. There is zero justification for calling it MiniGPT-4. A more honest name would be Visual-Vicuna or Son-of-BLIP.
    - sebzim4500 4y ago
      I don't see how it's misleading. MiniGPT-4 makes it sound like a smaller alternative to GPT-4, if it was based on GPT-4 there would be nothing 'mini' about it.
      - HarHarVeryFunny 4y ago
        It has more in common with GPT-3 than GPT-4 in terms of size, but in reality it's based on Vicuna/Llama which is 10x smaller than either, so as far as the LLM part of it goes its not mini-anything - it's just straight-up Vicuna 13B. The model as a whole is just BLIP-2 with a larger linear layer, and using Vicuna as the LLM. If you look at their code it's literally using the entire BLIP-2 encoder (Salesforce code). https://arxiv.org/pdf/2301.12597.pdf https://arxiv.org/pdf/2301.12597.pdf

tarruda 4y ago

Looking forward to the next generation of cheap GPUs with enough VRAM to run models like Vicuna-13 locally.

valine 4y ago
Cheap is relative I suppose. I’m running Vicuna 13b 16f locally and it needs 26GB of VRAM, which won’t even fit on a single RTX 4090. The next gen RTX Titan might have enough vram but that won’t come cheap. I’m expecting a price point above $2500.
- 22c 4y ago
  I'm not sure if it's the point GP is trying to make, but I would like to see GPUs with extra VRAM that don't have the extra compute. eg. similar performance of a 4070Ti but with 24GB or 32GB of VRAM. I don't see a really good reason why OEMs couldn't do that now, in the past there have been OEM cards that have more VRAM than the reference design. I'm sure there's an appetite for cards like that for people who don't want to refinance their home loan to get 2 x RTX 4090 cards.
  - Tepix 4y ago
    I'm fairly sure that NVIDIA is making sure that consumer cards are no good alternative to their $10000 80GB VRAM A100 cards.
- MacsHeadroom 4y ago
  13B in GPTQ 4bit has practically no quality loss and runs in 8GB. I get 8 tokens/second on my laptop CPU and 4 tokens/second on my phone CPU. Even 33B only needs 20GB of VRAM in GPTQ 4bit. 8bit has zero perplexity loss, so there's really no reason to run in 16bit. Even a $200 P40 24GB is enough to run 33B at extremely high speeds in GPTQ 4bit.
adam_arthur 4y ago
A lot of people are running fairly powerful models directly on the CPU these days... seems like inference will not be a GPU exclusive activity going forward. Given that RAM is the main bottleneck at this point, running on CPU seems more practical for most end users See: https://news.ycombinator.com/item?id=35602234 https://news.ycombinator.com/item?id=35602234

m00dy 4y ago

Hi CHEN, Invoker here, I would like to have a chat or send me an email @ community@invoker.network

endorphine 4y ago
DotA?
- m00dy 4y ago
  I'm way better than him ;)

xnzakg 4y ago

The ramen example is kind of hilarious. Wonder if it would make more sense with a bigger model.

m348e912 4y ago
Ok I'll bite. What makes the Raman example so funny?
- Thomashuet 4y ago
  The recipe it suggests is completely off. Just search for "ramen recipe" and compare. For instance it tells you to add soy sauce, sesame oil and green onions to the broth (so far so good) but then tells you to throw all of it away ("once the noodles are cooked, drain them"). The problem here is that you should cook the noodles separately in plain water, not a broth with soy sauce.
- burkaman 4y ago
  The recipe says to make a broth, cook the noodles in the broth, then throw the broth away and add the noodles to a pan with meat. You'll end up with burned crispy noodles and meat in a dry bowl.

atgctg 4y ago

It's hard to distinguish non-Google projects with Google Sans in their templates from actual Google Research papers, as the font is meant to be exclusively used by Google[1]. [1] https://developers.google.com/fonts/faq#how_can_i_get_a_license_for_google_sans https://developers.google.com/fonts/faq#how_can_i_get_a_lice...

sebzim4500 4y ago
Surely most people would read the authors list to determine provenance rather than the font?
- pxeger1 4y ago
  I didn't think about it consciously but I think I did implicitly assume it was a Google project because of the font

dom96 4y ago

Can this decode CAPTCHAs? Are we finally in a place where CAPTCHAs are useless?

alex_duf 4y ago
would it be a good things for captchas to be useless? Because in my career I've been a few times on the receiving end of spam attacks and various nastiness and it's nice to be able to integrate such a thing - as unpleasant as it is for the user - and solve the problem at hand. if this disappears then spammers and the various botnets will have the upper hand again.
- dom96 4y ago
  It wouldn't be good, did I imply it would? I'm interested to know if we are now in a place where they do become useless
dalys 4y ago
I was also thinking about this the other day. Idea one: Captchas are to become pretty useless as a "is this a human" tactic soon. Maybe it already is, I don't know. What other things could we think off to prove someone is human? I was watching Lex Fridman and Max Tegmark and they were remarking on how Twitter using payment as a differentiator between human and bot is actually really good. And maybe the only way we can reliably determine if someone is a human or not right now. Just by the virtue that having thousands of bots doing something, that suddenly costs $5 per event will deter most attacks. Integrating online identification systems from various countries could be one tactic (such as https://en.wikipedia.org/wiki/BankID https://en.wikipedia.org/wiki/BankID that we use in Sweden to log in to basically any online service). New startup: Un-botable authentication as a service. Idea two: Since captchas are useless, we'll be able to do bots that can do almost everything on the web. No need for writing automation scripts, headless browsers, regexp etc. Just feed real visual data from browser to GPT-4 (or MiniGPT-4 or similar). Give instructions like "You need to accomplish this task: Go to facebook.com and create a user account and be friends with 100 people and act like a human. Follow the instructions on the website.". Then let the bot figure out where to move the mouse and send click events, keyboard events etc. Obviously much slower than a scripted bot, but would be very hard to detect as a bot. Good for E2E testing also? New startup: WebGPT: Bots that can do any task on any website. TestGPT: E2E testing service. Just write test instructions like "Make sure the login works on /login" and you're done! And you thought flaky tests were a thing of the past... Would be kind of cool for UX testing also. "You are a 43 year old truck driver in Spain, please go to news.ycombinator.com and see how easy it is for you to register an account, log in, go to any news story and make a new comment. Report anything you think is confusing."

devinprater 4y ago

Holy crap! If this has image to text, and can run on a CPU, even with 16 GB RAM, one could hook up an NVDA addon, straight Python so it shouldn't be too hard, and have AI read whatever the user wants. Pictures, video frames, UI's, and if it can break the UI down into something the screen reader can understand, boom!

felipesabino 4y ago

the lost me when the demo bot chose php for the website builder demo /s

thatsadude 4y ago

How is this a paper, they simply fine-tune the fully-connected layer of BLIP-2 https://arxiv.org/pdf/2301.12597.pdf https://arxiv.org/pdf/2301.12597.pdf

macrolime 4y ago

A modified BLIP2 that works with diffusion models rather than vision transformers would be pretty cool. Using Vicuna-13B or another large language model as the language model of Stable Diffusion, instead of the text encoder of CLIP, would be a game changer, it would completely change how you prompt Stable Diffusion from something kinda like a tag list to it being able to follow instructions in English instead.

simonw 4y ago

Hate to be the person complaining about the name, but we already saw how this plays out with DALL-E mini: if you name your project directly after something else like this, no matter how much extra explanatory text you attach to it a large number of people will assume it's an "official" variant of the thing it was named after. Eventually you'll have to rename it, either to resolve the confusion or because OpenAI pressure you to do so, or both. So better to pick a less confusing name from the start. (This one is even more confusing because it's about image inputs, but GPT4 with image inputs had not actually been released to anyone yet - similar in fact to how DALL-E mini got massive attention because DALL-E itself was still in closed preview)

xiphias2 4y ago
,,So better to pick a less confusing name from the start.'' I'm not sure if that's better from a marketing standpoint though....it works, you still remember DALL-E mini

vlovic 4y ago

Do I understand this correctly: they just took Blip2 and replaced the LLM with Vicuna, and to do that they just added a single linear layer to translate between frozen vision encoder and (frozen) Vicuna? Additionally, and importantly, they manually create a high quality dataset for finetuning their model. If that is the case, then this is really a very, very simple paper. But I guess simple things can lead to great improvements, and indeed their results seem very impressive. Goes to show how much low hanging fruit there must be in deep learning these days by leveraging the amazing, and amazingly general, capabilities of LLMs.

visarga 4y ago
Yes, model composability magic.

aabajian 4y ago

From a radiology world this is fascinating. I'm not worried about job security as I'm an interventionalist. What I'm wondering is about go-to-market strategies for diagnostics. I do some diagnostic reads and I would love to have something like this pre-draft reports (especially for X-Rays). There are tons of "AI in rads" companies right now, none of which have models that come anywhere close to GPT-4 or even this. Personally, I would not start such a company because the barrier to entry is now so low. I think there needs to be an open-source project that just subtrains these models on rads images and is available to all radiologists.

euroderf 4y ago
Your profession and... a few hundred others ?

jiwidi 4y ago

a bit nasty to call something minigpt-4 when it comes from llama

lxe 4y ago

Why Vicuna over something like oasst-pythia? Would love to see a table comparing all the new models side by side.

GaggiX 4y ago
All Pythia models were trained on 300B tokens, LLaMa models were trained on 1/1.4T tokens.
lhl 4y ago
Fabrice Bellard has run a standard set of benchmarks w/ lm-eval on a big chunk of open models here: https://bellard.org/ts_server/ https://bellard.org/ts_server/ - Flan T5 XXL and GPT-NeoX 20B both outperform Pythia 12B on average (LLaMA 13B+ tops the charts).
- deleted 4y ago
  [deleted]

m3kw9 4y ago

This is basically Image to Text, then from the text, start a GPT type chat. No need to overthink anything

jokoon 4y ago

Are there more university/research centric resource of communities regarding AI, I mean things on a more epistemic and higher level, that disregards current ML techniques and implementations? I have no problem when there is a lot of money spent to implement things that can be used and tested, but I wish I could read about the next step of AI, something that involves science and not software engineering. I've yet to read anything related to the analysis of trained data or trained neural networks, the structure of the brains of small insects, psychology on general intelligence, AI and emotions, the real ability of an AI to think and write meaningful code or understand what is a problem/solution etc, I mean anything that is related to AI that can be treated by computer science. I have no problems with algorithms and implementations, but don't university/professors also try to go beyond, with epistemology, to connect the dots with biology and neurology? Unless I see science cover several fields of research, it's not real, worthy research in my view, which is why the reason why I'm so stubbornly skeptical of chatGPT.

thewarrior 4y ago
We’ve tried but we understand even less about how “real” intelligence works. Atleast in a way that we can replicate it. The people doing so called alchemy are making far better systems that anyone who’s trying to actually understand things. And we’ve been trying for decades now.
visarga 4y ago
If the engineering part is of secondary importance for you, then at least remember about the dataset. It is in the composition and quality of the training data that all the skills find their origin. Most of our discussions are about model size, but few about dataset. Yet all the scaling laws hint at the great usefulness of more data. Sometimes even little data can have a great impact in the fine-tuning phase. In the end it is the training data that transforms a random init into the model.

ggregoire 4y ago

Could you use something like this to summarize the content of a video? Or would you first need to extract the images from the video (like 1 fps, with ffmpeg) and tell the AI to analyze all the images and summarize them like if it was a video?

all2 4y ago
I'd do this using both streams of information: audio and video. I'd segment the audio semantically based on the topic of discussion, and I'd segment the video based on editing, subjects in scene, etc. We could start simply and just have a "timestamp": [ subjects, in, frame] key-value. It'd take some fiddling to sort how to mesh these two streams of data back together. The first thing I'd try is segment by time chunks (the resolution of which would depend on min/max segment lengths in video and audio streams) and then clump the time chunks together based on audio+video content.

nobrains 4y ago

I think the real news is where this research is coming from. It is from King Abdullah University of Science and Technology in Saudi. The arab world has recently been quite behind in tech research, but this is good news overall for the arab world, which will inspire other universities and researchers in the region to one-up their colleagues, which is overall good for the entire world.

sterlind 4y ago
I would be more thrilled if it were literally any other country in the Arab world, but maybe that will come.

23459872345 4y ago

Has anyone gotten this running in Colab?

Imnimo 4y ago

I'm pretty surprised that the frozen encoder has enough information in its representations to do the "design the website from a hand-written whiteboard mock-up" task, and that that information can be effectively accessed by the language model with just a small projection layer. I really would have guessed that you needed to be training the image encoder simultaneously.

deeplstm 4y ago

Here's the video summary https://youtu.be/uPV9Gk3IC-g https://youtu.be/uPV9Gk3IC-g

djangovm 4y ago

If I have to learn how to be able to read this code and understand what it and its dependencies are doing, where do I start? Is reading their paper an effective strategy? https://github.com/Vision-CAIR/MiniGPT-4/blob/main/MiniGPT_4.pdf https://github.com/Vision-CAIR/MiniGPT-4/blob/main/MiniGPT_4...

HansHamster 4y ago

Just tried it with a random schematic: > The image shows a block diagram of a power amplifier circuit with a power supply and a load resistor. The power supply is connected to the input of the amplifier, which is then connected to the output. The load resistor is connected to the output of the amplifier and the power supply. The amplifier is designed to amplify the input signal to a higher level, which is then sent to the load resistor to power the load. The power supply provides the necessary voltage to operate the amplifier. On the one hand I'm impressed that it recognises an amplifier, but just about everything in the description is (slightly) wrong. It is not a power amplifier, the power supply is not connected to the input (why should it be?), there is no load resistor...

mftb 4y ago
I also got middling results. I gave it a novel NASA picture of the moon passing in front of the earth, taken by the Deep Space Climate Observatory Satellite, so an unusual vantage point. First it was just wrong, and said that it was a picture of a crescent moon taken from a mountain top, or something like that. Not terrible at least it recognized the moon, but wrong. I tried to give it a chance and corrected it with the additional info that the picture was taken from space. It then did better, but on the last line it threw in that it was not a photograph, but a rendering of some sort, which again, is just wrong. More and more I get the impression that people are so taken with the chat interface that they are grading these models more favorably than is warranted.
- HansHamster 4y ago
  And after pointing out all the mistakes in the description, it took a very ... interesting approach to address my remarks: > I apologize for the confusion. The image shows a block diagram of a basic amplifier circuit with a power supply, but it does not show any input stages, output stages, or load resistors. [...] The input stage is made up of an operational amplifier, which is used to amplify the input signal. The output of the input stage is then connected to the power supply, which provides the necessary voltage to operate the amplifier. The amplifier is designed to amplify the input signal to a higher level, which is then sent to the load resistor to power the load. So it acknowledges the remarks, describes everything that is _not_ in the image and then puts the stuff back into the description anyway :)

morelight18 4y ago

[dead]

cloudking 4y ago

It doesn't seem to recognize text from a screenshot very well, I gave detailed error messages from a Windows screenshot: https://filestore.community.support.microsoft.com/api/images/485f9156-e955-4acc-a399-60d16cc64521 https://filestore.community.support.microsoft.com/api/images... and prompted "Describe any issues found in this screenshot and steps to resolve them" while it correctly identified it as a screenshot from a computer, it gave a very generic response and didn't identify the error messages correctly: "The screenshot shows a computer screen with a warning message that reads, “The system is currently unavailable. Please try again later.” This indicates that there is an issue with the computer system that needs to be resolved. To resolve this issue, you can try the following steps: 1. Check the system status: Check the system status to see if there are any errors or issues that need to be resolved. 2. Restart the computer: Try restarting the computer to see if that resolves the issue. 3. Check for updates: Check for any updates that may have been released for the computer system and apply them if necessary. 4. Check for viruses: Run a virus scan on the computer to check for any viruses that may be causing the issue. 5. Check for hardware issues: Check for any hardware issues that may be causing the issue. This could include issues with the power supply, hard drive, or other components. 6. Contact technical support: If none of the above steps resolve the issue, contact technical support for further assistance. They may be able to provide additional troubleshooting steps or help you resolve the issue."

MacsHeadroom 4y ago
I can barely read the text in that screenshot, and I'm a human (I think). It is both low resolution and filled with jpeg artifacts.

raylad 4y ago

I gave it a picture of a bottle, which it did a fairly good job of identifying (although it misread some of the text, mistaking a 1 for a 4). Then I asked it what are the likely ingredients of the product. It still hadn't replied after 2274s so I gave up on it.

nyanpasu64 4y ago

I hope that someday language models will be trained to accurately reflect their training data and real-world text, and not be hard-coded to spit out disclaimers like "As an artificial intelligence, I do not have personal feelings or emotions" in response to any corporate-unfriendly inputs.

no_time 4y ago

https://imgur.com/a/jEQzRzT https://imgur.com/a/jEQzRzT Great job at OCR-ing the text and recognizing the context, but it still can't do doom and gloom like us eastern Europeans.

smashfizzle7 4y ago

[dead]

monsoon11 4y ago

[dead]

blooms112 4y ago

[dead]

floufrouu11 4y ago

[dead]

morelight19 4y ago

[dead]

sharinggenes1 4y ago

[dead]