17 ms·
Visual ChatGPT
- zhangyiwu 4y agoI have tried to use my Macbook pro(M2 pro) to run it out, but failed to download the massive file。
- iamflimflam1 4y agoLinked paper is available here: https://arxiv.org/abs/2303.04671 https://arxiv.org/abs/2303.04671
- spagoop 4y agoVery cool. It's almost as if that chat session is a terminal, but instead of running commands you run prose. Very much a new HCI paradigm.
- copperx 4y agoI wonder what's more difficult: engineering a good prompt, or remembering commands and arguments? Also, is there a ChatGPT terminal where I can enter a prompt and get a fully-fledged command in response? Seems like very low hanging fruit.
- Loeffelmann 4y agoMade a simple python script: https://gist.github.com/Loeffeldude/41f3a71570f4bad1aff2749be2ea653b https://gist.github.com/Loeffeldude/41f3a71570f4bad1aff2749b... Works suprisingly well! pip install openai && export OPENAI_API_KEY=yourkey python3 chatgpt-terminal.py "make a dir test"
- danielbln 4y agoThere are plenty of GPT shell integrations, a quick Github search will reveal plenty. Check out this one, it seems rather new and integrates Dall-E as well as the ChatGPT API: https://github.com/0xacx/chatGPT-shell-cli https://github.com/0xacx/chatGPT-shell-cli
- seydor 4y agoThe new google images. I wonder how Pinterest will spam this
- dalbasal 4y agoYes, and I am fascinated to see what happens in this space as we go. Terminal based applications have always had the advantage of playing nice with eachother. That's at play here. Produces an easy way to have multiple models interact. Btw... Now that everyone is monkey typing prompts at got all day, is got going to start emulating a gpt user... autocompleting and auto generating prompt sequences?
- userbinator 4y ago"ChatGPT, I meant a desk with legs." For a second, I thought this was a Visual Studio-related plugin.
- ilaksh 4y ago[flagged]
- shubhamjain 4y agoJust to let you know, I am downvoting this post because it doesn't feel like a good plug. Every one of your recent post talks about your own project aidev.codes. That's not good way to market your product.
- ilaksh 4y agoI spent 3 months working on it, couldn't even get a single person to talk about it on here. Totally out of money. Submission completely ignored. Desperate. I think HN failed and all of the hate I feel just as strongly back.
- Regnore 4y agoNo offence but this comes across as falling into the fallacy of “I spent time on it therefore it must have value!” > I think HN failed Is it really HN that failed here? If you really believe that HN is the issue, may I suggest that you consider marketing your product somewhere else?
- yesenadam 4y agoHate? Sounds like total indifference. But 3 people commented last time you submitted it 12 hours ago, saying it's great. ...Although, those 3 commenters are brand new accounts with 1 comment only. Are they you?! The way you say you couldn't get a single person to talk about it on here makes me suspicious. Not an expert, but it sounds like you forgot what I'd say is the #1 startup/business lesson HN teaches - not to spend time/money developing a product before you know there's demand for it, people who will pay for it. That's the advice people would give, I guess, had you asked for help instead of writing this contemptible comment, which I flagged.
- ilaksh 4y agoThey are not me they are clients/friends. You are totally wrong. Its impossible that there is not demand for this. People still make web pages. People still want ChatGPT to output things related to their Stable Diffusion images or visa versa. The problem is that almost no one has even seen the web page.
- spaceman_2020 4y agoMan, Microsoft is kicking ass at AI. Maybe the others have great AI models too but haven’t seen any large company release product after product with AI.
- ksd482 4y agoYes. It's their strategy that makes them stand out. Ever since Satya took over, they have really turned the company around and reinvented themselves in terms of community involvement when it comes to software; embracing open source and .NET Core are just a few examples.
- Kibae 4y agoI think it's largely due to Sam Altman bringing the YC ethos to OpenAI. Building something people want, releasing products quickly, etc.
- muzani 4y agoIt's not a uniquely YC thing. Every major company did this well at some point. Microsoft is in that sweet spot where they're big enough to fund whatever they want, but not FAANG-tier, which tends to attract political types.
- DoctorOW 4y agoMicrosoft is the same size as the biggest companies in FAANG and the office politics are notoriously pretty competitive.
- Der_Einzige 4y agoThey also pay better than Amazon and have (or had based on recent blind reviews) excellent WLB
- tester756 4y agoMSFT is second biggest FAANG right after Apple and significantly ahead of #3, wtf?
- tomohelix 4y agoI guess one of the advantage of being early is that Microsoft get to pick all the low hanging fruit first. All of these products are very useful and interesting by itself but it is still too early to know if MS can continue to refine and maintain a competitive edge. Dall-E basically died in a few months, unable to compete. Hopefully these other stuff will have better fate.
- sharkjacobs 4y agoWe're at the point where these generative AIs are good enough that they're doing things which are really surprising and unexpected and kind of exciting, but they're bad enough that almost everything they create falls somewhere between mediocre and dogshit. I really hope, if these this stuff is going to be ubiquitous, that there are big strides made in improving the quality of the output, very soon. The novelty of seeing fake screencaps of Disney's Beauty and the Beast directed by David Cronenberg is wearing off fast, and aside from some very niche use cases (write some boiler plate code for this common design pattern in this very popular language) I haven't found much it's actually useful for
- sharkjacobs 4y agoI should probably clarify, this is a response to the visceral negative reaction I felt upon seeing the cat and dog images generated in the demo. They look so bad that it's actually upsetting to me, which is a reaction that I've had to a lot of AI art.
- swatcoder 4y agoDon’t sweat the downvotes. People have very heated opinions about this technology and any strong statement will have more vote-volatility than usual. I appreciate you putting a voice to what you said. The historical novelty of what these generative AI’s produce is striking, but the variety of style and form is still narrow enough that you start seeing the signature of CharGPT/SD/etc once that historical novelty wears off. They each have a strong, specific voice and it gets as tiring as any other art style that’s overexposed.
- zirgs 4y agoNobody uses SD base models any more. There are thousands of custom models available now and each of them can be extended even further using LORAs, text inversion, hypernetworks, ControlNet and so on. Your imagination (and your VRAM) is the limit.
- 4y ago
- p-e-w 4y agoThe "memory usage" section of the README highlights the surprising fact that image generation models need much less memory than text-based language models. ChatGPT itself is by far the most resource-hungry part of the system. Why is that so? It seems counterintuitive. A single picture snapped with a phone takes more space to store than the text of all the books in a typical home library, yet Stable Diffusion runs with 5 GB of RAM while LLAMA needs 130 GB. Can someone illuminate what's going on here?
- DaiPlusPlus 4y agoI have zero qualifications to talk about this subject, but from what little I do know it's because Stable Diffusion's network has "only" ~900m parameters, whereas ChatGPT has 175 billion parameters.
- flangola7 4y agoI wonder what a 100B image model will create.
- cyanf 4y agoImages, most likely.
- tomohelix 4y agoI dread that. You heard about Loab? That stuff is haunting. I am scared of what a 100B param model can create in the deep recesses of its system.
- DaiPlusPlus 4y agoI thought Loab was just a creeopypasta?
- JeremyBanks 4y agoNo, Loab was an extreme value in parameter space which certain inputs could converge too.
- lwneal 4y agoThe most incredible thing about this system is that it uses Stable Diffusion (the open source AI art generator), rather than DALL-E (the proprietary closed art generator owned by OpenAI). The fact that even Microsoft, which partially owns OpenAI, is giving up on DALL-E shows the power of building an open-source community around models with published, downloadable weights.
- MuffinFlavored 4y agoSo why hasn't there been a "as popular as ChatGPT" open source version?
- zamnos 4y agoTwo things stand out. Stable Diffusion; the weights for it and the supporting Python code was open sourced and released to the public. Anyone can find the cpkt file and the Python code online and download it and (with a bit of work) run it on their own computer if they have the hardware (any reasonable GPU) for it. There has been no such release by OpenAI, the closest we got is Facebook's leaked LLama model, and that's not a chat bot. So we don't have a model to run. Stability AI/Emad paid to train the model, which cost like a half million dollars in GPU time (he obviously didn't pay the retail price of $600k, but also it's not something you'd get right on the first shot either) and then gave the output from that away. It's not clear how much it would cost to train a comparable chat bot to ChatGPT but the impression is that would take much more. The second thing is that it's not clear that we, the Internet at large would actually benefit from the model's release. StableDiffusion is 4 gigs and able to run on all sorts of consumer grade hardware, leading to such a Renaissance. ChatGPT makes liberal use of Nvidia A100 GPUs that are available to them to use as compute in Azure. (AWS and GCP, along with many AI focused smaller cloud companies also offer these.) One of those costs, like, $10k. And you need several of them to be able to run ChatGPT. Which means even if OpenAI were to live up to their name and release ChatGPT's model, only businesses and research labs would actually have the hardware to run it, so it would be awesome to have the weights, but you wouldn't have the same army of developers able to work on it. There are open source LLM chat bots out there, so I think we will see one become popular, but at least that says why it'll be a second before we do.
- hackerlight 4y agoThere are more examples in the paper: https://arxiv.org/pdf/2303.04671.pdf https://arxiv.org/pdf/2303.04671.pdf
- kilgnad 4y agoNow is a really good time to make a start up called skynet.
- tuanx5 4y agoThis reminds me of Christina's workstation in Westworld Season 4
- est 4y agoMicrosoft is releasing second toy while Google had trouble launching its first.
- doctoboggan 4y agoWow, this is very timely! I just finished up a script that uses ChatGPT (via openAI APIs) to read my customer support messages on Etsy and generate a response. Since I often send and receive images via Etsy support (my customers can customize the product with images) I have been searching for a way to let ChatGPT "know" what the image is. Current the script just inserts the text "<uploaded image>", but I was just hacking together something using stable-diffusion-webui's API (interrogate using CLIP), but was struggling with a few things. I took a break to browse HN and this pops up! I will definitely be taking a look to see how this works and will try to get it integrated with my script.
- genewitch 4y agoCLIP itself is available by itself (BLIP, etc) via python; useful for "output a csv containing filename, filesize, and description of every image in this directory". is the stable-diffusion-webui API... stable, yet? I was looking at the json it's chucking back and forth from the frontend to the python backend, and it looks like it should be able to be parsed without a library.
- totetsu 4y agoAre there any recommendable resources for learning about designing these kind of system architectures?
- swyx 4y agoi have been trying for an hour and am completely unable to run this project. currently facing a "Building wheel for numpy (pyproject.toml) did not run successfully." error. the state of python dependency management and project distribution is just abjectly horrible. --- update: perhaps spoke too soon. just made it work! https://github.com/microsoft/visual-chatgpt/issues/37 https://github.com/microsoft/visual-chatgpt/issues/37
- muyuu 4y agohow much disk space does it take in total?
- swyx 4y agomacs are very hostile to calcing disk space but i think 44GB? $ df -h . Filesystem Size Used Avail Capacity iused ifree %iused Mounted on /dev/disk3s5 926Gi 354Gi 522Gi 41% 2722533 5478019360 0% /System/Volumes/Data $ du -sh . 44G . looks like most of it is the ControlNet folder which holds all the models
- swyx 4y agomanaged to get this working - images in github comment: https://github.com/microsoft/visual-chatgpt/issues/37#issuecomment-1463613300 https://github.com/microsoft/visual-chatgpt/issues/37#issuec... as can be expected the results look extremely cherry picked
- trompetenaccoun 4y agoEndless new possibilities for online scammers. Bright times ahead.
- mmq 4y agoI think the chat interface is a bit restrictive when it comes to multimodal models. A much cleaner interface would be an "AI notebook" where the user can move, compare, rerun blocks. Also sharing, versioning and collaborating with others on notebooks is more straightforward.
- pedrovhb 4y agoThat's neat, but it's not doing anything in the latent space of ChatGPT, is it? As I understand, it basically teaches the assistant to use SD for generating images/descriptions, but comes with all the limitations of the image model being used (as opposed to a leap in results quality such as GPT 3.5 itself was). Teaching it to use tools is of course an interesting concept itself, though.
- shp0ngle 4y agoI think they ate using StableDiffusion and not Dall-E? Which makes it kind of funny
- aaronrobert 4y agoChatGPT now is not only a simple standalone AI model, but a powerful AI core engine, and more and more people or companies will develop more and more interesting things based on ChatGPT. Like this awesome visual ChatGPT.
- Havoc 4y agoHappy that this is <8gb vram. Neatly fits into medium/highish consumer GPUs
- iandanforth 4y agoThis feels like it owes more to LangChain than a link at the bottom of the page. Compare their prompt: https://github.com/microsoft/visual-chatgpt/blob/main/visual_chatgpt.py#L51 https://github.com/microsoft/visual-chatgpt/blob/main/visual... With that of the LangChain ReAct conversational agent: https://github.com/hwchase17/langchain/blob/master/langchain/agents/conversational/prompt.py https://github.com/hwchase17/langchain/blob/master/langchain... Also it seems appropriate to cite the original ReAct paper (from Google mainly) https://arxiv.org/abs/2210.03629 https://arxiv.org/abs/2210.03629
- LastTrain 4y agoI think you mean here: https://github.com/hwchase17/langchain/blob/master/langchain/agents/conversational/prompt.py https://github.com/hwchase17/langchain/blob/master/langchain...
- iandanforth 4y agoFixed thanks!
- mcaledonensis 4y agoYes, in this integration GPT3 connects with the image space only through text and doesn't consume or emit image space embeddings / tokens. With GPT4 that perception modality will be available directly in the model. And it'll be a better integration.
- skybrian 4y agoHow do you know? I’ve seen multiple claims about GPT4 online but people never say where they came from.
- osigurdson 4y agoI think GPT is super useful but can't seem to eke any value out of DAL-E. Yes, it can draw a bear in a business suit on the beach well, which is impressive but I can't think of how to utilize this. As an example, I've tried to get it to draw architecture diagrams, it draws a few boxes but then places the strangest text on those boxes.
- sebzim4500 4y agoYeah, I think simply generating an image from a prompt is of dubious value for most usecases. Inpainting, etc. with SD and control net is really handy for image editing though. You can make changes in seconds that would normally take a professional hours in photoshop.
- userbinator 4y agowhich is impressive but I can't think of how to utilize this Art is probably the best use. The "filler images" and other miscellaneous background art that adorn lots of articles aren't really expected to be 100% accurate or even worth looking at for more than a moment, and I think this is where AI-generated art will mostly fit.
- status200 4y agoI do this for my blog, i love to type in a conceptual prompt related to the topics covered and get a bizarre amalgamation to use as filler art.
- deleted 4y ago[deleted]
- pornel 4y agoI've found it super useful for making app icons. It's also really good at picking pleasing color schemes. It's useful for brainstorming ideas. If you need a cover for a book about Business Bears, you can quickly evaluate many variants, especially if you're not quite sure what you want, and you'll only know when you see it.
- gavi 4y agoIf you are trying to run this on a single GPU, please be aware the models take up a lot of memory. You can reduce the number of tools by modifying the self.tools portion of the python script
- harveywi 4y agoMeta will probably soon release a competing technology. It will be called "DALL-E LLaMA".
- deleted 4y ago[deleted]
- walthamstow 4y agoIf this an original joke? If so, chapeau.
- JoeJonathan 4y agoKevin Roose made some version of it on the Hard Fork podcast.
- layer8 4y agoThat will get it banned in China. ;)
- SilentM68 4y agoLOL
- pmarreck 4y agothe pace of all this is astonishing, this is amazing
- qntmfred 4y agohmmm can I use this to see how far away we are now https://karpathy.github.io/2012/10/22/state-of-computer-vision/ https://karpathy.github.io/2012/10/22/state-of-computer-visi...
- amccloud 4y agoIve created a little api to grab images from pages to embed in chats. Was surprisingly easy to control with natural language. https://aimgsrc.com https://aimgsrc.com
- yazzku 4y agoThe shit has an MIT license... then requires an API key. Open source all the way, guys! Microsoft loves Open Source!
- golol 4y agofuture AI systems based on LLMs and other foundation models might think less like individuals and more like companies. Ironically, LLMs might finally make symbolic AI possible! The way I see it, symbolic AI was always missing a small sprinkle of "general intelligence" too amooth things out, to grease the gears and connect interfaces. I feel like LLMs have that little bit of magical "generality" so we can start building "symbolic" AI systems which produce work by managing a number of black box models. It is like a company: protocols and management structures are a sort of symbolic AI that connects black box humans to eachother.
- OliverZt 4y ago[flagged]
- razodactyl 4y agoThe comprehension thrown around in this thread is beautiful. Love the passion.