7 ms·
Talk = GPT-2 and Whisper and WASM
- tomthe 4y agoThis would of course be even more fun with ChatGPT, but it is a nice and funny demo of their whisper.cpp library. The second video is worth watching: https://user-images.githubusercontent.com/1991296/202914175-115793b1-d32e-4aaa-a45b-59e313707ff6.mp4 https://user-images.githubusercontent.com/1991296/202914175-...
- sheeeep86 4y agoIt's interesting that the english language model is loaded and it's clearly trying to pronounce things in a spanish way.
- dr_kiszonka 4y agoI think LaMBDA would be really fun. If you asked ChatGPT what movies it likes, it would tell that it is a large language model trained by OpenAI and it can't have opinions yada yada yada.
- pmontra 4y agoI understood that this limitation is circumvented with prompts like Imagine there is a guy that likes watching movies. Which ones would he like most in 2022? That context persists for a while.
- Rickvst 4y agoI implemented whisper + chatgpt + pyttsx3 and it worked. But then suddenly the chatgpt wrapper that I found on github stopped working. edit: whisper is awesome
- localhost 4y agoIt looks like the ChatGPT APIs that work well are the ones that are implemented as a browser extension and reusing the bearer token that you get by signing into ChatGPT from the same browser. I'm guessing since you're using pyttsx3 that you wrote a Python app instead and not in the browser?
- lhuser123 4y agoCool. Would like to see that.
- swyx 4y agoThe total data that the page will have to load on startup (probably using Fetch API) is: - 74 MB for the Whisper tiny.en model - 240 MB for the GPT-2 small model - Web Speech API is built-in in modern browsers cool but im now wondering what it would take to bring this down enough to put this in real apps? anyone talking about this?
- arcturus17 4y ago~314mb is a lot for a web app but small for a desktop or even a mobile app.
- agolio 4y agoI really liked how the page tells you the size it is planning to download, and prompts you before downloading. Coming from a limited bandwidth contract, I hate when I click a link and it instantly starts downloading a huge file. Great work OP!
- justanotheratom 4y agoPerhaps it will be built-in to browsers soon
- petercooper 4y agoGiven Whisper is open source, I'd be surprised if it's not. It would be cool for Web Speech API's SpeechRecognition to simply use it, though that would make browser downloads a little beefier.
- boredemployee 4y agoofftopic but what are the real limitations of gpt2 vs gpt3? (i know that gpt2 is free)
- mcbuilder 4y agoSize of the model is a big one. GPT-3 has over 10x as many parameters for example. Training data would be another huge one. Architecturally, they aren't that different if I recall correctly, it's a decoder stack of transformer like self-attention. Real world capability has GPT-3 giving much better answers, it was a big step up from GPT-2.
- boredemployee 4y agogot it. thanks! is there any application that gpt2 would be enough and could work as well as gpt3?
- namrog84 4y agoSo how 'big' is GPT-3? Is it anywhere near being able to be run on local consumer hardware? How long until we can have the GPT3 or 3.5 chatbot locally like we have StableDiffusion locally for image generation? I've been spoiled by having it accessible offline and with community built support/modifications to it. GPT-3 is super neat but feels like too many guard rails or the custom playground is too pricey.
- lossolo 4y ago> So how 'big' is GPT-3? For inference so basically running it you need multiple GPUs and hundreds of GBs of GPU memory. As to model size it's around 100x bigger than SD. You can forget about running it locally unless you have dozens of high-end GPUs or you want to wait hours/days/weeks (depending on your hardware) for a single response.
- zwaps 4y agoIt's almost the same model architecture, but GPT3 is much better trained. GPT3 is coherent, while GPT2 is prone to generating gibberish or getting stuck in a loop. The advantage is pretty significant for longer generations. That being said, neither GPT3 nor GPT2 are "efficient" models. On the one hand, they use inefficient architectures - starting with using a BPE Tokenizer, to having dense attention without any modifications, to being a decoder only architecture etc. Research has come up with many more fancy ideas on how to make all this run better and with less compute. But there is a reason why GPT2/3 are architecturally simple and inefficient: we know how to train these models reliably (more or less) on thousands of GPUs, whereas the same might not be true for more modern and efficient implementations. For instance, when training OPT, Facebook started using more fancy ideas but finally ended up going back to GPT-3 esque basics, simply because training on thousands of machines is a lot harder than it seems in theory. On the other hand, these models have far too many parameters compared to the data they were trained on. You might say they are undertrained - or they lean heavily on available compute to make up for missing data. In any case, much smaller models (like Chinchilla by DeepMind) match their performance with less parameters (and hence compute or model size) by using more and better data. In closing, there are better models for edge devices. This includes GPT clones like GPT-J in 8bit, or distilled version thereof. Similarly, there is still a lot of gains that will happen when all the numerous efficiency improvements get implemented in a model that operates at the data/parameter efficiency frontier. Still, even when considering efficient models like Chinchilla and then even more architecturally efficient versions thereof - we are still talking about a lot of $$$ to train these models. And so we are yet further from having OpenSource implementations of these models than we are from someone (like DeepMind) having them... With time, you can expect to run coherent models on your edge device. But not quite yet.
- iandanforth 4y agoTechnically this seems to work, and mad props to the author for getting to this point. On my computer (MacBook Pro) it's very slow but there are enough visual hints that it's thinking to make the wait ok. I have plenty of complaints about the output but most of that is GPT-2's problem.
- Terretta 4y agoListening to that demo, it's incredible how far we've come! Or, not. Racter was commercially released for Mac in December 1985: Racter strings together words according to "syntax directives", and the illusion of coherence is increased by repeated re-use of text variables. This gives the appearance that Racter can actually have a conversation with the user that makes some sense, unlike Eliza, which just spits back what you type at it. Of course, such a program has not been written to perfection yet, but Racter comes somewhat close. Since some of the syntactical mistakes that Racter tends to make cannot be avoided, the decision was made to market the game in a humorous vein, which the marketing department at Mindscape dubbed "tongue-in-chip software" and "artificial insanity". https://www.mobygames.com/game/macintosh/racter https://www.mobygames.com/game/macintosh/racter https://www.myabandonware.com/game/racter-4m/play-4m https://www.myabandonware.com/game/racter-4m/play-4m It's only amazing that chatGPT backed by GPT-3 is the first thing since then to do enough better that everyone is engaged. I owned that in 1985, and having studied AI/ML previously I've been (and remain something of) an AGI skeptic. But now in 2022, I finally think “this changes everything” ... not because it's AI, but because it's making the application of matching probabilistic patterns across mass knowledge practical and useful for everyday work, particularly as a structured synthesis assistant.
- Centigonal 4y agowell, the AI Winter happened in the intervening years, so that might help explain https://en.wikipedia.org/wiki/AI_winter https://en.wikipedia.org/wiki/AI_winter
- make3 4y agoGPT-2 is really by far massively stronger than anything in 1985. I suggest that you try using https://chat.openai.com/chat https://chat.openai.com/chat
- rozularen 4y agoOpenAI chat uses GPT-3 which, as some other user already pointed out, is not even close to GPT-2 in terms of generating text
- atum47 4y ago> whisper: number of tokens: 2, 'Hello?' > gpt-2: I want to have you on my lap. this GPT-2 better chill
- hanoz 4y agoWhat are some good things to try? I can't get any sense out of it at all so far.
- ggerganov 4y agoThis is the smallest GPT-2 model so it usually generates gibberish. Maybe some better prompting could improve the results. Currently, the strategy is to simply prepend 8 lines of text (prompt/context) and keep appending every new transcribed line at the end: https://github.com/ggerganov/whisper.cpp/blob/master/examples/talk.wasm/index-tmpl.html#L560 https://github.com/ggerganov/whisper.cpp/blob/master/example...
- rahimnathwani 4y agoI'm curious how they chose between: A) ggml https://github.com/ggerganov/ggml/tree/master/examples/gpt-2 https://github.com/ggerganov/ggml/tree/master/examples/gpt-2 B) Fabrice Bellard's GPT2C https://bellard.org/libnc/gpt2tc.html https://bellard.org/libnc/gpt2tc.html
- ggerganov 4y agoHey author here - I implemented `ggml` as a learning exercise. It allows me to easily port it to WebAssembly or iOS for example.
- rahimnathwani 4y agoOops - I didn't spot it was your own libary! Kudos!
- bilater 4y agoI've been thinking of doing something like this but hooked up with ChatGPT/GPT-3-daviinci003. Obviously model will not load in the browser but we cna call the API. Could be a neat way to interact with the bot.
- simonw 4y agoAnyone found a sentence that GPT-2 returns a good response for? My experiments have been not great so far. (LOVE this demo.)
- thundergolfer 4y agoThis guy's doing really great work recently. Keep it up, Georgi!