9 ms·
Serverless Video Transcription inspired by Cyberpunk 2077
- daedalus2027 4y agoIt seems there is a niche market in this...
- jacquesm 4y agoMore than just a niche. Imagine a pair of glasses a tourist would wear that would translate in real time what that waiter just said.
- vorpalhex 4y agoThis would be a killer app for a google glass type device. A near realtime closed captioning device with translation.
- fennecfoxy 4y agoIt made me so sad to see how Glass was neglected/spurned by the public. Especially because it had a camera. I don't think people understand yet that this stuff is inevitable; whether it takes 20 years or 50 years eventually we (or the next generation) will _all_ be wearing these. So why not do it now? For all the bad it could do, it could do so much more good/be useful. & if people are worried about the cameras on them; they should be worried about the camera that they _can't_ see, like the OmniVision OV6948.
- chinchilla2020 4y agoThis is impressive and so is the writeup. How did you create the diagrams in the readme? I really like that visual format for diagrams.
- super_linear 4y agoFor the hand-drawn style ones, I think it's https://excalidraw.com/ https://excalidraw.com/
- navanchauhan 4y agoPretty darn cool! I almost want to try and implement this in OpenCVjs and see if it is possible to build a browser extension to do this in the browser itself
- ArekDymalski 4y agoNow this is both cool piece of tech and practically useful. Very nice!
- totetsu 4y ago>Matching faces to voices relies on simple co-occurence heuristic, and will not work in certain scenarios (e.g. if the whole conversation between two people is recorded from a single angle) This seems like the really hard part. maybe if there was a way to find the time lips moves for a face. or guess gender and age of both face and voice.. or If the audio is a stereo mix, using relative position
- photoGrant 4y agoThere's this that can differentiate speakers https://github.com/CorentinJ/Real-Time-Voice-Cloning https://github.com/CorentinJ/Real-Time-Voice-Cloning
- oslem 4y agoI think you are referring to GE2E [1], which is the speaker identification model used in that project. [1] https://arxiv.org/abs/1710.10467 https://arxiv.org/abs/1710.10467
- melony 4y agoThat's from an eternity ago. Speaker diarization has come a long way.
- ChrisKnott 4y agoWhat's the best pretrained model available? The best I've tried is pyannote.
- totetsu 4y agoBut matching a speaker to a face is a different problem differentiating between speech in a recording
- holler 4y agoThis is awesome, impressed you threw this together over a weekend! What did you use to make that entity diagram? edit: answered below
- deleted 4y ago[deleted]
- abidlabs 4y agoNeat use of Gradio & Modal
- throwaway74932 4y ago[flagged]
- Mizza 4y agoI'm really hoping somebody comes along and puts this in a format that I can attach to my head inconspicuously.
- shadowgovt 4y agoGoogle Glass has already been rejected by the public; I imagine anything more inconspicuous would arouse even deeper suspicion.
- pyinstallwoes 4y agoIt’s inevitable though. Is it not? Contact lenses, eye, and so on.
- colordrops 4y agoIt was rejected because it didn't do anything useful. Certainly there are many really useful things we could do now that were not possible when it was first released, no?
- fennecfoxy 4y agoIt didn't do anything useful because it was a prototype, nobody really got the time to build great apps for it, and it didn't get rejected for anything except: it was bulky (can be solved) and it has a camera on it so privacy implications (ie naive hoomans/public who don't realise this is inevitable/how much they are already being recorded in public).
- ajuc 4y agoIt was ahead of its times (and wasn't that useful), in 10 years it will be perfectly acceptable.
- fennecfoxy 4y agoWhat's funny is that the opposite is true. People will cry about the big obvious camera on the glass, but won't cry about a pair of thick framed but normal looking glasses because they have no idea that it has a camera on it. As mentioned in another post chuck a OmniVision OV6948 or similar in the glasses and nobody will bat an eye because they simply won't know.
- jacquesm 4y agoMindblowing that this was so easy to build. Coming soon to a phone near you, and in realtime.
- piyh 4y agoPeople complain that the magic of programming has been lost because all we do is stitch together APIs. They're oversimplying the work done, those people are cynical, this is amazing.
- coding123 4y agoI think part of the magic is still lost, as we used to be very open source and free centric. These days it's all based on paid API calls.
- ccn0p 4y agoagree with parent re: stitching together -- but also, someone's gotta program the API!
- johnathandos 4y agoSomeone is out there building and maintaining the APIs
- yencabulator 4y agoOne hopes. If they decide to stop providing/maintaining the API, you're out of luck. Just using APIs also misses out on the learning, community, and collaboration of open source software.
- fragmede 4y agoThe magic is still there! In this case, the model for OpenAI's Whisper, which is arguably doing the bulk of the work here, is Open Source (under the MIT licence), and freely available for download at https://github.com/openai/whisper https://github.com/openai/whisper. You can run it wherever you want, though something with a GPU will let you do 5x realtime (or better!) transcription.
- beecafe 4y agothose APIs would be worthless if nobody wanted to stitch them together
- danso 4y agoClever hack/solution, and the thorough documentation is appreciated!
- MetallicCloud 4y agoVery cool. Zack Friedman made something similar to wear on a hoody. It wasn't perfect, but it worked. https://youtu.be/mTK8dIBJIqg https://youtu.be/mTK8dIBJIqg
- bee_rider 4y agoHow easy is it to translate word by word like that? I was under the impression that generally it is hard because different languages have different word orders. Is it not necessary to have the whole sentence before starting? (Or maybe Polish is just conveniently similar to English in word order?)
- ehnto 4y agoYes it's not the best way to translate, and for some languages it won't make much sense since the word order is different, as in English to Japanese translation. To say nothing of all the myriad nuances of language like context changing word meaning and colloquial meanings of words. In the game Cyberpunk, when doing the inline translations between english and japanese it is actually swapping fully translated sentences out, so not word for word, the effect just looks similar.
- shadowgovt 4y agoI really enjoy that aspect of the visuals because it's like a tiny reflection of the kernel of cyberpunk (the genre, not the game). Here you have a technology that would work in a way totally different to the way people assume... But someone put the effort into making it look like it does something else because the skeuomorphic expectations of the human makes that inaccurate representation feel more correct. That gap between the way tech works and the way our perceptions and psychology drive us to assume it works is very real, and it underpins piles of tech we use everyday, from the icons for making calls and hanging up to whether browsers blank the screen when navigating to another site.
- ehnto 4y agoAbsolutely, and I am perhaps morbidly excited to see what this next generation of technologists come up with, as they have never lived in a world without tech dominating day to day life. Has our idea of the human/technology interface shifted far enough that the idea of information technology implants will seem a sensible optimization of day to day life? Will skuemorphisms change over time? Maybe the symbol for a neural implant phone call will look like a smartphone (I think it does in Cyberpunk 2077 but I can't recall)
- NetOpWibby 4y agoThis is incredible
- warangal 4y agoVideos encode so much information, and it looks pretty cool when such projects extract higher level information to play with. And recent models like whisper and clip are amazing to help make sense of that information even for personal projects. We are also trying to do something similar[0], but still a lot of work remaining. Idea is to allow real time processing of any video using just a CPU. [0] https://www.youtube.com/watch?v=E7UPj9blnWc https://www.youtube.com/watch?v=E7UPj9blnWc
- unraveller 4y agoNice demo, any chance this tech gets ported to an android library soon? Real-time face detection via CPU would be way better than constantly uploading your video feed to aws kinesis for ML processing.
- sj8822 4y agoThis is utterly incredible. I have so many ideas after reading way too much scifi and watching Ridley Scott’s scifi series, Raised By Wolves (what if we could create a benevolent, kind and caring ai to help humans grow and navigate into this world, like Father in the series)? I want to jump in, badly. How would one go about picking up the skills needed to create stuff like this? As pragmatically, concretely and efficiently as possible without getting sidetracked in overly theoretical distractions? I’m a fullstack engineer and have an MS in CS and pretty good math chops, but I sadly only took 1 machine learning course in all of my formal education. How do I get into this (gpt-3, chatgpt are also on my mind)? Please, any books, moocs, etc
- DerSaidin 4y ago> I have so many ideas > I want to jump in, badly. How would one go about picking up the skills needed to create stuff like this? Have you tried to make any of your ideas? If not, pick one and try to make it.
- sj8822 4y agoThis is a great approach. I want to build a benevolent, kind, caring and nurturing AI that one can go to for learning, understanding and support in navigating this world and being human. I want to dump key knowledge into it that I have been fortunate enough to discern during my lifetime and knowledge from others that I regard highly. Kind of like what characterai is doing, but highly fined tuned and customized. What skills do I need to do this? What tech do I need to learn? What books, moocs, etc. or even just words to look into? Just looking to be pointed in the right direction, I’ll take care of the rest. Definitely not trying to reinvent gpt3 or anything like that, but looking to leverage these tools and customize and fine tune them highly
- jamesblonde 4y agoThis is a great example of serverless machine learning with modal.
- ReFruity 4y agoNice! Also would be cool to see the text positioned above the face and translated just like in the cyberpunk gif.
- flemhans 4y agoIt's actually incredible doing all this work and resist implementing the last few lines of code that would do that.
- jamesblonde 4y agoI developed an online course on serverless machine learning, where you can learn some of the principles of refactoring ML systems into separate feature/training/inference pipelines: https://github.com/featurestoreorg/serverless-ml-course https://github.com/featurestoreorg/serverless-ml-course Some of the students have built similar systems, for example chaining Whisper and ChatGPT or translation or sentiment analysis of transcribed text, such as here (transcribe Swedish and tell me the sentiment of the text): https://huggingface.co/spaces/Chrysoula/voice_to_text_swedish https://huggingface.co/spaces/Chrysoula/voice_to_text_swedis...
- SiempreViernes 4y agoWhy is this type of cloud based computing called serverless? To me "putting all the work on remote machines you don't manage" seems pretty cloudy, and I naively expected "serverless" to mean something like runs locally, not "uses a shit town of someone else's servers".
- jamesblonde 4y agoGreat question. I think of it as follows - if I can just write some operational Python code and "deploy" it, without needing to install/upgrade/maintain systems software (k8s/DB/feature-store/etc), then I have built a serverless system. What's cool now is lots of these services have great free tier and you can run serverless ML systems without paying anything.
- jwr 4y agoGood question. For me it's in the same category as "hoverboards" that don't actually hover.
- rospaya 4y agoA good analogy I heard is that serverless uses servers as much as wireless uses wires. It does, but it's not part of your responsibility.
- jamesblonde 4y ago
- fareesh 4y agoDoes anyone have a link to something similar for real-time captions (audio only - (.g. Google Meet, Twitter Spaces)? Or is that typically done on the device itself
- input_sh 4y agoIf I understood you correctly, there's a Chrome accessibility feature that you should check out: https://support.google.com/chrome/answer/10538231 https://support.google.com/chrome/answer/10538231 Think of it as YouTube's auto-generated captions, but on a browser level.
- fareesh 4y agoI was mainly interested in studying implementations of real time transcription
- schizo89 4y agoThis is cool. I guess it's possible to this all in single multi-task NN
- slowmotiony 4y agoIs it possible to have this tool run locally and use it for myself? I don't see any instructions on the Readme, all I see are ways to set it up for development and ways to deploy it to some 3rd party cloud solutions.
- EchoReflection 4y agoSo I've taken to saving many, many posts from HN with https://web.archive.org/save/ https://web.archive.org/save/ and this is the amusing result I got when trying to save this one: "This host has been already captured 100,091.0 times today. Please try again tomorrow. Please email us at "info@archive.org" if you would like to discuss this more." Haha.
- yDogbone 4y agoI would like to point out: As someone who is a newbie coder, I had a ton of fun learning how your code works with the help of ChatGPT. Even learning about unfamiliar topics like serverless apps, nlp transformers, or yaml vs json lol