16 ms·
Live-caption glasses let deaf people read conversations [video]
- Blackthorn 4y agoThis is the sort of thing I expected Google Glass to be able to do. But Google apparently lacks sufficient imagination, so they just unceremoniously canned it. Really hope this takes off.
- dymk 4y agoGoogle literally announced this feature 10 months ago at I/O 2022
- lern_too_spel 4y agoNot with Google Glass but with some new vaporware glasses. https://www.theverge.com/2022/5/13/23068759/google-io-2022-ar-glasses-translation-concept-challenges-vaporware https://www.theverge.com/2022/5/13/23068759/google-io-2022-a...
- zoklet-enjoyer 4y agoI was so excited for the possibilities of AR when I first heard about Google Glass. I imagined navigating foreign cities with signs auto-translated to English, turn by turn directions, translated subtitles, etc
- giraffe_lady 4y agoVery cool, especially for people who lose their hearing later in life. For other deaf people it's important to remember that written english is not a form of their native sign language¹, so this would be like (because it is) reading captions in a second language. Still potentially useful but with more limitations. Not that there's necessarily a technological way around those limitations either. ¹ Afaik this applies to all other sign languages outside english too. Signed Exact English exists and probably other-language equivalents too but I've never met a native speaker.
- spullara 4y agoFolks that use ASL use fingerspelling which is of course just written english, no? https://www.lifeprint.com/asl101/fingerspelling/fingerspelling.htm https://www.lifeprint.com/asl101/fingerspelling/fingerspelli...
- giraffe_lady 4y agoI have only basic ASL and am by no means an authority. But I think between native ASL speakers its use would be very rare, mostly just to clarify an exact spelling for something that was going to be written down in english. Native ASL speakers who are completely illiterate in english certainly exist, and I'm not sure at all if they know or use finger spelling.
- craigbaker 4y agoFingerspelling is just a path for borrowing individual words from English. It is not a part of native ASL vocabulary or grammar; that is to say, ASL does not consist of fingerspelled English sentences.
- deleted 4y ago[deleted]
- kiba 4y agoNo. Fingerspelled words are not written English, but literally spelled out English words. For example, it would be like me saying, "'h', 'e', 'l', 'l', 'o'" instead of whatever translated 'hello' in ASL.
- Tarragon 4y agoNope. https://www.signingsavvy.com/article/45/The+difference+between+ASL+and+English+signs https://www.signingsavvy.com/article/45/The+difference+betwe... Here's a fun example. ASL allows, maybe even requires, negation after the statement. An interpreter friend of mine was interpreting Wayne's World in a mixed crowd. The whole "<statement>... NOT!" joke gets laughs from the hearing audience and the Deaf audience doesn't understand why.
- eclipxe 4y agohttps://www.youtube.com/watch?v=SQd394a4qEo https://www.youtube.com/watch?v=SQd394a4qEo
- flanbiscuit 4y agoAnyone know the status of these Google AR glasses? The form factor of Google's AR glasses look much much closer to a normal pair of glasses than the glasses in the top video (which look like heavy sunglasses with a wire connecting to your phone) You posted a CNET link, here's a direct link to Google's: https://www.youtube.com/watch?v=lj0bFX9HXeE https://www.youtube.com/watch?v=lj0bFX9HXeE In the description it says: "This device has not been authorized as required by the rules of the Federal Communications Commission. This device is not, and may not be, offered for sale or lease, or sold or leased, until authorization is obtained." So I guess they might be waiting for that?
- quantumquetzal 4y agoBackground: researched this space for a graduate degree. There are a few issues that are unanswered by this video (which isn't intended to be a technical deep dive, but I don't see any related links in the video description): 1. How do these glasses handle multiple simultaneous speakers? Based on the display I saw, it shows the speakers' words sequentially, which starts to fall apart in real-world environments, especially group conversations. This is a big problem, and wider adoption is contingent on handling this elegantly. 2. These appear to be the classic "smart glasses" display style that's pervasive in consumer head-worn displays today, where content is projected at a fixed depth in front of the wearer. Because the captions aren't anchored at the same focal distance as the speaker, the wearer's eyes will swap between the captions and the speaker's faces, which is a tiring activity, and can make the wearer feel like they're not part of the conversation or being rude. 3. As mentioned by another commenter, this is a useful idea for people who lose their hearing later in life. That said, this is less (although certainly still) useful for people who have congenital hearing loss and primarily communicate via ASL. All in all, it's exciting to see growing interest in this space, as it's easily extendable to people learning a new language or navigating a foreign country. I think offloading the speech-to-text to a tethered mobile device is a good choice (though it would be nice to do low-latency wireless transmission).
- ape4 4y agoIt would be nice if it put the captions over the speaker (in a speech bubble?)
- danscarfe 4y agoYou pin the captions next to the person. We did try speech bubbles, but it didn't look great, so we went back to simple subtitles.
- quantumquetzal 4y agoThere’s a significant amount of evidence pointing to the fact that “anchored” captions (a la speech bubbles, like a comic book) is a preferred way to render captions, assuming the bubbles are at the same focal distance as the speaker. This is solvable with “true” AR, but getting that compute into the lightweight form factor of the original video is an unsolved problem, and is a ways off.
- danscarfe 4y agoThanks for the shout-out! To try and answer your questions: 1) The cocktail party problem is still a WIP. This is a very hard problem to solve. 2) These are not 'viewer' glasses they are 3DOF glasses which support moving and pinning the subtitles in 3D space 3) Whilst we targeted the Deaf and HOF to begin, we see broad applicability beyond this 4) You don't need glasses to test it, just an Android 12+ phone. Download and try it. We'd love your feedback https://link.xrai.glass/app https://link.xrai.glass/app
- ogig 4y agoHi there: Some feedback from a sing language interpreter, my wife, as I showed her this. > 4) You don't need glasses to test it, just an Android 12+ phone. This exactly pointed her. Some deaf people will use any dictation software on the phone, and look at the phone when needed. This glasses instead will cover some field of view. Note that view for deaf people is more important and used than for the rest of us. She couldn't see the improvement of using bulky glasses instead of lowering your eyes to the phone. Personally I think the endeavor is admirable and wish you best of luck. Also, as other comments say, I think this product might be more desirable for HOF and late in life hearing loss sectors than born deaf people.
- danscarfe 4y agoThe advantage of using AR glasses is you can still look at the person, see their facial expressions, the reactions etc without always having to look down at your phone. The glasses aren't very bulky or heavy. We're just providing an option for those that want it. It's a magical experience :)
- raldi 4y agoI'm impressed at how not-awful the glasses look. I do wish there had been a real-life shot showing how the text appears to the wearer, though.
- danscarfe 4y agoYou can see how it appears in our promo video: https://www.youtube.com/watch?v=iilN0368vQU https://www.youtube.com/watch?v=iilN0368vQU
- raldi 4y agoThe video gives the general idea, but it only shows a graphical simulation of what the wearer sees. I'm curious to see an actual photo or video of the text appearing on the lens of the glasses, in reality, with no special effects.
- danscarfe 4y agoIt doesn't appear on the glasses. It appears in 3D space infront of you. It's almost impossible to show via video.
- raldi 4y agoOh, interesting. It's not being projected onto the lenses, but straight into the eye or something?
- danscarfe 4y agoThe light is bounced straight into your eyes using something called a birdbath lens
- baseline-shift 4y agoI don't understand this page https://www.nreal.ai/compatibility-list/ https://www.nreal.ai/compatibility-list/ What do the glasses need to be compatible with? I don't recognize or own any of these products listed. I have an iPhone. How does it work with a device as part of the deal?
- elil17 4y agoI'm a hearing person and I've spent a summer interning in a 50/50 mixed Deaf and hearing research group. My take is that this is a huge UI improvement for AI speech to text, which a lot of Deaf people are already using to listen to conversations. It seems particularity great because it allows this technology to provide situational awareness while, for example, walking. It's important to remember, though that, for conversations where you're trying to include a Deaf person who isn't good at speaking or chooses not to speak, speech to text is a fundamentally unequal communication modality. They will be able to "receive" but they won't be able to "transmit", which makes for extremely lopsided conversations. There is no substitute for taking the time to learn a sign language or to have conversations via writing (no substitute for sign language as it requires a lot of patience from both parties).
- jackblemming 4y agoWhy can’t they also make glasses that translate sign language into English audio or text then?
- elil17 4y agoTranslating sign languages to spoken languages is really, really hard. Sign languages have a lot of features that don't really show up in spoken languages. Let's take an example: the sign for "send email" in ASL (http://www.lifeprint.com/asl101/pages-signs/e/email.htm http://www.lifeprint.com/asl101/pages-signs/e/email.htm). If I point at you at the end of the sign it could mean "I will send you an email." If I point at myself it could mean "You should send me an email" or "Did you send me an email" depending on my facial expression. If I point off into space it could mean "I'm sending an email." If I start by pointing at you and then end the sign by pointing off into space it could mean "You should send an email." So your translator AI needs not only to understand the facial expressions and movements of the signer, but also the spacial relationships of everyone in the conversation. And that is just one aspect of the difficulty - there are many other features of sign languages that are just as hard to translate. Perhaps this is the sort of thing that future AI systems could do. But it is quite complex.
- dymk 4y ago
- sigwinch28 4y agoI hope (reatively immature) solutions like this will not be used as an excuse to remove accessible infrastructure from the world (e.g. captions at the cinema, live subtitles at the theatre, text displays on public transport).
- drjasonharrison 4y agoThere is a huge population group that I am hoping will demand and make accessibility much more refined. Everything from the size of text to lighting levels to subtitles
- sigwinch28 4y agoWhich group is that? Disabled people, or their allies, or someone else?
- jhugo 4y agoMany countries have aging populations, so the proportion of people who have a personal reason to care about issues like sight degeneration is going to increase significantly over time.
- drjasonharrison 4y agobingo
- Andrew_nenakhov 4y agoMod with speech bubbles would be high on my priorities list, and a bonus would be a comic book font.
- Mizza 4y agoWould love a version for translation. I don't need the glasses, just read the translation to me and I'll wear an ear bud. And if you could make it work at a loud party, that'd be just perfect..
- tsimionescu 4y agoSpeech to text barely works in the best conditions. Combine it with bad audio, automatic translation, and text to speech, and you'll be lucky if you understand 10% of what's being said.
- omoikane 4y agoLooks like this product exists: https://support.google.com/googlepixelbuds/answer/7573100 https://support.google.com/googlepixelbuds/answer/7573100
- zoklet-enjoyer 4y agoMy Pixel phone can do this. Why are the glasses so expensive and bulky?
- NovaDudely 4y ago1. They don't have the same kind of budget as Google to get the scale down as effectively. 2. If these are the ones I have heard about before, all the speech to text is done on the device not in the cloud. This is for privacy reasons. Means it needs a bit more bulk for the gear to work.
- zoklet-enjoyer 4y agoThese glasses are connected to a phone app
- danscarfe 4y agoCompared to contemporary AR glasses ($2k+), these Nreal Airs are fairly cost effective at $379. We plan to support lots more glasses as they come out, especially ones with waveguide technology that doesn't require the shared lenses.
- stubish 4y agoWhat apps on the phone do you use for live conversation captioning? I'm thinking a tablet that does this would be ideal for in-home use by some elderly relatives. AR glasses won't be suitable for some time for the elderly, as it seems eyesight starts going before hearing.
- WastingMyTime89 4y agoReading the comments here, I think people are missing how many people are losing hearing while aging and how alienating it is. Even if it only works with one speaker at a time, it could mean a massive quality of life improvement for a large and growing share of the population. Sure it won’t solve the issues faced by the deaf community but that’s only a tiny portion of the people handicapped by difficulty hearing.
- autoexec 4y agoWhat are the privacy/security issues with this? Does this mean every conversation a person wearing these has (or that occurs within earshot) is being collected and harvested by someone? Will the AR be used to insert ads into people's conversations or plaster images of ads all over the place? Will certain words or phrases be automatically censored? This is cool tech, that could be used to help people, but it comes with lots of potential for new forms of evil that were not possible without it. Considering that I can't remember the last time I bought a product using a new technology that wasn't also designed to work against my interests, I'm immediately skeptical of any device that can't be used offline and especially one that requires being connected to cell phone apps.
- lazyeye 4y agoCompletely agree. There seems to be such a gap in the market for privacy-first hardware but very few companies are doing anything.
- Kalium 4y agoThe key question is always a painfully simple one: how much are people willing to pay as a privacy-first premium? The answers to this determine everything. Treating privacy-first as the moral and ethical default we should expect everyone to start from is a wonderful idea, rooted in compassion, kindness, and a foundational respect for human rights. It has also been an abject failure to date. We should not expect the future to be different unless we are willing to be realistic about the economics at work. Otherwise the market gap will remain in the realm of the wonderfully hypothetical forever.
- autoexec 4y ago> The key question is always a painfully simple one: how much are people willing to pay as a privacy-first premium? the fact is that no amount of money a customer can pay will ever be worth more than taking that customer's money and then also selling every scrap of data that your product can get its hands on on top of it. We need to stop accepting "But I can make more money by screwing you" as a valid excuse for how things are. It's true, but still not okay. If the economics are always going to favor exploiting people, than I guess if we don't want to be exploited we need some very powerful regulations and all of the oversight and enforcement that requires in place to change that situation. I'd prefer that to blind faith and optimism. Until then, there are a number of things we should be insisting on in products as consumers to help protect ourselves. "works offline" is a really good one in a lot of situations.
- ElijahLynn 4y agoVideo doesn't have a link to product. Here it is: https://xrai.glass/ https://xrai.glass/
- JoeAltmaier 4y agoOther applications? Travelers, translate Spanish to English etc. Reconstruct voice in noisy environments. ???
- 1-6 4y agoCan they make the glasses without tint? I can imagine users going, "I'm deaf, not blind." Magic Leap has screens that can adjust opacity at the pixel-level.
- danscarfe 4y agoThe current generation of glasses (such as these Nreal Airs) use a birdbath technology which requires a tint. The next generation of waveguide glasses won't require this but they are currently not as good visual quality, especially for reading text. They are also a lot more money. Everything is a trade off right now. The next couple of years will be transformational.
- tossaway0 4y agoI admittedly don’t have much experience with deaf people; I had one acquaintance in high school who was deaf. Hanging out with him made me very aware of how isolating it can be to only be able to participate in conversations where you are actively trying to pay attention. If these can let people hang out and participate without having to actively track each speaker in a group setting it will go a long way.
- allanrbo 4y agoBought a pair of these glasses, Nreal Air, a few months ago. I find them useful for laptop coding without straining my neck. It's awesome to see more creative use cases for them!
- ghotli 4y agoSo I dug around a little bit and figured I'd just ask. As just some guy on the internet, can I buy one of these and write a hello world to have text show up in front of my eyes of my own choosing? Does it have an API or will it?
- danscarfe 4y agoYou can indeed. It's called the Nreal sdk
- andsoitis 4y agoNext step: babel fish
- s-xyz 4y agoThe metaverse is coming!
- baseline-shift 4y agoI am pretty deaf yet somehow have wound up interviewing people for a living (go figure). I currently use googlemeet and recently switched to Otter.ai for recording/transcription. Unlike the previous transcription tools for journalists that I used in the past - Otter.ai generates the text live on the laptop screen while we are talking and even corrects itself to make sense as the speaker reveals contextual clues. It is a huge help, and I had wished there was something for real life conversations like this is for screen conversation. Good news for deaf people. You only have to watch newscasters speaking through a pasted-on permanent grin - as if every word is ee - to know that lip reading is garbage.
- deleted 4y ago[deleted]
- polygot 4y agoIt's powered by an app apparently. Could be interesting to connect it to GPT-4 (e.g., someone asks you a question and then you could just tell them the answer from GPT.) Or, if it had OCR capabilities, you could just hold a sheet of paper in front of yourself and say "what's this?" and it would explain the text to you.
- yosito 4y agoI'm not deaf, but I would love to have this for language learning.
- Animats 4y agoWas that available for Google Glass?
- captain3d 4y agoThe number one thing these glasses/software need to solve is that the words match the speech for a one to one conversation in a quiet environment. Eg doctors visit. I think they are very close. We just got the Nreal/Xrai setup a few days ago for deaf from birth wife (hearing husband) She grew up lipreading but integrated more with signing and deaf community as an adult. She has a cochlear implant but can not understand language from sound alone. And really doesn't enjoy hearing that much unless we are watching a movie etc where the sound is 100% linked to the visual. Initial reaction to the setup is. 1. Impressed, hopeful, excited 2. A bit complicated technically. More stuff to deal with. not an everyday thing. 3. Phone battery usage high. Maybe 3 - 4 hours 4. In the right situation they will be really powerful. 5. Need more control over the interface eg show/hide 'listening' icon etc. Can be distracting. Move subtitle position (maybe you can) 6. Processing delay can make you more an observer of the conversation. Response time is delayed enough to interrupt the flow a conversation. (satellite tv connection interviews) The number one barrier to using them is having everything ready for the moment they are needed. You need to plan ahead. Takes a few mins to set up. All the other high end ideas can be set aside while the core function is dialed in. We really appreciate the effort and hope to contribute.
- DrKeithDuggar 4y agoThank you for this feedback! Btw, we'll support better adjustment of the subtitle position very soon. We did just add many additional font size options as well. If you haven't already, please consider joining our Discord server to provide feedback at any time: https://discord.gg/7HjyDJ3JAz https://discord.gg/7HjyDJ3JAz
- wbm1 4y agoThis has the potential to be the killer app for mixed reality once instant / realtime translation is possible. Imagine being able to understand every language in the world - and if two users of this product meet, being able to converse without learning each other's language. It would be a technological Babelfish.
- TRiG_Ireland 4y agoAll these "aids" always have to be born by the Deaf people, not by the hearing. At least these have some chance of actually working, I suppose, unlike those ridiculous gloves one sometimes sees celebrated.