6 ms·
I'm Deaf and read lips. Thanks for this extremely intriguing idea to use emoji to depict a talking person visually. You know, some animated films get it right
by _nalply 9y ago
I'm Deaf and read lips.
Thanks for this extremely intriguing idea to use emoji to depict a talking person visually.
You know, some animated films get it right. Sometimes I clearly see utterances like «Thank you!», «Okay», but more often not. Especially older animations don't care and just let the character move the mouth in very simplistic ways: «Babbabbabaa Baababba ba baaa.»
Similarly for that emoji. It is moving the mouth in quite arbitrary ways. It looks like: «Sobbabbabee be <grin> seebee <grin> <frown> babbaa».
To solve this problem we need about twelve emoji for utterances: «ah», «ay», «ee», «oh», «oo», «b», «d», «f», «l», «n», «r» and «s». The «r» emoji must be animated (so we see the trilling tongue). The other sounds are either invisible like «k» or cover several different sounds like «b» also covering «m» and «p».
«Okay» would be rendered by: «oh», «oo», «oo» (standing for «k» wich is invisible) «ay» and «ee».
Edit: Add «th». This is an extremely simple sound to lipread. I remember my delight when I learnt English in my youth and in an movie suddenly realised that an actor said «Thank you».
- twiss 9y agoHalfway through the article, he shows a picture with mouths for a-z, sh and th: https://cdn-images-1.medium.com/max/1600/0*gwbWfYm-iQIPEXXP%2E https://cdn-images-1.medium.com/max/1600/0*gwbWfYm-iQIPEXXP%... There are others if you search images for "lip sync animation". It would be interesting to see if the result is understandable if you simply replace the emojis with those images.
- _nalply 9y agoLipreading is different. For example the «sh» sound is made by slightly pursing the lips but not all speakers do this. In German it gives a very clear visual picture.
- falsedan 9y agoI would only purse my lips if I were performing a shush. In words like ‘hash’ I’d just close my mouth slightly & leave lips parted and wide from the ‘ha’ sound. I agree that the mouth shapes are garbage in this demo. A shame because that is the most interesting & important part of animating speech.
- maury91 9y agoThis demo is done with my nearly 0 knowledge in lips reading, the effect I was searching was to make it more realistic, but with the limited amount of usable emojis I cannot have it very realistic, just better than random. At the end of the article I propose some gotchas to improve it: use better images, use a better map. I'm happy that this article has triggered some people curiosity and can maybe evolve in something more advanced
- _nalply 9y agoThat's why I am bad at lipreading in English. People have mouth shapes all over the place. I'm sure I could learn lipreading a friend I meet often well, however this doesn't help lipreading a stranger. When traveling in the US I always have a notebook ready.
- nurettin 9y agoThat image does not map to letters, it maps to the phonetic alphabet. English has different phonetics for words when compared to their written counterparts. For example "money" needs to be converted to "mahnee" before being converted to emoji counterpart. Otherwise it is gibberish.
- yesenadam 9y ago'mah-' is US English, not English :P It's 'mah' in one country. p.s. Seems every day on here I'm reminding people from the US that their country is not the world, their version of English is spoken and spelt in just one country, etc etc etc. Maybe I should give up doing that. The most classic maybe was a linguist (yes, amazing) who on her blog was gloating that her variety of US English was particularly rich, in having almost every vowel sound in English. I looked at her list.. it didn't include the 'o' sound in 'pot'..
- viridian 9y agoEither you don't understand that he means the pheonetic alphabet, or you say money in a way that is foreign to almost any person in the UK as well as the US. Do you say mooney or mohney? If not I think your phoneme almost certainly must be mah.
- namelost 9y agoBritish English has at least two pronunciations of money that I know of, /mʌni/ and /mʊni/
- viridian 9y agoThe former is the most common way for Americans to pronounce it, and matches the mouth shape of ah, and the latter, while not common in the US, also matches the ah mouth shape. I really don't know how yesenadam is pronouncing it such that his mouth doesn't take the ah shape.
- 9y ago
- k__ 9y agoI once read lipreading doesn't work very reliable, even for very skilled lipreaders. Maybe that's why most animation studios don't bother.
- _nalply 9y agoSome do bother, because even some hearing people can tell the mouth movements are off. This adds to the authenticity of a movie.
- k__ 9y agoI see, didn't think of that :)
- kuschku 9y ago> Maybe that's why most animation studios don't bother. Dubs. 90% of your viewers will see it in a foreign language, dubbed, where all your lipsyncing effort is useless anyway. In fact, lipsyncing ut to english makes it look worse in the dub than having random nonsense movements.
- maury91 9y agoThat's false, I'm from Italy and our dubbing is made with big effort, and lips syncing is an important part, so important that sometimes they don't use literal translation, so they can choose words that have a similar pronunciation to the English ones and the lips look in sync
- icebraining 9y agoSame here, but that's a reverse process (syncing the script to the animation, rather than the animation to the script).
- kuschku 9y agoI’m from Germany, and there’s a lot of effort spent on lip syncing, but it’s not even close to what you get with the native language. As child (due to deafness until I was almost 5) I relied a lot on lip reading, and it was very noticeable if something was dubbed.
- evanb 9y agoYou could actually propose a whole set of Unicode characters that are making realistic-for-lip-reading movements.
- orbital-decay 9y agoThat would be a rabbit hole though, if you consider that Unicode isn't English or even Indo-European centric. *(just another rabbit hole actually, Unicode got plenty of them already)
- evanb 9y agoNo reason not to have one for each IPA character.
- scrollaway 9y agoThe entire IPA is available in unicode. You could treat those as ligatures with a special combinator character, which would work a lot like flags, skintones etc work. eg: [special begin character] + NEUTRAL FACE (U+1F610) + [IPA characters] + (special end character) = talking face! hurray! "text engine developers hate him"
- frik 9y agoJesus man, have some respect for other humans, and some leniency where it's warranted.
- chrisseaton 9y ago> Unicode isn't English or even Indo-European centric It definitely is! The Latin alphabet gets a special place at the start of the range, in order to be compatible with ASCII, an English-language character set.
- throwanem 9y agoThat doesn't say Unicode is English-centric. That just says it sought backwards compatibility with a then-dominant character set that was. Seeing as otherwise Unicode would likely have been regarded as a pointless exercise in ocean-boiling in many more quarters than it initially was, and would have faced a commensurately steeper slope to reach wide adoption, I'm not sure what better alternative you see the initial designers having forgone to choose as they did.
- ajanuary 9y agoI always thought an interesting project would be to use facial recognition to detect key mouth shapes to make better gifs of people talking.
- falsedan 9y agoOnly a few sounds use lip shapes. Most sounds are based on tongue positioning. Compare ‘m’ & ‘n’ to… not making any sound with your lips together or slightly parted. The only sounds I would be very confident at recovering sounds from images for ‘th’, ‘w’, ‘o’, & indistinguishable voiced/unvoiced pairs like ‘b’/‘p’ or ‘v’/‘f’. Basically, the sounds a ventriloquist has to cheat.
- thrownhwn 9y agoInteresting, thanks for the insight!
- ggggtez 9y agoI don't read lips but it's painfully obviously that emoji animation is not right. I seriously looked at it and thought: wow that looks terrible.