5 ms·

End-To-end Audiovisual Speech Recognition

54 points by ghosthamlet 9y ago

rahimnathwani 9y ago
Wow: "In presence of high levels of noise, the end-to-end audiovisual model significantly outperforms both audio-only models."
diminish 9y ago
I am curious what the success rate would be without any audio - by just reading lips and face.
- IshKebab 9y ago
  I'm going to guess "very bad" given that even experts can't do this very well. I don't think there is enough information. Edit: It does say in the paper actually. The "classification rate" (I assume % of words correctly identified - they only recognise single works from a dictionary of 500) is 82 for visual only and 98 for audio, and audio-visual (without noise). Without noise the audio is good enough to be near perfect (I assume 100 is perfect) so the video doesn't really help. It helps when there is noise (which matches real life experience - people lip read in noisy bars).
  - mwcampbell 9y ago
    > people lip read in noisy bars Maybe that explains why I, as a visually impaired person who can't lip read, have trouble carrying on conversations in noisy bars. Good to know!
nmca 9y ago
Y'all didn't cite some extremely relevant work: https://arxiv.org/abs/1709.00572 https://arxiv.org/abs/1709.00572 (disclaimer - they're from my lab)
- radarsat1 9y ago
  What is HNs impression of the etiquette of "demanding" citations? I agree with nmca that it's the author's onus to research prior work as widely as possible, but if something is missed, presumably both by the authors and the reviewers, and therefore clearly was not actually a reference for the work in question, is the author really at fault? Is it "ok" to demand a citation like this?
  - singularity2001 9y ago
    >> Is it "ok" to demand a citation like this? Sure, why not (especially with disclaimer given)
  - syllogism 9y ago
    Closed-form peer review is basically a (very tight) spam filter in ML at the moment. Everyone wants the papers to appear quickly, so reviewers can't meaningfully require changes --- papers are in or out. Overall this is net better. Requiring changes leads to bike-shedding, and really long publication processes. Most review is community review, of exactly this sort of form. So we really don't want to add a lot of politeness constraints around requesting citations. The literature is moving so fast that authors can get away with assuming readers won't know about relevant work. If we let that thrive, we'll reward bad actors who "forget" prior work and claim novelty.
    - amygdyl 9y ago
      What would it take to make authors interested in a publication in magazine physical format, of extended abstracts and generalized discussion of how findings fit in the literature and from viewpoints of practice and research worlds? Do authors have the capacity in time for writing commissioned articles for example to place their work in parenthetical context? This is a hypothetical posed by a colleague of mine recently, arising from the obstacle of references requiring journal access, I was asked for my opinion whether I thought it would be commercially viable for instance to license reproduction of the references cited in a new paper, the I availablity the convenience of company executives with distant academic backgrounds, or very small r&d shops unable to justify the extensive subscriptions for journals to follow a year's developments in real time. The very first advertising sale I made was to Iterated Systems, a vendor of software fractal image compression who continued as a niche video archive business. At the time, January in 1994 I seem to remember now rather vividly, C-cubed Corporation was in its quiet period prior to IPO. C-cubed designed and sold ASICS for JPEG comprehension. I believe the NeXT Station was equipped with one. I made a sale because I was able to make a sufficiently valliant effort at differentiating the respective theories of operation,, the managing director at their EMEA hq went quiet. I closed a few more shortly after, just on the improbability to my prospect's view, of making a actually intelligible argument for the relevance of their products to our readers, that I have specialised in roughly the similar approach ever since. I say this because there does exist in the C suite, a intense interest in the details of the most arcane components of any strategic computing asset, and the problem now is simply a paucity of suitable publications for placing the advertisements. I see more need to understand performance in context of costs of a different magnitude variation in many cases all set against a backdrop without any publisher pursuing the potential. I seek out the professional association and industry body journals that are closest to a audience. But while I get margins to be happy with, initially, without broader sales activity this kind of title never sees the investment necessary to develop the coverage. Because of two particularly generous tax deals, the first which boils down to the HMRC adding forty pence for your every sixty, come three years of holding the stock in a start-up British media business. The second one is particularly important to my constituency showing interest in the similar to that which I just posited, which is the fact that all AIM quoted shares are inherited without application of inheritance tax before the beneficiaries are recipient from probate. The circumstances: retired holders of PhDs in various fields whom are interested in applying expertise more lightly than full employment, teaching difficult for age and or health, but active minds missing the stimulus of professional interests. I immediately thought of how it might be a business capable of keeping in touch the intellect of a employee on gardening leave, who may be willing to accept necessarily pro rata editing and writing fees far below their salary level, but given the opportunity to discuss interests in a comparable community and obtain a professional exposure through publication of their writing or review. I did not only think about but talked to a lawyer who manages leave for a big broker, who was reservedly enthusiastic that I wouldn't be open to disclosure violation specific liability given my outfit runs the same level of communications audit and compliance as a broker so I can say that our systems were not used to communicate anything proprietary. But I am in London, where the employee is favoured by the law. (At least in tribunals I will not make similar closing if you are defending IP theft in the Chancery Division of the Royal Courts) Maybe I'm pining for Byte magazine again and have it bad. But with the business in selling hardware to run ML work, given consistency of the aim and product of such a publication, I'd have to try to not sell out the as pages. (American publishers have to devote two thirds of the pages to qualify for bulk mail rates. This shouldn't matter for small circulation, but it was always the first and last thing I heard when I was hoping to launch titles in the USA.

54 points by ghosthamlet 9y ago

Wow: "In presence of high levels of noise, the end-to-end audiovisual model significantly outperforms both audio-only models."

diminish 9y ago

I am curious what the success rate would be without any audio - by just reading lips and face.

IshKebab 9y ago
I'm going to guess "very bad" given that even experts can't do this very well. I don't think there is enough information. Edit: It does say in the paper actually. The "classification rate" (I assume % of words correctly identified - they only recognise single works from a dictionary of 500) is 82 for visual only and 98 for audio, and audio-visual (without noise). Without noise the audio is good enough to be near perfect (I assume 100 is perfect) so the video doesn't really help. It helps when there is noise (which matches real life experience - people lip read in noisy bars).
- mwcampbell 9y ago
  > people lip read in noisy bars Maybe that explains why I, as a visually impaired person who can't lip read, have trouble carrying on conversations in noisy bars. Good to know!

nmca 9y ago

Y'all didn't cite some extremely relevant work: https://arxiv.org/abs/1709.00572 https://arxiv.org/abs/1709.00572 (disclaimer - they're from my lab)

radarsat1 9y ago
What is HNs impression of the etiquette of "demanding" citations? I agree with nmca that it's the author's onus to research prior work as widely as possible, but if something is missed, presumably both by the authors and the reviewers, and therefore clearly was not actually a reference for the work in question, is the author really at fault? Is it "ok" to demand a citation like this?
- singularity2001 9y ago
  >> Is it "ok" to demand a citation like this? Sure, why not (especially with disclaimer given)
- syllogism 9y ago
  Closed-form peer review is basically a (very tight) spam filter in ML at the moment. Everyone wants the papers to appear quickly, so reviewers can't meaningfully require changes --- papers are in or out. Overall this is net better. Requiring changes leads to bike-shedding, and really long publication processes. Most review is community review, of exactly this sort of form. So we really don't want to add a lot of politeness constraints around requesting citations. The literature is moving so fast that authors can get away with assuming readers won't know about relevant work. If we let that thrive, we'll reward bad actors who "forget" prior work and claim novelty.
  - amygdyl 9y ago
    What would it take to make authors interested in a publication in magazine physical format, of extended abstracts and generalized discussion of how findings fit in the literature and from viewpoints of practice and research worlds? Do authors have the capacity in time for writing commissioned articles for example to place their work in parenthetical context? This is a hypothetical posed by a colleague of mine recently, arising from the obstacle of references requiring journal access, I was asked for my opinion whether I thought it would be commercially viable for instance to license reproduction of the references cited in a new paper, the I availablity the convenience of company executives with distant academic backgrounds, or very small r&d shops unable to justify the extensive subscriptions for journals to follow a year's developments in real time. The very first advertising sale I made was to Iterated Systems, a vendor of software fractal image compression who continued as a niche video archive business. At the time, January in 1994 I seem to remember now rather vividly, C-cubed Corporation was in its quiet period prior to IPO. C-cubed designed and sold ASICS for JPEG comprehension. I believe the NeXT Station was equipped with one. I made a sale because I was able to make a sufficiently valliant effort at differentiating the respective theories of operation,, the managing director at their EMEA hq went quiet. I closed a few more shortly after, just on the improbability to my prospect's view, of making a actually intelligible argument for the relevance of their products to our readers, that I have specialised in roughly the similar approach ever since. I say this because there does exist in the C suite, a intense interest in the details of the most arcane components of any strategic computing asset, and the problem now is simply a paucity of suitable publications for placing the advertisements. I see more need to understand performance in context of costs of a different magnitude variation in many cases all set against a backdrop without any publisher pursuing the potential. I seek out the professional association and industry body journals that are closest to a audience. But while I get margins to be happy with, initially, without broader sales activity this kind of title never sees the investment necessary to develop the coverage. Because of two particularly generous tax deals, the first which boils down to the HMRC adding forty pence for your every sixty, come three years of holding the stock in a start-up British media business. The second one is particularly important to my constituency showing interest in the similar to that which I just posited, which is the fact that all AIM quoted shares are inherited without application of inheritance tax before the beneficiaries are recipient from probate. The circumstances: retired holders of PhDs in various fields whom are interested in applying expertise more lightly than full employment, teaching difficult for age and or health, but active minds missing the stimulus of professional interests. I immediately thought of how it might be a business capable of keeping in touch the intellect of a employee on gardening leave, who may be willing to accept necessarily pro rata editing and writing fees far below their salary level, but given the opportunity to discuss interests in a comparable community and obtain a professional exposure through publication of their writing or review. I did not only think about but talked to a lawyer who manages leave for a big broker, who was reservedly enthusiastic that I wouldn't be open to disclosure violation specific liability given my outfit runs the same level of communications audit and compliance as a broker so I can say that our systems were not used to communicate anything proprietary. But I am in London, where the employee is favoured by the law. (At least in tribunals I will not make similar closing if you are defending IP theft in the Chancery Division of the Royal Courts) Maybe I'm pining for Byte magazine again and have it bad. But with the business in selling hardware to run ML work, given consistency of the aim and product of such a publication, I'd have to try to not sell out the as pages. (American publishers have to devote two thirds of the pages to qualify for bulk mail rates. This shouldn't matter for small circulation, but it was always the first and last thing I heard when I was hoping to launch titles in the USA.