5 ms·

>~97% accuracy over hour-long conversations. I'm sure it's been an absolute godsend for law enforcement 97% accuracy means roughly three or four errors per min

>~97% accuracy over hour-long conversations. I'm sure it's been an absolute godsend for law enforcement 97% accuracy means roughly three or four errors per minute of speech. That seems potentially extremely problematic for something like law enforcement use where decisions with significant impact on people's day and/or life might be made on the basis of "evidence".

golem14 4y ago
One would think that the few crucial bits of information gleaned are listened to manually, and the machine translation is not the only thing the judge or a jury sees.
- thfuran 4y ago
  You have absolutely ruined someone's day way before they're sitting in front of a jury.
  - formerly_proven 4y ago
    Stuff like that is a very good tell that someone has zero experience with law enforcement.
hadlock 4y ago
Microsoft announced their voice transcription technology a couple years ago and were also touting ~97-98% accuracy which was actually better than human transcription error rates. The errors are usually in part people garbling their own speech, or they move their head while talking and the microphone misses a syllable. Anything in that error bar would probably fall under "reasonable doubt"
- kyriakos 4y ago
  If its anything like Microsoft teams transcription I doubt the 97%+ accuracy.
gs17 4y ago
Yeah, I tried to use automated transcription for a research project and we had to do it all manually because the few errors (I would say it did pretty well given our recording quality) were often dropping words like "not", which changed the whole meaning of a sentence! It was a useful assistance during transcription, but I really hope they would verify it was correct before arresting anyone based on it.
anigbrowl 4y ago
No it isn't. That just means 2-3% of your content needs to be double-checked by a person at the audio level, saving huge amounts of time - equally true of human transcription, in which individual words are often [UNINTELLIGEBLE]. Would you want to review this fully before going into court, absolutely - because you'd want to play the recording to a jury for emotional impact. Can you rely on it when you want to quickly read through hours of conversation and make decisions about whether to invest further resources (which might just mean another hour of listening back to the original audio)? Also absolutely. Bear in mind that a lot of these errors have little to no semantic impact, being on the same level as typos or misspellings in a written communication. Bear in mind too that if law enforcement (honest or not) is so interested in you that they're willing to record your conversations, your day is already ruined, you just don't know it yet. The change here is one of scale rather than quality.
- dotancohen 4y ago
  Sometimes even human will disagree about what was said in a recording - I had this happen recently. I heard a specific sentence, the other person heard the exact opposite. I cannot say who was right, even after listening to the recording several times on headphones and speakers I'm as certain of my interpretation as was the other party.
- wging 4y ago
  Doesn't it mean 100% of your content needs to be double-checked? You can't easily identify which 2-3% of your content has errors. I'm aware that errors are more likely when the model is less confident of its predictions, but that shouldn't be enough. (edit for clarification: errors are not always something like "[UNINTELLIGIBLE]", where the system knows it doesn't know; they can also be misrecognitions that the system believes in with high confidence.)
  - u8 4y ago
    I had to do a lot of manual transcription in Journalism school. Using a tool like Descript saved HOURS of my life. Generally it was 80% accurate, but going over an two-hour-long recording again at 3x speed while reading over the transcript, fixing errors from memory or pausing took a five hour job down to 30-40 minutes. Either way, somebody is going to have to listen to the recording. This just removes a layer of grunt work.
selfmodruntime 4y ago
I've worked with similar technology in the law enforcement space and the software is never used to make decisions. You can make out critical timestamps in conversations and a law enforcement officer will always manually confirm the software's assessments.
- JohnFen 4y ago
  Given that law enforcement has made similar claims about technology use in the past that turned out to be false, I have no faith in this claim.
  - selfmodruntime 4y ago
    In all honesty, this is the correct mindset to have. I have limited expertise in this topic, and you should be aware that other law enforcement agencies probably do not handle this the same way.
CTDOCodebases 4y ago
I imagine a certain percentage of a given population is on a voice call at any one time. 1. Set up a computer with voice recognition software that flags certain patterns. 2. Connect computer to voice call communication network. 3. Configure computer to switch between calls every x number of seconds. Think of it like a system to generate leads for law enforcement that can be integrated with other systems to produce the best quality leads.
- NaturalPhallacy 4y ago
  This is called "a fishing expedition" and is wildly unconstitutional in the US. >The right of the people to be secure in their persons, houses, papers, and effects, against unreasonable searches and seizures, shall not be violated, and no Warrants shall issue, but upon probable cause, supported by Oath or affirmation, and particularly describing the place to be searched, and the persons or things to be seized.
  - jjoonathan 4y ago
    Yes, it is wildly unconstitutional, but in practice don't the courts endorse the asinine "it's not a search unless we find something" argument from the NSA? Power always just finds a way to rationalize what it wants to do.
    - kurisufag 4y ago
      see: Operation PRISM
  - CTDOCodebases 4y ago
    Are you sure about that? [0] Besides I wasn't talking about the USA when I said this. I was remembering a conversation I once had with a person who worked as a technician in a telephone exchange. [0] - https://en.wikipedia.org/wiki/Jewel_v._NSA https://en.wikipedia.org/wiki/Jewel_v._NSA
Thorentis 4y ago
Not really. Imagine that they do simple keyword matching on the text. Anything that's missed (part of the 97%) the criminals get away with. Anything that matches in the 3% is then checked by a human (by listening to the audio at that time stamp). So you only need to manually check the 3%, and even then only if something you're interested in is found.

golem14 4y ago

One would think that the few crucial bits of information gleaned are listened to manually, and the machine translation is not the only thing the judge or a jury sees.

thfuran 4y ago
You have absolutely ruined someone's day way before they're sitting in front of a jury.
- formerly_proven 4y ago
  Stuff like that is a very good tell that someone has zero experience with law enforcement.

hadlock 4y ago

Microsoft announced their voice transcription technology a couple years ago and were also touting ~97-98% accuracy which was actually better than human transcription error rates. The errors are usually in part people garbling their own speech, or they move their head while talking and the microphone misses a syllable. Anything in that error bar would probably fall under "reasonable doubt"

kyriakos 4y ago
If its anything like Microsoft teams transcription I doubt the 97%+ accuracy.

gs17 4y ago

Yeah, I tried to use automated transcription for a research project and we had to do it all manually because the few errors (I would say it did pretty well given our recording quality) were often dropping words like "not", which changed the whole meaning of a sentence! It was a useful assistance during transcription, but I really hope they would verify it was correct before arresting anyone based on it.

anigbrowl 4y ago

No it isn't. That just means 2-3% of your content needs to be double-checked by a person at the audio level, saving huge amounts of time - equally true of human transcription, in which individual words are often [UNINTELLIGEBLE]. Would you want to review this fully before going into court, absolutely - because you'd want to play the recording to a jury for emotional impact. Can you rely on it when you want to quickly read through hours of conversation and make decisions about whether to invest further resources (which might just mean another hour of listening back to the original audio)? Also absolutely. Bear in mind that a lot of these errors have little to no semantic impact, being on the same level as typos or misspellings in a written communication. Bear in mind too that if law enforcement (honest or not) is so interested in you that they're willing to record your conversations, your day is already ruined, you just don't know it yet. The change here is one of scale rather than quality.

dotancohen 4y ago
Sometimes even human will disagree about what was said in a recording - I had this happen recently. I heard a specific sentence, the other person heard the exact opposite. I cannot say who was right, even after listening to the recording several times on headphones and speakers I'm as certain of my interpretation as was the other party.
wging 4y ago
Doesn't it mean 100% of your content needs to be double-checked? You can't easily identify which 2-3% of your content has errors. I'm aware that errors are more likely when the model is less confident of its predictions, but that shouldn't be enough. (edit for clarification: errors are not always something like "[UNINTELLIGIBLE]", where the system knows it doesn't know; they can also be misrecognitions that the system believes in with high confidence.)
- u8 4y ago
  I had to do a lot of manual transcription in Journalism school. Using a tool like Descript saved HOURS of my life. Generally it was 80% accurate, but going over an two-hour-long recording again at 3x speed while reading over the transcript, fixing errors from memory or pausing took a five hour job down to 30-40 minutes. Either way, somebody is going to have to listen to the recording. This just removes a layer of grunt work.

selfmodruntime 4y ago

I've worked with similar technology in the law enforcement space and the software is never used to make decisions. You can make out critical timestamps in conversations and a law enforcement officer will always manually confirm the software's assessments.

JohnFen 4y ago
Given that law enforcement has made similar claims about technology use in the past that turned out to be false, I have no faith in this claim.
- selfmodruntime 4y ago
  In all honesty, this is the correct mindset to have. I have limited expertise in this topic, and you should be aware that other law enforcement agencies probably do not handle this the same way.

CTDOCodebases 4y ago

I imagine a certain percentage of a given population is on a voice call at any one time. 1. Set up a computer with voice recognition software that flags certain patterns. 2. Connect computer to voice call communication network. 3. Configure computer to switch between calls every x number of seconds. Think of it like a system to generate leads for law enforcement that can be integrated with other systems to produce the best quality leads.

NaturalPhallacy 4y ago
This is called "a fishing expedition" and is wildly unconstitutional in the US. >The right of the people to be secure in their persons, houses, papers, and effects, against unreasonable searches and seizures, shall not be violated, and no Warrants shall issue, but upon probable cause, supported by Oath or affirmation, and particularly describing the place to be searched, and the persons or things to be seized.
- jjoonathan 4y ago
  Yes, it is wildly unconstitutional, but in practice don't the courts endorse the asinine "it's not a search unless we find something" argument from the NSA? Power always just finds a way to rationalize what it wants to do.
  - kurisufag 4y ago
    see: Operation PRISM
- CTDOCodebases 4y ago
  Are you sure about that? [0] Besides I wasn't talking about the USA when I said this. I was remembering a conversation I once had with a person who worked as a technician in a telephone exchange. [0] - https://en.wikipedia.org/wiki/Jewel_v._NSA https://en.wikipedia.org/wiki/Jewel_v._NSA

Thorentis 4y ago

Not really. Imagine that they do simple keyword matching on the text. Anything that's missed (part of the 97%) the criminals get away with. Anything that matches in the 3% is then checked by a human (by listening to the audio at that time stamp). So you only need to manually check the 3%, and even then only if something you're interested in is found.