3 ms·

Expecting intelligence and accuracy to "emerge" from a statistical process is absurb. In others words, LLMs are only clearly useful if the results don't really

Expecting intelligence and accuracy to "emerge" from a statistical process is absurb. In others words, LLMs are only clearly useful if the results don't really matter or they can and will be externally verified. LLMs negate a fundamental argument for computing --- instead of accurate results at low cost, we now have inaccurate results at high cost. There is undoubtedly some utility to be had here but it is not at all clear or obvious that this will be widely transformative.

pachorizons 2y ago
Isn't that what is promised though? What is the benefit to automated transcription if each and every single transcription must be manually audited? Where is the cost or labor saving?
- 39896880 2y ago
  It is much easier to correct a transcription than to generate it wholesale. As well, the task of audio transcription correction has long since been commoditized because of the deployment of speech recognition on every smartphone. It’s not quite a solved problem but it’s close.
  - jqpabc123 2y ago
    It’s not quite a solved problem but it’s close. As long as the results don't really matter and no one is auditing, it appears more "solved" than it actually is.
Ukv 2y ago
I don't think it's necessarily absurd to expect accuracy from statistical methods - in many domains (including voice transcription) they blow the accuracy of traditional non-statistical approaches out of the water, and in some areas even surpass human-level accuracy. Main thing is to measure the accuracy of the approach (regardless of whether it's traditional, statistical, or human) to determine if it's fit for purpose. In this case it sounds like the transcription shouldn't be solely relied on for high-risk decisions in its current state, but could be useful for something like searching through the reference audio if it were available. That the issue tends to be from "pauses, background sounds or music playing" also makes me suspect a lot of the cases could be relatively low hanging fruit - check the noise gate and normalization on the microphones, or potentially have the model output a quality score for each word so that low-confidence background noise can be displayed to the end user as smaller fainter text for instance, instead of part of the conversation.

pachorizons 2y ago

Isn't that what is promised though? What is the benefit to automated transcription if each and every single transcription must be manually audited? Where is the cost or labor saving?

39896880 2y ago
It is much easier to correct a transcription than to generate it wholesale. As well, the task of audio transcription correction has long since been commoditized because of the deployment of speech recognition on every smartphone. It’s not quite a solved problem but it’s close.
- jqpabc123 2y ago
  It’s not quite a solved problem but it’s close. As long as the results don't really matter and no one is auditing, it appears more "solved" than it actually is.

Ukv 2y ago

I don't think it's necessarily absurd to expect accuracy from statistical methods - in many domains (including voice transcription) they blow the accuracy of traditional non-statistical approaches out of the water, and in some areas even surpass human-level accuracy. Main thing is to measure the accuracy of the approach (regardless of whether it's traditional, statistical, or human) to determine if it's fit for purpose. In this case it sounds like the transcription shouldn't be solely relied on for high-risk decisions in its current state, but could be useful for something like searching through the reference audio if it were available. That the issue tends to be from "pauses, background sounds or music playing" also makes me suspect a lot of the cases could be relatively low hanging fruit - check the noise gate and normalization on the microphones, or potentially have the model output a quality score for each word so that low-confidence background noise can be displayed to the end user as smaller fainter text for instance, instead of part of the conversation.