11 ms·
An OCR cliche: Into his/her anus (2009)
- robin_reala 8y agoAfter stopping laughing I went back and checked the Standard Ebooks corpus to see if any of this mistake had slipped through; luckily it seems that in the intervening 9 years someone at Gutenberg and / or archive.org has corrected this particular issue in the source transcriptions.
- mcguire 8y agoGutenberg is designed to avoid this sort of thing, although some slip through: originally, they didn't use OCR and now they use the distributed proofreader thing. Archive.org is kind of a mess, though.
- robin_reala 8y agoYeah, I usually submit about 10-15 corrections to Gutenberg per book I proof; generally they’re in good shape. The bigger problem with Gutenberg is that older books omit all accents, which is a huge problem for who series of books. I’ve been trying to produce Maurice Leblanc’s series of Arsène Lupin stories for Standard Ebooks and Gutenberg generally spells the titular protagonist’s name wrong.
- visarga 8y agoFunny The arms/anus confusion should be fixed with a language model on top of the letter predicting network.
- duskwuff 8y agoWithout a very long-range model I don't think that would help. "in his/her anus" and "in his/her arms" can both be correct in the right circumstances; it takes quite a bit of surrounding context to tell which one is more likely. (While doing some research in Google Books I even found a couple that looked like OCR errors until I read beyond the search snippet.)
- abecedarius 8y agoHow does OCR software integrate letter and language models? Do they first make a best guess at the letters and then try to correct it with the language model? https://en.wikipedia.org/wiki/Optical_character_recognition#Techniques https://en.wikipedia.org/wiki/Optical_character_recognition#... gives me that impression, but I'm not sure. Brains are said to have a lot of feedback from higher levels of sensory processing to lower. Maybe you don't need as good a language model if its evidence is integrated more tightly with the rest.
- amelius 8y agoBut that would completely ruin the article, if it were OCR'ed.
- mort96 8y agoOnly without human oversight. Human proofreading might not be economically feasible, but maybe detecting likely OCR errors and making humans decide whether it's an actual error or not would be feasible.
- qwerty456127 8y agoOCR-ed texts should really be proof-read before being published. Pirates usually do this, that's funny Google doesn't. Also Markov chains can help by highlighting unusual word combinations, I doubt anal children occur often in correct texts.
- lozenge 8y agoGoogle scanned 25 million books, and would show the user the original scan anyway. The text was only used for the search index.
- dmm 8y agoGoogle also provides epubs, which show the OCRed text.
- robin_reala 8y agoProofing transcriptions is difficult. You can go a long way with computer models, but eventually you’ll get a feeling about a word not quite fitting and need to check back to the source text. I guess you could run a high quality / more iterations OCR at this point, but more likely you’re relying on human proofing, and it doesn’t scale if you’re relying on humans. There’s a reason the Standard Ebooks project I contribute to has <250 books compared to Gutenberg’s 50k+: quality takes effort.
- jakobegger 8y agoMaybe the problem are said Markov chains? It's possible the phrase "in her anus" occurs more frequently on the internet than "in her arms".
- dmichulke 8y agoIn other words: Just because 50% of the internet are porn, you shouldn't use a 50% porn corpus to train your Markov chains.
- qwerty456127 8y ago
- DoreenMichele 8y agoGiven the examples in the piece involving children, I wonder if there is any danger of this resulting in a problem where a site gets accused of child pornography or gets blocked because of it sounding so wildly inappropriate or something.
- Tharkun 8y agoI don't know, children do have anuses, and they are known for their curiosity. I'm sure many a parent has had to dig lego bricks out of various orifices.
- esrauch 8y agoChildren have a few things that depicting or discussing would result in being blocked from most schools, even when stopping well short of pornographic depictions.
- deleted 8y ago[deleted]
- elbrownos 8y ago"You can lead a horse to water but you can't stop it from sticking Lego up its bum." Probably my favourite quote from the Inbetweeners.
- billfruit 8y agoWhat is the article saying about 'pertistent' vs 'persistent' ? Is that a word, what is its meaning?
- Kliment 8y agoLikely because of the long s in historical texts - it's often misrecognized as an f or t
- thaumasiotes 8y agoNo, the book says "pertistent", and the transcriber mistakenly wrote "persistent". Neither of those is a recognition error. Rather, the error is that the transcriber did recognize what was meant, and ignored what was actually written.
- codetrotter 8y agoIndeed. https://books.google.com/books?id=7JfT9yq0zAQC&pg=PA78&lpg=PA78&dq=uncle+tom%27s+cabin+%22pertistent%22 https://books.google.com/books?id=7JfT9yq0zAQC&pg=PA78&lpg=P... > If another copy from the same edition has the error corrected, such cues may help to identify early and late printings and contribute to a more comprehensive account of the book’s printing history. In other words, when transcribing books you want to preserve misspellings that occur in the source text. It’s actually quite interesting because that means that automatic spellchecking of OCRed text while helping to improve the quality of the transcript could also introduce unwanted corrections. But doing like the OP did and comparing their transcripts with those of Google Books was clever.
- thaumasiotes 8y agoTangent: scribal errors are often classified as to whether they are committed by scribes who do understand the language they're copying or scribes who don't. (Some errors can be committed by either kind of scribe, but will still tend to lean one way or the other.) Copying "persistent" where the text has "pertistent" is a good example of a kind of error that only a scribe who understands the text will make. (Though this particular case might not even be considered a scribal error.)
- billfruit 8y agoI am reminded of the anecdote of one of the first mass printings of the Bible in London in the 1600s having a grave misprint: "Though shall commit adultery". Proof reading is as ever important.
- TipVFL 8y agoThis reminds of an eBook of Neuromancer that I read, it was occasionally missing the letter f. For the most part I just added it back mentally without really thinking about it, but then sometimes I hit a passage like this: "He turned, pulled his jacket on, and licked the cobra to full extension." That one took a moment.
- jobigoud 8y agoProbably the original text was using ligatures for fi and fl and they got lost in conversion. https://en.wikipedia.org/wiki/Typographic_ligature#Stylistic_ligatures https://en.wikipedia.org/wiki/Typographic_ligature#Stylistic...
- superkuh 8y agoYup. I have to manually detect and correct for all the possible ligatures in all possible unicode in my text to speech pre-processor scripts. I hate them.
- ahazred8ta 8y ago"this gives us efficient space-time trade-offs" :-(
- dotancohen 8y agoThose are HTML entities. Most modern programming languages come with tools to decode this, e.g. in python: text = urllib.parse.unquote(text)
- jwilk 8y agourllib.parse.unquote() is unrelated to HTML. It undoes URL-encoding: https://docs.python.org/3/library/urllib.parse.html#urllib.parse.unquote https://docs.python.org/3/library/urllib.parse.html#urllib.p... In Python ≥ 3.4, you can use html.unescape() to decode HTML entities: https://docs.python.org/3/library/html.html#html.unescape https://docs.python.org/3/library/html.html#html.unescape
- tilt_error 8y agoI have a work injury, acquired from 30+ years of coding :) As a youngster, I read a lot and at great speed, but after I started coding my read speed dropped dramatically. Attention to detail while writing or reading code seems to have re-wired my brain for accuracy instead of speed ;) What jumps at me in the article is not the misinterpretation of arms, which results in funny but somewhat "working" language, but rather "[..] hitting lier feet against stones" where the 'h' is interpreted as 'li'. That brought me to a full stop. Also makes me think of "kerning" vs. "keming" :)
- rzzzt 8y agoMaven documentation porn.xml
- taneq 8y ago> Also makes me think of "kerning" vs. "keming" :) The game Path of Exile once had a line in the patch notes which simply said "Fixed keming." Made my day.
- deleted 8y ago[deleted]
- chrisweekly 8y agoA designer at my former workplace had a full-zip hoodie. To one side of the zipper: "Ker", to the other: "ning".
- projectramo 8y agoAfter you guys fix this with Markov chains or whatever, I look forward to reading: the proctologist was thorough but found no sign of blockage in her arms.
- sthgrau 8y agoThat would be a clbuttic mistake, indeed.
- reaperducer 8y agoThe Wesleyan-Methodist Magazine – Page 433 “carried this child in his anus to Derry” I guess so the child could smell the Derry air.
- Infernal 8y agoToo fine a pun to let pass without remark.
- EForEndeavour 8y agoI want to believe that the secret purpose of Wesleyan-Methodist Magazine was to set up the reading machines of a future civilization for this pun.
- softawre 8y agoA pun so funny that you didn't get downvoted or a user/mod commenting about how this isn't reddit. GG
- gumoro 8y agoReminds me of "Don't kick a man when he's clown"; Google finds 2 PDFs with this, due to bad OCR: https://www.google.com/search?q="Don't+kick+a+man+when+he's+clown" https://www.google.com/search?q="Don't+kick+a+man+when+he's+... Credit: https://twitter.com/ObeyComputer/status/1050131788830560258 https://twitter.com/ObeyComputer/status/1050131788830560258
- tyingq 8y agoApparently, "feces" where "faces" should be is a thing as well. "feces sticking out of large pipes, looking hungrily at the camera" https://archive.org/stream/The-Colonel-Who-Would-Not-Repent/The%20Colonel%20Who%20Would%20Not%20Repent%20-%20The%20Bangladesh%20War%20and%20Its%20Unquiet%20Legacy%20-%20Salil%20Tripathi_djvu.txt https://archive.org/stream/The-Colonel-Who-Would-Not-Repent/...
- userbinator 8y agoI like how if you read it out of context, the first part of that sentence seems perfectly fine in something like a text about sewage; and then the second part catches you by surprise.
- dfboyd 8y agoThe Kindle version of "A Game Of Thrones" (first book in the series) has "Dome" everywhere instead of "Dorne" (the name of the kingdom in the south). Apparently it was OCR'ed from the printed book.
- kzrdude 8y agoHow is that acceptable past a month of its release, is nobody correcting it?
- deleted 8y ago[deleted]
- iiiggglll 8y agoWelcome to the 21st century, where quality, accuracy, and precision are sacrificed at the altar of "scale".
- userbinator 8y agoThis is one of those cases where applying some kind of additional dictionary algorithms to "autocorrect" OCR (which may have been what actually happened) won't help, since the wrong reading is an actual dictionary word while the correct one is a name.
- lafar6502 8y agoSlap in the face of computerkind
- TazeTSchnitzel 8y agoReminds me of how when xkcd looked at which days of the month were most common, the 1st, 10th, 11th, 21st and 31st were more or less common than they should have been due to OCR error: https://drhagen.com/blog/the-missing-11th-of-the-month/ https://drhagen.com/blog/the-missing-11th-of-the-month/
- HocusLocus 8y agoAt the family printshop my mother experienced some typing glitches, she would sometimes type "simulate" instead of "stimulate" and "muck" instead of "mark". This led to two disasters, which required us to stop the presses (the pressman was a great proofreader!), In a political brochure: "...will introduce bills to simulate progress..." In a funeral booklet: "...he left his muck upon us all." She asked me to go into computer's master dictionary and patch it to disable the words 'simulate' and 'muck' so it would bring these mistakes to her attention.
- pxtail 8y agoI'm pretty sure that for many political brochures these words could be used interchangeably
- deleted 8y ago[deleted]
- mrob 8y agoAnother example: https://books.google.com/ngrams/graph?content=fuck&year_start=1600&year_end=2000&corpus=15&smoothing=3&direct_url=t1%3B%2Cfuck%3B%2Cc0 https://books.google.com/ngrams/graph?content=fuck&year_star... If you don't know what's going on it looks like the word "fuck" was more common in the 17th century than today. But actually it's the word "suck" written with a long s ("ſuck"), which you can see is easily OCRed incorrectly.
- robbrown451 8y agoYou'd think that the OCR process would somehow call attention to words that have a high probability of being wrong and especially of being wrong in a problematic way. You don't want to require humans to read and sign off on everything, but with something like that, it shouldn't be that hard to have something that is very quick for a human to see the scanned image and compare it to the transcription, simply on the basis of the word "anus" being in there.
- lathiat 8y agoI was reading “Creative Selection” by Ken Kocienda last week. Goes behind the scenes of him designing the iPhone keyboard early in its development (good read) In any case he mentioned there is a hate word dictionary specifically so that the autocorrect never suggests such words even if they seem to be a close match. You basically have to type those words perfectly. In another related bug Xerox document centres which weren’t even technically doing OCR were changing numbers from 1 thing to another in scanned IMAGES due to high level compression substituting numbers - much more dangerous! https://www.theregister.co.uk/2013/08/06/xerox_copier_flaw_means_dodgy_numbers_and_dangerous_designs/ https://www.theregister.co.uk/2013/08/06/xerox_copier_flaw_m...
- drcongo 8y agoImagine how Clint Eastwood feels. https://www.google.com/search?tbm=bks&q=cunt+eastwood https://www.google.com/search?tbm=bks&q=cunt+eastwood
- yesenadam 8y agoThis online version of Miles Davis' Autobiography features a character called "dark Terry" i.e. Clark Terry. http://yanko.lib.ru/books/bio/miles.htm http://yanko.lib.ru/books/bio/miles.htm