7 ms·
The writing is on the wall for handwriting recognition
- coolness 1y agoGreat post and amazing progress in this field! However, I have to wonder if some of these letters were part of the training data for Gemini, since they are well-known and someone has probably already done the painstaking work of transcribing them...
- suddenlybananas 1y agoShhhhh no one cares about data contamination anymore.
- spwa4 1y agoThen write something down yourself and upload a picture to gemini.google.com or chatgpt. Hell, combine it. Make yourself a quick math test, print it, solve with pen and ask these models to correct it. They're very good at it.
- timdiggerm 1y agoFor that to be relevant to this post, they would need to write with secretary hand.
- suddenlybananas 1y agoI don't know how to write like a 19th century mathematician, nor anyone earlier. I'm not sure OCR on Carolingian Miniscule has been solved, let alone more ancient styles like Roman cursive or, god forbid, things like cuneiform. Especially since the corpora on these styles is so small, dataset contamination /is/ a major issue!
- MrSkelter 1y agoI have a personal corpus of letters between my grandparents in WW2. My grandfather fighting in Europe and my grandmother in England. The ability of Claude and ChatGPT to transcribe them is extremely impressive. Though I haven’t worked on them in months and this uses older models. At that time neither system could properly organize pages though and chatGPT would sometimes skip a paragraph.
- vertnerd 1y agoI've also been working on half a dozen crates of old family letters. ChatGPT does well with them and is especially good at summarizing the letters. Unfortunately, all the output still has to be verified because it hallucinates words and phrases and drops lines here and there. So at this point, I still transcribe them by hand, because the verification process is actually more tiresome than just typing them up in the first place. Maybe I should just have ChatGPT verify MY transcriptions instead.
- embedding-shape 1y agoIt helps when you can see the confidence of each token, which downloadable weights usually gives you. Then whenever you (your software) detects a low confidence token, run over that section multiple times to generate alternatives, and either go with the highest confidence one, or manually review the suggestions. Easier than having to manually transcribe those parts at least.
- seidleroni 1y agoIs there any way to do this with the frontier LLM's?
- red75prime 1y agoAsk them to mark low confidence words.
- 1y ago
- dmd 1y agoPossibly, but given it can also read my handwriting- which is much, MUCH worse than Boole’s - with better accuracy than any human I’ve given it to- that’s probably not the explanation.
- lccerina 1y agoMost likely, and probably inferring the structure on texts with "similar" writing forms. Tried with my handwriting (in italian) and the performance wasn't that stellar. More annoyingly, it is still a LLM and not a "pure" OCR, so some sentences were partially rephrased with different words than the one in the text. This is crucially problematic if they would be used to transcribe historical documents
- embedding-shape 1y ago> Tried with my handwriting (in italian) and the performance wasn't that stellar. Same here, for diaries/journals written in mixed Swedish/English/Spanish and with absolutely terrible hand-writing. I'd love for the day where the writing is on the wall for handwriting recognition, which is something I bet on when I started with my journals, but seems that day has yet to come. I'm eager to get there though so I can archive all of it!
- GaggiX 1y agoAre you sure to have used the Gemini 3.0 pro model? Maybe try increasing the media resolution on the AI studio if the text is small
- butlike 1y agoSo it doesn't work is what you're saying, right?
- pbronez 1y ago"it is still a LLM and not a "pure" OCR" When does a character model become a language model? If you're looking at block text with no connections between letter forms, each character mostly stands on its own. Except capital letters are much more likely at the beginning of a word or sentence than elsewhere, so you probably get a performance boost if you incorporate that. Now we're considering two-character chunks. Cursive script connects the letterforms, and the connection changes based on both the source and target. We can definitely get a performance boost from looking at those. Hmm you know these two-letter groupings aren't random. "ng" is much more likely if we just saw an "i". Maybe we need to take that into account. Hmm actually whole words are related to each other! I can make a pretty good guess at what word that four-letter-wide smudge is if I can figure out the word before and after... and now it's an LLM.
- iamflimflam1 1y agoIf I went back in time to the 90s when I was doing my PhD I would absolutely blow my mind with how well handwriting OCR works now.
- th0ma5 1y agoMy question for OCR automation is always which digits within the numbers being read are allowed to be incorrect?
- pjmlp 1y agoMaybe for English, for the other human languages I use, it is still kind of hit and miss, just like speaking recognition, even with English it suffices to have an accent that is off the standard TV one.
- NitpickLawyer 1y agoee lay vhen!
- macleginn 1y agoAs always, this depends on the amount of training data available. Japanese is another success story: https://digitalorientalist.com/2020/02/18/cursive-japanese-and-ocr-using-kuronet/ https://digitalorientalist.com/2020/02/18/cursive-japanese-a...
- pjmlp 1y agoInteresting, thanks for sharing.
- williamjsdavis 1y agoAgree here. I've had successes with 18th century Dutch, but again quite a few failures and mistakes
- cubefox 1y agoIs "it" Gemini 3 Pro?
- __alexs 1y agoCall me when it can do Russian Cursive.
- decimalenough 1y agoSeems to do an OK job: https://g.co/gemini/share/e173d18d1d80 https://g.co/gemini/share/e173d18d1d80 This is a random image from Twitter with no transcript or English translation provided, so it's not going to be in the training data.
- shatsky 1y agoNo, transcription has nothing to do with written text, it guessed few words here and there but not even general topic. That's doctors note about patient visit, beginning with "Прием: состояние удовл., t*, но кашель / patient visit: condition is OK, t(temperature normal?) but coughing". But unreadable doctors handwriting is a meme...
- GaggiX 1y agoThat's Gemini 2.5 Flash btw The result from Gemini 3 Pro using the default media resolution (the medium one): "(Заголовок / Header): Арсеньев (Фамилия / Surname - likely "Arsenyev") Состояние удовл- t N, кожные покровы чистые, [л/у не увел.] В зеве умерен. [умеренная] гипер. [гиперемия] В легких дыха- ние жесткое, хрипов нет. Тоны серд- [ца] [ритм]ичные. Живот мяг- кий, б/б [безболезненный]. мочеисп. [мочеиспускание] своб. [свободное] Ds: ОРЗ [или ОРВИ]" and with the translation: "Arsenyev Condition satisfactory. Temp normal, skin coverings [skin] are clean, lymph nodes not enlarged. In the throat [pharynx], moderate hyperemia [redness]. In the lungs, breathing is rigid [hard], no rales [crackles/wheezing]. Heart tones are rhythmic. Abdomen is soft, painless. Urination is free [unhindered]. Diagnosis: ARD (Acute Respiratory Disease)."
- __alexs 1y agoOk fine I'm impressed
- 1y ago
- tigerlily 1y agoSurely the true prize is to be able to ditch computers altogether and just write with pencil on paper.
- DarkNova6 1y ago> Here’s Transkribus’s best guess at George’s letter to Maryann, above: Transkribus got a new model architecture around the corner and the results look impressive. Not only for trivial cases like text, but also for table structures and layouting. Best of all, you can train it on your own corpus of text to support obscure languages and handwriting systems. Really looking forward to it.
- macleginn 1y agoI became convinced of this after the release of KuroNet: https://arxiv.org/pdf/1910.09433 https://arxiv.org/pdf/1910.09433 (High-quality OCR of Japanese manuscripts, which look almost impossible to read.)
- nikanj 1y agoThe writing is on the wall for handwriting. Zoomers use speech recognition or touchscreen keyboards, millennials use keyboards. Boomers use pens
- deleted 1y ago[deleted]
- lccerina 1y agoI call out the Lindy effect. Handwriting survived printed characters, typewriters, and the last 50-70 years of computers and keyboards, it will survive this too.
- djmips 1y agoI love how you fit right into the current meme that Gen-X never gets mentioned.
- shaftway 1y agoWhile I agree (and strongly identify with, and like this position), one could amend the original to be "Gen-X uses pens and print, Boomers use pens and cursive"
- sph 1y agoSilly comment. Handwriting is proven to be correlated with much better memory retention, which ultimately means much greater degree of association with existing memories and the creation of novel ideas. "The comparison between handwriting and typing reveals important differences in their neural and cognitive impacts. Handwriting activates a broader network of brain regions involved in motor, sensory, and cognitive processing, contributing to deeper learning, enhanced memory retention, and more effective engagement with written material. Typing, while more efficient and automated, engages fewer neural circuits, resulting in more passive cognitive engagement. These findings suggest that despite the advantages of typing in terms of speed and convenience, handwriting remains an important tool for learning and memory retention, particularly in educational contexts." https://pmc.ncbi.nlm.nih.gov/articles/PMC11943480/ https://pmc.ncbi.nlm.nih.gov/articles/PMC11943480/ You are literally handicapping yourself by not thinking with pen and paper, or keeping paper notes. The future is handwriting with painless digitization for searchability, until we invent a better input device for text that leverages our motor-memory facilities in the brain.
- lifestyleguru 1y agoIt feels unbelievable that in Europe literacy rate could be 10% of lower. Then I look at documents even as young as 150 years... fraktur, blackletter, elaborate handwritting. I guess I'm illiterate now. Hopefully next generations will feel the same about legal contracts, law in general, and Java code bases. They're incomprehensible not because of fonts but because of unfathomable complexity.
- sph 1y agoWhich Europe and which century do you live in where literacy rate is below 10%?
- lifestyleguru 1y agoSpeaking about the past centuries.
- tpm 1y agoYou can learn fraktur or blackletter in a day and cyrillic in a few days, if you already know the latin alphabet.
- lifestyleguru 1y ago> learn fraktur or blackletter in a day and cyrillic in a few days Not a chance, sorry.
- tpm 1y agoWhy? The former are just different typefaces (I learned to read them by myself when I was 10 while looking at our old books) and the latter I sort of picked up while travelling through Serbia and Bulgaria (I don't speak the languages).
- sph 1y agoAny self-hosted open source solution? I would like to digitize my paper notebooks but I do not want to use anything proprietary or that uses external services. What is the state of the art on the FOSS side? Ideally something that I can train with my own handwriting. I had a look at Tesseract, wondering if there’s anything better out there.
- vintermann 1y agoRegular handwriting there are many. Historical handwriting, Gemini 3 is the only one which gave a decent result on a 19th century minutes from a town court in Northern Norway (Danish gothic handwriting with bleed through). I'm not 100% sure it's correct, but that's because it's so dang hard to read it to verify it. At least I see it gets many names, dates and locations right. I've been waiting a long time for this.
- sph 1y ago> Regular handwriting there are many. Please share. I am out of the loop and my searches have not pointed me to the state of the art, which has seen major steps forward in the past 3 or 4 years but most of it seems to be closed or attached to larger AI products. Is it even still called OCR?
- fragmede 1y agoTotally not what you asked, but making an OCR model is a learning exercise for AI research students. Using the Kaggle-hosted dataset https://www.kaggle.com/datasets/landlord/handwriting-recognition/data https://www.kaggle.com/datasets/landlord/handwriting-recogni... and a tutorial, eg https://pyimagesearch.com/2020/08/17/ocr-with-keras-tensorflow-and-deep-learning/ https://pyimagesearch.com/2020/08/17/ocr-with-keras-tensorfl... you can follow along and train your own OCR model!
- driscoll42 1y agoThe best open source OCR model for handwriting in my experience is surya-v2 or nougat, really depends on the docs which is better, each got about 90% accuracy (cosine similarity) in my tests. I have not tried Deepseek-OCR, but mean to at some point.
- girvo 1y ago> "transmitted": In the second line of the body, the word "transmitted" is crossed out in the original text Am I nuts or is this wrong, not “perfect”? It doesn’t look crossed out at all to me in the image, just some bleeding? Still very impressive, of course
- williamscales 1y agoI agree, I noticed the same thing. To my eye it appears smudged.
- benterix 1y ago> If AI can diminish some of the monotony of research, perhaps we can spend more time thinking, writing, playing piano, and taking walks — with other people. Whenever any progress is made, this is the logical conclusion. And yet, those who decide about how your time is being used, have an opposing view.
- ulbu 1y agomaybe akin to how faster conputers bred programs that are slower than before.
- volemo 1y agoBetter “thinking” computers will breed worse thinking people, huh?
- palmotea 1y ago> Better “thinking” computers will breed worse thinking people, huh? I actually think that will be the case. We're designing society for the technology, not the technology for the people in it. The human brain wasn't built to fit whatever gap is left by AI, regardless of how many words the technologists spew to claim otherwise. For instance: AI already is undermining education by enabling mental laziness students (why learn the material when ChatGPT can do your homework for you). It seems the current argument is that AI will replace entry-level roles but leave space for experienced and skilled people (but block the path to get there). Some of the things LLMs do a mediocre but often acceptable job at are the things one needs to do to build and hone higher-level skills.
- spiritplumber 1y agoDr. Walter Gibbs: Won't that be grand? Computers and the programs will start thinking and the people will stop.
- themaninthedark 1y agoLook at GPS and then "self-driving" cars. With GPS we have seen people confidently drive past road closed signs and around barriers off bridges. With self-driving technology, we have seen them defeat safe guards so they can sit in the back while the car accelerates up to 70 in a subdivision.
- zkmon 1y agoIt's painful to see that beautiful hand-writing of the past is now pretty much extinct. For me, handwriting of a person speaks a lot about them, not just their mind, but physical state as well.
- SoftTalker 1y agoPenmanship used to be a topic of instruction in school. It hasn't been for quite a long time. Even in the 1970s when we still had to write in cursive, we didn't spend time learning to make it elegant, with a lot of flourishes. We just learned a very plain standard form of writing. By high school everyone had diverged a bit with their own personal style but nothing like the writing of the 19th century. I once visited a high school where they had a wall of signatures from every graduating senior going back to the 1920s or so. The "personality" evident in the signatures showed a steady decline, from very stylish in the oldest ones to mostly just poorly printed names in the 2020s.
- ferguess_k 1y agoDon't worry, handwriting itself has diminished throughout the decades since the introduction of computers an especially smart phones. Ah, maybe I'll pick up Qin seal when I retire, if I retire.
- tokai 1y agoAnyone knows how the models do on Russian cursive?
- euroderf 1y agoUltimate Test
- zelphirkalt 1y agoQuite certain my doctor can still produce writing, that the models don't stand a chance to be able to recognize.
- stronglikedan 1y agoI'm just excited that I may finally be able to decipher my meeting notes from yesterday!
- RationPhantoms 1y agoAnecdata inbound but my PCP, thankfully, used Nuance's speech-to-text platform remarkably well for adding his own commentary on things. It was a refreshing thing to see and I hope my clinicians use it.
- nottorp 1y agoI thought handwriting recognition is on the wall because no one knows how to write cursive any more
- taeric 1y agoI confess this largely surprises me for reasons that I think should not surprise me. I would expect current AI is largely best at guessing at what some writing was based on expectations of other things it has managed to "read." As such, I would think it is largely not going to be much better at hand writing than any other tool. Yet, it occurs to me that that "guess and check" is exactly what I'm doing when trying to read my 6yo's writing. Often I will do a pass to detect the main sounds, but then I start thinking of what was current on his thoughts and see if I can make a match. Not surprisingly, often I do.
- shevy-java 1y agoI can't recognize my handwriting anymore. :(
- RationPhantoms 1y agoWouldn't it be easier to train a vLLM on the handwriting style of the historical person in question? An agent graphologist if you will. Surely there is a lot of pattern matching in the way things are written. Then again, getting this result from a heavily-generalized SOTA model is pretty incredible too.
- supersrdjan 1y agoNow, onto the next frontier: handwriting recognition for shorthand. Let's start from Orthic :)
- canopi 1y agoIf that's not a proof of the 10/90 rule in machine learning. The last 10% of accuracy are harder than the first 90 (and that goes recursively). We almost solved OCR 20 years ago. Then we spent 20 years on the last percentage. We see the same in self-driving cars.