7 ms·

Real-Time Recognition of Handwritten Chinese Characters

115 points by aeontech 9y ago

DelightOne 9y ago
Is there an API for this or is this for Scribble/Apple Apps only?
- paxswill 9y ago
  I think this is only exposed in iOS with the Chinese keyboards (and Scribble on watchOS), so not programmatically.
  - DelightOne 9y ago
    Ah I see thanks!
  - peterburkimsher 9y ago
    Yes, and also on Mac OS. (Show Trackpad Handwriting)
soared 9y ago
I was going to ask if anyone could pinpoint when apple started publishing, but I found their first post - July 2017. They even have an email inviting students, researchers, and devs to ask questions. https://machinelearning.apple.com/2017/07/01/Welcome.html https://machinelearning.apple.com/2017/07/01/Welcome.html
Isamu 9y ago
Hey, I understand this! Because I read An Intuitive Explanation of CNN's yesterday! (https://news.ycombinator.com/item?id=15328516 https://news.ycombinator.com/item?id=15328516) Thanks HN!
Scaevolus 9y ago
The input is a 48x48 image. For handwriting recognition, many systems use stroke information (online) to augment the purely image based (offline) data. This significantly reduces error rates. For the smaller ICDAR 2013 Chinese Handwriting Recognition Competition (~8,000 characters), CR(1) accuracy went from 94.77% (offline) to 97.39% (online). Human recognition rates were 95.19% (offline) and 96.13% (online).
- raverbashing 9y ago
  Yes, but the article mentions there's a lot of stroke order inconsistency (errors), so they don't rely on that
  - taneq 9y ago
    Surely you could still use it for hints? (ie. "stroke X followed by stroke Y has a Z% higher chance of being this character rather than that one", much like some phone keyboards use the prior/next words entered to correct words so they make sense in context).
    - raverbashing 9y ago
      Problem is that when these hints push you in the wrong direction One of the examples there was a Chinese character being drawn as something like a 2. The character looks nothing like a 2
- olalonde 9y ago
  Somewhat unrelated but it's funny how we talk about human recognition rates being below 100%. Obviously, a human somewhere had to decide what all the correct answers were. I understand that it has a specific meaning in the context of machine learning though.
rootbear 9y ago
Over twenty years ago, I was in a conversation with a developer who was working on Japanese handwriting recognition for the Go PenPoint OS. He told me that stroke order, which is supposed to be consistent in Japanese writing, was a big help. It's interesting that this article says that there was a need to "achieve stroke-order independence". I'm wondering what the issues are here and why stroke order was useful in the 90s but not considered so now. I'm also not sure if the developer I spoke to was concerned with Chinese input as well, or just Japanese, and how that might affect the methods used.
- fredliu 9y ago
  I don't know much about Japanese, but in written Chinese, the stroke order is less likely to be consistent. One of the challenges for early leaners of writing Chinese characters is to "get the stroke order right", and many, even sufficiently educated adults could struggle getting all orders right. Also, writing Chinese characters in cursive often requires different stroke orders for the same character due to aesthetic reasons, and the boundary between print and cursive style in hand writings could be very blurry. Therefore, I could see stroke-order becomes a very noisy, inconsistent and less reliable input.
- smallnamespace 9y ago
  Standard stroke order for any given character is occasionally different between Chinese and Japanese [1]. Characters also have different variants, e.g. the radical on top of 'grass' 草 has variants with 3, 4, or even 6 strokes [1]. There are two 4-stroke variants that look different and have different stroke order. Plus cursive handwriting will often elide and merge strokes to make things flow better. Full stroke order independence is probably unrealistic (some parts of a character will always be written last), but allowing for some local inversions is probably necessary. [1] https://en.wikipedia.org/wiki/Stroke_order#Stroke_order_per_polity https://en.wikipedia.org/wiki/Stroke_order#Stroke_order_per_... [2] https://en.wiktionary.org/wiki/%E8%89%B9 https://en.wiktionary.org/wiki/%E8%89%B9
- tobltobs 9y ago
  With handwriting recognition you differ between online and offline recognition. With online you have the stroke order, which helps a lot for character segmentation. Character segmentation is the unsolved problem for handwriting recognition, at least for Latin scripts. With offline you only have the image of a word. If you depend on online information you can't recognize text from images taken with a camera.
ww520 9y ago
What would be really cool is to have an augmented VR setup where the user is writing in midair and the VR shows light trace of it, while the OCR recognizing the writing.
FabHK 9y ago
slightly off-topic, but the images in the article illustrate it beautifully: I think many people don't realise (I didn't) what an amazing skill it is to be able to read handwriting (cursive). I took it for granted to be able to read hand-written German, English, Spanish, and totally forgot that it must have taken me some time as a kid to acquire that skill (beyond reading and writing generally). However, when I studied Chinese for a bit, the (printed) characters are tricky to start with, but reading a fluent writer's handwriting ("unconstrained", as they call it in the article) is nigh impossible. I think stroke order is hugely important, in that it forms the conventions for fast abbreviated writing. For example, the "mouth" radical, a square, "口", when written quickly basically looks like a lower-case "b", because the left lower stroke comes first, then top and right in one stroke, then the bottom stroke. More complicated characters basically become a squiggle, and it's really magical, if you think about it, that you can just read it. I mean, look at figures 5, 6, 7 in that article. For me, a nice illustration of Moravec's paradox of AI [1]: what we think is difficult (chess, differentiation, ...) is easy, and what we think is easy (reading handwriting, putting an egg in the fridge, walking (!) and opening a door [2]) is hard... [1] https://en.wikipedia.org/wiki/Moravec%27s_paradox https://en.wikipedia.org/wiki/Moravec%27s_paradox [2] https://www.youtube.com/watch?v=g0TaYhjpOfo https://www.youtube.com/watch?v=g0TaYhjpOfo ("A Compilation of Robots Falling Down at the [2015] DARPA Robotics Challenge")
- imron 9y ago
  > but reading a fluent writer's handwriting ("unconstrained", as they call it in the article) is nigh impossible. It is when you are starting out, but it's not so bad once you get used to it. The best way to learn to read Chinese handwriting is to learn to write Chinese handwriting, and I don't just mean practising writing by hand, but to work through a textbook aimed at teaching cursive Chinese. This book is a good introduction in English [0], and if you are comfortable with Chinese only, this book is excellent and also far more comprehensive [1]. Once you've gone through those books, you'll be able to read most handwritten stuff without too much struggle. 0: https://www.amazon.com/Chinese-Cursive-Script-Introduction-Publications/dp/0887100333 https://www.amazon.com/Chinese-Cursive-Script-Introduction-P... 1: https://www.amazon.cn/图书/dp/B00C907XM2 https://www.amazon.cn/图书/dp/B00C907XM2
Steko 9y ago
I applaud 'Handwriting Recognition Team' on the excellent work they're doing.
peterburkimsher 9y ago
I'm very interested in this field! But I didn't stop at 30,000 characters - I'm making a dataset of 75,000 characters from 1500 different fonts. Does anyone want me to publish it? I use Apple's Chinese handwriting keyboard almost every day, living in Taiwan. Unfortunately it only shows 4 possible choices, and often the character I need is not one of those. So I use my own Pingtype keyboard. https://pingtype.github.io https://pingtype.github.io Enter a character into the bottom text field, click Search, and see the character decomposition above, and the characters & words containing that character below. Then when you're ready, click Insert to add it to the text field at the top for translation. I have no idea how to do marketing for this tool, and commenting here is the only way I've had ~1300 views in the last 6 months since I published it.
- lathiat 9y ago
  Sounds like you should apply for a Job at Apple, or sell your "company" :-)
ocrimgproc 9y ago
Has anyone done any work on English handwriting recognition? I'm very interested in this field.
- rabboRubble 9y ago
  Chinese language student, native English speaker. I haven't found any handwriting recognition commercially available for the English language that compares to what is available for Chinese Traditional / Simplified. The best English handwriting recognition software I've found is the old Palm Pilot Graffiti writing system. It's not really developed since the late 90's / 00s. Back in the day, it was blazing fast (and pretty accurate after learning the method) compared to shitty small mechanical keyboards of the era. After diving into Chinese I realized how deficient the English HR landscape was and how badly under-researched the area is. I think the standard QWERTY keyboard is just too efficient for the Euro zone languages to make the research investment worthwhile. My 2 cents / guess. There are Graffiti like apps available on IOS. What I've found is that with english keyboard auto-correct, my typing is so much faster without Graffiti using the standard Apple IOS English keyboard. Kind of a disappointment... Another conjecture, I suspect much of the morbidity in English HR research has something to do with Graffiti patent lock up between what ever remains of Palm, Xerox, and some heretofore unheard of company called Communication Intelligence Corporation.
  - ocrimgproc 9y ago
    I meant offline, not online recognition.
  - tonfa 9y ago
    On android, https://play.google.com/store/apps/details?id=com.google.android.apps.handwriting.ime&hl=en https://play.google.com/store/apps/details?id=com.google.and... supports a bunch of languages (and emoji).
    - rabboRubble 9y ago
      I hated the Android phone I had 5 years ago and haven't looked back, so apparently I'm not up to speed on developments in the Android marketplace. I stand corrected on everything stated above! I'll have to borrow a friend's phone to play with this and see how it compares to the old Graffiti.
wibr 9y ago
Unfortunately they don't provide any details about the model or the training. Would be nice to have an example as a baseline for real-time performance on iOS devices.

115 points by aeontech 9y ago

DelightOne 9y ago

Is there an API for this or is this for Scribble/Apple Apps only?

paxswill 9y ago
I think this is only exposed in iOS with the Chinese keyboards (and Scribble on watchOS), so not programmatically.
- DelightOne 9y ago
  Ah I see thanks!
- peterburkimsher 9y ago
  Yes, and also on Mac OS. (Show Trackpad Handwriting)

soared 9y ago

I was going to ask if anyone could pinpoint when apple started publishing, but I found their first post - July 2017. They even have an email inviting students, researchers, and devs to ask questions. https://machinelearning.apple.com/2017/07/01/Welcome.html https://machinelearning.apple.com/2017/07/01/Welcome.html

Isamu 9y ago

Hey, I understand this! Because I read An Intuitive Explanation of CNN's yesterday! (https://news.ycombinator.com/item?id=15328516 https://news.ycombinator.com/item?id=15328516) Thanks HN!

Scaevolus 9y ago

The input is a 48x48 image. For handwriting recognition, many systems use stroke information (online) to augment the purely image based (offline) data. This significantly reduces error rates. For the smaller ICDAR 2013 Chinese Handwriting Recognition Competition (~8,000 characters), CR(1) accuracy went from 94.77% (offline) to 97.39% (online). Human recognition rates were 95.19% (offline) and 96.13% (online).

raverbashing 9y ago
Yes, but the article mentions there's a lot of stroke order inconsistency (errors), so they don't rely on that
- taneq 9y ago
  Surely you could still use it for hints? (ie. "stroke X followed by stroke Y has a Z% higher chance of being this character rather than that one", much like some phone keyboards use the prior/next words entered to correct words so they make sense in context).
  - raverbashing 9y ago
    Problem is that when these hints push you in the wrong direction One of the examples there was a Chinese character being drawn as something like a 2. The character looks nothing like a 2
olalonde 9y ago
Somewhat unrelated but it's funny how we talk about human recognition rates being below 100%. Obviously, a human somewhere had to decide what all the correct answers were. I understand that it has a specific meaning in the context of machine learning though.

rootbear 9y ago

Over twenty years ago, I was in a conversation with a developer who was working on Japanese handwriting recognition for the Go PenPoint OS. He told me that stroke order, which is supposed to be consistent in Japanese writing, was a big help. It's interesting that this article says that there was a need to "achieve stroke-order independence". I'm wondering what the issues are here and why stroke order was useful in the 90s but not considered so now. I'm also not sure if the developer I spoke to was concerned with Chinese input as well, or just Japanese, and how that might affect the methods used.

fredliu 9y ago
I don't know much about Japanese, but in written Chinese, the stroke order is less likely to be consistent. One of the challenges for early leaners of writing Chinese characters is to "get the stroke order right", and many, even sufficiently educated adults could struggle getting all orders right. Also, writing Chinese characters in cursive often requires different stroke orders for the same character due to aesthetic reasons, and the boundary between print and cursive style in hand writings could be very blurry. Therefore, I could see stroke-order becomes a very noisy, inconsistent and less reliable input.
smallnamespace 9y ago
Standard stroke order for any given character is occasionally different between Chinese and Japanese [1]. Characters also have different variants, e.g. the radical on top of 'grass' 草 has variants with 3, 4, or even 6 strokes [1]. There are two 4-stroke variants that look different and have different stroke order. Plus cursive handwriting will often elide and merge strokes to make things flow better. Full stroke order independence is probably unrealistic (some parts of a character will always be written last), but allowing for some local inversions is probably necessary. [1] https://en.wikipedia.org/wiki/Stroke_order#Stroke_order_per_polity https://en.wikipedia.org/wiki/Stroke_order#Stroke_order_per_... [2] https://en.wiktionary.org/wiki/%E8%89%B9 https://en.wiktionary.org/wiki/%E8%89%B9
tobltobs 9y ago
With handwriting recognition you differ between online and offline recognition. With online you have the stroke order, which helps a lot for character segmentation. Character segmentation is the unsolved problem for handwriting recognition, at least for Latin scripts. With offline you only have the image of a word. If you depend on online information you can't recognize text from images taken with a camera.

ww520 9y ago

What would be really cool is to have an augmented VR setup where the user is writing in midair and the VR shows light trace of it, while the OCR recognizing the writing.

FabHK 9y ago

slightly off-topic, but the images in the article illustrate it beautifully: I think many people don't realise (I didn't) what an amazing skill it is to be able to read handwriting (cursive). I took it for granted to be able to read hand-written German, English, Spanish, and totally forgot that it must have taken me some time as a kid to acquire that skill (beyond reading and writing generally). However, when I studied Chinese for a bit, the (printed) characters are tricky to start with, but reading a fluent writer's handwriting ("unconstrained", as they call it in the article) is nigh impossible. I think stroke order is hugely important, in that it forms the conventions for fast abbreviated writing. For example, the "mouth" radical, a square, "口", when written quickly basically looks like a lower-case "b", because the left lower stroke comes first, then top and right in one stroke, then the bottom stroke. More complicated characters basically become a squiggle, and it's really magical, if you think about it, that you can just read it. I mean, look at figures 5, 6, 7 in that article. For me, a nice illustration of Moravec's paradox of AI [1]: what we think is difficult (chess, differentiation, ...) is easy, and what we think is easy (reading handwriting, putting an egg in the fridge, walking (!) and opening a door [2]) is hard... [1] https://en.wikipedia.org/wiki/Moravec%27s_paradox https://en.wikipedia.org/wiki/Moravec%27s_paradox [2] https://www.youtube.com/watch?v=g0TaYhjpOfo https://www.youtube.com/watch?v=g0TaYhjpOfo ("A Compilation of Robots Falling Down at the [2015] DARPA Robotics Challenge")

imron 9y ago
> but reading a fluent writer's handwriting ("unconstrained", as they call it in the article) is nigh impossible. It is when you are starting out, but it's not so bad once you get used to it. The best way to learn to read Chinese handwriting is to learn to write Chinese handwriting, and I don't just mean practising writing by hand, but to work through a textbook aimed at teaching cursive Chinese. This book is a good introduction in English [0], and if you are comfortable with Chinese only, this book is excellent and also far more comprehensive [1]. Once you've gone through those books, you'll be able to read most handwritten stuff without too much struggle. 0: https://www.amazon.com/Chinese-Cursive-Script-Introduction-Publications/dp/0887100333 https://www.amazon.com/Chinese-Cursive-Script-Introduction-P... 1: https://www.amazon.cn/图书/dp/B00C907XM2 https://www.amazon.cn/图书/dp/B00C907XM2

Steko 9y ago

I applaud 'Handwriting Recognition Team' on the excellent work they're doing.

peterburkimsher 9y ago

I'm very interested in this field! But I didn't stop at 30,000 characters - I'm making a dataset of 75,000 characters from 1500 different fonts. Does anyone want me to publish it? I use Apple's Chinese handwriting keyboard almost every day, living in Taiwan. Unfortunately it only shows 4 possible choices, and often the character I need is not one of those. So I use my own Pingtype keyboard. https://pingtype.github.io https://pingtype.github.io Enter a character into the bottom text field, click Search, and see the character decomposition above, and the characters & words containing that character below. Then when you're ready, click Insert to add it to the text field at the top for translation. I have no idea how to do marketing for this tool, and commenting here is the only way I've had ~1300 views in the last 6 months since I published it.

lathiat 9y ago
Sounds like you should apply for a Job at Apple, or sell your "company" :-)

ocrimgproc 9y ago

Has anyone done any work on English handwriting recognition? I'm very interested in this field.

rabboRubble 9y ago
Chinese language student, native English speaker. I haven't found any handwriting recognition commercially available for the English language that compares to what is available for Chinese Traditional / Simplified. The best English handwriting recognition software I've found is the old Palm Pilot Graffiti writing system. It's not really developed since the late 90's / 00s. Back in the day, it was blazing fast (and pretty accurate after learning the method) compared to shitty small mechanical keyboards of the era. After diving into Chinese I realized how deficient the English HR landscape was and how badly under-researched the area is. I think the standard QWERTY keyboard is just too efficient for the Euro zone languages to make the research investment worthwhile. My 2 cents / guess. There are Graffiti like apps available on IOS. What I've found is that with english keyboard auto-correct, my typing is so much faster without Graffiti using the standard Apple IOS English keyboard. Kind of a disappointment... Another conjecture, I suspect much of the morbidity in English HR research has something to do with Graffiti patent lock up between what ever remains of Palm, Xerox, and some heretofore unheard of company called Communication Intelligence Corporation.
- ocrimgproc 9y ago
  I meant offline, not online recognition.
- tonfa 9y ago
  On android, https://play.google.com/store/apps/details?id=com.google.android.apps.handwriting.ime&hl=en https://play.google.com/store/apps/details?id=com.google.and... supports a bunch of languages (and emoji).
  - rabboRubble 9y ago
    I hated the Android phone I had 5 years ago and haven't looked back, so apparently I'm not up to speed on developments in the Android marketplace. I stand corrected on everything stated above! I'll have to borrow a friend's phone to play with this and see how it compares to the old Graffiti.

wibr 9y ago

Unfortunately they don't provide any details about the model or the training. Would be nice to have an example as a baseline for real-time performance on iOS devices.