5 ms·
PyCantonese: Cantonese Linguistics and NLP in Python
- totoglazer 4y agoSeems wild to build a new python NLP library totally disconnected from the current ecosystem. Particularly spaCy, but really any of NLTK, gensim, huggingface. Would love to understand the context of that choice.
- deleted 4y ago[deleted]
- solarmist 4y agoBecause Academia. "Lee, Jackson L., Litong Chen, Charles Lam, Chaak Ming Lau, and Tsz-Him Tsui. 2022. PyCantonese: Cantonese Linguistics and NLP in Python. Proceedings of the 13th Language Resources and Evaluation Conference." That said parsing is not a solved problem in a lot of languages (particularly ones without spaces) and frameworks like spaCy just integrate existing solutions into a common pipeline and datastructures.
- kylebgorman 4y agoThe first author's an independent scholar so I don't think that's right.
- throwaway290 4y agoCantonese is in my opinion a richer and more interesting language than Mandarin, yet much more difficult in terms of finding resources and tools for working with it. I wish I could support this project but for some reason none of Github's paid features, including sponsoring, seem to work on my account.
- fartcannon 4y agoNice to see some work being done with Cantonese. Large part of North American Chinese speak Cantonese but its rarely supported officially in any tools.
- kelvie 4y agoGoogle's cantonese IME is a godsend for us in this group (that never learned other pingyin)
- contingencies 4y agoWhenever someone talks about Cantonese I instantly think of Ah To's cartoons of Cantonese proverbs: https://says.com/my/fun/81-chinese-cantonese-proverbs-in-one-cartoon-picture-by-ah-to https://says.com/my/fun/81-chinese-cantonese-proverbs-in-one... https://writecantonese8.wordpress.com/2014/02/25/cantonese-proverbs-in-one-picture/ https://writecantonese8.wordpress.com/2014/02/25/cantonese-p...
- closed 4y agoAs a person learning cantonese, pycantonese and cantonese.sheik.co.uk were so helpful! The trickiest thing to explain to people is that Cantonese is a diaglossika. It's spoken, but to write you would use mandarin. So there aren't a ton of giant written language corpuses to train on. AFAICT that's why tools like this often use simple heuristics to do word segmentation. I remember going down a deep rabbit hole, before finally just packaging a small tool to do cantonese word segmentation using the A* algorithm: https://github.com/machow/cantocut https://github.com/machow/cantocut
- kevinstubbs 4y agoI thought the same thing as you, but spurred by the other commenters, found this: https://en.wikipedia.org/wiki/Written_Cantonese https://en.wikipedia.org/wiki/Written_Cantonese
- closed 4y agoThis is a minor point, but I'm cantonese and aware it can be written. (This is why I linked to a tool to segment written cantonese :).
- phantomathkg 4y agoCantonese can be written, it is just that the character is not as common as those you see in Mandarin.
- Starfox0518 4y agoNot 100% Accurate. A lot of us who speak Cantonese as mother tongue use "Cantonese" as written languages on online forums / medias. Written Mandarin normally appear on news or more formal environment.
- gaudat 4y agoWow the authors are from the newest universities in Hong Kong. Hope they can make it big without supports from bigger unis like CUHK!