3 ms·

> We have a dictionary in PDF form The file contains a copyright notice by the Summer Institute for Linguistics. That might cause legal trouble, although if th

by yorwba 6y ago

> We have a dictionary in PDF form The file contains a copyright notice by the Summer Institute for Linguistics. That might cause legal trouble, although if the Australian government can host the PDF without repercussions, you'll probably fly under the radar. > I'm currently manually transcribing it all into YAML The scan looks relatively clean, so I think it should be possible to speed this up massively using OCR. > Also, for anyone who works in this space, any resource recommendations or tips for writing a translation engine. I don't work in this space, but I know about Apertium https://www.apertium.org/ https://www.apertium.org/ , a framework for writing rule-based translation engines. As for machine-learning approaches, some are simple enough to be used as tutorials for ML frameworks, e.g. https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html https://pytorch.org/tutorials/intermediate/seq2seq_translati... Note that this tutorial uses data from Tatoeba https://tatoeba.org/eng/ https://tatoeba.org/eng/ , which currently only has a single sentence in an Australian Aboriginal language, Noongar: https://tatoeba.org/eng/sentences/show_all_in/nys/und https://tatoeba.org/eng/sentences/show_all_in/nys/und For languages that are not closely related, you'll need much more training data. (A friend of mine tried to make it work for English-Korean with a few thousand sentences, but failed.)

thomasfromcdnjs 6y ago
Ah, you are an absolute legend. I will look into automating with OCR but it is also kind of cool to re-type it all to get a feel for the language myself. Apertium looks awesome, and I'm going through all the docs now. I think it will make sense for me to structure things like Apertium as it looks like a pretty active and relevant project. And yeah, the odds of getting some sweet ML system working seem quite low, as the languages are almost dead anyway so will be nigh impossible to get a data set. I have family member on the language advisory group so I will double check the copy right.
- yorwba 6y ago
  > I will look into automating with OCR but it is also kind of cool to re-type it all to get a feel for the language myself. I ran the PDF through OCRmyPDF https://ocrmypdf.readthedocs.io/en/latest/ https://ocrmypdf.readthedocs.io/en/latest/ with the language set to English and it actually did pretty well even at recognizing Kuku Yalanji words. Getting it into a structured format will require some custom processing to parse the layout, though. > the languages are almost dead anyway so will be nigh impossible to get a data set. One way you might be able to slightly boost usage is to create a mobile keyboard based on the dictionary, making it more convenient to type in the language. Keyman https://keyman.com/ https://keyman.com/ is an open-source tool for creating keyboard layouts, coincidentally created by the Summer Institute for Linguistics.
  - thomasfromcdnjs 6y ago
    It seems like you have done this before aha I will try the OCR too and make it work some how.
- anewhnaccount2 6y ago
  Just to mention, the Apertium community are very collaborative and always interested in all sort of minority languages, endangered and extinct languages and language revival. You should join their IRC channel #apertium and show them what you have already --- I'm sure they'd be interested and also possibly even have some ideas about things that could help.
  - thomasfromcdnjs 6y ago
    It looks like an awesome org, I will be definitely reaching out after reading your endorsement.
- throwaway_pdp09 6y ago
  > also kind of cool to re-type it all to get a feel for the language myself Don't. Or do, but in any event be really clear of your goals. Is your aim to get something done? Then do it the most efficient way (not by hand). Do you want to get a feel for the language? Then hand-type it all, but at the cost of slowing you down greatly. Decide up front what your goals are else you will mix your goals and try to ride two horses, as the saying goes, and make achieving either much, much harder. I've made this mistake many times, so take it as wisdom derived from really painful experience. Focus! (this may come across as lecturing and presumptuous but I've screwed so much up by lack of clear goals and focus).
  - hombre_fatal 6y ago
    Good point. As someone who often took the transcribe-by-hand route on my language projects to sneak in some incidental language practice, you also have to budget for all the typos and inexplicable mistakes you will make. You're going to have to go line by line in the source either way to proofread, so I recommend machine transcription over ape transcription.

by yorwba 6y ago

thomasfromcdnjs 6y ago

Ah, you are an absolute legend. I will look into automating with OCR but it is also kind of cool to re-type it all to get a feel for the language myself. Apertium looks awesome, and I'm going through all the docs now. I think it will make sense for me to structure things like Apertium as it looks like a pretty active and relevant project. And yeah, the odds of getting some sweet ML system working seem quite low, as the languages are almost dead anyway so will be nigh impossible to get a data set. I have family member on the language advisory group so I will double check the copy right.

yorwba 6y ago
> I will look into automating with OCR but it is also kind of cool to re-type it all to get a feel for the language myself. I ran the PDF through OCRmyPDF https://ocrmypdf.readthedocs.io/en/latest/ https://ocrmypdf.readthedocs.io/en/latest/ with the language set to English and it actually did pretty well even at recognizing Kuku Yalanji words. Getting it into a structured format will require some custom processing to parse the layout, though. > the languages are almost dead anyway so will be nigh impossible to get a data set. One way you might be able to slightly boost usage is to create a mobile keyboard based on the dictionary, making it more convenient to type in the language. Keyman https://keyman.com/ https://keyman.com/ is an open-source tool for creating keyboard layouts, coincidentally created by the Summer Institute for Linguistics.
- thomasfromcdnjs 6y ago
  It seems like you have done this before aha I will try the OCR too and make it work some how.
anewhnaccount2 6y ago
Just to mention, the Apertium community are very collaborative and always interested in all sort of minority languages, endangered and extinct languages and language revival. You should join their IRC channel #apertium and show them what you have already --- I'm sure they'd be interested and also possibly even have some ideas about things that could help.
- thomasfromcdnjs 6y ago
  It looks like an awesome org, I will be definitely reaching out after reading your endorsement.
throwaway_pdp09 6y ago
> also kind of cool to re-type it all to get a feel for the language myself Don't. Or do, but in any event be really clear of your goals. Is your aim to get something done? Then do it the most efficient way (not by hand). Do you want to get a feel for the language? Then hand-type it all, but at the cost of slowing you down greatly. Decide up front what your goals are else you will mix your goals and try to ride two horses, as the saying goes, and make achieving either much, much harder. I've made this mistake many times, so take it as wisdom derived from really painful experience. Focus! (this may come across as lecturing and presumptuous but I've screwed so much up by lack of clear goals and focus).
- hombre_fatal 6y ago
  Good point. As someone who often took the transcribe-by-hand route on my language projects to sneak in some incidental language practice, you also have to budget for all the typos and inexplicable mistakes you will make. You're going to have to go line by line in the source either way to proofread, so I recommend machine transcription over ape transcription.