4 ms·

I've always wanted a program or set of tools that I can use to generate my own dictionaries - not just a glossary, but real dictionaries with multiple definitio

I've always wanted a program or set of tools that I can use to generate my own dictionaries - not just a glossary, but real dictionaries with multiple definitions for words, derivations, example sentences, and so on. And, one other very important feature I'd need, would be to be able to run the dictionary on itself, to find words that are used in the descriptions for which there isn't yet a definition. This would be amazing, for example, to run on a large corpus, generate the dictionary, and then run it again to find words that are used but not defined - not just in the original corpus but in the definitions too. I've yet to find anything like this and have managed, over the years, to do this with some cobbled-together sed and awk hacks .. but I still think this is something that would be quite a viable commercial product - especially useful for international translations and creating properly-defined glossaries for documentation, etc. Anyone know of such tools? I'd love to have my own Dictionary builder, proper, and stop fantasizing about turning sed and awk scripts into a proper app ..

gajomi 5y ago
There are a couple starting points you could take. I spent a weekend hacking out a program that generates fake word/definition pairs with a transformer model set against a dictionary: https://youtu.be/XnJ2TKAn-Vk?t=1547 https://youtu.be/XnJ2TKAn-Vk?t=1547. If you substitute fake words for real words and have a sufficiently accurate model you could quickly generate reasonable and novel definitions. There are more complete versions of this kind of thing publicly available: https://github.com/turtlesoupy/this-word-does-not-exist https://github.com/turtlesoupy/this-word-does-not-exist > This would be amazing, for example, to run on a large corpus, generate the dictionary, and then run it again to find words that are used but not defined - not just in the original corpus but in the definitions too. I think this would be how you would gauge success of the model. That is to say, you would evaluate model accuracy on a set of held-out words with definitions that never appeared in your dictionary training set but appeared in context in your corpus. You would have to manually annotate whether or not the generated definition of these held out words was acceptable.
- boffinAudio 5y ago
  Thanks - that is indeed very interesting, and I will spend my weekend checking it out. >I think this would be how you would gauge success of the model. Yes, exactly. I think there would definitely be edge-cases, but the general rule is that there should not be any undefined terms/words in the final dictionary. The degree to which this can be achieved is of course related to the cyclomatic complexity of the original materials. But this is why I want this tool - to see how effective it is for creating training materials that prepare students for obtuse subjects.
solarmist 5y ago
The closest thing linguists have come up with, to my knowledge, are wordnets. They build definitions by the words, directly and indirectly, associated with them. You could then use those clusters to create individual definitions in some way. Have you looked at them? I think the biggest problem is that definitions are semantic which computers are terrible at. We’re at the infancy of being able to with transformers and large language models nowadays. So is start looking around for the proto-tools that will lead to what you're thinking about.
- boffinAudio 5y ago
  I have skirted around wordnets, because I'm not really a linguist - more of a technical documentation writer - but I will check it out now that I have a clue what to search for. Do you have any pointers to functional software tools for the creation of wordnets that you'd recommend?
knadh 5y ago
https://alar.ink https://alar.ink (Kannada-English dictionary) was built using https://github.com/knadh/dictmaker https://github.com/knadh/dictmaker It addresses some of the things you’ve mentioned.
- boffinAudio 5y ago
  Thanks very much for the link to dictmaker, I will spend my weekend playing with it and see how I get along. Much appreciated! EDIT: after visiting your site, somehow my browser language has been switched to Kannada and I'm now finding myself highly amused at seeing Twitter threads in this script. Would love to know how you did that!
  - knadh 5y ago
    That's strange. Apart from the fact that the dictionary website renders utf8 Kannada text, there's nothing else to it. Can't imagine how the browser managed to switch its language based on text found on a webpage!
wordreference 5y ago
I run a popular dictionary website (mostly for translation dictionaries). I have no shortage of terms that need definition or translation, but I would be happy to work with you on something like what you suggest.
- boffinAudio 5y ago
  Cool! This is only a hobby for me, as a technical documentation writer I find that a properly formed glossary/dictionary is key to the effort - but I'm very interested in the technology around this subject, so I'd be quite happy to see your website and get familiar with what you've done. Can you post a link here publically?
  - wordreference 5y ago
    Please see my profile or type my username into Google.

gajomi 5y ago

There are a couple starting points you could take. I spent a weekend hacking out a program that generates fake word/definition pairs with a transformer model set against a dictionary: https://youtu.be/XnJ2TKAn-Vk?t=1547 https://youtu.be/XnJ2TKAn-Vk?t=1547. If you substitute fake words for real words and have a sufficiently accurate model you could quickly generate reasonable and novel definitions. There are more complete versions of this kind of thing publicly available: https://github.com/turtlesoupy/this-word-does-not-exist https://github.com/turtlesoupy/this-word-does-not-exist > This would be amazing, for example, to run on a large corpus, generate the dictionary, and then run it again to find words that are used but not defined - not just in the original corpus but in the definitions too. I think this would be how you would gauge success of the model. That is to say, you would evaluate model accuracy on a set of held-out words with definitions that never appeared in your dictionary training set but appeared in context in your corpus. You would have to manually annotate whether or not the generated definition of these held out words was acceptable.

boffinAudio 5y ago
Thanks - that is indeed very interesting, and I will spend my weekend checking it out. >I think this would be how you would gauge success of the model. Yes, exactly. I think there would definitely be edge-cases, but the general rule is that there should not be any undefined terms/words in the final dictionary. The degree to which this can be achieved is of course related to the cyclomatic complexity of the original materials. But this is why I want this tool - to see how effective it is for creating training materials that prepare students for obtuse subjects.

solarmist 5y ago

The closest thing linguists have come up with, to my knowledge, are wordnets. They build definitions by the words, directly and indirectly, associated with them. You could then use those clusters to create individual definitions in some way. Have you looked at them? I think the biggest problem is that definitions are semantic which computers are terrible at. We’re at the infancy of being able to with transformers and large language models nowadays. So is start looking around for the proto-tools that will lead to what you're thinking about.

boffinAudio 5y ago
I have skirted around wordnets, because I'm not really a linguist - more of a technical documentation writer - but I will check it out now that I have a clue what to search for. Do you have any pointers to functional software tools for the creation of wordnets that you'd recommend?

knadh 5y ago

https://alar.ink https://alar.ink (Kannada-English dictionary) was built using https://github.com/knadh/dictmaker https://github.com/knadh/dictmaker It addresses some of the things you’ve mentioned.

boffinAudio 5y ago
Thanks very much for the link to dictmaker, I will spend my weekend playing with it and see how I get along. Much appreciated! EDIT: after visiting your site, somehow my browser language has been switched to Kannada and I'm now finding myself highly amused at seeing Twitter threads in this script. Would love to know how you did that!
- knadh 5y ago
  That's strange. Apart from the fact that the dictionary website renders utf8 Kannada text, there's nothing else to it. Can't imagine how the browser managed to switch its language based on text found on a webpage!

wordreference 5y ago

I run a popular dictionary website (mostly for translation dictionaries). I have no shortage of terms that need definition or translation, but I would be happy to work with you on something like what you suggest.

boffinAudio 5y ago
Cool! This is only a hobby for me, as a technical documentation writer I find that a properly formed glossary/dictionary is key to the effort - but I'm very interested in the technology around this subject, so I'd be quite happy to see your website and get familiar with what you've done. Can you post a link here publically?
- wordreference 5y ago
  Please see my profile or type my username into Google.