5 ms·

European Languages Datasets

adontz 6y ago
I absolutely do not understand purpose of the folowing https://live.european-language-grid.eu/catalogue/#/resource/service/ld/898 https://live.european-language-grid.eu/catalogue/#/resource/... https://live.european-language-grid.eu/catalogue/#/resource/service/ld/904 https://live.european-language-grid.eu/catalogue/#/resource/... Since when strict eBNF grammars are useful for natural language processing?
- emteycz 6y ago
  "Finance English" may not be a natural language
  - bierjunge 6y ago
    Every language has something like that. In Germany it's called "Behördendeutsch" (administration german). Even as a native speaker with good language skills, you have to read all forms and letters at least twice to make sense of it.
- PeterisP 6y ago
  There's a research direction on 'controlled natural language' which is essentially a limited subset of a natural lanugage that still allows to express all that you need for a particular problem domain. They have their uses in natural language generation, where you may want to output some data in a way that's more readable to humans, and in various specialized query languages. For example, in some tasks you may prefer a voice command system that has more flexibility than mere keywords, where the instructions are phrases matching the users' language (you might want the same product to support many languages) but the system needs to understand only a very limited subset of language that can be expressed with a strict grammar - mainly because its ability to do stuff is also limited to what that subset can express. And this provides reliability - you can verify that the limited set of expressions that the system can understand get understood properly and those who aren't clear get rejected. This is bad for some use cases and good for others; picking a 'best effort' most likely interpretation (which many state of art methods do now) might be desirable or completely unacceptable depending on your use case. The benefit of a strict grammar over (for example) NN transformer architectures for NLG and NLU is that it's relatively straightforward to map the structures of that grammar to the structured data that your non-NLP code is using for the business logic, you can have a clear and debuggable 1-to-1 mapping for the semantics of these phrases.
visarga 6y ago
There's also "The Big Bad NLP Database", a much larger resource. https://datasets.quantumstat.com/ https://datasets.quantumstat.com/
- qayxc 6y ago
  I get eye cancer from the date format they use...
gumby 6y ago
Though this is an .eu domain, the languages covered are more than just the official EU ones (A quick glance showed entries for Basque (somewhat politically controversial) and Turkish (not part of the EU, though partially in continental Europe).
- maze-le 6y ago
  How is Basque "politically controversial"? It is an official language in the Basque Autonomous Community of Spain. Turkish is also a minority language in Greece. It doesn't have official status, but they have their own government mandated Turkish-language schools.
  - gumby 6y ago
    I was not aware that Turkish-language schools were in place in Greece. That's great! Admittedly I don't know the current state of the politics of Basque but when I lived in France its use was not encouraged in the Basque region.
    - AnimalMuppet 6y ago
      For these purposes, I'm not sure that its political status is relevant. It's a language; there are a significant number of people in Europe who speak it. So it belongs on a list of European languages.
  - disown 6y ago
    Which brings up an interesting point. What makes a language european? English, French, etc are official languages in many african and asian colonies due to conquest. Does it make french and english asian or african language? Is russian an asian language or a european language?
    - qayxc 6y ago
      > What makes a language european? That one's simple: it's a native language in a European country. > Does it make french and english asian or african language? No, since the official language is not the native language of and in these countries. Afrikaans, on the other hand is an African language, though it originated from (and is still very close to) Dutch. > Is russian an asian language or a european language? Russian is still a European language, despite most of Russia being located in Asia. The Asian parts of Russia have their own regional native languages (35 or so in total), with more than 20 official ones.

I absolutely do not understand purpose of the folowing https://live.european-language-grid.eu/catalogue/#/resource/service/ld/898 https://live.european-language-grid.eu/catalogue/#/resource/... https://live.european-language-grid.eu/catalogue/#/resource/service/ld/904 https://live.european-language-grid.eu/catalogue/#/resource/... Since when strict eBNF grammars are useful for natural language processing?

emteycz 6y ago
"Finance English" may not be a natural language
- bierjunge 6y ago
  Every language has something like that. In Germany it's called "Behördendeutsch" (administration german). Even as a native speaker with good language skills, you have to read all forms and letters at least twice to make sense of it.
PeterisP 6y ago
There's a research direction on 'controlled natural language' which is essentially a limited subset of a natural lanugage that still allows to express all that you need for a particular problem domain. They have their uses in natural language generation, where you may want to output some data in a way that's more readable to humans, and in various specialized query languages. For example, in some tasks you may prefer a voice command system that has more flexibility than mere keywords, where the instructions are phrases matching the users' language (you might want the same product to support many languages) but the system needs to understand only a very limited subset of language that can be expressed with a strict grammar - mainly because its ability to do stuff is also limited to what that subset can express. And this provides reliability - you can verify that the limited set of expressions that the system can understand get understood properly and those who aren't clear get rejected. This is bad for some use cases and good for others; picking a 'best effort' most likely interpretation (which many state of art methods do now) might be desirable or completely unacceptable depending on your use case. The benefit of a strict grammar over (for example) NN transformer architectures for NLG and NLU is that it's relatively straightforward to map the structures of that grammar to the structured data that your non-NLP code is using for the business logic, you can have a clear and debuggable 1-to-1 mapping for the semantics of these phrases.

visarga 6y ago

There's also "The Big Bad NLP Database", a much larger resource. https://datasets.quantumstat.com/ https://datasets.quantumstat.com/

qayxc 6y ago
I get eye cancer from the date format they use...

gumby 6y ago

Though this is an .eu domain, the languages covered are more than just the official EU ones (A quick glance showed entries for Basque (somewhat politically controversial) and Turkish (not part of the EU, though partially in continental Europe).

maze-le 6y ago
How is Basque "politically controversial"? It is an official language in the Basque Autonomous Community of Spain. Turkish is also a minority language in Greece. It doesn't have official status, but they have their own government mandated Turkish-language schools.
- gumby 6y ago
  I was not aware that Turkish-language schools were in place in Greece. That's great! Admittedly I don't know the current state of the politics of Basque but when I lived in France its use was not encouraged in the Basque region.
  - AnimalMuppet 6y ago
    For these purposes, I'm not sure that its political status is relevant. It's a language; there are a significant number of people in Europe who speak it. So it belongs on a list of European languages.
- disown 6y ago
  Which brings up an interesting point. What makes a language european? English, French, etc are official languages in many african and asian colonies due to conquest. Does it make french and english asian or african language? Is russian an asian language or a european language?
  - qayxc 6y ago
    > What makes a language european? That one's simple: it's a native language in a European country. > Does it make french and english asian or african language? No, since the official language is not the native language of and in these countries. Afrikaans, on the other hand is an African language, though it originated from (and is still very close to) Dutch. > Is russian an asian language or a european language? Russian is still a European language, despite most of Russia being located in Asia. The Asian parts of Russia have their own regional native languages (35 or so in total), with more than 20 official ones.