6 ms·

can't they use the zipf law ? https://en.wikipedia.org/wiki/Zipf%27s_law https://en.wikipedia.org/wiki/Zipf%27s_law The decreasing exponential law should allow

can't they use the zipf law ? https://en.wikipedia.org/wiki/Zipf%27s_law https://en.wikipedia.org/wiki/Zipf%27s_law The decreasing exponential law should allow to find "the" and some closed form POS words, so yeah determiners, prepositions and conjonctions.

jcranmer 4y ago
The total extant corpus of Linear A amounts to fewer than 10,000 characters (and this is, I believe, the largest corpus of any undeciphered script). There's not enough text to do statistical analysis.
- adrian_b 4y ago
  If all that text would have been that of a story, there would still have been a chance to decipher it. Even worse than the small number of texts is that all, or almost all, are just bookkeeping records, so they contain few words besides numbers, symbols for useful goods, e.g. wine, olive oil, barley, wool and so on, and proper names of places or people. So even if there might be a few hundreds of texts, most just reproduce the same phrases, only with different numbers and names substituted in them. Any statistics on this handful of stereotype phrases will offer no information about the statistics of the words of the Minoan language as used in a normal conversation or story telling.
luma 4y ago
Can we assume that any of those features would be part of this language?
- SemanticStrengh 4y ago
  a language without thoses would be hella weird and primitive, like stereotypical robotic talking. To answer you question, I don't know, do linear B have them?
  - tgv 4y ago
    Latin doesn't have articles (the/a), and frequently drops the verb. Aramaic encodes the article in a suffix. Arabic and Hebrew omit the vowels, leaving the interpretation depending on contextual clues. There are languages without auxiliary verbs. And there are tons of other constructions that English doesn't have.
    - Keysh 4y ago
      Written Arabic and Hebrew often omit vowels; the actual spoken languages do not, of course.
      - bigbillheck 4y ago
        The problem isn't in deciphering spoken Linear A.
  - greenyoda 4y ago
    There are languages used today that don't have a separate word for "the", such as Hebrew (which uses a prefix to denote "the"), or Chinese, which apparently doesn't use articles.[1] Also, knowing what the most common words are wouldn't really help you much if you didn't know what the documents are about. For example, if they were trade records, they might contain a lot of text saying something like "X agrees to buy 20 pounds of olives from Y for $50 if delivered by next week". But if they were historical records of wars, other words may be more common. [1] https://mylanguages.org/chinese_articles.php https://mylanguages.org/chinese_articles.php
  - JadeNB 4y ago
    > a language without thoses would be hella weird and primitive, like stereotypical robotic talking. To answer you question, I don't know, do linear B have them? I think there are very few assumptions of the form "every reasonable language has […]" that hold up even for all current languages, let alone historical ones.
    - jhgb 4y ago
vjerancrnjak 4y ago
I wonder can they construct a deep learning model that encodes human languages from script shapes and then somehow figure out the God language from which Linear A script/language is derived. There's too little data for Linear A. But it might be enough if there's a God language oracle waiting to be fed new descendant languages.
- liliumregale 4y ago
  This is a God (language)-of-the-gaps argument: we can't figure out this rarely language, but maybe we can figure out an entirely unattested language instead, and also learn the correspondence between it and Linear A. Deep learning can predict plenty of phenomena in the world, sure, but it needs data, not aspirations.
  - vjerancrnjak 4y ago
    > figure out an entirely unattested language I did not say that. Human languages evolve in similar ways, use similar vocabulary, grammar etc. Linguistics has already unraveled the structure of many languages and the structure of evolution of language through time. I am not saying DL is THE approach to take, but given that there's only ~10k characters of Linear A, it is hard to tackle the problem without common representation of multiple languages that are close to it. That's the whole point of DL, how to build better and better representations, not how to accurately model uncertainty (which is what you get by doing statistics). I would say XLM [0] builds a common representation of a collection of languages and then works better on machine translation for languages for which the data is scarce but that are related to the languages in the model. (what it also does is discover and represent the structure of part-of-speech, grammar, entities etc. without being told about those particular things) Does there exist an abundance of data for languages close to Linear A? If not, then I admire the work of all that try to untangle this with their brains alone. 0: https://github.com/facebookresearch/XLM https://github.com/facebookresearch/XLM
    - Contexti 4y ago
      > Does there exist an abundance of data for languages close to Linear A? If not, then I admire the work of all that try to untangle this with their brains alone. In the article, Dr. Ester Salgarella says: "we have not yet identified the linguistic family the Minoan language belongs to (unless it has to be taken as an ‘isolated’ language)" If we knew that the Minoan language belonged to some extant language family and we had an abundance of data, the mystery of Linear A would already have been solved decades ago. In general, there's very little data for any of the Palaeo-European languages that got replaced by Indo-European languages. Linguistic relatives of the Minoan language could have gone extinct when their speakers shifted to Greek or some other Indo-European language. It is also possible that other Minoan languages died out centuries or millenia before the arrival of Indo-Europeans. I don't believe we will ever know.
mtlmtlmtlmtl 4y ago
You really think that in decades of linguists studying Linear A, no one has thought of trying Zipf's law? If scientists have studied something for this long, and you come up with an idea that fits in a single paragraph, it's probably been tried and didn't work. Unless you're the field's leading expert in which case you would be off doing it, not posting it on HN :) Edit: typos
- escape_goat 4y ago
  I understand the impulse to point out the obvious, but when the question is asked honestly rather than arrogantly or dismissively, it is even better to wait for someone to provide the specific answer; in this case, the reason that Zipf's law is of no help.
  - mtlmtlmtlmtl 4y ago
    It wasn't my intent to be overly dismissive. But I see this sort of thing all the time, and I find this phenomenon interesting, so I wanted to engage with that aspect of it, specifically. I agree with you in general though. Dismissing these things out of hand isn't helpful either. But multiple people had already made substantive replies to the actual content of their idea, anyway.
    - SemanticStrengh 4y ago
      useless dismissal, I made a question not an affirmation. Besides it allow for an exploration of the search space of solutions, which stimulate the depth of the discussion and might allow finer grained questions that would then become possibly innovative
      - mtlmtlmtlmtl 4y ago
        Hardly useless, since it apparently inspired people to dig deeper and find more info :)
- wheelinsupial 4y ago
heavenlyblue 4y ago
I think the issue with Linear A is that the amount of preserved units of culture in that language is incredibly small, so whichever the statistics you can obtain from it are limited in use.
eklitzke 4y ago
I'm skeptical of the claim that this would work at all even if there was a larger corpus. Let's say you had a million pages of classical Chinese text, but absolutely no context about what the text meant or was about. By looking at it closely and using statistical analysis you could certainly determine various rules of the grammar, and you might even be able to guess that certain characters are grammatical constructions representing things like conjunctions and prepositions. But this isn't really going to let you translate anything.
- bloak 4y ago
  My guess is that if you had a really big, wide-ranging and high-quality corpus of a completely unknown human language then you probably would be able to decipher and translate it. If you could deduce or guess the grammatical structure the next step might be to look at which nouns can be subjects of which verbs, for example, and it might then be possible to guess which nouns refer to humans and which verbs describe actions that can normally only be performed by a human, and then ... well, there's lots of statistical stuff you can do with a really huge corpus ... It's an interesting problem to think about but it's not a problem we're ever likely to encounter in real life. It's more likely we'll discover a corpus of some alien, non-human language than a huge corpus of a completely unknown human language.
  - inglor_cz 4y ago
    The Voynich manuscript is fairly extensive (over 200 pages), illustrated, and we still do not know what the heck of a language it is written in. https://en.wikipedia.org/wiki/Voynich_manuscript https://en.wikipedia.org/wiki/Voynich_manuscript
    - ncmncm 4y ago
      ...assuming it is really language at all, and not just deliberate gibberish.
      - SemanticStrengh 4y ago
yk 4y ago
That makes assumptions about the language, for the languages I know: German has three definite articles, Latin doesn't have any, so it is not obvious what looking for "the" would result in either.
- sramsay 4y ago
  I think people (here and below) are getting hung up on definite articles, but Zipf's Law makes no such observation. It says only that a word's frequency in a natural language corpus tends to be in inverse proportion to its rank in a frequency table. In English, the most frequent words are articles, but the general observation about word frequency holds across languages (whether those languages have articles or not).
  - seoaeu 4y ago
    "The most frequently appearing words in this pile of un-translateable text are the most common words in the language it is written in" seems like it falls somewhere between blindingly obvious, and entirely useless. Unless you have some clue what those words mean, how does that observation help you?
    - sramsay 4y ago
      It doesn't (in this case), and I didn't say it did. And there's nothing "blindingly obvious" about the ubiquity of the Zipf curve.
    - bee_rider 4y ago
      Just from skimming the wikipedia article, it doesn't seem useful for translating. But it is slightly stronger than "The most frequently appearing words in this pile of un-translateable text are the most common words in the language it is written in." It tells you that, for example, the most popular word should be about twice as popular as the second most popular word. It doesn't tell you what those words are, but it is a pretty specific observation about the frequency/rank relationship. So, as the wikipedia article liked about points out, it can tell us that the Voynich Manuscript was probably written in a language (of course, it could be a cypher of a real language or something made up, like elvish in Lord of the Rings, but it probably isn't just a random collection of symbols because it is unlikely that a random collection of symbols would happen to follow this distribution).
cge 4y ago
In addition to the other comments on difficulties of such analyses, an additional difficulty may be the type of inscriptions in the corpus. We understand Linear B, for example, because it is early Greek. But the texts are not narrative prose or poetry: they're administrative records, mostly lists and inventories. If Linear A texts are of similar types, then trying to decipher the language from them alone may be challenging or impossible, unless it can be linked to a known language: the forms of speech used may simply be too limited. Trying to understand English grammar by looking only at bare financial statements would likely be extremely hard.
scheme271 4y ago
The language might not have that. E.g. latin doesn't have a word that corresponds to "the". Languages may indicate conjunctions or some types of prepositions or determiners by changing the case of a noun or the conjungation of a verb.
bigbillheck 4y ago
Zipf's law is a statement about the results of a frequency analysis. Frequency analysis has been a core part of Linear A efforts for a long time, as you'd expect considering how important it was to Ventris in deciphering Linear B. See for example "On the Language of Linear A" from 1958 (https://gredos.usal.es/bitstream/handle/10366/73246/On_the_Language_of_Linear_A.pdf?sequence=1 https://gredos.usal.es/bitstream/handle/10366/73246/On_the_L...) or "Greek-like Elements in Linear A" from 1963 (https://grbs.library.duke.edu/article/viewFile/11991/4031 https://grbs.library.duke.edu/article/viewFile/11991/4031). For something more recent, there's "Linear A and Linear B: Structural and contextual concerns" from 2017 (https://core.ac.uk/download/pdf/200196986.pdf https://core.ac.uk/download/pdf/200196986.pdf). (One of the authors of that one, Steele, also has a blog: https://crewsproject.wordpress.com/tag/linear-a/ https://crewsproject.wordpress.com/tag/linear-a/)

jcranmer 4y ago

The total extant corpus of Linear A amounts to fewer than 10,000 characters (and this is, I believe, the largest corpus of any undeciphered script). There's not enough text to do statistical analysis.

adrian_b 4y ago
If all that text would have been that of a story, there would still have been a chance to decipher it. Even worse than the small number of texts is that all, or almost all, are just bookkeeping records, so they contain few words besides numbers, symbols for useful goods, e.g. wine, olive oil, barley, wool and so on, and proper names of places or people. So even if there might be a few hundreds of texts, most just reproduce the same phrases, only with different numbers and names substituted in them. Any statistics on this handful of stereotype phrases will offer no information about the statistics of the words of the Minoan language as used in a normal conversation or story telling.

luma 4y ago

Can we assume that any of those features would be part of this language?

SemanticStrengh 4y ago
a language without thoses would be hella weird and primitive, like stereotypical robotic talking. To answer you question, I don't know, do linear B have them?
- tgv 4y ago
  Latin doesn't have articles (the/a), and frequently drops the verb. Aramaic encodes the article in a suffix. Arabic and Hebrew omit the vowels, leaving the interpretation depending on contextual clues. There are languages without auxiliary verbs. And there are tons of other constructions that English doesn't have.
  - Keysh 4y ago
    Written Arabic and Hebrew often omit vowels; the actual spoken languages do not, of course.
    - bigbillheck 4y ago
      The problem isn't in deciphering spoken Linear A.
- greenyoda 4y ago
  There are languages used today that don't have a separate word for "the", such as Hebrew (which uses a prefix to denote "the"), or Chinese, which apparently doesn't use articles.[1] Also, knowing what the most common words are wouldn't really help you much if you didn't know what the documents are about. For example, if they were trade records, they might contain a lot of text saying something like "X agrees to buy 20 pounds of olives from Y for $50 if delivered by next week". But if they were historical records of wars, other words may be more common. [1] https://mylanguages.org/chinese_articles.php https://mylanguages.org/chinese_articles.php
- JadeNB 4y ago
  > a language without thoses would be hella weird and primitive, like stereotypical robotic talking. To answer you question, I don't know, do linear B have them? I think there are very few assumptions of the form "every reasonable language has […]" that hold up even for all current languages, let alone historical ones.
  - jhgb 4y ago

vjerancrnjak 4y ago

I wonder can they construct a deep learning model that encodes human languages from script shapes and then somehow figure out the God language from which Linear A script/language is derived. There's too little data for Linear A. But it might be enough if there's a God language oracle waiting to be fed new descendant languages.

liliumregale 4y ago
This is a God (language)-of-the-gaps argument: we can't figure out this rarely language, but maybe we can figure out an entirely unattested language instead, and also learn the correspondence between it and Linear A. Deep learning can predict plenty of phenomena in the world, sure, but it needs data, not aspirations.
- vjerancrnjak 4y ago
  > figure out an entirely unattested language I did not say that. Human languages evolve in similar ways, use similar vocabulary, grammar etc. Linguistics has already unraveled the structure of many languages and the structure of evolution of language through time. I am not saying DL is THE approach to take, but given that there's only ~10k characters of Linear A, it is hard to tackle the problem without common representation of multiple languages that are close to it. That's the whole point of DL, how to build better and better representations, not how to accurately model uncertainty (which is what you get by doing statistics). I would say XLM [0] builds a common representation of a collection of languages and then works better on machine translation for languages for which the data is scarce but that are related to the languages in the model. (what it also does is discover and represent the structure of part-of-speech, grammar, entities etc. without being told about those particular things) Does there exist an abundance of data for languages close to Linear A? If not, then I admire the work of all that try to untangle this with their brains alone. 0: https://github.com/facebookresearch/XLM https://github.com/facebookresearch/XLM
  - Contexti 4y ago
    > Does there exist an abundance of data for languages close to Linear A? If not, then I admire the work of all that try to untangle this with their brains alone. In the article, Dr. Ester Salgarella says: "we have not yet identified the linguistic family the Minoan language belongs to (unless it has to be taken as an ‘isolated’ language)" If we knew that the Minoan language belonged to some extant language family and we had an abundance of data, the mystery of Linear A would already have been solved decades ago. In general, there's very little data for any of the Palaeo-European languages that got replaced by Indo-European languages. Linguistic relatives of the Minoan language could have gone extinct when their speakers shifted to Greek or some other Indo-European language. It is also possible that other Minoan languages died out centuries or millenia before the arrival of Indo-Europeans. I don't believe we will ever know.

mtlmtlmtlmtl 4y ago

You really think that in decades of linguists studying Linear A, no one has thought of trying Zipf's law? If scientists have studied something for this long, and you come up with an idea that fits in a single paragraph, it's probably been tried and didn't work. Unless you're the field's leading expert in which case you would be off doing it, not posting it on HN :) Edit: typos

escape_goat 4y ago
I understand the impulse to point out the obvious, but when the question is asked honestly rather than arrogantly or dismissively, it is even better to wait for someone to provide the specific answer; in this case, the reason that Zipf's law is of no help.
- mtlmtlmtlmtl 4y ago
  It wasn't my intent to be overly dismissive. But I see this sort of thing all the time, and I find this phenomenon interesting, so I wanted to engage with that aspect of it, specifically. I agree with you in general though. Dismissing these things out of hand isn't helpful either. But multiple people had already made substantive replies to the actual content of their idea, anyway.
  - SemanticStrengh 4y ago
    useless dismissal, I made a question not an affirmation. Besides it allow for an exploration of the search space of solutions, which stimulate the depth of the discussion and might allow finer grained questions that would then become possibly innovative
    - mtlmtlmtlmtl 4y ago
      Hardly useless, since it apparently inspired people to dig deeper and find more info :)
wheelinsupial 4y ago

heavenlyblue 4y ago

I think the issue with Linear A is that the amount of preserved units of culture in that language is incredibly small, so whichever the statistics you can obtain from it are limited in use.

eklitzke 4y ago

I'm skeptical of the claim that this would work at all even if there was a larger corpus. Let's say you had a million pages of classical Chinese text, but absolutely no context about what the text meant or was about. By looking at it closely and using statistical analysis you could certainly determine various rules of the grammar, and you might even be able to guess that certain characters are grammatical constructions representing things like conjunctions and prepositions. But this isn't really going to let you translate anything.

bloak 4y ago
My guess is that if you had a really big, wide-ranging and high-quality corpus of a completely unknown human language then you probably would be able to decipher and translate it. If you could deduce or guess the grammatical structure the next step might be to look at which nouns can be subjects of which verbs, for example, and it might then be possible to guess which nouns refer to humans and which verbs describe actions that can normally only be performed by a human, and then ... well, there's lots of statistical stuff you can do with a really huge corpus ... It's an interesting problem to think about but it's not a problem we're ever likely to encounter in real life. It's more likely we'll discover a corpus of some alien, non-human language than a huge corpus of a completely unknown human language.
- inglor_cz 4y ago
  The Voynich manuscript is fairly extensive (over 200 pages), illustrated, and we still do not know what the heck of a language it is written in. https://en.wikipedia.org/wiki/Voynich_manuscript https://en.wikipedia.org/wiki/Voynich_manuscript
  - ncmncm 4y ago
    ...assuming it is really language at all, and not just deliberate gibberish.
    - SemanticStrengh 4y ago

yk 4y ago

That makes assumptions about the language, for the languages I know: German has three definite articles, Latin doesn't have any, so it is not obvious what looking for "the" would result in either.

sramsay 4y ago
I think people (here and below) are getting hung up on definite articles, but Zipf's Law makes no such observation. It says only that a word's frequency in a natural language corpus tends to be in inverse proportion to its rank in a frequency table. In English, the most frequent words are articles, but the general observation about word frequency holds across languages (whether those languages have articles or not).
- seoaeu 4y ago
  "The most frequently appearing words in this pile of un-translateable text are the most common words in the language it is written in" seems like it falls somewhere between blindingly obvious, and entirely useless. Unless you have some clue what those words mean, how does that observation help you?
  - sramsay 4y ago
    It doesn't (in this case), and I didn't say it did. And there's nothing "blindingly obvious" about the ubiquity of the Zipf curve.
  - bee_rider 4y ago
    Just from skimming the wikipedia article, it doesn't seem useful for translating. But it is slightly stronger than "The most frequently appearing words in this pile of un-translateable text are the most common words in the language it is written in." It tells you that, for example, the most popular word should be about twice as popular as the second most popular word. It doesn't tell you what those words are, but it is a pretty specific observation about the frequency/rank relationship. So, as the wikipedia article liked about points out, it can tell us that the Voynich Manuscript was probably written in a language (of course, it could be a cypher of a real language or something made up, like elvish in Lord of the Rings, but it probably isn't just a random collection of symbols because it is unlikely that a random collection of symbols would happen to follow this distribution).

cge 4y ago

In addition to the other comments on difficulties of such analyses, an additional difficulty may be the type of inscriptions in the corpus. We understand Linear B, for example, because it is early Greek. But the texts are not narrative prose or poetry: they're administrative records, mostly lists and inventories. If Linear A texts are of similar types, then trying to decipher the language from them alone may be challenging or impossible, unless it can be linked to a known language: the forms of speech used may simply be too limited. Trying to understand English grammar by looking only at bare financial statements would likely be extremely hard.

scheme271 4y ago

The language might not have that. E.g. latin doesn't have a word that corresponds to "the". Languages may indicate conjunctions or some types of prepositions or determiners by changing the case of a noun or the conjungation of a verb.

bigbillheck 4y ago

Zipf's law is a statement about the results of a frequency analysis. Frequency analysis has been a core part of Linear A efforts for a long time, as you'd expect considering how important it was to Ventris in deciphering Linear B. See for example "On the Language of Linear A" from 1958 (https://gredos.usal.es/bitstream/handle/10366/73246/On_the_Language_of_Linear_A.pdf?sequence=1 https://gredos.usal.es/bitstream/handle/10366/73246/On_the_L...) or "Greek-like Elements in Linear A" from 1963 (https://grbs.library.duke.edu/article/viewFile/11991/4031 https://grbs.library.duke.edu/article/viewFile/11991/4031). For something more recent, there's "Linear A and Linear B: Structural and contextual concerns" from 2017 (https://core.ac.uk/download/pdf/200196986.pdf https://core.ac.uk/download/pdf/200196986.pdf). (One of the authors of that one, Steele, also has a blog: https://crewsproject.wordpress.com/tag/linear-a/ https://crewsproject.wordpress.com/tag/linear-a/)