3 ms·

I was wondering, has anyone used the zipf law to help deciphering? E.g one would expect the to be one or the top 3 most used word

timoth 5y ago
In general, assuming some (types of) words are more common is reasonable, but the specifics could be tricky. I know it was only an example, but "the", for example, might not exist in the language -- it might not have definite articles, or any articles equivalent to English articles at all.
lazyasciiart 5y ago
Not all languages even have articles - Finnish is an example.
vidarh 5y ago
A lot of languages lack an equivalent to "the". E.g Norwegian "shop" -> "butikk", "the shop" -> "butikken". Conversely, if you started with a North Germanic language like Norwegian instead of English as your reference, you might be tempted to assume gendered articles. E.g. "en sol" -> "a sun" masculine, "ei sol" -> "a sun" feminine, to confuse matters Norwegian bokmål (Norwegian has two written languages; they're very similar; bokmål is one of them) allows both masculine and feminine most places now (but not all - "butikk"/"shop" is unambiguously masculine - imagine trying to guess at how things like that affects stats without knowing the language), but also has a neuter gender which is less interchangeable. It gets worse. Imagine you'd like to try to look for possible suffixes corresponding to "the" as an alternative (I don't know any languages that uses other methods, but I'm sure they exist). Imagine now if your "unknown" language was similar to Norwegian. In Norwegian "the sun" can be either "solen" or "sola" depending on gender, and you can't assume a writer is even consistent (it's bad form to mix different gender for the same word in the same text, but that doesn't stop people), which would throw off your counts/stats. We have a pretty good catalogue of possible variations based on modern languages, so you certainly could try to construct a theory of a mapping based on known possible options (and you're not wrong to think in that direction as one approach worth testing), but there are many possible variations, so it's unfortunately not an easy one.
- Someone 5y ago
  > I don't know any languages that uses other methods, but I'm sure they exist Reading https://en.wikipedia.org/wiki/Article_(grammar)#Tables https://en.wikipedia.org/wiki/Article_(grammar)#Tables, Maltese uses prefixes.
CogitoCogito 5y ago
In addition to the grammatical issues other posters have replied with, another issue is that script in tablets probably would drop words that aren't strictly necessary for understanding which may mean very compact writing since the context could allow high compression (e.g. "John green house" or possibly "John g h" could be perfectly clear to the readers, but still mean a million different things if you don't have the context).

timoth 5y ago

In general, assuming some (types of) words are more common is reasonable, but the specifics could be tricky. I know it was only an example, but "the", for example, might not exist in the language -- it might not have definite articles, or any articles equivalent to English articles at all.

lazyasciiart 5y ago

Not all languages even have articles - Finnish is an example.

vidarh 5y ago

A lot of languages lack an equivalent to "the". E.g Norwegian "shop" -> "butikk", "the shop" -> "butikken". Conversely, if you started with a North Germanic language like Norwegian instead of English as your reference, you might be tempted to assume gendered articles. E.g. "en sol" -> "a sun" masculine, "ei sol" -> "a sun" feminine, to confuse matters Norwegian bokmål (Norwegian has two written languages; they're very similar; bokmål is one of them) allows both masculine and feminine most places now (but not all - "butikk"/"shop" is unambiguously masculine - imagine trying to guess at how things like that affects stats without knowing the language), but also has a neuter gender which is less interchangeable. It gets worse. Imagine you'd like to try to look for possible suffixes corresponding to "the" as an alternative (I don't know any languages that uses other methods, but I'm sure they exist). Imagine now if your "unknown" language was similar to Norwegian. In Norwegian "the sun" can be either "solen" or "sola" depending on gender, and you can't assume a writer is even consistent (it's bad form to mix different gender for the same word in the same text, but that doesn't stop people), which would throw off your counts/stats. We have a pretty good catalogue of possible variations based on modern languages, so you certainly could try to construct a theory of a mapping based on known possible options (and you're not wrong to think in that direction as one approach worth testing), but there are many possible variations, so it's unfortunately not an easy one.

Someone 5y ago
> I don't know any languages that uses other methods, but I'm sure they exist Reading https://en.wikipedia.org/wiki/Article_(grammar)#Tables https://en.wikipedia.org/wiki/Article_(grammar)#Tables, Maltese uses prefixes.

CogitoCogito 5y ago

In addition to the grammatical issues other posters have replied with, another issue is that script in tablets probably would drop words that aren't strictly necessary for understanding which may mean very compact writing since the context could allow high compression (e.g. "John green house" or possibly "John g h" could be perfectly clear to the readers, but still mean a million different things if you don't have the context).