4 ms·

How would you identify similarity though when you know nothing about the encoding?

by usrusr 5y ago

eru 5y ago
In general, you could try many encodings roughly in order of algorithmic complexity. Not sure if that's any use here. The really big problem is lack of data. With sufficient amounts of data and the assumption that we are dealing with human writing, you can probably figure out the encoding.
raverbashing 5y ago
By statistics: Word length, frequency of symbols, etc It is trivial to decode English (or any other alphabetical language) gone through a substitution cypher. Now, of course, it's not that simple with older languages, but this is a good starting step.
- hnbad 5y ago
  The assumption is both that this hasn't been tried and that an AI would perform significantly better at this despite lacking any of the necessary training data.
- rvense 5y ago
  > It is trivial to decode English (or any other alphabetical language) gone through a substitution cypher. As far as I know, it's only trivial if the substitution is symbol to symbol. If you mess with the letter frequencies, it becomes a lot harder. If you, say, use an orthography that's more in line with the phonology of language in its current form - or devise one that is as quirky, but in a different way. We are talking about a script with 700+ symbols, mathing it together with one that is written in 26 is nonsensical. This is emphatically not a cryptographic problem. It would really suit ML'ers to learn to take of their shoes and listen a lot more than they speak when they walk into a new scientific discipline. (BTW, I don't think many modern linguists would accept the term "alphabetic language"; a language is not defined by its writing system, and many written languages are routinely used with multiple writing systems)
  - raverbashing 5y ago
    > it's only trivial if the substitution is symbol to symbol Yes it will be more difficult if you play around and change the syntax, join letters, etc. But you usually can figure it out. > "alphabetic language"; a language is not defined by its writing system What I mean is a language not written with an abjad, or a syllabary. As in, the actual thing you're trying to decode.
    - rvense 5y ago
      You will maybe be able to "figure it out" if you know English. The use of maths to compare languages hinges on the frequencies being comparable, which is as much a trait of the script used as it is of the language itself.
  - mcswell 5y ago
    Along these lines: a script with over 700 symbols is definitely not an alphabetic script. Possibly syllabic, although most syllabic writing systems have fewer symbols than that. And of course, it may not be a script at all.

How would you identify similarity though when you know nothing about the encoding?

by usrusr 5y ago

How would you identify similarity though when you know nothing about the encoding?

eru 5y ago

In general, you could try many encodings roughly in order of algorithmic complexity. Not sure if that's any use here. The really big problem is lack of data. With sufficient amounts of data and the assumption that we are dealing with human writing, you can probably figure out the encoding.

raverbashing 5y ago

By statistics: Word length, frequency of symbols, etc It is trivial to decode English (or any other alphabetical language) gone through a substitution cypher. Now, of course, it's not that simple with older languages, but this is a good starting step.

hnbad 5y ago
The assumption is both that this hasn't been tried and that an AI would perform significantly better at this despite lacking any of the necessary training data.
rvense 5y ago
> It is trivial to decode English (or any other alphabetical language) gone through a substitution cypher. As far as I know, it's only trivial if the substitution is symbol to symbol. If you mess with the letter frequencies, it becomes a lot harder. If you, say, use an orthography that's more in line with the phonology of language in its current form - or devise one that is as quirky, but in a different way. We are talking about a script with 700+ symbols, mathing it together with one that is written in 26 is nonsensical. This is emphatically not a cryptographic problem. It would really suit ML'ers to learn to take of their shoes and listen a lot more than they speak when they walk into a new scientific discipline. (BTW, I don't think many modern linguists would accept the term "alphabetic language"; a language is not defined by its writing system, and many written languages are routinely used with multiple writing systems)
- raverbashing 5y ago
  > it's only trivial if the substitution is symbol to symbol Yes it will be more difficult if you play around and change the syntax, join letters, etc. But you usually can figure it out. > "alphabetic language"; a language is not defined by its writing system What I mean is a language not written with an abjad, or a syllabary. As in, the actual thing you're trying to decode.
  - rvense 5y ago
    You will maybe be able to "figure it out" if you know English. The use of maths to compare languages hinges on the frequencies being comparable, which is as much a trait of the script used as it is of the language itself.
- mcswell 5y ago
  Along these lines: a script with over 700 symbols is definitely not an alphabetic script. Possibly syllabic, although most syllabic writing systems have fewer symbols than that. And of course, it may not be a script at all.