4 ms·

I found surprising that shorthand compresses enough of the writing to resist statistical techniques, knowing approximately the corpus of language that the theol

I found surprising that shorthand compresses enough of the writing to resist statistical techniques, knowing approximately the corpus of language that the theologian would have used, and the source material he would usually quote from.

analog31 8y ago
I wonder if it resists cracking by logical processes because it's not logical. For instance, the author may have not followed a system, but chose symbols that were partly pictographic, or based on word associations that occurred to him at the moment. Plus, for the author, it didn't have to be archival, because he would have been aided by his own memory when reading it. My best chance at reading my own handwriting is to ask: "Now, what would I have meant here?" ;-) I hope nobody ever tries to read my notebooks.
- gnulinux 8y ago
  Languages can be deciphered by statistical techniques not because they are "logical" or have a "system" (well uhm you're sort of right, actually). In natural languages there is a certain distribution of each phoneme and syllable (let's say due to an unknown reason, but this is observed in all languages[1]), which makes this distribution to somewhat correlate with certain letters and symbols if we're lucky. E.g. suppose we know /i/ phoneme is 90% of vowels in a language and we see that the letter <e> is used about 70% of the time in a large corpus of this language. Our best guess is /i/ is transcribed as <e>. If this phoneme to symbol correlation is too noisy (e.g. in the case of Voynich Script or in this case) then we'll have harder time using statistical techniques. [1] There are certain universal phonological "Universals" that applies to 99.99% of languages. https://en.wikipedia.org/wiki/Linguistic_universal https://en.wikipedia.org/wiki/Linguistic_universal But still these don't let us find the entire structure of a language, and we have to use other techniques. Such as comparative study to other languages, finding loan words we know (e.g. finding the word "computer" or "Jesus" in a text of unknown language with unknown alphabet)
- seanp2k2 8y ago
  >Now, what would I have meant here? Interestingly (to me, at least), this is also how I remember where I put something if I've misplaced it. I ask myself, and really try to visualize, if I were putting that thing away right now, where would I put it, and what was I just doing with it that might give me a bit more context about where I might leave it for myself cleverly next time? It usually works.
  - Tor3 8y ago
    Heh, that's exactly what I do as well.. so it must be common to some extent.
sudosteph 8y ago
I imagine it's possible someone would have eventually cracked it that way. From my experience learning a few shorthand techniques though, I can see how it would be difficult regardless. - many shorthands, especially phonetic ones, are abjads and lack the full metadata around vowels which make context extremely important to understanding meaning. - It's typical for a shorthand writer to make have unique markings for words they write frequently which may not match the phonetic pattern (if that's even being used) which are hard to guess - shorthand is hard to parse in general because the strokes typically value flow over precision, so secondary and third strokes are usually are formed in relation to the prior ones rather than any absolute positioning scheme But really, I just don't think there are many people actually working on trying to solve old short hands with modern statistics + ML techniques. Maybe I'm wrong though and someone gave it a good go but something here really was particularly hard to solve.
- fernly 8y ago
  TIL a new word, abjad, "a type of writing system where each symbol or glyph stands for a consonant, leaving the reader to supply the appropriate vowel." Thanks! https://en.wikipedia.org/wiki/Abjad https://en.wikipedia.org/wiki/Abjad
  - schoen 8y ago
    You might also be interested in https://en.wikipedia.org/wiki/Abugida https://en.wikipedia.org/wiki/Abugida Also, I was wondering if someone had already made the "f y cn rd ths" joke about abjads; the answer is yes! http://www.learnclassicalhebrew.net/blog/?p=84 http://www.learnclassicalhebrew.net/blog/?p=84
PavlikPaja 8y ago
Shorthands may consist not only of the symbols themselves, but also their positions, or the thickness of the line (nib pens allowed that), so it's clear they could be hard to decipher for those who have no idea what to look for. https://archive.org/details/crossseclecticsh00crosiala https://archive.org/details/crossseclecticsh00crosiala

analog31 8y ago

I wonder if it resists cracking by logical processes because it's not logical. For instance, the author may have not followed a system, but chose symbols that were partly pictographic, or based on word associations that occurred to him at the moment. Plus, for the author, it didn't have to be archival, because he would have been aided by his own memory when reading it. My best chance at reading my own handwriting is to ask: "Now, what would I have meant here?" ;-) I hope nobody ever tries to read my notebooks.

gnulinux 8y ago
Languages can be deciphered by statistical techniques not because they are "logical" or have a "system" (well uhm you're sort of right, actually). In natural languages there is a certain distribution of each phoneme and syllable (let's say due to an unknown reason, but this is observed in all languages[1]), which makes this distribution to somewhat correlate with certain letters and symbols if we're lucky. E.g. suppose we know /i/ phoneme is 90% of vowels in a language and we see that the letter <e> is used about 70% of the time in a large corpus of this language. Our best guess is /i/ is transcribed as <e>. If this phoneme to symbol correlation is too noisy (e.g. in the case of Voynich Script or in this case) then we'll have harder time using statistical techniques. [1] There are certain universal phonological "Universals" that applies to 99.99% of languages. https://en.wikipedia.org/wiki/Linguistic_universal https://en.wikipedia.org/wiki/Linguistic_universal But still these don't let us find the entire structure of a language, and we have to use other techniques. Such as comparative study to other languages, finding loan words we know (e.g. finding the word "computer" or "Jesus" in a text of unknown language with unknown alphabet)
seanp2k2 8y ago
>Now, what would I have meant here? Interestingly (to me, at least), this is also how I remember where I put something if I've misplaced it. I ask myself, and really try to visualize, if I were putting that thing away right now, where would I put it, and what was I just doing with it that might give me a bit more context about where I might leave it for myself cleverly next time? It usually works.
- Tor3 8y ago
  Heh, that's exactly what I do as well.. so it must be common to some extent.

sudosteph 8y ago

I imagine it's possible someone would have eventually cracked it that way. From my experience learning a few shorthand techniques though, I can see how it would be difficult regardless. - many shorthands, especially phonetic ones, are abjads and lack the full metadata around vowels which make context extremely important to understanding meaning. - It's typical for a shorthand writer to make have unique markings for words they write frequently which may not match the phonetic pattern (if that's even being used) which are hard to guess - shorthand is hard to parse in general because the strokes typically value flow over precision, so secondary and third strokes are usually are formed in relation to the prior ones rather than any absolute positioning scheme But really, I just don't think there are many people actually working on trying to solve old short hands with modern statistics + ML techniques. Maybe I'm wrong though and someone gave it a good go but something here really was particularly hard to solve.

fernly 8y ago
TIL a new word, abjad, "a type of writing system where each symbol or glyph stands for a consonant, leaving the reader to supply the appropriate vowel." Thanks! https://en.wikipedia.org/wiki/Abjad https://en.wikipedia.org/wiki/Abjad
- schoen 8y ago
  You might also be interested in https://en.wikipedia.org/wiki/Abugida https://en.wikipedia.org/wiki/Abugida Also, I was wondering if someone had already made the "f y cn rd ths" joke about abjads; the answer is yes! http://www.learnclassicalhebrew.net/blog/?p=84 http://www.learnclassicalhebrew.net/blog/?p=84

PavlikPaja 8y ago

Shorthands may consist not only of the symbols themselves, but also their positions, or the thickness of the line (nib pens allowed that), so it's clear they could be hard to decipher for those who have no idea what to look for. https://archive.org/details/crossseclecticsh00crosiala https://archive.org/details/crossseclecticsh00crosiala