There is an excellent short story from the 50s, "Omnilingual" by H. Beam Piper, that explores deciphering a literally alien language. It is mercifully in the pu

There is an excellent short story from the 50s, "Omnilingual" by H. Beam Piper, that explores deciphering a literally alien language. It is mercifully in the public domain because it was written before copyright became infinite in the 1970s. You can read it here: https://gutenberg.org/ebooks/19445 https://gutenberg.org/ebooks/19445 I would be interested to read the book you're talking about, if you have the title! (Your math example reminds me a bit of Lambda Calculus, but I guess you'd have a hard time making a real language out of it.)