4 ms·
For n-gram indexes where n>1, I wonder if some of the sequence alignment techniques from bioinformatics could reconstruct the original documents?
by bane 5y ago
For n-gram indexes where n>1, I wonder if some of the sequence alignment techniques from bioinformatics could reconstruct the original documents?
- anitil 5y agoExactly my thought. I haven't delved in to what's here but I suspect this copyright workaround is not meant to be foolproof
- rmbyrro 5y agoAlso crossed my mind. We'd still would be left without illustrations, table organization, charts, etc, which in a considerable number of cases are of extreme relevance, though
- marginalia_nu 5y agoYou would typically require some sort of filtering in an N-gram index to prevent the index from growing larger than the corpus. Maybe something like a tf-idf-threshold, or some other heuristic. Otherwise you get a lot of useless "of that", "in an", "as a"-type word salad that is of no use to anyone, while what's actually useful sounds more like N-grams out of Tamarian.