General purpose compressors and information distance measures have become super interesting to me while I've been investigating alternative language models. I'

General purpose compressors and information distance measures have become super interesting to me while I've been investigating alternative language models. I've been playing around with an attention mechanism that combines the idea of using normalized compression distance (gzip) with discrete convolution between candidate sequences (sliding window of N bytes over each). Another round of normalization over the convolution outputs - accommodating varying lengths - allows for us to compare candidate sequences for relevant information on equal grounds. The NCD formula I am using right now: NCD(x,y) = (C(xy) - MIN(C(x),C(y))) / MAX(C(x),C(y)) No weird parameters or any other things to tune. The only parameters are the source documents and the input context/query.