These matches suffer from a very common problem in recommendation engines, which is a tendency toward matching with the long tail of randomness. The problem is

These matches suffer from a very common problem in recommendation engines, which is a tendency toward matching with the long tail of randomness. The problem is that your matching engine assumes perfect accuracy in vectorization, but actually, the vectorization is a sample from a distribution. The distribution of the vectorization comes from the idea that it is trained on a mix of information and randomness, and this model is just one instance of the randomness. There are some sources of randomness in the model that you control, and others in the data. To get better recommendations, you first need to get the distributions for each element of the vectorization. The simplest approximation would be a variance around the model output. Each item should have a variance determined by the amount of information available for that sample. Now you have another problem, which is finding the closest vector. A p-test will fail you because that is telling you the probability that you came from the same distribution, which is going to be the random distribution for almost all pairs. You might ask something like, the probability that the two points come from a distribution that is distinct from the random distribution. You’d have to form that distribution, asses probability of membership, and then probability of rejection from randomness. You could also consider doing this for each vector element and returning a negative sum of log probability to represent the amount of information shared between vectors. But ultimately, you need a truth set to test these methods against. This is easy. You just split each person into N people by randomly assigning their comments to one of N identities. This could also be used as an element of the variance. An easy way to start is by bootstrapping the randomness. In this case, there is probably a random initialization vector in the model. You can bootstrap the distribution of each vector element w.r.t. the IV by running the model many times with random initialization vectors. You are still using a fixed point for the training data noise, but this is a start. To bootstrap the training noise variance, you can train many models using random selection of the data, and the same IV. A good heuristic split is to decide how many times you will run the model, x, and then do random selection by 1/sqrt(x). Ex. Run 100 times with 1/10th of the data in each run. Then you have two distributions for each vector element, one for the data randomness, and one for the model randomness, but the mean from the model randomness is the most informative. Now add the data variance / sqrt(x) to get a rough approximation of the true variance. These are all hacky ways to get a decent improvement, and the formal methods will also give great improvement on top of that, but are easy to mess up and often quite expensive to bootstrap.