This is immensely useful for my vying attempt to produce a compact lookup-tableless perfect hash generator for big datasets using ML. Naively one might think,

This is immensely useful for my vying attempt to produce a compact lookup-tableless perfect hash generator for big datasets using ML. Naively one might think, why not just do a standard loss - a point to point metric like mean squared error. But this is deeply flawed. Because it requires assigning each sample to a specific natural number, effectively reducing the solution space by an order of magnitude, n!. In practice, believe me, I have tried - the network never converges because the mapping is entirely arbitrary and has nothing to do with the samples. To remedy this, we need an innovation in loss functions / mathematics. The loss function for the output of the net needs to be a set function [1]. This set function should measure the distance between the Set of outputs of the net and the Set {1,2,...,n-1,n}. This is different from KL and all the other standard loss metrics, because we do not care about the point to point mappings, and have no ability to histogram or compute the probability distribution since those are non differentiable operations. On sorting: tensorflow has a differentiable sort but it is a hack and simply propagates the loss backwards to the position the original data was in before it ended up in its sorted position. This loss of dist(sort(Y_pred),[1,n]) provides better results but still fails for large data sets due to the fakeness of the sort derivative. I have a hunch that there is a mathematical way to uniquely measure some arithmetic quality to optimize for, that is maximum when the output set is the discrete uniform distribution. The mean, standard deviation and other statistical measures are terrible identifiers and actually, via statistical theory, we would need n moments for a dataset of size n, to uniquely identify the distribution..so scratch those off the list [2]. So there are two ways to achieve this milestone in ML: 1) a truly differentiable distance metric between two sets d(S,T) 2) a differentiable measure of ideal dispersion / density that forces the output set S to converge to the discrete uniform distribution (this is more problem specific to perfect hashes.) Perhaps this sort is the key to doing #1 generically so we can have a new type of NN based on the output Set instead of the specific points. It is late here but I am excited to hear feedback. [1] https://en.wikipedia.org/wiki/Set_function https://en.wikipedia.org/wiki/Set_function [2] https://en.wikipedia.org/wiki/Hausdorff_moment_problem https://en.wikipedia.org/wiki/Hausdorff_moment_problem