48 ms·

So this is only tangentially related in that it's ML related. I'm working on a system where a user uploads a single document that corresponds to several documen

So this is only tangentially related in that it's ML related. I'm working on a system where a user uploads a single document that corresponds to several documents we already have I'm the system (and we know which documents those are) and we are supposed to map pages from the single document to pages in the other know documents. I'm having a hard time finding literature on problems like this. I think the closest thing it's like is the stable matching problem, but I'm not sure. My current algorithm doesn't allow for missing or reordered pages. Does anyone have magic words I could google to find pertinent research/algorithms? I think if worst comes to worse I'll have to try to model the incoming page stream as something like a bayseian ... graph? It's been so long since I've done something like that (5 years), and that was in school.

KidComputer 6y ago
I'd generate document embeddings for each page then put the embedding vectors in a similarity search database like Facebook Research's Faiss.
setr 6y ago
It sounds like you're just trying to do similarity search? In which case, I imagine the key terms you want to be looking for are Information Retrieval and Recommendation Algorithms The most basic thing I'd imagine (assuming text docs) is something like index each document by TF-IDF, and then the new document by TF-IDF, and then rank results by similarity You're basically doing a google search on documents, but with a really big query (another document)
sixhobbits 6y ago
If the pages are ordered it sounds like a text alignment problem. Lots of literature on that especially in machine translation
sgt101 6y ago
If I were you I would measure reconstruction errors from transformer predictions using input from the source document and then difference vs the target documents. The challenge would be to get a good measure of the difference, perhaps us some cosine difference again from the transformer? If you have specialist vocab you can train the transformers - so maybe you are doing legal matching?

KidComputer 6y ago

I'd generate document embeddings for each page then put the embedding vectors in a similarity search database like Facebook Research's Faiss.

setr 6y ago

It sounds like you're just trying to do similarity search? In which case, I imagine the key terms you want to be looking for are Information Retrieval and Recommendation Algorithms The most basic thing I'd imagine (assuming text docs) is something like index each document by TF-IDF, and then the new document by TF-IDF, and then rank results by similarity You're basically doing a google search on documents, but with a really big query (another document)

sixhobbits 6y ago

If the pages are ordered it sounds like a text alignment problem. Lots of literature on that especially in machine translation

sgt101 6y ago

If I were you I would measure reconstruction errors from transformer predictions using input from the source document and then difference vs the target documents. The challenge would be to get a good measure of the difference, perhaps us some cosine difference again from the transformer? If you have specialist vocab you can train the transformers - so maybe you are doing legal matching?