3 ms·
For my RAG projet [WDoc](https://github.com/thiswillbeyourgithub/WDoc/tree/dev https://github.com/thiswillbeyourgithub/WDoc/tree/dev) I use multiple pdf parser
by Ey7NFZ3P0nzAe 2y ago
For my RAG projet [WDoc](https://github.com/thiswillbeyourgithub/WDoc/tree/dev https://github.com/thiswillbeyourgithub/WDoc/tree/dev) I use multiple pdf parser then use heuristics the keep the best one. The code is at https://github.com/thiswillbeyourgithub/WDoc/blob/654c05c5b23f334d953955a0834e4e0637212ae5/WDoc/utils/loaders.py#L1910 https://github.com/thiswillbeyourgithub/WDoc/blob/654c05c5b2...
And the heurstics are partly based on using fasttext to detecr languages : https://github.com/thiswillbeyourgithub/WDoc/blob/654c05c5b23f334d953955a0834e4e0637212ae5/WDoc/utils/misc.py#L417 https://github.com/thiswillbeyourgithub/WDoc/blob/654c05c5b2...
It's probably crap for tables but I don't want to rely on external parsers.