5 ms·

Can anyone recommend how to build the following solution? - Full-text search on modern era PDFs (i.e no need for OCR) - Exact word search would suffice (fuzzy

by divan 4y ago

Can anyone recommend how to build the following solution? - Full-text search on modern era PDFs (i.e no need for OCR) - Exact word search would suffice (fuzzy/contextual search actually is less desirable) - Cross-platform frontend part that highlights and jumps to the found text within the document. Frontend should be embeddable (i.e. not a SaaS or just standalone UI) - As lightweight as possible (i.e. no Java, Python or Ruby) - Long-term oriented stack (i.e. minimum dependencies, ideally promise of compatibility) I'm looking at Mellisearch or Bleve for indexing/backend, and Syncfusion Flutter PDF viewer for frontend, but it still needs a lot of gluing code and I would love to explore more options. Google Pinpoint is pretty cool, and I use it a lot, but there is only hosted Google version, plus it's too smart (still can't get it to do exact word search).

deleted 4y ago
[deleted]
capableweb 4y ago
> - As lightweight as possible (i.e. no Java, Python or Ruby) I don't have suggestions for you, but I do have a question regarding this point. Why wouldn't Java be considered lightweight? Java literally runs on your SIM card, which is a very bare-bones environment to run something on, I'd probably consider something like that pretty lightweight.
- divan 4y ago
  Ha, I'm from that generation of developers who have the mental model of what is actually happening on the hardware level when you run the program. Doesn't necesarilly mean I overoptimize or think about struct fields offsets or cache branching, but I do have this in my mental model and just can't unlearn it. When I think about how many stuff needs to be moved in cpu/memory/io bus just to launch simple "Hello, World" in Java - I just cannot accept it. I do realize that for large programs that overhead is small, but still the JVM concept is something I want to avoid as much as possible. Plus the sheer scale of Java SDK and amount of legacy and complexity behind it exceeds my treshold of "avoiding complexity" by orders of magnitude. And the nail to the coffin of "no java" stance is, of course, experience with desktop Java applications. Consistenly the worst UX experience and performance I've seen in 25 years among desktop apps.
  - alexcg1 4y ago
    Don't remind me of desktop Java. What was that toolkit, swing(?) that was used in all the apps back in the day. PDFs have a special place in Hell, but Java desktop UXen deserve a whole special circle
    - divan 4y ago
      PDF history is pretty amazing, actually. The fact that PDF survived over so many decades is something worth reflecting upon :)
      - alexcg1 4y ago
        Cholera has survived for thousands of years. Doesn't mean I want to deal with it :) Seriously though, I agree. The fact that a file format can stick around that long is impressive.
snowstormsun 4y ago
pdfgrep with some formatting to add links open the correct page?
- alexcg1 4y ago
  Getting the URI of original PDF would be straightforward enough - I could whack that into the code tomorrow with a few lines. Opening up the correct page? I don't know of any standardized PDF reader that supports that kind of thing. And the format has such a history that even if it were supported (technically by Adobe - don't even get me started on what PDF readers support what formats), there's no guarantee the file itself would even have that cooked in.
simonw 4y ago
If you hadn't ruled out Python I'd be suggesting using Datasette + SQLite FTS - I've been building a whole bunch of different search engines on that (including ones for searching within OCRd PDF files) and the cost to host is trivial, since you just need to run a Python process somewhere with a binary SQLite database file. I usually use Vercel, Cloud Run or Fly for that. One example of a search engine I've built like this is the one on the Datasette website: https://datasette.io/-/beta?q=fts https://datasette.io/-/beta?q=fts - I wrote about how that works here: https://simonwillison.net/2020/Dec/19/dogsheep-beta/ https://simonwillison.net/2020/Dec/19/dogsheep-beta/
- divan 4y ago
  Interesting, thanks! I'll take a look (datasette is amazing).
alexcg1 4y ago
- Modern PDFs - if you wanna extract text and images, then the PDFSegmenter used in my example will work. If tables too, might need some additional jiggery-pokery, but definitely doable. I know other ppl using the same framework (Jina) who've accomplished it. - Exact word search - pretty simple. I've focused on more advanced stuff because color vs colour is same same but different. Also just because it's pretty easy since I'm just using pre-defined building blocks, not manually integrating stuff - Cross platform frontend - I've seen a lyrics search frontend [0] and I've built stuff in Streamlit before. Jina offers RESTful/gRPC/WebSockets gateways so it can't be too tough - Lightweight? I mean how lightweight do you want it? C? Bash? Assembly? I've found Python good for text parsing - Long-term: The notebook I wrote has a few (each of which have their own), but compared to others they're relatively lightweight. - Gluing code: I've been using pre-existing building blocks, and writing new Executors (i.e. building blocks) is relatively straightforward, and then scaling them up with shards, replicas, etc is just a parameter away. I'm more into the search side then the PDF stuff. The PDF side I've had experience with through bitter suffering and torment. Not a fun format to work with (unless you're into sado-masochism) [0] https://github.com/jina-ai/examples/tree/master/multires-lyrics-search https://github.com/jina-ai/examples/tree/master/multires-lyr...
- divan 4y ago
  Thanks for elaborated answer. Most of my use cases have to deal with 10-100 PDF small documents, some – 1000-2000, but I don't want the solution to choke on 10GB of huge PDFs (I was just uploading those to Google Pinpoint). So Go or Rust for backend should be good fit. By cross-platform frontend I meant web/ios/android/desktop. It's probably only Flutter, but I'm looking for other plugins than Syncfusion's one to try. I know that sounds like overkill for many people (website with search suffice), but I already have cross-platform apps that would benefit from this functionality, and web is a fallback there, not the main option.
  - alexcg1 4y ago
    I know folks with thousands to millions of PDFs using the Jina framework and it works fine. I hear what you're saying about frontends and lightweight though. Jina doesn't come with any cross-platform frontends, though Jina NOW has a Streamlit interface that's responsive (so works across devices)

by divan 4y ago

deleted 4y ago

[deleted]

capableweb 4y ago

> - As lightweight as possible (i.e. no Java, Python or Ruby) I don't have suggestions for you, but I do have a question regarding this point. Why wouldn't Java be considered lightweight? Java literally runs on your SIM card, which is a very bare-bones environment to run something on, I'd probably consider something like that pretty lightweight.

divan 4y ago
Ha, I'm from that generation of developers who have the mental model of what is actually happening on the hardware level when you run the program. Doesn't necesarilly mean I overoptimize or think about struct fields offsets or cache branching, but I do have this in my mental model and just can't unlearn it. When I think about how many stuff needs to be moved in cpu/memory/io bus just to launch simple "Hello, World" in Java - I just cannot accept it. I do realize that for large programs that overhead is small, but still the JVM concept is something I want to avoid as much as possible. Plus the sheer scale of Java SDK and amount of legacy and complexity behind it exceeds my treshold of "avoiding complexity" by orders of magnitude. And the nail to the coffin of "no java" stance is, of course, experience with desktop Java applications. Consistenly the worst UX experience and performance I've seen in 25 years among desktop apps.
- alexcg1 4y ago
  Don't remind me of desktop Java. What was that toolkit, swing(?) that was used in all the apps back in the day. PDFs have a special place in Hell, but Java desktop UXen deserve a whole special circle
  - divan 4y ago
    PDF history is pretty amazing, actually. The fact that PDF survived over so many decades is something worth reflecting upon :)
    - alexcg1 4y ago
      Cholera has survived for thousands of years. Doesn't mean I want to deal with it :) Seriously though, I agree. The fact that a file format can stick around that long is impressive.

snowstormsun 4y ago

pdfgrep with some formatting to add links open the correct page?

alexcg1 4y ago
Getting the URI of original PDF would be straightforward enough - I could whack that into the code tomorrow with a few lines. Opening up the correct page? I don't know of any standardized PDF reader that supports that kind of thing. And the format has such a history that even if it were supported (technically by Adobe - don't even get me started on what PDF readers support what formats), there's no guarantee the file itself would even have that cooked in.

simonw 4y ago

If you hadn't ruled out Python I'd be suggesting using Datasette + SQLite FTS - I've been building a whole bunch of different search engines on that (including ones for searching within OCRd PDF files) and the cost to host is trivial, since you just need to run a Python process somewhere with a binary SQLite database file. I usually use Vercel, Cloud Run or Fly for that. One example of a search engine I've built like this is the one on the Datasette website: https://datasette.io/-/beta?q=fts https://datasette.io/-/beta?q=fts - I wrote about how that works here: https://simonwillison.net/2020/Dec/19/dogsheep-beta/ https://simonwillison.net/2020/Dec/19/dogsheep-beta/

divan 4y ago
Interesting, thanks! I'll take a look (datasette is amazing).

alexcg1 4y ago

- Modern PDFs - if you wanna extract text and images, then the PDFSegmenter used in my example will work. If tables too, might need some additional jiggery-pokery, but definitely doable. I know other ppl using the same framework (Jina) who've accomplished it. - Exact word search - pretty simple. I've focused on more advanced stuff because color vs colour is same same but different. Also just because it's pretty easy since I'm just using pre-defined building blocks, not manually integrating stuff - Cross platform frontend - I've seen a lyrics search frontend [0] and I've built stuff in Streamlit before. Jina offers RESTful/gRPC/WebSockets gateways so it can't be too tough - Lightweight? I mean how lightweight do you want it? C? Bash? Assembly? I've found Python good for text parsing - Long-term: The notebook I wrote has a few (each of which have their own), but compared to others they're relatively lightweight. - Gluing code: I've been using pre-existing building blocks, and writing new Executors (i.e. building blocks) is relatively straightforward, and then scaling them up with shards, replicas, etc is just a parameter away. I'm more into the search side then the PDF stuff. The PDF side I've had experience with through bitter suffering and torment. Not a fun format to work with (unless you're into sado-masochism) [0] https://github.com/jina-ai/examples/tree/master/multires-lyrics-search https://github.com/jina-ai/examples/tree/master/multires-lyr...

divan 4y ago
Thanks for elaborated answer. Most of my use cases have to deal with 10-100 PDF small documents, some – 1000-2000, but I don't want the solution to choke on 10GB of huge PDFs (I was just uploading those to Google Pinpoint). So Go or Rust for backend should be good fit. By cross-platform frontend I meant web/ios/android/desktop. It's probably only Flutter, but I'm looking for other plugins than Syncfusion's one to try. I know that sounds like overkill for many people (website with search suffice), but I already have cross-platform apps that would benefit from this functionality, and web is a fallback there, not the main option.
- alexcg1 4y ago
  I know folks with thousands to millions of PDFs using the Jina framework and it works fine. I hear what you're saying about frontends and lightweight though. Jina doesn't come with any cross-platform frontends, though Jina NOW has a Streamlit interface that's responsive (so works across devices)