4 ms·
For what it's worth, I've built an entire data ingest pipeline with PostgreSQL that stores tens of millions of PDFs in a giant database table and...it works fin
by snuxoll 3y ago
For what it's worth, I've built an entire data ingest pipeline with PostgreSQL that stores tens of millions of PDFs in a giant database table and...it works fine.
Now we are working on scooping up a bunch of small (1-2KB) files we want to index with metadata and chucking them into a SQLite file, because as fast as `grep` is searching through the text the constant open() and close() calls are much more impactful.