4 ms·
Ask HN: What is the best database to use heavy text processing?
i started out working with relational database and
have stuck to them on my various clients and projects.
I started out with Sybase and moved over to SQL Server where I have been until.
In projects where it made sense, I have also used Elasticsearch,
Now I am part of starting a new open source project that will be almost entirely text manipulation and links. Initially our sample is about 100GB of non-compressed text.
Once (if) we prove our ideas we will make an announcement of some sort.
Now we need to store the text that is not so difficult.
My understanding is that serval NoSQL databases lend themselves well to that.
My problem now is identifying candidate databases (or something else) that will be able to store large amounts of text and a huge amounts of links between all the content.
There will be millions of links, and the links are not static.
The value in what we are developing are the links not really the text.
My present old fashioned idea is to use Postgres, break it all down to words and let links define everything else.
I was hoping some smart souls out there can give me better ideas.