I'm doing this kind of thing for hundreds of billion records stored across hundreds of Terabytes (our business is to generate fingerprints based on binary files

I'm doing this kind of thing for hundreds of billion records stored across hundreds of Terabytes (our business is to generate fingerprints based on binary files). The added advantages: -- less failure points in case of data corruption -- any tool or any language can be used for parsing data -- easy to partition and move data elsewhere as needed -- grow as you see needed (only need disk space) -- dependable disk space, no caches nor anything else needed So, will depend on your context but you are not alone in case deciding to follow that route. On our side we are happy with the approach, sqlite gets slow when adding billions of records.