We are using Sqlite as a primary datastorage for "file blobs" and implement a software based replication using our queue to multiple servers for redundancy. Th

We are using Sqlite as a primary datastorage for "file blobs" and implement a software based replication using our queue to multiple servers for redundancy. The use case is storing trace/profiling data, where we use one sqlite file for each customer per day. This way its easy to implement retention based cleanup and also there is little contention in write locking. We store about 1 terrabyte of data over the course of 2 weeks this way. Metadata is stored in Elasticsearch for querying the search results and then displaying a trace hits the Sqlite database. As looking at traces is a somewhat rare occurence we iterate over all fileservers and query them for trace data given an ID until we find the result. Reference https://www.sqlite.org/fasterthanfs.html https://www.sqlite.org/fasterthanfs.html