3 ms·
> The performance difference arises (we believe) because when working from an SQLite database, the open() and close() system calls are invoked only once, wherea
by nightfly 5y ago
> The performance difference arises (we believe) because when working from an SQLite database, the open() and close() system calls are invoked only once, whereas open() and close() are invoked once for each blob when using blobs stored in individual files. It appears that the overhead of calling open() and close() is greater than the overhead of using the database. The size reduction arises from the fact that individual files are padded out to the next multiple of the filesystem block size, whereas the blobs are packed more tightly into an SQLite database.
- sumtechguy 5y agoOne trick I used to use is to put a small read/write cache in front of file operations. I would usually pick something like the size of a sector or cluster. Windows is pretty bad at small writes, and mearly 'ok' at reads because you usually can just get away with the windows file store cache in that case. I came across the write cache bit accidently when trying to minimize wear on an embedded flash device. It was borderline I thought I had done something wrong. Minimizing open/close and keeping your reads/writes close to sector/cluster size on many filesystems can produce some very nice results. As you can minimize the context switches from user space to kernel. In the case of adding in a 'db' layer packing probably helps as well as slack on small files is huge percentage wise of the total file. So you would be more likely to hit the file cache as well as any in built ones for your stack.