> The performance difference arises (we believe) because when working from an SQLite database, the open() and close() system calls are invoked only once Yes, o

> The performance difference arises (we believe) because when working from an SQLite database, the open() and close() system calls are invoked only once Yes, open() can introduce significant overhead, especially when doing it sequentially on a remote filesystem, e.g. NFS. But that's using the filesystems in the most inefficient way possible, there are better ways to do it. E.g. you can pre-touch the files or even pre-open them in parallel and then point to /proc/self/fd/<N> instead, which turns an open() into a dup(). I was able to make some properietary software run 5x faster that way simply by feeding it already open file descriptors.