3 ms·
stat to get filesize is a pathlogically bad pattern on Windows. If I remove the call to fileSize() in kvtest and instead replace it with 16k (since the max size
by muststopmyths 7y ago
stat to get filesize is a pathlogically bad pattern on Windows. If I remove the call to fileSize() in kvtest and instead replace it with 16k (since the max size in the post is 12k), I can significantly improve the time measured on Windows 10.
Before:
C:\temp>kvtest run test1.dir --count 100k --blob-api
--count 100000 --max-id 1000 --asc
Total elapsed time: 9.495
Microseconds per BLOB read: 94.950
Content read rate: 104.0 MB/s
After:
C:\temp>kvtest run test1.dir --count 100k --blob-api
--count 100000 --max-id 1000 --asc
Total elapsed time: 5.218
Microseconds per BLOB read: 52.180
Content read rate: 313.3 MB/s
showing here the stable-ish numbers after a couple of runs of each version to let the cache warm up.
Windows is a completely different beast than Unix and you need to know the beast to tame it.
You might say that stat+open+read+close should be compared/measured together. In that case my answer would be that you would not read all files in a directory in this way on Windows anyway. You'd use overlapped I/O with buffers of FS cache granularity (which is 64Kb, unless it has changed in recent versions of Windows). It wouldn't matter for smaller files anyway (since the OS aggressively reads ahead when you open a file) and would be better for larger files.
It is a bugbear of mine when I see complaints about Windows from people who never actually tried to optimize for the OS.
reading blobs in Sqlite from db is still ~3x faster on my Surface Pro 2017 (don't call it Surface Pro 5!), but that's to be expected.
- SQLite 7y agoThanks for this. I did not realize that fileSize() was a performance issue on Windows. I'll attempt to update kvtest.c to deal with that, rerun the tests, and update the page, as I have opportunity.
- pdimitar 7y agoApologies for a hijack but since you're here, I really wanted to say thank you for sqlite! Saved me many headaches and made me a better programmer. You're amazing for creating it. And if you don't mind a question: are you guys up for making a strictly typed variant of sqlite? I'm aware it is going to be a completely different beast and many wouldn't expect backwards compatibility. Just curious if such a strongly-typed variant is on your radar at all.
- muststopmyths 7y agoAlso, that test program for the above arguments just loops through the first 1000 files 100 times. (iMax is 1000). Would be good to point that out a bit more clearly if it's intentional. Curious to know why if it is.