5 ms·
This is weird benchmarketing. They are comparing reading/writing 100,000 individual files vs writing 100,000 entries into a single file(sqlite database). For c
by tarasglek 9y ago
This is weird benchmarketing. They are comparing reading/writing 100,000 individual files vs writing 100,000 entries into a single file(sqlite database).
For comparison one could concatenate the same data into a single big file even faster than into sqlite.
They then do not offer logical analysis as to why things are faster.
My understanding is that reads are probably faster due to operating system readahead being able to predict reads better when they are within a single file. Writes are faster because they do one bulk COMMIT instead of many individual fsyncs.
- codehusker 9y agoThe third paragraph might offer the logical analysis you're looking for: > The performance difference arises (we believe) because when working from an SQLite database, the open() and close() system calls are invoked only once, whereas open() and close() are invoked once for each blob when using blobs stored in individual files. It appears that the overhead of calling open() and close() is greater than the overhead of using the database. The size reduction arises from the fact that individual files are padded out to the next multiple of the filesystem block size, whereas the blobs are packed more tightly into an SQLite database.
- imron 9y ago> For comparison one could concatenate the same data into a single big file even faster than into sqlite. One could, but then one would have great difficulty retrieving the individual files back when needed. The point is not about what has the greatest raw speed, the point is that that for applications that read lots of small files from the filesystem, they'll possibly get better performance and almost certainly use less disk space by sticking the files in SQLite instead.
- nimchimpsky 9y ago> have great difficulty retrieving Not that great, and the point is its comparing apples to oranges, pointless.
- function_seven 9y agoWriting your own methods to retrieve arbitrary chunks of data from a monolithic file would be a lot of work. Using map tiles as the example, how do you easily retrieve just the tiles in a specific region? Ok, how about all tiles that have the “hasLand” attribute set? Or the “containsCoastline” attribute? You’d end up rewriting your own version of a database.
- akira2501 9y ago> You’d end up rewriting your own version of a database. This is the hell we just crawled out of in one of our codebases.. it's not so much the access for a single process, but when you want to share the data in the file, that's when it seriously starts to get messy and breaking through otherwise clean abstraction layers and/or forcing recompile on compatibility breaking changes.
- adventured 9y agoIf you don't mind answering, out of sheer curiosity how was the data stored in the individual file/s?
- akira2501 9y agoStraight array of structs. The complication was that there were several of them, they were stored in bus-attached MRAM which was accessed directly (e.g. mmap of /dev/ram) by several independent processes. The idea came from previous generations of the product which generally ran on microcontrollers coupled with custom FPGA designs, so, not entirely unwarranted.. but completely incorrect for a newer product with much greater computing power than their previous devices.
- inetknght 9y ago> Writing your own methods to retrieve arbitrary chunks of data from a monolithic file would be a lot of work. Oh goodness have we come a long way if interacting with a file is harder than interacting with a SQL database. (it really depends on what language and/or library(ies) you use and for what purpose)
- deleted 9y ago[deleted]
- ori_b 9y agoA big part of it is because they're repeatedly walking the same path while reading and writing it. I got a 20% performance improvement on FreeBSD+ZFS with this change, which merely opens an fd, and uses that to stat and write, cutting the number of path traversals in half: https://eigenstate.org/paste/d41e3b69d258e3b52f7cee7a9921f027 https://eigenstate.org/paste/d41e3b69d258e3b52f7cee7a9921f02...
- justinclift 9y agoCool, have you pointed them at it? :)
- visarga 9y agoNo, no, if it is faster than the filesystem it must mean that it has a faster storage medium. Maybe it is writing to memory, or the aether. /s Now, seriously, how can the foot be faster than the shoe?
- geon 9y agoA reading/writing a single file can be faster than multiple files. Also querying a db can be faster than querying the fs.
- std_throwaway 9y agoWhy stop there? One could use the raw device to get even greater performance!
- nine_k 9y agoA database on a raw device is a thing. E.g. Oracle has this option for decades; MySQL, for some time, too.
- bigger_cheese 9y agoFrom this old stack overflow topic: https://stackoverflow.com/questions/6916011/how-do-i-get-windows-to-go-as-fast-as-linux-for-compiling-c https://stackoverflow.com/questions/6916011/how-do-i-get-win... Conclusion seems to be that filesystems on windows are not well optimized for the usecase of reading and writing from many small files at once. Which could explain poor windows performance in the benchmark (writing to 100,000 individual files).
- AstralStorm 9y agoWhich is why before making such broad statements they should check on the other platforms, which have much less broken file systems. (Check Ext4, ZFS, UFS2, HPFS+ and APFS at least.) What they did is check an untuned Ext4, UBIFS, HPFS+ and NTFS vs a tuned database in a single thread scenario. Especially misleading in the mmap case where synchronization would just kill the performance dead.
- majewsky 9y agoI find that comparison justified: How many people are even aware of the fact that filesystems can be tuned (except for, maybe, noatime and discard for SSDs)?
- aruggirello 9y agoExactly my thought. A possibly fairer comparison could be SQLite blobs vs. some archive format (with and without compression).