3 ms·
Write a program that creates 100,000 1k-8k byte files. Write a program that creates 100,000 1k-8k byte files in a zip file with no compression. Run multiple c
by fleitz 15y ago
Write a program that creates 100,000 1k-8k byte files.
Write a program that creates 100,000 1k-8k byte files in a zip file with no compression.
Run multiple copies of these programs, see which writes files faster.
Keep running the program(s) until the drive is 90% full. (Put the files in different directories if you feel this is fair.)
Run the programs again.
You'll notice that on a decent 7200 RPM drive the ZIP code is writing about 40-50MB/sec on a good drive.
You'll probably be lucky to get 4-5MB/sec from the file code (unless you're on an SSD, even then it will probably be half)
If you feel like it also create a database and do the same with a file table like "CREATE TABLE data (id int NOT NULL PRIMARY KEY, data varbinary(8000))"
Inject 1000 to 2000 rows between commits.
The DB should also outperform the file code, but probably not the ZIP because it has to write the data twice.
The reason is that writing file to disk quickly produces a random IO pattern as the FS tries to allocate files all over the disk. Also, everytime you create a file you have to update a whole bunch of file descriptors, a modern file system is a B-tree and journal. Sounds awfully like a database right? Except this database is optimized for large files, at least 32-64kbytes and preferably in the hundreds of MB.
- dmpk2k 15y agoThis is specific to the type of filesystem in use. Something like ZFS does not exhibit these artifacts.
- gnaritas 15y agoYou're biassing the test towards small files and particular file systems that don't deal with them as well, and edge cases where you think the db is highly optimized; I'd imagine that's because you know outside those bounds the raw file system will kick its butt, especially given the tools that let you avoid the application entirely by streaming the file from disk.
- AlisdairO 15y agoIn fairness, while his test is biased towards dbs, it's also testing the claims that he made in his top level comment, which specified smaller files.
- gnaritas 15y agoWhich again, presumes I have to use multiple files. I can append small data to a log file faster than I can insert small rows of data to a database, and with much less effort. And my main point was, speed is not the reason we use databases, transactional semantics and query capabilities are.
- AlisdairO 15y agoRight, but he was talking about image files. edit: It's not that your point is wrong by any means, it's just that you're arguing against a point that he wasn't really emphasising. I don't think anyone, including him, would try to tell you that constant appending to a log file would be faster through a db than just a direct write to the file.