6 ms·
What's a use case for that final example? I can't help but thinking it would only impede overall performance rather than enhance it. It may be faster when writ
by zepolen 16y ago
What's a use case for that final example? I can't help but thinking it would only impede overall performance rather than enhance it.
It may be faster when writing, but most systems are write little, read a lot.
- fleitz 16y agoBut read load generally follows square law, your most popular files are going to be used an order of magnitude more than your least popular files. If you have 1TB of files you can probably get away with 1GB of RAM for caching with a 90% hit rate. This means that your read speed is something on the order of 11GB/sec. (eg. more than your 1Gbps network) Because ZIP files are essentially append only you can read from a ZIP file while it's being written to as long as you cache the file table. This means that you can have hundreds of readers from the ZIP file and only one writer. UNIX/Windows file locking semantics support this beautifully, readers don't lock the file and writers obtain a write lock. The basic design of the system is that the readers register an event handler with the writer and when the writer finishes writing a file it updates the readers with the location of the file in the ZIP file. Once you have that information it's a simple read call to the ZIP file.
- zepolen 16y agoWhat you described is still not going to be faster than just using the file system. It's just an unnecessary layer on top of it.
- fleitz 16y agoYou should probably test that assumption.
- zepolen 16y agoI want to, but it's hard to make an artificial test for this. You need to have a dataset that's larger than available ram and a read queue that follows real life access patterns but also uses different enough data to force disk reads to happen. If the theory checked out I would go ahead and implement it - but the theory says it would be slower.