5 ms·

I couldn't see any fsync (or similar) in your filesystem code (sorry if I missed them) -- it doesn't seem like a completely fair comparison, as sqlite has vario

I couldn't see any fsync (or similar) in your filesystem code (sorry if I missed them) -- it doesn't seem like a completely fair comparison, as sqlite has various promises about commits being completely made, etc, which you won't get from just chucking files on a filesystem.

bogota 5y ago
If you only care about storing data and not doing any complex operations or query against it why are you using something like sqlite as well? It’s not a comparison as being in sqlite makes the ability to access this data significantly easier. This is comparing apples and dogs and i don’t see the merits.
- kevingadd 5y ago
  It's not uncommon to see people advocate for using sqlite as an alternative to flat files or json blobs. In particular, transactions provide nice properties vs a web of separate flat files, and the stronger schema can be a good alternative to json. There's a lot of great existing sqlite tooling, too.
  - samwillis 5y ago
    Not only other people, the creators also advocate for it as a file format: https://www.sqlite.org/aff_short.html https://www.sqlite.org/aff_short.html https://www.sqlite.org/appfileformat.html https://www.sqlite.org/appfileformat.html
    - tyingq 5y ago
      Sqlite archive files are somewhat interesting too: https://www.sqlite.org/sqlar.html https://www.sqlite.org/sqlar.html
  - le-mark 5y ago
    The history around mongrel2 using SQLite as a config file format may be instructive. Seems like there was a lot of resistance in that case.
- electroly 5y ago
  In one application I use sqlite to store JPEG thumbnails. That's it; there's nothing else in there. It's super handy; I specifically needed to reduce the number of files that I open and close because that's slow on Windows/NTFS. SQLite made this trivial. I could have managed a binary pack format on my own but I didn't have to.
  - jbluepolarbear 5y ago
    You could have easily done that with a single file with a tail header for look ups.
    - baq 5y ago
      Easier than SQLite? With less bugs and better tests? SQLite advertises itself as an fopen replacement. Sounds like a perfect match for parent’s use case.
      - KerrAvon 5y ago
        Depends on the indexing needs, right? Something like IFF should be fine if you need a sequence of images to store that you don’t need to actually query for individually.
        baq 5y ago
        Always, but I find it really hard to see how not using SQLite in this use case is the better option. Maybe if you need every single percentile of performance? In 99% of other cases just go with SQLite.
      - jbluepolarbear 5y ago
        Yes easier, as in time to integrate and get working correctly. I’ve used SQLite extensively over the last 10 years and yes it’s a good solution, but not a replacement for fopen. A flat file would be easier to integrate, test, and harden over SQLite. Would take about the same time as integrating SQLite into a system for the same purpose and would be easily extendable to support features as the system grows.
- baq 5y ago
  SQLite makes seeking in a file using a lookup table look like a complex operation from the developer’s perspective, which is absolutely a good thing unless you have a strong alternative rationale for your use case.
christophilus 5y ago
Yeah. I mentioned that in the final test where I write to a temporary file, then rename. This is much slower, probably due to an implicit fsync.
- electroly 5y ago
  You can disable the fsync calls in sqlite if you wanted to do a little better with this benchmark. You're also explicitly choosing the WAL in your go-sqlite3 configuration which is not at all replicated by your filesystem test. I think, honestly, that you're just going to mislead and confuse people who don't know any better with this writeup. I can write faster to /dev/null, too, but that isn't a very interesting comparison.
  - HideousKojima 5y ago
    Does /dev/null support sharding?
    - pa7ch 5y ago
      Yes.
    - grogers 5y ago
      Even better, it's embarrassingly parallel!
- cafxx 5y ago
  I am not aware of implicit fsyncs. Can you please link to what you are referring to?
  - prussian 5y ago
    Probably in reference to Ext4's[0] configurable rename-replace behavior under auto_da_alloc [0]: https://www.kernel.org/doc/Documentation/filesystems/ext4.txt https://www.kernel.org/doc/Documentation/filesystems/ext4.tx... Edit: note this isn't a universal property either, so it's still wrong: https://btrfs.wiki.kernel.org/index.php/FAQ#What_are_the_crash_guarantees_of_overwrite-by-rename.3F https://btrfs.wiki.kernel.org/index.php/FAQ#What_are_the_cra...

bogota 5y ago

If you only care about storing data and not doing any complex operations or query against it why are you using something like sqlite as well? It’s not a comparison as being in sqlite makes the ability to access this data significantly easier. This is comparing apples and dogs and i don’t see the merits.

kevingadd 5y ago
It's not uncommon to see people advocate for using sqlite as an alternative to flat files or json blobs. In particular, transactions provide nice properties vs a web of separate flat files, and the stronger schema can be a good alternative to json. There's a lot of great existing sqlite tooling, too.
- samwillis 5y ago
  Not only other people, the creators also advocate for it as a file format: https://www.sqlite.org/aff_short.html https://www.sqlite.org/aff_short.html https://www.sqlite.org/appfileformat.html https://www.sqlite.org/appfileformat.html
  - tyingq 5y ago
    Sqlite archive files are somewhat interesting too: https://www.sqlite.org/sqlar.html https://www.sqlite.org/sqlar.html
- le-mark 5y ago
  The history around mongrel2 using SQLite as a config file format may be instructive. Seems like there was a lot of resistance in that case.
electroly 5y ago
In one application I use sqlite to store JPEG thumbnails. That's it; there's nothing else in there. It's super handy; I specifically needed to reduce the number of files that I open and close because that's slow on Windows/NTFS. SQLite made this trivial. I could have managed a binary pack format on my own but I didn't have to.
- jbluepolarbear 5y ago
  You could have easily done that with a single file with a tail header for look ups.
  - baq 5y ago
    Easier than SQLite? With less bugs and better tests? SQLite advertises itself as an fopen replacement. Sounds like a perfect match for parent’s use case.
    - KerrAvon 5y ago
      Depends on the indexing needs, right? Something like IFF should be fine if you need a sequence of images to store that you don’t need to actually query for individually.
      - baq 5y ago
        Always, but I find it really hard to see how not using SQLite in this use case is the better option. Maybe if you need every single percentile of performance? In 99% of other cases just go with SQLite.
    - jbluepolarbear 5y ago
      Yes easier, as in time to integrate and get working correctly. I’ve used SQLite extensively over the last 10 years and yes it’s a good solution, but not a replacement for fopen. A flat file would be easier to integrate, test, and harden over SQLite. Would take about the same time as integrating SQLite into a system for the same purpose and would be easily extendable to support features as the system grows.
baq 5y ago
SQLite makes seeking in a file using a lookup table look like a complex operation from the developer’s perspective, which is absolutely a good thing unless you have a strong alternative rationale for your use case.

christophilus 5y ago

Yeah. I mentioned that in the final test where I write to a temporary file, then rename. This is much slower, probably due to an implicit fsync.

electroly 5y ago
You can disable the fsync calls in sqlite if you wanted to do a little better with this benchmark. You're also explicitly choosing the WAL in your go-sqlite3 configuration which is not at all replicated by your filesystem test. I think, honestly, that you're just going to mislead and confuse people who don't know any better with this writeup. I can write faster to /dev/null, too, but that isn't a very interesting comparison.
- HideousKojima 5y ago
  Does /dev/null support sharding?
  - pa7ch 5y ago
    Yes.
  - grogers 5y ago
    Even better, it's embarrassingly parallel!
cafxx 5y ago
I am not aware of implicit fsyncs. Can you please link to what you are referring to?
- prussian 5y ago
  Probably in reference to Ext4's[0] configurable rename-replace behavior under auto_da_alloc [0]: https://www.kernel.org/doc/Documentation/filesystems/ext4.txt https://www.kernel.org/doc/Documentation/filesystems/ext4.tx... Edit: note this isn't a universal property either, so it's still wrong: https://btrfs.wiki.kernel.org/index.php/FAQ#What_are_the_crash_guarantees_of_overwrite-by-rename.3F https://btrfs.wiki.kernel.org/index.php/FAQ#What_are_the_cra...