5 ms·

As someone who has spent time consulting for different kinds of "we do stuff on the internet" companies, I can confidently say that this is a premature optimiza

As someone who has spent time consulting for different kinds of "we do stuff on the internet" companies, I can confidently say that this is a premature optimization for 99.999% of the companies/projects. The companies simply don't have enough IO traffic to need it. Is it cool? Sure. Is it sexy? Maybe. Is it needed? Nope. Do boring stuff. Use files. If you send your small files over the internet via web and you need to squeeze additional performance, do boring stuff again: use Varnish.

dana321 7y ago
First thing you can do is just add more RAM, any free ram will be used to cache disk access. Varnish will decrease the latency only if you are using some scripting between the file and its access.
- notyourday 7y ago
  If you are at the point where you need to handle more IO, throwing additional memory onto a file origin will not in my experience do much because filesystems's idea of what to cache is vastly different from user's requests. Not doing any disk IO for hot files is vastly preferable than to doing some. This has been demonstrated in spades at LiveJournal which extensively stuffed files into memcache before Varnish became a thing.
fvdessen 7y ago
> Do boring stuff. Use files. I have the opposite experience. Files are far from 'boring', they have too many gotchas to count. Putting files in the db is for many use-cases actually easier than putting them on the filesystem. By putting all data in the db you have less moving parts, transactions, referential integrity, well defined behaviours, etc. The speed increase is just a nice side effect.
- notyourday 7y ago
  > By putting all data in the db you have less moving parts, transactions, referential integrity, well defined behaviours, etc If you are doing it for 100k files and below you are using a Ferrari to pick up eggs in a corner store. If you are using it at 100k files and above, you are using a Ferrari to move a pile of paving stones one by one.
  - derefr 7y ago
    Show me a filesystem that can efficiently hold onto all the inode information for a blockchain represented as files and directories. (Hint: not even LevelDB can hold onto all the trie information efficiently; solutions are being sought that pack things tighter than LevelDB.)
    - notyourday 7y ago
      You don't need to do it efficiently. You are optimizing for non-existent problem. Here's what you actually need to optimize for: you have a hundred million files. The are all somehow reachable via http://origin/someuniqueurl http://origin/someuniqueurl. You have multiple copies because you aren't an idiot and you know that users actually hate either losing or getting corrupted files back. You have fingerprints associated with every copy. Something happens and you can't reach part of the tree or you are getting incorrect hashes ( which you know because your origin computes the hashes every time someone requests and one and if those hashes don't match it triggers a re-request to a backup copy ). And now you need to (a) ensure you still have the needed protection factor ( you you went from 3x to 2x as one copy is dead ) for all affected files (b) minimize the time needed to remove the "thing" ( probably a disk or a node ) that caused the failure. (c) minimize the cost of ensuring (a) and (b) People spent lots of time figuring out very clever ways of doing it via DB. It neither scaled nor worked well when real life (i.e. disk crashes/nodes going away/bad data was returned) happened. The only thing that you really need to know is (a) where the data and its copies are stored. (b) what is the hash of the data that is stored there (c) how fast you can recover the information about where that data is stored in event of the issue with whatever the system that is used to keep track between (a) and (b). (d) preferably you also want to know what other data objects ( files ) could be affected when some of the other objects are misbehaving ( if you know a file at nodeA/volumeB had a fingerprint Ykntr8H8pL9PyAtCwdw/CB5tToXTPf55+hKSZb0uhV0 when it was written and now for some reason it says its fingerprint is sw0SI4jkU5VVk6CH4oPYtwx+bK2hIlrw8hVM7i9zmNk while the rest of the copies are saying their fingerprint is still Ykntr8H8pL9PyAtCwdw/CB5tToXTPf55+hKSZb0uhV0 you could decide that you want to trash the nodeX/volumeB because you no longer trust it.
      - 7y ago
pdimitar 7y ago
The main added value of sqlite3 when I work with it is mostly compartmentalisation of your data in one DB and one file. You not only store files as blobs, you can also store a lot of metadata on top of them, and then, when your needs inevitably grow, start using other SQL tables for the rest of your application configuration or state. And it's all in one neat tidy package. For the scenario you describe I'd generate an nginx container -- or an Ansible script -- by pulling data from a sqlite3 DB. Technically your approach and mine are not mutually exclusive at all. And, as a sibling commenter said, the speedy nature of sqlite3 is just a bonus. It's not always the chased-after feature (although with WAL and a few other flags enabled, sqlite3 is damn fast and not many DB engines can compete with it).
- notyourday 7y ago
  > You not only store files as blobs, you can also store a lot of metadata on top of them, and then, when your needs inevitably grow, start using other SQL tables for the rest of your application configuration or state. And it's all in one neat tidy package That's a great solution in search of a problem for a small dataset. For a large dataset it is a terrible solution, because disks fail and bit-rot happens. Ability to easily restore a tiny subset of data ( single file or multiple files ) using standard commands ( cp/rsync/etc ) is highly underappreciated.
  - pdimitar 7y ago
    Why of course. This entire discussion should always be complemented with the sqlite3's intended use cases. Even the authors admit it's not every app. So yes, definitely reach for a richer DB system when your needs grow beyond the point where sqlite3's weaknesses start to show. I believe the discussion in the entire HN thread is on the general topic of "is sqlite3 really that good for small-to-mid-sized projects", to which I believe the answer is definitely a yes.
ysleepy 7y ago
While I agree with the sentiment, using the unix file API is wrought with peril if you try to do it properly. https://www.usenix.org/system/files/conference/osdi14/osdi14-paper-pillai.pdf https://www.usenix.org/system/files/conference/osdi14/osdi14... sqlite does a good job to provide expected consistency even in the face of power loss. Half overwritten files or causal ordering violations can occur if you do not call fsync at the appropriate places.
- notyourday 7y ago
  Sorry, that's junk peddled by people who have no idea how often disks become "unhealthy" in the most spectacular way. Bitrot is real. Disks returning funky data is real. Sqlite does not even have online remote backup that would allow one to easily fix/tweak a portion of a database. When you start storing a few TB of data in every blob you are going to learn some very painful lessons that grey beards learned thirty years ago, except that you will learn them at a real misery producing scale. Is it an efficient way of querying structured data that can easily be blown away/recreated? Absolutely. Storing bytestreams that should not change in SQLite? That's insanity. Edgy and hipster-esque insanity but insanity none the less.

dana321 7y ago

First thing you can do is just add more RAM, any free ram will be used to cache disk access. Varnish will decrease the latency only if you are using some scripting between the file and its access.

notyourday 7y ago
If you are at the point where you need to handle more IO, throwing additional memory onto a file origin will not in my experience do much because filesystems's idea of what to cache is vastly different from user's requests. Not doing any disk IO for hot files is vastly preferable than to doing some. This has been demonstrated in spades at LiveJournal which extensively stuffed files into memcache before Varnish became a thing.

fvdessen 7y ago

> Do boring stuff. Use files. I have the opposite experience. Files are far from 'boring', they have too many gotchas to count. Putting files in the db is for many use-cases actually easier than putting them on the filesystem. By putting all data in the db you have less moving parts, transactions, referential integrity, well defined behaviours, etc. The speed increase is just a nice side effect.

notyourday 7y ago
> By putting all data in the db you have less moving parts, transactions, referential integrity, well defined behaviours, etc If you are doing it for 100k files and below you are using a Ferrari to pick up eggs in a corner store. If you are using it at 100k files and above, you are using a Ferrari to move a pile of paving stones one by one.
- derefr 7y ago
  Show me a filesystem that can efficiently hold onto all the inode information for a blockchain represented as files and directories. (Hint: not even LevelDB can hold onto all the trie information efficiently; solutions are being sought that pack things tighter than LevelDB.)
  - notyourday 7y ago
    You don't need to do it efficiently. You are optimizing for non-existent problem. Here's what you actually need to optimize for: you have a hundred million files. The are all somehow reachable via http://origin/someuniqueurl http://origin/someuniqueurl. You have multiple copies because you aren't an idiot and you know that users actually hate either losing or getting corrupted files back. You have fingerprints associated with every copy. Something happens and you can't reach part of the tree or you are getting incorrect hashes ( which you know because your origin computes the hashes every time someone requests and one and if those hashes don't match it triggers a re-request to a backup copy ). And now you need to (a) ensure you still have the needed protection factor ( you you went from 3x to 2x as one copy is dead ) for all affected files (b) minimize the time needed to remove the "thing" ( probably a disk or a node ) that caused the failure. (c) minimize the cost of ensuring (a) and (b) People spent lots of time figuring out very clever ways of doing it via DB. It neither scaled nor worked well when real life (i.e. disk crashes/nodes going away/bad data was returned) happened. The only thing that you really need to know is (a) where the data and its copies are stored. (b) what is the hash of the data that is stored there (c) how fast you can recover the information about where that data is stored in event of the issue with whatever the system that is used to keep track between (a) and (b). (d) preferably you also want to know what other data objects ( files ) could be affected when some of the other objects are misbehaving ( if you know a file at nodeA/volumeB had a fingerprint Ykntr8H8pL9PyAtCwdw/CB5tToXTPf55+hKSZb0uhV0 when it was written and now for some reason it says its fingerprint is sw0SI4jkU5VVk6CH4oPYtwx+bK2hIlrw8hVM7i9zmNk while the rest of the copies are saying their fingerprint is still Ykntr8H8pL9PyAtCwdw/CB5tToXTPf55+hKSZb0uhV0 you could decide that you want to trash the nodeX/volumeB because you no longer trust it.
    - 7y ago

pdimitar 7y ago

The main added value of sqlite3 when I work with it is mostly compartmentalisation of your data in one DB and one file. You not only store files as blobs, you can also store a lot of metadata on top of them, and then, when your needs inevitably grow, start using other SQL tables for the rest of your application configuration or state. And it's all in one neat tidy package. For the scenario you describe I'd generate an nginx container -- or an Ansible script -- by pulling data from a sqlite3 DB. Technically your approach and mine are not mutually exclusive at all. And, as a sibling commenter said, the speedy nature of sqlite3 is just a bonus. It's not always the chased-after feature (although with WAL and a few other flags enabled, sqlite3 is damn fast and not many DB engines can compete with it).

notyourday 7y ago
> You not only store files as blobs, you can also store a lot of metadata on top of them, and then, when your needs inevitably grow, start using other SQL tables for the rest of your application configuration or state. And it's all in one neat tidy package That's a great solution in search of a problem for a small dataset. For a large dataset it is a terrible solution, because disks fail and bit-rot happens. Ability to easily restore a tiny subset of data ( single file or multiple files ) using standard commands ( cp/rsync/etc ) is highly underappreciated.
- pdimitar 7y ago
  Why of course. This entire discussion should always be complemented with the sqlite3's intended use cases. Even the authors admit it's not every app. So yes, definitely reach for a richer DB system when your needs grow beyond the point where sqlite3's weaknesses start to show. I believe the discussion in the entire HN thread is on the general topic of "is sqlite3 really that good for small-to-mid-sized projects", to which I believe the answer is definitely a yes.

ysleepy 7y ago

While I agree with the sentiment, using the unix file API is wrought with peril if you try to do it properly. https://www.usenix.org/system/files/conference/osdi14/osdi14-paper-pillai.pdf https://www.usenix.org/system/files/conference/osdi14/osdi14... sqlite does a good job to provide expected consistency even in the face of power loss. Half overwritten files or causal ordering violations can occur if you do not call fsync at the appropriate places.

notyourday 7y ago
Sorry, that's junk peddled by people who have no idea how often disks become "unhealthy" in the most spectacular way. Bitrot is real. Disks returning funky data is real. Sqlite does not even have online remote backup that would allow one to easily fix/tweak a portion of a database. When you start storing a few TB of data in every blob you are going to learn some very painful lessons that grey beards learned thirty years ago, except that you will learn them at a real misery producing scale. Is it an efficient way of querying structured data that can easily be blown away/recreated? Absolutely. Storing bytestreams that should not change in SQLite? That's insanity. Edgy and hipster-esque insanity but insanity none the less.