6 ms·

Well, here's some good reasons, your OS doesn't want you to store tons of small pieces of data in many different files. If you're storing a 100 byte record in a

Well, here's some good reasons, your OS doesn't want you to store tons of small pieces of data in many different files. If you're storing a 100 byte record in a file you're doing it wrong because: 1. Your FS has a minimum blocksize, often somewhere around 4k, you'll waste disk space with small files 2. Your OS MMAPs/FS caches pages with its page size, ~4-8k, you'll waste ram with small files. (MongoDB also makes this mistake) 3. Reading/writing data from files directly requires syscalls, a DB can cache a lot in memory and minimize syscalls. 4. Atomic transactions are hard, even with a database. Are you actually sure you know how fsync works? (see the famous ext4 fsync issue: http://blogs.gnome.org/alexl/2009/03/16/ext4-vs-fsync-my-take/ http://blogs.gnome.org/alexl/2009/03/16/ext4-vs-fsync-my-tak...) 5. Distributed operation is hard, take a look at a databases like couchbase, elasticsearch or S3, where data is replicated across machines and datacenters in a highly reliable way 6. Consistent backups on a live app are hard. Using postgres or another MVCC DB? Just perform a dump, it will be a true snapshot. 7. Do you need to coordinate writes across multiple app instances (E.G. every web-app ever), you'll have to figure out your own record locking system. Need a high-performance counter atomically incrementable from multiple clients? Easy in SQL (UPDATE foo SET counter=counter+1 WHERE id=1), try doing that in a simple way with plain-text files. The list just goes on and on and on. The kicker is even if you understand these issues, it's hard enough building a non-buggy implementation of these basics. Oh, and lastly, even if you do build your own mediocre DB, future devs will have to learn your crazy system rather than leveraging all the existing SQL knowledge they currently have.

larrys 13y ago
"future devs will have to learn your crazy system rather than leveraging all the existing SQL knowledge they currently have." Super important point that sometimes trumps everything else when considering any new technology that people often forget. Some of the first servers that I purchased were SGI running Irix. Turns out that the person who advised that didn't consider the fact that while the Sun's weren't "as good" he also didn't consider that there were so many more packages available on Sun's as well as sysadmin's as well as information (at that time) on the net. What I've found over time is that unless there is a compelling reason to do otherwise you go with the ubiquitous solution to a problem. Back in the day: Or the solution that there was an O'Reilly book on or a row of books on the bookshelf explaining (when that is essentially the knowledge base).
lysium 13y ago
I think your points are valid only for very different data scenarios. 1-2 scenario: many many small files. 3: continuously accessing your data 4: concurrent connections 5: distributed application 6: never-stopping application 7: concurrent writes to mutable data You may need a DB, if you want several of these to happen. This might be true for many applications, but for many applications this is equally not true. Most often than youd' expect, using a DB makes understanding an application _harder_ because the data in the application tends to get structured as in the database: only ids, ints, strings and maybe date, no graphs or algebraic data types such as sum types, optional types, etc.
- icebraining 13y ago
  There are plenty of databases with support for more complex types, and in any case, it's not like the filesystem understands those either, so if you want to store something to disk, you'll need some kind of translation/serialization anyway.
  - lysium 13y ago
    I'm interested in the databases that support more complex types. Pointers?
    - dkersten 13y ago
      For graphs, try a graph database like Neo4j, OrientDB, Titan, InfiniteGraph or Dex.
    - icebraining 13y ago
      Nothing very specific, I was specifically thinking about graph databases, since you mentioned them. There's also domain-specific datatypes, such as those implemented by PostGIS.
- andrewvc 13y ago
  That's a fair point, for many scenarios this stuff doesn't matter, like a video-game. However, most apps fall into either the "serialize a bunch of crap to disk" category, or the "multi-user data changes all the time category". Nearly everything in the second category works best with a proper database.
guard-of-terra 13y ago
Modern file systems can store several files' "tails" in one block, therefore they don't round file sizes to block size.
- emillon 13y ago
  Interesting. How modern are we talking exactly? More ext2-modern, XFS-modern or ZFS-modern?
  - lysium 13y ago
    ext4 http://kernelnewbies.org/Linux_3.8#head-372b38979138cf2006bd0114ae97f889f67ef46a http://kernelnewbies.org/Linux_3.8#head-372b38979138cf2006bd...
    - emillon 13y ago
      Thanks!
    - Dylan16807 13y ago
      That link talks about putting the entirety of very small files into their inodes. I don't see anything about combining the tails of multiple files in a single block.
      - lysium 13y ago
        That's true, but OP talked about storing '100 byte records' into a single file. For large files (x times block size), wasting a block for the tail is usually negligible.
      - guard-of-terra 13y ago
        I believe ReiserFS did that.
KaiserPro 13y ago
1) 100% yes 2) yup 3) Frequently read/written files will be stored in ram 4) This still affects DBs, as they are still a layer above the FS. 5) GPFS, GFS2, ceph, cleversafe, tahoe-lafs, all make this go away. 6) ZFS has snapshots. Snapshots are amazing. Granted they are not nice as SQL dump. 7) If you're doing this with files, you're in trouble. Having said that, your last point is 100% valid. Don't reinvent the wheel. SQL is good for most things. Even SQLite is super awesome for slightly busy websites(depending on how you use it).
- andrewvc 13y ago
  Agreed, but remember, syscall overhead is syscall overhead, cached in ram or not.
deleted 13y ago
[deleted]
read 13y ago
I noticed when a discussion like this comes up people separate themselves into two camps that mock each other's assumptions while both talking about the same thing. It would be nice to talk more concretely with benchmark numbers. 1. What about a database saves you disk space? Whatever it is, the filesystem can do the same thing. Perhaps you had a preconceived notion of a specific filesystem and OS in mind when you wrote this. 2. Your database MMAPs/caches pages with its custom page buffer too, and that wastes ram with small records. 3. Reading/writing data from a database requires syscalls too. An OS can cache a lot in memory and minimize disk writes. Arguably the database is duplicating the caching logic of the OS and slows things down. 4. Filesystems have atomic transactions too. You can use the flock call to lock files, and rename is atomic. It's a rather befitting hack for HackerNews which "atomically" renames a file to update it. 5, 6 and 7: You can get distribution, replication, fault-tolerance, availability, and snapshots for free out of a filesystem like GlusterFS too. (Also note SQL is crap; it's a bad serialization of a data structure.) I wonder if the real reason we have better databases than better filesystems is people didn't want to program for the kernel, which is where filesystems had typically ran. People not grasping you can store data on a filesystem is probably one of the most valuable things a new programming language would have going for it. It would be a staggering reduction in complexity if a language made data manipulation easy, perhaps at the dismay of programmers stripped of the amusing complexity of using this separate thing called a database.
- AlisdairO 13y ago
  Could you clarify on 2? I mean, there's of course a per-row overhead on a DB, but many rows get put into the same page - so I'm not sure how that's remotely as wasteful as a hypothetical filesystem that uses a minimum of a page-per-file. Of course, that's not a requirement for FS behaviour - just historically a lot of them have worked that way. > Reading/writing data from a database requires syscalls too Not for stuff that's in the DB's page cache. > Arguably the database is duplicating the caching logic of the OS and slows things down. That's a pretty biased assessment. A 'real' database like Postgres will use calls that bypass the OS page cache, so there's no duplication there. It also understands its usage patterns better, so it can do generally better than a general-purpose cache. > and rename is atomic Assuming your file system doesn't atomically give you a truncated file post-system-crash :-). > (Also note SQL is crap; it's a bad serialization of a data structure.) Agreed that SQL is not the best language, but is this an argument against set-based query languages, or just SQL specifically? Databases make some things that are really hard less hard. Managing concurrent read/writes over sets of files (say) is currently difficult. Sure, your FS could have deadlock detection, MVCC and so on built into it, but that would be turning the file system into even more of a database system than it already is. Database systems are complicated because people often want to do complicated things.

larrys 13y ago

"future devs will have to learn your crazy system rather than leveraging all the existing SQL knowledge they currently have." Super important point that sometimes trumps everything else when considering any new technology that people often forget. Some of the first servers that I purchased were SGI running Irix. Turns out that the person who advised that didn't consider the fact that while the Sun's weren't "as good" he also didn't consider that there were so many more packages available on Sun's as well as sysadmin's as well as information (at that time) on the net. What I've found over time is that unless there is a compelling reason to do otherwise you go with the ubiquitous solution to a problem. Back in the day: Or the solution that there was an O'Reilly book on or a row of books on the bookshelf explaining (when that is essentially the knowledge base).

lysium 13y ago

I think your points are valid only for very different data scenarios. 1-2 scenario: many many small files. 3: continuously accessing your data 4: concurrent connections 5: distributed application 6: never-stopping application 7: concurrent writes to mutable data You may need a DB, if you want several of these to happen. This might be true for many applications, but for many applications this is equally not true. Most often than youd' expect, using a DB makes understanding an application _harder_ because the data in the application tends to get structured as in the database: only ids, ints, strings and maybe date, no graphs or algebraic data types such as sum types, optional types, etc.

icebraining 13y ago
There are plenty of databases with support for more complex types, and in any case, it's not like the filesystem understands those either, so if you want to store something to disk, you'll need some kind of translation/serialization anyway.
- lysium 13y ago
  I'm interested in the databases that support more complex types. Pointers?
  - dkersten 13y ago
    For graphs, try a graph database like Neo4j, OrientDB, Titan, InfiniteGraph or Dex.
  - icebraining 13y ago
    Nothing very specific, I was specifically thinking about graph databases, since you mentioned them. There's also domain-specific datatypes, such as those implemented by PostGIS.
andrewvc 13y ago
That's a fair point, for many scenarios this stuff doesn't matter, like a video-game. However, most apps fall into either the "serialize a bunch of crap to disk" category, or the "multi-user data changes all the time category". Nearly everything in the second category works best with a proper database.

guard-of-terra 13y ago

Modern file systems can store several files' "tails" in one block, therefore they don't round file sizes to block size.

emillon 13y ago
Interesting. How modern are we talking exactly? More ext2-modern, XFS-modern or ZFS-modern?
- lysium 13y ago
  ext4 http://kernelnewbies.org/Linux_3.8#head-372b38979138cf2006bd0114ae97f889f67ef46a http://kernelnewbies.org/Linux_3.8#head-372b38979138cf2006bd...
  - emillon 13y ago
    Thanks!
  - Dylan16807 13y ago
    That link talks about putting the entirety of very small files into their inodes. I don't see anything about combining the tails of multiple files in a single block.
    - lysium 13y ago
      That's true, but OP talked about storing '100 byte records' into a single file. For large files (x times block size), wasting a block for the tail is usually negligible.
    - guard-of-terra 13y ago
      I believe ReiserFS did that.

KaiserPro 13y ago

1) 100% yes 2) yup 3) Frequently read/written files will be stored in ram 4) This still affects DBs, as they are still a layer above the FS. 5) GPFS, GFS2, ceph, cleversafe, tahoe-lafs, all make this go away. 6) ZFS has snapshots. Snapshots are amazing. Granted they are not nice as SQL dump. 7) If you're doing this with files, you're in trouble. Having said that, your last point is 100% valid. Don't reinvent the wheel. SQL is good for most things. Even SQLite is super awesome for slightly busy websites(depending on how you use it).

andrewvc 13y ago
Agreed, but remember, syscall overhead is syscall overhead, cached in ram or not.

deleted 13y ago

[deleted]

read 13y ago

I noticed when a discussion like this comes up people separate themselves into two camps that mock each other's assumptions while both talking about the same thing. It would be nice to talk more concretely with benchmark numbers. 1. What about a database saves you disk space? Whatever it is, the filesystem can do the same thing. Perhaps you had a preconceived notion of a specific filesystem and OS in mind when you wrote this. 2. Your database MMAPs/caches pages with its custom page buffer too, and that wastes ram with small records. 3. Reading/writing data from a database requires syscalls too. An OS can cache a lot in memory and minimize disk writes. Arguably the database is duplicating the caching logic of the OS and slows things down. 4. Filesystems have atomic transactions too. You can use the flock call to lock files, and rename is atomic. It's a rather befitting hack for HackerNews which "atomically" renames a file to update it. 5, 6 and 7: You can get distribution, replication, fault-tolerance, availability, and snapshots for free out of a filesystem like GlusterFS too. (Also note SQL is crap; it's a bad serialization of a data structure.) I wonder if the real reason we have better databases than better filesystems is people didn't want to program for the kernel, which is where filesystems had typically ran. People not grasping you can store data on a filesystem is probably one of the most valuable things a new programming language would have going for it. It would be a staggering reduction in complexity if a language made data manipulation easy, perhaps at the dismay of programmers stripped of the amusing complexity of using this separate thing called a database.

AlisdairO 13y ago
Could you clarify on 2? I mean, there's of course a per-row overhead on a DB, but many rows get put into the same page - so I'm not sure how that's remotely as wasteful as a hypothetical filesystem that uses a minimum of a page-per-file. Of course, that's not a requirement for FS behaviour - just historically a lot of them have worked that way. > Reading/writing data from a database requires syscalls too Not for stuff that's in the DB's page cache. > Arguably the database is duplicating the caching logic of the OS and slows things down. That's a pretty biased assessment. A 'real' database like Postgres will use calls that bypass the OS page cache, so there's no duplication there. It also understands its usage patterns better, so it can do generally better than a general-purpose cache. > and rename is atomic Assuming your file system doesn't atomically give you a truncated file post-system-crash :-). > (Also note SQL is crap; it's a bad serialization of a data structure.) Agreed that SQL is not the best language, but is this an argument against set-based query languages, or just SQL specifically? Databases make some things that are really hard less hard. Managing concurrent read/writes over sets of files (say) is currently difficult. Sure, your FS could have deadlock detection, MVCC and so on built into it, but that would be turning the file system into even more of a database system than it already is. Database systems are complicated because people often want to do complicated things.