4 ms·

I noticed when a discussion like this comes up people separate themselves into two camps that mock each other's assumptions while both talking about the same th

by read 13y ago

I noticed when a discussion like this comes up people separate themselves into two camps that mock each other's assumptions while both talking about the same thing. It would be nice to talk more concretely with benchmark numbers. 1. What about a database saves you disk space? Whatever it is, the filesystem can do the same thing. Perhaps you had a preconceived notion of a specific filesystem and OS in mind when you wrote this. 2. Your database MMAPs/caches pages with its custom page buffer too, and that wastes ram with small records. 3. Reading/writing data from a database requires syscalls too. An OS can cache a lot in memory and minimize disk writes. Arguably the database is duplicating the caching logic of the OS and slows things down. 4. Filesystems have atomic transactions too. You can use the flock call to lock files, and rename is atomic. It's a rather befitting hack for HackerNews which "atomically" renames a file to update it. 5, 6 and 7: You can get distribution, replication, fault-tolerance, availability, and snapshots for free out of a filesystem like GlusterFS too. (Also note SQL is crap; it's a bad serialization of a data structure.) I wonder if the real reason we have better databases than better filesystems is people didn't want to program for the kernel, which is where filesystems had typically ran. People not grasping you can store data on a filesystem is probably one of the most valuable things a new programming language would have going for it. It would be a staggering reduction in complexity if a language made data manipulation easy, perhaps at the dismay of programmers stripped of the amusing complexity of using this separate thing called a database.

AlisdairO 13y ago
Could you clarify on 2? I mean, there's of course a per-row overhead on a DB, but many rows get put into the same page - so I'm not sure how that's remotely as wasteful as a hypothetical filesystem that uses a minimum of a page-per-file. Of course, that's not a requirement for FS behaviour - just historically a lot of them have worked that way. > Reading/writing data from a database requires syscalls too Not for stuff that's in the DB's page cache. > Arguably the database is duplicating the caching logic of the OS and slows things down. That's a pretty biased assessment. A 'real' database like Postgres will use calls that bypass the OS page cache, so there's no duplication there. It also understands its usage patterns better, so it can do generally better than a general-purpose cache. > and rename is atomic Assuming your file system doesn't atomically give you a truncated file post-system-crash :-). > (Also note SQL is crap; it's a bad serialization of a data structure.) Agreed that SQL is not the best language, but is this an argument against set-based query languages, or just SQL specifically? Databases make some things that are really hard less hard. Managing concurrent read/writes over sets of files (say) is currently difficult. Sure, your FS could have deadlock detection, MVCC and so on built into it, but that would be turning the file system into even more of a database system than it already is. Database systems are complicated because people often want to do complicated things.
- read 13y ago
  A way to execute set-based queries is a major point you bring up here. It's also the most baffling. When tens of filesystems with sophisticated features completely omit what is one of the most common features used by database users, presumably it's not because filesystem designers are not capable of implementing it but of some more mundane reason. > Not for stuff that's in the DB's page cache. The database has to log what was written. It does use syscalls to write the write-ahead log. Agreed it can save read syscalls. Then again one process sending a query to another process to read something generates even more syscalls than having everything (app+db) all in the same process. You are right about 2. I'm playing devils advocate here jumping around assumptions. I could ask why you assume an OS will use a page per file, you can say Linux does this, I can say why is your OS so heavy and point to Exokernel, you can say that's side-stepping the issue. I could say user-space filesystem having its own page cache; in the end we are talking about the same thing. In the end: the less layers the better. When I look at the kernel I see something like the Berlin wall. A barrier that was there since you were born and which you never asked for. A kernel being hard to hack and monolithic is bound to push developers away, but it won't stop developers from building what they want in the end.
cafard 13y ago
"It would be a staggering reduction in complexity if a language made data manipulation easy," So, you mean, like a query language? "perhaps at the dismay of programmers stripped of the amusing complexity of using this separate thing called a database." Or facing the prospect of learning another database system, and unsure what the gain is.

by read 13y ago

AlisdairO 13y ago

Could you clarify on 2? I mean, there's of course a per-row overhead on a DB, but many rows get put into the same page - so I'm not sure how that's remotely as wasteful as a hypothetical filesystem that uses a minimum of a page-per-file. Of course, that's not a requirement for FS behaviour - just historically a lot of them have worked that way. > Reading/writing data from a database requires syscalls too Not for stuff that's in the DB's page cache. > Arguably the database is duplicating the caching logic of the OS and slows things down. That's a pretty biased assessment. A 'real' database like Postgres will use calls that bypass the OS page cache, so there's no duplication there. It also understands its usage patterns better, so it can do generally better than a general-purpose cache. > and rename is atomic Assuming your file system doesn't atomically give you a truncated file post-system-crash :-). > (Also note SQL is crap; it's a bad serialization of a data structure.) Agreed that SQL is not the best language, but is this an argument against set-based query languages, or just SQL specifically? Databases make some things that are really hard less hard. Managing concurrent read/writes over sets of files (say) is currently difficult. Sure, your FS could have deadlock detection, MVCC and so on built into it, but that would be turning the file system into even more of a database system than it already is. Database systems are complicated because people often want to do complicated things.

read 13y ago
A way to execute set-based queries is a major point you bring up here. It's also the most baffling. When tens of filesystems with sophisticated features completely omit what is one of the most common features used by database users, presumably it's not because filesystem designers are not capable of implementing it but of some more mundane reason. > Not for stuff that's in the DB's page cache. The database has to log what was written. It does use syscalls to write the write-ahead log. Agreed it can save read syscalls. Then again one process sending a query to another process to read something generates even more syscalls than having everything (app+db) all in the same process. You are right about 2. I'm playing devils advocate here jumping around assumptions. I could ask why you assume an OS will use a page per file, you can say Linux does this, I can say why is your OS so heavy and point to Exokernel, you can say that's side-stepping the issue. I could say user-space filesystem having its own page cache; in the end we are talking about the same thing. In the end: the less layers the better. When I look at the kernel I see something like the Berlin wall. A barrier that was there since you were born and which you never asked for. A kernel being hard to hack and monolithic is bound to push developers away, but it won't stop developers from building what they want in the end.

cafard 13y ago

"It would be a staggering reduction in complexity if a language made data manipulation easy," So, you mean, like a query language? "perhaps at the dismay of programmers stripped of the amusing complexity of using this separate thing called a database." Or facing the prospect of learning another database system, and unsure what the gain is.