3 ms·

I know you've acknowledged the limitations of your knowledge in your comment, but it's tremendously limiting to take an attitude of 'I don't like databases' in

I know you've acknowledged the limitations of your knowledge in your comment, but it's tremendously limiting to take an attitude of 'I don't like databases' in the presence of such limited knowledge and on the basis of a single implementation.

jacques_chester 13y ago
I noticed a few years ago that "joins suck" is usually a thin abstraction layer over "joins in MySQL < 5.6 suck"
millstone 13y ago
Maybe I was too glib. I should have said, I think that databases are overused. There's a bias in favor of databases in the comments here and in the StackOverflow page, which isn't properly tempered by negative experiences. I also see a lot of misinformation, for example, "A database is needed if you have multiple processes modifying the data." I have seen some desktop apps with terrible performance, that issue disproportionately huge amounts of disk IO, and where backtraces show SQLite functions. I conclude that using SQLite correctly in a desktop app is harder that people say. (Other RDBMSs aren't really in the running on the desktop.) I have also seen a lot of dog-slow WordPress sites, including my own. Wouldn't you agree that many of them would be faster and more robust if they used a flat-file generator like Jekyll? Mine sure was! So if databases are overused, what is the explanation? Well, look at the top voted reply: use a database and now your project is fast, ACID, fault-tolerant, can handle very large data sets, is concurrent, and can scale well! And when should you use files? The second reply answers that: it's when you "don't care about scalability or reliability." If you believe these things, it will lead you to use databases in inappropriate contexts, and make the mistake of thinking that now your I/O performance is good. If anything, it's the opposite: databases abstract I/O, so properly evaluating performance means you need to work harder to understand what's really going on.
- AlisdairO 13y ago
  I think my own answer would be that a database system is very often the right solution if you know what you're doing with them. In this situation, diagnosing performance problems is quite easy (thanks to diagnostics), concurrency is certainly easier, and scaling can be easier too - because problems like concurrent file access for writing are hard in a flat-file situation and easy on the database, and that's exactly the kind of problem you often run into when you scale. Unfortunately, when you don't know what you're doing with them, they're basically a black box where magic happens, and you don't know how to fix any problems that occur. The thing is, that applies to many things. When you don't know what you're doing with concurrent file access, you'll run into even bigger problems than you will on an RDBMS - just last week I was in fact rewriting an otherwise-competent colleague's code, because concurrent read/write to files caused massive unrecoverable corruption issues. I don't think your wordpress comparison is apples to apples - wordpress is known to be dog-slow, and if you tried similar access patterns over file systems it would be at least as bad. There's little incentive for them to fix it, though, because wordpress sites are read-mostly, and all you have to do to fix all your problems is stick one of the wordpress caches in front of it. Frankly it boggles my mind that they don't ship with a cache on by default. There's absolutely a place for no-db - if your content is static or write-once, doesn't need indexing over, and you can guarantee that those requirements won't change. In this case, there's no need at all to use a db, and it will simplify your life not to. I'm writing such a site as we speak, in fact! If you have dynamic content, well, I'd use a DBMS personally. tl;dr: understanding how to use RDBMSs will make the average software engineer's life much easier. edit: Just a note on SQLite specifically: SQLite can be slow compared to the common file system trick of write new temporary file -> rename, because the file system trick (on ext3 at least) doesn't require an fsync. It does mean that the change is not guaranteed durable, but often for browser data that's OK! An equivalent in database-land would be not writing every single transaction to the transaction log when it commits. That means that on a crash you lose everything that hasn't hit the disk yet, but you maintain consistency. Pre 3.7 SQLite this wasn't supported because SQLite didn't use write-ahead logging, which meant you had to fsync to maintain consistency. I believe in WAL mode this behaviour is easily available now though!

jacques_chester 13y ago

I noticed a few years ago that "joins suck" is usually a thin abstraction layer over "joins in MySQL < 5.6 suck"

millstone 13y ago

Maybe I was too glib. I should have said, I think that databases are overused. There's a bias in favor of databases in the comments here and in the StackOverflow page, which isn't properly tempered by negative experiences. I also see a lot of misinformation, for example, "A database is needed if you have multiple processes modifying the data." I have seen some desktop apps with terrible performance, that issue disproportionately huge amounts of disk IO, and where backtraces show SQLite functions. I conclude that using SQLite correctly in a desktop app is harder that people say. (Other RDBMSs aren't really in the running on the desktop.) I have also seen a lot of dog-slow WordPress sites, including my own. Wouldn't you agree that many of them would be faster and more robust if they used a flat-file generator like Jekyll? Mine sure was! So if databases are overused, what is the explanation? Well, look at the top voted reply: use a database and now your project is fast, ACID, fault-tolerant, can handle very large data sets, is concurrent, and can scale well! And when should you use files? The second reply answers that: it's when you "don't care about scalability or reliability." If you believe these things, it will lead you to use databases in inappropriate contexts, and make the mistake of thinking that now your I/O performance is good. If anything, it's the opposite: databases abstract I/O, so properly evaluating performance means you need to work harder to understand what's really going on.

AlisdairO 13y ago
I think my own answer would be that a database system is very often the right solution if you know what you're doing with them. In this situation, diagnosing performance problems is quite easy (thanks to diagnostics), concurrency is certainly easier, and scaling can be easier too - because problems like concurrent file access for writing are hard in a flat-file situation and easy on the database, and that's exactly the kind of problem you often run into when you scale. Unfortunately, when you don't know what you're doing with them, they're basically a black box where magic happens, and you don't know how to fix any problems that occur. The thing is, that applies to many things. When you don't know what you're doing with concurrent file access, you'll run into even bigger problems than you will on an RDBMS - just last week I was in fact rewriting an otherwise-competent colleague's code, because concurrent read/write to files caused massive unrecoverable corruption issues. I don't think your wordpress comparison is apples to apples - wordpress is known to be dog-slow, and if you tried similar access patterns over file systems it would be at least as bad. There's little incentive for them to fix it, though, because wordpress sites are read-mostly, and all you have to do to fix all your problems is stick one of the wordpress caches in front of it. Frankly it boggles my mind that they don't ship with a cache on by default. There's absolutely a place for no-db - if your content is static or write-once, doesn't need indexing over, and you can guarantee that those requirements won't change. In this case, there's no need at all to use a db, and it will simplify your life not to. I'm writing such a site as we speak, in fact! If you have dynamic content, well, I'd use a DBMS personally. tl;dr: understanding how to use RDBMSs will make the average software engineer's life much easier. edit: Just a note on SQLite specifically: SQLite can be slow compared to the common file system trick of write new temporary file -> rename, because the file system trick (on ext3 at least) doesn't require an fsync. It does mean that the change is not guaranteed durable, but often for browser data that's OK! An equivalent in database-land would be not writing every single transaction to the transaction log when it commits. That means that on a crash you lose everything that hasn't hit the disk yet, but you maintain consistency. Pre 3.7 SQLite this wasn't supported because SQLite didn't use write-ahead logging, which meant you had to fsync to maintain consistency. I believe in WAL mode this behaviour is easily available now though!