3 ms·

Came here looking for every one to say just rewrite it in Rust like every other post, but surprisingly found someone who said just use a database for everything

Came here looking for every one to say just rewrite it in Rust like every other post, but surprisingly found someone who said just use a database for everything. Files are easier to backup, version and transparently encrypt. Try syncing a large database continuously with Google Drive, vs being able to sync new files. Files can be read by simply opening them in a text editor, just like email should. Email clients shouldn't have to connect to a database just to view your email. Linux solved authentication and permissions before MySQL.

jcranmer 7y ago
A filesystem is nothing more than a database, albeit one that is usually very crappily implemented. For example, locking a file is a joke in POSIX. On Windows, locking a file will keep other processes from being able to read or write it, but POSIX will only keep other processes from being able to lock it. And it doesn't work on NFS anyways, so you have to use lock files in practice if you care about potentially working on NFS. This of course means that backing up files while running a process that might be using them is extremely dangerous--there's no guarantee that the backup system will capture the filesystem in a consistent state. A proper database will come with its own backup utility that is capable of backing up databases without taking them offline. > Linux solved authentication and permissions before MySQL. POSIX file permissions are a joke. You get to define just three levels of permissions (user, group, and everybody), and they apply coarsely to an entire file. Database permissions can define permissions for multiple users or sets of users, and some of them will even let you specify permissions on a per-record basis. I've worked with email clients, so I know how they're implemented, and how the protocols work, and how (some) servers work. Email needs to be stored in a database; it's what users expect in their UX; it's what protocol implementations require. Building that database on top of a file system might have been best practice in 1990, but it's 2020 now, and we know that filesystem-based databases are a disaster waiting to happen, so you'd be an idiot to do so today.
- techntoke 7y ago
  > A filesystem is nothing more than a database, albeit one that is usually very crappily implemented. Then why do you still need a file system for your database files? Windows registry is a good example of how storing configurations in a database is a bad idea. > For example, locking a file is a joke in POSIX. I don't even see how that applies here, but you are talking about a bunch of different software vs a single database client used for a single purpose. There are mandatory locks if enabled and other solutions, but generally this isn't a problem unless you have unknown software accessing the same data as your program. > A proper database will come with its own backup utility that is capable of backing up databases without taking them offline. A file that is being written to hasn't been saved, therefore I would expect to only get a snapshot of the previous version. I don't see how that again applies to emails, often saved as individual files. > POSIX file permissions are a joke. You get to define just three levels of permissions (user, group, and everybody), and they apply coarsely to an entire file. There are ACLs. > Email needs to be stored in a database No it doesn't. It is best to cache metadata, but email has traditionally not been saved in a database and there are lot of benefits of saving emails as files that can be read from the file system. It is much easier to copy and sync files that store relevant information, than to take continuous versioned backups of a database that stores gigabytes of data.
  - jcranmer 7y ago
    > It is best to cache metadata And the thing you cache it in is called a database. You may choose to eschew an SQL database, or a no-SQL database, and roll your own implementation of a database, but you are building a database in some fashion. > there are lot of benefits of saving emails as files that can be read from the file system And now you're using the filesystem as key-value store, aka, database. Except since you're preserving metadata in a distinct database, now you have to maintain ACID qualities across two databases, one of which is pretty notorious for lying to you about durability. Or you can recognize that you're probably not very good at actually dealing with databases, and delegate the task to a proper database who has people on staff who actually worry about these sorts of scenarios. > generally this isn't a problem unless you have unknown software accessing the same data as your program. The most common such program is... backup software. Especially the kind that says backups are "copy all files in these directories," which I'm guessing is the kind of backup software that you advocate for. But you'll find in general that many applications have people who'll want to read your innards. If you're a browser, your cookie jar is valuable; an email client, your email storage and address book. > It is much easier to copy and sync files that store relevant information It is easier to write such an application. It is harder to make such an application correct. To give a concrete example of where things can go wrong: Suppose I'm running my email client when the backup process is running. The backup process finishes copying the contents of my mail folder, and my email client gets a new message, which it dutifully files into that folder. This creates a new file in said folder, and also updates the cached metadata file. Now the backup process indexes the metadata file. Were I to try to restore from this backup, the resulting state is inconsistent, and the message may well be gone forever.

jcranmer 7y ago

A filesystem is nothing more than a database, albeit one that is usually very crappily implemented. For example, locking a file is a joke in POSIX. On Windows, locking a file will keep other processes from being able to read or write it, but POSIX will only keep other processes from being able to lock it. And it doesn't work on NFS anyways, so you have to use lock files in practice if you care about potentially working on NFS. This of course means that backing up files while running a process that might be using them is extremely dangerous--there's no guarantee that the backup system will capture the filesystem in a consistent state. A proper database will come with its own backup utility that is capable of backing up databases without taking them offline. > Linux solved authentication and permissions before MySQL. POSIX file permissions are a joke. You get to define just three levels of permissions (user, group, and everybody), and they apply coarsely to an entire file. Database permissions can define permissions for multiple users or sets of users, and some of them will even let you specify permissions on a per-record basis. I've worked with email clients, so I know how they're implemented, and how the protocols work, and how (some) servers work. Email needs to be stored in a database; it's what users expect in their UX; it's what protocol implementations require. Building that database on top of a file system might have been best practice in 1990, but it's 2020 now, and we know that filesystem-based databases are a disaster waiting to happen, so you'd be an idiot to do so today.

techntoke 7y ago
> A filesystem is nothing more than a database, albeit one that is usually very crappily implemented. Then why do you still need a file system for your database files? Windows registry is a good example of how storing configurations in a database is a bad idea. > For example, locking a file is a joke in POSIX. I don't even see how that applies here, but you are talking about a bunch of different software vs a single database client used for a single purpose. There are mandatory locks if enabled and other solutions, but generally this isn't a problem unless you have unknown software accessing the same data as your program. > A proper database will come with its own backup utility that is capable of backing up databases without taking them offline. A file that is being written to hasn't been saved, therefore I would expect to only get a snapshot of the previous version. I don't see how that again applies to emails, often saved as individual files. > POSIX file permissions are a joke. You get to define just three levels of permissions (user, group, and everybody), and they apply coarsely to an entire file. There are ACLs. > Email needs to be stored in a database No it doesn't. It is best to cache metadata, but email has traditionally not been saved in a database and there are lot of benefits of saving emails as files that can be read from the file system. It is much easier to copy and sync files that store relevant information, than to take continuous versioned backups of a database that stores gigabytes of data.
- jcranmer 7y ago
  > It is best to cache metadata And the thing you cache it in is called a database. You may choose to eschew an SQL database, or a no-SQL database, and roll your own implementation of a database, but you are building a database in some fashion. > there are lot of benefits of saving emails as files that can be read from the file system And now you're using the filesystem as key-value store, aka, database. Except since you're preserving metadata in a distinct database, now you have to maintain ACID qualities across two databases, one of which is pretty notorious for lying to you about durability. Or you can recognize that you're probably not very good at actually dealing with databases, and delegate the task to a proper database who has people on staff who actually worry about these sorts of scenarios. > generally this isn't a problem unless you have unknown software accessing the same data as your program. The most common such program is... backup software. Especially the kind that says backups are "copy all files in these directories," which I'm guessing is the kind of backup software that you advocate for. But you'll find in general that many applications have people who'll want to read your innards. If you're a browser, your cookie jar is valuable; an email client, your email storage and address book. > It is much easier to copy and sync files that store relevant information It is easier to write such an application. It is harder to make such an application correct. To give a concrete example of where things can go wrong: Suppose I'm running my email client when the backup process is running. The backup process finishes copying the contents of my mail folder, and my email client gets a new message, which it dutifully files into that folder. This creates a new file in said folder, and also updates the cached metadata file. Now the backup process indexes the metadata file. Were I to try to restore from this backup, the resulting state is inconsistent, and the message may well be gone forever.