3 ms·

Good luck with getting rid of your steenking database, when it comes to the real world. Like figuring out a reasonable access path between tables when you're d

Good luck with getting rid of your steenking database, when it comes to the real world. Like figuring out a reasonable access path between tables when you're dealing with billions of rows Like guaranteeing consistency between your relations. While the join command sounds useful (I didn't know it and I'll look into it) it's not really a replacement for all the constraints, which a database provides to ensure that you don't fubar your data Backup, recovery? Just hope to you really grab everything. No only the data, but also your carefully crafted queries. Sounds like a bummer to figure them out anew Implementing ACID also sounds like quite a challenge A database is not just a shipping container, where you cram in your data and provides a few tools to organize it. There's a lot more under the hood, which is just not supported in plain old files.

wallstprog 9y ago
Umm, the title was meant to be, like, a joke?
haney 9y ago
You're assuming quite a lot about the use case. I personally don't have a problem that's solved by this particular utility, but it's not uncommon for our analytics team to use pandas to join CSV files on their way into our data warehouse (or to otherwise reshape small datasets). I could definitely see the utility in plain text relational tools.
richman777 9y ago
You didn't read the article did you?
Arizhel 9y ago
I've got a question about databases: why don't they use raw disk partitions instead of files? If you think about it, a modern disk filesystem is really a form of database (but not the relational kind): various kinds of data (filename, date, access time, etc., and finally the actual data) are stored in a schema on the disk (which itself is abstracted by the disk controller) in a way that it can be looked up and retrieved quickly. Most modern databases I've heard of use regular filesystems to store their data, so it's a database using another database to store data. Wouldn't it be faster to eliminate the filesystem altogether, and just have the SQL database go directly to the disk, just like Linux swap partitions do? After a little googling, it does seem that Oracle used to do this (but no more), and MySQL's InnoDB still offers this. Others claim that there's no advantage to this because the kernel handles dealing with raw devices better than the DB does, but I think it's more likely that the DBs just don't have enough development resources to try doing it better; filesystems are necessarily going to incur a certain amount of overhead, so it only makes sense that theoretically you could go directly to the raw device and cut out some of that overhead, though you would be replicating some of what the kernel's filesystem driver is doing, but in a way that's more optimal for the DB.

wallstprog 9y ago

Umm, the title was meant to be, like, a joke?

haney 9y ago

You're assuming quite a lot about the use case. I personally don't have a problem that's solved by this particular utility, but it's not uncommon for our analytics team to use pandas to join CSV files on their way into our data warehouse (or to otherwise reshape small datasets). I could definitely see the utility in plain text relational tools.

richman777 9y ago

You didn't read the article did you?

Arizhel 9y ago

I've got a question about databases: why don't they use raw disk partitions instead of files? If you think about it, a modern disk filesystem is really a form of database (but not the relational kind): various kinds of data (filename, date, access time, etc., and finally the actual data) are stored in a schema on the disk (which itself is abstracted by the disk controller) in a way that it can be looked up and retrieved quickly. Most modern databases I've heard of use regular filesystems to store their data, so it's a database using another database to store data. Wouldn't it be faster to eliminate the filesystem altogether, and just have the SQL database go directly to the disk, just like Linux swap partitions do? After a little googling, it does seem that Oracle used to do this (but no more), and MySQL's InnoDB still offers this. Others claim that there's no advantage to this because the kernel handles dealing with raw devices better than the DB does, but I think it's more likely that the DBs just don't have enough development resources to try doing it better; filesystems are necessarily going to incur a certain amount of overhead, so it only makes sense that theoretically you could go directly to the raw device and cut out some of that overhead, though you would be replicating some of what the kernel's filesystem driver is doing, but in a way that's more optimal for the DB.