3 ms·

That's on you, it happens quite often :)

dragontamer 4y ago
I think most people know that filesystems make for poor databases. However, most people don't seem to realize that Excel spreadsheets are rather poor databases... they're the _real_ issue.
- dsr_ 4y ago
  There are lots of situations where filesystems are good databases. The problem is that relational databases are flexible in ways that filesystems are not, so when your requirements change, a relational database is likely to be able to support the change whereas the filesystem may not. Excel spreadsheets are terrible as databases and are also terrible as a file format for sending database records.
  - dragontamer 4y ago
    Filesystems already fail at step#1 of a database, which is concurrent read / concurrent writes. Its pretty obscure and arcane for the rules of read/write ordering on Linux/Windows systems and filesystems. You've got FILE* fwrites interacting with mmap regions, and locks don't exactly do what you think in Linux. In both Linux / Windows, you end up needing to create a Filesystem mutex of some kind (Linux flock, Windows Mutex). Then, and only then, is it really safe to read or write to a file.
    - whatshisface 4y ago
      That complexity only exists if you create it. The regular read and write calls are serialized by the operating system.
      - dragontamer 4y ago
        I'm sure you're familiar with the ATM example? Lets say Bob's money in an ATM is being held in your "database" (be it a file, or SQL database, or whatever). Thread#1: Depositing money -------------- int bob_balance = readfile(); bob_balance += 100; // Add 100 to Bob's account. writefile(bob_balance); Thread#2: Withdrawing money ------------- int bob_balance = readfile(); bob_balance -= 100; // Add 100 to Bob's account. writefile(bob_balance); ------------ Lets say Bob started with X-dollars. What is Bob's value of his account after Thread#1 and Thread#2 run concurrently? More than likely, Bob's account is either X, X-100, or X+100. But under the worst case scenario, a full corruption of Bob's account value could happen. If Bob's account straddles across a filesystem boundary, like across 4096 hard drive cluster or something, then the "writefile" itself could be corrupted and interleaved between the two threads.
        whatshisface 4y ago
        Most of Linux's userspace uses files as databases. There are a lot of cases where you won't ever have two of the same process running at once. One example is those that bind a specific port.
        dragontamer 4y ago
        People are talking about MongoDB, PostgreSQL, MySQL and the like in this thread of discussion. And suddenly you're talking about local userspace programs on Linux? There's a web-bias in the discussion. Database here is innately concurrent, such as when you and I submit a comment to this webpage (Hacker News), the ordering of our submissions are ill specified and effectively random. A proper database organizes the data (ex: PostgreSQL) so that its in a form that's well defined. This is true for even the smallest of web servers. Concurrency and concurrent access is fundamental to web applications today. I know PHP has access to raw files if you ever wanted them, but its definitely a dangerous proposition to be doing raw file access on web apps. There's just too much risk of data corruption.
        hot_gril 4y ago
        Well the irony of this example is that Postgres and MySQL struggle with it too because they use row-level locking by default, not serializable (full ACID). You'd be ok if you repeatedly updated a balance row as a way to lock things, but if you relied on summing withdrawal/deposit rows to determine balances as you would in a fully normalized schema, you'd get a race condition allowing Bob to overdraw. You can put the transaction in serializable mode, but it becomes very slow and non-scalable. So you have to work around this with a little denormalization or separate pending/active balances. But yes it'd be awful doing this with files.
    - dcow 4y ago
      What in the world are you doing where you need everything you described? Writing a custom cache for a FS?
      - dragontamer 4y ago
        Just a webserver. If two threads write to a file at the same time, you start getting wonky issues. You know, like if you and I make a comment at the same time here on Hacker News, the server's hard drive has to serialize the data so that all the other readers can handle it.
        dcow 4y ago
        I was thinking more about what mmap has to do with anything. Anyway it depends on the API you use. libc read/write are thread safe for instance. Obviously you still have to synchronize the higher level concept of writes if you are not writing an object all at once, but this is rather trivial to do.
    - dathinab 4y ago
      Through there are use cases (e.g. some ad-hoc special purpose corner case tooling) which: - do not need or do concurrent anything - do not need any form of transactions - do only need some simple key value(blob) store - are as ad-hoc scripts written in e.g. bash, making it just easier to access the file system for this Sure most use cases do not fit this and should not use the file system.

I think most people know that filesystems make for poor databases. However, most people don't seem to realize that Excel spreadsheets are rather poor databases... they're the _real_ issue.

dsr_ 4y ago
There are lots of situations where filesystems are good databases. The problem is that relational databases are flexible in ways that filesystems are not, so when your requirements change, a relational database is likely to be able to support the change whereas the filesystem may not. Excel spreadsheets are terrible as databases and are also terrible as a file format for sending database records.
- dragontamer 4y ago
  Filesystems already fail at step#1 of a database, which is concurrent read / concurrent writes. Its pretty obscure and arcane for the rules of read/write ordering on Linux/Windows systems and filesystems. You've got FILE* fwrites interacting with mmap regions, and locks don't exactly do what you think in Linux. In both Linux / Windows, you end up needing to create a Filesystem mutex of some kind (Linux flock, Windows Mutex). Then, and only then, is it really safe to read or write to a file.
  - whatshisface 4y ago
    That complexity only exists if you create it. The regular read and write calls are serialized by the operating system.
    - dragontamer 4y ago
      I'm sure you're familiar with the ATM example? Lets say Bob's money in an ATM is being held in your "database" (be it a file, or SQL database, or whatever). Thread#1: Depositing money -------------- int bob_balance = readfile(); bob_balance += 100; // Add 100 to Bob's account. writefile(bob_balance); Thread#2: Withdrawing money ------------- int bob_balance = readfile(); bob_balance -= 100; // Add 100 to Bob's account. writefile(bob_balance); ------------ Lets say Bob started with X-dollars. What is Bob's value of his account after Thread#1 and Thread#2 run concurrently? More than likely, Bob's account is either X, X-100, or X+100. But under the worst case scenario, a full corruption of Bob's account value could happen. If Bob's account straddles across a filesystem boundary, like across 4096 hard drive cluster or something, then the "writefile" itself could be corrupted and interleaved between the two threads.
      - whatshisface 4y ago
        Most of Linux's userspace uses files as databases. There are a lot of cases where you won't ever have two of the same process running at once. One example is those that bind a specific port.
        dragontamer 4y ago
        People are talking about MongoDB, PostgreSQL, MySQL and the like in this thread of discussion. And suddenly you're talking about local userspace programs on Linux? There's a web-bias in the discussion. Database here is innately concurrent, such as when you and I submit a comment to this webpage (Hacker News), the ordering of our submissions are ill specified and effectively random. A proper database organizes the data (ex: PostgreSQL) so that its in a form that's well defined. This is true for even the smallest of web servers. Concurrency and concurrent access is fundamental to web applications today. I know PHP has access to raw files if you ever wanted them, but its definitely a dangerous proposition to be doing raw file access on web apps. There's just too much risk of data corruption.
      - hot_gril 4y ago
        Well the irony of this example is that Postgres and MySQL struggle with it too because they use row-level locking by default, not serializable (full ACID). You'd be ok if you repeatedly updated a balance row as a way to lock things, but if you relied on summing withdrawal/deposit rows to determine balances as you would in a fully normalized schema, you'd get a race condition allowing Bob to overdraw. You can put the transaction in serializable mode, but it becomes very slow and non-scalable. So you have to work around this with a little denormalization or separate pending/active balances. But yes it'd be awful doing this with files.
  - dcow 4y ago
    What in the world are you doing where you need everything you described? Writing a custom cache for a FS?
    - dragontamer 4y ago
      Just a webserver. If two threads write to a file at the same time, you start getting wonky issues. You know, like if you and I make a comment at the same time here on Hacker News, the server's hard drive has to serialize the data so that all the other readers can handle it.
      - dcow 4y ago
        I was thinking more about what mmap has to do with anything. Anyway it depends on the API you use. libc read/write are thread safe for instance. Obviously you still have to synchronize the higher level concept of writes if you are not writing an object all at once, but this is rather trivial to do.
  - dathinab 4y ago
    Through there are use cases (e.g. some ad-hoc special purpose corner case tooling) which: - do not need or do concurrent anything - do not need any form of transactions - do only need some simple key value(blob) store - are as ad-hoc scripts written in e.g. bash, making it just easier to access the file system for this Sure most use cases do not fit this and should not use the file system.