3 ms·

That complexity only exists if you create it. The regular read and write calls are serialized by the operating system.

dragontamer 4y ago
I'm sure you're familiar with the ATM example? Lets say Bob's money in an ATM is being held in your "database" (be it a file, or SQL database, or whatever). Thread#1: Depositing money -------------- int bob_balance = readfile(); bob_balance += 100; // Add 100 to Bob's account. writefile(bob_balance); Thread#2: Withdrawing money ------------- int bob_balance = readfile(); bob_balance -= 100; // Add 100 to Bob's account. writefile(bob_balance); ------------ Lets say Bob started with X-dollars. What is Bob's value of his account after Thread#1 and Thread#2 run concurrently? More than likely, Bob's account is either X, X-100, or X+100. But under the worst case scenario, a full corruption of Bob's account value could happen. If Bob's account straddles across a filesystem boundary, like across 4096 hard drive cluster or something, then the "writefile" itself could be corrupted and interleaved between the two threads.
- whatshisface 4y ago
  Most of Linux's userspace uses files as databases. There are a lot of cases where you won't ever have two of the same process running at once. One example is those that bind a specific port.
  - dragontamer 4y ago
    People are talking about MongoDB, PostgreSQL, MySQL and the like in this thread of discussion. And suddenly you're talking about local userspace programs on Linux? There's a web-bias in the discussion. Database here is innately concurrent, such as when you and I submit a comment to this webpage (Hacker News), the ordering of our submissions are ill specified and effectively random. A proper database organizes the data (ex: PostgreSQL) so that its in a form that's well defined. This is true for even the smallest of web servers. Concurrency and concurrent access is fundamental to web applications today. I know PHP has access to raw files if you ever wanted them, but its definitely a dangerous proposition to be doing raw file access on web apps. There's just too much risk of data corruption.
- hot_gril 4y ago
  Well the irony of this example is that Postgres and MySQL struggle with it too because they use row-level locking by default, not serializable (full ACID). You'd be ok if you repeatedly updated a balance row as a way to lock things, but if you relied on summing withdrawal/deposit rows to determine balances as you would in a fully normalized schema, you'd get a race condition allowing Bob to overdraw. You can put the transaction in serializable mode, but it becomes very slow and non-scalable. So you have to work around this with a little denormalization or separate pending/active balances. But yes it'd be awful doing this with files.
  - kortilla 4y ago
    They don’t “struggle” with it at all. You presented two approaches that work fine (a balance row or serialized transactions). It can also be done with monotonic “account transaction” counters with unique constraints to prevent the concurrent inserts of withdrawals. It’s trivially solvable and it never results in complete DB corruption by getting the ordering wrong.
    - hot_gril 4y ago
      Serialized transactions don't work fine. That's one thing Postgres itself struggles with. Your xacts will conflict and both fail, and you might even exceed queue limits as you retry. It's just not optimized for serializable xacts, which is a conscious decision because you inevitably sacrifice too much performance with it. The working approaches aren't so obvious to users unfamiliar with the details of Postgres. People think they have full ACID, and it's not apparent that rows and uniqueness constraints are magically transaction-safe while aggregates are not. If you're following the usual advice of schema normalization, you will get something with race conditions. Postgres is still my favorite DBMS regardless. Maybe there can be a clearer way to advise users.
      - kortilla 4y ago
        > Serialized transactions don't work fine. That's one thing Postgres itself struggles with. Your xacts will conflict and both fail, and you might even exceed queue limits as you retry. Of course they work fine. You’re getting hung up in optimizing for scale when you’re already way beyond solving the two threads one file issue.

I'm sure you're familiar with the ATM example? Lets say Bob's money in an ATM is being held in your "database" (be it a file, or SQL database, or whatever). Thread#1: Depositing money -------------- int bob_balance = readfile(); bob_balance += 100; // Add 100 to Bob's account. writefile(bob_balance); Thread#2: Withdrawing money ------------- int bob_balance = readfile(); bob_balance -= 100; // Add 100 to Bob's account. writefile(bob_balance); ------------ Lets say Bob started with X-dollars. What is Bob's value of his account after Thread#1 and Thread#2 run concurrently? More than likely, Bob's account is either X, X-100, or X+100. But under the worst case scenario, a full corruption of Bob's account value could happen. If Bob's account straddles across a filesystem boundary, like across 4096 hard drive cluster or something, then the "writefile" itself could be corrupted and interleaved between the two threads.

whatshisface 4y ago
Most of Linux's userspace uses files as databases. There are a lot of cases where you won't ever have two of the same process running at once. One example is those that bind a specific port.
- dragontamer 4y ago
  People are talking about MongoDB, PostgreSQL, MySQL and the like in this thread of discussion. And suddenly you're talking about local userspace programs on Linux? There's a web-bias in the discussion. Database here is innately concurrent, such as when you and I submit a comment to this webpage (Hacker News), the ordering of our submissions are ill specified and effectively random. A proper database organizes the data (ex: PostgreSQL) so that its in a form that's well defined. This is true for even the smallest of web servers. Concurrency and concurrent access is fundamental to web applications today. I know PHP has access to raw files if you ever wanted them, but its definitely a dangerous proposition to be doing raw file access on web apps. There's just too much risk of data corruption.
hot_gril 4y ago
Well the irony of this example is that Postgres and MySQL struggle with it too because they use row-level locking by default, not serializable (full ACID). You'd be ok if you repeatedly updated a balance row as a way to lock things, but if you relied on summing withdrawal/deposit rows to determine balances as you would in a fully normalized schema, you'd get a race condition allowing Bob to overdraw. You can put the transaction in serializable mode, but it becomes very slow and non-scalable. So you have to work around this with a little denormalization or separate pending/active balances. But yes it'd be awful doing this with files.
- kortilla 4y ago
  They don’t “struggle” with it at all. You presented two approaches that work fine (a balance row or serialized transactions). It can also be done with monotonic “account transaction” counters with unique constraints to prevent the concurrent inserts of withdrawals. It’s trivially solvable and it never results in complete DB corruption by getting the ordering wrong.
  - hot_gril 4y ago
    Serialized transactions don't work fine. That's one thing Postgres itself struggles with. Your xacts will conflict and both fail, and you might even exceed queue limits as you retry. It's just not optimized for serializable xacts, which is a conscious decision because you inevitably sacrifice too much performance with it. The working approaches aren't so obvious to users unfamiliar with the details of Postgres. People think they have full ACID, and it's not apparent that rows and uniqueness constraints are magically transaction-safe while aggregates are not. If you're following the usual advice of schema normalization, you will get something with race conditions. Postgres is still my favorite DBMS regardless. Maybe there can be a clearer way to advise users.
    - kortilla 4y ago
      > Serialized transactions don't work fine. That's one thing Postgres itself struggles with. Your xacts will conflict and both fail, and you might even exceed queue limits as you retry. Of course they work fine. You’re getting hung up in optimizing for scale when you’re already way beyond solving the two threads one file issue.