4 ms·

"Maybe READ COMMITTED is good enough or maybe people don’t know how dirty their data actually is…" I'm going to go with the latter option. And further posit th

"Maybe READ COMMITTED is good enough or maybe people don’t know how dirty their data actually is…" I'm going to go with the latter option. And further posit that on the whole most people working as "full stack" devs in our industry don't really get databases. In particular they don't get the relational model and its advantages at a fundamental level and are perfectly willing to paper over it with ORMs, statically structured "model" objects, and break their (logically connected) data up into disjoint microservices that end up doing inefficient server-side software joins by web service call and... Putting it another way, they want the DB to play second fiddle to frameworks, architectures, and techniques that operate with different approaches (usually OO & SOA) than the logical/relational modelling approach that an RDBMS is best for. And chaos and misuse is the result. That's my usual rant anyways... And, so yeah, it's also not a surprise that people aren't really "getting" transactions and concurrency. Certainly the wave of "NoSQL" stores (thankfully waning) showed this. People were willing to toss coherence in favour of scalability (often even when they didn't need it or hadn't proved that they did.) But often the "NoSQL" thing was simply a product of a mentality that went something like "I just want a data store" and "my object doesn't fit in a table", not getting what a table really is... And so we got Redis and Mongo, systems that didn't offer any performance advantages really, but looked on the surface like they'd remove complexity ... but ... And then I'd say that this general lack of enthusiasm around the relational model and ACID means that people haven't demanded the right things from their database tooling. Ok this rant is going to go on too long ... I just had my morning coffee...

wiz21c 3y ago
> its advantages at a fundamental level I used to be a dev and I wonder what you have in mind here ??? For me the two important points with a DB are: - entities connections (foreign keys), data types and a few other things are specified declaratively (so you don't have to think about enforcing them, the DB does it for you, so you never, ever forget). So I liked to have very strong DB models (no denormalization :-)) as the ultimate line of defense of my data quality. In my business, that was worth millions of euros (and politics). - transactions: Read commited was good enough in most cases (clients didn't access the same data) adn sometimes serializable was necessary (for example, toi generate ID's unique acorss the database and across all the clients)
RugnirViking 3y ago
im so glad I got taught database normal forms in high school because I sure as anything didnt learn it at university. That, in the long and short, is probably approximately where the problem lies.
turtleyacht 3y ago
Would this be a possible alternative? Careful, deliberate consideration of a handrolled data model, with plenty of prototyping up-front with Excel and the customer, playing out various scenarios.
jayd16 3y ago
You make some ok points but it's not hard to see why Redis and Mongo took off. During their hype cycle, they could cluster/shard more easily than RDBs. You could just throw hardware at a problem.
- cmrdporcupine 3y ago
  I think it's an unfortunate history that the boom in popular "high scalability" type applications happened before SSDs became commonplace. Feeling I/O bound and having to "go horizontal" to increase IOPS was something everybody felt they had to do. I remember reaching for Cassandra to solve a problem for a startup that needed to do what seemed like massive concurrent writes; the workload simply saturated what a single Postgres instance could be expected to do. But since then, SSD & nVME happened, and, well it's mind boggling. Just rough numbers. 7200 RPM HD IOPS: ~100 nVME IOPS: ~500,000 Now, one doesn't always have a properly designed storage engine & DB that can take advantage of this huge differential. But what a yawning gap. Solid state storage changed everything. Honestly, horizontal scaling is often reached for too quickly. For redundancy, I get it (but that's more a replication story.) For performance reasons? For most regular OLTP workloads for most average sized website type applications, I am not convinced these days.
marcosdumay 3y ago
Well, it really doesn't help that from the top-5 engines on his slides, only Postgres doesn't have a completely horrible performance profile for serializable transactions. So, people that try to learn about it run away quite quickly. But I'm not sure if most systems need it anyway. Personally, I remember very few times where I had to change from the default transaction model. As a rule, when people need full serializability, they create it with some other, more flexible mechanism.
- cmrdporcupine 3y ago
  Perhaps fair, but I think though this ties back to one point I was making: "And then I'd say that this general lack of enthusiasm around the relational model and ACID means that people haven't demanded the right things from their database tooling." In general there are plenty of solutions in database research and implementation. But because the broad community of developers doesn't put data first, and have a low expectation and knowledge of what a DBMS can and should do... It took us years to get to the place where Postgres is common place instead of e.g. MySQL; and back in the day "serious" folks spent tens of thousands on Oracle licenses... I guess that's progress, but there's so much more that can be done in this sphere.
RegW 3y ago
I also agree with the later option. I've been using databases for a very long time. For the past 2 decades there has been a drive by framework providers to transparently handle such things and relieve developers of any difficult decisions. Often there isn't an option or it is deeply hidden. I've arrived on jobs to find Spring applications where database transactions have been unwittingly disabled (or not enabled) and no one has realised. There is also a lack of upfront thinking about types of data and the required consistency. Some items really should be consistent and some don't need to be. In one of Martin Fowler's earlier books he describes the difference between knowledge and operational data. Knowledge data is almost like configuration - perhaps the description of a currency and number of decimal places it has. Operational data might be an invoice generated using that currency. The description of a currency changes rarely (unlike a currency rate), it could be cached a long time for use by many processes without issue. However, the invoice must be consistent, but only one process will be involved in creating it and nothing else is held up waiting for it, and once it is created - it will never change. In practice most data items lie somewhere along this spectrum. However, you choose your tool and then everthing is either ACID or NoSQL.

wiz21c 3y ago

> its advantages at a fundamental level I used to be a dev and I wonder what you have in mind here ??? For me the two important points with a DB are: - entities connections (foreign keys), data types and a few other things are specified declaratively (so you don't have to think about enforcing them, the DB does it for you, so you never, ever forget). So I liked to have very strong DB models (no denormalization :-)) as the ultimate line of defense of my data quality. In my business, that was worth millions of euros (and politics). - transactions: Read commited was good enough in most cases (clients didn't access the same data) adn sometimes serializable was necessary (for example, toi generate ID's unique acorss the database and across all the clients)

RugnirViking 3y ago

im so glad I got taught database normal forms in high school because I sure as anything didnt learn it at university. That, in the long and short, is probably approximately where the problem lies.

turtleyacht 3y ago

Would this be a possible alternative? Careful, deliberate consideration of a handrolled data model, with plenty of prototyping up-front with Excel and the customer, playing out various scenarios.

jayd16 3y ago

You make some ok points but it's not hard to see why Redis and Mongo took off. During their hype cycle, they could cluster/shard more easily than RDBs. You could just throw hardware at a problem.

cmrdporcupine 3y ago
I think it's an unfortunate history that the boom in popular "high scalability" type applications happened before SSDs became commonplace. Feeling I/O bound and having to "go horizontal" to increase IOPS was something everybody felt they had to do. I remember reaching for Cassandra to solve a problem for a startup that needed to do what seemed like massive concurrent writes; the workload simply saturated what a single Postgres instance could be expected to do. But since then, SSD & nVME happened, and, well it's mind boggling. Just rough numbers. 7200 RPM HD IOPS: ~100 nVME IOPS: ~500,000 Now, one doesn't always have a properly designed storage engine & DB that can take advantage of this huge differential. But what a yawning gap. Solid state storage changed everything. Honestly, horizontal scaling is often reached for too quickly. For redundancy, I get it (but that's more a replication story.) For performance reasons? For most regular OLTP workloads for most average sized website type applications, I am not convinced these days.

marcosdumay 3y ago

Well, it really doesn't help that from the top-5 engines on his slides, only Postgres doesn't have a completely horrible performance profile for serializable transactions. So, people that try to learn about it run away quite quickly. But I'm not sure if most systems need it anyway. Personally, I remember very few times where I had to change from the default transaction model. As a rule, when people need full serializability, they create it with some other, more flexible mechanism.

cmrdporcupine 3y ago
Perhaps fair, but I think though this ties back to one point I was making: "And then I'd say that this general lack of enthusiasm around the relational model and ACID means that people haven't demanded the right things from their database tooling." In general there are plenty of solutions in database research and implementation. But because the broad community of developers doesn't put data first, and have a low expectation and knowledge of what a DBMS can and should do... It took us years to get to the place where Postgres is common place instead of e.g. MySQL; and back in the day "serious" folks spent tens of thousands on Oracle licenses... I guess that's progress, but there's so much more that can be done in this sphere.

RegW 3y ago

I also agree with the later option. I've been using databases for a very long time. For the past 2 decades there has been a drive by framework providers to transparently handle such things and relieve developers of any difficult decisions. Often there isn't an option or it is deeply hidden. I've arrived on jobs to find Spring applications where database transactions have been unwittingly disabled (or not enabled) and no one has realised. There is also a lack of upfront thinking about types of data and the required consistency. Some items really should be consistent and some don't need to be. In one of Martin Fowler's earlier books he describes the difference between knowledge and operational data. Knowledge data is almost like configuration - perhaps the description of a currency and number of decimal places it has. Operational data might be an invoice generated using that currency. The description of a currency changes rarely (unlike a currency rate), it could be cached a long time for use by many processes without issue. However, the invoice must be consistent, but only one process will be involved in creating it and nothing else is held up waiting for it, and once it is created - it will never change. In practice most data items lie somewhere along this spectrum. However, you choose your tool and then everthing is either ACID or NoSQL.