3 ms·

> Why assume you need ACID and transactions when they're overwhelmingly likely to bring you nothing but trouble Maybe this for consumer web applications, or fo

by ekidd 5y ago

> Why assume you need ACID and transactions when they're overwhelmingly likely to bring you nothing but trouble Maybe this for consumer web applications, or for CRUD apps. But my experience with B2B and SaaS applications is that sooner or later, I always need transactions (or locks) for something. Maybe it's for the billing code. Maybe it's for batch jobs or workflow logic. And that's when I'm really happy to have a transactional database. The alternative to having transactions is setting up a Raft or Paxos server to handle distributed locks, and those require a lot more ops effort than a copy of PostgreSQL.

lmm 5y ago
Well if it's not distributed then it's a single point of failure, almost by definition. I hate maintaining a Zookeeper cluster as much as anyone, but anything that runs on a single server is not a real alternative (and if a single-server SPoF is acceptable, then just running the batch job / workflow processor / whatever it is on a single server works just as well - maybe with a pool of workers to scale out the actual processing, but a single coordinator, sort of like how Jenkins or Hadoop do it).
- ekidd 5y ago
  My most common batch/workflow usecases tend to involve a few hundred workers, and tolerance for brief downtime (especially if jobs don't fail outright). PostgreSQL works surprisingly well for something like this. It has a rich set of coordination and locking primitives, and it can usually be recovered and restarted in under 10 minutes if the server fails. Yes, connection overhead from 300 workers is a real problem, so it doesn't hurt to put a thin layer in front of it. One thing that an RDBMS buys me is flexible feature set and a solid ecosystem. Want write-ahead logs and automatic point-in-time recovery? It's out there. Need a specific locking primitive? PostgreSQL probably supports it. Etc. If I were looking at 2,000 workers, yeah, it would be time to set up a Raft server. But a good RDBMS will do the job for a surprisingly long time.
  - lmm 5y ago
    I wouldn't say "flexible" exactly - RDBMSes come with a lot of stuff, but it's all coupled in surprising ways and you have to use it the way the system expects. They remind me of those do-everything application frameworks that come with a built in web server and task queue and distributed transaction framework and what have you, and you can do these amazing demos in 10 lines of code but as soon as you want to customize the behaviour or step slightly off the beaten path you're in for a world of pain. I do see where you're coming from, but I find they're more trouble than they're worth in the long term. YMMV I guess.

by ekidd 5y ago

lmm 5y ago

Well if it's not distributed then it's a single point of failure, almost by definition. I hate maintaining a Zookeeper cluster as much as anyone, but anything that runs on a single server is not a real alternative (and if a single-server SPoF is acceptable, then just running the batch job / workflow processor / whatever it is on a single server works just as well - maybe with a pool of workers to scale out the actual processing, but a single coordinator, sort of like how Jenkins or Hadoop do it).

ekidd 5y ago
My most common batch/workflow usecases tend to involve a few hundred workers, and tolerance for brief downtime (especially if jobs don't fail outright). PostgreSQL works surprisingly well for something like this. It has a rich set of coordination and locking primitives, and it can usually be recovered and restarted in under 10 minutes if the server fails. Yes, connection overhead from 300 workers is a real problem, so it doesn't hurt to put a thin layer in front of it. One thing that an RDBMS buys me is flexible feature set and a solid ecosystem. Want write-ahead logs and automatic point-in-time recovery? It's out there. Need a specific locking primitive? PostgreSQL probably supports it. Etc. If I were looking at 2,000 workers, yeah, it would be time to set up a Raft server. But a good RDBMS will do the job for a surprisingly long time.
- lmm 5y ago
  I wouldn't say "flexible" exactly - RDBMSes come with a lot of stuff, but it's all coupled in surprising ways and you have to use it the way the system expects. They remind me of those do-everything application frameworks that come with a built in web server and task queue and distributed transaction framework and what have you, and you can do these amazing demos in 10 lines of code but as soon as you want to customize the behaviour or step slightly off the beaten path you're in for a world of pain. I do see where you're coming from, but I find they're more trouble than they're worth in the long term. YMMV I guess.