8 ms·

No it doesn’t. They scale amazingly well if you throw money at the problem. Most people never get there. When you do you will know. I’ve been there. When you’re

by hughrr 5y ago

No it doesn’t. They scale amazingly well if you throw money at the problem. Most people never get there. When you do you will know. I’ve been there. When you’re spending $3 million on hardware and licenses a year you either have a viable business or fucked up badly. That’s the real decider. The answer is to start siloing customers or application concerns out into separate clusters depending on your operating model. If you spent the last decade writing shitty cross database SQL then you’ll have to silo by customer. That’s the only real constraint.

tremon 5y ago
I'd phrase it differently: the database is the litmus test of your supposed "good idea". If your good idea doesn't survive referential integrity or even data consistency, your idea doesn't deserve the label "good". Of course, the next "good idea" is to sacrifice data integrity in the name of performance. That can work, but usually it's just a technical form of growth hacking. Sacrificing data integrity without understanding the problem space is a disaster waiting to happen (but luckily, it may not happen on your watch so you can continue hacking).
- hughrr 5y ago
  Very good points there. I have directly experienced the problems associated with blindly assuming that you can write your own consistency guarantees in Redis. I didn't do it but I had to write some tooling to rewrite the AOF to get some data back...
- HWR_14 5y ago
  I'm sorry, are you saying that a good idea should survive the loss of referential integrity or data consistency? I don't feel that's what you mean, and it's probably my fault for misreading your comment.
  - erik_seaberg 5y ago
    I read it the other way: if abandoning correctness is the only way to make your idea feasible, your idea kinda sucks.
  - tremon 5y ago
    Yeah, I probably could have phrased that better. The sibling comment is correct, I meant that if your idea requires abandoning referential integrity or data consistency, it's probably not a good idea. Since it's the database that actually enforces those constraints, it may seem that the database causes the problem. But in most cases, the problem is the data model or the idea itself.
    - HWR_14 5y ago
      I see. I mean, one of my ideas requires many transactions and records that require data consistency. I have no idea how to make sure it can scale to Internet scale. Do you know any good resources? Or is it just "get mysql, pay thousands a month for either a cloud provider or colocated hardware"
      - xemdetia 5y ago
        I would say that most resources would be a mixed bag in most situations but much of the advice skips over something very basic that experienced people tend to assume you know: you are always going to be limited by the number of times you can actually sync a write to a disk and/or how fast you can read back from the disk and these are the main upper bounds. One mistake I see inexperienced engineers make in modern environments is not realizing that in their particular cloud instance they are explicitly or implicitly using the cloud vendor's network storage and they are entirely hamstrung on that. The other is not configuring the database to use all the RAM- but those are usually covered by the existing tuning guides. The next from a design perspective is to leave natural places to shard/split the data up so you could represent the data consistently on multiple systems. Sometimes it's pretty bonehead things like your database tables related to login are affecting your critical path transactions and/or also tied to the referential integrity of your application data in ways that are hard to split apart. It goes back to the IO thing where if you are reading and writing login or UI traffic you sure as heck are not writing some other critical time sensitive transaction especially if it's to the same disk. The other obvious thing is to be trying to run analytic queries (aka look at many records) on a database tuned for mass INSERT/write traffic and having no plan to do that with a different resource. If you are still worried about it I would suggest just prototyping it using a currently modern API/RPC server in front of whatever datastore you come up with. As systems have gotten more distributed the straight line speed of connecting to the database directly makes less and less sense in my opinion and having an API server you control allows you to be clever later in ways that might not be obvious today (especially keeping application-aware counters so that you can see what your application is actually doing as query-style logs for databases are extremely expensive in performance). Either way if you sit down and consider the performance characteristics of the underlying system, think about each class of transactions are you considering from your application in a write/read context (hopefully from measured data), and using tools like EXPLAIN you'll quickly start finding what pieces of the puzzle might need to move to start scaling. The answer also might just be some parts of your application would be better served something other than a database like a message queue, a more straightforward ISAM or key-value store, or depending on your situation just flat files written onto the fastest local storage you can get to then be later transformed and loaded into a database sometime later.
Ozzie_osman 5y ago
> The answer is to start siloing customers or application concerns out into separate clusters depending on your operating model. This is such an underrated solution (siloing or sharding your data in some way). I think people don't do it because: 1. The tooling doesn't make it super-easy (e.g. good luck sharding Postgres unless you're willing to pay for Citus) 2. "Trendy" companies in the past decade have been network-type products (social networks, etc), where the structure of the data makes it much harder to silo (people need to follow each other, interact with each other's content, etc) 3. We as an industry took a several-year detour over to NoSQL land as a promised solution to scalability. Life would be a lot easier if you could say something like: * I want a Postgres node. * I'm happy to shard my data by some key (customerId, city, etc) and am willing to accept responsibility for thinking through that sharding key. * My application has some logic that easily knows which DB to read/write from depending on shard key. * There's some small amount of "global application data" that might need to live on a single node.
- simonw 5y ago
  For your "life would be easier" version, isn't that pretty much what Citus does? And Citus is open source so paying for the managed version is optional.
  - cellis 5y ago
    saying "Citus is open source" doesn't mean its as simple as running `yarn install citus; yarn run citus`. Its is much much more complicated than typical open source. You pay for support for a very good reason.
- rad_gruchalski 5y ago
  > The tooling doesn't make it super-easy (e.g. good luck sharding Postgres unless you're willing to pay for Citus) There’s YugabyteDB.
- seanhunter 5y ago
  Sharding by customer has a lot of benefits for b2b applications also. You can run each customer in a seperate instance in their own vpc if you want to, and guarantee to your customers that their data is never commingled with another customer. This makes a lot of sense if you have relatively low number relatively high value customers. It radically reduces the infosec/screw up blast radius. For example it means however badly you screw up it is just about impossible to accidentally show data from customer A to a user from customer B. You can let customers specify region/availability zone and bring their own keys for encryption making lots of enterprise security compliance things easier (for regulated use cases). There is no one true way of scaling that suits all possible use cases and as engineers, we should think for ourselves and build a solution that works well in our situation, rather than thinking you have to do a certain thing because that's what google/facebook etc do. Their scaling problems are most likely really different to yours.
hobs 5y ago
I just automatically silo by customer because I am so scarred by so much shitty SQL written by ??? that it just make sense for privacy reasons alone. Row based permissions are not great imo, either performance is crap or you have some weird bug.
- hughrr 5y ago
  Totally agree there as well.

by hughrr 5y ago

tremon 5y ago

I'd phrase it differently: the database is the litmus test of your supposed "good idea". If your good idea doesn't survive referential integrity or even data consistency, your idea doesn't deserve the label "good". Of course, the next "good idea" is to sacrifice data integrity in the name of performance. That can work, but usually it's just a technical form of growth hacking. Sacrificing data integrity without understanding the problem space is a disaster waiting to happen (but luckily, it may not happen on your watch so you can continue hacking).

hughrr 5y ago
Very good points there. I have directly experienced the problems associated with blindly assuming that you can write your own consistency guarantees in Redis. I didn't do it but I had to write some tooling to rewrite the AOF to get some data back...
HWR_14 5y ago
I'm sorry, are you saying that a good idea should survive the loss of referential integrity or data consistency? I don't feel that's what you mean, and it's probably my fault for misreading your comment.
- erik_seaberg 5y ago
  I read it the other way: if abandoning correctness is the only way to make your idea feasible, your idea kinda sucks.
- tremon 5y ago
  Yeah, I probably could have phrased that better. The sibling comment is correct, I meant that if your idea requires abandoning referential integrity or data consistency, it's probably not a good idea. Since it's the database that actually enforces those constraints, it may seem that the database causes the problem. But in most cases, the problem is the data model or the idea itself.
  - HWR_14 5y ago
    I see. I mean, one of my ideas requires many transactions and records that require data consistency. I have no idea how to make sure it can scale to Internet scale. Do you know any good resources? Or is it just "get mysql, pay thousands a month for either a cloud provider or colocated hardware"
    - xemdetia 5y ago
      I would say that most resources would be a mixed bag in most situations but much of the advice skips over something very basic that experienced people tend to assume you know: you are always going to be limited by the number of times you can actually sync a write to a disk and/or how fast you can read back from the disk and these are the main upper bounds. One mistake I see inexperienced engineers make in modern environments is not realizing that in their particular cloud instance they are explicitly or implicitly using the cloud vendor's network storage and they are entirely hamstrung on that. The other is not configuring the database to use all the RAM- but those are usually covered by the existing tuning guides. The next from a design perspective is to leave natural places to shard/split the data up so you could represent the data consistently on multiple systems. Sometimes it's pretty bonehead things like your database tables related to login are affecting your critical path transactions and/or also tied to the referential integrity of your application data in ways that are hard to split apart. It goes back to the IO thing where if you are reading and writing login or UI traffic you sure as heck are not writing some other critical time sensitive transaction especially if it's to the same disk. The other obvious thing is to be trying to run analytic queries (aka look at many records) on a database tuned for mass INSERT/write traffic and having no plan to do that with a different resource. If you are still worried about it I would suggest just prototyping it using a currently modern API/RPC server in front of whatever datastore you come up with. As systems have gotten more distributed the straight line speed of connecting to the database directly makes less and less sense in my opinion and having an API server you control allows you to be clever later in ways that might not be obvious today (especially keeping application-aware counters so that you can see what your application is actually doing as query-style logs for databases are extremely expensive in performance). Either way if you sit down and consider the performance characteristics of the underlying system, think about each class of transactions are you considering from your application in a write/read context (hopefully from measured data), and using tools like EXPLAIN you'll quickly start finding what pieces of the puzzle might need to move to start scaling. The answer also might just be some parts of your application would be better served something other than a database like a message queue, a more straightforward ISAM or key-value store, or depending on your situation just flat files written onto the fastest local storage you can get to then be later transformed and loaded into a database sometime later.

Ozzie_osman 5y ago

> The answer is to start siloing customers or application concerns out into separate clusters depending on your operating model. This is such an underrated solution (siloing or sharding your data in some way). I think people don't do it because: 1. The tooling doesn't make it super-easy (e.g. good luck sharding Postgres unless you're willing to pay for Citus) 2. "Trendy" companies in the past decade have been network-type products (social networks, etc), where the structure of the data makes it much harder to silo (people need to follow each other, interact with each other's content, etc) 3. We as an industry took a several-year detour over to NoSQL land as a promised solution to scalability. Life would be a lot easier if you could say something like: * I want a Postgres node. * I'm happy to shard my data by some key (customerId, city, etc) and am willing to accept responsibility for thinking through that sharding key. * My application has some logic that easily knows which DB to read/write from depending on shard key. * There's some small amount of "global application data" that might need to live on a single node.

simonw 5y ago
For your "life would be easier" version, isn't that pretty much what Citus does? And Citus is open source so paying for the managed version is optional.
- cellis 5y ago
  saying "Citus is open source" doesn't mean its as simple as running `yarn install citus; yarn run citus`. Its is much much more complicated than typical open source. You pay for support for a very good reason.
rad_gruchalski 5y ago
> The tooling doesn't make it super-easy (e.g. good luck sharding Postgres unless you're willing to pay for Citus) There’s YugabyteDB.
seanhunter 5y ago
Sharding by customer has a lot of benefits for b2b applications also. You can run each customer in a seperate instance in their own vpc if you want to, and guarantee to your customers that their data is never commingled with another customer. This makes a lot of sense if you have relatively low number relatively high value customers. It radically reduces the infosec/screw up blast radius. For example it means however badly you screw up it is just about impossible to accidentally show data from customer A to a user from customer B. You can let customers specify region/availability zone and bring their own keys for encryption making lots of enterprise security compliance things easier (for regulated use cases). There is no one true way of scaling that suits all possible use cases and as engineers, we should think for ourselves and build a solution that works well in our situation, rather than thinking you have to do a certain thing because that's what google/facebook etc do. Their scaling problems are most likely really different to yours.

hobs 5y ago

I just automatically silo by customer because I am so scarred by so much shitty SQL written by ??? that it just make sense for privacy reasons alone. Row based permissions are not great imo, either performance is crap or you have some weird bug.

hughrr 5y ago
Totally agree there as well.