5 ms·

The more you put in the database black box, the more scalability problems you will encounter in the future. It's more economical to scale out than buy bigger da

The more you put in the database black box, the more scalability problems you will encounter in the future. It's more economical to scale out than buy bigger databases servers. Please just don't do it.

kiwidrew 14y ago
Happily, Postgres keeps improving performance with each release. According to benchmarks [1], the upcoming version 9.2 will scale up to 64 cores for read-heavy workloads. Unless your application is going to grow really fast -- keeping in mind that the hardware capabilities will continue to improve each year -- then it's just premature optimization to worry about horizontal scaling. Just upgrade your database server once a year and reap the benefits of Moore's law. [1] http://rhaas.blogspot.com/2012/04/did-i-say-32-cores-how-about-64.html http://rhaas.blogspot.com/2012/04/did-i-say-32-cores-how-abo...
- eckyptang 14y ago
  I love your optimism. The real world doesn't work like that. The real world punishes you for every shitty feature you pick and every bad chunk of code. Our application is 15 years old, and we're on a 64-core machine with 768Gb of RAM and 35Tb of disk. We're running at 80% capacity. Where do we go from here? Yes, we rewrite and scale out for the measly cost of £450k. That cost would have been avoided with the appropriate due diligence. That is not a cost anyone wants to swallow. Then again I don't work disposable CRUD applications...
alexro 14y ago
It really depends on the case at hands, sometimes it's more practical to scale up, especially if there is no indefinite grows anticipated. http://www.codinghorror.com/blog/2009/06/scaling-up-vs-scaling-out-hidden-costs.html http://www.codinghorror.com/blog/2009/06/scaling-up-vs-scali...
- eckyptang 14y ago
  That's hardly a great reference, especially when he immediately shot himself by not checking vendor limits... http://www.codinghorror.com/blog/2009/07/oh-you-wanted-awesome-edition.html http://www.codinghorror.com/blog/2009/07/oh-you-wanted-aweso...
  - alexro 14y ago
    The general point still stands - scaling up is often a preferred solution and life proves it every day. Not so many companies enjoy facebook style grows, and even aren't viral at all.
jeffdavis 14y ago
"The more you put in the database black box, the more scalability problems you will encounter in the future." I don't see a obvious relationship between using arrays to store product tags and scalability. What method of solving this problem are you suggesting, and what are the scalability characteristics of that solution? Databases don't go out of their way to hurt scalability. Databases are shared state, and scaling that is just hard. Unless the problem is easy, in which case it's easy no matter what you use.
- eckyptang 14y ago
  Looking at the way arrays are implemented, they can readily turn operations into O(M*N) rather than O(N). Even the PgSQL documentation suggests cases that will hurt you. I would suggest that the relational model is maintained as there are no cases in which it isn't valid where arrays are and the optimiser is likely to come up with a better solution. Relational databases (well all databases) have convenient tools which in the short term look good, but in the long term will hurt you. Careful testing and feature selection is the alternative I am suggesting (from 20 years of experience with RDBMS platforms).
  - jeffdavis 14y ago
    "can readily turn operations" Which operations, specifically? We have the product tags example to work from, so it would help to be more concrete. "into O(M*N) rather than O(N)" I assume here that your target is "scalability to higher numbers of tags for a given product"? If so, please state that explicitly. Again, M and N aren't necessary, because we have an example to work from. Are the operations still susceptible to worse scalability on this axis if you employ a GIN index? "PgSQL documentation suggests cases that will hurt you." Please elaborate. "I would suggest that the relational model is maintained" I don't see how your solution is relational and the other one is not. Can you please explain? How is using an array different from using, say, a string, which is essentially an array of characters? How is it different from using a date, the components of which you can access (e.g. grouping by the month)? "the optimiser is likely to come up with a better solution." Please elaborate.

kiwidrew 14y ago

Happily, Postgres keeps improving performance with each release. According to benchmarks [1], the upcoming version 9.2 will scale up to 64 cores for read-heavy workloads. Unless your application is going to grow really fast -- keeping in mind that the hardware capabilities will continue to improve each year -- then it's just premature optimization to worry about horizontal scaling. Just upgrade your database server once a year and reap the benefits of Moore's law. [1] http://rhaas.blogspot.com/2012/04/did-i-say-32-cores-how-about-64.html http://rhaas.blogspot.com/2012/04/did-i-say-32-cores-how-abo...

eckyptang 14y ago
I love your optimism. The real world doesn't work like that. The real world punishes you for every shitty feature you pick and every bad chunk of code. Our application is 15 years old, and we're on a 64-core machine with 768Gb of RAM and 35Tb of disk. We're running at 80% capacity. Where do we go from here? Yes, we rewrite and scale out for the measly cost of £450k. That cost would have been avoided with the appropriate due diligence. That is not a cost anyone wants to swallow. Then again I don't work disposable CRUD applications...

alexro 14y ago

It really depends on the case at hands, sometimes it's more practical to scale up, especially if there is no indefinite grows anticipated. http://www.codinghorror.com/blog/2009/06/scaling-up-vs-scaling-out-hidden-costs.html http://www.codinghorror.com/blog/2009/06/scaling-up-vs-scali...

eckyptang 14y ago
That's hardly a great reference, especially when he immediately shot himself by not checking vendor limits... http://www.codinghorror.com/blog/2009/07/oh-you-wanted-awesome-edition.html http://www.codinghorror.com/blog/2009/07/oh-you-wanted-aweso...
- alexro 14y ago
  The general point still stands - scaling up is often a preferred solution and life proves it every day. Not so many companies enjoy facebook style grows, and even aren't viral at all.

jeffdavis 14y ago

"The more you put in the database black box, the more scalability problems you will encounter in the future." I don't see a obvious relationship between using arrays to store product tags and scalability. What method of solving this problem are you suggesting, and what are the scalability characteristics of that solution? Databases don't go out of their way to hurt scalability. Databases are shared state, and scaling that is just hard. Unless the problem is easy, in which case it's easy no matter what you use.

eckyptang 14y ago
Looking at the way arrays are implemented, they can readily turn operations into O(M*N) rather than O(N). Even the PgSQL documentation suggests cases that will hurt you. I would suggest that the relational model is maintained as there are no cases in which it isn't valid where arrays are and the optimiser is likely to come up with a better solution. Relational databases (well all databases) have convenient tools which in the short term look good, but in the long term will hurt you. Careful testing and feature selection is the alternative I am suggesting (from 20 years of experience with RDBMS platforms).
- jeffdavis 14y ago
  "can readily turn operations" Which operations, specifically? We have the product tags example to work from, so it would help to be more concrete. "into O(M*N) rather than O(N)" I assume here that your target is "scalability to higher numbers of tags for a given product"? If so, please state that explicitly. Again, M and N aren't necessary, because we have an example to work from. Are the operations still susceptible to worse scalability on this axis if you employ a GIN index? "PgSQL documentation suggests cases that will hurt you." Please elaborate. "I would suggest that the relational model is maintained" I don't see how your solution is relational and the other one is not. Can you please explain? How is using an array different from using, say, a string, which is essentially an array of characters? How is it different from using a date, the components of which you can access (e.g. grouping by the month)? "the optimiser is likely to come up with a better solution." Please elaborate.