5 ms·

This article is also very useful in showing just how far you can push Postgres _without_ reaching for any of these optimizations. I've seen too many projects w

This article is also very useful in showing just how far you can push Postgres _without_ reaching for any of these optimizations. I've seen too many projects worry about these things very early on in their lifecycle, when in reality they are no where close to having enough traffic to cause a problem.

dfox 9y ago
And almost certainly you can push postgresql even more by throwing hardware (ie. RAM) at the problem, although this is probably the point when it stops being worthwhile for reasonably sane applications (I've seen deployments where single pgsql server with 2TB+ RAM was cheaper solution than this kind of optimisation, due to all the technical debt in applications)
mamcx 9y ago
Great point. Is easy to overlook how much you can do with a decent RDBMS. And for the small data that gitlab use, I believe still exist a lot of big wins on performance. Is just that the new generation not pay much attention to Sql databases... P.D: I don't mean the gitlab developers, just on general
nickh9000 9y ago
The whole point why you should worry from the beginning is so that you don't have to re-work everything and put a huge risk to the business when you have to do it.
- RandomBK 9y ago
  Certainly. However, there's a difference between keeping scalability in mind, and actually implementing/maintaining a complex database scaling system at the onset of a project. Edit: I would also add that most projects I've seen tend to undergo one or more large refactors / redesigns before growing to a size where complex DB scaling is needed. This is, of course, speaking from the perspective of a small startup or pet project. If you're writing a new feature for an existing product where you can assume that it'll have lots of users, then you would obviously build scaling right from the start.
- jsmeaton 9y ago
  Instead you appear to be advocating for over engineering a solution for a problem you don’t yet have where the opportunity costs are probably features customers can use. That is also a huge risk to a business, but one that is immediate.
  - nickh9000 9y ago
    Thinking about how you would shard your data is hardly over engineering. On some storage systems you are forced to anyway: Cassandra, dynamodb. And the cost of building the solution from the onset is hardly a killer. The pain usually comes from operational complexity.
- ris 9y ago
  No. No. Never do this. You never actually know where your bottlenecks are going to be until they arrive and by designing everything with a "super scalable" architecture you will be making development ten times as painful and expensive as it needs to be while throwing away nice things that come "for free" and "just work" at the mid-low end like transactions. Amazon and Netflix don't want to have to use their hideously complex and inefficient service architectures - they're forced to because of their scale. Most people who engineer their systems for hyper-scale from the get-go never see a whiff of anything that looks remotely like high traffic. Often they go out of business before they get anywhere near that. And once you get to serious scale, you really shouldn't still be running the code from back when you didn't really know what your business/product was.
YorickPeterse 9y ago
Yes, you can typically push PostgreSQL very far while still using a fairly simple setup (e.g. no sharding). Unfortunately too many times people have this mindset that a slow application is the result of a slow/bad database (as in "it's an RDBMS and RDBMS' don't scale"), and not the result of it being misused (e.g. badly written queries, lack of proper indexes, that sort of thing). At GitLab it took quite a while to get everybody to see this.
- awj 9y ago
  1. Write web application with little to no idea how an RDBMS works 2. Run into inevitable performance problems 3. Decide that your database is at fault, rewrite in NoSQL 4. Buy into NoSQL hype, push 5x resources on your NoSQL solution 5. Profit? There are NoSQL options that are great for lots of things. It takes some significant expertise to know if your thing is one of those things. Expertise you probably don't have if you don't even have a moderately deep knowledge of RDBMS-es.
  - YorickPeterse 9y ago
    > Buy into NoSQL hype, push 5x resources on your NoSQL solution NoSQL is so last year, NewSQL is the future!
- philliphaydon 9y ago
  Lack of indexes vs way too many indexes that writes suffer.
deepsun 9y ago
Exactly. When they confessed they didn't use connection pooling I almost fell off my chair.

dfox 9y ago

And almost certainly you can push postgresql even more by throwing hardware (ie. RAM) at the problem, although this is probably the point when it stops being worthwhile for reasonably sane applications (I've seen deployments where single pgsql server with 2TB+ RAM was cheaper solution than this kind of optimisation, due to all the technical debt in applications)

mamcx 9y ago

Great point. Is easy to overlook how much you can do with a decent RDBMS. And for the small data that gitlab use, I believe still exist a lot of big wins on performance. Is just that the new generation not pay much attention to Sql databases... P.D: I don't mean the gitlab developers, just on general

nickh9000 9y ago

The whole point why you should worry from the beginning is so that you don't have to re-work everything and put a huge risk to the business when you have to do it.

RandomBK 9y ago
Certainly. However, there's a difference between keeping scalability in mind, and actually implementing/maintaining a complex database scaling system at the onset of a project. Edit: I would also add that most projects I've seen tend to undergo one or more large refactors / redesigns before growing to a size where complex DB scaling is needed. This is, of course, speaking from the perspective of a small startup or pet project. If you're writing a new feature for an existing product where you can assume that it'll have lots of users, then you would obviously build scaling right from the start.
jsmeaton 9y ago
Instead you appear to be advocating for over engineering a solution for a problem you don’t yet have where the opportunity costs are probably features customers can use. That is also a huge risk to a business, but one that is immediate.
- nickh9000 9y ago
  Thinking about how you would shard your data is hardly over engineering. On some storage systems you are forced to anyway: Cassandra, dynamodb. And the cost of building the solution from the onset is hardly a killer. The pain usually comes from operational complexity.
ris 9y ago
No. No. Never do this. You never actually know where your bottlenecks are going to be until they arrive and by designing everything with a "super scalable" architecture you will be making development ten times as painful and expensive as it needs to be while throwing away nice things that come "for free" and "just work" at the mid-low end like transactions. Amazon and Netflix don't want to have to use their hideously complex and inefficient service architectures - they're forced to because of their scale. Most people who engineer their systems for hyper-scale from the get-go never see a whiff of anything that looks remotely like high traffic. Often they go out of business before they get anywhere near that. And once you get to serious scale, you really shouldn't still be running the code from back when you didn't really know what your business/product was.

YorickPeterse 9y ago

Yes, you can typically push PostgreSQL very far while still using a fairly simple setup (e.g. no sharding). Unfortunately too many times people have this mindset that a slow application is the result of a slow/bad database (as in "it's an RDBMS and RDBMS' don't scale"), and not the result of it being misused (e.g. badly written queries, lack of proper indexes, that sort of thing). At GitLab it took quite a while to get everybody to see this.

awj 9y ago
1. Write web application with little to no idea how an RDBMS works 2. Run into inevitable performance problems 3. Decide that your database is at fault, rewrite in NoSQL 4. Buy into NoSQL hype, push 5x resources on your NoSQL solution 5. Profit? There are NoSQL options that are great for lots of things. It takes some significant expertise to know if your thing is one of those things. Expertise you probably don't have if you don't even have a moderately deep knowledge of RDBMS-es.
- YorickPeterse 9y ago
  > Buy into NoSQL hype, push 5x resources on your NoSQL solution NoSQL is so last year, NewSQL is the future!
philliphaydon 9y ago
Lack of indexes vs way too many indexes that writes suffer.

deepsun 9y ago

Exactly. When they confessed they didn't use connection pooling I almost fell off my chair.