8 ms·

Having recently heard a lot of about PostgreSQL in Kubernetes (cloudNativePG for example) it always makes me wonder about the actual load and the complexity of

by radimm 4y ago

Having recently heard a lot of about PostgreSQL in Kubernetes (cloudNativePG for example) it always makes me wonder about the actual load and the complexity of the cluster in the question. > This is the reason why we were able to easily cope with 2M+ requests in less than 24h when Midnight Society launched This gives the answer, while it's probably not evenly distributed gives 23 req/sec (guess peak 60 - 100 might be already stretching it). Always wonder about use cases around 3 - 5k req/sec as minimum. [edit] PS: not really ditching neither k8s pg nor AWS RDS or similar solutions. Just being curious.

Nextgrid 4y ago
> 23 req/sec (guess peak 60 - 100 might be already stretching it) That kind of load is something a decent developer laptop with an NVME drive can serve, nothing to write home about. It is sad that the "cloud" and all these supposedly "modern" DevOps systems managed to redefine the concept of "performance" for a large chunk of the industry.
- rrampage 4y ago
  It depends a lot on the backend architecture. Number of DB requests per web request can also be high due to the pathological cases in some ORMs which can result in N+1 query problems or eagerly fetching entire object hierarchies. Such problems in application code can get brushed under the carpet due to "magical" autoscaling (be it RDS or K8s). There can also be fanout to async services/job queues which will in turn run even more DB queries.
  - AccountAccount1 4y ago
    Hey, this is not a problem for us at Nhost since most of the interfacing with Postgres is through Hasura (a GraphQL SQL-to-GraphQL) it solves the n+1 issue by compiling a performant sql statement from the gql query (it's also written in haskell, you can read more here https://hasura.io/blog/architecture-of-a-high-performance-graphql-to-sql-server-58d9944b8a87/ https://hasura.io/blog/architecture-of-a-high-performance-gr...)
  - robertlagrant 4y ago
    I don't think K8s at least will autoscale quickly enough to mask something like that.
    - raffraffraff 4y ago
      Autoscaling is slow. If you're using AWS autoscaling group, decisions are based on several different metrics that are typically averaged over a period. If the instance pool size is increased, that fact gets picked up by yet another event loop that runs periodically, and actually starts instances. So there are multiple chained delays before the instance is actually launched. In practice, even if your instances have extremely fast start-up and can begin processing the queue quickly, the job in the queue could be waiting 4+ minutes to get picked up, in a scale-to-zero situation. You've also got things like cooldown periods to ensure that you are not flapping. With k8s you have more control over knobs and switches, and you don't have an instance start-up delay, but the same type of metrics and event loops are used, particularly if you're using an external metric (eg SQS queue depth) in your calculations. Some type of predictive and/or scheduled scaling can reduce delays at the expense of potentially higher cost.
- singron 4y ago
  RDS tops out at about 18000 IOPS since it uses a single ebs volume. Any decent ssd will do much better. E.g. a 970 evo will easily do >100K IOPS and can do more like 400K in ideal conditions. You can get that many IOPS with aurora, but the cost is exorbitant.
  - mcbain 4y ago
    I don't think it has been a single EBS volume for a while, but in any case, 256k is more than 18k. https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Storage.html#USER_PIOPS https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_...
  - raffraffraff 4y ago
    I believe RDS automatically stripes EBS volumes under the hood, but don't expose that information to you unless you enable enhanced metrics (it's shown under "Physical Device I/O", where you can infer the number of volumes in the stripe). I have no idea when the striping kicks in - presumable some specific volume size for gp2, or some provisioned IOPS setting. According to the link below, provisioned IOPS tops out at 256000: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Storage.html#:~:text=Amazon%20RDS%20provides%20that%20IOPS,operations%20per%20second%20(IOPS) https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_....
    - singron 4y ago
      Provisioned IOPS is insanely expensive. Running 256K IOPS is $25K/month. You could outright buy multiple 970 EVOs every day and spend less money.
  - StreamBright 4y ago
    And this is very rarely the only dimension we chose technology by.
- mhuffman 4y ago
  It does depend on the architecture and framework they are using imo. I have a single Hetzer machine with spinning plate HDs that serves between 1-2 million requests per day hitting DB and ML models and rarely every gets over 1% CPU usage. I have pressure-tested it to around 3k reqs/sec. On the other hand I have seen WP and CodeIgniter setups that even with 5 copies running on the largest AWS instances available, "optimized" to the hilt, caching everywhere possible, etc. absolute crumble under the load of 3k req per min. (not sec ... min). Many frameworks that make early development easy fuck you later during growth with ORM calls, tons of unnecessary text in the DB, etc.
  - Nextgrid 4y ago
    Keep in mind that your Hetzner instance has locally-attached storage and a real CPU as opposed to networked storage and a slice of a CPU, so I'm not surprised at all that this beats an AWS setup even on the more expensive instances. Yes, frameworks can be a problem (although including WP in the list is an insult to other, actually decent frameworks), but I would bet good money if they moved their setup to a Hetzner setup it would still fly. Non-optimal ORM calls can be optimized manually without necessarily dropping the framework altogether.
    - marcosdumay 4y ago
      Hum... The Hetzner instance is very likely cheaper than any AWS setup, so while there is a point in that part, it's not a very relevant one. (And that's exactly the issue with the "modern DevOps" tooling.)
  - acdha 4y ago
    > On the other hand I have seen WP and CodeIgniter setups that even with 5 copies running on the largest AWS instances available, "optimized" to the hilt, caching everywhere possible, etc. absolute crumble under the load of 3k req per min. (not sec ... min). This sounds like some other architectural problems - running nowhere near the largest instances available that was single node performance on EC2 in the 2000s. There are concerns switching from local to SAN storage, of course, but that’s also shifting the problem if you care about durability.
- jerf 4y ago
  I can't blame it on "cloud", though it's not helping that there are an awful lot of cloud services that claim to be "high performance" and are often mediumish at best. But in general I see a lot of ignorance in the developer community as to how fast things should be able to run, even in terms of reading local files and doing local manipulations with no "cloud" in sight. Honestly, if I had to pin it on just one thing, I'd blame networking everything. Cloud would fit as a subset of that. Networking slows things down at the best of times, and the latency distribution can be a nightmare at the worst. Few developers think about the cost of using the network, and even fewer can think about it holistically (e.g., to avoid making 50 network transactions spread throughout the system when you could do it all in one transaction if you rearranged things).
  - geggam 4y ago
    Are you talking about the cloud host to cloud host networking or the POD networking inside the single host ? The dizzying amount of NAT layers has to be killing performance. I haven't had the chance to ever sit down and unravel a system running a good load. The lack of TCP tuning combined with the required connection tracking is interesting to think about
    - kazen44 4y ago
      i still dont understand why nearly all CNI's are so hell bent on implementing a dozen layers of NAT to tunnel their overlay networks, instead of implementing a proper control plane to automate it all away between routes. Calico seems to be doing it semi-okeish, and even their the control plane is kind of unfinished? The only software based solution which seem to properly have this figured out is VMware NSX-T. (i am not counting all the traditional overlay networks in use by ISP's based on MPLS/BGP).
      - geggam 4y ago
        Before you even get to the CNI, I think AWS VM to internet is at least 3 NAT layers. So we have 3 layers from container to pod. The virtual host kernel is tracking those layers. Once connection to one container is 3 tracked connections. Then you have whatever else you put on top to go in and out of the internet. The funny think to me is HaProxy recommended getting rid of connection tracking for performance while everyone is doubling down on that alone and calling it performant.
- derefr 4y ago
  Depends on the queries. Point queries that take 1ms each? Sure. Analytical queries that take 1000ms+ each? Not so much.
- c2h5oh 4y ago
  That kind of a load you can handle on spinning rust without breaking a sweat.
- ayende 4y ago
  You are off by a couple of orders of magnitude I have run 500+ req/sec on a raspberry pi using 4 TB dataset with 2 GB of RAM, with under 100ms for the 99.99 percentile A few hundreds req a second is basically nothing.
- deleted 4y ago
  [deleted]
- eptcyka 4y ago
  NVME? You can serve this from a raspberry pi.
- StreamBright 4y ago
  I see this problem a bit more nuanced. Why does everybody starting with the assumption that the solution is SQL? You can get very far with a k:v store like S3 for example. On the top of that, if you really need SQL you can use a lot of different systems (without k8s).
xani_ 4y ago
It's essentially just a process running in a cgroup so performance shouldn't be all that different than bare metal/VM postgresql. Main difference would be storage speed and how it exactly is attached to a container.
brand 4y ago
I’ve personally deployed O(TBs) and O(10^4 TPS) Postgres clusters on Kubernetes with a CNPG-style operator based deployment. There are some subtleties to it but it’s not exceeding complicated, and a good project like CNPG goes a long way to shaving off those sharp edges. As other commenters have suggested it’s good to really understand Kubernetes if you want to do it, though.
- radimm 4y ago
  Thanks for the confirmation. As mentioned I'm not saying no to it. It is really that "really understand" part which holds me back for now - mainly the observability and dealing with edge cases in high-throughput environment.
- remram 4y ago
  > O(TBs) and O(10^4 TPS) What does this syntax mean? Surely you wouldn't use big-o notation with a constant in it, especially to convey the same meaning as the thing without the O?
  - MikePlacid 4y ago
    Mathematically speaking the statement you are objecting to is correct: c1 is O(c2) for any constants c1, c2. English-language-ly speaking the statement you are objecting to is also correct: both you and I managed to get it’s correct meaning. No?
    - remram 4y ago
      No, I am not sure about the meaning at all. Did they deploy databases that will tend to be a TB in size as something tends to infinity? Or multiple TBs? I don't know if they know about the constant factor since they don't know what the notation mean. Maybe they know what it means and are using it for a clever lie, a 1kB database is O(1TB). So is an empty database. GGP is trying to be cool, and doing so stripped all meaning from their statement.
kccqzy 4y ago
> This is the reason why we were able to easily cope with 2M+ requests in less than 24h I thought this was referring to 2M+ requests per second over a ramp period of 24h, not 2M requests per 24h?
XCSme 4y ago
2M+ requests per day can be handled on a pretty cheap VPS even by MySQL, but it depends on the request complexity and, more importantly, the database size.
MuffinFlavored 4y ago
> Having recently heard a lot of about PostgreSQL in Kubernetes I could never get a straight answer on whether running a database in a container (and mounting the storage volume through a bind mount/network drive or whatever) came with a performance hit compared to running it as a systemd service for example.
- speedgoose 4y ago
  It does but it’s minimal. Especially compared to the high latency and low throughout network volumes provides (which are the defaults on cloud VMs).
- ahachete 4y ago
  In case you are interested, I blogged about it last year: https://thenewstack.io/kubernetes-will-revolutionize-enterprise-database-management/ https://thenewstack.io/kubernetes-will-revolutionize-enterpr... TL;DR performance impact should be negligible, could be even slightly negative compared to a VM (when running K8s on bare metal).
  - MuffinFlavored 4y ago
    What's the best way to simply mount the storage volume needed for Postgres to be performant?

by radimm 4y ago

Nextgrid 4y ago

> 23 req/sec (guess peak 60 - 100 might be already stretching it) That kind of load is something a decent developer laptop with an NVME drive can serve, nothing to write home about. It is sad that the "cloud" and all these supposedly "modern" DevOps systems managed to redefine the concept of "performance" for a large chunk of the industry.

rrampage 4y ago
It depends a lot on the backend architecture. Number of DB requests per web request can also be high due to the pathological cases in some ORMs which can result in N+1 query problems or eagerly fetching entire object hierarchies. Such problems in application code can get brushed under the carpet due to "magical" autoscaling (be it RDS or K8s). There can also be fanout to async services/job queues which will in turn run even more DB queries.
- AccountAccount1 4y ago
  Hey, this is not a problem for us at Nhost since most of the interfacing with Postgres is through Hasura (a GraphQL SQL-to-GraphQL) it solves the n+1 issue by compiling a performant sql statement from the gql query (it's also written in haskell, you can read more here https://hasura.io/blog/architecture-of-a-high-performance-graphql-to-sql-server-58d9944b8a87/ https://hasura.io/blog/architecture-of-a-high-performance-gr...)
- robertlagrant 4y ago
  I don't think K8s at least will autoscale quickly enough to mask something like that.
  - raffraffraff 4y ago
    Autoscaling is slow. If you're using AWS autoscaling group, decisions are based on several different metrics that are typically averaged over a period. If the instance pool size is increased, that fact gets picked up by yet another event loop that runs periodically, and actually starts instances. So there are multiple chained delays before the instance is actually launched. In practice, even if your instances have extremely fast start-up and can begin processing the queue quickly, the job in the queue could be waiting 4+ minutes to get picked up, in a scale-to-zero situation. You've also got things like cooldown periods to ensure that you are not flapping. With k8s you have more control over knobs and switches, and you don't have an instance start-up delay, but the same type of metrics and event loops are used, particularly if you're using an external metric (eg SQS queue depth) in your calculations. Some type of predictive and/or scheduled scaling can reduce delays at the expense of potentially higher cost.
singron 4y ago
RDS tops out at about 18000 IOPS since it uses a single ebs volume. Any decent ssd will do much better. E.g. a 970 evo will easily do >100K IOPS and can do more like 400K in ideal conditions. You can get that many IOPS with aurora, but the cost is exorbitant.
- mcbain 4y ago
  I don't think it has been a single EBS volume for a while, but in any case, 256k is more than 18k. https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Storage.html#USER_PIOPS https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_...
- raffraffraff 4y ago
  I believe RDS automatically stripes EBS volumes under the hood, but don't expose that information to you unless you enable enhanced metrics (it's shown under "Physical Device I/O", where you can infer the number of volumes in the stripe). I have no idea when the striping kicks in - presumable some specific volume size for gp2, or some provisioned IOPS setting. According to the link below, provisioned IOPS tops out at 256000: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Storage.html#:~:text=Amazon%20RDS%20provides%20that%20IOPS,operations%20per%20second%20(IOPS) https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_....
  - singron 4y ago
    Provisioned IOPS is insanely expensive. Running 256K IOPS is $25K/month. You could outright buy multiple 970 EVOs every day and spend less money.
- StreamBright 4y ago
  And this is very rarely the only dimension we chose technology by.
mhuffman 4y ago
It does depend on the architecture and framework they are using imo. I have a single Hetzer machine with spinning plate HDs that serves between 1-2 million requests per day hitting DB and ML models and rarely every gets over 1% CPU usage. I have pressure-tested it to around 3k reqs/sec. On the other hand I have seen WP and CodeIgniter setups that even with 5 copies running on the largest AWS instances available, "optimized" to the hilt, caching everywhere possible, etc. absolute crumble under the load of 3k req per min. (not sec ... min). Many frameworks that make early development easy fuck you later during growth with ORM calls, tons of unnecessary text in the DB, etc.
- Nextgrid 4y ago
  Keep in mind that your Hetzner instance has locally-attached storage and a real CPU as opposed to networked storage and a slice of a CPU, so I'm not surprised at all that this beats an AWS setup even on the more expensive instances. Yes, frameworks can be a problem (although including WP in the list is an insult to other, actually decent frameworks), but I would bet good money if they moved their setup to a Hetzner setup it would still fly. Non-optimal ORM calls can be optimized manually without necessarily dropping the framework altogether.
  - marcosdumay 4y ago
    Hum... The Hetzner instance is very likely cheaper than any AWS setup, so while there is a point in that part, it's not a very relevant one. (And that's exactly the issue with the "modern DevOps" tooling.)
- acdha 4y ago
  > On the other hand I have seen WP and CodeIgniter setups that even with 5 copies running on the largest AWS instances available, "optimized" to the hilt, caching everywhere possible, etc. absolute crumble under the load of 3k req per min. (not sec ... min). This sounds like some other architectural problems - running nowhere near the largest instances available that was single node performance on EC2 in the 2000s. There are concerns switching from local to SAN storage, of course, but that’s also shifting the problem if you care about durability.
jerf 4y ago
I can't blame it on "cloud", though it's not helping that there are an awful lot of cloud services that claim to be "high performance" and are often mediumish at best. But in general I see a lot of ignorance in the developer community as to how fast things should be able to run, even in terms of reading local files and doing local manipulations with no "cloud" in sight. Honestly, if I had to pin it on just one thing, I'd blame networking everything. Cloud would fit as a subset of that. Networking slows things down at the best of times, and the latency distribution can be a nightmare at the worst. Few developers think about the cost of using the network, and even fewer can think about it holistically (e.g., to avoid making 50 network transactions spread throughout the system when you could do it all in one transaction if you rearranged things).
- geggam 4y ago
  Are you talking about the cloud host to cloud host networking or the POD networking inside the single host ? The dizzying amount of NAT layers has to be killing performance. I haven't had the chance to ever sit down and unravel a system running a good load. The lack of TCP tuning combined with the required connection tracking is interesting to think about
  - kazen44 4y ago
    i still dont understand why nearly all CNI's are so hell bent on implementing a dozen layers of NAT to tunnel their overlay networks, instead of implementing a proper control plane to automate it all away between routes. Calico seems to be doing it semi-okeish, and even their the control plane is kind of unfinished? The only software based solution which seem to properly have this figured out is VMware NSX-T. (i am not counting all the traditional overlay networks in use by ISP's based on MPLS/BGP).
    - geggam 4y ago
      Before you even get to the CNI, I think AWS VM to internet is at least 3 NAT layers. So we have 3 layers from container to pod. The virtual host kernel is tracking those layers. Once connection to one container is 3 tracked connections. Then you have whatever else you put on top to go in and out of the internet. The funny think to me is HaProxy recommended getting rid of connection tracking for performance while everyone is doubling down on that alone and calling it performant.
derefr 4y ago
Depends on the queries. Point queries that take 1ms each? Sure. Analytical queries that take 1000ms+ each? Not so much.
c2h5oh 4y ago
That kind of a load you can handle on spinning rust without breaking a sweat.
ayende 4y ago
You are off by a couple of orders of magnitude I have run 500+ req/sec on a raspberry pi using 4 TB dataset with 2 GB of RAM, with under 100ms for the 99.99 percentile A few hundreds req a second is basically nothing.
deleted 4y ago
[deleted]
eptcyka 4y ago
NVME? You can serve this from a raspberry pi.
StreamBright 4y ago
I see this problem a bit more nuanced. Why does everybody starting with the assumption that the solution is SQL? You can get very far with a k:v store like S3 for example. On the top of that, if you really need SQL you can use a lot of different systems (without k8s).

xani_ 4y ago

It's essentially just a process running in a cgroup so performance shouldn't be all that different than bare metal/VM postgresql. Main difference would be storage speed and how it exactly is attached to a container.

brand 4y ago

I’ve personally deployed O(TBs) and O(10^4 TPS) Postgres clusters on Kubernetes with a CNPG-style operator based deployment. There are some subtleties to it but it’s not exceeding complicated, and a good project like CNPG goes a long way to shaving off those sharp edges. As other commenters have suggested it’s good to really understand Kubernetes if you want to do it, though.

radimm 4y ago
Thanks for the confirmation. As mentioned I'm not saying no to it. It is really that "really understand" part which holds me back for now - mainly the observability and dealing with edge cases in high-throughput environment.
remram 4y ago
> O(TBs) and O(10^4 TPS) What does this syntax mean? Surely you wouldn't use big-o notation with a constant in it, especially to convey the same meaning as the thing without the O?
- MikePlacid 4y ago
  Mathematically speaking the statement you are objecting to is correct: c1 is O(c2) for any constants c1, c2. English-language-ly speaking the statement you are objecting to is also correct: both you and I managed to get it’s correct meaning. No?
  - remram 4y ago
    No, I am not sure about the meaning at all. Did they deploy databases that will tend to be a TB in size as something tends to infinity? Or multiple TBs? I don't know if they know about the constant factor since they don't know what the notation mean. Maybe they know what it means and are using it for a clever lie, a 1kB database is O(1TB). So is an empty database. GGP is trying to be cool, and doing so stripped all meaning from their statement.

kccqzy 4y ago

> This is the reason why we were able to easily cope with 2M+ requests in less than 24h I thought this was referring to 2M+ requests per second over a ramp period of 24h, not 2M requests per 24h?

XCSme 4y ago

2M+ requests per day can be handled on a pretty cheap VPS even by MySQL, but it depends on the request complexity and, more importantly, the database size.

MuffinFlavored 4y ago

> Having recently heard a lot of about PostgreSQL in Kubernetes I could never get a straight answer on whether running a database in a container (and mounting the storage volume through a bind mount/network drive or whatever) came with a performance hit compared to running it as a systemd service for example.

speedgoose 4y ago
It does but it’s minimal. Especially compared to the high latency and low throughout network volumes provides (which are the defaults on cloud VMs).
ahachete 4y ago
In case you are interested, I blogged about it last year: https://thenewstack.io/kubernetes-will-revolutionize-enterprise-database-management/ https://thenewstack.io/kubernetes-will-revolutionize-enterpr... TL;DR performance impact should be negligible, could be even slightly negative compared to a VM (when running K8s on bare metal).
- MuffinFlavored 4y ago
  What's the best way to simply mount the storage volume needed for Postgres to be performant?