4 ms·

Usually it's the inefficiencies of architectural decisions (your algos, layouts, etc.) you put into that rails app.

by rf15 2y ago

scosman 2y ago
Sometimes. But if you have any real processing to do, it can be hard/impossible to do it quickly in ruby. You end up writing c extensions, or another app server in another language for the hard stuff.
ndriscoll 2y ago
Right. Simple test: check the throughput your database can do with bulk queries. Do a single query select of an xml or json struct that outputs the page rails would return (so one rendered page per row). Select enough rows for the query to take 1-10s to get a sense for rendered pages/second. Do the same for inserts/updates. If you're not close to what postgres alone can do with bulk queries, you're obviously not IO bound. Other sanity check: nvme drives are ~10000x faster than hard disks were. Are you doing 10000x more useful disk IO than people were doing with hard disks?
- anarazel 2y ago
  For typical rails like applications that are "bottlenecked on the database", latency is a much bigger factor than actual query processing performance. Even if careful about placement of the database and "application servers". Just as an example, here's the results of pgbench -S of a single client, with pgbench running on different servers (a single pkey lookup): pgbench on database server: 36233 QPS pgbench on a different system (local 10gbit network): 2860 QPS It's rare to actually have that low latency to the database in the real world IME. If the same pkey lookup is executed utilizing pipelining, I get ~145k QPS from both locally and remotely.
  - ndriscoll 2y ago
    That's still an architectural thing. If CPU isn't an issue, why would you be running your application server on a different machine from your database? Connecting to a unix domain socket will give better latency and throughput and security as long as you can scale vertically. Even if you have it on another machine, it's on the same switch, right (also an architectural choice)? So latency should be sub millisecond, and irrelevant for a single request. So we assume we're talking about many requests, but then why not combine/batch your queries (also an architectural choice)? Now latency (and other things like database latches and fsyncs. Very important.) is amortized and again irrelevant. You should be able to hit close to the pure pg bulk query case (and e.g. a rust application server can do this in practice for me with a 4 core machine and an old SATA SSD with ~50k read IOPS). Point is, the optimal case for building a web page is probably to do pg queries that return the xml of the page (letting you do subselects to combine everything into one query). So see how fast that is, and if your application can't match that, the database is not your bottleneck.
    - anarazel 2y ago
      > That's still an architectural thing. If CPU isn't an issue, why would you be running your application server on a different machine from your database? Because that'll be too much of a scalability limitation. Rails etc are rather CPU heavy. In cloud environments it's also typically much more feasible to scale the stateless parts up and down than the database. > Even if you have it on another machine, it's on the same switch, right (also an architectural choice)? Yep, was on the same switch in my example. > So latency should be sub millisecond, and irrelevant for a single request. In my testcase it was well below a millisecond (2.8k QPS on a single non-pipelined connection would not be possible, it implies a RTT <= 0.35ms), but I don't at all agree that that makes it irrelevant for a single request.
whstl 2y ago
IME the Rails ecosystem and community pushes teams into architectural decisions that don't scale quite well. In small apps it's easy to keep track of the number of queries going to the DB per http request and their cost, but with bigger apps what I often find is ActiveRecord, services, policies, all calling the database way too much and too often. Monitoring which individual queries are slow is easy, but it doesn't help if operations are overcomplicated. Even if you chase all obvious N+1s, you might still have "sets of queries that could have been joins" but are in different classes, so it's difficult to notice.

Usually it's the inefficiencies of architectural decisions (your algos, layouts, etc.) you put into that rails app.

by rf15 2y ago

Usually it's the inefficiencies of architectural decisions (your algos, layouts, etc.) you put into that rails app.

Sometimes. But if you have any real processing to do, it can be hard/impossible to do it quickly in ruby. You end up writing c extensions, or another app server in another language for the hard stuff.

ndriscoll 2y ago

Right. Simple test: check the throughput your database can do with bulk queries. Do a single query select of an xml or json struct that outputs the page rails would return (so one rendered page per row). Select enough rows for the query to take 1-10s to get a sense for rendered pages/second. Do the same for inserts/updates. If you're not close to what postgres alone can do with bulk queries, you're obviously not IO bound. Other sanity check: nvme drives are ~10000x faster than hard disks were. Are you doing 10000x more useful disk IO than people were doing with hard disks?

anarazel 2y ago
For typical rails like applications that are "bottlenecked on the database", latency is a much bigger factor than actual query processing performance. Even if careful about placement of the database and "application servers". Just as an example, here's the results of pgbench -S of a single client, with pgbench running on different servers (a single pkey lookup): pgbench on database server: 36233 QPS pgbench on a different system (local 10gbit network): 2860 QPS It's rare to actually have that low latency to the database in the real world IME. If the same pkey lookup is executed utilizing pipelining, I get ~145k QPS from both locally and remotely.
- ndriscoll 2y ago
  That's still an architectural thing. If CPU isn't an issue, why would you be running your application server on a different machine from your database? Connecting to a unix domain socket will give better latency and throughput and security as long as you can scale vertically. Even if you have it on another machine, it's on the same switch, right (also an architectural choice)? So latency should be sub millisecond, and irrelevant for a single request. So we assume we're talking about many requests, but then why not combine/batch your queries (also an architectural choice)? Now latency (and other things like database latches and fsyncs. Very important.) is amortized and again irrelevant. You should be able to hit close to the pure pg bulk query case (and e.g. a rust application server can do this in practice for me with a 4 core machine and an old SATA SSD with ~50k read IOPS). Point is, the optimal case for building a web page is probably to do pg queries that return the xml of the page (letting you do subselects to combine everything into one query). So see how fast that is, and if your application can't match that, the database is not your bottleneck.
  - anarazel 2y ago
    > That's still an architectural thing. If CPU isn't an issue, why would you be running your application server on a different machine from your database? Because that'll be too much of a scalability limitation. Rails etc are rather CPU heavy. In cloud environments it's also typically much more feasible to scale the stateless parts up and down than the database. > Even if you have it on another machine, it's on the same switch, right (also an architectural choice)? Yep, was on the same switch in my example. > So latency should be sub millisecond, and irrelevant for a single request. In my testcase it was well below a millisecond (2.8k QPS on a single non-pipelined connection would not be possible, it implies a RTT <= 0.35ms), but I don't at all agree that that makes it irrelevant for a single request.

whstl 2y ago

IME the Rails ecosystem and community pushes teams into architectural decisions that don't scale quite well. In small apps it's easy to keep track of the number of queries going to the DB per http request and their cost, but with bigger apps what I often find is ActiveRecord, services, policies, all calling the database way too much and too often. Monitoring which individual queries are slow is easy, but it doesn't help if operations are overcomplicated. Even if you chase all obvious N+1s, you might still have "sets of queries that could have been joins" but are in different classes, so it's difficult to notice.