4 ms·

I've fixed a lot of performance problems over my career. This is not true, and not what we talk about when we say it's almost always the db. The usual cause o

I've fixed a lot of performance problems over my career. This is not true, and not what we talk about when we say it's almost always the db. The usual cause of database problems is a single big query that is causing a slow page, or a lot of medium size queries inside a poorly written loop. Whether it be bad joins, missing indexes, select statements in the select statements, operations in select statements that force the db to run the query line by line instead of as a set (the technical term suddenly escapes me), it's the queries that almost always cause db slowdowns. It rarely has anything to do with writes, and then in my experience usually had to do with escalating locks, and I haven't seen that problem in ages as hardwares got faster.

dorfsmay 7y ago
> This is not true, and not what we talk about when we say it's almost always the db. It depends who's talking and what they are talking about. If we're talking about fixing slow website, then sure, bad big ugly queries are one dominant issue, but not the only one. If we're talking about architecture of a greenfield project, then balancing between concurrent writes and consistency is going to be your ultimate problem, everything else can be "duplicated" (read-duplicated DB servers, caching strategies, multiple apps server, etc...).
- mattmanser 7y ago
  And who ever needs to think about that stuff? A tiny number of companies. Again, I just think you're not considering the vast majority of us here work in terms of thousands, 10s of thousands or millions of users, not the sort of scales where anything you're saying ever becomes relevant. For most of us, if you spend any time thinking about that, you're utterly wasting it. It's pointless. Greenfield or not.
  - scarface74 7y ago
    I have no idea where the mindset comes from that only large companies have large datasets.
    - bcrosby95 7y ago
      In my experience, a dataset is considered large if its the largest dataset the company has ever seen. It could be 2GB and be considered "large".
- dorfsmay 7y ago
  Replying to mattmanser in the sibling comment since I cannot replying to it directly: It comes in faster than you think. One of the company I work for was a small company, at the time I think we were 4 or 5 including the 2 founders, they sell some specific information as a service through APIs. They charge per call, so you have to update the customer funds in their account on each API call. Yes, people can want and make lots of APIs call very fast, and in a business which charges per API call, this is a good thing, but you have to find strategies and make business decisions on how to handle accepting and replying to new API calls while maintaining their account balance.
  - nostrademons 7y ago
    Load the remaining customer funds into RAM, and decrement them on each API. Periodically flush the remaining balance to DB at set intervals (say every hour). If the server goes down, well, your customers got free API calls for an hour to compensate them for their trouble. With any reasonable uptime you'll save more on DB traffic than you lose in lost revenues. (BTW, the "can't reply" issue happens on HN when there's a long comment thread between just two people that's being rapidly updated. If you wait 10 minutes or so you'll be able to reply.)
    - jimktrains2 7y ago
      > Load the remaining customer funds into RAM, and decrement them on each API. Periodically flush the remaining balance to DB at set intervals (say every hour). So you only ever have one app server? Sure, this specific issues could be handled other ways, but in general global state is hard.
      - nostrademons 7y ago
        If you have more than one app server load the customer funds into Redis and use atomic operations to decrement it. Serving out of RAM is going to be orders of magnitude faster than updating disk regardless of how much you spend on network overhead. If you have so many customers that they blow the RAM limit on a Redis box....well, you won't, because that's billions of customers. You can get surprisingly far on one app server (or sometimes one app server with 2 mirrors for failover). See eg. the latest TechEmpower benchmarks [1], where common Java and C++ web frameworks can serve 300k+ reqs/sec off of bare-metal hardware. The OP indicated that these API requests are basically straight queries anyway, and only need to write to update the billing information. Reads can run incredibly fast (both because they can usually be served out of cache and because they can be mirrored), so if your only bottleneck is the write, take the write out of the critical path. In general global state is hard. Don't solve general problems, solve specific ones. Atomic counters have a well-understood and highly performant solution. [1] https://www.techempower.com/benchmarks/#section=data-r17&hw=ph&test=fortune https://www.techempower.com/benchmarks/#section=data-r17&hw=...
merpnderp 7y ago
Nothing is slower than accessing third party databases you can't optimize. And a near 100% likelihood in any enterprise operation. We still work around this by caching tables where possible, but often we must access the single source of truth regardless of performance.
asark 7y ago
There are an awful lot of low-hanging-fruit performance problems out there that are just a result of developers not using the DB correctly—usually doing too much in whatever slow scripting language they're using, rather than letting the DB do as much as it can. Sorting, counting, combining, a bunch of queries in a loop where one or two cleverer ones would do. I think part of why low-feature, specific-usecase DBs like Mongo can get such traction as a general purpose datastore (when marketed as such, of course, to sell more licenses, no matter how bad an idea it is—looking at you, Neo4j) is because so many devs don't know what a half-decent SQL DB can (and should) do for them in the first place.
- didericis 7y ago
  Is there an alternative to Neo4j you would recommend, or do you think graph dbs are bad in general? I have a pet project that works with data that’s very graphy (timetabling app with lots of interdependent events); I tried using postges at first since I’m used to it, but found myself writing really ugly looking recursive queries. Neo4j seems to fit my use case a lot better, and their graphql plugin has been really useful.
  - asark 7y ago
    N4J's a good option if you have a dense graph (lots of edges) and need to do one of the handful of things it's good at with said graph (shortest path, that sort of thing). I would not recommend using it as a general datastore, even if your data's a little "graphy", because it's not great at stuff you'd think it might be, like "find this subgraph, then return some data from nodes related to each node", performance-wise, and it lacks tons of features something like Postgres has, including simply providing lots of built-in data types. I'd recommend using it in combination with an authoritative datastore if you just need to do some "graphy" things, using it as a secondary datastore just for certain purposes—it'll be safer and likely perform better. [EDIT] it's also not nearly as well-supported so you'll find a lot of supporting libraries with multi-datastore support don't support N4J yet, or do but only in some crippled or poorly-tested (not widely deployed) fashion. [EDIT EDIT] it's also not a great fit if you have a lot of constraints on or structure to your graph. At least as of when I used it ~1 year ago it had essentially no support for expressing things like "this type of node should only permit two outbound edges". You can work around that but the solutions will be less safe and/or suck.
WhiteOwlLion 7y ago
line by line: table scan?

dorfsmay 7y ago

> This is not true, and not what we talk about when we say it's almost always the db. It depends who's talking and what they are talking about. If we're talking about fixing slow website, then sure, bad big ugly queries are one dominant issue, but not the only one. If we're talking about architecture of a greenfield project, then balancing between concurrent writes and consistency is going to be your ultimate problem, everything else can be "duplicated" (read-duplicated DB servers, caching strategies, multiple apps server, etc...).

mattmanser 7y ago
And who ever needs to think about that stuff? A tiny number of companies. Again, I just think you're not considering the vast majority of us here work in terms of thousands, 10s of thousands or millions of users, not the sort of scales where anything you're saying ever becomes relevant. For most of us, if you spend any time thinking about that, you're utterly wasting it. It's pointless. Greenfield or not.
- scarface74 7y ago
  I have no idea where the mindset comes from that only large companies have large datasets.
  - bcrosby95 7y ago
    In my experience, a dataset is considered large if its the largest dataset the company has ever seen. It could be 2GB and be considered "large".
dorfsmay 7y ago
Replying to mattmanser in the sibling comment since I cannot replying to it directly: It comes in faster than you think. One of the company I work for was a small company, at the time I think we were 4 or 5 including the 2 founders, they sell some specific information as a service through APIs. They charge per call, so you have to update the customer funds in their account on each API call. Yes, people can want and make lots of APIs call very fast, and in a business which charges per API call, this is a good thing, but you have to find strategies and make business decisions on how to handle accepting and replying to new API calls while maintaining their account balance.
- nostrademons 7y ago
  Load the remaining customer funds into RAM, and decrement them on each API. Periodically flush the remaining balance to DB at set intervals (say every hour). If the server goes down, well, your customers got free API calls for an hour to compensate them for their trouble. With any reasonable uptime you'll save more on DB traffic than you lose in lost revenues. (BTW, the "can't reply" issue happens on HN when there's a long comment thread between just two people that's being rapidly updated. If you wait 10 minutes or so you'll be able to reply.)
  - jimktrains2 7y ago
    > Load the remaining customer funds into RAM, and decrement them on each API. Periodically flush the remaining balance to DB at set intervals (say every hour). So you only ever have one app server? Sure, this specific issues could be handled other ways, but in general global state is hard.
    - nostrademons 7y ago
      If you have more than one app server load the customer funds into Redis and use atomic operations to decrement it. Serving out of RAM is going to be orders of magnitude faster than updating disk regardless of how much you spend on network overhead. If you have so many customers that they blow the RAM limit on a Redis box....well, you won't, because that's billions of customers. You can get surprisingly far on one app server (or sometimes one app server with 2 mirrors for failover). See eg. the latest TechEmpower benchmarks [1], where common Java and C++ web frameworks can serve 300k+ reqs/sec off of bare-metal hardware. The OP indicated that these API requests are basically straight queries anyway, and only need to write to update the billing information. Reads can run incredibly fast (both because they can usually be served out of cache and because they can be mirrored), so if your only bottleneck is the write, take the write out of the critical path. In general global state is hard. Don't solve general problems, solve specific ones. Atomic counters have a well-understood and highly performant solution. [1] https://www.techempower.com/benchmarks/#section=data-r17&hw=ph&test=fortune https://www.techempower.com/benchmarks/#section=data-r17&hw=...

merpnderp 7y ago

Nothing is slower than accessing third party databases you can't optimize. And a near 100% likelihood in any enterprise operation. We still work around this by caching tables where possible, but often we must access the single source of truth regardless of performance.

asark 7y ago

There are an awful lot of low-hanging-fruit performance problems out there that are just a result of developers not using the DB correctly—usually doing too much in whatever slow scripting language they're using, rather than letting the DB do as much as it can. Sorting, counting, combining, a bunch of queries in a loop where one or two cleverer ones would do. I think part of why low-feature, specific-usecase DBs like Mongo can get such traction as a general purpose datastore (when marketed as such, of course, to sell more licenses, no matter how bad an idea it is—looking at you, Neo4j) is because so many devs don't know what a half-decent SQL DB can (and should) do for them in the first place.

didericis 7y ago
Is there an alternative to Neo4j you would recommend, or do you think graph dbs are bad in general? I have a pet project that works with data that’s very graphy (timetabling app with lots of interdependent events); I tried using postges at first since I’m used to it, but found myself writing really ugly looking recursive queries. Neo4j seems to fit my use case a lot better, and their graphql plugin has been really useful.
- asark 7y ago
  N4J's a good option if you have a dense graph (lots of edges) and need to do one of the handful of things it's good at with said graph (shortest path, that sort of thing). I would not recommend using it as a general datastore, even if your data's a little "graphy", because it's not great at stuff you'd think it might be, like "find this subgraph, then return some data from nodes related to each node", performance-wise, and it lacks tons of features something like Postgres has, including simply providing lots of built-in data types. I'd recommend using it in combination with an authoritative datastore if you just need to do some "graphy" things, using it as a secondary datastore just for certain purposes—it'll be safer and likely perform better. [EDIT] it's also not nearly as well-supported so you'll find a lot of supporting libraries with multi-datastore support don't support N4J yet, or do but only in some crippled or poorly-tested (not widely deployed) fashion. [EDIT EDIT] it's also not a great fit if you have a lot of constraints on or structure to your graph. At least as of when I used it ~1 year ago it had essentially no support for expressing things like "this type of node should only permit two outbound edges". You can work around that but the solutions will be less safe and/or suck.

WhiteOwlLion 7y ago

line by line: table scan?