6 ms·

Why do we promote articles like this that have nice graphs and are well written, when they should get a grade 'F' as an actual benchmark study. The way it is pr

Why do we promote articles like this that have nice graphs and are well written, when they should get a grade 'F' as an actual benchmark study. The way it is presented, a casual reader would think Postgres is 2/3rds the performance of Redis. Good god. He even admits Postgres maxxed out its 2 cores, but Redis was bottlenecked by the HTTP server. We need more of an academic, not a hacker, culture for benchmarks.

dizzyVik 1y ago
There's a reason this is on my blog and not a paper in a journal. This isn't supposed to show the absolute speed of either tool, the benchmark is not set up for that. I do state that redis has more performance on the table in the blog post.
- jamesblonde 1y ago
  [flagged]
  - adamhartenz 1y ago
    That can't have felt great having your tantrum spotlighted by the author.
- vasco 1y ago
  It's not a paper or a journal but you could at least try to run a decent benchmark. As it is this serves no purpose other than reinforcing whatever point you started with. Didn't even tweak postgres buffers, literally what's the point.
  - dizzyVik 1y ago
    I still end up recommending using postgres though, don't I?
    - pcthrowaway 1y ago
      "I'll use postgres" was going to be your conclusion no matter what I guess? I mean what if an actual benchmark showed Redis is 100X as fast as postgres for a certain use case? What are the constraints you might be operating with? What are the characteristics of your workload? What are your budgetary constraints? Why not just write a blog post saying "Unoptimized postgres vs redis for the lazy, running virtualized with a bottleneck at the networking level" I even think that blog post would be interesting, and might be useful to someone choosing a stack for a proof of concept. For someone who to scale to large production workloads (~10,000 requests/second or more), this isn't a very useful article, so the criticism is fair, and I'm not sure why you're dismissing it off hand.
      - dizzyVik 1y ago
        I completely agree that this is not relevant for anyone running such workloads, the article is not aimed at them at all. Within the constraints of my setup, postgres came out slower but still fast enough. I don't think I can quantify what fast enough is though. Is it 1000 req/s? Is it 200? It all depends on what you're doing with it. For many of my hobby projects which see tens of requests per second it definitely is fast enough. You could argue that caching is indeed redundant in such cases, but some of those have quite a lot of data that takes a while to query.
      - motorest 1y ago
        > "I'll use postgres" was going to be your conclusion no matter what I guess? Would it bother you as well if the conclusion was rephrased as "based on my observations, I see no point in rearchitecting the system to improve the performance by this much"? I think you are too tied to a template solution that not only you don't stop to think why you're using it or even if it is justified at all. Then, when you are faced with observations that challenge your unfounded beliefs, you somehow opt to get defensive? That's not right.
    - vasco 1y ago
      That's the point, you put no effort and decided to do what you had decided already to do before.
- deleted 1y ago
  [deleted]
- a_c 1y ago
  I find your article valuable. It shows me what amount of configuration is needed for a reasonable expectation of performance. In real world, I’m not going to spend effort maxing out configuring a single piece of tool. Not being the most performing config on either of the tools is the least of my concern. Picking either of them, or as you suggested, Postgres, and then worry about getting one billion requests to the service is far more important
- lemagedurage 1y ago
  The main issue is that a reader might mistake Redis as a 2X faster postgres. Memory is 1000X faster than disk (SSD) and with network overhead Redis can still be 100X as fast as postgres for caching workloads. Otherwise, the article does well to show that we can get a lot of baseline performance either way. Sometimes a cache is premature optimisation.
  - phiresky 1y ago
    If your cache fits in Redis then it fits in RAM, if your cache fits in RAM then Postgres will serve it from RAM just as well. Writes will go to RAM as well if you have synchronous=off.
    - senorrib 1y ago
      Not necessarily true. If you're sharing the database with your transaction workload your cache will be paged out eventually.
      - jgalt212 1y ago
        This was my take as well, but I'm a MySQL / Redis shop. I really have no idea what tables MySQL has in RAM at any given moment, but with Redis I know what's in RAM.
  - motorest 1y ago
    > The main issue is that a reader might mistake Redis as a 2X faster postgres. Memory is 1000X faster than disk (SSD) and with network overhead Redis can still be 100X as fast as postgres for caching workloads. Your comments suggest that you are definitely missing some key insights onto the topic. If you, like the whole world, consume Redis through a network connection, it should be obvious to you that network is in fact the bottleneck. Furthermore, using a RDBMS like Postgres may indeed imply storing data in a slower memory. However, you are ignoring the obvious fact that a service such as Postgres also has its own memory cache, and some query results can and are indeed fetched from RAM. Thus it's not like each and every single query forces a disk read. And at the end of the day, what exactly is the performance tradeoff? And does it pay off to spend more on an in-memory cache like Redis to buy you the performance Delta? That's why real world benchmarks like this one are important. They help people think through the problem and reassess their irrational beliefs. You may nitpick about setup and configuration and test patterns and choice of libraries. What you cannot refute are the real world numbers. You may argue they could be better if this and that, but the real world numbers are still there.
- rollcat 1y ago
  Thank you for the article. My own conclusions from your data: - Under light workloads, you can get away with Postgres. 7k RPS is fine for a lot of stuff. - Introducing Redis into the mix has to be carefully weighted against increased architectural complexity, and having a common interface allows us to change that decision down the road. Yeah maybe that's not up to someone else's idea of a good synthetic benchmark. Do your load-testing against actual usage scenarios - spinning up an HTTP server to serve traffic is a step in the right direction. Kudos.
whateveracct 1y ago
most people with blogs don't know what they're doing. or don't care to know? sadly they get hired at companies and everyone does what they say cuz they have a blog. i've seen some shit in that department it's wild how much some people really are imposters.
- motorest 1y ago
  > most people with blogs don't know what they're doing. or don't care to know? I don't see any point to this blend of cynical contrarianism. If you feel you can do better, put your money where your mouth is. Lashing at others because they went through the trouble of sharing something they did is something that's absurd and creates no value. Also, maintaining a blog doesn't make anyone an expert, but not maintaining a blog doesn't mean you are suddenly more competent than those who do.
  - whateveracct 1y ago
    just an observation :)
motorest 1y ago
> He even admits Postgres maxxed out its 2 cores, but Redis was bottlenecked by the HTTP server. What exactly is your point? That you can further optimize either option? Well yes, that comes at no suprise. I mean, the latencies alone are in the range of some transcontinental requests. Were you surprised that Redis outperformed Postgres? I hardly think so. So what's the problem? The main point that's proven is that there is indeed diminishing returns in terms of performance. For applications where you can afford an extra 20ms when hitting a cache, caching using a persistent database is an option. For some people, it seems this fact was very surprising. That's food for thought, isn't it?
- hvb2 1y ago
  I've done this many times in AWS leveraging dynamodb. Comes with ttl support (which isn't precise so you still need to check expiration on read), and can support long TTLs as there's essentially no limit to the storage. All of this at a fraction of the cost of HA redis Only if you need that last millisecond of performance and have done all other optimizations should one consider redis imho
  - motorest 1y ago
    > I've done this many times in AWS leveraging dynamodb. Exactly. I think nosql offerings from any cloud provider already supports both TTL and conditional requests out-of-the-box, and the performance of basic key-value CRUD operations is often <10ms. I've seem some benchmarks advertise memory cache services as having latencies around 1ms. Yeah, this would mean the latency of a database is 10 times higher. But relative numbers matter nothing. What matters is absolute numbers, as they are the ones that drive tradeoff analysis. Does a feature afford an extra 10ms in latency, and is that performance improvement worth paying a premium?
  - re-thc 1y ago
    > All of this at a fraction of the cost of HA redis This depends on your scale. Dynamodb is pay per request and the scaling isn’t as smooth. At certain scales Redis is cheaper. Then if you don’t have high demand maybe it’s ok without HA for Redis and it can still be cheaper.
lelanthran 1y ago
I'm not seeing your point. This wouldn't get an F, purely because all the parameters are documented. Conclusions aren't incorrect either, so what's the problem?
- m000 1y ago
  The use case is not representative of a real-life scenario, so the value of the presented results are minimal. A takeaway could be that you can dedicate a postgres instance for caching and have acceptable results. But who does that? Even for a relatively simple intranet app, your #1 cost when deploying in Google Cloud would probably be running Postgres. Redis OTOH is dirt cheap.
  - lelanthran 1y ago
    > The use case is not representative of a real-life scenario, so the value of the presented results are minimal. Maybe I'm reading the article wrong, but it is representative of any application that uses a PosgreSQL server for data, correct? In what way is that not a real-life scenario? I've deployed Single monolith + PostgreSQL to about 8 different clients in the last 2.5 years. It's my largest source of income.
    - m000 1y ago
      When you run a relational database, you typically do it for the joins, aggregations, subqueries, etc. So a real-life scenario would include some application actually putting some stress on postgres. If your don't mind overprovisioning your postgres, yes I guess the presented benchmarks are kind of representative. But they also don't add anything that you didn't know without reading the article.
      - lelanthran 1y ago
        > If your don't mind overprovisioning your postgres Why would I mind it? I'm not using overpriced hosted PostgreSQL, after all.
      - 1y ago
ENGNR 1y ago
There’s too many hackers on hacker news!
zer00eyz 1y ago
You might not have been here 25 years ago when the dot com bubble burst. A lot of us ate shit to stay in the Bay Area, to stay in computing. I have stories of great engineers doing really crappy jobs and "contracting" on the side. I couldn't really have a 'startup' out of my house and a slice of rented hosting. Hardware was expensive and nothing was easy. Today I can set up a business and thrive on 1000 users at 10 bucks a month. Thats a viable and easy to build business. It's an achievable metric. But Im not going to let amazon and its infinite bill you for everything at 2012 prices so it can be profitable hosting be my first choice. Im not going to do that when I can get fixed cost hosting. For me, all the interesting things going on in tech aren't coming out of FB, Google and hyperscalers. They aren't AI or ML. We dont need another Kubernetes or Kafka or react (no more Conways law projects). There is more interesting work going on down at the bottom. In small 2 and 3 man shops solving their problems on limited time and budget with creative "next step" solutions. Their work is likely more applicable to most people reading HN than another well written engineering blog from cloud flare about their latest massive rust project.
positron26 1y ago
A lot of great benchmarking probably dies inside internal tuning. When we're lucky, we get a blog post, but if the creator isn't incentivized or is even discouraged by an employer from sharing the results, it will never see the light of day.
oulipo2 1y ago
The main point was not to fully benchmark and compare both, but just to get a rough sense of whether a Postgres cache was fast enough to be useful in practice. The comparison with Redis was more a crutch to get a sense of that, than really something that pretends to be "rock-solid benchmarking"
KronisLV 1y ago
I feel like the outrage is unwarranted. > The way it is presented, a casual reader would think Postgres is 2/3rds the performance of Redis. If a reader cares about the technical choice, they'll probably at least read enough to learn of the benchmarks in this popular use case, or even just the conclusion: > Redis is faster than postgres when it comes to caching, there’s no doubt about it. It conveniently comes with a bunch of other useful functionality that one would expect from a cache, such as TTLs. It was also bottlenecked by the hardware, my service or a combination of both and could definitely show better numbers. Surely, we should all use Redis for our caching needs then, right? Well, I think I’ll still use postgres. Almost always, my projects need a database. Not having to add another dependency comes with its own benefits. If I need my keys to expire, I’ll add a column for it, and a cron job to remove those keys from the table. As far as speed goes - 7425 requests per second is still a lot. That’s more than half a billion requests per day. All on hardware that’s 10 years old and using laptop CPUs. Not many projects will reach this scale and if they do I can just upgrade the postgres instance or if need be spin up a redis then. Having an interface for your cache so you can easily switch out the underlying store is definitely something I’ll keep doing exactly for this purpose. I might take an issue with the first sentence (might add "...at least when it comes to my hardware and configuration."), but the rest seems largely okay. As a casual reader, you more or less just get: * Oh hey, someone's experience and data points. I won't base my entire opinion upon it, but it's cool that people are sharing their experiences. * If I wanted to use either, I'd probably also need to look into bottlenecks, even the HTTP server, something you might not look into at first! * Even without putting in a lot of work into tuning, both of the solutions process a lot of data and are within an order of magnitude when it comes to performance. * So as a casual reader, for casual use cases, it seems like the answer is - just pick whatever feels the easiest. If I wanted to read super serious benchmarks, I'd go looking for those (which would also have so many details that they would no longer be a casual read, short of just the abstract, but them I'm missing out on a lot anyways), or do them myself. This is more like your average pop-sci article, nothing wrong with that, unless you're looking for something else. Eliminating the bottlenecks would be a cool followup post though!
lomase 1y ago
This site is called Hackernews btw.
novoreorx 1y ago
it's called blog for a reason

dizzyVik 1y ago

There's a reason this is on my blog and not a paper in a journal. This isn't supposed to show the absolute speed of either tool, the benchmark is not set up for that. I do state that redis has more performance on the table in the blog post.

jamesblonde 1y ago
[flagged]
- adamhartenz 1y ago
  That can't have felt great having your tantrum spotlighted by the author.
vasco 1y ago
It's not a paper or a journal but you could at least try to run a decent benchmark. As it is this serves no purpose other than reinforcing whatever point you started with. Didn't even tweak postgres buffers, literally what's the point.
- dizzyVik 1y ago
  I still end up recommending using postgres though, don't I?
  - pcthrowaway 1y ago
    "I'll use postgres" was going to be your conclusion no matter what I guess? I mean what if an actual benchmark showed Redis is 100X as fast as postgres for a certain use case? What are the constraints you might be operating with? What are the characteristics of your workload? What are your budgetary constraints? Why not just write a blog post saying "Unoptimized postgres vs redis for the lazy, running virtualized with a bottleneck at the networking level" I even think that blog post would be interesting, and might be useful to someone choosing a stack for a proof of concept. For someone who to scale to large production workloads (~10,000 requests/second or more), this isn't a very useful article, so the criticism is fair, and I'm not sure why you're dismissing it off hand.
    - dizzyVik 1y ago
      I completely agree that this is not relevant for anyone running such workloads, the article is not aimed at them at all. Within the constraints of my setup, postgres came out slower but still fast enough. I don't think I can quantify what fast enough is though. Is it 1000 req/s? Is it 200? It all depends on what you're doing with it. For many of my hobby projects which see tens of requests per second it definitely is fast enough. You could argue that caching is indeed redundant in such cases, but some of those have quite a lot of data that takes a while to query.
    - motorest 1y ago
      > "I'll use postgres" was going to be your conclusion no matter what I guess? Would it bother you as well if the conclusion was rephrased as "based on my observations, I see no point in rearchitecting the system to improve the performance by this much"? I think you are too tied to a template solution that not only you don't stop to think why you're using it or even if it is justified at all. Then, when you are faced with observations that challenge your unfounded beliefs, you somehow opt to get defensive? That's not right.
  - vasco 1y ago
    That's the point, you put no effort and decided to do what you had decided already to do before.
deleted 1y ago
[deleted]
a_c 1y ago
I find your article valuable. It shows me what amount of configuration is needed for a reasonable expectation of performance. In real world, I’m not going to spend effort maxing out configuring a single piece of tool. Not being the most performing config on either of the tools is the least of my concern. Picking either of them, or as you suggested, Postgres, and then worry about getting one billion requests to the service is far more important
lemagedurage 1y ago
The main issue is that a reader might mistake Redis as a 2X faster postgres. Memory is 1000X faster than disk (SSD) and with network overhead Redis can still be 100X as fast as postgres for caching workloads. Otherwise, the article does well to show that we can get a lot of baseline performance either way. Sometimes a cache is premature optimisation.
- phiresky 1y ago
  If your cache fits in Redis then it fits in RAM, if your cache fits in RAM then Postgres will serve it from RAM just as well. Writes will go to RAM as well if you have synchronous=off.
  - senorrib 1y ago
    Not necessarily true. If you're sharing the database with your transaction workload your cache will be paged out eventually.
    - jgalt212 1y ago
      This was my take as well, but I'm a MySQL / Redis shop. I really have no idea what tables MySQL has in RAM at any given moment, but with Redis I know what's in RAM.
- motorest 1y ago
  > The main issue is that a reader might mistake Redis as a 2X faster postgres. Memory is 1000X faster than disk (SSD) and with network overhead Redis can still be 100X as fast as postgres for caching workloads. Your comments suggest that you are definitely missing some key insights onto the topic. If you, like the whole world, consume Redis through a network connection, it should be obvious to you that network is in fact the bottleneck. Furthermore, using a RDBMS like Postgres may indeed imply storing data in a slower memory. However, you are ignoring the obvious fact that a service such as Postgres also has its own memory cache, and some query results can and are indeed fetched from RAM. Thus it's not like each and every single query forces a disk read. And at the end of the day, what exactly is the performance tradeoff? And does it pay off to spend more on an in-memory cache like Redis to buy you the performance Delta? That's why real world benchmarks like this one are important. They help people think through the problem and reassess their irrational beliefs. You may nitpick about setup and configuration and test patterns and choice of libraries. What you cannot refute are the real world numbers. You may argue they could be better if this and that, but the real world numbers are still there.
rollcat 1y ago
Thank you for the article. My own conclusions from your data: - Under light workloads, you can get away with Postgres. 7k RPS is fine for a lot of stuff. - Introducing Redis into the mix has to be carefully weighted against increased architectural complexity, and having a common interface allows us to change that decision down the road. Yeah maybe that's not up to someone else's idea of a good synthetic benchmark. Do your load-testing against actual usage scenarios - spinning up an HTTP server to serve traffic is a step in the right direction. Kudos.

whateveracct 1y ago

most people with blogs don't know what they're doing. or don't care to know? sadly they get hired at companies and everyone does what they say cuz they have a blog. i've seen some shit in that department it's wild how much some people really are imposters.

motorest 1y ago
> most people with blogs don't know what they're doing. or don't care to know? I don't see any point to this blend of cynical contrarianism. If you feel you can do better, put your money where your mouth is. Lashing at others because they went through the trouble of sharing something they did is something that's absurd and creates no value. Also, maintaining a blog doesn't make anyone an expert, but not maintaining a blog doesn't mean you are suddenly more competent than those who do.
- whateveracct 1y ago
  just an observation :)

motorest 1y ago

> He even admits Postgres maxxed out its 2 cores, but Redis was bottlenecked by the HTTP server. What exactly is your point? That you can further optimize either option? Well yes, that comes at no suprise. I mean, the latencies alone are in the range of some transcontinental requests. Were you surprised that Redis outperformed Postgres? I hardly think so. So what's the problem? The main point that's proven is that there is indeed diminishing returns in terms of performance. For applications where you can afford an extra 20ms when hitting a cache, caching using a persistent database is an option. For some people, it seems this fact was very surprising. That's food for thought, isn't it?

hvb2 1y ago
I've done this many times in AWS leveraging dynamodb. Comes with ttl support (which isn't precise so you still need to check expiration on read), and can support long TTLs as there's essentially no limit to the storage. All of this at a fraction of the cost of HA redis Only if you need that last millisecond of performance and have done all other optimizations should one consider redis imho
- motorest 1y ago
  > I've done this many times in AWS leveraging dynamodb. Exactly. I think nosql offerings from any cloud provider already supports both TTL and conditional requests out-of-the-box, and the performance of basic key-value CRUD operations is often <10ms. I've seem some benchmarks advertise memory cache services as having latencies around 1ms. Yeah, this would mean the latency of a database is 10 times higher. But relative numbers matter nothing. What matters is absolute numbers, as they are the ones that drive tradeoff analysis. Does a feature afford an extra 10ms in latency, and is that performance improvement worth paying a premium?
- re-thc 1y ago
  > All of this at a fraction of the cost of HA redis This depends on your scale. Dynamodb is pay per request and the scaling isn’t as smooth. At certain scales Redis is cheaper. Then if you don’t have high demand maybe it’s ok without HA for Redis and it can still be cheaper.

lelanthran 1y ago

I'm not seeing your point. This wouldn't get an F, purely because all the parameters are documented. Conclusions aren't incorrect either, so what's the problem?

m000 1y ago
The use case is not representative of a real-life scenario, so the value of the presented results are minimal. A takeaway could be that you can dedicate a postgres instance for caching and have acceptable results. But who does that? Even for a relatively simple intranet app, your #1 cost when deploying in Google Cloud would probably be running Postgres. Redis OTOH is dirt cheap.
- lelanthran 1y ago
  > The use case is not representative of a real-life scenario, so the value of the presented results are minimal. Maybe I'm reading the article wrong, but it is representative of any application that uses a PosgreSQL server for data, correct? In what way is that not a real-life scenario? I've deployed Single monolith + PostgreSQL to about 8 different clients in the last 2.5 years. It's my largest source of income.
  - m000 1y ago
    When you run a relational database, you typically do it for the joins, aggregations, subqueries, etc. So a real-life scenario would include some application actually putting some stress on postgres. If your don't mind overprovisioning your postgres, yes I guess the presented benchmarks are kind of representative. But they also don't add anything that you didn't know without reading the article.
    - lelanthran 1y ago
      > If your don't mind overprovisioning your postgres Why would I mind it? I'm not using overpriced hosted PostgreSQL, after all.
    - 1y ago

ENGNR 1y ago

There’s too many hackers on hacker news!

zer00eyz 1y ago

You might not have been here 25 years ago when the dot com bubble burst. A lot of us ate shit to stay in the Bay Area, to stay in computing. I have stories of great engineers doing really crappy jobs and "contracting" on the side. I couldn't really have a 'startup' out of my house and a slice of rented hosting. Hardware was expensive and nothing was easy. Today I can set up a business and thrive on 1000 users at 10 bucks a month. Thats a viable and easy to build business. It's an achievable metric. But Im not going to let amazon and its infinite bill you for everything at 2012 prices so it can be profitable hosting be my first choice. Im not going to do that when I can get fixed cost hosting. For me, all the interesting things going on in tech aren't coming out of FB, Google and hyperscalers. They aren't AI or ML. We dont need another Kubernetes or Kafka or react (no more Conways law projects). There is more interesting work going on down at the bottom. In small 2 and 3 man shops solving their problems on limited time and budget with creative "next step" solutions. Their work is likely more applicable to most people reading HN than another well written engineering blog from cloud flare about their latest massive rust project.

positron26 1y ago

A lot of great benchmarking probably dies inside internal tuning. When we're lucky, we get a blog post, but if the creator isn't incentivized or is even discouraged by an employer from sharing the results, it will never see the light of day.

oulipo2 1y ago

The main point was not to fully benchmark and compare both, but just to get a rough sense of whether a Postgres cache was fast enough to be useful in practice. The comparison with Redis was more a crutch to get a sense of that, than really something that pretends to be "rock-solid benchmarking"

KronisLV 1y ago

I feel like the outrage is unwarranted. > The way it is presented, a casual reader would think Postgres is 2/3rds the performance of Redis. If a reader cares about the technical choice, they'll probably at least read enough to learn of the benchmarks in this popular use case, or even just the conclusion: > Redis is faster than postgres when it comes to caching, there’s no doubt about it. It conveniently comes with a bunch of other useful functionality that one would expect from a cache, such as TTLs. It was also bottlenecked by the hardware, my service or a combination of both and could definitely show better numbers. Surely, we should all use Redis for our caching needs then, right? Well, I think I’ll still use postgres. Almost always, my projects need a database. Not having to add another dependency comes with its own benefits. If I need my keys to expire, I’ll add a column for it, and a cron job to remove those keys from the table. As far as speed goes - 7425 requests per second is still a lot. That’s more than half a billion requests per day. All on hardware that’s 10 years old and using laptop CPUs. Not many projects will reach this scale and if they do I can just upgrade the postgres instance or if need be spin up a redis then. Having an interface for your cache so you can easily switch out the underlying store is definitely something I’ll keep doing exactly for this purpose. I might take an issue with the first sentence (might add "...at least when it comes to my hardware and configuration."), but the rest seems largely okay. As a casual reader, you more or less just get: * Oh hey, someone's experience and data points. I won't base my entire opinion upon it, but it's cool that people are sharing their experiences. * If I wanted to use either, I'd probably also need to look into bottlenecks, even the HTTP server, something you might not look into at first! * Even without putting in a lot of work into tuning, both of the solutions process a lot of data and are within an order of magnitude when it comes to performance. * So as a casual reader, for casual use cases, it seems like the answer is - just pick whatever feels the easiest. If I wanted to read super serious benchmarks, I'd go looking for those (which would also have so many details that they would no longer be a casual read, short of just the abstract, but them I'm missing out on a lot anyways), or do them myself. This is more like your average pop-sci article, nothing wrong with that, unless you're looking for something else. Eliminating the bottlenecks would be a cool followup post though!

lomase 1y ago

This site is called Hackernews btw.

novoreorx 1y ago

it's called blog for a reason