3 ms·

great explanation by the Postgres crew. it makes me wonder, though: if I had a table with 50k rows, updated hundreds of times per second and used in joins thro

great explanation by the Postgres crew. it makes me wonder, though: if I had a table with 50k rows, updated hundreds of times per second and used in joins throughout the database, is there any way I can just stick that whole table into memcached or redis? I know there are some cases where this works, some where it doesn't. curious if this option was explored.

lemcoe9 10y ago
Is this not what Redis was designed for? In this situation, I would value Redis over memcached just based on its performance values and the cheapness of RAM in the cloud right now.
bsaul 10y ago
Completely agreee. Highly mutable => in memory ( then add a secondary stream for colder storage backups just in case). A relational db is originaly meant for fetching data from a slow storage to a fast one, and the other way around, in a smart way. Doing it 500 times a second isn't a scenario for any db. Build your own in memory data + process tructure, maybe using something like and agent network and using eg akka or erlang for failovers. I'm curious as to why uber had to rely on a relational db for this case...
- tylermauthe 10y ago
  It's often simpler to keep data in a relational DB for reporting, especially if you've got data lake tooling -- it's just easier to get wider insights if your data is stored relationally. I know it's possible to do this without relational DB, but you get it for free with SQL. As you say though, put it in memory first and write it out to a DB every now and then.
qopp 10y ago
Just to keep it in memory? Postgres does this already-- it will put tables or parts of tables in memory to speed up reads [1]. [1] Source: https://momjian.us/main/writings/pgsql/hw_performance/ https://momjian.us/main/writings/pgsql/hw_performance/
- cobbzilla 10y ago
  It's not about "just" keeping it in memory, it's about handling hundreds of updates per second without creating un-vacuumable bloat (per description in OP's link).
taurath 10y ago
I'd really like to know why they DIDN'T consider something like Redis for this - if I'm thinking of my own apps where I just take redis==sessions for granted, why would they make a technology change of this magnitude in order to cover a usecase like that? Maybe there was a lot more?

lemcoe9 10y ago

Is this not what Redis was designed for? In this situation, I would value Redis over memcached just based on its performance values and the cheapness of RAM in the cloud right now.

bsaul 10y ago

Completely agreee. Highly mutable => in memory ( then add a secondary stream for colder storage backups just in case). A relational db is originaly meant for fetching data from a slow storage to a fast one, and the other way around, in a smart way. Doing it 500 times a second isn't a scenario for any db. Build your own in memory data + process tructure, maybe using something like and agent network and using eg akka or erlang for failovers. I'm curious as to why uber had to rely on a relational db for this case...

tylermauthe 10y ago
It's often simpler to keep data in a relational DB for reporting, especially if you've got data lake tooling -- it's just easier to get wider insights if your data is stored relationally. I know it's possible to do this without relational DB, but you get it for free with SQL. As you say though, put it in memory first and write it out to a DB every now and then.

qopp 10y ago

Just to keep it in memory? Postgres does this already-- it will put tables or parts of tables in memory to speed up reads [1]. [1] Source: https://momjian.us/main/writings/pgsql/hw_performance/ https://momjian.us/main/writings/pgsql/hw_performance/

cobbzilla 10y ago
It's not about "just" keeping it in memory, it's about handling hundreds of updates per second without creating un-vacuumable bloat (per description in OP's link).

taurath 10y ago

I'd really like to know why they DIDN'T consider something like Redis for this - if I'm thinking of my own apps where I just take redis==sessions for granted, why would they make a technology change of this magnitude in order to cover a usecase like that? Maybe there was a lot more?