5 ms·

I had a project where I just needed to be saving a constant flow of data (id + 100 bytes records) somewhere, no persistence needed, no sharding (single machine)

by amenod 6y ago

I had a project where I just needed to be saving a constant flow of data (id + 100 bytes records) somewhere, no persistence needed, no sharding (single machine) - but lots of it. I first turned to Redis and it worked nicely, but since I had PostgreSQL running, I made a performance test with it too. To my surprise, once I optimized both (memory only, pg WAL disabled, persistent connections,...) there was almost no difference in insert rate, so I ended up using PgSQL only. This is anecdotal, of course, and it is possible that I simply didn't optimize Redis enough (though I did give it my best!). Let me also say that Redis is great otherwise, it was just in this specific case that PostgreSQL surprised me and fulfilled the need itself.

HelloNurse 6y ago
Your data-saving test was probably limited by how fast you could save to disk a roughly equal amount of data with the two databases; queries might work and perform more differently.
- amenod 6y ago
  That's what I meant by "no persistence needed". In both cases, I wasn't actually saving anything to disk. I needed an in-memory storage for a never-ending stream of incoming data that would be read (and purged) by a separate process, no biggie if the data got lost occasionally. I also simply inserted and read the data, no complex queries or anything. Key-value store is all I needed and for this particular case PostgreSQL was as good at it as Redis.
  - kinos 6y ago
    Why did you use a database instead of just creating a global map and running in memory? Sharding?
    - amenod 6y ago
      No - different processes were reading and writing. And once you implement a global map which is accessible from multiple processes, you might as well use an existing solution, like Redis. Or, as I've learned, pgSQL - which gave me the same performance with less moving parts (because I was already using it) when I disabled WAL.

by amenod 6y ago

HelloNurse 6y ago

Your data-saving test was probably limited by how fast you could save to disk a roughly equal amount of data with the two databases; queries might work and perform more differently.

amenod 6y ago
That's what I meant by "no persistence needed". In both cases, I wasn't actually saving anything to disk. I needed an in-memory storage for a never-ending stream of incoming data that would be read (and purged) by a separate process, no biggie if the data got lost occasionally. I also simply inserted and read the data, no complex queries or anything. Key-value store is all I needed and for this particular case PostgreSQL was as good at it as Redis.
- kinos 6y ago
  Why did you use a database instead of just creating a global map and running in memory? Sharding?
  - amenod 6y ago
    No - different processes were reading and writing. And once you implement a global map which is accessible from multiple processes, you might as well use an existing solution, like Redis. Or, as I've learned, pgSQL - which gave me the same performance with less moving parts (because I was already using it) when I disabled WAL.