5 ms·

The blog reiterates that using Redis as your primary database is totally possible, and a good idea even. I think that's the biggest Redis anti-pattern out there

by cmckn 5y ago

The blog reiterates that using Redis as your primary database is totally possible, and a good idea even. I think that's the biggest Redis anti-pattern out there. I would never put data in Redis that I wasn't comfortable losing at some point. There are way too many other perfectly good, OSS options designed for durable storage (and the access patterns that come along with "primary databases").

paxys 5y ago
People always repeat this but I have never understood why. Is there any reason to believe that a well configured and maintained instance of Redis is more prone to data loss than a similar setup of any traditional database software?
- tomnipotent 5y ago
  > Is there any reason to believe Yes, by the very nature of how redis implements persistence (RDF snapshot format + AOF) compared to traditional btree-backed storage with WALs.
  - samhw 5y ago
    Could you explain why that has a bearing on the risk of data loss? Note that AOF is a WAL. (Also, I think you mean RDB – unless you're talking about something else entirely of which I'm unaware?)
    - tomnipotent 5y ago
      > Note that AOF is a WAL The redis AOF is written to after the command is executed and in-memory data is modified. It is not a WAL.
      - samhw 5y ago
        Ah, fair enough, that's true. It's written to the AOF before processing the next query (https://github.com/redis/redis/blob/unstable/src/server.c#L4396 https://github.com/redis/redis/blob/unstable/src/server.c#L4...), so I believe you get read-your-writes consistency per node (in terms of the AOF), but I suppose it is technically written afterwards rather than before.
- hardwaresofton 5y ago
  > well configured and maintained Well there's the rub -- table stakes for traditional database software (the "boring technology") is not losing your data. Redis has AOF which can be tuned and this SO post[0] is a good intro to how they work and the different settings involved. Pieces of software like Postgres have settings that get you to dont-lose-my-data on day 1. Also people generally rant and rave about the performance of Redis when AOF-write-on-every-write (aka WAL) is not turned on. It also kind of annoys me how AOF is caleld AOF when there is a very good well known term for this already -- WAL. [0]: https://stackoverflow.com/questions/40939756/difference-between-redis-aof-and-tarantool-wal-log https://stackoverflow.com/questions/40939756/difference-betw...
  - tomnipotent 5y ago
    The redis AOF is a post-write log, so it's not a WAL.
    - hardwaresofton 5y ago
      Ah thanks for the correction, it's not a Write Ahead Log at all, and they made the right choice in not overloading it to mean Write After or anything silly like that. That irk was over nothing. Given that being true then there is no way to run Redis without the spectre of data loss, period -- just a question of how much you are OK with possibly losing (which to be fair is probably fairly small with most healthy systems).
- deleted 5y ago
  [deleted]
- deleted 5y ago
  [deleted]
- rakoo 5y ago
  Redis as a main store is acceptable if you are ready to lose all data between two writes to disk, and you don't care about when data is actually written. It might be ok if all you're storing is likes and votes, but probably not if you're storing posts and comments.
  - spaceheater 5y ago
    Not a db expert. How would such failure occur? How other db would handle it? It's not like other db's don't get corrupted, you can search `database-name corrupted` and find people trying to get help, usually the suggestion is use a backup instead of trying to fix it, so what does it matter how often it fsyncs?. Would PostgreSQL help me in OVH data center fire? What does it matter if it writes before or after, the op was either success or failure, and then the server crashed with whatever state was in RAM (which is moot by now, since the server is down).
    - dikei 5y ago
      It's all about minimizing the risk of data loss. The risk is always there, but some DB are much safer than other. For example, Server crash is a much rarer issue than process restart. In Redis, if the process supervisors (systemd, docker, k8s, etc..) decide to restart your Redis process for whatever reasons, you risk losing ACKed data, because Redis ACKed before the data is written to disk. In a safer database system, the data is not ACKed until it can safely survive a process restart. Furthermore, many database systems can be configured so that the data is safely persisted in multiple servers before ACKing, and therefore survive even a server lost incident.

by cmckn 5y ago

paxys 5y ago

People always repeat this but I have never understood why. Is there any reason to believe that a well configured and maintained instance of Redis is more prone to data loss than a similar setup of any traditional database software?

tomnipotent 5y ago
> Is there any reason to believe Yes, by the very nature of how redis implements persistence (RDF snapshot format + AOF) compared to traditional btree-backed storage with WALs.
- samhw 5y ago
  Could you explain why that has a bearing on the risk of data loss? Note that AOF is a WAL. (Also, I think you mean RDB – unless you're talking about something else entirely of which I'm unaware?)
  - tomnipotent 5y ago
    > Note that AOF is a WAL The redis AOF is written to after the command is executed and in-memory data is modified. It is not a WAL.
    - samhw 5y ago
      Ah, fair enough, that's true. It's written to the AOF before processing the next query (https://github.com/redis/redis/blob/unstable/src/server.c#L4396 https://github.com/redis/redis/blob/unstable/src/server.c#L4...), so I believe you get read-your-writes consistency per node (in terms of the AOF), but I suppose it is technically written afterwards rather than before.
hardwaresofton 5y ago
> well configured and maintained Well there's the rub -- table stakes for traditional database software (the "boring technology") is not losing your data. Redis has AOF which can be tuned and this SO post[0] is a good intro to how they work and the different settings involved. Pieces of software like Postgres have settings that get you to dont-lose-my-data on day 1. Also people generally rant and rave about the performance of Redis when AOF-write-on-every-write (aka WAL) is not turned on. It also kind of annoys me how AOF is caleld AOF when there is a very good well known term for this already -- WAL. [0]: https://stackoverflow.com/questions/40939756/difference-between-redis-aof-and-tarantool-wal-log https://stackoverflow.com/questions/40939756/difference-betw...
- tomnipotent 5y ago
  The redis AOF is a post-write log, so it's not a WAL.
  - hardwaresofton 5y ago
    Ah thanks for the correction, it's not a Write Ahead Log at all, and they made the right choice in not overloading it to mean Write After or anything silly like that. That irk was over nothing. Given that being true then there is no way to run Redis without the spectre of data loss, period -- just a question of how much you are OK with possibly losing (which to be fair is probably fairly small with most healthy systems).
deleted 5y ago
[deleted]
deleted 5y ago
[deleted]
rakoo 5y ago
Redis as a main store is acceptable if you are ready to lose all data between two writes to disk, and you don't care about when data is actually written. It might be ok if all you're storing is likes and votes, but probably not if you're storing posts and comments.
- spaceheater 5y ago
  Not a db expert. How would such failure occur? How other db would handle it? It's not like other db's don't get corrupted, you can search `database-name corrupted` and find people trying to get help, usually the suggestion is use a backup instead of trying to fix it, so what does it matter how often it fsyncs?. Would PostgreSQL help me in OVH data center fire? What does it matter if it writes before or after, the op was either success or failure, and then the server crashed with whatever state was in RAM (which is moot by now, since the server is down).
  - dikei 5y ago
    It's all about minimizing the risk of data loss. The risk is always there, but some DB are much safer than other. For example, Server crash is a much rarer issue than process restart. In Redis, if the process supervisors (systemd, docker, k8s, etc..) decide to restart your Redis process for whatever reasons, you risk losing ACKed data, because Redis ACKed before the data is written to disk. In a safer database system, the data is not ACKed until it can safely survive a process restart. Furthermore, many database systems can be configured so that the data is safely persisted in multiple servers before ACKing, and therefore survive even a server lost incident.