5 ms·

If the setup is that only one local process will use on-machine Redis as an in-memory cache, you're better off using the data structures available in your progr

If the setup is that only one local process will use on-machine Redis as an in-memory cache, you're better off using the data structures available in your programming language.

teaearlgraycold 2y ago
This is the vanillajs.com of data stores.
threeseed 2y ago
Not really. Because then once you restart that process you lose everything. And it's far more likely you are continuously upgrading your process than Redis.
- bastawhiz 2y ago
  If you need that, you can use an embedded data store like leveldb/rocksdb or sqlite. Why bring another application running as its own process into the equation?
  - nine_k 2y ago
    Running in its own process, and, better yet, in its own cgroup (container) makes potential bugs in it, including RCEs, harder to exploit. It also makes it easier to limit resources it consumes, monitor its functioning, etc. Upgrading it does not require you to rebuild and redeploy your app, which may be important if a bug or performance regression occurs is triggered, and you need a quick upgrade or downgrade with (near) zero downtime. Ideally every significant part should live in its own universe, only interacting with other parts via well-defined interfaces. Sadly, it's either more expensive (even as Unix processes, to say nothing of Windows), slower (Erlang / Elixir), or both (microservices).
    - LtWorf 2y ago
      At the cost of requiring a lot of IPC and memory copying.
  - 112233 2y ago
    But having separate applications can give you better perfomance and isolation than one single threaded process? Or do you compare using separate processes to having multiple threads in a single process?
    - bastawhiz 2y ago
      There's no universe where a single threaded embedded persistence implementation is slower than a single threaded application synchronously talking to a single threaded database over the network stack. As far as isolation goes, if you are worried about the properties of reading and writing data to the disk then I simply don't know what to tell you. Isolation from what?
      - 112233 2y ago
        Why network stack? On same host IPC over shared memory is a normal thing. Perfomance-wise, I do not know of a nice portable way of flushing changes to disk securely that does not block (like, e.g. fsync does). If you own the whole system and can tune whole kernel and userspace to run single application, sure, why overengineer. Otherwise software faults (bugs, crash due to memory overcommit, oom killer etc.) take down single process, and that can be less disruptive than full stop/start.
        bastawhiz 2y ago
        > On same host IPC over shared memory is a normal thing. Not for Redis. > I do not know of a nice portable way of flushing changes to disk securely that does not block (like, e.g. fsync does). If you use Redis, you're either not waiting for writes to be acknowledged or you're waiting on fsync. You always fsync no matter whether it's in process or not or you're risking losing data. Which process blocks doesn't affect performance, it's getting the data there in the first place. > Otherwise software faults (bugs, crash due to memory overcommit, oom killer etc.) take down single process, and that can be less disruptive than full stop/start. Even worse: Redis crashes and now your application (which hasn't crashed) can't read or write data, perhaps in the middle of ongoing operations. You have a whole new class of failure modes.
- throwaway81523 2y ago
  Either fork the process so the forked copy can dump its data (I think Redis itself does something like this), or launch a new process (with updated code if desired), then migrate the data and open sockets to it through some combination of unix domain sockets and maybe mmap. Or if we had OS's that really used x86 segmentation capability as it was designed for (this is one thing the 386 and later did cleverly) it could all be done with segments.
  - mcv 2y ago
    Now you're making it complicated again. I think a good rule of thumb is: * If you need it from a single process and it doesn't need to survive process restarts, just use basic in-process data structures. * If you need it from multiple processes or it needs to survive process restarts, but not system restarts, use redis. * If it needs persistence across system restarts, use a real DB.
    - throwaway81523 2y ago
      Redis is nice but you take a huge speed hit depending on what you're doing, compared to using in-memory structures. Note that Redis can also persist to disk, either by forking or by writing an append-only op log that can be replayed for reloading or replication. Anyway, you'veve forgotten the case where you want not only the data structures, but also the open network connections to persist across process restarts. That's what passing file descriptors through unix domain sockets lets you do. It gets you the equivalent of Erlang's hot upgrades.
  - xxs 2y ago
    >I think Redis itself does something like this) It does... and it's a blocking operation (Bad). It's used when storing data to a file to allow restarts w/o losing the said data.
    - throwaway81523 2y ago
      Do you mean fork is a blocking operation? That's surprising but hmm ok. I didn't realize that. I thought that the process's pages all became copy-on-write, so that page updates could trigger faults handled by in-memory operations. Maybe those could be considered blocking (I hadn't thought of it like that) or maybe you mean something different?
zamalek 2y ago
Redis can cluster alongside your backend cluster, so you'll have the advantage that sometimes the data will be local.
baq 2y ago
Why? Most languages are less memory efficient than redis, even in a separate process, and I don’t want to reimplement half of its features.
- cstadnij 2y ago
  Because it's really, really slow compared to in-language variables? I've not yet ran out of memory because of program variables, and speed shouldn't be an argument for low memory footprint when it's already at best 1/10 of it's performance.
  - speedgoose 2y ago
    Redis locally is good to do IPC though. You have many scenarios where multi local processes have to exchange data. You have alternatives but Redis works and can scale up.
- deleted 2y ago
  [deleted]
- mgraczyk 2y ago
  You can do a mutex in like 10 bytes, how could redis be more memory efficient?
  - baq 2y ago
    you are using the minority language if that's the case for you.
bryanrasmussen 2y ago
all the other reasons beside - because at some point I might want to move the data cache off of the single machine and then I can do that with a couple of minutes work, yes then I am back to do I need redis why not use postgres, but I got months of better performance out of redis than I would have with postgres, if I need to move to separate machine I am probably doing good enough I can afford it. Redis helps you get to the point (need cache on separate machines) where you might want to consider moving to postgres, or not because you don't feel it warrants the investment of programmer hours at that point. on edit: got downvoted for ..? I guess what I said was so supremely stupid I deserved it. I do admit it was perhaps slightly obvious, but not so obvious that it warranted downvoting
- ryandvm 2y ago
  Don't build today what you might need tomorrow.
  - bryanrasmussen 2y ago
    the comment was quite clear that you are building today what you need today, and when you need to have another solution tomorrow you don't need to rebuild everything but just move it and update some connections - but if you build the other solution - keeping cache in the application - when you need the other solution you will need to rebuild everything. How could that very clear statement be misunderstood as to be well I am building a bunch of stuff I might need to tomorrow, it is the exact opposite of that! I guess it's my fault, nobody on HN seems to understand a single thing I say even though the words seem reasonably clear to me.
edanm 2y ago
Yes, that's assuming a single local process. Redis bills itself as an in-memory data structure store, and at least in my memory, it was mainly pitched for doing interprocess communication with real data structures.
jvanderbot 2y ago
Unless you need persistence across process respawning, ipc across lots of subprocesses, or want to trivially inspect and namespace your entire in memory data. But yeah redis is not an answer for most single machine questions.

If the setup is that only one local process will use on-machine Redis as an in-memory cache, you're better off using the data structures available in your programming language.

teaearlgraycold 2y ago

This is the vanillajs.com of data stores.

threeseed 2y ago

Not really. Because then once you restart that process you lose everything. And it's far more likely you are continuously upgrading your process than Redis.

bastawhiz 2y ago
If you need that, you can use an embedded data store like leveldb/rocksdb or sqlite. Why bring another application running as its own process into the equation?
- nine_k 2y ago
  Running in its own process, and, better yet, in its own cgroup (container) makes potential bugs in it, including RCEs, harder to exploit. It also makes it easier to limit resources it consumes, monitor its functioning, etc. Upgrading it does not require you to rebuild and redeploy your app, which may be important if a bug or performance regression occurs is triggered, and you need a quick upgrade or downgrade with (near) zero downtime. Ideally every significant part should live in its own universe, only interacting with other parts via well-defined interfaces. Sadly, it's either more expensive (even as Unix processes, to say nothing of Windows), slower (Erlang / Elixir), or both (microservices).
  - LtWorf 2y ago
    At the cost of requiring a lot of IPC and memory copying.
- 112233 2y ago
  But having separate applications can give you better perfomance and isolation than one single threaded process? Or do you compare using separate processes to having multiple threads in a single process?
  - bastawhiz 2y ago
    There's no universe where a single threaded embedded persistence implementation is slower than a single threaded application synchronously talking to a single threaded database over the network stack. As far as isolation goes, if you are worried about the properties of reading and writing data to the disk then I simply don't know what to tell you. Isolation from what?
    - 112233 2y ago
      Why network stack? On same host IPC over shared memory is a normal thing. Perfomance-wise, I do not know of a nice portable way of flushing changes to disk securely that does not block (like, e.g. fsync does). If you own the whole system and can tune whole kernel and userspace to run single application, sure, why overengineer. Otherwise software faults (bugs, crash due to memory overcommit, oom killer etc.) take down single process, and that can be less disruptive than full stop/start.
      - bastawhiz 2y ago
        > On same host IPC over shared memory is a normal thing. Not for Redis. > I do not know of a nice portable way of flushing changes to disk securely that does not block (like, e.g. fsync does). If you use Redis, you're either not waiting for writes to be acknowledged or you're waiting on fsync. You always fsync no matter whether it's in process or not or you're risking losing data. Which process blocks doesn't affect performance, it's getting the data there in the first place. > Otherwise software faults (bugs, crash due to memory overcommit, oom killer etc.) take down single process, and that can be less disruptive than full stop/start. Even worse: Redis crashes and now your application (which hasn't crashed) can't read or write data, perhaps in the middle of ongoing operations. You have a whole new class of failure modes.
throwaway81523 2y ago
Either fork the process so the forked copy can dump its data (I think Redis itself does something like this), or launch a new process (with updated code if desired), then migrate the data and open sockets to it through some combination of unix domain sockets and maybe mmap. Or if we had OS's that really used x86 segmentation capability as it was designed for (this is one thing the 386 and later did cleverly) it could all be done with segments.
- mcv 2y ago
  Now you're making it complicated again. I think a good rule of thumb is: * If you need it from a single process and it doesn't need to survive process restarts, just use basic in-process data structures. * If you need it from multiple processes or it needs to survive process restarts, but not system restarts, use redis. * If it needs persistence across system restarts, use a real DB.
  - throwaway81523 2y ago
    Redis is nice but you take a huge speed hit depending on what you're doing, compared to using in-memory structures. Note that Redis can also persist to disk, either by forking or by writing an append-only op log that can be replayed for reloading or replication. Anyway, you'veve forgotten the case where you want not only the data structures, but also the open network connections to persist across process restarts. That's what passing file descriptors through unix domain sockets lets you do. It gets you the equivalent of Erlang's hot upgrades.
- xxs 2y ago
  >I think Redis itself does something like this) It does... and it's a blocking operation (Bad). It's used when storing data to a file to allow restarts w/o losing the said data.
  - throwaway81523 2y ago
    Do you mean fork is a blocking operation? That's surprising but hmm ok. I didn't realize that. I thought that the process's pages all became copy-on-write, so that page updates could trigger faults handled by in-memory operations. Maybe those could be considered blocking (I hadn't thought of it like that) or maybe you mean something different?

zamalek 2y ago

Redis can cluster alongside your backend cluster, so you'll have the advantage that sometimes the data will be local.

baq 2y ago

Why? Most languages are less memory efficient than redis, even in a separate process, and I don’t want to reimplement half of its features.

cstadnij 2y ago
Because it's really, really slow compared to in-language variables? I've not yet ran out of memory because of program variables, and speed shouldn't be an argument for low memory footprint when it's already at best 1/10 of it's performance.
- speedgoose 2y ago
  Redis locally is good to do IPC though. You have many scenarios where multi local processes have to exchange data. You have alternatives but Redis works and can scale up.
deleted 2y ago
[deleted]
mgraczyk 2y ago
You can do a mutex in like 10 bytes, how could redis be more memory efficient?
- baq 2y ago
  you are using the minority language if that's the case for you.

bryanrasmussen 2y ago

all the other reasons beside - because at some point I might want to move the data cache off of the single machine and then I can do that with a couple of minutes work, yes then I am back to do I need redis why not use postgres, but I got months of better performance out of redis than I would have with postgres, if I need to move to separate machine I am probably doing good enough I can afford it. Redis helps you get to the point (need cache on separate machines) where you might want to consider moving to postgres, or not because you don't feel it warrants the investment of programmer hours at that point. on edit: got downvoted for ..? I guess what I said was so supremely stupid I deserved it. I do admit it was perhaps slightly obvious, but not so obvious that it warranted downvoting

ryandvm 2y ago
Don't build today what you might need tomorrow.
- bryanrasmussen 2y ago
  the comment was quite clear that you are building today what you need today, and when you need to have another solution tomorrow you don't need to rebuild everything but just move it and update some connections - but if you build the other solution - keeping cache in the application - when you need the other solution you will need to rebuild everything. How could that very clear statement be misunderstood as to be well I am building a bunch of stuff I might need to tomorrow, it is the exact opposite of that! I guess it's my fault, nobody on HN seems to understand a single thing I say even though the words seem reasonably clear to me.

edanm 2y ago

Yes, that's assuming a single local process. Redis bills itself as an in-memory data structure store, and at least in my memory, it was mainly pitched for doing interprocess communication with real data structures.

jvanderbot 2y ago

Unless you need persistence across process respawning, ipc across lots of subprocesses, or want to trivially inspect and namespace your entire in memory data. But yeah redis is not an answer for most single machine questions.