4 ms·

There is a HUGE overhead when going through the network. Even if you don't (localhost), there's overhead when using TCP/IP. Even if you don't, there's a overhea

by gizzlon 15y ago

There is a HUGE overhead when going through the network. Even if you don't (localhost), there's overhead when using TCP/IP. Even if you don't, there's a overhead when using UNIX sockets or whatever you use for Inter-process-communication. It probably doesn't matter though.. And yes, Redis is very fast and you gain a lot when using it compared to just a Hash in the same process.

tfb 15y ago
So are you saying that because of the overhead involved with "talking" to redis, the fastest datastore would actually be an implementation of my own version (or a readily available version like node-lru-cache) of an LRU cache in my node app script - the datastore would essentially be a simple JSON object embedded directly in the script with methods specific to LRU data sets, gets, and backup?
- gizzlon 15y ago
  well, the point is only that In-process is the fastest there is. By using something like Redis you give up a some speed but gain features: ques, pub/sub etc etc.. But that doesn't mean you have to implement those features yourself, they might be available as a library for your language of choice as well. The bigger win, IMHO, is that you gain flexibility: Since Redis (or whatever) is decoupled from you process it can run on another processor, another machine or perhaps run on many machines etc.. Not sure how in-process cache would work in node, being async and all, but yes in-process is faster. But then you have to think about stuff like: - how do you avoid loosing everything when node crashes / restarts? - what if another process needs to read write to the cache? - what if you need more memory than a single machine provides (probably not going to happen). - implementation bugs In-process: Faster but probably harder to scale. But then again, it might be so fast you don't have to.
  - willvarfar 15y ago
    Would Redis evicting something from the LRU make some older mails unreadable?
    - gizzlon 15y ago
      Don't know.. But you make it sound like a bad thing. If you're out of memory you kind of have to evict stuff, don't you? I get the feeling you are kind of anti-Redis and I don't get why? Redis is a very cool project and could be useful for a lot of things.. It's not Redis' fault some people misuse it..
      - willvarfar 15y ago
        I am not at all anti-redis. Its not at all redis's fault it gets misused either; I say so that in the blog-post, even. So why would you compress something that you can only decompress if its recently-reused? How would you do mailinator with your strings in redis - and taking O(n) calls to redis to recover them to decompress an email where n is the number of lines (or consecutive lines, granted) in the email?
        gizzlon 15y ago
        "I am not at all anti-redis. Its not at all redis's fault it gets misused either; I say so that in the blog-post, even." Yeah, you actually do.. sorry. "So why would you compress something that you can only decompress if its recently-reused?" Not sure I understand your questions, and I've just started looking at Redis. But I guess you could do it the same way, but the added latency may make it infeasible. But the better answer is probably that you don't: You would modify the implementation to fit Redis' (or whatever) strength and weaknesses.
        willvarfar 15y ago
        Trying to work out how fit mailinator into Redis rather than questioning if Redis fits into mailinator is exactly the cargo-cult cool-kids zombism I was ranting against, though ;) You really can process mailinator- quantities of email with a simple Java server using a synchronized hash-map and linked list LRU and have some CPUs left over for CPU-intensive opportunistic LZMAing. Trying to do it with IPC TCP ping-pong for each and every line though; well I'm not sure you could process mailinator quantities of email within any reasonable hardware budget... Luckily you have a chance to see the error of your ways :)

by gizzlon 15y ago

tfb 15y ago

So are you saying that because of the overhead involved with "talking" to redis, the fastest datastore would actually be an implementation of my own version (or a readily available version like node-lru-cache) of an LRU cache in my node app script - the datastore would essentially be a simple JSON object embedded directly in the script with methods specific to LRU data sets, gets, and backup?

gizzlon 15y ago
well, the point is only that In-process is the fastest there is. By using something like Redis you give up a some speed but gain features: ques, pub/sub etc etc.. But that doesn't mean you have to implement those features yourself, they might be available as a library for your language of choice as well. The bigger win, IMHO, is that you gain flexibility: Since Redis (or whatever) is decoupled from you process it can run on another processor, another machine or perhaps run on many machines etc.. Not sure how in-process cache would work in node, being async and all, but yes in-process is faster. But then you have to think about stuff like: - how do you avoid loosing everything when node crashes / restarts? - what if another process needs to read write to the cache? - what if you need more memory than a single machine provides (probably not going to happen). - implementation bugs In-process: Faster but probably harder to scale. But then again, it might be so fast you don't have to.
- willvarfar 15y ago
  Would Redis evicting something from the LRU make some older mails unreadable?
  - gizzlon 15y ago
    Don't know.. But you make it sound like a bad thing. If you're out of memory you kind of have to evict stuff, don't you? I get the feeling you are kind of anti-Redis and I don't get why? Redis is a very cool project and could be useful for a lot of things.. It's not Redis' fault some people misuse it..
    - willvarfar 15y ago
      I am not at all anti-redis. Its not at all redis's fault it gets misused either; I say so that in the blog-post, even. So why would you compress something that you can only decompress if its recently-reused? How would you do mailinator with your strings in redis - and taking O(n) calls to redis to recover them to decompress an email where n is the number of lines (or consecutive lines, granted) in the email?
      - gizzlon 15y ago
        "I am not at all anti-redis. Its not at all redis's fault it gets misused either; I say so that in the blog-post, even." Yeah, you actually do.. sorry. "So why would you compress something that you can only decompress if its recently-reused?" Not sure I understand your questions, and I've just started looking at Redis. But I guess you could do it the same way, but the added latency may make it infeasible. But the better answer is probably that you don't: You would modify the implementation to fit Redis' (or whatever) strength and weaknesses.
        willvarfar 15y ago
        Trying to work out how fit mailinator into Redis rather than questioning if Redis fits into mailinator is exactly the cargo-cult cool-kids zombism I was ranting against, though ;) You really can process mailinator- quantities of email with a simple Java server using a synchronized hash-map and linked list LRU and have some CPUs left over for CPU-intensive opportunistic LZMAing. Trying to do it with IPC TCP ping-pong for each and every line though; well I'm not sure you could process mailinator quantities of email within any reasonable hardware budget... Luckily you have a chance to see the error of your ways :)