5 ms·

The bigger they are, the harder they f(etch)all

16 points by itamarhaber 12y ago

andrewvc 12y ago
Doesn't this point to a design issue with redis? The single threaded, blocking operation of it works, but it seems to present more problems than it solves for a large quantity of use cases. While it works, and works quickly for a narrow range of use cases, it comes at a huge loss in flexibility when compared to an SQL DB. While I think it is possible to use redis well, I'm at a loss for what redis does for the average user, that a decent use of SQL doesn't do. My gut feeling is that if people would spend 30 minutes reading up on how to read a postgres query plan maybe they wouldn't reach for redis so quickly.
- itamarhaber 12y ago
  I don't think single- vs. multi-threaded is not relevant in this context. Any database, regardless its architectural design choices, will probably have a hard time if you try process this much information. As for when to use SQL and when not to, well, that really depends on the problems that you're trying to solve, your preferences and a whole myriad of other considerations. I fully agree there are tasks that can be handled well by more than one solution, while others are best tackled with specialized tools.
  - andrewvc 12y ago
    It is very relevant because with a multithreaded DB you can survey your pkeys while only minimally interfering with the day to day operations of the database. I understand that it's a judgement call, I'm merely asserting that most people make the wrong judgement because they want to play with new tech.
    - thezilch 12y ago
      You can do the same with Redis. Regardless of the fact you can simply hang a Redis slave and work on that; Redis has a *SCAN commands that can incrementally iterate over your entire keyspace or keys of any Redis collection (hashes, sets, etc).
- lmm 12y ago
  For a simple "fetch one row by pkey" lookup a traditional SQL DB spends something like 2/3 of the execution time parsing the SQL. So using a binary protocol's always going to win on performance. (You may be right, maybe the main value of redis is as a way to work around stupid corporate/DBA policies? I know in theory I could have a single table key / json value in Postgres, but I've never found a company willing to actually do that. Whereas in Redis it's "no, I can't define a schema for the data, because the tool doesn't have one.")
  - andrewvc 12y ago
    You are 100% right about the perf there, but most people don't really need to optimize those lookups, they just like to because engineers like shiny things.
    - lmm 12y ago
      I think it's more that engineers are (rightly) lazy. Redis feels very simple - install it, run it, do your key/value operations.
- Zikes 12y ago
  No database system is perfect for all use cases, that's why we have options.
  - andrewvc 12y ago
    I agree, I just think redis is mostly used as a premature optimization
- damon_c 12y ago
  Redis offers several data structures that make it far simpler a solution than SQL for certain uses. One example is its lists: http://redis.io/commands#list http://redis.io/commands#list
  - andrewvc 12y ago
    I'm actually quite familiar with Redis, but most of the commands aren't meaningfully more simple. Redis' greatest advantage is performance.
dfc 12y ago
This is a first: a title change that is dramatically less informative than the original title. Please do not change the title of submissions. The guidelines are fairly clear on the policy and all you end up doing is making more work for "the ever vigilant deity of community goodness and truth in titling," aka dang. > please use the original title, unless it is misleading or linkbait. http://ycombinator.com/newsguidelines.html http://ycombinator.com/newsguidelines.html
- itamarhaber 12y ago
  Sorry - no intent to mislead here... we were debating internally between the two titles and although we finally opted for the 500M one, I guess I liked this one better and it stuck in my subconscious, hence the submission. The original title: I Have 500 Million Keys But What’s In My Redis DB?!?
euphemize 12y ago
From the example: Size range 0-49 600-649 ... Why does the range skip from 0-49 to 600-649? I'm not sure I get this.
- mordocai 12y ago
  I'd assume, without further information, that the random samples the script grabbed didn't have any sizes in range 50-599 inclusive. While that seems a bit odd, it doesn't seem impossible to me.
- itamarhaber 12y ago
  An alternate explanation (and in this case also the real one :)) is that the database doesn't have any keys at that specific size range.
  - euphemize 12y ago
    Ah! heh, thanks for the clarification - I somehow expected it to have a row, with a count "0" :)

16 points by itamarhaber 12y ago

Doesn't this point to a design issue with redis? The single threaded, blocking operation of it works, but it seems to present more problems than it solves for a large quantity of use cases. While it works, and works quickly for a narrow range of use cases, it comes at a huge loss in flexibility when compared to an SQL DB. While I think it is possible to use redis well, I'm at a loss for what redis does for the average user, that a decent use of SQL doesn't do. My gut feeling is that if people would spend 30 minutes reading up on how to read a postgres query plan maybe they wouldn't reach for redis so quickly.

itamarhaber 12y ago
I don't think single- vs. multi-threaded is not relevant in this context. Any database, regardless its architectural design choices, will probably have a hard time if you try process this much information. As for when to use SQL and when not to, well, that really depends on the problems that you're trying to solve, your preferences and a whole myriad of other considerations. I fully agree there are tasks that can be handled well by more than one solution, while others are best tackled with specialized tools.
- andrewvc 12y ago
  It is very relevant because with a multithreaded DB you can survey your pkeys while only minimally interfering with the day to day operations of the database. I understand that it's a judgement call, I'm merely asserting that most people make the wrong judgement because they want to play with new tech.
  - thezilch 12y ago
    You can do the same with Redis. Regardless of the fact you can simply hang a Redis slave and work on that; Redis has a *SCAN commands that can incrementally iterate over your entire keyspace or keys of any Redis collection (hashes, sets, etc).
lmm 12y ago
For a simple "fetch one row by pkey" lookup a traditional SQL DB spends something like 2/3 of the execution time parsing the SQL. So using a binary protocol's always going to win on performance. (You may be right, maybe the main value of redis is as a way to work around stupid corporate/DBA policies? I know in theory I could have a single table key / json value in Postgres, but I've never found a company willing to actually do that. Whereas in Redis it's "no, I can't define a schema for the data, because the tool doesn't have one.")
- andrewvc 12y ago
  You are 100% right about the perf there, but most people don't really need to optimize those lookups, they just like to because engineers like shiny things.
  - lmm 12y ago
    I think it's more that engineers are (rightly) lazy. Redis feels very simple - install it, run it, do your key/value operations.
Zikes 12y ago
No database system is perfect for all use cases, that's why we have options.
- andrewvc 12y ago
  I agree, I just think redis is mostly used as a premature optimization
damon_c 12y ago
Redis offers several data structures that make it far simpler a solution than SQL for certain uses. One example is its lists: http://redis.io/commands#list http://redis.io/commands#list
- andrewvc 12y ago
  I'm actually quite familiar with Redis, but most of the commands aren't meaningfully more simple. Redis' greatest advantage is performance.

dfc 12y ago

This is a first: a title change that is dramatically less informative than the original title. Please do not change the title of submissions. The guidelines are fairly clear on the policy and all you end up doing is making more work for "the ever vigilant deity of community goodness and truth in titling," aka dang. > please use the original title, unless it is misleading or linkbait. http://ycombinator.com/newsguidelines.html http://ycombinator.com/newsguidelines.html

itamarhaber 12y ago
Sorry - no intent to mislead here... we were debating internally between the two titles and although we finally opted for the 500M one, I guess I liked this one better and it stuck in my subconscious, hence the submission. The original title: I Have 500 Million Keys But What’s In My Redis DB?!?

euphemize 12y ago

From the example: Size range 0-49 600-649 ... Why does the range skip from 0-49 to 600-649? I'm not sure I get this.

mordocai 12y ago
I'd assume, without further information, that the random samples the script grabbed didn't have any sizes in range 50-599 inclusive. While that seems a bit odd, it doesn't seem impossible to me.
itamarhaber 12y ago
An alternate explanation (and in this case also the real one :)) is that the database doesn't have any keys at that specific size range.
- euphemize 12y ago
  Ah! heh, thanks for the clarification - I somehow expected it to have a row, with a count "0" :)