4 ms·

>What could possibly be causing it to use that much memory? I considered the length of my keys...Maybe Redis was eating memory by storing tens of thousands of r

>What could possibly be causing it to use that much memory? I considered the length of my keys...Maybe Redis was eating memory by storing tens of thousands of really long key names in RAM? I decided to try shortening them to a more reasonable format: u:1234:lb for example. Wow ... why would you reengineer your naming conventions without at least doing some back-of-a-napkin arithmetic beforehand and compare the potential savings? That strikes me as ... insane.

davidcelis 14y ago
> Optimization is a rabbithole I’ve not had to go down very often. I am hardly an expert. But really, the answer to this is because it's easy to copy a database in Redis and then just do a mass RENAME. It took all of two minutes with a Ruby script and Redis' RENAME command. I could have done a theoretical back-of-a-napkin attempt, but I wanted some real numbers. And if it's easy to get them, why not?
- macspoofing 14y ago
  Nevertheless, it was effort for dubious statistics (is knowing that the memory savings are exactly 10MB any better than estimating that the savings would be around 8MB to15MB, working with a data set that's 1.5GB big?). Anyway, it clearly doesn't really matter in this case. You're playing around with a side-project that you can just blow away anytime you feel like. However, when the calculations are this trivial, it's probably good to get into the habit.
  - jholman 14y ago
    Fermi estimates are a terrific technique, and indeed in this case we can see that "tens of thousands" x "five for each user" x 20B saved = 10MB, so that worked out nicely. So yeah, that would have been smart. BUT... When Fermi estimates fail, they fail in one of two ways. Sometimes they fail by a constant factor, as expected, and that's fine. But sometimes they fail because you forgot one or more important factors entirely, and in that case, they can be arbitrarily wrong. (In fact, it seems pretty likely that the so-called Fermi paradox is an example of this). It's not just 8MB vs 15MB vs 10MB.... it's 10MB vs some chance of totally utterly wrong, like maybe 1GB or something. So when the test for real data is cheap (as OP says it is in this case, not that I know squit about Redis), it's "probably good to get into the habit" of doing it right, instead of getting a perfectly good napkin dirty.
    - badgar 14y ago
      This isn't a Fermi problem. It's analysis. Math. How many keys do you have? Not something you should guess. How many bytes per key are you saving? Not something you should guess.
      - ihsw 14y ago
        The 10MB difference can be attributed to fragmentation, so the savings may in fact be zero. In such systems mathematical estimates don't always reflect reality so real testing would be more expedient and appropriate.
    - macspoofing 14y ago
      I know what you're saying but even if you're off by an order of a magnitude, at least it's some sort of starting point. Estimating that your key-space in Redis is between 5MB and 500MB isn't particularly useful but it is a big improvement over having no idea. I don't disagree with you that in most cases, there is no substitute for real world data, especially if you can come across it without much pain, but I maintain you should always start with some sort of estimate, though wrong it may be. It's good to take the time to try to reason out a system or problem, and then seeing if your mental model matches the real world reality. In this particular example the napkin analysis is so trivial and so accurate and the 'real world' data is so expensive ( linear look up of the entire key-space, followed by data destruction ), that it immediately jumps out at me.
      - jholman 14y ago
        When you forget even one factor, you can be off by many orders of magnitude. And you're measuring the cost of the real data in totally useless terms. Who gives a shit about linear scans and "data destruction", lolz. What matters is how long it took the OP to figure out what command to write, and write it, and write the cleanup commands. If he already knew how to do it, then it was basically free. If he didn't already know how to do it, then he learned something, which pays for most of the cost of doing it. And the napkin analysis is so trivial, but you don't EVER know that it's accurate, because there's no error bound on "oh I forgot the important part".

davidcelis 14y ago

> Optimization is a rabbithole I’ve not had to go down very often. I am hardly an expert. But really, the answer to this is because it's easy to copy a database in Redis and then just do a mass RENAME. It took all of two minutes with a Ruby script and Redis' RENAME command. I could have done a theoretical back-of-a-napkin attempt, but I wanted some real numbers. And if it's easy to get them, why not?

macspoofing 14y ago
Nevertheless, it was effort for dubious statistics (is knowing that the memory savings are exactly 10MB any better than estimating that the savings would be around 8MB to15MB, working with a data set that's 1.5GB big?). Anyway, it clearly doesn't really matter in this case. You're playing around with a side-project that you can just blow away anytime you feel like. However, when the calculations are this trivial, it's probably good to get into the habit.
- jholman 14y ago
  Fermi estimates are a terrific technique, and indeed in this case we can see that "tens of thousands" x "five for each user" x 20B saved = 10MB, so that worked out nicely. So yeah, that would have been smart. BUT... When Fermi estimates fail, they fail in one of two ways. Sometimes they fail by a constant factor, as expected, and that's fine. But sometimes they fail because you forgot one or more important factors entirely, and in that case, they can be arbitrarily wrong. (In fact, it seems pretty likely that the so-called Fermi paradox is an example of this). It's not just 8MB vs 15MB vs 10MB.... it's 10MB vs some chance of totally utterly wrong, like maybe 1GB or something. So when the test for real data is cheap (as OP says it is in this case, not that I know squit about Redis), it's "probably good to get into the habit" of doing it right, instead of getting a perfectly good napkin dirty.
  - badgar 14y ago
    This isn't a Fermi problem. It's analysis. Math. How many keys do you have? Not something you should guess. How many bytes per key are you saving? Not something you should guess.
    - ihsw 14y ago
      The 10MB difference can be attributed to fragmentation, so the savings may in fact be zero. In such systems mathematical estimates don't always reflect reality so real testing would be more expedient and appropriate.
  - macspoofing 14y ago
    I know what you're saying but even if you're off by an order of a magnitude, at least it's some sort of starting point. Estimating that your key-space in Redis is between 5MB and 500MB isn't particularly useful but it is a big improvement over having no idea. I don't disagree with you that in most cases, there is no substitute for real world data, especially if you can come across it without much pain, but I maintain you should always start with some sort of estimate, though wrong it may be. It's good to take the time to try to reason out a system or problem, and then seeing if your mental model matches the real world reality. In this particular example the napkin analysis is so trivial and so accurate and the 'real world' data is so expensive ( linear look up of the entire key-space, followed by data destruction ), that it immediately jumps out at me.
    - jholman 14y ago
      When you forget even one factor, you can be off by many orders of magnitude. And you're measuring the cost of the real data in totally useless terms. Who gives a shit about linear scans and "data destruction", lolz. What matters is how long it took the OP to figure out what command to write, and write it, and write the cleanup commands. If he already knew how to do it, then it was basically free. If he didn't already know how to do it, then he learned something, which pays for most of the cost of doing it. And the napkin analysis is so trivial, but you don't EVER know that it's accurate, because there's no error bound on "oh I forgot the important part".