3 ms·

Are Cloud Based Memory Architectures the Next Big Thing?

19 points by randomwalker 18y ago

rgrieselhuber 18y ago
Enjoyed this post. Most of the in-memory solutions seemed focused around Java / .NET. What are the major open source (if any) solutions for Python / Rails / PHP?
- njharman 18y ago
  memcached? There's memory only versions/addons to Tokyo Cabinet which I can't find right now. And I've recently read about a couple other memory (although disk backed) key-value systems. They're the new hotness. Or am I not getting what author means by "cloud based memory"? It sounds like nothing more than what Erlang + mnesia has been doing for ages. Even MySQL has it's in memory Cluster engine. Although, I doubt that scales to cloud levels.
lsb 18y ago
Here's 10 minutes of your life: 1) RAM is a lot faster than disk, keep things there for speed. 2) "RAM is the new disk." "Or is disk the new RAM?" 3) 5 ads for Java companies. Nowhere does it say that Facebook is running the world's largest memcached installation. Or, for that matter, that HN is all in memory.
- jwilliams 18y ago
  It mentions Facebook? One distinction this article never fleshed out is the difference between memory caching and memory as a System of Record - this is mentioned very early on, but then the rest of the article seems to be a mixture of these techniques. Not sure I'd be keen to use memory as a SoR at this stage. You leave yourself vulnunerable to bugs, and deployment would be a nightmare... So - given that - If you want to snapshot some kind of consistent state, then I don't see why you wouldn't stick with disk for the moment.
  - toddh 18y ago
    Yes, early in the article it says "Facebook has northwards of 800 memcached servers creating a reservoir of 28 terabytes of memory enabling a 99% cache hit rate." I've covered this stuff before so I don't explore the topic. It's a device to quickly show how memory is being used by serious companies to do serious things. Caching is mentioned as related to but not the same as SoR, but you are right, I don't spend a lot of time on it. I understand why you wouldn't be keen, but it is at least something to think about.
oomkiller 18y ago
I think this article is close, but misses an important player, SSDs. SSDs bring super-fast data rates that can be amplified by traditional techniques (striping etc), but work enough like disk systems so that applications don't have to be rewritten. I believe that we will see SSD speeds continue to increase, along with widespread adoption and price reduction. On the software side, I believe that many commonly used applications will begin to detect SSDs and (un)optimize their code for them. In my opinion, memory will always be the fastest, but SSD gives the reliability and backwards computability that will cause mass-adoption. I think we'll see (and have already seen) the early-adopters and power users start adopting SSDs at a high rate. Business adoption usually follows after a few years. So my prediction is that we'll see 1TB or larger sized SSDs, that are increasingly faster, within a few years. I think we'll start seeing DB arrays with multiple SSDs in a Raid0+1 or similar, probably up to 10TB or larger. Does anyone else feel this way.

19 points by randomwalker 18y ago

Enjoyed this post. Most of the in-memory solutions seemed focused around Java / .NET. What are the major open source (if any) solutions for Python / Rails / PHP?

njharman 18y ago
memcached? There's memory only versions/addons to Tokyo Cabinet which I can't find right now. And I've recently read about a couple other memory (although disk backed) key-value systems. They're the new hotness. Or am I not getting what author means by "cloud based memory"? It sounds like nothing more than what Erlang + mnesia has been doing for ages. Even MySQL has it's in memory Cluster engine. Although, I doubt that scales to cloud levels.

lsb 18y ago

Here's 10 minutes of your life: 1) RAM is a lot faster than disk, keep things there for speed. 2) "RAM is the new disk." "Or is disk the new RAM?" 3) 5 ads for Java companies. Nowhere does it say that Facebook is running the world's largest memcached installation. Or, for that matter, that HN is all in memory.

jwilliams 18y ago
It mentions Facebook? One distinction this article never fleshed out is the difference between memory caching and memory as a System of Record - this is mentioned very early on, but then the rest of the article seems to be a mixture of these techniques. Not sure I'd be keen to use memory as a SoR at this stage. You leave yourself vulnunerable to bugs, and deployment would be a nightmare... So - given that - If you want to snapshot some kind of consistent state, then I don't see why you wouldn't stick with disk for the moment.
- toddh 18y ago
  Yes, early in the article it says "Facebook has northwards of 800 memcached servers creating a reservoir of 28 terabytes of memory enabling a 99% cache hit rate." I've covered this stuff before so I don't explore the topic. It's a device to quickly show how memory is being used by serious companies to do serious things. Caching is mentioned as related to but not the same as SoR, but you are right, I don't spend a lot of time on it. I understand why you wouldn't be keen, but it is at least something to think about.

oomkiller 18y ago

I think this article is close, but misses an important player, SSDs. SSDs bring super-fast data rates that can be amplified by traditional techniques (striping etc), but work enough like disk systems so that applications don't have to be rewritten. I believe that we will see SSD speeds continue to increase, along with widespread adoption and price reduction. On the software side, I believe that many commonly used applications will begin to detect SSDs and (un)optimize their code for them. In my opinion, memory will always be the fastest, but SSD gives the reliability and backwards computability that will cause mass-adoption. I think we'll see (and have already seen) the early-adopters and power users start adopting SSDs at a high rate. Business adoption usually follows after a few years. So my prediction is that we'll see 1TB or larger sized SSDs, that are increasingly faster, within a few years. I think we'll start seeing DB arrays with multiple SSDs in a Raid0+1 or similar, probably up to 10TB or larger. Does anyone else feel this way.