3 ms·

I've been wondering for a while why we don't see more often systems with a huge amount of RAM and asynchronous persistent updates, with a more, how should I put

I've been wondering for a while why we don't see more often systems with a huge amount of RAM and asynchronous persistent updates, with a more, how should I put it, computer science-ish architecture inside. MVC works ok, but does it really mean we _have_ to use it always? Is it so hard to make system with a TeraByte of RAM? Why even use memcache? Why not make a clear decision that stuff like user statuses will never even see the inside of a hard disk and make use of the simplification it brings? Is there a clear reason, or simply it's not fashionable?

thomaspaine 17y ago
User statuses should never touch hard disk? What happens when your server restarts and everything you were storing in memory disappears?
- spc476 17y ago
  The greylist daemon I wrote (http://www.x-grey.com/ http://www.x-grey.com/) keeps everything in memory, with a checkpoint (everything gets written out to disk) every hour (the main process forks, and it's the child that does the writing). A similar approach might be applicable here. It could very well be a RAM based approach might be viable once memristros (http://en.wikipedia.org/wiki/Memristor#Potential_applications http://en.wikipedia.org/wiki/Memristor#Potential_application...) are used for memory.
- radu_floricica 17y ago
  Exactly how often do servers restart? Your average commodity server - maybe twice a year. For user data it's too much, but for user statuses, I don't think so. And if you save the data in a controlled restart and only consider crashes, with a bit of effort you can probably go down to about once every couple of years or less. As for the benefit... there was a picture some time back here on HN: http://imgur.com/X1Hi1.gif http://imgur.com/X1Hi1.gif
- cd34 17y ago
  Based on another article on highscalability, Twitter has one master MySQL server that is used for backups only. If a node crashes, it reloads that status from MySQL. That MySQL server handles 300 tweets per second @ roughly 2400 qps. They now use Scala/JVM rather than rails for the backend, Rails for the frontend. http://highscalability.com/scaling-twitter-making-twitter-10000-percent-faster http://highscalability.com/scaling-twitter-making-twitter-10...
peterwwillis 17y ago
You can't store everything in memory all the time. There simply isn't a computer built which can do that. Horizontal scaling with commodity boxes and layers of cache are the most cheap and effective method of returning a high rate of requests as quickly as possible.
kmavm 17y ago
You can buy a machine with a terabyte of RAM. If your check doesn't bounce, IBM, HP, Sun, et al. will deliver it via forklift tomorrow. However, that box will cost much, much more than 16 boxes with 64GB each. Those 16 boxes will also have more CPU, network cards, and aggregate bandwidth to RAM than the terabox. The big honkin' box full of RAM is in most respects an unbalanced system; you can do a lot more with a lot less money if it's acceptable to give up on the single-system image programming model.

thomaspaine 17y ago

User statuses should never touch hard disk? What happens when your server restarts and everything you were storing in memory disappears?

spc476 17y ago
The greylist daemon I wrote (http://www.x-grey.com/ http://www.x-grey.com/) keeps everything in memory, with a checkpoint (everything gets written out to disk) every hour (the main process forks, and it's the child that does the writing). A similar approach might be applicable here. It could very well be a RAM based approach might be viable once memristros (http://en.wikipedia.org/wiki/Memristor#Potential_applications http://en.wikipedia.org/wiki/Memristor#Potential_application...) are used for memory.
radu_floricica 17y ago
Exactly how often do servers restart? Your average commodity server - maybe twice a year. For user data it's too much, but for user statuses, I don't think so. And if you save the data in a controlled restart and only consider crashes, with a bit of effort you can probably go down to about once every couple of years or less. As for the benefit... there was a picture some time back here on HN: http://imgur.com/X1Hi1.gif http://imgur.com/X1Hi1.gif
cd34 17y ago
Based on another article on highscalability, Twitter has one master MySQL server that is used for backups only. If a node crashes, it reloads that status from MySQL. That MySQL server handles 300 tweets per second @ roughly 2400 qps. They now use Scala/JVM rather than rails for the backend, Rails for the frontend. http://highscalability.com/scaling-twitter-making-twitter-10000-percent-faster http://highscalability.com/scaling-twitter-making-twitter-10...

peterwwillis 17y ago

You can't store everything in memory all the time. There simply isn't a computer built which can do that. Horizontal scaling with commodity boxes and layers of cache are the most cheap and effective method of returning a high rate of requests as quickly as possible.

kmavm 17y ago

You can buy a machine with a terabyte of RAM. If your check doesn't bounce, IBM, HP, Sun, et al. will deliver it via forklift tomorrow. However, that box will cost much, much more than 16 boxes with 64GB each. Those 16 boxes will also have more CPU, network cards, and aggregate bandwidth to RAM than the terabox. The big honkin' box full of RAM is in most respects an unbalanced system; you can do a lot more with a lot less money if it's acceptable to give up on the single-system image programming model.