4 ms·

So as long as Moore’s law can give us capacity jumps like that, we can keep the entire working set in memory and all will be good. I'm not going to say "this g

by trezor 18y ago

So as long as Moore’s law can give us capacity jumps like that, we can keep the entire working set in memory and all will be good. I'm not going to say "this guy does not know what he is talking about", but what kind of shitty database are you running if you need to have the entire database cached in memory? Isn't the point of having a proper RDBMS that it should be able to handle data more efficiently than what would be "logically" possible? I've run databases with 100s of gigs of content on a 2-node cluster where each node "only" had 8GBs of RAM and performance nor memory usage wasn't an issue at all. One point of real pain we’ve suffered, though, is that migrating a database schema in MySQL on a huge table takes forever and a day. Oh right. MySQL. That explains it.

chrisbolt 18y ago
That's great if your data access is localized or infrequent enough that you aren't exceeding your disks' I/O limits, or if you have lots of I/O capacity by adding lots of spindles, but even adding spindles isn't going to be as good as adding RAM. Sharding just makes each shard's local RAM cache more efficient, and 37signals is choosing to have one 128GB server instead of 4 32GB shards, which is great as long as Moore's law keeps up with your growth.
- trezor 18y ago
  My point was more that adding RAM isn't to be as good as having an actually good DB which doesn't rape your disk and CPU by using inefficient lookup methods, employing overly broad locks and which fails at transactions. His IO-problem is because he is using a shitty database. Only in the MySQL-world is it common to need the same amount of RAM as the dataset you are working on. And if you are going that way memory-wise, you might as well you use an in-memory flat textfile or persisted objects. I'm not saying having excess RAM for caching is bad, because that is obviously silly, but when you need to have the entire DB in ram, you should realize something is horribly wrong.

by trezor 18y ago

chrisbolt 18y ago

That's great if your data access is localized or infrequent enough that you aren't exceeding your disks' I/O limits, or if you have lots of I/O capacity by adding lots of spindles, but even adding spindles isn't going to be as good as adding RAM. Sharding just makes each shard's local RAM cache more efficient, and 37signals is choosing to have one 128GB server instead of 4 32GB shards, which is great as long as Moore's law keeps up with your growth.

trezor 18y ago
My point was more that adding RAM isn't to be as good as having an actually good DB which doesn't rape your disk and CPU by using inefficient lookup methods, employing overly broad locks and which fails at transactions. His IO-problem is because he is using a shitty database. Only in the MySQL-world is it common to need the same amount of RAM as the dataset you are working on. And if you are going that way memory-wise, you might as well you use an in-memory flat textfile or persisted objects. I'm not saying having excess RAM for caching is bad, because that is obviously silly, but when you need to have the entire DB in ram, you should realize something is horribly wrong.