15 ms·
"A 2 terabyte hard drive costs $94.99, 4 terabytes is $169.99. Buy one and stick it in a desktop computer or server. Then install Postgres on it." Done! Althou
by eksith 13y ago
"A 2 terabyte hard drive costs $94.99, 4 terabytes is $169.99. Buy one and stick it in a desktop computer or server. Then install Postgres on it."
Done! Although with more drives and a backup server. Right now, we're pushing 15Tb with no loss in performance.
- raverbashing 13y agoYou would probably want some RAID + Backups, (and a very generous amount of RAM there) but this is ok. In fact, I wouldn't doubt that for several "big data" users even SQLite would be enough.
- eksith 13y agoI wouldn't doubt that either. We do use RAID 6 (I think) + tape backups and the backup server itself is a hot standby. If you're curious, this is the chassis : http://www.supermicro.com/products/chassis/3U/836/SC836A-R1200.cfm http://www.supermicro.com/products/chassis/3U/836/SC836A-R12... The motherboard is also Supermicro running dual AMD Opterons ( single processor on the backup server ) with 256Gb RAM.
- StavrosK 13y agoI migrated from MongoDB to SQLite for a project a few years ago, it was much more performant: http://www.stavros.io/posts/using-sqlite-for-great-science-part-1-or-3-i-guess/ http://www.stavros.io/posts/using-sqlite-for-great-science-p... I think the data size ended up being a GB or so.
- theorique 13y agoWhat is the price / performance tradeoff for using SSD in these applications nowadays?
- asabjorn 13y agoFrom my experience SQLite is great when only one process with no parallelism needs to write to the DB and where no other process will frequently overlap reads with writes. However, for CPU-heavy computations that needs multiple cores to compute and write to the DB PostgreSQL has a much better fit for me even for fairly small data-sets. And the ease of use and setup of PostgreSQL is not that much more than SQLite.
- xradionut 13y agoAs date guy that uses both, there is a higher level of expertise required to properly maintain PostgreSQL. SQLite is something that I can call instantly from Python without much forethought.
- daxelrod 13y agoDo you use standard off-the-shelf consumer hard drives at those prices? Most companies I've worked for have shelled out quite a bit more for "enterprise" class hard drives. I've always struggled to understand what these bring to the table, and my understanding is that it's some combination of greater reliability and a service agreement. It's always seemed to me, in my software-centric naivete, that it would be more cost-effective to get off-the-shelf drives, RAID them for redundancy to increase uptime, and make regular backups to increase data longevity.
- rplnt 13y agoMTTF should (and probably is) higher. Perhaps some other things like reallocated sectors count can be higher without affecting performance/stability much. But from what I've seen, the most important thing is the service agreement, i.e. having new disk ready the next day without any questions asked (or money spent for a new drive).
- rplnt 13y ago> or money spent for a new drive To put it other way - expected vs. unexpected expenses.
- AlisdairO 13y agoOne of the most important things is that enterprise drives are much more likely to be telling you the truth when they tell you your data is written to disk. Even if it hasn't actually hit the platter, they'll have a battery or cap backup power supply that allows them to write anything still pending to disk before they run out of power. Consumer drives often lie horribly about whether data is truly durable.
- yummyfajitas 13y agoNot really an issue for building a reporting/aggregation database. But certainly something to think about when processing realtime financial transactions.
- 13y ago