4 ms·

I don't see Amazon SimpleDB in this list, rather Postgres and Redis. Can someone who knows more about various DBs opine on this. Is it better to run your own D

by yumraj 14y ago

I don't see Amazon SimpleDB in this list, rather Postgres and Redis. Can someone who knows more about various DBs opine on this. Is it better to run your own DB as Instagram seems to be doing or is relying on SimpleDB good enough if you don't need such high performance. Also, as happens with many startups, how easy/difficult is data migration when startups try to scale and need to scale fast.

dangrossman 14y ago
SimpleDB is not an RDBMS. It has very specific use-cases where it makes sense, and most database use does not fit into those use-cases. Perhaps you're thinking of Amazon RDS, which is their managed MySQL/Oracle/SQL Server deployments. RDS is very expensive compared to running your own database, even on the very same EC2/EBS platform -- but you get reliability, replication and backups that some employee would have to be managing otherwise. In theory, at least. RDS didn't live up to its promises when I used it, so I moved back to running databases on physical servers at half the cost.
- genwin 14y ago
  Please elaborate on how RDS didn't live up to its promises?
  - dangrossman 14y ago
    When EBS failed in US-East a year or two ago, my RDS instance did not remain available and did not automatically fail over to its replica, despite my paying for these benefits. Even when it was working correctly, the RDS instances had low capacity compared to physical servers, and there were often spikes of significant latency that would lead to overloaded web servers as DB queries backed up unanswered.
    - genwin 14y ago
      Good to know. I read here on HN that during the storm event in June, the same thing happened (no auto fail over for those who paid for it).
seldo 14y ago
AWS has more than one DB option. http://aws.amazon.com/simpledb/ http://aws.amazon.com/simpledb/ SimpleDB is a non-relational store that automatically indexes everything, but can only store up to 10GB. It's very flexible but limited, good for prototyping. http://aws.amazon.com/dynamodb/ http://aws.amazon.com/dynamodb/ DynamoDB is like SimpleDB's grown-up version. At greater cost and with more up-front configuration, it scales automatically to huge workloads. It's still a non-relational store, with the drawbacks that implies. http://aws.amazon.com/rds/ http://aws.amazon.com/rds/ Relational Database Service is literally managed MySQL instances. Amazon spins them up for you, manages configuration, backups and restoration. One of RDS's primary value-adds is that it can automatically partition your data across multiple EBS volumes (like hard drives). This helps get around the relatively low I/O performance of EBS volumes. So pick your poison -- if you don't need high performance, simpleDB or a small RDS instance will work; it depends whether you want relational data or not. I can't speak for the difficulty of migrating; we stuck with running our own MySQL instances from the start. For a lot of reasons (that I can enumerate if you'd like), we think running your own DBs is the best option.
- cluda01 14y ago
  I would very much appreciate it if you elaborate on why managing your own MySQL database is the best option. I'm currently moving from back end proprietary systems development to web development and would like to hear your considerations.
  - icebraining 14y ago
    Seldo said they run their own DBs, no that they run their own MySQL instances. Being able to run other engines (e.g. PostgreSQL) is no doubt a benefit of running your own DBs.
  - tlack 14y ago
    If you run your own stack, you get lower prices and more flexibility. On the flip side, if you don't know how to administer MySQL, you'll have some learning to do. Dedicated server pricing is 1/2 or less of what Amazon offers you, and you get better performance to boot. Seems like a no brainer to me (but then again I've been doing "dev ops" stuff since the late 90s and learned many lessons the hard way).
jparise 14y ago
Mike gave a great talk to the San Francisco PostgreSQL User Group in which he discussed their PostgreSQL setup. Slides: http://media.postgresql.org/sfpug/instagram_sfpug.pdf http://media.postgresql.org/sfpug/instagram_sfpug.pdf Video: http://www.ustream.tv/recorded/21929154 http://www.ustream.tv/recorded/21929154

by yumraj 14y ago

dangrossman 14y ago

SimpleDB is not an RDBMS. It has very specific use-cases where it makes sense, and most database use does not fit into those use-cases. Perhaps you're thinking of Amazon RDS, which is their managed MySQL/Oracle/SQL Server deployments. RDS is very expensive compared to running your own database, even on the very same EC2/EBS platform -- but you get reliability, replication and backups that some employee would have to be managing otherwise. In theory, at least. RDS didn't live up to its promises when I used it, so I moved back to running databases on physical servers at half the cost.

genwin 14y ago
Please elaborate on how RDS didn't live up to its promises?
- dangrossman 14y ago
  When EBS failed in US-East a year or two ago, my RDS instance did not remain available and did not automatically fail over to its replica, despite my paying for these benefits. Even when it was working correctly, the RDS instances had low capacity compared to physical servers, and there were often spikes of significant latency that would lead to overloaded web servers as DB queries backed up unanswered.
  - genwin 14y ago
    Good to know. I read here on HN that during the storm event in June, the same thing happened (no auto fail over for those who paid for it).

seldo 14y ago

AWS has more than one DB option. http://aws.amazon.com/simpledb/ http://aws.amazon.com/simpledb/ SimpleDB is a non-relational store that automatically indexes everything, but can only store up to 10GB. It's very flexible but limited, good for prototyping. http://aws.amazon.com/dynamodb/ http://aws.amazon.com/dynamodb/ DynamoDB is like SimpleDB's grown-up version. At greater cost and with more up-front configuration, it scales automatically to huge workloads. It's still a non-relational store, with the drawbacks that implies. http://aws.amazon.com/rds/ http://aws.amazon.com/rds/ Relational Database Service is literally managed MySQL instances. Amazon spins them up for you, manages configuration, backups and restoration. One of RDS's primary value-adds is that it can automatically partition your data across multiple EBS volumes (like hard drives). This helps get around the relatively low I/O performance of EBS volumes. So pick your poison -- if you don't need high performance, simpleDB or a small RDS instance will work; it depends whether you want relational data or not. I can't speak for the difficulty of migrating; we stuck with running our own MySQL instances from the start. For a lot of reasons (that I can enumerate if you'd like), we think running your own DBs is the best option.

cluda01 14y ago
I would very much appreciate it if you elaborate on why managing your own MySQL database is the best option. I'm currently moving from back end proprietary systems development to web development and would like to hear your considerations.
- icebraining 14y ago
  Seldo said they run their own DBs, no that they run their own MySQL instances. Being able to run other engines (e.g. PostgreSQL) is no doubt a benefit of running your own DBs.
- tlack 14y ago
  If you run your own stack, you get lower prices and more flexibility. On the flip side, if you don't know how to administer MySQL, you'll have some learning to do. Dedicated server pricing is 1/2 or less of what Amazon offers you, and you get better performance to boot. Seems like a no brainer to me (but then again I've been doing "dev ops" stuff since the late 90s and learned many lessons the hard way).

jparise 14y ago

Mike gave a great talk to the San Francisco PostgreSQL User Group in which he discussed their PostgreSQL setup. Slides: http://media.postgresql.org/sfpug/instagram_sfpug.pdf http://media.postgresql.org/sfpug/instagram_sfpug.pdf Video: http://www.ustream.tv/recorded/21929154 http://www.ustream.tv/recorded/21929154