4 ms·

Relational databases are indeed excellent, and you are better off using a relational database if a single server can handle the load. But if you have more load

by petilon 12y ago

Relational databases are indeed excellent, and you are better off using a relational database if a single server can handle the load. But if you have more load than a single server can handle that's when you need things like MongoDB that support automatic sharding.

jsjohnst 12y ago
So I guess you think Facebook/Tumblr/Google/Yahoo/etc use MongoDB for their databases (or worse, that they run it on a single server)?
- tracker1 12y ago
  Realistically most websites could use a combination of caching with a single database server behind them. If the schema isn't too horrible. You can scale up a lot, and depending on your use case and revenue model, that's a valid plan. I'm pretty sure that Facebook is using a wide database (similar to Cassandra) with reverse indexes generated at the application layer. Yahoo, it depends on the developer group, but something like mongo in use wouldn't completely surprise me. Google has published a lot of white papers on how they use data (see Cassandra and Hadoop from some of those lessons). MongoDB is itself a bit of a bridge... replication setup is far easier than with a typical RDBMS, and you can break data into documents for simpler queries. Mongo as a persistent caching layer is also an option depending on your load and needs with SQL as authority, mongo for public search/display.
  - jsjohnst 12y ago
    I've worked for a reasonable subset of top Internet sites and have friends who work at the remainder I haven't been at and I can emphatically say that none of them run "a single database server" for any main portion of the site. To think that is simply ludicrous!
    - tracker1 12y ago
      I didn't say "top Internet sites" ... I said "most" sites... that being anything not in the top 10000 sites, and probably over half of those within the top 10000 could be the case. I also said, with a caching system in place. (data and/or output). It really depends on how interactive and static your needs are, which will vary... but most sites are mostly static and very cachable content... For an example, look at StackOverflow... they were really huge before they needed to start breaking their data up more... Of course their SQL servers are enormous by anyone's standards.. the point is you can scale vertically a lot, and if you can handle a certain amount of downtime, or reduced functionality (if db is down), you can go a long way on a single database server... add caching, add replication... all of this before needing to break data up. (In mostly read scenarios)
soapdog 12y ago
Check out the book "High Performance MySQL" at http://shop.oreilly.com/product/0636920022343.do http://shop.oreilly.com/product/0636920022343.do from chapter 9 and beyond it covers replication and high availability. All this for MySQL which is not event the shinning crown of RDBMS. If you think that automatic sharding is the answer to handle high loads then your are in for a very exciting surprise specially when things go awry and all that missing ACID compliance comes knocking...
- petilon 12y ago
  Replication works for managing read load, but it doesn't help manage write load. This is when you need automatic sharding. As far as ACID, many applications don't need it, for example in a game application users will forgive you if you lose their latest scores, but in a banking application users will not forgive you if you lose their last deposit. So it depends on the application.
  - cthalupa 12y ago
    >Replication works for managing read load, but it doesn't help manage write load. This is when you need automatic sharding. You've said 'need automatic sharding' multiple times now, but you've still yet to explain why you need it. Yes, sharding is important for write scaling. But it doesn't "need" to be automatic. Plenty of setups have completely capable manually sharded workloads running on MySQL. Some would consider the manual approach a plus, vs. relying on the database to automagically do it. It can be nice to have automatic sharding, assuming it works properly. But nice and needed are two different things, an I think it's an important distinction. You're basically saying if you shard you have to go with mongo and you can't go with anything that requires you manually handle it.

by petilon 12y ago

jsjohnst 12y ago

So I guess you think Facebook/Tumblr/Google/Yahoo/etc use MongoDB for their databases (or worse, that they run it on a single server)?

tracker1 12y ago
Realistically most websites could use a combination of caching with a single database server behind them. If the schema isn't too horrible. You can scale up a lot, and depending on your use case and revenue model, that's a valid plan. I'm pretty sure that Facebook is using a wide database (similar to Cassandra) with reverse indexes generated at the application layer. Yahoo, it depends on the developer group, but something like mongo in use wouldn't completely surprise me. Google has published a lot of white papers on how they use data (see Cassandra and Hadoop from some of those lessons). MongoDB is itself a bit of a bridge... replication setup is far easier than with a typical RDBMS, and you can break data into documents for simpler queries. Mongo as a persistent caching layer is also an option depending on your load and needs with SQL as authority, mongo for public search/display.
- jsjohnst 12y ago
  I've worked for a reasonable subset of top Internet sites and have friends who work at the remainder I haven't been at and I can emphatically say that none of them run "a single database server" for any main portion of the site. To think that is simply ludicrous!
  - tracker1 12y ago
    I didn't say "top Internet sites" ... I said "most" sites... that being anything not in the top 10000 sites, and probably over half of those within the top 10000 could be the case. I also said, with a caching system in place. (data and/or output). It really depends on how interactive and static your needs are, which will vary... but most sites are mostly static and very cachable content... For an example, look at StackOverflow... they were really huge before they needed to start breaking their data up more... Of course their SQL servers are enormous by anyone's standards.. the point is you can scale vertically a lot, and if you can handle a certain amount of downtime, or reduced functionality (if db is down), you can go a long way on a single database server... add caching, add replication... all of this before needing to break data up. (In mostly read scenarios)

soapdog 12y ago

Check out the book "High Performance MySQL" at http://shop.oreilly.com/product/0636920022343.do http://shop.oreilly.com/product/0636920022343.do from chapter 9 and beyond it covers replication and high availability. All this for MySQL which is not event the shinning crown of RDBMS. If you think that automatic sharding is the answer to handle high loads then your are in for a very exciting surprise specially when things go awry and all that missing ACID compliance comes knocking...

petilon 12y ago
Replication works for managing read load, but it doesn't help manage write load. This is when you need automatic sharding. As far as ACID, many applications don't need it, for example in a game application users will forgive you if you lose their latest scores, but in a banking application users will not forgive you if you lose their last deposit. So it depends on the application.
- cthalupa 12y ago
  >Replication works for managing read load, but it doesn't help manage write load. This is when you need automatic sharding. You've said 'need automatic sharding' multiple times now, but you've still yet to explain why you need it. Yes, sharding is important for write scaling. But it doesn't "need" to be automatic. Plenty of setups have completely capable manually sharded workloads running on MySQL. Some would consider the manual approach a plus, vs. relying on the database to automagically do it. It can be nice to have automatic sharding, assuming it works properly. But nice and needed are two different things, an I think it's an important distinction. You're basically saying if you shard you have to go with mongo and you can't go with anything that requires you manually handle it.