4 ms·

I seriously doubt that this is true. SQL databases are extremely easy to adjust. Hell you can add columns on your live database in the middle of the day if you

I seriously doubt that this is true. SQL databases are extremely easy to adjust. Hell you can add columns on your live database in the middle of the day if you want. Removing data is harder but normally you don't have to delete columns just to launch your new code. Another thing to remember is that most SQL systems are mature and there is pretty much always very good tools available to do any kind of changes you may need to do.

benblack 16y ago
The existence of great tools for relational databases is a compelling argument for using them. As I said in the article, starting off using a single, monolithic relational store is a successful approach employed by many, successful companies. I would suggest, though, that the rest of your comment indicates a lack of experience with relational databases at large-scale. One metric provided by Twitter in a presentation was that an ALTER TABLE command took 2 weeks to run on a previously centralized relational database. Perhaps someone from Twitter can add some color to that anecdote. tl;dr - Scale breaks your assumptions.
- Devilboy 16y ago
  Scale just breaks MySQL.
  - benblack 16y ago
    If the answer is "Spend millions on Oracle", people will be motivated to ask a different question.
    - gaius 16y ago
      Like I say, in business there is no cheap or expensive. There's worth the money, or not. Yes, Oracle costs money. But so does "rolling your own". How much of their VCs cash has say Twitter spent doing that? Yes, Oracle "locks you in". But your own legacy code locks you in whatever platform you've built it on.
      - benblack 16y ago
        In the specific case of FlockDB, that's actually not the case as the shards are modular. SQLShard is one implementation. There was an experimental Redis shard implementation, as well. You might be both overestimating the cost of building things like this and underestimating the enormous drag of using closed source components in these systems. If you've personally built online services this big with Oracle, well done. What did it cost?
        gaius 16y ago
        Once you have hash joins, shards start to look awfully restrictive compared to partitions... MySQL (et al, I don't know about FlockDB) can freely shard because they're not losing functionality they don't have in the first place. Maybe I'm not a good example because I've mainly worked in financial services, but nearly every Oracle project I've worked on has been wildly profitable, and most have been at the level (in terms of transactional throughput, and in terms of features that were cheaper to buy than build) at which there are only really two choices, Oracle or DB2.
        benblack 16y ago
        Vendors highly optimize for your case, and you find that their solutions work for you. Not a surprise. Different constraints and different requirements produce different solutions. Nobody is saying you are doing financial services storage and processing wrong. Why are you so insistent experts in a totally different field are doing their jobs wrong?
        gaius 16y ago
        I'm not. I'm insistent that the assertions that RDBMSs don't scale and that RDBMSs are too inflexible for rapid, iterative development are false.
        benblack 16y ago
        Absent a metric for "scale" it isn't false, it's meaningless. Here is some context to help you distinguish how things work for online services as opposed to financials: As you are not printing money, you care about cost efficiency. This means you are biased towards using white box hardware, and having as little variation as possible. In the financial world, they use name brand hardware and whatever configurations make sense for a specific application, which is good, because Oracle and IBM are not interested in supporting their products on white boxes. You're Twitter, you have 15 billion edges in this database. Each one is conservatively 24 bytes. 360GB of raw data. Everything you do depends on that data being always available, extremely tolerant of component failures, accessible with very low latency, so assume it all has to be in RAM on a bunch of machines. The Oracle answer is a cluster of fat, named-brand servers (totally unlike all the rest of your hardware) with a dedicated interconnect, a lot of license fees, support contracts, and dedicated DBAs (plural, since you'll need them on-call). Special hardware, headcount, license and support costs. Hit a bug? Call Oracle. Need a feature? Call Oracle. Wish that precious headcount was filled by engineers writing code instead of DBAs carrying pagers? Too bad. When scaling includes having to fit into the same model as all the rest of your infrastructure, and that model is not the Oracle model, then it becomes a bit clearer why engineers might dismiss Oracle. They are left with those inferior options like MySQL, which you've already agreed has various scaling issues (though Facebook somehow manages). So, they invest a few months of a few engineers and they have a purpose built system that does what they need, fits the rest of their model, for which they have the source, and for which the maintenance costs are likely to be far lower. Like I said in the article: absent a specific problem for the business, SQL vs NoSQL is just noise. Something like FlockDB is far simpler and cheaper than throwing Oracle at the problem. That's not a technical argument, it's a business argument. If you want to argue that the big players don't know how to run their businesses, I will not try to stop you.
    - Devilboy 16y ago
      You don't have to spend millions on Oracle. You can spend thousands on SQL Server or get PostgreSQL for nothing too.
  - 1337p337 16y ago
    What scale? How many concurrent users of your specific site break MySQL, and how many concurrent users do you have? Not answering or considering those questions is what leads to premature scaling.

benblack 16y ago

The existence of great tools for relational databases is a compelling argument for using them. As I said in the article, starting off using a single, monolithic relational store is a successful approach employed by many, successful companies. I would suggest, though, that the rest of your comment indicates a lack of experience with relational databases at large-scale. One metric provided by Twitter in a presentation was that an ALTER TABLE command took 2 weeks to run on a previously centralized relational database. Perhaps someone from Twitter can add some color to that anecdote. tl;dr - Scale breaks your assumptions.

Devilboy 16y ago
Scale just breaks MySQL.
- benblack 16y ago
  If the answer is "Spend millions on Oracle", people will be motivated to ask a different question.
  - gaius 16y ago
    Like I say, in business there is no cheap or expensive. There's worth the money, or not. Yes, Oracle costs money. But so does "rolling your own". How much of their VCs cash has say Twitter spent doing that? Yes, Oracle "locks you in". But your own legacy code locks you in whatever platform you've built it on.
    - benblack 16y ago
      In the specific case of FlockDB, that's actually not the case as the shards are modular. SQLShard is one implementation. There was an experimental Redis shard implementation, as well. You might be both overestimating the cost of building things like this and underestimating the enormous drag of using closed source components in these systems. If you've personally built online services this big with Oracle, well done. What did it cost?
      - gaius 16y ago
        Once you have hash joins, shards start to look awfully restrictive compared to partitions... MySQL (et al, I don't know about FlockDB) can freely shard because they're not losing functionality they don't have in the first place. Maybe I'm not a good example because I've mainly worked in financial services, but nearly every Oracle project I've worked on has been wildly profitable, and most have been at the level (in terms of transactional throughput, and in terms of features that were cheaper to buy than build) at which there are only really two choices, Oracle or DB2.
        benblack 16y ago
        Vendors highly optimize for your case, and you find that their solutions work for you. Not a surprise. Different constraints and different requirements produce different solutions. Nobody is saying you are doing financial services storage and processing wrong. Why are you so insistent experts in a totally different field are doing their jobs wrong?
        gaius 16y ago
        I'm not. I'm insistent that the assertions that RDBMSs don't scale and that RDBMSs are too inflexible for rapid, iterative development are false.
        benblack 16y ago
        Absent a metric for "scale" it isn't false, it's meaningless. Here is some context to help you distinguish how things work for online services as opposed to financials: As you are not printing money, you care about cost efficiency. This means you are biased towards using white box hardware, and having as little variation as possible. In the financial world, they use name brand hardware and whatever configurations make sense for a specific application, which is good, because Oracle and IBM are not interested in supporting their products on white boxes. You're Twitter, you have 15 billion edges in this database. Each one is conservatively 24 bytes. 360GB of raw data. Everything you do depends on that data being always available, extremely tolerant of component failures, accessible with very low latency, so assume it all has to be in RAM on a bunch of machines. The Oracle answer is a cluster of fat, named-brand servers (totally unlike all the rest of your hardware) with a dedicated interconnect, a lot of license fees, support contracts, and dedicated DBAs (plural, since you'll need them on-call). Special hardware, headcount, license and support costs. Hit a bug? Call Oracle. Need a feature? Call Oracle. Wish that precious headcount was filled by engineers writing code instead of DBAs carrying pagers? Too bad. When scaling includes having to fit into the same model as all the rest of your infrastructure, and that model is not the Oracle model, then it becomes a bit clearer why engineers might dismiss Oracle. They are left with those inferior options like MySQL, which you've already agreed has various scaling issues (though Facebook somehow manages). So, they invest a few months of a few engineers and they have a purpose built system that does what they need, fits the rest of their model, for which they have the source, and for which the maintenance costs are likely to be far lower. Like I said in the article: absent a specific problem for the business, SQL vs NoSQL is just noise. Something like FlockDB is far simpler and cheaper than throwing Oracle at the problem. That's not a technical argument, it's a business argument. If you want to argue that the big players don't know how to run their businesses, I will not try to stop you.
  - Devilboy 16y ago
    You don't have to spend millions on Oracle. You can spend thousands on SQL Server or get PostgreSQL for nothing too.
- 1337p337 16y ago
  What scale? How many concurrent users of your specific site break MySQL, and how many concurrent users do you have? Not answering or considering those questions is what leads to premature scaling.