5 ms·

That does assume that the company's non-SQL choices don't actually save time during development. In building a system the design of which is under constant flu

That does assume that the company's non-SQL choices don't actually save time during development. In building a system the design of which is under constant flux, there can be a lot of time lost dealing with schema rigidity.

Devilboy 16y ago
I seriously doubt that this is true. SQL databases are extremely easy to adjust. Hell you can add columns on your live database in the middle of the day if you want. Removing data is harder but normally you don't have to delete columns just to launch your new code. Another thing to remember is that most SQL systems are mature and there is pretty much always very good tools available to do any kind of changes you may need to do.
- benblack 16y ago
  The existence of great tools for relational databases is a compelling argument for using them. As I said in the article, starting off using a single, monolithic relational store is a successful approach employed by many, successful companies. I would suggest, though, that the rest of your comment indicates a lack of experience with relational databases at large-scale. One metric provided by Twitter in a presentation was that an ALTER TABLE command took 2 weeks to run on a previously centralized relational database. Perhaps someone from Twitter can add some color to that anecdote. tl;dr - Scale breaks your assumptions.
  - Devilboy 16y ago
    Scale just breaks MySQL.
    - benblack 16y ago
      If the answer is "Spend millions on Oracle", people will be motivated to ask a different question.
      - gaius 16y ago
        Like I say, in business there is no cheap or expensive. There's worth the money, or not. Yes, Oracle costs money. But so does "rolling your own". How much of their VCs cash has say Twitter spent doing that? Yes, Oracle "locks you in". But your own legacy code locks you in whatever platform you've built it on.
        benblack 16y ago
        In the specific case of FlockDB, that's actually not the case as the shards are modular. SQLShard is one implementation. There was an experimental Redis shard implementation, as well. You might be both overestimating the cost of building things like this and underestimating the enormous drag of using closed source components in these systems. If you've personally built online services this big with Oracle, well done. What did it cost?
        gaius 16y ago
        Once you have hash joins, shards start to look awfully restrictive compared to partitions... MySQL (et al, I don't know about FlockDB) can freely shard because they're not losing functionality they don't have in the first place. Maybe I'm not a good example because I've mainly worked in financial services, but nearly every Oracle project I've worked on has been wildly profitable, and most have been at the level (in terms of transactional throughput, and in terms of features that were cheaper to buy than build) at which there are only really two choices, Oracle or DB2.
        benblack 16y ago
        Vendors highly optimize for your case, and you find that their solutions work for you. Not a surprise. Different constraints and different requirements produce different solutions. Nobody is saying you are doing financial services storage and processing wrong. Why are you so insistent experts in a totally different field are doing their jobs wrong?
        16y ago
    - 1337p337 16y ago
      What scale? How many concurrent users of your specific site break MySQL, and how many concurrent users do you have? Not answering or considering those questions is what leads to premature scaling.
gaius 16y ago
What is "schema rigidity"? I'm genuinely curious, do the NoSQL types really believe that you can't add a column to a table on the fly in a RDBMS?
- cdavid 16y ago
  The problem is not whether you can, but how long does it take. At least with mysql, alter table for a large table will take a long time (hours, if not more), during which the table cannot be written to.
  - Devilboy 16y ago
    Other databases do not have this problem, MySQL is just a particularly bad example of a RDBMS
    - fauigerzigerk 16y ago
      I doubt that any DB that stores data physically in a row oriented format doesn't have issues with this particular type of change. But of course there are many other ways to work around that. For instance, you could just create a new table and a view to join the two. Or, if a particular table changes all the time, redesign it to store key value pairs instead. RDBMS can easily be used in a schemaless fashion if needed for particular scenarios.
    - benblack 16y ago
      Google and Facebook have all the money and talent required to deploy epic Oracle systems. Instead they use GFS and BigTable and Cassandra and HBase and Scribe and Hadoop and MySQL and a host of other systems. Amazon has massive Oracle deployments, so plenty of money and knowledge on the topic, but still built Dynamo and S3. There are billions of dollars riding on this for them. Instead of insisting they are wrong, ask yourself why they might be right.
      - gaius 16y ago
        Google and Facebook aren't good examples, because they have no hard transactional requirements for their main applications. If a web page isn't included in one search result but is in the same search executed on a different node 5 minutes later, who would notice or care? If a status update gets dropped, it might be annoying, but you can always just resend it. If you want to compare like for like, ask why Visa isn't using MongoDB for authorizations, or why American Airlines isn't using Redis for reservations.
fauigerzigerk 16y ago
Whether it saves time or not depends on a lot of things. If users actually determine the structure of your data then you're right. If it's just about schema evolution by developers I don't think that moving schema constraints into procedural code makes things simpler or more flexible.
jasonkester 16y ago
That's one of the tough myths to overcome when building on top of a relational database. Changing the Database has traditionally been Hard, so everybody has been trained to think that way, therefore nobody is allowed to change the database because Changing the Database is hard. Try it at your bigco and the old guys will do everything in their power to stop you (thus making changing the database hard to do). Ignore that rule and build yourself an environment where changing the database is easy. I make schema changes to my stuff all the time, and seldom push a release live that doesn't do so. The tools are in place to ensure that it's No Big Deal, so it just works. If you live in a world where changing your SQL database is easy, it sort of takes the wind out of the "start with NoSQL, because changing the database is easy" argument. You get all the speed advantages of being schema-flexible, and you can write ad-hoc queries when you want to, so you're flexible in that direction too.
- billybob 16y ago
  Would you mind telling some of the tools and techniques that make this easy for you?
  - jasonkester 16y ago
    All you really need to pull it off are: - Change scripts for every schema change, stored in source control. - An automated build/deploy that pulls down new change scripts and executes them in order. - (optional) a good way to generate your backend CRUD by looking at the existing database schema, or as a lesser option an ORM that does the same thing. So your workflow is: script out your schema change, check it in, apply it to your dev environment, regenerate the CRUD from your local schema, fix any compile errors that you've introduced. (and optionally make sure all your unit tests still pass). You'll notice that all that stuff above is just the basic workflow you should have in place anyway.
    - rbranson 16y ago
      This is exactly the pattern used by Rails apps with database migrations and Capistrano deployment.
fiveo 16y ago
RDBMS does require a bit of planning (not too much, and not too rigid) and previous experiences building good data model; both traits don't exist in hotshot hackers/startups these days because they either don't have the patience or much deeper experience.

Devilboy 16y ago

I seriously doubt that this is true. SQL databases are extremely easy to adjust. Hell you can add columns on your live database in the middle of the day if you want. Removing data is harder but normally you don't have to delete columns just to launch your new code. Another thing to remember is that most SQL systems are mature and there is pretty much always very good tools available to do any kind of changes you may need to do.

benblack 16y ago
The existence of great tools for relational databases is a compelling argument for using them. As I said in the article, starting off using a single, monolithic relational store is a successful approach employed by many, successful companies. I would suggest, though, that the rest of your comment indicates a lack of experience with relational databases at large-scale. One metric provided by Twitter in a presentation was that an ALTER TABLE command took 2 weeks to run on a previously centralized relational database. Perhaps someone from Twitter can add some color to that anecdote. tl;dr - Scale breaks your assumptions.
- Devilboy 16y ago
  Scale just breaks MySQL.
  - benblack 16y ago
    If the answer is "Spend millions on Oracle", people will be motivated to ask a different question.
    - gaius 16y ago
      Like I say, in business there is no cheap or expensive. There's worth the money, or not. Yes, Oracle costs money. But so does "rolling your own". How much of their VCs cash has say Twitter spent doing that? Yes, Oracle "locks you in". But your own legacy code locks you in whatever platform you've built it on.
      - benblack 16y ago
        In the specific case of FlockDB, that's actually not the case as the shards are modular. SQLShard is one implementation. There was an experimental Redis shard implementation, as well. You might be both overestimating the cost of building things like this and underestimating the enormous drag of using closed source components in these systems. If you've personally built online services this big with Oracle, well done. What did it cost?
        gaius 16y ago
        Once you have hash joins, shards start to look awfully restrictive compared to partitions... MySQL (et al, I don't know about FlockDB) can freely shard because they're not losing functionality they don't have in the first place. Maybe I'm not a good example because I've mainly worked in financial services, but nearly every Oracle project I've worked on has been wildly profitable, and most have been at the level (in terms of transactional throughput, and in terms of features that were cheaper to buy than build) at which there are only really two choices, Oracle or DB2.
        benblack 16y ago
        Vendors highly optimize for your case, and you find that their solutions work for you. Not a surprise. Different constraints and different requirements produce different solutions. Nobody is saying you are doing financial services storage and processing wrong. Why are you so insistent experts in a totally different field are doing their jobs wrong?
        16y ago
  - 1337p337 16y ago
    What scale? How many concurrent users of your specific site break MySQL, and how many concurrent users do you have? Not answering or considering those questions is what leads to premature scaling.

gaius 16y ago

What is "schema rigidity"? I'm genuinely curious, do the NoSQL types really believe that you can't add a column to a table on the fly in a RDBMS?

cdavid 16y ago
The problem is not whether you can, but how long does it take. At least with mysql, alter table for a large table will take a long time (hours, if not more), during which the table cannot be written to.
- Devilboy 16y ago
  Other databases do not have this problem, MySQL is just a particularly bad example of a RDBMS
  - fauigerzigerk 16y ago
    I doubt that any DB that stores data physically in a row oriented format doesn't have issues with this particular type of change. But of course there are many other ways to work around that. For instance, you could just create a new table and a view to join the two. Or, if a particular table changes all the time, redesign it to store key value pairs instead. RDBMS can easily be used in a schemaless fashion if needed for particular scenarios.
  - benblack 16y ago
    Google and Facebook have all the money and talent required to deploy epic Oracle systems. Instead they use GFS and BigTable and Cassandra and HBase and Scribe and Hadoop and MySQL and a host of other systems. Amazon has massive Oracle deployments, so plenty of money and knowledge on the topic, but still built Dynamo and S3. There are billions of dollars riding on this for them. Instead of insisting they are wrong, ask yourself why they might be right.
    - gaius 16y ago
      Google and Facebook aren't good examples, because they have no hard transactional requirements for their main applications. If a web page isn't included in one search result but is in the same search executed on a different node 5 minutes later, who would notice or care? If a status update gets dropped, it might be annoying, but you can always just resend it. If you want to compare like for like, ask why Visa isn't using MongoDB for authorizations, or why American Airlines isn't using Redis for reservations.

fauigerzigerk 16y ago

Whether it saves time or not depends on a lot of things. If users actually determine the structure of your data then you're right. If it's just about schema evolution by developers I don't think that moving schema constraints into procedural code makes things simpler or more flexible.

jasonkester 16y ago

That's one of the tough myths to overcome when building on top of a relational database. Changing the Database has traditionally been Hard, so everybody has been trained to think that way, therefore nobody is allowed to change the database because Changing the Database is hard. Try it at your bigco and the old guys will do everything in their power to stop you (thus making changing the database hard to do). Ignore that rule and build yourself an environment where changing the database is easy. I make schema changes to my stuff all the time, and seldom push a release live that doesn't do so. The tools are in place to ensure that it's No Big Deal, so it just works. If you live in a world where changing your SQL database is easy, it sort of takes the wind out of the "start with NoSQL, because changing the database is easy" argument. You get all the speed advantages of being schema-flexible, and you can write ad-hoc queries when you want to, so you're flexible in that direction too.

billybob 16y ago
Would you mind telling some of the tools and techniques that make this easy for you?
- jasonkester 16y ago
  All you really need to pull it off are: - Change scripts for every schema change, stored in source control. - An automated build/deploy that pulls down new change scripts and executes them in order. - (optional) a good way to generate your backend CRUD by looking at the existing database schema, or as a lesser option an ORM that does the same thing. So your workflow is: script out your schema change, check it in, apply it to your dev environment, regenerate the CRUD from your local schema, fix any compile errors that you've introduced. (and optionally make sure all your unit tests still pass). You'll notice that all that stuff above is just the basic workflow you should have in place anyway.
  - rbranson 16y ago
    This is exactly the pattern used by Rails apps with database migrations and Capistrano deployment.

fiveo 16y ago

RDBMS does require a bit of planning (not too much, and not too rigid) and previous experiences building good data model; both traits don't exist in hotshot hackers/startups these days because they either don't have the patience or much deeper experience.