7 ms·

> We changed the structure of our heaviest used models a couple times in the past year, and instead of going back and updating millions of old documents, we sim

• mitchellh 15y ago
  > Ugh, this sounds like a maintenance nightmare. How do you deal with adding extra field to the document? Do you ever feel the need of running on-the-fly migration of old versions? (But when you do, shouldn't running a migration for all documents a better idea?) Yes, we did on-the-fly migration as we loaded old data in. Doing full data migration was not really an option because querying from MongoDB on un-indexed data is so slow, and paging in all that data would purge hot data, exacerbating the problem. > I'll admit I'm a non-believer, but every time I see "Schemaless" in MongoDB, I think "oh, so you're implementing schema in your application?" That's exactly what happens.
  • judofyr 15y ago
    So it's actually 100% the same as you would do with on-the-fly migrations in SQL: (1) Add column and add code moves the old data when you access it. Deploy. (2) Let it run for a while. Run a background job that migrates the rest (this might be done days or months later). (3) Remove the column and the custom code. The more I hear about "schemaless" the more I realize that it doesn't make any difference at all.
    • scubaguy 15y ago
      For a large table, adding a new column can be non-trivial, each record is updated when the new column is added.
      • jeffdavis 15y ago
        If you choose to update all of the old records. As other posts have already mentioned, adding a new column to a table in postgres does so instantly unless you set a default (telling postgres that you want it to update all the old records too). If you add the default separately from adding the column, you get the default but don't rewrite all the old rows (which remain NULL).
    • raverbashing 15y ago
      Absolutely not!! It's more like this You have old user table with for example: login and user name In MongoDB this is a JSON object {login:'user', name:'User Name'} You want to add 'shoe size'. So you add 1 - the shoe size to the signup/user editing form 2 - next user created is like this: {login:'user', name:'User Name', 'shoe_size': 7} 3 - Old users DON'T get a shoe_size added automatically to their document, but next time they login they get asked "What's your shoe size". It's dynamic. if (!user.shoe_size), done. You add new columns on demand, and you almost never remove them (you don't need to add fields for migrating, except of course those you want to add) You (almost never) "run the migration" for a while adding shoe_size to existing documents. And it absolutely does make a huge difference, since you don't have to deal with the mess of relational DB migrations. I'll never develop a system using a relational DB again if I can avoid.
      • bradgessler 15y ago
        That's still a migration, its just incremental.
        • Jare 15y ago
          Incremental can make all the difference between zero- and hours of downtime. Do not underestimate the importance of this extra agility in the modern world of almost daily updates to web apps and backends.
          • bradgessler 15y ago
            I don't see where I underestimated that?
        • pbiggar 15y ago
          It's the same concept, but it's very different in practice. For example, adding a field with a default value in postgres will lock the table during the migration, which may be killer. If you use postgres for big data sets, what you'll end up implementing for your migrations looks a lot like what schemaless gives you for free.
    • pbiggar 15y ago
      That process sounds more like "implementing schemaless in SQL" than "implementing schema in NoSQL".
• fizx 15y ago
  There are many serialization formats (e.g. Apache Thrift) with versionable schemas. You can do a poor man's Thrift with json, mongo, etc. It's a common thing to throw thrift or protobuf values in giant key-value stores (cassandra, replicated memcache, hbase). You don't need to migrate unless you change the key format. If you want do on the fly migrations (mostly to save space, and have the ability to delete ancient code paths), you can do them a row at a time. And yes, we do occasionally write a script to loop through the database and migrate all the documents.
• bitops 15y ago
  > I'll admit I'm a non-believer, but every time I see "Schemaless" in MongoDB, I think "oh, so you're implementing schema in your application?" I think that is arguably one of the selling points of MongoDB. Yes, you do implement schema in your application, but should be doing that knowingly and embracing both the costs and benefits. The benefit is that you can very quickly change your "schema" since it's just however you're choosing to represent data in memory (through your objects or whatever). It also allows you to have part of your application running on one "version" of the schema while another part of the application catches up. The tradeoff is that you have to manage all this yourself. MongoDB does not know about your schema, nor does it want to, nor should it. It affords you a lot of power, but then you have to use it responsibly and understand what safety nets are not present (which you may be used to from a traditional RDBMS). To your point about migrations, there are a few different strategies. You can do a "big bang" migration where you change all documents at once. (I would argue that if you require all documents to be consistent at all times, you should not be using MongoDB). A more "Mongo" approach is to migrate data as you need to; e.g. you pull in an older document and at that time add any missing fields. So yes, in MongoDB, the schema effectively lives in your application. But that is by design and to use MongoDB effectively that's something you have to embrace.
  • gatesvp1 15y ago
    > A more "Mongo" approach is to migrate data as you need to; e.g. you pull in an older document and at that time add any missing fields. I think this works when you're talking about adding fields. But it really breaks down if you need to make some change regarding "sub-objects" or arrays of "sub-objects". If you have made a modeling mistake and you need to pull out a sub-object you generally have to do simply stop the system and migrate.
    • bitops 15y ago
      That's very true. Sometimes you can't do a live migration and you have to bite the bullet.
      • jeltz 15y ago
• radicalbyte 15y ago
  > oh, so you're implementing schema in your application? Isn't that where the schema belongs? Each document represents a conceptual whole. It doesn't contain fields which have to be NULL simply because they weren't in previous versions of the schema. I've been an rdbms guy (datawarehousing/ETL) for a long time now, I've seen a lot of large databases which have been in production for considerable time. They get messy. Really messy. They become basically unmaintainable. Apples, oranges and pears all squashed into a schema the shape of a banana. It's a pretty elegant solution, and is the problem XML/XSD were designed to solve. The cleanest solution that I've seen in production used a relational database as a blob storage for XML-serialized entities. Each table defined a basic interface for the models, but each model was free to use its own general schema. After 10 years it contained a set of very clean individual entities which were conceptually correct. As opposed to the usage as a serialization format for remoting, which has been largely replaced with JSON.
  • alexchamberlain 15y ago
    I am a schemaless believer. BUT (big BUT) have you ever tried to program to a schema that has a XML blob? It's really difficult and painfully slow on the app side. The ops guys like it, because it's easy for them to maintain.
    • radicalbyte 15y ago
      I find it much easier, but then I rely on a framework for serialization/de-serialization..
  • trimbo 15y ago
    > isn't that where the schema belongs? [In the application] Well, unless you have, you know, multiple applications accessing said data. Then it's kind of important to keep it in sync, which is why RDBMS exist and operate the way they do. In my experience, on a long enough timeline, the probability of needing multi-application access for your data goes to 1.
    • matthewcford 15y ago
• FuzzyDunlop 15y ago
  > I'll admit I'm a non-believer, but every time I see "Schemaless" in MongoDB, I think "oh, so you're implementing schema in your application?" I saw what may have well been 'schemaless' in an RBDMS recently, and the application code for it was far from pretty. I couldn't migrate at all; the results were far too inconsistent to pull it off reliably (you know something is wrong when boolean values are replaced with 'Y' and 'N', which of course both evaluate to 'true'). That being said, I tried to implement something else with Node.js and MongoDB, and I found it quite manageable. As long as the application implements a schema well, you should still be able to infer it when looking at the database direct. To that extent, I'd take that over using an RBDMS as a key/value store for serialised data, because that's typically useless without the application that parses it.