5 ms·

everyone gangsta until they have to feature flag a database schema change

jeffbee 6y ago
having a database schema at all is the root of that problem. i wish people who are writing systems from scratch would reflect on whether they actually need an RDBMS or RDBMS-like schemas. Everything I've worked on in the last few years would have been in far, far better shape if they had just started with a no-schema data management system like Bigtable and just adapted their backend code around its limitations. Imagining you are safe because you are smoke-checking DML statements in staging is just begging for a week-long outage.
- necovek 6y ago
  I can easily imagine majority of the developers trying to solve trivial serialization and locking issues on their own (which RDBMS provide) and failing spectacularly! Heck, I don't need to imagine, I have seen it time and time again.
- programmarchy 6y ago
  Curious about this line of thought because for most apps I reach for a RDBMS because I want strong data integrity and a data model that’s easy to reason about. Maybe you tend to work in different domains. Can you give an example of one of the projects better served by Bigtable?
  - jeffbee 6y ago
    Everything? When the topic is operability it is easily shown that a database schema that you have to change atomically and the change becomes visible to all clients instantly is very incompatible with the ideas of rolling releases, rollback safety, zero maintenance windows, etc. If you can see that centralizing the schema doesn’t work well then you can also see that the schema doesn’t belong in the database at all, it belongs in the clients where it can be rolled forward and backward safely if you never try to change a field, only add them, which is a best practice anyway. After you’re onboard with this philosophy then I’m ready to sell you the fact that you don’t need joins, indexes, or transactions. If a giant complicated service like gmail doesn’t need any of those things then very likely that your thing doesn’t need them either.
    - LoSboccacc 6y ago
      the intermediate layer between clients and the database is responsible to negotiate a protocol version, you shouldn't expose the storage model to the api layer, then schema versioning becomes a non problem.
      - jeffbee 6y ago
        I don't think you can hide schemas that way in practice. At Dropbox I saw that they had an abstract storage service in front of MySQL but every aspect of the underlying schema leaked out through the API in some way.
- LoSboccacc 6y ago
  first: I didn't say rdbms, I said database. that covers nosql as well. second: even if you don't have an explicit schema in the database, you still have to have a way to know which data goes in which column. just because it isn't written in a definition language doesn't mean the concept doesn't exist and doesn't require to be maintained as the application domain evolves.
- john-shaffer 6y ago
  This is a big reason why I like Datomic so much. I get the benefits of a schema, but the schema is basically append-only, so it's very easy to reason about. Updating the schema is instant. The last time I tried a project with a no-schema DB, it was a huge PITA. It sucks having a function fail on one single document because it had a number instead of a string. I love flexibility, but it is very valuable to have the right constraints in the right places. I quickly learned that a schema is just that.
  - doteka 6y ago
    I don’t see how you’d have a number instead of a spring in a document in a typed language. Do you just directly dump user supplied data in your database without parsing/processing it? Otherwise, how is this not caught when it enters your system?
    - loopz 6y ago
      It is caught by the schema and never stored in the first place. Or you can solve it for every column/field in your system manually, and probably introduce subtle bugs and inconsistencies along the way. I'm for schemas, but not necessarily RDBMS for everything.
    - john-shaffer 6y ago
      Most document databases have JSON editors, which are very useful at times but don't have any schema checks. By "typed language" I suppose you mean "statically typed language". This isn't relevant because A) JSON types exist and B) I've never had a problem with unexpected types from user data in a dynamic language, not even in PHP. > Otherwise, how is this not caught when it enters your system? It should be caught, and it's very important that it be caught all of the time and not have any issues due to bugs or version mismatches. That's why the schema should live in the DB.
- dagss 6y ago
  And everything I have worked on would have been in far, far better shape if they had just reached for RDBMS instead of following the NoSQL hype and reimplementing what the RDBMS gives you in each backend service. YMMV I guess.

having a database schema at all is the root of that problem. i wish people who are writing systems from scratch would reflect on whether they actually need an RDBMS or RDBMS-like schemas. Everything I've worked on in the last few years would have been in far, far better shape if they had just started with a no-schema data management system like Bigtable and just adapted their backend code around its limitations. Imagining you are safe because you are smoke-checking DML statements in staging is just begging for a week-long outage.

necovek 6y ago
I can easily imagine majority of the developers trying to solve trivial serialization and locking issues on their own (which RDBMS provide) and failing spectacularly! Heck, I don't need to imagine, I have seen it time and time again.
programmarchy 6y ago
Curious about this line of thought because for most apps I reach for a RDBMS because I want strong data integrity and a data model that’s easy to reason about. Maybe you tend to work in different domains. Can you give an example of one of the projects better served by Bigtable?
- jeffbee 6y ago
  Everything? When the topic is operability it is easily shown that a database schema that you have to change atomically and the change becomes visible to all clients instantly is very incompatible with the ideas of rolling releases, rollback safety, zero maintenance windows, etc. If you can see that centralizing the schema doesn’t work well then you can also see that the schema doesn’t belong in the database at all, it belongs in the clients where it can be rolled forward and backward safely if you never try to change a field, only add them, which is a best practice anyway. After you’re onboard with this philosophy then I’m ready to sell you the fact that you don’t need joins, indexes, or transactions. If a giant complicated service like gmail doesn’t need any of those things then very likely that your thing doesn’t need them either.
  - LoSboccacc 6y ago
    the intermediate layer between clients and the database is responsible to negotiate a protocol version, you shouldn't expose the storage model to the api layer, then schema versioning becomes a non problem.
    - jeffbee 6y ago
      I don't think you can hide schemas that way in practice. At Dropbox I saw that they had an abstract storage service in front of MySQL but every aspect of the underlying schema leaked out through the API in some way.
LoSboccacc 6y ago
first: I didn't say rdbms, I said database. that covers nosql as well. second: even if you don't have an explicit schema in the database, you still have to have a way to know which data goes in which column. just because it isn't written in a definition language doesn't mean the concept doesn't exist and doesn't require to be maintained as the application domain evolves.
john-shaffer 6y ago
This is a big reason why I like Datomic so much. I get the benefits of a schema, but the schema is basically append-only, so it's very easy to reason about. Updating the schema is instant. The last time I tried a project with a no-schema DB, it was a huge PITA. It sucks having a function fail on one single document because it had a number instead of a string. I love flexibility, but it is very valuable to have the right constraints in the right places. I quickly learned that a schema is just that.
- doteka 6y ago
  I don’t see how you’d have a number instead of a spring in a document in a typed language. Do you just directly dump user supplied data in your database without parsing/processing it? Otherwise, how is this not caught when it enters your system?
  - loopz 6y ago
    It is caught by the schema and never stored in the first place. Or you can solve it for every column/field in your system manually, and probably introduce subtle bugs and inconsistencies along the way. I'm for schemas, but not necessarily RDBMS for everything.
  - john-shaffer 6y ago
    Most document databases have JSON editors, which are very useful at times but don't have any schema checks. By "typed language" I suppose you mean "statically typed language". This isn't relevant because A) JSON types exist and B) I've never had a problem with unexpected types from user data in a dynamic language, not even in PHP. > Otherwise, how is this not caught when it enters your system? It should be caught, and it's very important that it be caught all of the time and not have any issues due to bugs or version mismatches. That's why the schema should live in the DB.
dagss 6y ago
And everything I have worked on would have been in far, far better shape if they had just reached for RDBMS instead of following the NoSQL hype and reimplementing what the RDBMS gives you in each backend service. YMMV I guess.