8 ms·

> "Saving" 15 minutes of dev time because you don't know what your schema is going to look like is going to cost you orders of magnitude more time down the road

by stouset 10y ago

> "Saving" 15 minutes of dev time because you don't know what your schema is going to look like is going to cost you orders of magnitude more time down the road asking yourself that same question. Yeah. In my experience, "schemaless" doesn't mean you have no schema. You still have a schema, it's just implicit and you don't have any tools available to actually operate on that schema.

smallnamespace 10y ago
This is pretty similar to the argument for static typing. Better to let the type system (help) prove correctness ahead of time, rather than wait until runtime to debug.
- hashkb 10y ago
  It's generally easier to fix an incorrect program than corrupt data. You're not wrong, but the magnitude doesn't compare.
  - digitalzombie 10y ago
    Seriously, eventually the data have to be structure some how so you can make use of it. Hence you end up writing map reduce down the road. Mind as well do it upfront with Relational DB.
  - adw 10y ago
    Yep. Everyone winds up with some war story about cleansing some multi-petabyte data store – but the better data engineers I know try very hard to avoid having two of them.
  - stouset 10y ago
    Yeah. I'm a huge fan of dynamically (yet strongly) typed languages like Ruby. They absolutely have their place. But weakly-typed data persistence or data interchange are absolutely terrible.
    - krylon 10y ago
      It's not just about type safety, though. A good RDBMS enforce things like referential integrity (FOREIGN KEY constraints) and allow you to express further constraints on your data (e.g. order.amount must be a positive number, a certain combination of columns must be unique across the table, etc.) And most (all?) RDBMS that have been around for a while have been tuned and optimized to support this efficiently. I remember reading that at some point (1980s-1990s-ish) DBMS vendors were buying compiler developers like crazy to help them with query optimization and such.
threeseed 10y ago
In almost all cases I've seen the schema is simply in the application layer as opposed to the database layer. The argument for schemas within databases was when you had dozens of users and applications all connected to a database. In the last few years we've seen micro services, Kafka queues and REST APIs replace the database as the primary integration points.
- throwaway91111 10y ago
  What sort of tooling does rethinkdb have for verifying the schema of existing data and inserts?
  - SamReidHughes 10y ago
    Soon, write hooks. Not the greatest solution though.
- 67726e 10y ago
  Your schema might sit in code, but you've got to reinvent the wheel on all sorts of validations and guarantees your database would provide. Whether you have queues and APIs is irrelevant, your persistent store should be a strong one.
virtuabhi 10y ago
Schemaless design enables the organizations to collect data now and decide what to do later. Though analysts who work on these datasets should be able to write code to parse the records when needed.
- 9gunpi 10y ago
  Offsetting database layout decisions is not having better flexibility, it is having less clue about database layout, unfortunately. Collecting data in some form and re/de-normalizing it in some useful fashion isn't exactly huge problem if you're not NetFlix or Google (and isn't huge problem even then). It just requires certain amount of education.
- naasking 10y ago
  > Though analysts who work on these datasets should be able to write code to parse the records when needed. Right, so the data conforms to a known schema that they can program against. Schemaless design doesn't somehow make this possible, it actually makes it difficult. If you already know what data you're capturing, then you have a schema.
  - digitalzombie 10y ago
    Schemaless imo is just kicking the can down the road. You always know what kind of data it is anyway. You don't just dump random data, at least not the majority of the time. When you grab data you're grabbing a specific data group.
  - hvidgaard 10y ago
    It's important to keep in mind that the "collect now and analyze later" is exactly that. You don't have to worry about constraints and integrity, you just capture what you think is of value, which is likely to be all the things unless you're working at really large scales. For all intents and purposes, storing everything as text files is just as good for this task.
    - naasking 10y ago
mickronome 10y ago
The problem with standard'ish SQL+RDBMS and their schemas is that types are not enforced well in SQL and most libraries and language integrations, thus we end up with the quagmire of static schemas that can not be reliably type checked when you use standard tooling. Also lacking in many DB systems is integrated support for tables withh heterogenous schemas that is supported by page/row-level cersioning and/or on line schema alteration. Having to rewrite a huge table only because you want to narrow a couple field for future data, it gets old quickly.
- bfrog 10y ago
  I've never had such issues with postgres personally. I know mysql is awful for schema migrations.
- jimktrains2 10y ago
  > types are not enforced well in SQL I'm not sure what you mean. Sane RDBMS do enforce types strictly. > language integrations, thus we end up with the quagmire of static schemas that can not be reliably type checked when you use standard tooling Languages that don't have static types to begin with maybe. > tables withh heterogenous schemas that is supported by page/row-level cersioning Can you give an example of what you mean? > on line schema alteration. > Having to rewrite a huge table only because you want to narrow a couple field for future data, it gets old quickly. Postgres doesn't need to lock the whole table for many types of alterations. It won't rewrite the table for null default fields.
- ankrgyl 10y ago
  > The problem with standard'ish SQL+RDBMS and their schemas is that types are not enforced well in SQL and most libraries and language integrations, thus we end up with the quagmire of static schemas that can not be reliably type checked when you use standard tooling. This used to be an issue with MySQL (and not really any other major RDBMS implementation), but with STRICT_MODE and the default settings on other implementations, SQL is fantastic at representing and enforcing static types. Furthermore, ORMs and tools like Apache Spark have really upped the integration between types in the database and types in languages. Practically speaking, RDBMS is really the only way you can have sane static typing in most applications (esp. ones that are built on dynamic languages). > Also lacking in many DB systems is integrated support for tables withh heterogenous schemas that is supported by page/row-level cersioning and/or on line schema alteration. Having to rewrite a huge table only because you want to narrow a couple field for future data, it gets old quickly. This is just not true. All of the major commercial RDBMS systems support both online schema alteration AND flexible/semi-structured types (JSON, XML). Again, MySQL and Postgres are behind the curve on this (although they both now support JSON). Commercial systems like SQL Server, Oracle, and MemSQL all have both.
pryelluw 10y ago
Worse is when the schema is written into the code. It turns tje codebase into a horrible mess.
- scaryclam 10y ago
  I'd probably change this a bit by saying that it's not a problem to have the schema in the code (it's got to be somewhere, preferably version controlled), it's a problem when your schema reflects (or even worse, is) your business logic. edit: s/you're/your/
  - pryelluw 10y ago
    Yes, exactly.

by stouset 10y ago

smallnamespace 10y ago

This is pretty similar to the argument for static typing. Better to let the type system (help) prove correctness ahead of time, rather than wait until runtime to debug.

hashkb 10y ago
It's generally easier to fix an incorrect program than corrupt data. You're not wrong, but the magnitude doesn't compare.
- digitalzombie 10y ago
  Seriously, eventually the data have to be structure some how so you can make use of it. Hence you end up writing map reduce down the road. Mind as well do it upfront with Relational DB.
- adw 10y ago
  Yep. Everyone winds up with some war story about cleansing some multi-petabyte data store – but the better data engineers I know try very hard to avoid having two of them.
- stouset 10y ago
  Yeah. I'm a huge fan of dynamically (yet strongly) typed languages like Ruby. They absolutely have their place. But weakly-typed data persistence or data interchange are absolutely terrible.
  - krylon 10y ago
    It's not just about type safety, though. A good RDBMS enforce things like referential integrity (FOREIGN KEY constraints) and allow you to express further constraints on your data (e.g. order.amount must be a positive number, a certain combination of columns must be unique across the table, etc.) And most (all?) RDBMS that have been around for a while have been tuned and optimized to support this efficiently. I remember reading that at some point (1980s-1990s-ish) DBMS vendors were buying compiler developers like crazy to help them with query optimization and such.

threeseed 10y ago

In almost all cases I've seen the schema is simply in the application layer as opposed to the database layer. The argument for schemas within databases was when you had dozens of users and applications all connected to a database. In the last few years we've seen micro services, Kafka queues and REST APIs replace the database as the primary integration points.

throwaway91111 10y ago
What sort of tooling does rethinkdb have for verifying the schema of existing data and inserts?
- SamReidHughes 10y ago
  Soon, write hooks. Not the greatest solution though.
67726e 10y ago
Your schema might sit in code, but you've got to reinvent the wheel on all sorts of validations and guarantees your database would provide. Whether you have queues and APIs is irrelevant, your persistent store should be a strong one.

virtuabhi 10y ago

Schemaless design enables the organizations to collect data now and decide what to do later. Though analysts who work on these datasets should be able to write code to parse the records when needed.

9gunpi 10y ago
Offsetting database layout decisions is not having better flexibility, it is having less clue about database layout, unfortunately. Collecting data in some form and re/de-normalizing it in some useful fashion isn't exactly huge problem if you're not NetFlix or Google (and isn't huge problem even then). It just requires certain amount of education.
naasking 10y ago
> Though analysts who work on these datasets should be able to write code to parse the records when needed. Right, so the data conforms to a known schema that they can program against. Schemaless design doesn't somehow make this possible, it actually makes it difficult. If you already know what data you're capturing, then you have a schema.
- digitalzombie 10y ago
  Schemaless imo is just kicking the can down the road. You always know what kind of data it is anyway. You don't just dump random data, at least not the majority of the time. When you grab data you're grabbing a specific data group.
- hvidgaard 10y ago
  It's important to keep in mind that the "collect now and analyze later" is exactly that. You don't have to worry about constraints and integrity, you just capture what you think is of value, which is likely to be all the things unless you're working at really large scales. For all intents and purposes, storing everything as text files is just as good for this task.
  - naasking 10y ago

mickronome 10y ago

The problem with standard'ish SQL+RDBMS and their schemas is that types are not enforced well in SQL and most libraries and language integrations, thus we end up with the quagmire of static schemas that can not be reliably type checked when you use standard tooling. Also lacking in many DB systems is integrated support for tables withh heterogenous schemas that is supported by page/row-level cersioning and/or on line schema alteration. Having to rewrite a huge table only because you want to narrow a couple field for future data, it gets old quickly.

bfrog 10y ago
I've never had such issues with postgres personally. I know mysql is awful for schema migrations.
jimktrains2 10y ago
> types are not enforced well in SQL I'm not sure what you mean. Sane RDBMS do enforce types strictly. > language integrations, thus we end up with the quagmire of static schemas that can not be reliably type checked when you use standard tooling Languages that don't have static types to begin with maybe. > tables withh heterogenous schemas that is supported by page/row-level cersioning Can you give an example of what you mean? > on line schema alteration. > Having to rewrite a huge table only because you want to narrow a couple field for future data, it gets old quickly. Postgres doesn't need to lock the whole table for many types of alterations. It won't rewrite the table for null default fields.
ankrgyl 10y ago
> The problem with standard'ish SQL+RDBMS and their schemas is that types are not enforced well in SQL and most libraries and language integrations, thus we end up with the quagmire of static schemas that can not be reliably type checked when you use standard tooling. This used to be an issue with MySQL (and not really any other major RDBMS implementation), but with STRICT_MODE and the default settings on other implementations, SQL is fantastic at representing and enforcing static types. Furthermore, ORMs and tools like Apache Spark have really upped the integration between types in the database and types in languages. Practically speaking, RDBMS is really the only way you can have sane static typing in most applications (esp. ones that are built on dynamic languages). > Also lacking in many DB systems is integrated support for tables withh heterogenous schemas that is supported by page/row-level cersioning and/or on line schema alteration. Having to rewrite a huge table only because you want to narrow a couple field for future data, it gets old quickly. This is just not true. All of the major commercial RDBMS systems support both online schema alteration AND flexible/semi-structured types (JSON, XML). Again, MySQL and Postgres are behind the curve on this (although they both now support JSON). Commercial systems like SQL Server, Oracle, and MemSQL all have both.

pryelluw 10y ago

Worse is when the schema is written into the code. It turns tje codebase into a horrible mess.

scaryclam 10y ago
I'd probably change this a bit by saying that it's not a problem to have the schema in the code (it's got to be somewhere, preferably version controlled), it's a problem when your schema reflects (or even worse, is) your business logic. edit: s/you're/your/
- pryelluw 10y ago
  Yes, exactly.