6 ms·

Can you explain in more detail about “my schema is my app layer”? EDIT: fixing autocorrect

toomuchtodo 8y ago
MongoDB by default is (was? It’s been a while since I’ve used it) schemaless, which means all of your data validation must take place in your app instead of the database. Your data integrity is then only as good as your weakest validation. Edit: scheme/schema autocorrect typos corrected. Thanks!
- taspeotis 8y ago
  > my scheme is my app layer ... MongoDB by default is (was? It’s been a while since I’ve used it) schemeless https://www.google.com/search?q=define%3Aschema https://www.google.com/search?q=define%3Aschema
  - deleted 8y ago
    [deleted]
- chisleu 8y ago
  Right, but most people using it use model or data repository patterns to ensure correctness. It does offer nearly infinite flexibility provided you use it correctly. You can add fields without any sort of DB work, you just start adding fields to rows as needed and let it catch up organically. There are use cases where mongo makes a lot of sense. It's very popular in the node.js / RAD world for sure. I certainly have never been a huge fan by any means. Only relatively recently did they solve distributed writes.
  - toomuchtodo 8y ago
    Unfortunately, I’d argue PostgreSQL gets you all the same benefits with JSON storage (fairly equivalent to Mongo docs), while also giving you all the goodness of a relational, transactional, schema enforcing RDBMS. PGSQL became Mongo faster than Mongo could become PGSQL.
    - WrtCdEvrydy 8y ago
      This is the same thing that happened in Java. Other languages started prototyping features... that eventually just end up being implemented in Java.
      - int_19h 8y ago
        Java did it way too slow, and that is a significant contributor to it being relegated to "legacy" in many areas. If it waited for the other languages to prototype stuff, it might have not been the case. The problem is that it waited for them to prototype it, refine it, release it, popularize it, and for their community to adopt it, before even starting to work on it in Java - which means that by the time they had it, most people who needed it were already elsewhere (not necessarily off JVM, just another language). Lambdas were a very good example - if you look at the closest competitor, C#, it got the first take on them back in 2005. Then a major refinement in 2008, adding type inference. By 2010, lambdas were idiomatic in C#. Java, in contrast, released the first version in 2014. And even then, they're still less powerful.
        kjeetgill 8y ago
        Huh. I didn't know they worked any differently in C#. In what ways are they less powerful in Java?
        int_19h 8y ago
        For example, in Java, lambdas cannot capture mutable variables from the outer scope - it must be final, or effectively final. C# always let you do that, from the very first implementation of lambdas.
        eropple 8y ago
        C# lets you do that because C# doesn't have a way to declare a local readonly/final variable at all. I significantly prefer features that encourage the use of `final` variables everywhere that it is possible in Java. I write C#, Java, and Kotlin in roughly equal measure. Each has its pluses. But the claim that Java's lambdas are worse because it doesn't let you--and this was a conscious design choice!--do something so potentially catastrophic and difficult to debug is an odd one.
- sonnyblarney 8y ago
  RDBMS only provides a limited degree of 'validation'. It still must exist fairly comprehensively in the app.
  - jacques_chester 8y ago
    Databases are, as their name suggests, closest to the data. Applications generally can't recreate ACID properties and specifically, they shouldn't be trying to.
    - sonnyblarney 8y ago
      "Applications generally can't recreate ACID properties" - why would they? ACID and 'data validation' are generally separate issues. Data generally has to be validated as it enters the business logic, before it gets stored in a DB. While a DB may in some cases ensure that data adheres to a schema, this usually does not fulfill all of the validation requirements.
      - jacques_chester 8y ago
        Validation often requires examine a model beyond "is this an int?". That model needs to be self-consistent. That requires atomic movements from consistent state to consistent state. You can do that yourself. Or let the database do it. For things where you can't express it in a database schema, sure. But you'd be surprised how far it gets you.
  - int_19h 8y ago
    On the contrary, RDBMS provides far more opportunity for validation, because it has all the data at its disposal, which can be queried as needed without the expense of crossing the boundary.
    - Bucephalus355 8y ago
      What is the purpose of validation would you say with modern computers? At one time, specifying the exact number of chars was good for squeezing out as much storage as possible, but less so today.
- bunderbunder 8y ago
  I've always preferred the terms "schema on write" and "schema on read" to schemaful/schemaless. At some point, you are always going to have to get the data into some sort of consistent model, so that you can operate on it in a predictable and sane way. So there's no question of there being a schema, even if it's only implicit. The question is, do you apply the schema once, when you write to the data store, so that the data at rest is consistently structured? Or do you allow it to be inconsistent in the storage layer, and instead apply the schema and re-validate the data every time you read from it? There are valid reasons why one might choose either approach. Which is not to say that valid reasons always play in to the decision to choose one approach or the other.
  - 616c 8y ago
    Excellent perspective. I start using this at work!
  - bpye 8y ago
    I'm tired and haven't often dealt with database systems. I'm struggling to see significant benefits for schema on read style systems - maybe progressive migration? I'm not convinced...
    - pryce 8y ago
      Not having to know all / as many of the structural details up-front could be of value in some use-cases. It can translate to reducing time-to-start cutting code, which can (in some cases) be a business priority, and can lead to identifying critical dependency problems earlier in development. I'd happily agree that's an inappropriate model in close to 99% of cases, and that even if it was the right model one could (and most likely should) still use a decent database for this anyway.
    - mikekchar 8y ago
      When you want to do validation depends on when you can do something about it. I work with a NO-SQL DB at work and while it wouldn't be my choice for most things I would use a DB for, the lack of validation has some benefits. A good example is where you have no ability to validate input from a user, but where you need to store the data anyway. The last thing you want is your noisy data being kicked out by the DB because it doesn't follow a DB constraint. Sometimes you want to go in afterwards and say, "Show me all the data which is incorrect". This is also useful for dealing with important data sent by other systems which have been coded by people other than you. The get the data wrong (or are using older versions of specs, etc) but you want to store what they sent you anyway. Then you can go in later and sort it out by hand. I don't think that kind of thing is particularly common, but there are definite use cases. In our particular case we use it for financial data where we want the data we are given even if it is flawed. I think the OP is 100% correct. You have to write that validation somewhere or else you are in big trouble. Usually it is easier and more convenient to do it at the DB layer, but sometimes you choose to do it somewhere else.

MongoDB by default is (was? It’s been a while since I’ve used it) schemaless, which means all of your data validation must take place in your app instead of the database. Your data integrity is then only as good as your weakest validation. Edit: scheme/schema autocorrect typos corrected. Thanks!

taspeotis 8y ago
> my scheme is my app layer ... MongoDB by default is (was? It’s been a while since I’ve used it) schemeless https://www.google.com/search?q=define%3Aschema https://www.google.com/search?q=define%3Aschema
- deleted 8y ago
  [deleted]
chisleu 8y ago
Right, but most people using it use model or data repository patterns to ensure correctness. It does offer nearly infinite flexibility provided you use it correctly. You can add fields without any sort of DB work, you just start adding fields to rows as needed and let it catch up organically. There are use cases where mongo makes a lot of sense. It's very popular in the node.js / RAD world for sure. I certainly have never been a huge fan by any means. Only relatively recently did they solve distributed writes.
- toomuchtodo 8y ago
  Unfortunately, I’d argue PostgreSQL gets you all the same benefits with JSON storage (fairly equivalent to Mongo docs), while also giving you all the goodness of a relational, transactional, schema enforcing RDBMS. PGSQL became Mongo faster than Mongo could become PGSQL.
  - WrtCdEvrydy 8y ago
    This is the same thing that happened in Java. Other languages started prototyping features... that eventually just end up being implemented in Java.
    - int_19h 8y ago
      Java did it way too slow, and that is a significant contributor to it being relegated to "legacy" in many areas. If it waited for the other languages to prototype stuff, it might have not been the case. The problem is that it waited for them to prototype it, refine it, release it, popularize it, and for their community to adopt it, before even starting to work on it in Java - which means that by the time they had it, most people who needed it were already elsewhere (not necessarily off JVM, just another language). Lambdas were a very good example - if you look at the closest competitor, C#, it got the first take on them back in 2005. Then a major refinement in 2008, adding type inference. By 2010, lambdas were idiomatic in C#. Java, in contrast, released the first version in 2014. And even then, they're still less powerful.
      - kjeetgill 8y ago
        Huh. I didn't know they worked any differently in C#. In what ways are they less powerful in Java?
        int_19h 8y ago
        For example, in Java, lambdas cannot capture mutable variables from the outer scope - it must be final, or effectively final. C# always let you do that, from the very first implementation of lambdas.
        eropple 8y ago
        C# lets you do that because C# doesn't have a way to declare a local readonly/final variable at all. I significantly prefer features that encourage the use of `final` variables everywhere that it is possible in Java. I write C#, Java, and Kotlin in roughly equal measure. Each has its pluses. But the claim that Java's lambdas are worse because it doesn't let you--and this was a conscious design choice!--do something so potentially catastrophic and difficult to debug is an odd one.
sonnyblarney 8y ago
RDBMS only provides a limited degree of 'validation'. It still must exist fairly comprehensively in the app.
- jacques_chester 8y ago
  Databases are, as their name suggests, closest to the data. Applications generally can't recreate ACID properties and specifically, they shouldn't be trying to.
  - sonnyblarney 8y ago
    "Applications generally can't recreate ACID properties" - why would they? ACID and 'data validation' are generally separate issues. Data generally has to be validated as it enters the business logic, before it gets stored in a DB. While a DB may in some cases ensure that data adheres to a schema, this usually does not fulfill all of the validation requirements.
    - jacques_chester 8y ago
      Validation often requires examine a model beyond "is this an int?". That model needs to be self-consistent. That requires atomic movements from consistent state to consistent state. You can do that yourself. Or let the database do it. For things where you can't express it in a database schema, sure. But you'd be surprised how far it gets you.
- int_19h 8y ago
  On the contrary, RDBMS provides far more opportunity for validation, because it has all the data at its disposal, which can be queried as needed without the expense of crossing the boundary.
  - Bucephalus355 8y ago
    What is the purpose of validation would you say with modern computers? At one time, specifying the exact number of chars was good for squeezing out as much storage as possible, but less so today.
bunderbunder 8y ago
I've always preferred the terms "schema on write" and "schema on read" to schemaful/schemaless. At some point, you are always going to have to get the data into some sort of consistent model, so that you can operate on it in a predictable and sane way. So there's no question of there being a schema, even if it's only implicit. The question is, do you apply the schema once, when you write to the data store, so that the data at rest is consistently structured? Or do you allow it to be inconsistent in the storage layer, and instead apply the schema and re-validate the data every time you read from it? There are valid reasons why one might choose either approach. Which is not to say that valid reasons always play in to the decision to choose one approach or the other.
- 616c 8y ago
  Excellent perspective. I start using this at work!
- bpye 8y ago
  I'm tired and haven't often dealt with database systems. I'm struggling to see significant benefits for schema on read style systems - maybe progressive migration? I'm not convinced...
  - pryce 8y ago
    Not having to know all / as many of the structural details up-front could be of value in some use-cases. It can translate to reducing time-to-start cutting code, which can (in some cases) be a business priority, and can lead to identifying critical dependency problems earlier in development. I'd happily agree that's an inappropriate model in close to 99% of cases, and that even if it was the right model one could (and most likely should) still use a decent database for this anyway.
  - mikekchar 8y ago
    When you want to do validation depends on when you can do something about it. I work with a NO-SQL DB at work and while it wouldn't be my choice for most things I would use a DB for, the lack of validation has some benefits. A good example is where you have no ability to validate input from a user, but where you need to store the data anyway. The last thing you want is your noisy data being kicked out by the DB because it doesn't follow a DB constraint. Sometimes you want to go in afterwards and say, "Show me all the data which is incorrect". This is also useful for dealing with important data sent by other systems which have been coded by people other than you. The get the data wrong (or are using older versions of specs, etc) but you want to store what they sent you anyway. Then you can go in later and sort it out by hand. I don't think that kind of thing is particularly common, but there are definite use cases. In our particular case we use it for financial data where we want the data we are given even if it is flawed. I think the OP is 100% correct. You have to write that validation somewhere or else you are in big trouble. Usually it is easier and more convenient to do it at the DB layer, but sometimes you choose to do it somewhere else.