11 ms·

Smug self-righteousness

toomuchtodo 8y ago
I’ll allow it. Years later, I’m still miffed at Graylog (centralized logging engine) for having required MongoDB for a small bit of auth and meta storage that could’ve easily been done in MySQL or PostgreSQL (RDS even), forcing the need for that much more ops work for a small Mongo cluster for HA. Everyone deprecating the use of Mongo is a welcoming turn of events. I shall recall these dark days to the next generation as “NoSQL Madness”, or more colloquially, “my schema is my app layer”.
- Bucephalus355 8y ago
  Can you explain in more detail about “my schema is my app layer”? EDIT: fixing autocorrect
  - toomuchtodo 8y ago
    MongoDB by default is (was? It’s been a while since I’ve used it) schemaless, which means all of your data validation must take place in your app instead of the database. Your data integrity is then only as good as your weakest validation. Edit: scheme/schema autocorrect typos corrected. Thanks!
    - taspeotis 8y ago
      > my scheme is my app layer ... MongoDB by default is (was? It’s been a while since I’ve used it) schemeless https://www.google.com/search?q=define%3Aschema https://www.google.com/search?q=define%3Aschema
      - deleted 8y ago
        [deleted]
    - chisleu 8y ago
      Right, but most people using it use model or data repository patterns to ensure correctness. It does offer nearly infinite flexibility provided you use it correctly. You can add fields without any sort of DB work, you just start adding fields to rows as needed and let it catch up organically. There are use cases where mongo makes a lot of sense. It's very popular in the node.js / RAD world for sure. I certainly have never been a huge fan by any means. Only relatively recently did they solve distributed writes.
      - toomuchtodo 8y ago
        Unfortunately, I’d argue PostgreSQL gets you all the same benefits with JSON storage (fairly equivalent to Mongo docs), while also giving you all the goodness of a relational, transactional, schema enforcing RDBMS. PGSQL became Mongo faster than Mongo could become PGSQL.
        WrtCdEvrydy 8y ago
        This is the same thing that happened in Java. Other languages started prototyping features... that eventually just end up being implemented in Java.
        int_19h 8y ago
        Java did it way too slow, and that is a significant contributor to it being relegated to "legacy" in many areas. If it waited for the other languages to prototype stuff, it might have not been the case. The problem is that it waited for them to prototype it, refine it, release it, popularize it, and for their community to adopt it, before even starting to work on it in Java - which means that by the time they had it, most people who needed it were already elsewhere (not necessarily off JVM, just another language). Lambdas were a very good example - if you look at the closest competitor, C#, it got the first take on them back in 2005. Then a major refinement in 2008, adding type inference. By 2010, lambdas were idiomatic in C#. Java, in contrast, released the first version in 2014. And even then, they're still less powerful.
    - sonnyblarney 8y ago
      RDBMS only provides a limited degree of 'validation'. It still must exist fairly comprehensively in the app.
      - jacques_chester 8y ago
        Databases are, as their name suggests, closest to the data. Applications generally can't recreate ACID properties and specifically, they shouldn't be trying to.
        sonnyblarney 8y ago
        "Applications generally can't recreate ACID properties" - why would they? ACID and 'data validation' are generally separate issues. Data generally has to be validated as it enters the business logic, before it gets stored in a DB. While a DB may in some cases ensure that data adheres to a schema, this usually does not fulfill all of the validation requirements.
        jacques_chester 8y ago
        Validation often requires examine a model beyond "is this an int?". That model needs to be self-consistent. That requires atomic movements from consistent state to consistent state. You can do that yourself. Or let the database do it. For things where you can't express it in a database schema, sure. But you'd be surprised how far it gets you.
      - int_19h 8y ago
        On the contrary, RDBMS provides far more opportunity for validation, because it has all the data at its disposal, which can be queried as needed without the expense of crossing the boundary.
        Bucephalus355 8y ago
        What is the purpose of validation would you say with modern computers? At one time, specifying the exact number of chars was good for squeezing out as much storage as possible, but less so today.
    - bunderbunder 8y ago
      I've always preferred the terms "schema on write" and "schema on read" to schemaful/schemaless. At some point, you are always going to have to get the data into some sort of consistent model, so that you can operate on it in a predictable and sane way. So there's no question of there being a schema, even if it's only implicit. The question is, do you apply the schema once, when you write to the data store, so that the data at rest is consistently structured? Or do you allow it to be inconsistent in the storage layer, and instead apply the schema and re-validate the data every time you read from it? There are valid reasons why one might choose either approach. Which is not to say that valid reasons always play in to the decision to choose one approach or the other.
      - 616c 8y ago
        Excellent perspective. I start using this at work!
      - bpye 8y ago
        I'm tired and haven't often dealt with database systems. I'm struggling to see significant benefits for schema on read style systems - maybe progressive migration? I'm not convinced...
        pryce 8y ago
        Not having to know all / as many of the structural details up-front could be of value in some use-cases. It can translate to reducing time-to-start cutting code, which can (in some cases) be a business priority, and can lead to identifying critical dependency problems earlier in development. I'd happily agree that's an inappropriate model in close to 99% of cases, and that even if it was the right model one could (and most likely should) still use a decent database for this anyway.
        mikekchar 8y ago
        When you want to do validation depends on when you can do something about it. I work with a NO-SQL DB at work and while it wouldn't be my choice for most things I would use a DB for, the lack of validation has some benefits. A good example is where you have no ability to validate input from a user, but where you need to store the data anyway. The last thing you want is your noisy data being kicked out by the DB because it doesn't follow a DB constraint. Sometimes you want to go in afterwards and say, "Show me all the data which is incorrect". This is also useful for dealing with important data sent by other systems which have been coded by people other than you. The get the data wrong (or are using older versions of specs, etc) but you want to store what they sent you anyway. Then you can go in later and sort it out by hand. I don't think that kind of thing is particularly common, but there are definite use cases. In our particular case we use it for financial data where we want the data we are given even if it is flawed. I think the OP is 100% correct. You have to write that validation somewhere or else you are in big trouble. Usually it is easier and more convenient to do it at the DB layer, but sometimes you choose to do it somewhere else.
- throwaway5752 8y ago
  "my scheme is my app layer" To be fair you can do use schema validators in Mongo. Not sure it's widespread in practice. And there are other distributed databases that aren't document stores that have schemas and various subsets of SQL implemented.
  - deleted 8y ago
    [deleted]
- bane 8y ago
  I once worked at a place that, at some point in the past, had developed some kind of semantic graph database that never gained market traction. The author of it had ended up as CTO and kept seeking out uses for his work and ended up finding all kinds of odd places for it to live, including as the auth database in a large scale document analytics system and running part of the payroll. We were constantly running into all kinds of scalability issues where this 12 year old component was often the source of our pain, but he'd never even entertain a conversation to eliminate it and consolidate or replace it with something better.
  - seibelj 8y ago
    This is the definition of software engineering hubris distilled into a few paragraphs
  - duxup 8y ago
    I worked for a company that was old but had a fancy new product. It was just about to be released when.... A competitor bought us for stupid money. They fired most folks (not my small department) had their CTO decide between our fancy new product, and the product he created. There was no question, our product was amazing, his Frankenstein was two pieces of equipment cabled together in three items the footprint ... and still did less, and was quite a ways away from being "ready". If there was a silver lining, the Frankenstein product doomed that company and we got bought by the far more competent competitor they had.
    - qbaqbaqba 8y ago
      Happy ending.
      - duxup 8y ago
        It's surprising how that works. I've been through numerous mergers, acquisitions, been in the company acquiring a company. Lots of hair pulling and I can say that it was really hard to predict the outcome for any individual positive or negative in every case until long afterward.
- djrobstep 8y ago
  My pet theory is that NoSQL took off purely because people were sick of having to manage schema changes. Unfortunately, people reacted to being (justifiably) frustrated with schemas by throwing strict schemas out entirely, instead of making better schema management/migration tools.
  - staticassertion 8y ago
    Tell that to AWS. They've banned relational databases for specific workloads because Dynamo (nosql) provides more consistent performance, and is easier to operate. Tons of conflation of Mongo's problems with those of nosql in this thread.
    - djrobstep 8y ago
      Your average dev is not making decisions based on what might work best for one particular problem Amazon has.
      - staticassertion 8y ago
        Do you think all of the companies that chose Cassandra and Dynamo were wrong to do so? There's no use case for NoSQL? There were no lessons learned, value adds from NoSQL? How do you explain the 'NewSQL' approach, which seems to be so clearly borne of what we've learned from NoSQL? It should be obvious that NoSQL has value, regardless of the issues with one of the earlier NoSQL DBs.
        pjmlp 8y ago
        I don't see a value other than fashion driven development, specially when comparing the bare bones browser GUI for Dynamo with something like SQL Server Management Studio or that whole story with primary and secondary indexes, with prices being set by index usage.
        menacingly 8y ago
- baroffoos 8y ago
  Setting up graylog was one of the worst mistakes I made. It took forever to get all the required software installed and configured and then it was taking up all the ram on the server doing fuck all.
  - SteveNuts 8y ago
    That was just Java doing Java things. Business as usual.
    - RhodesianHunter 8y ago
      This is a nonsense comment doing language troll things.
  - deleted 8y ago
    [deleted]
  - atm0sphere 8y ago
    There were single script 1-click installs for it in bash all over for me..but yeah jvm is a hog.
    - yjftsjthsd-h 8y ago
      My experience with those one-liner installs is that they usually work... strictly speaking. They don't scale, they don't deal with edge cases, they know nothing of your environment. They install one piece of software (in an "interesting" way that won't upgrade), and that's it.
      - sunnymanokl 8y ago
        My last pay check was $9500 working 12 hours a week online. My sisters friend has been averaging 15k for months now and she works about 20 hours a week. I can't believe how easy it was once I tried it out. This is what I do, HERE►► http://www.worktoday33.com http://www.worktoday33.com
      - 8y ago
- Sandra_lisa4 8y ago
  XzXzXZXX
- dominotw 8y ago
  It was just a discovery phase.
- pwm 8y ago
  I agree with the general sentiment just wanted to point out that "my schema is my app layer" has some valid use-cases. At my current job we deal with highly complex schemas (modelling insurance contracts). Correctness is paramount but at the same time you need flexibility in terms of change over time. Defining these schemas at the DB level would be painful, a bit like writing web apps in assembly. Languages like Haskell (which we use) help here with their rich and expressive typing capabilities, so we can model these complex domains expressively and use the DB only for persistence. Admittedly it does have downsides, like having to write your own migration layer, but for this use-case the benefits outweigh the pains. PS.: We do use Postgres though ;) as it has 1st class json support and at the same time we have the luxury of using its relational capabilities where needed (think of a hybrid model).
- speedplane 8y ago
  Maybe I’m a rare exception, but I chose NoSQL early on when it was still “hot” and have never looked back. We’ve grown from a couple megabytes of data do several dozen terabytes and have had countless issues, but scaling our database was never one of them.
- twic 8y ago
  Well, happily, those days are gone for good. Who needs NoSQL when you have the blockchain!
  - ixtli 8y ago
    Glorious, cursed comment.
deleted 8y ago
[deleted]
bostonpete 8y ago
Wish we had HN gold for comments like this.

I’ll allow it. Years later, I’m still miffed at Graylog (centralized logging engine) for having required MongoDB for a small bit of auth and meta storage that could’ve easily been done in MySQL or PostgreSQL (RDS even), forcing the need for that much more ops work for a small Mongo cluster for HA. Everyone deprecating the use of Mongo is a welcoming turn of events. I shall recall these dark days to the next generation as “NoSQL Madness”, or more colloquially, “my schema is my app layer”.

Bucephalus355 8y ago
Can you explain in more detail about “my schema is my app layer”? EDIT: fixing autocorrect
- toomuchtodo 8y ago
  MongoDB by default is (was? It’s been a while since I’ve used it) schemaless, which means all of your data validation must take place in your app instead of the database. Your data integrity is then only as good as your weakest validation. Edit: scheme/schema autocorrect typos corrected. Thanks!
  - taspeotis 8y ago
    > my scheme is my app layer ... MongoDB by default is (was? It’s been a while since I’ve used it) schemeless https://www.google.com/search?q=define%3Aschema https://www.google.com/search?q=define%3Aschema
    - deleted 8y ago
      [deleted]
  - chisleu 8y ago
    Right, but most people using it use model or data repository patterns to ensure correctness. It does offer nearly infinite flexibility provided you use it correctly. You can add fields without any sort of DB work, you just start adding fields to rows as needed and let it catch up organically. There are use cases where mongo makes a lot of sense. It's very popular in the node.js / RAD world for sure. I certainly have never been a huge fan by any means. Only relatively recently did they solve distributed writes.
    - toomuchtodo 8y ago
      Unfortunately, I’d argue PostgreSQL gets you all the same benefits with JSON storage (fairly equivalent to Mongo docs), while also giving you all the goodness of a relational, transactional, schema enforcing RDBMS. PGSQL became Mongo faster than Mongo could become PGSQL.
      - WrtCdEvrydy 8y ago
        This is the same thing that happened in Java. Other languages started prototyping features... that eventually just end up being implemented in Java.
        int_19h 8y ago
        Java did it way too slow, and that is a significant contributor to it being relegated to "legacy" in many areas. If it waited for the other languages to prototype stuff, it might have not been the case. The problem is that it waited for them to prototype it, refine it, release it, popularize it, and for their community to adopt it, before even starting to work on it in Java - which means that by the time they had it, most people who needed it were already elsewhere (not necessarily off JVM, just another language). Lambdas were a very good example - if you look at the closest competitor, C#, it got the first take on them back in 2005. Then a major refinement in 2008, adding type inference. By 2010, lambdas were idiomatic in C#. Java, in contrast, released the first version in 2014. And even then, they're still less powerful.
  - sonnyblarney 8y ago
    RDBMS only provides a limited degree of 'validation'. It still must exist fairly comprehensively in the app.
    - jacques_chester 8y ago
      Databases are, as their name suggests, closest to the data. Applications generally can't recreate ACID properties and specifically, they shouldn't be trying to.
      - sonnyblarney 8y ago
        "Applications generally can't recreate ACID properties" - why would they? ACID and 'data validation' are generally separate issues. Data generally has to be validated as it enters the business logic, before it gets stored in a DB. While a DB may in some cases ensure that data adheres to a schema, this usually does not fulfill all of the validation requirements.
        jacques_chester 8y ago
        Validation often requires examine a model beyond "is this an int?". That model needs to be self-consistent. That requires atomic movements from consistent state to consistent state. You can do that yourself. Or let the database do it. For things where you can't express it in a database schema, sure. But you'd be surprised how far it gets you.
    - int_19h 8y ago
      On the contrary, RDBMS provides far more opportunity for validation, because it has all the data at its disposal, which can be queried as needed without the expense of crossing the boundary.
      - Bucephalus355 8y ago
        What is the purpose of validation would you say with modern computers? At one time, specifying the exact number of chars was good for squeezing out as much storage as possible, but less so today.
  - bunderbunder 8y ago
    I've always preferred the terms "schema on write" and "schema on read" to schemaful/schemaless. At some point, you are always going to have to get the data into some sort of consistent model, so that you can operate on it in a predictable and sane way. So there's no question of there being a schema, even if it's only implicit. The question is, do you apply the schema once, when you write to the data store, so that the data at rest is consistently structured? Or do you allow it to be inconsistent in the storage layer, and instead apply the schema and re-validate the data every time you read from it? There are valid reasons why one might choose either approach. Which is not to say that valid reasons always play in to the decision to choose one approach or the other.
    - 616c 8y ago
      Excellent perspective. I start using this at work!
    - bpye 8y ago
      I'm tired and haven't often dealt with database systems. I'm struggling to see significant benefits for schema on read style systems - maybe progressive migration? I'm not convinced...
      - pryce 8y ago
        Not having to know all / as many of the structural details up-front could be of value in some use-cases. It can translate to reducing time-to-start cutting code, which can (in some cases) be a business priority, and can lead to identifying critical dependency problems earlier in development. I'd happily agree that's an inappropriate model in close to 99% of cases, and that even if it was the right model one could (and most likely should) still use a decent database for this anyway.
      - mikekchar 8y ago
        When you want to do validation depends on when you can do something about it. I work with a NO-SQL DB at work and while it wouldn't be my choice for most things I would use a DB for, the lack of validation has some benefits. A good example is where you have no ability to validate input from a user, but where you need to store the data anyway. The last thing you want is your noisy data being kicked out by the DB because it doesn't follow a DB constraint. Sometimes you want to go in afterwards and say, "Show me all the data which is incorrect". This is also useful for dealing with important data sent by other systems which have been coded by people other than you. The get the data wrong (or are using older versions of specs, etc) but you want to store what they sent you anyway. Then you can go in later and sort it out by hand. I don't think that kind of thing is particularly common, but there are definite use cases. In our particular case we use it for financial data where we want the data we are given even if it is flawed. I think the OP is 100% correct. You have to write that validation somewhere or else you are in big trouble. Usually it is easier and more convenient to do it at the DB layer, but sometimes you choose to do it somewhere else.
throwaway5752 8y ago
"my scheme is my app layer" To be fair you can do use schema validators in Mongo. Not sure it's widespread in practice. And there are other distributed databases that aren't document stores that have schemas and various subsets of SQL implemented.
- deleted 8y ago
  [deleted]
bane 8y ago
I once worked at a place that, at some point in the past, had developed some kind of semantic graph database that never gained market traction. The author of it had ended up as CTO and kept seeking out uses for his work and ended up finding all kinds of odd places for it to live, including as the auth database in a large scale document analytics system and running part of the payroll. We were constantly running into all kinds of scalability issues where this 12 year old component was often the source of our pain, but he'd never even entertain a conversation to eliminate it and consolidate or replace it with something better.
- seibelj 8y ago
  This is the definition of software engineering hubris distilled into a few paragraphs
- duxup 8y ago
  I worked for a company that was old but had a fancy new product. It was just about to be released when.... A competitor bought us for stupid money. They fired most folks (not my small department) had their CTO decide between our fancy new product, and the product he created. There was no question, our product was amazing, his Frankenstein was two pieces of equipment cabled together in three items the footprint ... and still did less, and was quite a ways away from being "ready". If there was a silver lining, the Frankenstein product doomed that company and we got bought by the far more competent competitor they had.
  - qbaqbaqba 8y ago
    Happy ending.
    - duxup 8y ago
      It's surprising how that works. I've been through numerous mergers, acquisitions, been in the company acquiring a company. Lots of hair pulling and I can say that it was really hard to predict the outcome for any individual positive or negative in every case until long afterward.
djrobstep 8y ago
My pet theory is that NoSQL took off purely because people were sick of having to manage schema changes. Unfortunately, people reacted to being (justifiably) frustrated with schemas by throwing strict schemas out entirely, instead of making better schema management/migration tools.
- staticassertion 8y ago
  Tell that to AWS. They've banned relational databases for specific workloads because Dynamo (nosql) provides more consistent performance, and is easier to operate. Tons of conflation of Mongo's problems with those of nosql in this thread.
  - djrobstep 8y ago
    Your average dev is not making decisions based on what might work best for one particular problem Amazon has.
    - staticassertion 8y ago
      Do you think all of the companies that chose Cassandra and Dynamo were wrong to do so? There's no use case for NoSQL? There were no lessons learned, value adds from NoSQL? How do you explain the 'NewSQL' approach, which seems to be so clearly borne of what we've learned from NoSQL? It should be obvious that NoSQL has value, regardless of the issues with one of the earlier NoSQL DBs.
      - pjmlp 8y ago
        I don't see a value other than fashion driven development, specially when comparing the bare bones browser GUI for Dynamo with something like SQL Server Management Studio or that whole story with primary and secondary indexes, with prices being set by index usage.
        menacingly 8y ago
baroffoos 8y ago
Setting up graylog was one of the worst mistakes I made. It took forever to get all the required software installed and configured and then it was taking up all the ram on the server doing fuck all.
- SteveNuts 8y ago
  That was just Java doing Java things. Business as usual.
  - RhodesianHunter 8y ago
    This is a nonsense comment doing language troll things.
- deleted 8y ago
  [deleted]
- atm0sphere 8y ago
  There were single script 1-click installs for it in bash all over for me..but yeah jvm is a hog.
  - yjftsjthsd-h 8y ago
    My experience with those one-liner installs is that they usually work... strictly speaking. They don't scale, they don't deal with edge cases, they know nothing of your environment. They install one piece of software (in an "interesting" way that won't upgrade), and that's it.
    - sunnymanokl 8y ago
      My last pay check was $9500 working 12 hours a week online. My sisters friend has been averaging 15k for months now and she works about 20 hours a week. I can't believe how easy it was once I tried it out. This is what I do, HERE►► http://www.worktoday33.com http://www.worktoday33.com
    - 8y ago
Sandra_lisa4 8y ago
XzXzXZXX
dominotw 8y ago
It was just a discovery phase.
pwm 8y ago
I agree with the general sentiment just wanted to point out that "my schema is my app layer" has some valid use-cases. At my current job we deal with highly complex schemas (modelling insurance contracts). Correctness is paramount but at the same time you need flexibility in terms of change over time. Defining these schemas at the DB level would be painful, a bit like writing web apps in assembly. Languages like Haskell (which we use) help here with their rich and expressive typing capabilities, so we can model these complex domains expressively and use the DB only for persistence. Admittedly it does have downsides, like having to write your own migration layer, but for this use-case the benefits outweigh the pains. PS.: We do use Postgres though ;) as it has 1st class json support and at the same time we have the luxury of using its relational capabilities where needed (think of a hybrid model).
speedplane 8y ago
Maybe I’m a rare exception, but I chose NoSQL early on when it was still “hot” and have never looked back. We’ve grown from a couple megabytes of data do several dozen terabytes and have had countless issues, but scaling our database was never one of them.
twic 8y ago
Well, happily, those days are gone for good. Who needs NoSQL when you have the blockchain!
- ixtli 8y ago
  Glorious, cursed comment.

deleted 8y ago

[deleted]

bostonpete 8y ago

Wish we had HN gold for comments like this.