12 ms·

What, you'd rather drop data on the floor? Your scientists would prefer you keep all measurements, even the bad ones. Sometimes those "bad" data actually have i

by xapata 9y ago

What, you'd rather drop data on the floor? Your scientists would prefer you keep all measurements, even the bad ones. Sometimes those "bad" data actually have information. Why not have two databases, one with all data without constraints and another, or even many which impose various constraints?

stouset 9y ago
Bad data of this for is almost always the result of bugs, not of real data failing to conform to your modeling. In this case, yes, you should absolutely be dropping it on the floor (and alerting the devs with an exception) rather than creating a mess that needs cleaning up later.
- xapata 9y ago
  How would you know if you'd made a false negative if you drop all "bad" data? It's more of a mess when you realize you've accidentally been losing records for the last few years.
  - mixmastamyk 9y ago
    The data is not dropped silently but very loudly, immediately. Unless of course you are using Mysql, well then...
    - xapata 9y ago
      So you stop the program immediately and allow no more transactions while you debug? Unlikely. Keep the data. Log that it's weird.
      - stouset 9y ago
        What is your software development background? I ask because it would not have once occurred to me in the past fifteen years of software development that “enforcing database constraints” in any way implies shutting everything down and stopping transactions until the cause of a violation is rooted out. If you’re approaching it from the position of collecting raw data from sensors or something, that might make sense. But from the perspective of a webapp, allowing duplicate (or empty) usernames or failing to keep foreign key relations valid often violate basic assumptions of soundness that the application is based upon. In these cases, the right answer is to drop the data on the floor (loudly) so a developer can look at it. Inconveniencing a customer once here is infinitely better than inconveniencing them constantly due to the site being buggy or broken as a result of the invalid data, and prevents needing to migrate the data in the future which can actually be impossible to do correctly in some cases.
        xapata 9y ago
        > background Web dev for some years, then data science. > implies shutting everything down That's what "loudly" implied to me. Loud means crash, no? > basic assumptions of soundness I like to enforce that at the application layer, so that I can quickly swap storage without worrying. Whether I'm writing to database or a mock object, I'd hope the exceptions that arise in the application are the same. Things like unique index are easy to implement with a mock object, but more complex constraints I'd rather enforce in code that I find easier to read. > buggy or broken All I'm saying is not to be an absolutist. Some things are good to have as database constraints, others are good as application logic. I'm arguing against the idea that "validations are [never] enough." In contrast, database constraints are almost never enough to manage the complexity of data integrity.
        mixmastamyk 9y ago
        App should not crash but return/log the database error, ideally translated to layman terms on the customer end. Logs sent high priority to dev team for fixing.
        xapata 9y ago
        Indeed, I agree.
      - Slansitartop 9y ago
        > So you stop the program immediately and allow no more transactions while you debug? Unlikely. You stop the transaction on a constraint violation, not usually the entire system. > Keep the data. Log that it's weird. I've worked on systems that do that, and they're nightmares. The problem with that style can be summed up in "if it doesn't look broke, people don't bother fixing it." No one actually watches those logs, and users don't try to fix data that was "accepted" with no errors. The result is bad data piles up, and often causes problems in other parts of the system. Maybe you (as a developer) love working on support tickets to manually fix bad data the users entered or a buggy process created, but I sure as hell don't. The solution to this is to fail fast: https://en.wikipedia.org/wiki/Fail-fast https://en.wikipedia.org/wiki/Fail-fast.
        deleted 9y ago
        [deleted]
        xapata 9y ago
        I think our practices are closer than you realize. Maybe I didn't explain myself well. The idea that troubles me is including more strict constraints in the database than in the application itself, because that indicates the application isn't fully tested. If the application is well tested, then it doesn't matter where the logic resides, db or app, so there's no standing to the claim that application logic is never enough. Also, there are many kinds of data. Transactions, sure, reject if it doesn't match. Telemetry data, I'd like to hang on to it regardless of how weird it seems.
midev 9y ago
> Your scientists would prefer you keep all measurements, even the bad ones. Sometimes those "bad" data actually have information. A constraint can allow "bad" data if the "bad" data is still semantically valid for that column. A freeform text column where the scientist writes notes might not need any constraints. But a column used to send measurements to a robot to manufacture something, must have some protections around what type of data it can store. And no, the scientist doesn't get to input "bad" data that makes the robot lose its mind. Sometimes yes, you drop data on the floor.
- xapata 9y ago
  I've seen it go the other way plenty of times -- the dev thinks a system should only receive numbers, but occasionally a string is valid as well.
  - always_good 9y ago
    Then the data model was wrong and they migrate their scheme appropriately. I don't think you understand that this is a core part of why you want a strict database: so you can perceive mismatches between your assumptions and reality and then respond precisely. Do you think it's superior to be in a situation where you thought all of your IDs were numbers, you built your system with that expectation, and it turns out strings have been silently making their way into your data? The difference between the two scenarios is that the person that used constraints knows when and why and where they got unexpected data and they can take more decisive, informed action. Meanwhile the other guy is desperately writing queries to find unexpected data, shooting around in the dark, and wondering what else slipped through.
    - hinkley 9y ago
      Unfortunately he sounds like about half the Node developers I work with. Nobody wants to make concrete assertions. To decide anything. Usually this count field contains a number, but sometimes... It’s a miracle our stuff works as well as it does. A more patient man could probably create a good deal of programming literature trying to unpack that paradox. Instead I’m hunting for something where people are a little less concerned about job security and a little more open to mentoring, robustness, and low drama development processes.
always_good 9y ago
I don't understand the distinction you're making. Bad data in the context of database constraints is data that violates your data model which, in the absence of constraints, is implicit and tucked away around the application layer in comments and functions and word of mouth. If you're cleaning up data, that means your data integrity has been compromised. If you're collecting unstructured data from scientists, there's no contradiction. Maybe you make a nullable byte blob column for it. That's generally the most nonthreatening data in the database. Data constraints don't mean you need perfect data. But the fact that your mental image is the weirdly specific scenario someone deleting data while "their scientists" go "aw snap, I wanted that" indicates a misunderstanding on your part about what's being discussed. It sounds like you think that the opposite of data constraints is a windfall of valuable data that someone wants. No, exactly the opposite is what you're protecting yourself from with constraints.
- xapata 9y ago
  Two different tasks can have two different levels of "integrity" desired. For example, the task of mailing a thank-you note to all customers might not care if one customer accidentally has two addresses or none. For the task of billing customers, we probably want an exact one-to-one customer to mailing address. If someone protects me with the constraint of one-to-one customer to address, then maybe I lose the fact that the customer has just opened a second office. All I'm saying is that database constraints go both ways, they reveal bugs and they cause bugs.

by xapata 9y ago

stouset 9y ago

Bad data of this for is almost always the result of bugs, not of real data failing to conform to your modeling. In this case, yes, you should absolutely be dropping it on the floor (and alerting the devs with an exception) rather than creating a mess that needs cleaning up later.

xapata 9y ago
How would you know if you'd made a false negative if you drop all "bad" data? It's more of a mess when you realize you've accidentally been losing records for the last few years.
- mixmastamyk 9y ago
  The data is not dropped silently but very loudly, immediately. Unless of course you are using Mysql, well then...
  - xapata 9y ago
    So you stop the program immediately and allow no more transactions while you debug? Unlikely. Keep the data. Log that it's weird.
    - stouset 9y ago
      What is your software development background? I ask because it would not have once occurred to me in the past fifteen years of software development that “enforcing database constraints” in any way implies shutting everything down and stopping transactions until the cause of a violation is rooted out. If you’re approaching it from the position of collecting raw data from sensors or something, that might make sense. But from the perspective of a webapp, allowing duplicate (or empty) usernames or failing to keep foreign key relations valid often violate basic assumptions of soundness that the application is based upon. In these cases, the right answer is to drop the data on the floor (loudly) so a developer can look at it. Inconveniencing a customer once here is infinitely better than inconveniencing them constantly due to the site being buggy or broken as a result of the invalid data, and prevents needing to migrate the data in the future which can actually be impossible to do correctly in some cases.
      - xapata 9y ago
        > background Web dev for some years, then data science. > implies shutting everything down That's what "loudly" implied to me. Loud means crash, no? > basic assumptions of soundness I like to enforce that at the application layer, so that I can quickly swap storage without worrying. Whether I'm writing to database or a mock object, I'd hope the exceptions that arise in the application are the same. Things like unique index are easy to implement with a mock object, but more complex constraints I'd rather enforce in code that I find easier to read. > buggy or broken All I'm saying is not to be an absolutist. Some things are good to have as database constraints, others are good as application logic. I'm arguing against the idea that "validations are [never] enough." In contrast, database constraints are almost never enough to manage the complexity of data integrity.
        mixmastamyk 9y ago
        App should not crash but return/log the database error, ideally translated to layman terms on the customer end. Logs sent high priority to dev team for fixing.
        xapata 9y ago
        Indeed, I agree.
    - Slansitartop 9y ago
      > So you stop the program immediately and allow no more transactions while you debug? Unlikely. You stop the transaction on a constraint violation, not usually the entire system. > Keep the data. Log that it's weird. I've worked on systems that do that, and they're nightmares. The problem with that style can be summed up in "if it doesn't look broke, people don't bother fixing it." No one actually watches those logs, and users don't try to fix data that was "accepted" with no errors. The result is bad data piles up, and often causes problems in other parts of the system. Maybe you (as a developer) love working on support tickets to manually fix bad data the users entered or a buggy process created, but I sure as hell don't. The solution to this is to fail fast: https://en.wikipedia.org/wiki/Fail-fast https://en.wikipedia.org/wiki/Fail-fast.
      - deleted 9y ago
        [deleted]
      - xapata 9y ago
        I think our practices are closer than you realize. Maybe I didn't explain myself well. The idea that troubles me is including more strict constraints in the database than in the application itself, because that indicates the application isn't fully tested. If the application is well tested, then it doesn't matter where the logic resides, db or app, so there's no standing to the claim that application logic is never enough. Also, there are many kinds of data. Transactions, sure, reject if it doesn't match. Telemetry data, I'd like to hang on to it regardless of how weird it seems.

midev 9y ago

> Your scientists would prefer you keep all measurements, even the bad ones. Sometimes those "bad" data actually have information. A constraint can allow "bad" data if the "bad" data is still semantically valid for that column. A freeform text column where the scientist writes notes might not need any constraints. But a column used to send measurements to a robot to manufacture something, must have some protections around what type of data it can store. And no, the scientist doesn't get to input "bad" data that makes the robot lose its mind. Sometimes yes, you drop data on the floor.

xapata 9y ago
I've seen it go the other way plenty of times -- the dev thinks a system should only receive numbers, but occasionally a string is valid as well.
- always_good 9y ago
  Then the data model was wrong and they migrate their scheme appropriately. I don't think you understand that this is a core part of why you want a strict database: so you can perceive mismatches between your assumptions and reality and then respond precisely. Do you think it's superior to be in a situation where you thought all of your IDs were numbers, you built your system with that expectation, and it turns out strings have been silently making their way into your data? The difference between the two scenarios is that the person that used constraints knows when and why and where they got unexpected data and they can take more decisive, informed action. Meanwhile the other guy is desperately writing queries to find unexpected data, shooting around in the dark, and wondering what else slipped through.
  - hinkley 9y ago
    Unfortunately he sounds like about half the Node developers I work with. Nobody wants to make concrete assertions. To decide anything. Usually this count field contains a number, but sometimes... It’s a miracle our stuff works as well as it does. A more patient man could probably create a good deal of programming literature trying to unpack that paradox. Instead I’m hunting for something where people are a little less concerned about job security and a little more open to mentoring, robustness, and low drama development processes.

always_good 9y ago

I don't understand the distinction you're making. Bad data in the context of database constraints is data that violates your data model which, in the absence of constraints, is implicit and tucked away around the application layer in comments and functions and word of mouth. If you're cleaning up data, that means your data integrity has been compromised. If you're collecting unstructured data from scientists, there's no contradiction. Maybe you make a nullable byte blob column for it. That's generally the most nonthreatening data in the database. Data constraints don't mean you need perfect data. But the fact that your mental image is the weirdly specific scenario someone deleting data while "their scientists" go "aw snap, I wanted that" indicates a misunderstanding on your part about what's being discussed. It sounds like you think that the opposite of data constraints is a windfall of valuable data that someone wants. No, exactly the opposite is what you're protecting yourself from with constraints.

xapata 9y ago
Two different tasks can have two different levels of "integrity" desired. For example, the task of mailing a thank-you note to all customers might not care if one customer accidentally has two addresses or none. For the task of billing customers, we probably want an exact one-to-one customer to mailing address. If someone protects me with the constraint of one-to-one customer to address, then maybe I lose the fact that the customer has just opened a second office. All I'm saying is that database constraints go both ways, they reveal bugs and they cause bugs.