3 ms·

This completely ignores my comment in its entirety. Do people make mistakes: yes. Does software ever have defects: yes. Typically, people making mistakes in

This completely ignores my comment in its entirety. Do people make mistakes: yes. Does software ever have defects: yes. Typically, people making mistakes in software is called a software defect. It is absolutely critical in a discussion like this to understand that defects in software are not necessarily defects in data. This conversation is about data, and checks for processing data are already present in various forms. For example when was the last time you complained about weak type systems when working with JSON, XML, or SQL tables?

Quekid5 8y ago
> For example when was the last time you complained about weak type systems when working with JSON, XML, or SQL tables? Well, I don't know about the person you're replying to, but I certainly complain about these all the time. Interestingly and counter to your point, I think, XML grew a "type system" of sorts with XML Schema (which is widely used IME, esp. where interfacing with external services as in SOA), and SQL actually has a reasonably strong type system for most implementations albeit dynamically checked at query planning time. (SQLite seems to be the outlier here.) JSON is actually the outlier here, but even with JSON there's been attempts at a sort-of type system with JSON Schema. Why? Well, because beyond a certain size the "just chuck some data somewhere in there" ceases to be workable when you want stable and maintainable interfaces between software components. Ultimately type systems are invented and used for almost entirely pragmatic reasons.
- spiralx 8y ago
  > ... XML grew a "type system" of sorts with XML Schema (which is widely used IME, esp. where interfacing with external services as in SOA) ... The type system came from both XML Schema and XPath and was codified with XSLT 2, then expanded when XQuery was defined. It's not been a type system "of sorts" since XSLT 2, which filled in a lot of gaps and undefined behaviours.
arcticbull 8y ago
SQL has a schema, and the way I use it, I generate interfaces with strict types from my schema, specifically because I've run into trouble not doing so. XML, well, there's the schema language for that, but that's fairly "outdated" and I haven't touched it in ages, but if you've ever looked at an HTML parser, it's borderline sentient because of weak typing. JSON? That's a huge source of issues. I've run into them professionally and personally, and they cause outages all the time. Thats why protobufs are great (schema) and GraphQL with generated interface is great.
arcticbull 8y ago
I didn't really ignore your comment either though. >> You don't need static typing to program defensively. Static typing instead ensures a certain defensive position, but as a developer you can choose to program in such a way even in a dynamic language. Correct, but in a statically typed language defensive programming is not required on your part to the same extent (because the compiler enforces a whole set of defenses on your behalf). You'll also forget sometimes, mistype sometimes, check the wrong things sometimes, forget to update some defenses. The compiler won't. >> That being said it is foolish to make a blanket assumption that data is doomed to confusion. The state and shape of data is determined by the authoring application and reinforced by the receiving application. If data is shaped improperly then a defect is present in the system, so fix the defect. The type system enforces you checked the data (by for instance, providing a conversion function `fn convert(input: String) -> Result<StructuredData, Error>` which then forces you to handle your error there, and then you don't need to check anything else later. >> If a dose of common sense is still not enough... then program in TypeScript. That's half a solution because unless your entire ecosystem is typescript those types are at best aspirational. An improvement to be sure, but still aspirational. Which, IMO, can all be summed up by "we have automated checks so humans don't have to write them and make mistakes doing so as they go". Outsource the checks to an automated system.
crimsonalucard 8y ago
Data in itself can be typed. For JSON objects the equivalent type is usually called a Record. Within a single strictly typed program, data is transferred between methods and functions and the shape of this data would be described by a type. There is not much of a difference, theoretically speaking, when data is passed between functions inside an app, or between apps via some network api. What you describe is actually a real issue in typing. Types are lost across systems. JSON is not strictly typed, neither is XML or SQL. So if I have a typed application that spits out data to these formats I lose type checking. If you think about it, even jumping across abstractions from one typed language (say rust on the backend) to another typed language (say typescript on the frontend) you will lose type safety due to incompatibilities between languages. I don't know if there's research into or thought going into solving this issue. But a universal type language/syntax that can be translated into respective type systems across api abstractions could lend a lot to safety of entire ecosystems. There is no type safety across http services and designing a system to overcome this issue is very possible and to my knowledge not something people have thought about yet. New idea? Imagine a language/framework that compiles into web apps and microservices on some cloud provider. The language has syntax that allows me to decorate functions to specify which microservice it runs on. Data transfer between microservices can now be placed in some binary format or maybe the language has syntax that allows me to specify human readability. Either way the language abstracts away the devops and webapp layer and combines it into a single source code... this would allow type checking of the shape of data to not happen just within an app, but across services. Is there such a framework/language that does this? I'm positive it can be done. I would totally love to work on an idea like this. The microservice fad talked up modularity between services but in practice people realized that microservices are too complicated. Perhaps an entire language framework that combines the web app layer and dev ops layer and adds type checking across services would solve this issue.

Quekid5 8y ago

> For example when was the last time you complained about weak type systems when working with JSON, XML, or SQL tables? Well, I don't know about the person you're replying to, but I certainly complain about these all the time. Interestingly and counter to your point, I think, XML grew a "type system" of sorts with XML Schema (which is widely used IME, esp. where interfacing with external services as in SOA), and SQL actually has a reasonably strong type system for most implementations albeit dynamically checked at query planning time. (SQLite seems to be the outlier here.) JSON is actually the outlier here, but even with JSON there's been attempts at a sort-of type system with JSON Schema. Why? Well, because beyond a certain size the "just chuck some data somewhere in there" ceases to be workable when you want stable and maintainable interfaces between software components. Ultimately type systems are invented and used for almost entirely pragmatic reasons.

spiralx 8y ago
> ... XML grew a "type system" of sorts with XML Schema (which is widely used IME, esp. where interfacing with external services as in SOA) ... The type system came from both XML Schema and XPath and was codified with XSLT 2, then expanded when XQuery was defined. It's not been a type system "of sorts" since XSLT 2, which filled in a lot of gaps and undefined behaviours.

arcticbull 8y ago

SQL has a schema, and the way I use it, I generate interfaces with strict types from my schema, specifically because I've run into trouble not doing so. XML, well, there's the schema language for that, but that's fairly "outdated" and I haven't touched it in ages, but if you've ever looked at an HTML parser, it's borderline sentient because of weak typing. JSON? That's a huge source of issues. I've run into them professionally and personally, and they cause outages all the time. Thats why protobufs are great (schema) and GraphQL with generated interface is great.

arcticbull 8y ago

I didn't really ignore your comment either though. >> You don't need static typing to program defensively. Static typing instead ensures a certain defensive position, but as a developer you can choose to program in such a way even in a dynamic language. Correct, but in a statically typed language defensive programming is not required on your part to the same extent (because the compiler enforces a whole set of defenses on your behalf). You'll also forget sometimes, mistype sometimes, check the wrong things sometimes, forget to update some defenses. The compiler won't. >> That being said it is foolish to make a blanket assumption that data is doomed to confusion. The state and shape of data is determined by the authoring application and reinforced by the receiving application. If data is shaped improperly then a defect is present in the system, so fix the defect. The type system enforces you checked the data (by for instance, providing a conversion function `fn convert(input: String) -> Result<StructuredData, Error>` which then forces you to handle your error there, and then you don't need to check anything else later. >> If a dose of common sense is still not enough... then program in TypeScript. That's half a solution because unless your entire ecosystem is typescript those types are at best aspirational. An improvement to be sure, but still aspirational. Which, IMO, can all be summed up by "we have automated checks so humans don't have to write them and make mistakes doing so as they go". Outsource the checks to an automated system.

crimsonalucard 8y ago

Data in itself can be typed. For JSON objects the equivalent type is usually called a Record. Within a single strictly typed program, data is transferred between methods and functions and the shape of this data would be described by a type. There is not much of a difference, theoretically speaking, when data is passed between functions inside an app, or between apps via some network api. What you describe is actually a real issue in typing. Types are lost across systems. JSON is not strictly typed, neither is XML or SQL. So if I have a typed application that spits out data to these formats I lose type checking. If you think about it, even jumping across abstractions from one typed language (say rust on the backend) to another typed language (say typescript on the frontend) you will lose type safety due to incompatibilities between languages. I don't know if there's research into or thought going into solving this issue. But a universal type language/syntax that can be translated into respective type systems across api abstractions could lend a lot to safety of entire ecosystems. There is no type safety across http services and designing a system to overcome this issue is very possible and to my knowledge not something people have thought about yet. New idea? Imagine a language/framework that compiles into web apps and microservices on some cloud provider. The language has syntax that allows me to decorate functions to specify which microservice it runs on. Data transfer between microservices can now be placed in some binary format or maybe the language has syntax that allows me to specify human readability. Either way the language abstracts away the devops and webapp layer and combines it into a single source code... this would allow type checking of the shape of data to not happen just within an app, but across services. Is there such a framework/language that does this? I'm positive it can be done. I would totally love to work on an idea like this. The microservice fad talked up modularity between services but in practice people realized that microservices are too complicated. Perhaps an entire language framework that combines the web app layer and dev ops layer and adds type checking across services would solve this issue.