20 ms·

Parse, Don’t Validate

• magical_h4x 7y ago
  Nice article! It got me thinking about an issue I've noticed in dynamically typed languages (namely JavaScript) where it's very easy to end up doing lots of (potentially redundant) validation all over the place because it's much more difficult / unwieldy to pass that information along.
  • undreren 7y ago
    Yeah, the author mentions that too. For me, the funny thing about this article is that I kind of had the core insight just yesterday while hacking together a cli framework, but like the author, I couldn't quite explain what I learned.
  • mamcx 7y ago
    Yep. For example you call: config.load_from_file(.... And you can't know if it check if the file exist first or not. From pure habit you add checks... (what? looking at the code of the library? thats crazy!).
    • makecheck 7y ago
      The “exists” state is dynamic (something else in the system could be in the process of deleting the file since it was last checked). It’s more reliable if you don’t check in advance and simply try to open the file every time you need it. You handle errors while opening the file.
      • stoops 7y ago
        This is really interesting. For systems programming, files usually have clear ownership and modification context. If I call exists(), it should still exists for as long as that thread is paused. It's an anti-pattern to throw errors for expected behavior.
        • Izkata 7y ago
          The idea appears to me to have been popularized by python, where it's straight called an antipattern to do the check beforehand [0]. The language also uses exceptions for standard control flow - for example, there's one called StopIteration that the language uses to indicate to loops that the current generator or iterator has reached its end. [0] https://docs.quantifiedcode.com/python-anti-patterns/readability/asking_for_permission_instead_of_forgiveness_when_working_with_files.html https://docs.quantifiedcode.com/python-anti-patterns/readabi...
        • Sharlin 7y ago
          Which is why you don't throw errors but return a result type such as Rust's Result, Haskell's Either, or C++'s upcoming std::expected :) On the other hand, non-atomically asking for permission first in any concurrent (or re-entrant!) context is not just an antipattern but simply wrong and always a possible race condition. It is impossible to write a routine that tries to open a file and is statically guaranteed to succeed.
  • ricardobeat 7y ago
    In this case not even TypeScript can rescue you. You might naively implement: const head = (input: A[]) => input[0] and it will happily infer a return type of A. Then you'll fall flat on your face the first time you encounter an empty array: head([]) // -> undefined To make it correct you need to explicitly define a return type of (input: A[]): A | undefined, just as the Maybe. It's obviously impossible for TS to guarantee that an array will not be empty at runtime, but I wish this specific case triggered some help from the type checker.
    • tylerhou 7y ago
      > It's obviously impossible for TS to guarantee that an array will not be empty at runtime, but I wish this specific case triggered some help from the type checker. type NonEmptyArray<T> = { head: T, rest: T[] }; function makeNonEmptyArray<T>(array: T[]): NonEmptyArray<T> | null { if (array.length == 0) return null; return { head: array[0], rest: array.slice(1, array.length) }; } function head<T>(array: T[]): T | null; function head<T>(array: NonEmptyArray<T>): T; function head(array: any): any { if (Array.isArray(array)) { if (array.length === 0) return null; return array[0]; } if (typeof array === "object") { return array.head; } } http://www.typescriptlang.org/play/#code/C4TwDgpgBAcg9gOwKIFsygIICcsEMQA8AKgHxQC8UA3lABYS4AmAXFEQDRRYQDOwrRANoBdKAF8A3ACgpAMwCuCAMbAAloigpcAawjxkaTDnzESAClzGQAkQEpW+1OhDY8hUlAA+UBPIA2ftRSUFCqslAWVgB0fhAIAObAtBSUAAy2XBDA8lgIPv5+0iHc2bnUdAwsUJZugqnCnNx8rDX4UTx+qkoQZgCMnK0gMXGJtBmSUmIyCspqGvRMppFuNsL2bF75AdIzKup5C4xLgw6ITkZuputEO4p785XL+C0IIOu4r0EhYRGubao8P4gJ5vDJUYIhULhEHDBJJFJpDIlHJ5XzbCHFLIo6pWOrCIriGTfaGgSBwcKDBFQABEcAARgArCAqalgjGZUp5QZRQ4EqZiIA http://www.typescriptlang.org/play/#code/C4TwDgpgBAcg9gOwKIF... It's impossible for Haskell to guarantee an array will not be empty at runtime as well; that's why we can write a new type + a smart constructor to track runtime properties.
      • s_severus 7y ago
        A possibly more elegant TypeScript solution: type NonEmpty<T> = [T, ...T[]]; function head<T>(input: NonEmpty<T>): T { return input[0]; } https://www.typescriptlang.org/play/?ssl=5&ssc=2&pln=1&pc=1#code/C4TwDgpgBAcg9gOwKIFsygDwBUB8UC8UA2lgDRQB0VWRAurQNwCwAUKwGYCuCAxsAJaIoACwgBDACbYcACn4IwnYAC5YiVOhDSAlKqxQA3qyhQAThGCdTCKPMXAiABkasAvq1ajJMutuYsAegCoAEFTAHNOFAgEYCg4dihQSCgAcjpU2wBnKAQ4OLEsrP5whDEAIwAbaGA4KDAxUzFo4AhTeMTk6FT4ZDRMbgBrPIB3BBxUilYgkwAFUzhIU1A0x0z+HJQN4oRw2xsutIyocqUzCABHTn5zCX2k8G7ejQGEYbgxiam2Fi8JHwATLQ-FAZnBBgBCDwsHiILJxBBRcptLKqREoZGmOgEYgARhcv3E-3RmKyfmmwTCkWisQ6DxSqRJbWOG1y+SghWKpQq1SSdQaTRabTphx66n6WiZpi+0NhCHhbL6mhgSJRqmeEowUrwhCI+P8fxkeSVoBVGJRILBkKAA https://www.typescriptlang.org/play/?ssl=5&ssc=2&pln=1&pc=1#...
  • dudul 7y ago
    Everybody notices this. That's why most dynamically typed languages, after having reached good adoption, try to shoehorn type safety to fix that.
  • BurningFrog 7y ago
    To me, unit tests replaces almost all I got from type safety. And that's just a side effect even!
    • NieDzejkob 7y ago
      Tests can only prove the presence of bugs, not their absence.
      • BurningFrog 7y ago
        That's also true of type safety, and every other technique...
        • NieDzejkob 7y ago
          Not if you take into account the bug classes the technique intends to eliminate.
• matheusmoreira 7y ago
  Language-theoretic security was the first thing that came to mind when I read the title. Was pleasantly surprised to see it referenced at the end of the article. http://langsec.org http://langsec.org The idea is to formally specify the structure of inputs and reject invalid data instead of trying to process it.
  • danharaj 7y ago
    langsec is directly cited in the article :)
    • tome 7y ago
      The citation of langsec is directly mentioned in the comment :)
      • danharaj 7y ago
        That's what I get for not reading the whole comment :^)
• hcarvalhoalves 7y ago
  I would like to see an example on something less trivial than encoding `not empty` on the type. At some point the difference between what the author calls "parsing" and "validating" gets blurry, or the type system can't encode.
  • danharaj 7y ago
    You can always use abstract data types to indicate in the type system that you know something that can't directly be expressed. The point is not that "parsing" and "validating" are distinct concepts, but that they're two different points of view on the same problem.
  • lexi-lambda 7y ago
    See my other comment about this question here: https://news.ycombinator.com/item?id=21478427 https://news.ycombinator.com/item?id=21478427
• QuinnWilton 7y ago
  I really enjoyed this article! I think that leveraging the type system to enforce invariants about your data model is one of the most effective things you can do as an engineer. I gave a talk on this topic at my workplace, using Elixir and Haskell as examples. It's a little haphazard, since it wasn't written for public consumption, but someone might find it interesting: https://github.com/QuinnWilton/programming-with-types-talk https://github.com/QuinnWilton/programming-with-types-talk
  • verttii 7y ago
    Really good stuff, is that talk uploaded anywhere?
    • QuinnWilton 7y ago
      It's not unfortunately, I just gave it over lunch to my team about a year ago. Thanks though!
• giomasce 7y ago
  Using this programming style requires a rather powerful type system, if you want to go past the simple examples the author shows and encode more complex validity conditions. I am still learning these things, but AFAIU the extreme edition of all this is the Calculus of Constructions used in Coq and Lean, where you can encode theorems and algorithm specifications as very complicated type; proving a theorem corresponds to showing an element of the theorem itself (seen as a type), and if you find an element of a type encoding a certain specification, Coq can work out for you a program that implements that specification. This is what I understood, at least. Things are quite complicated!
  • QuinnWilton 7y ago
    > Using this programming style requires a rather powerful type system, if you want to go past the simple examples the author shows and encode more complex validity conditions. You need a strong type system if you want to encode complex constraints at compile time, but you can still encode a surprising number of conditions in weaker type systems. And if you can't enforce a constraint you care about at compile time, capturing that constraint in an ADT using sum types and runtime assertions can still provide a lot of value to anyone reading or maintaining your code. This sort of style requires a lot more diligence and verbosity than the same code in Haskell would, but I think dynamically typed programs can make a lot more use of their type system than they usually do. At the very least, humans can try to pick up the slack where a compiler is insufficient, and while designing your data model in Ruby as if it were in Haskell may not help the interpreter at all, it can give the programmer a lot more context about how the code is intended to work, and what assumptions it is making about its data.
  • stoops 7y ago
    A pragmatic interpretation of this is to: * subclass or compose String whenever possible. (Id, Name, Address, AddressLine1) * subclass or compose Number/Matrix types whenever possible (e.g. Users, Packages, Widgets) * Use immutable collections (e.g. Google Guava library in Java) I have built very powerful software with small teams using these principles. At scale, your day to day programming becomes easy as the compiler is doing all the error checking. It is also very slow to program this way. You write A LOT of boilerplate, and there's only so many code-generation libraries you can throw at it before it becomes a pain to setup your IDE. But it is worthwhile for complex applications. We did not have bugs.
• Izkata 7y ago
  This is describing what I've known as "Typed Hungarian notation", and have seen a few times before, though I can't seem to find now. The original intent of Hungarian notation was to encode metadata into the variable name - for example, "validData = validate(rawData); showValidData(validData)". The idea was that the prefixes would make it just look wrong to the programmer if someone accidentally used mismatched names, such as "showValidData(rawData)", indicating a likely bug. This notation mutated into the far more simplistic and arguably less useful "prefix with the datatype", such as iFoo indicating Foo is an integer. This mutation became known as Systems Hungarian, while the original became Apps Hungarian. The suggestion in this post is taking Apps Hungarian, and instead of relying on variable names, encoding it into the type system itself. The first time I recall seeing something like this suggested was actually in Java, with examples in that blog post involving class attributes: public String Name; public String Address; And creating classes to represent these values, preventing them from being used the wrong way: public NameType Name; public AddressType Address; ...which is why I remember this as "Typed Hungarian" - making the language's type system handle the work that a person would normally have to think about in Apps Hungarian.
  • mason55 7y ago
    Another good example of this is having separate classes for something like unsafe strings vs. safe strings in a web app. The functions which interact with the outside world accept unsafe strings and emit safe strings to the rest of the application. Then the rest of the application only works with safe strings. Anything that accepts a safe string can make an assumption that it doesn't need to do any validation (or "parsing" in the context of the OP), which lets you centralize validation logic. And since you can't turn an unsafe string into a safe string without sending it through the validator, it prevents unsafe strings from leaking into the rest of the app by accident. This concept can be used for pretty much anything where you are doing data validation or transformation.
    • user5994461 7y ago
      What are safe and unsafe strings supposed to mean? All strings seem like normal string to me, a "DELETE * FROM db" is no different from any other string until it's given to a SQL query.
• schoen 7y ago
  I learned a lot from this article and hope to revisit it. I had one quarrel: > Consider: what is a parser? Really, a parser is just a function that consumes less-structured input and produces more-structured output. By its very nature, a parser is a partial function—some values in the domain do not correspond to any value in the range—so all parsers must have some notion of failure. I think it's reasonable to include some total functions as parsers as well, because some grammars -- structures that are recognized by the parser and that specify languages -- generate all strings as the corresponding language. In that case, there are no failure cases because every string is a valid input. I came up with some more-trivial examples, but I think a less-trivial example is the split() function in many languages. Its Haskell type signature is String -> [String], and it is a total function. It handles the case where the input is a nonempty string with no delimiters (the output is a list containing a single token equal to the input), as well as the case where input is an empty string (the output is an empty list), and the case where the input consists only of delimiters (in many implementations, the output is a list containing n+1 empty strings). Another trivial-ish case is where the grammar allows an arbitrary trailing string following the object of interest (similar to C's atoi). In that case, a parser will always return the empty object if there is no interesting or relevant data in the input. (For example, atoi returns 0 if the input string doesn't begin with an integer.) This could be viewed as a kind of error, but there may be applications in which this behavior is useful.
  • joppy 7y ago
    This seems needlessly pedantic - we could also notice that a total function is just a special case of a partial function, and so total functions are already included by the article.
  • lexi-lambda 7y ago
    I think this is a reasonable point, and your split() example is a good one. I wasn’t sure while I was writing the blog post if I considered isomorphisms to be parsers, and I’m still not entirely sure if I do or not, but there’s an argument to be made that they are simply parsers where the set of possible failures is empty. I don’t have strong feelings about that distinction one way or the other, though.
• privethedge 7y ago
  > head :: [a] -> a > This function returns the first element from a list. Is it possible to implement? But the type itself doesn't say that it must return the first element. > If you see why parseNonEmpty is preferable, you understand what I mean by the mantra “parse, don’t validate.” Okay, what about more complex cases? E.g. how do you describe type "such value doesn't exist in the db table yet"?
  • mason55 7y ago
    > Okay, what about more complex cases? E.g. how do you describe type "such value doesn't exist in the db table yet"? If that's supposed to be an error state then you just throw a different error. If it's not supposed to be an error state, then you can have something like a function that accepts a type of "Input" and returns a type of "DBTrackedInput" and in that function you do your saving to the DB. Then things which only want to work with values that are in the DB accept types of DBTrackedInput. That prevents you from passing something which hasn't been saved in the DB to those functions. And you can expand from there, maybe the function that saves to the DB only accepts input of type "ValidatedInput". So you first have a function that accepts an "Input", validates it, and emits a "ValidatedInput". Then you have other functions, like your DB saver, which can say "I only want validated data" and still other functions which can say "I only want data that's been saved to the DB".
    • privethedge 7y ago
      What if there is another application (or e.g. a stored procedure) that uses this DB table so you cannot control all saving operations from your application?
      • lmm 7y ago
        Don't do that. A DB table is too big and complex a surface to use as an interface between multiple applications. Each table should have one clear owner; if multiple components need to access the same table, put it behind a service rather than having them access it directly.
  • 7y ago
• bretthopper 7y ago
  I just watched a good Elm conference talk about this same concept called "Making Impossible States Impossible" - https://www.youtube.com/watch?v=IcgmSRJHu_8 https://www.youtube.com/watch?v=IcgmSRJHu_8
• beders 7y ago
  It's pretty cute. Still no static types for "Strings that end with a dot" or "Strings that match [a-z]+" or "Strings that are proper nouns". ;) I'm not sure I understand the benefits of this compared to a traditional parser generator. "Use a data structure that makes illegal states unrepresentable." is not effort spent well. (And given Gödels incompleteness theorems unachievable in the general case.) A proper parser generator using a grammar will give you the structural guarantees you need, a lexer will reject invalid terms. And a transformation step gives you the target structure. Eventually you will have to validate though at runtime.
  • lukebitts 7y ago
    > It's pretty cute. Still no static types for "Strings that end with a dot" or "Strings that match [a-z]+" or "Strings that are proper nouns". ;) That’s not true, you can wrap these in types that can only be constructed from valid data, and from that point on you can be sure everything is correct.
  • lexi-lambda 7y ago
    > Still no static types for "Strings that end with a dot" or "Strings that match [a-z]+" Sure there are. :) Technically speaking, anyway. Here’s a type for “strings that end with a dot”: -- | A string that ends in the character '.', -- represented as a string without its trailing dot. newtype StringEndingInDot = StringEndingInDot String fromString :: String -> Maybe StringEndingInDot fromString str = case reverse str of '.':cs -> Just $ StringEndingInDot (reverse cs) _ -> Nothing toString :: StringEndingInDot -> String toString (StringEndingInDot str) = str ++ "." And here’s a type for “strings that match [a-z]+”: data Letter = A | B | C | D | ... | X | Y | Z type StringAZ = NonEmpty Letter Now, admittedly, I would never use either of these types in real code, which is probably your actual point. :) But that’s a situation where there is a pragmatic “escape hatch” of sorts, since you can create an abstract datatype to represent these sorts of things in the type system without having to genuinely prove them: module StringEndingInDot(StringEndingInDot, fromString, toString) where newtype StringEndingInDot = StringEndingInDot { toString :: String } fromString :: String -> Maybe StringEndingInDot fromString str = case reverse str of '.':_ -> Just $ StringEndingInDot str _ -> Nothing You may rightly complain that’s a lot of boilerplate just to represent this one property, and it doesn’t compose. That’s where the “Ghosts of Departed Proofs” paper (https://kataskeue.com/gdp.pdf https://kataskeue.com/gdp.pdf) cited in the conclusion can come in. It provides a technique like the above that’s a little more advanced, but brings back composition.
• iddan 7y ago
  Dear awesome Haskell writers. You are writing great articles but I can’t understand the code examples as Haskell is too far from the programming languages. Can you provide examples in TypeScript / Rust / Swift / Reason or another C like language? If not, I’m curious why? Love, Iddan
  • voxl 7y ago
    Dead iddan, Haskell is in fact a programming language, despite what the phrasing of your comment might imply. Translation from language to language is often difficult, we don't do typically do it from English to say Spanish not because it wouldn't be valued but because we may not have the same expertise in Spanish and it would be expensive to hire a person to do it. Expertise is important, because key ideas may get lost in translation.
  • lexi-lambda 7y ago
    I have provided a translation of the NonEmpty datatype into Java in this comment: https://news.ycombinator.com/item?id=21478322 https://news.ycombinator.com/item?id=21478322 However, to answer your “why” question: mostly just because I write Haskell for a living, and my primary personal motivation for writing this blog post was to share with my coworkers and to point to during code reviews. Therefore, it makes the most sense for the examples to be in Haskell! Haskell is also especially well-suited for communicating this sort of thing if you already understand the notation, as you can sort of see from the Java translation: it’s significantly more verbose just to get the same idea across. Still, it’d be great if someone were to provide a “translation” of the same ideas to other languages, no doubt!
  • NieDzejkob 7y ago
    I believe I understood the article completely, even though I haven't written a line of Haskell in my life.
  • papln 7y ago
    One of Haskell's virtues is that it is concise enough to make pleasant reading in a blog post. Come learn Haskell! It's fun and edificational.
  • 7y ago
• z3t4 7y ago
  What is the equivalent in TypeScript? const head = (arr: Array<number>):number => arr[0]; happily returns undefined if you pass an empty array (with strict null checks)
  • scrollaway 7y ago
    This SO answer is pretty clever: https://stackoverflow.com/a/56006703 https://stackoverflow.com/a/56006703 type NonEmptyArray<T> = [T, ...T[]]; That is: A type of "NonEmptyArray" of T is an array of one element of type T, followed by any number of type-T elements.
  • lexi-lambda 7y ago
    TypeScript is sadly very unsound by design, so doing this kind of thing in TypeScript is always going to be more “best effort” and less “exhaustive proof.” That’s not necessarily wrong or bad per se, as after all, Haskell has escape hatches, too (and so do dependently-typed languages, for that matter). However, there are philosophical differences between the way the languages’ type systems are designed. When I’ve talked to people who develop and use TypeScript, the general impression I’ve gotten from them is that TS is actually as much about tooling as it is about correctness. TS enables things like autocomplete, jump to definition, and quick access to documentation, and it does that by leveraging the typing information to “see through” the dynamic dispatch that makes that sort of thing infeasible in dynamically-typed JavaScript. The TS type system does provide some correctness benefits, don’t get me wrong, but where ease of use and correctness are in conflict, usually ease of use is preferred. This is true even with all the “strictness” options enabled, like strict null checking. For some examples of TypeScript unsoundness, see [1] and [2]. Flow is actually generally a lot better than TS in this regard (though it does still have some major soundness holes), but it looks like TS has basically “won,” for better or for worse. But again: TS won because its tooling was better, not because its ability to actually typecheck JS programs was better, which I think reinforces what I said in the previous paragraph on what TS is really about. [1] https://twitter.com/lexi_lambda/status/1068704405124452352 https://twitter.com/lexi_lambda/status/1068704405124452352 [2] https://twitter.com/lexi_lambda/status/1068705142109810688 https://twitter.com/lexi_lambda/status/1068705142109810688
  • 7y ago
• darkkindness 7y ago
  Yes, yes yes! Encode invariants in your data, don't enforce invariants on your data. I particularly think this point needs to be stressed more, because practicably speaking, not every language has the typechecking faculties of Haskell: > Sometimes, making an illegal state truly unrepresentable is just plain impractical given the tools Haskell provides, such as ensuring an integer is in a particular range. In that case, use an abstract newtype with a smart constructor to “fake” a parser from a validator. For instance, few type systems will let you encode prime numbers as datatypes, so you'd have to do something in your language like newtype Prime = Integer mkPrime :: Integer -> Maybe Prime mkPrime | isPrime p = Just (Prime p) | otherwise = Nothing which is parsing!
  • mc3 7y ago
    This is why learning Elm has been valuable for me. Without the convoluted type system of Haskell, you have to do this more, and it is a lot simpler to understand and maintain. You are never going to be able to use the type system to guarantee everything about your data, so let your runtime code check it, add lots of tests and keep it isolated (Elm calls them "Opaque Types") so it is easy to reason about. The equivalent is possible in OO with classes, but with the caveat that only if you make everything immutable and ideally have a layer of compile time null checking on top. In short, with Haskell I am learning and relearning language features constantly and banging my head. With Elm I grok the language and I am focused on solving the problem.
    • lexi-lambda 7y ago
      I think it’s fantastic that Elm works well for you; it’s a wonderful language and there’s no doubt that it is much simpler than Haskell. I agree with you that, in many cases, the technique you’re describing is sufficient, and it requires much less sophisticated machinery than some of the fancier Haskell techniques. That said, I do think what you’re describing is different from what is described in the blog post. Opaque/abstract types reduce the trusted code significantly, which is fantastic, but the example provided in the blog post is actually even better than that: because NonEmpty is correct by construction, the amount of trusted code goes down to zero. A lot of the tools Haskell provides, like GADTs and type families, can be used to capture more invariants in a correct-by-construction way, rather than relying on abstraction boundaries. There are advantages and disadvantages to both. I certainly don’t have any ill will toward anyone who finds the complexity of Haskell not worth the extra benefits, so to be quite clear, I’m not trying to sell you on Haskell instead of Elm! I do think being aware of the distinction is still useful, though.
• jes5199 7y ago
  Woah cool. Back when I was using Haskell (uh, ten years ago, now), people kept telling me that they were _pretty sure_ there was a flag that detected incomplete pattern matches at compile time, but no one could ever tell me what it was. -Wincomplete-patterns . Now I know. Thanks!
  • papln 7y ago
    https://downloads.haskell.org/~ghc/latest/docs/html/users_guide/flags.html https://downloads.haskell.org/~ghc/latest/docs/html/users_gu...
• noobiemcfoob 7y ago
  I like the concept of encoding properties, like a boolean of non-empty, in an object. Beyond that, this article further convinced me type systems are for the pedantic. A given function signature is impossible? Seems like just another strength of a dynamic language.
  • aidenn0 7y ago
    What does your dynamic language do when you take the first element of an empty list? There is no obvious "correct" thing to do. Furthermore, whatever you do return is unlikely to be the same sort of thing that is returned for a non-empty list. A dynamic language will have a behavior that corresponds to some sort of type signature, and it's not possible to write behavior that corresponds with the type signatures given as examples of "impossible" in the article. A type signature is merely a statement about behavior, so it's nonsensical to make a false statement about the behavior, and Haskell catches this.
    • noobiemcfoob 7y ago
      It throws an error. Errors are a type of behavior. Some of those error behaviors you can cope with. You catch those. Others you can't. You raise those and either the system can cope or it can't and you crash. What part is nonsensical?
      • aidenn0 7y ago
        Throwing an error should be part of the function signature. Otherwise the statement "This function takes a list of objects of type A, and returns an object of type A" is false; sometimes it will return an object of type A, other times it will signal an error.
        • scarface74 7y ago
          Nice in theory but if you want to see how bad it is in practice, look no further than Java.
• phreack 7y ago
  I love this kind of work having the compiler prevent you from creating business logic bugs. If you're into this and Kotlin, take a look at the contracts experiment - https://blog.kotlin-academy.com/understanding-kotlin-contracts-f255ded41ef2 https://blog.kotlin-academy.com/understanding-kotlin-contrac...
• qtplatypus 7y ago
  I agree broadly with this however in many cases that I have been dealing with I don't think that there can be a type system that would be rich enough to encode the validation laws. For example how would you encode "This purchase request must be made before the closing date for the purchase"?
  • undreren 7y ago
    This sounds like the Nirvana fallacy. Just because parsing over validation is not perfect, it can still lead to significantly more maintainable code and safer and stabler system. Pure validation has a lot of problems, namely that you have to validate the data on all levels for the invariants required at that level.
  • lkitching 7y ago
    This isn't about encoding the validation rules in the type system, only the requirement that validation with respect to some properties has been done. So in your example you would just define a function createPurchaseRequest :: UserRequest -> Purchase -> Maybe PurchaseRequest now everywhere in the code taking a PurchaseRequest can assume it is valid.
• jillesvangurp 7y ago
  Kotlin's smart cast is nice in this context. One example is with nullable types. if you have val foo: Whatever? // might be null and Whatever and Whatever? are two separate types. foo.someMethod()// compilation error because foo is not of type Whatever if(foo != null) { foo.someMethod() // works because foo was smart cast to Whatever after the null check } In the same way doing a switch on a type smart casts the variable to that type. Smart casting of nullable types also gets rid of a lot of clumsy things like Optional, Maybe, etc. you'd need in other languages where you'd have to call a method to extract the value, assign it to a new variable, etc.
• larusso 7y ago
  I understand the concept of wrapping values in specific types which gives you certain guarantees at compile time. And I really like this concept and will play around with this some more because the empty something issue is something I myself struggle with. But what really urgs me is the usage of throw in the exceptional case. My goto type in these situations would be Either rather than a throw. But this would create nearly the same issues on the caller site as an Maybe would create. Now one could argue that this is an exceptional case the user or Programm can’t possibly handle. So how do you handle this then? My main usage is in Rust and here the panic! handle seems to be used as often as unsafe raw pointers :)
• cryptica 7y ago
  It doesn't make sense to use JSON as the interchange format for a statically typed language. There is an impedence mismatch between the two. You are forced to infer types which are not actually there. The reason why JSON is so popular is the same reason why dynamic typed languages became popular. The interfaces between components are simple and human-readable. A library written in a dynamically typed language can be effectively integrated into any system without the possibility of type mismatches. If you have an API written with Protocol Buffers, every system which interacts with yours needs to agree with your type system; this creates tighter coupling.
  • johnday 7y ago
    > It doesn't make sense to use JSON as the interchange format for a statically typed language. Two problems here. One, there are multiple statically typed languages with different type systems and it is nontrivial to translate between them. It's not "having a language" versus "not having a language"; it's Type system A vs Type system B vs Type system C vs none. Secondly, this is only true if you assume that no dynamically typed language will ever want to interoperate with your code - which is almost always incorrect to presume.
  • whateveracct 7y ago
    Doesn't this hold true for all serialization ever? Replace JSON with "bytes" (aka any serialized data) and it still holds: It doesn't make sense to use bytes as the interchange format for a statically typed language. There is an impedence mismatch between the two. You are forced to infer types which are not actually there.
• z3t4 7y ago
  Not only was this the first article about Haskel that I could actually understand. But it was also the first article where type annotations makes sense. That it actually helps you think about edge cases, instead of just annoy you. My personal experience with type annotations (not to be confused with type systems) makes the code harder to read, for very little benefit. I would like a type checker that, instead of you having to babysit the type checker, the type checker should babysit you. Instead of the type checker crying over trivial issues, and needs your comfort. The type checker should comfort me when I'm sad. While manually writing human friendly error message helps, you still have to discover edge cases yourself. It would be nice with a tool, not necessary baked into the types/system, that automatically detects edge cases. For example detect when a variable has the possibility of becoming undefined/null. Maybe achieved via fussing. A college once asked me "How do I get rid of the damn nulls?" Null's are basically a lazy error/exception. Error handling is so laborious we often "forget" to do it. And this is very confusing for beginners. There are basically two camps, one that think errors are exceptions, and the other that thinks errors should be first class citizens. But there are different domains of course. I don't want my CPU to maybe return a error message when I ask it to multiply two numbers, I want that to always work, I don't care if the result get shifted (that would be a job for the type checker to detect, or better; automatically use mult64 instead of mult32), because handling errors at that level and detail would be too annoying and bad for performance. Writing network services however, it is crucial that I get noticed if a request failed.
  • tejon 7y ago
    For what it's worth: it's _very_ rare for a type annotation to be required in Haskell. It's just considered best practice (supported by compiler warning options) to have them on all top-level declarations, for two reasons: - it's a good extra layer of _human-readable_ documentation about basic behavior and requirements (as in this article); and - it lets the compiler give better error messages. The compiler is _always_ going to do its own inference, whether you give it a signature or not. If it infers a valid type _which isn't the one you thought you wrote_, and you didn't specify what the type _should_ be, that function/statement will compile -- you won't get an error until later, when you attempt to consume it and the types don't match. This can be harder to trace, especially if there's a bunch of polymorphic code in the middle where any type will pass. Supplying a manual annotation lets the compiler immediately say "hey, these don't match."
• mehrdadn 7y ago
  Am I right to suspect the only reason "there can’t be any performance overhead" is that this is being done in a lazy language like Haskell? Meaning the statement won't hold in >99% of practical cases? Or did I misunderstand something?
  • spuz 7y ago
    No, this doesn't have anything to do with Haskell's lazy evaluation (at least not in the NonEmpty list example presented). The general idea is that if you are going to perform validation checks at some point in your dynamic language, you won't lose any performance performing those checks up-front in your static language.
    • mehrdadn 7y ago
      But how can this possibly be true in general? As an example, just imagine I want to check that a string represents a list of space-delimited integers, where each one is also less than 10 digits. It's far more trivial to verify that than to actually parse the integers. And by performing the verification pass separately before the parsing pass, I can reject invalid inputs early, leading to much faster rejections than if I parse them as I validate them. The only way I can see there being zero cost difference is if everything is implicitly lazy, such that at runtime the verification won't even happen until the parsing needs to be performed too. Right?
      • papln 7y ago
        It's true that in the failure case, you can get faster reults by short-circuiting. If failure cases are a substantial portion of your runtime, then yes, doing a fast pre-pass can be more efficient. I'd speculate that in the real world, 99% of cases have time dominated by success-cases. Exceptions would be things like DOS attacks.
        • mehrdadn 7y ago
          No that's just wrong. It's not just failures. Imagine if I verified they were all zero then I wouldn't have to do a full parse. Or if I verify they have only a few digits then I would avoid bignum parsing. I think this proves what I'm saying - this simply isn't true in general.
• bambax 7y ago
  I have done, and still do, a lot of "shotgun validation" ;-( But after having read the article it's still not clear how to log illegal values or entities if they can't even be represented in the data model? How do you talk about things you can't name?
  • thedufer 7y ago
    You can log them by serializing the pre-parsed data model, unless I've misunderstood the question.
  • matt_kantor 7y ago
    You could return an error type wrapping the original value (and any other useful context) using Either instead of Maybe.
• anasbarg 7y ago
  When we started writing the code for Heavenly-x (https://heavenlyx.com https://heavenlyx.com), the first thing we needed to write before anything else is the definition of a data structure that represents the requirements as close as possible (a concrete syntax tree). We’re building a tool with a custom DSL for building CRUD GraphQL APIs, with auth and data validation and transformation. So our architecture consists of three phases: - Parser - Setup - Runtime There’s no way the parser would succeed if the input is not valid. We’re writing our software in Scala and we are using parboiled2 for parsing the DSL into a concrete syntax tree, so if it succeeds then it’s valid and if it fails, it fails early and we don’t have to worry about validation in later phases. We wrote some custom validation logic that traverses the concrete syntax tree after the parser to validate for some requirements that we couldn’t encode in the concrete syntax tree, but it’s really a small portion of our codebase and it’s easy to maintain. At the Setup and the Runtime phase we assume that the concrete syntax tree is valid. At the Runtime phase we have a running GraphQL server so we have a parsing phase too but it’s handled by Sangria so we don’t have to worry about it. We are also building a UI for those who don’t like using our language. It’s a React app where the state data structure looks exactly like our concrete syntax tree. We’re launching soon. You can subscribe for early access here: https://heavenlyx.com/get-started https://heavenlyx.com/get-started
• jdonaldson 7y ago
  Haxe has a nice mechanism to handle this called "abstract types". https://haxe.org/manual/types-abstract.html https://haxe.org/manual/types-abstract.html The critical validation step happens when the abstract type is created, not when it is used or passed to other functions...similar to the example in TFA. The added benefit is that you get a type that behaves pretty closely to a class, but adds no typing or runtime memory overhead. E.g. here's an example for an "unsafe" string : https://gist.github.com/jdonaldson/9a09287e540e7392857f https://gist.github.com/jdonaldson/9a09287e540e7392857f Another benefit is that you abstract types can define relationships between multiple types in this way, making it possible to perform validation for functions that need to handle "Either<T>"-style types.