23 ms·

Parsix: Parse Don't Validate

153 points by Iazel 5y ago

didibus 5y ago
I have to be honest, I'm not seeing what problem this is trying to solve. Anyone can enlighten me? Edit: Ok I think I understand... It seems the problem would be that if you're implementing a function that takes a user email as a string, and that function is in a lower layer of the application, like inside the data access layer. It is difficult at this point to know if the email string you will be passed as input has already been validated or not. Thus you might be tempted to re-implement validation for it at your level inside this function as well and have an assertValidEmail check. This can lead to a littering of validation throughout the code base, as each implemented function worries that the input isn't validated and re-validates it, possibly using slightly different rules each time. Furthermore, if you decide to not validate it again, you might be left wondering, but am I sure it'll have been validated prior? How can I be sure? Someone in the future could easily start calling my function and forget to validate the email before calling it? This could eventually lead to a security issue or just a bug, by introducing a code path that doesn't ever validate the email string. Thus if instead you'd re-write your function so it takes an email as a ValidEmail type (or object), and not as a string, you force the caller to for sure remember to validate the email first. And you also can safely assume if you're getting an email as a ValidEmail type that it has been validated. It could also technically allow you to localize the validation logic to the ValidEmail type constructor, avoiding possible duplicate attempts at validating email with different rules. And it seems the latter "style" the author calls "Parsing" while the former they call "Validating", in the sense that since the function validating returns a modified structure it "parsed" it, because a string became a ValidEmail, thus parsing a string into a ValidEmail, as opposed to simply validating that the string is valid as an email. And finally, this is a little library to help make use of this pattern in Kotlin.
- jinwoo68 5y ago
  As they said in README, it’s inspired by Alex King’s Parse, don’t validate [1]. Basically, rather than write a validation function, write a parser that returns a result of a specific type and use that type everywhere else. Then you can make sure the raw inputs are always validated. [1] https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/ https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-va...
  - anaerobicover 5y ago
    Small correcting, the author is named Alexis
    - jinwoo68 5y ago
      Whoops. Thanks for correcting.
- steventhedev 5y ago
  There's an entire class of vulnerabilities caused by having separate verification and parsing logic, typically with fields that usually only one is used, but the format supports multiple. The verifier checks the first one but the parser uses the last one.
- Smaug123 5y ago
  If you mean "why Parse, Don't Validate", you should read the original blog post, linked at the top of the article. It's… transformative, if you aren't already aware of the principle. https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/ https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-va... If you mean "why this library", well, I guess parser combinators are nice! Some may say that a declarative statement of the parsing restrictions is better than a procedural implementation, on general principles.
- finnh 5y ago
  It encourages people to use strongly typed classes rather than primitives, even if the type simply wraps a primitive. As a result you can't pass a invalid (say) accountID deep into your code, bc validity is guaranteed to be checked early when you "parse" an input string into the "AccountId" type. So: internal interfaces defined using non-primitive types, so internal methods don't need to keep validating their input. Conversion to said types happens early and predictably, catching bad values before they (eg) hit the database.
- coderintherye 5y ago
  The linked blog post explains it pretty well. Essentially, it seems to be solving for unexpected cases or incorrect validation by using of static typing and passing the expected type back in the return rather than a boolean. I'm not sure I've encountered enough issues with validation functions to use this pattern, but it does seem like a more robust way of writing them.
- jpeloquin 5y ago
  Paraphrased from the repo's readme: Suppose you have a program that consumes user input. Users often give bad input so the program needs to validate user input before acting on it. One way to validate is to call a function (e.g., `check_input`) on the user input and if it doesn't raise an error the input is safe for consumption by the rest of the program. The repo author considers this approach to be risky because the programmer can inadvertently omit or bypass `check_input` and the program still compiles and runs without complaint. The repo presents an alternative validation approach, which is to parse the user input into a data type (or, not quite equivalently, into a class). The parsing process serves as validation. Consumer functions are written such that they only accept the parsed data type. Therefore it is now impossible for the programmer to inadvertently omit or bypass validation of user input. The library is a set of convenience functions for actually writing these parsing / validation functions.
  - atoav 5y ago
    So in short: instead of representing user input (e.g. a Email address) as a string – which you can forget to validate – the idea here is to create a own data type for it, and use the validation step to create said data type. The rest of your program then works with this data type instead of the string and this way you will get a type error whenever you accidentally use unvalidated data. A nice idea that goes into a similar direction is to expand on this and create more types for different levels of trust. E.g. you could have the data types ValidatedEmail, VerifiedEmail and TrustedEmail and define precisely how one becomes the other. This way your typesystem will already tell you what is valid and what is not and you can't accidental mix them up.
    - TeMPOraL 5y ago
      You can also further generalize this idea by noticing you can encode all kinds of life cycle information in your type system. As you transform some data in a sequence of steps, you can use types to document and enforce the steps are always executed in order. In this example, the user input validation step is f(String) -> ValidatedEmail, then the process of verifying it is f(ValidatedEmail) -> VerifiedEmail. But the same principle can apply to e.g. append() operation being f(List[T], T) -> NonEmptyList[T], and you can write code accepting NonEmptyList to save yourself an emptiness check. Or, take a multi-step algorithm that gets a list of users, filters them by some criterion, sorts the list, and sends these users e-mails. Type-wise, it's a flow of Users -> EligibleUsers -> SortedEligibleUsers -> ContactedEligibleUsers. And then, why should types be singular anyway? You should be able to tag data with properties, and then filter on or transform a subtag of them. This is the area of theory I'm not familiar with yet, but I imagine you should be able to do things like: List[User] -> List[User, NonEmpty] -> List[User[Eligible], NonEmpty] -> List[User[Eligible], NonEmpty, Sorted[Asc]] -> List[User[Contacted], Sorted[Asc]]. Or, Email -> Email[Validated] -> Email[Validated, Verified] -> Email[Validated, Verified, Trusted]. I'm sure there's a programming language that does that, and then there's probably lots of reasons that this doesn't work in practice. I'd love to know about them, as I haven't encountered anything like it in practice, except bits and pieces of compiler code that can sometimes propagate such information "in the background", for optimization and correctness checking.
- cle 5y ago
  There are lots of siblings explaining why “parse don’t validate”. But also, it’s not always wise to take this to an extreme. I’ve seen over the years many scenarios where dev teams were over-enthusiastic about this and parsed themselves into a corner by making system components over-strict and enforcing invariants that weren’t necessary to enforce, making them much harder to change later. The right answer is, of course, somewhere in the middle, and depends on your domain and situation.
  - Iazel 5y ago
    hi, @cle! Curious to hear more about that, were they actually running validation/assertions in constructors?
    - cle 5y ago
      That can be a case of that yeah. Using your example, a lot of devs might use that email parsing logic in various independent components of the same system. Eg if you have a reporting component that sends you business reports, that component really shouldn’t be validating the structure of email addresses…if you need to refine the parsing logic now you’ve got to do coordinated deployments, possibly backfills, etc., whereas if you just treated it as an opaque string in that system you’d be better off. This isn’t really a criticism of the approach, it’s super useful, just that it needs to be applied judiciously. “Parse all the things” isn’t always the best advice.
      - Kranar 5y ago
        Your response isn't clear. >that component really shouldn’t be validating the structure of email addresses That's right, as the article says you no longer validate data, instead you parse data. Components don't take strings and validate them, they take an EMail address. >If you need to refine the parsing logic now you’ve got to do coordinated deployments If you need to refine the parsing logic, you modify the constructor of the EMail class to do whatever refinements you need. I have no doubt you may have seen codebases that did something wrong, and that wrong thing may have been related to parsing or validation, but nothing you've said indicates that there is something wrong with replacing validation with parsing, so that code always operates on semantically meaningful data types that are valid by construction, instead of opaque binary blobs that need to constantly be properly interpreted in ad-hoc ways.
- mirekrusin 5y ago
  Imagine you're writing typescript project. You type everything and have type safety. This type safety is an illusion on I/O boundaries – whenever ie. JSON.parse(...) from file/websocket/http happens. To preserve type safety, you want to use something like [0] to do runtime type assertions. Once i/o boundaries are parsing unknown types at runtime into what is defined as static types, your type safety is guaranteed. [0] https://github.com/appliedblockchain/assert-combinators https://github.com/appliedblockchain/assert-combinators
- TheAceOfHearts 5y ago
  Refining types so they encode all desired constraints before use. This is explained in the linked article: Parse, don’t validate [0]. It helps reduce the risk of using invalid inputs by representing constraints over the value as part of the type. For example: a common problem in web development security is that query parameters aren't properly validated which can lead to denial of service attacks. As a trivial example of this, consider a web server which paginates some data using "offset" and "limit" by passing those parameters directly to a database query; an attacker could set "limit" to some incredibly high value and cause the server to crash. If you're just doing validation on your inputs it's possible that some usage could end up being overlooked. [0] https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/ https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-va...
  - gregors 5y ago
    So real question - in the "offset" "limit" example what makes it any more safe if at first the programmer sets those types to be integers? The same problem persists does it not? Does the explicit creation of a type add this introspection? I'm not convinced that it does. Now once you fix this bug, encoding it in a type prevents it from creeping into other parts of the the code. This seems more like DRY principles in action.
    - didibus 5y ago
      Yeah, it seems to be more about guarantees as a code base grows larger and more people touch it. If there's a Limit class whose constructor and setter all check that the range is between say 5 to 100, and all existing code that needs the limit uses the instance of Limit, it just becomes less likely a code change is made that uses the limit input as it was directly provided by the user (and thus possibly out of range). But you'd still need to have had someone be smart enough to make sure the Limit class does prevent limits that could cause DB crashes. In practice I'm thinking, ok, so someone must have thought... Hey we should validate this user input and put in some logic for it. So I think what this says is, validation works by having all external input validated as they are received. But it can be easy to make a code change at the boundary where you forget to add proper validation. If all existing functions in the lower layers, like in the data access layer, are designed to take a Limit object, the person who took a limit as external input and was about to pass it to the query function will get a compile error and realize... Oh I need to first parse my integer limit into a Limit, and thus reminds them to use the thing that enforces the valid range. If instead the code had a util function called assertValidLimit, and the query function took a limit as an integer, it be easy for that person to forget to add a call to assertValidLimit when getting the limit from the user and then pass that unvalidated to the query and possibly cause a vulnerability. And lastly, it seems they argue, if you were to validate instead in the query function itself, thus it wouldn't matter if others forget to validate since where it matters would, but then it is hard to fail at that layer, since you might have already made other changes and that can leave your state corrupted. So basically it seems the argument is: "It is best to validate external input at the boundary as soon as it is received, but it can be easy to forget to do so and that's dangerous. So to help you not forget, have all implementing functions take a different type then the type of the external input, which will remind people... Oh right I need to parse this thing first and in doing so assert it's valid as well.
- Iazel 5y ago
  Cool to see you perfectly got the point in the end! I wonder though, were you confused by the README? What made it clear for you?
  - didibus 5y ago
    Hum, it was the people here who replied to my question, and also reading the linked article. I think my confusion was in trying to frame things as parsing VS validating. While I now appreciate that use of word, now that I understand, it also caused my biggest source of confusion. That's because I think most people think of parsing as conversion, like I turn a String to an Int. Where as in your case, you're simply wanting to tag a type as having been validated, but you don't really convert the type itself, so you simply wrap it in another type in order to tag it as having been validated simply because the language offers no other way to tag the type with meta-information for the compiler to assert statically. So because it seemed more like you're just wrapping the input, but still all code will be using the input value as it is, extracting it out of your wrapped type, the idea that you were "Parsing" and not "Validating" well just confused me.
    - sullyj3 5y ago
      It's true that here the data type was just a tag. There's always a decision to be made about how much structure you want to enforce by the construction of the data type. You'd still consider it to be "parsing" because you now statically know that an Email is valid according to the rules of the parser - in a loose sense, it's a "conversion" between a type you know nothing about, and one you know a lot about. If you wanted to go further you could start going the "correct by construction" route - having a data type that enforces more invariants. For example, you might store the recipient name, domain name, and tld of your email in separate fields. Then your parser would more obviously be a parser. I think of the "mere tag" type as a kind of degenerate case of this, rather than something totally separate.
    - Iazel 5y ago
      I see, thanks for sharing. I will improve the README by adding a clear definition for parsing, validation and deserialization, so that we can all be on the same page ;)
- rdedev 5y ago
  I find this approach combined with phantom data types really cool. Now you can easily introduce a semantic differentiation between two instances of the same data type but without much overhead
- GordonS 5y ago
  If it helps, here's a related blog post but with a C# slant: https://andrewlock.net/using-strongly-typed-entity-ids-to-avoid-primitive-obsession-part-1 https://andrewlock.net/using-strongly-typed-entity-ids-to-av... The author refers to using primitives everywhere as "primitive obsession", and proposed using types instead.
  - dmux 5y ago
    Similar to the idea of "microtypes" (I've most often seen it used in Java circles): https://www.markhneedham.com/blog/2009/03/10/oo-micro-types/ https://www.markhneedham.com/blog/2009/03/10/oo-micro-types/
- StreamBright 5y ago
  Interesting naming. Strongly typed languages (especially in the ML family) have best practices that include using types instead of strings as function parameters. Email type itself is enough to skip validation in each function accepting that particular type. I think this is great first step using functional languages but you can go much much deeper than that. https://www.slideshare.net/ScottWlaschin/the-power-of-composition-ndc-oslo-2020 https://www.slideshare.net/ScottWlaschin/the-power-of-compos...
- matheusmoreira 5y ago
  This also has security implications. The input handling layer is critical. Bugs in parsing and validation code are responsible for a huge number of vulnerabilities. More details: http://langsec.org/ http://langsec.org/ > The Language-theoretic approach (LANGSEC) regards the Internet insecurity epidemic as a consequence of ad hoc programming of input handling at all layers of network stacks, and in other kinds of software stacks. > LANGSEC posits that the only path to trustworthy software that takes untrusted inputs is treating all valid or expected inputs as a formal language, and the respective input-handling routines as a recognizer for that language.
ledauphin 5y ago
I've been looking for a solid Typescript implementation of "parse don't validate" that performs runtime parsing using semantics attached to the defined Typescript types themselves. In other words, much like attrs for Python, I want to be able to define a low/no-boilerplate type, and then register parsers for those types that will work recursively to parse my data, resulting in the specified Typescript type. Has anyone seen or written something like this?
- bmuon 5y ago
  I've been using this small library inspired by Elm/Swift decoders [1]. It works, but it's not low boilerplate. I'm gravitating towards GraphQL now because strict parsing is built into it, so there is no need for all this boilerplate. https://www.npmjs.com/package/@mojotech/json-type-validation https://www.npmjs.com/package/@mojotech/json-type-validation
  - ledauphin 5y ago
    we use GraphQL for this purpose as well, but I'd also like to be able to validate across other boundaries. However, as I'm saying this, I wonder if I've been looking at this problem wrong. Since we already generate types from GraphQL schemas, maybe I should figure out how to use the same client side parser that's already in my GraphQL client, define a GraphQL schema for the types I'm interested in, and then just generate and use those types. One thing that doesn't necessarily give me is the ability to define custom parsers corresponding to custom types. At least, I think most of that sort of thing is usually done server side with GraphQL. So, thank you for the link and also the inspiration for considering an alternative.
- renke1 5y ago
  Not exactly what you want, I think, but there is zod [0]. I really would like to see nominal typing support in TypeScript. Currently, it's hard to validate a piece of data (or parse for that matter) once and have other functions only operate on that validated data. There are (ugly?) workarounds though [1]. [0]: https://github.com/colinhacks/zod https://github.com/colinhacks/zod [1]: https://gist.github.com/dcolthorp/aa21cf87d847ae9942106435bf47565d https://gist.github.com/dcolthorp/aa21cf87d847ae9942106435bf...
billytetrud 5y ago
To me this just looks like they're arguing for using class types rather than raw strings. The parsing seems kind of orthogonal and a special case of the kinds of validation you might want to do. It's also misleading in that the code is still doing validation, just in a different place.
- alserio 5y ago
  I mean, yes, the point is that it is a better place to do the validation step. Also, parse is generic to mean from a representation to a more structured one.
- Iazel 5y ago
  Yes, it is basically a combination of proofing some data has been validated by encoding this proof in a specific type, like Email :) We want to popularize this idea and make it easier to work with it by offering some nice, type-safe abstraction.
- didibus 5y ago
  Yeah, but I think it's even more so, they're arguing that you should model the fact that something has been validated or not, and functions should indicate if they expect a validated form of input or not. In that sense, using types is only one way to do this, but you could model that in other ways. For example: var foo = "123" foo = validFoo(foo) print(foo) > {"value" : "123", "valid?" : true} And now you could have: function bar(validFoo) { if (!validFoo.get("valid?")) throw new InvalidInputException("Foo must be validated prior to calling bar.") ... } Now types are a convenient way to do this that also gives you static checking for it, but I believe the idea is more to model that things were validated and expects validated input or fail. That allows you to push all validation at the boundary, and make sure that no one ever forgets to validate the input, because if they do, the inner functions will fail reminding the caller: Please remember to validate this!
  - billytetrud 5y ago
    Makes sense. It just seems like parsing is kind of a separate issue and shouldn't be entangled with the concept of input validation.
smnrchrds 5y ago
Fun fact: Parsix was the name of a Linux distro optimized for Persian speakers.
skybrian 5y ago
This library seems to be providing a framework and doesn’t include any interesting parsers. (There is no email address parser, despite the example.) It seems to allow for some composition of parsers, but the basic idea is a design pattern that’s simple enough that it doesn’t obviously require a framework. So it seems like most of the value comes from standardizing on domain types like Username, Email, and so on. Using a framework doesn’t get you there, and it adds a dependency on the framework.
- Iazel 5y ago
  Hi skybrian, would you mind explaining why do you see this as a framework? About missing interesting parsers, you are right, for now only the core part is done. Based on community interest, we will work on complementary packages, like more common parsers, easy integration with a web framework like ktor, effectful parsers based on coroutine, etc... Lots of work ahead :D
  - skybrian 5y ago
    Well, it's a minimal framework. Parsers are supposed to implement a particular function definition [1] and use the "Parsed" type for their return value, which if widely adopted, will result in references to the core library's types appearing in a lot of API's. The advantage is that you can write code using generics that works with any parser, but I'm a little skeptical about the value of generic code. [1] https://github.com/parsix/parsix#build-your-own-parse https://github.com/parsix/parsix#build-your-own-parse
    - Iazel 5y ago
      I see, but isn't that true for any library? xD In the end, `Parsed` should only be handled whereever you receive the initial input, so the rest of your business logic will be free of it :) In the end, this is just a tool and it's fine if it isn't fitting your particular style or use case ;)
      - skybrian 5y ago
throwawayboise 5y ago
I do as much of this as I can with database constraints. Foreign key constraints, or check constraints, or even triggers if necessary (though I do try to avoid them). Databases tend to outlive application code, or may be fronted by different applications (internal vs external for example). Keeping the constraints with the data is the best way to ensure that your data remains consistent within itself.
- Iazel 5y ago
  I see, this is also an interesting approach and definitely have its usages. Thinking about it, though, it has its own limitations when it comes to scalability and business requirements naturally out from the database box, eg: how would you ensure an S3 file reference is actually valid and it does exist?
  - mixedCase 5y ago
    You can't, since the file can be deleted from under you after verification anyway. But you can treat it as an "S3FileReference" or something arbitrarily generic like "RemoteFile" and have associated procedures/methods for verifying the properties you want at any point without confusing it with "just a string"
- jhardy54 5y ago
  I do this too, but I’m always frustrated by the mismatch between database constraints and application constraints. For example, when using Django you can declare a field as varchar(32) but that constraint isn’t checked until you actually insert the row into the database. I suppose maybe that’s not a problem in languages with more mature type safety ecosystems?
  - Iazel 5y ago
    Yeah, I've also worked with weak type systems in the past too (PHP, Ruby, JS), so I can definitely share the pain! I learned the hard way how much easier it is to build complex systems when you have a compiler helping you ;)
alex_duf 5y ago
I think this can be summarised by "model your domain by using types, then let the compiler ensure you're not doing anything silly"
- hinkley 5y ago
  Also known as Stop Using Stringly Typed Data
Waterluvian 5y ago
I developed a pattern in typescript (I'm sure it's not original) where I have an interface describing an API entity and a class of the same name with only static methods, one of which is Foo.fromApi() that validates and parses. I haven't seen any need to bring a library in to handle this. Though it would be nice to marry the worlds of TS, API, and Json Schema.
- lhnz 5y ago
  It doesn't use json schema but you might be interested in something like this: https://gcanti.github.io/io-ts/ https://gcanti.github.io/io-ts/ (You can define runtime encoder/decoders which produce typed values.)
  - brundolf 5y ago
    io-ts is fantastic (I linked it myself above). The killer feature is that it infers the static types of your runtime schemas for you, so you don't have to define them twice. You make a change to the schema, the rest of your code will typecheck against it.
twic 5y ago
Great, but why do you need a library for this? I just write classes with a falliable static parse method and a private constructor. It looks like this library was written by someone labouring under the mistaken belief that it's better to build and use a DSL to create the illusion of declarativity than to just write a line or two of normal code (eg the focusedParse stuff). Also, i demur somewhat at calling this parsing. It's tracking validation using typestate.
- mixedCase 5y ago
  You'll want to read on "parser combinator" as a concept. A parser combinator library can provide a good repertoire of primitives, standard interfaces for defining and using the parsers, and utilities for working with those interfaces. User-friendly error reporting comes to mind.
  - slver 5y ago
    This only matters when you parse complex syntax. Most input isn’t.
    - mixedCase 5y ago
      Boring Line-of-Business Applications have their bread and butter by parsing JSON and treating it with more meaning than what `string | number | boolean | object` can express. Therefore, they can benefit from parsers. If you think JSON primitives are "enough to express most input" then you missed the point; because of course you can, but no person who has learnt to use types to their advantage would want to write error-prone code and an unnecessary amount of unit tests that deal with ambiguous behavior for no good reason.
      - slver 5y ago
        Parsing JSON with various slightly specialized string fields like "email" or "url" and so on is precisely what I had in mind by simple input. There's absolutely nothing there for "parser combinators" to do. JSON parsing is a standalone step, and the values retrieved from it can be passed to another EXISTING (mind you) validators to handle the rest. I live and breath this every day as I deal with forms and APIs for a living. And I never was like "oh man this is hard, if I only had parser combinators". > If you think JSON primitives are "enough to express most input" then you missed the point Unclear why you're trying to put words in my mouth, but JSON is enough to describe the composition of input (in terms of lists, dictionaries and leaf scalar values). Which as simple as it is, turns out to be a very significant part of the task. Of course you have additional constraints on top of that. But no need for "parser combinators". To JSON, a string is a string. Any additional operations performed on that string happen AFTER it's decoded from a JSON string format to an in-memory native string literal. Those are entirely independent steps. A simple function composition would do i.e. json => string => email. And that's more or less how I parse arbitrary data input. All of which is simple to parse. Relative to, you know, say parsing a full programming language.
brundolf 5y ago
Similar thing for TypeScript: https://github.com/gcanti/io-ts https://github.com/gcanti/io-ts
samatman 5y ago
As a minor point of order, the exact phrase "parse, don't validate" has been conventional wisdom in langsec circles since I got involved, so 2014 at the earliest. I asked around on the work Matrix as to who actually coined it, but it's the weekend. This is not to take anything away from @lexi_lambda, who cited her sources and documented an interesting type-theoretic approach to applying the principle. She did a great job! If anyone wants to do a deeper dive, look into langsec, language-theoretic security. There's a lot of prior art to explore.
- matt-noonan 5y ago
  The principle was certainly known, but I think Alexis really does deserve the credit for the catchy "parse, don't validate" wording. A Google search for that phrase, restricted to October 2019 and earlier, has no results (or rather, the results that do show up all are more recent additions such as comments, appended to previously-existing content)
  - samatman 5y ago
    I assure you this isn't the case, I have personally heard it said as early as 2014. Meredith Patterson got back to me and attributes it to Sergey Bratus. We're a little vague on when, but it was quite some time ago. It's a great blog post, and it popularized the slogan, which is the important part. She was quite clear in the original post to cite langsec, everyone here is on a collegial basis. My point was not really about 'credit', it was about langsec. If the ideas in this library, and that post, are interesting, there's a lot more to discover in langsec. That's it.
- Iazel 5y ago
  Cool, didn't know that, thanks for sharing!
araknafobia 5y ago
What forces you to use validator function and not just do Email(input)? This is how you actually do it in Scala if anyone interested https://github.com/fthomas/refined https://github.com/fthomas/refined
- agbell 5y ago
  Yeah, refined types are a great solution to this problem, but can lead to slow compile times. Just having private constructor for a case class and a public apply method that is an Option or Either or something can go a long way in my experience.
andrenarchy 5y ago
For TypeScript there is the pure functional https://github.com/paperhive/fefe https://github.com/paperhive/fefe Less boilerplate than the other solutions I know... for example it allows you to define types easily just by writing a parser/validator ("transformation" in fefe terminology).
slver 5y ago
Wrapping primitives comes with its own set of problems. Like having to unbox it all the time for things like basic display. So you either validate it all the time or you unbox it all the time. Also with dedicated primitives you need to validate the email when you read it from the dB to hydrate it. So you removed couple of validations here but added a couple there. My solution is stick to basic DTO and validate in the end at the service receiving the data, and let the errors, if any, propagate up the chain in a way that preserves where the input came from. In most cases I validate once. So this issue we’re talking about. Doesn’t happen.
- Iazel 5y ago
  I guess it depends on how you structure your code. If you validate at constructor level, then you have this problem. However if you actually parse it, then this will have to be external to the actual class and you are left with a simple Value Object. This means that you can apply constraints only when it makes sense. Most of the time, there is little reason to validate something coming from the DB, so I would rather skip it. About unboxing, that depends. I would expect for the overall business logic to always use the more structured types, probably this will only be needed once you need to serialize them :) That's said, if you are happy with your current way, please keep doing it! I would still suggest to give this style a try though :)
jart 5y ago
Didn't really understand the linked blog posts, but "Parse Don't Validate" sounds like another way of stating Postel's Maxim, which is to be liberal in what you accept but conservative in what you send. I generally try to follow that principle with the way I've written the redbean web server. When it comes to things like URLs and HTTP messages, there's a whole bunch of characters in each field that the RFCs say aren't allowed, but the parser code is still able to work fine even if they're permitted. For example, the RFC says ` isn't allowed in a path, but why should the parser care? It's not like that's a syntactic element. There's no reason to consult a map of validate characters, which slows things down, until you want to re-encode the parsed URL. The re-encoded version will turn the parsed ` into %60 since that's what you might send back to the network, and that needs to be perfect, since you don't know if the other end has one of those strict as a matter of policy parsers, that ignore what's possible and enforces only what's legal.
- Iazel 5y ago
  Hey there, I wonder if this article I wrote about the same concept would make it more clear? It's on medium, but free :) https://pelligra-s.medium.com/parse-dont-validate-in-kotlin-afbac505a04f https://pelligra-s.medium.com/parse-dont-validate-in-kotlin-... I honestly think the two arguments are different though. We could say that Postel's Maxim is about "how much you should parse/validate", while the problem we are addressing is rather "how you should do it" :)
  - jart 5y ago
    Oh I see. You define parsing more in the yacc sense. Carry on. I've been working with char arrays too long.
- IshKebab 5y ago
  I'm not sure if you're aware but Postel's Maxim is fairly universally viewed as a huge mistake. It leads to security issues and poor reliability.
  - jart 5y ago
devit 5y ago
"Parse don't validate is suboptimal" What you ideally want is a first step that "validates", i.e. creates a representation from text that is easy to use but also succeeds for any input, and then a further "parsing" step that converts it to another representation such that the parsing only succeeds for valid input and is invertible for any value of the representation type, and then finally code that checks constraints that aren't captured by the type system. This way you can support IDEs that need to edit partially incorrect code.
- Iazel 5y ago
  Hey, I fail to see how this is suboptimal. Some points I don't understand: * What's the goal of a validation that always succeed? * What's the value of checking constraints after you parsed it and not capture this information? In the end, parsing is just a combination of what you described: ensure something fit a particular shape and constraints :)
robertlagrant 5y ago
Interesting stuff. Having read Parse, Don't Validate, it would be good to know how to face potential practical pitfalls, such as: 1) how to serialise these types into different formats (e.g. "how do I save you in a SQL database?) - inside the type, which requires one size fits all, or as separate mapping functions, which could result in significant code proliferation? 2) how to cope with partial validation, e.g. do I need a wrapper class for (say) some form inputs that have these types in its fields, but override nullable on if it's off? 3) sort of a combo of (1) and (2), but how do I serialise groups of these typed fields in different ways? E.g. I want to save a model made up of these typed fields via SQLAlchemy and I also want to publish it to RabbitMQ. How do I manage the potentially varied formats in a type-safe way? When everything was Just Strings™ it appeared to Just Work™. 4) how to compose these things with monads - e.g. if I have an optional field, should Maybe treat the value as nullable, or should the type itself contain that option for validation purposes elsewhere? Even as I'm writing these questions I'm coming up with a potential pattern for using this sort of thing, but I'm curious to hear thoughts.
- Iazel 5y ago
  Hey, thanks for starting this discussion, I think it will be an interesting one :) 1) Just to be clear, when I read "serialise" I understand that you want to parse some object into a format suitable for storage. I would honestly go with the common flow: use some library like kotlinx.serialization or whatever your database library support. If this isn't what you meant, please let me know. 2) I personally like well defined type. For example, if you have a case where Email must be like Email(local, domain, tld) and another where you just want a String out of your email, I would rather use two different types for the two different use cases and therefore different parsing logic (which will probably share most code, but still produce different results). On the other hand, if you want to model the fact that some piece of information could be provided or not by the external source, than yes, I would just go with nullable types, which in Kotlin is like `Something?`, but in other language are known as Maybe or Optional. 3) Just to be clear, I will strongly advice against having validation in object constructors. Once this is out of the picture, you are left with simple Value Objects: simple, immutable, bundle of data with a defined structure and semantic meaning. When you work with these simple objects, it's very easy to derive different views for whatever use case you need. I'm not familiar with SQLAlchemy and Python ecosystem in general, but I guess there will be some way of converting a specific type into one that is suitable for DB consumption. 4) Both Parse and Parsed are actually Monads ;) And we know something about Monads, is that composing different Monads together is usually not straightforward xD That's said, in your particular example of having an optional field, I would see the parser produce a Maybe<Something> (or Something? in Kotlin) Let me know if I got you right and what you think about it :)

153 points by Iazel 5y ago

I have to be honest, I'm not seeing what problem this is trying to solve. Anyone can enlighten me? Edit: Ok I think I understand... It seems the problem would be that if you're implementing a function that takes a user email as a string, and that function is in a lower layer of the application, like inside the data access layer. It is difficult at this point to know if the email string you will be passed as input has already been validated or not. Thus you might be tempted to re-implement validation for it at your level inside this function as well and have an assertValidEmail check. This can lead to a littering of validation throughout the code base, as each implemented function worries that the input isn't validated and re-validates it, possibly using slightly different rules each time. Furthermore, if you decide to not validate it again, you might be left wondering, but am I sure it'll have been validated prior? How can I be sure? Someone in the future could easily start calling my function and forget to validate the email before calling it? This could eventually lead to a security issue or just a bug, by introducing a code path that doesn't ever validate the email string. Thus if instead you'd re-write your function so it takes an email as a ValidEmail type (or object), and not as a string, you force the caller to for sure remember to validate the email first. And you also can safely assume if you're getting an email as a ValidEmail type that it has been validated. It could also technically allow you to localize the validation logic to the ValidEmail type constructor, avoiding possible duplicate attempts at validating email with different rules. And it seems the latter "style" the author calls "Parsing" while the former they call "Validating", in the sense that since the function validating returns a modified structure it "parsed" it, because a string became a ValidEmail, thus parsing a string into a ValidEmail, as opposed to simply validating that the string is valid as an email. And finally, this is a little library to help make use of this pattern in Kotlin.

jinwoo68 5y ago
As they said in README, it’s inspired by Alex King’s Parse, don’t validate [1]. Basically, rather than write a validation function, write a parser that returns a result of a specific type and use that type everywhere else. Then you can make sure the raw inputs are always validated. [1] https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/ https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-va...
- anaerobicover 5y ago
  Small correcting, the author is named Alexis
  - jinwoo68 5y ago
    Whoops. Thanks for correcting.
steventhedev 5y ago
There's an entire class of vulnerabilities caused by having separate verification and parsing logic, typically with fields that usually only one is used, but the format supports multiple. The verifier checks the first one but the parser uses the last one.
Smaug123 5y ago
If you mean "why Parse, Don't Validate", you should read the original blog post, linked at the top of the article. It's… transformative, if you aren't already aware of the principle. https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/ https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-va... If you mean "why this library", well, I guess parser combinators are nice! Some may say that a declarative statement of the parsing restrictions is better than a procedural implementation, on general principles.
finnh 5y ago
It encourages people to use strongly typed classes rather than primitives, even if the type simply wraps a primitive. As a result you can't pass a invalid (say) accountID deep into your code, bc validity is guaranteed to be checked early when you "parse" an input string into the "AccountId" type. So: internal interfaces defined using non-primitive types, so internal methods don't need to keep validating their input. Conversion to said types happens early and predictably, catching bad values before they (eg) hit the database.
coderintherye 5y ago
The linked blog post explains it pretty well. Essentially, it seems to be solving for unexpected cases or incorrect validation by using of static typing and passing the expected type back in the return rather than a boolean. I'm not sure I've encountered enough issues with validation functions to use this pattern, but it does seem like a more robust way of writing them.
jpeloquin 5y ago
Paraphrased from the repo's readme: Suppose you have a program that consumes user input. Users often give bad input so the program needs to validate user input before acting on it. One way to validate is to call a function (e.g., `check_input`) on the user input and if it doesn't raise an error the input is safe for consumption by the rest of the program. The repo author considers this approach to be risky because the programmer can inadvertently omit or bypass `check_input` and the program still compiles and runs without complaint. The repo presents an alternative validation approach, which is to parse the user input into a data type (or, not quite equivalently, into a class). The parsing process serves as validation. Consumer functions are written such that they only accept the parsed data type. Therefore it is now impossible for the programmer to inadvertently omit or bypass validation of user input. The library is a set of convenience functions for actually writing these parsing / validation functions.
- atoav 5y ago
  So in short: instead of representing user input (e.g. a Email address) as a string – which you can forget to validate – the idea here is to create a own data type for it, and use the validation step to create said data type. The rest of your program then works with this data type instead of the string and this way you will get a type error whenever you accidentally use unvalidated data. A nice idea that goes into a similar direction is to expand on this and create more types for different levels of trust. E.g. you could have the data types ValidatedEmail, VerifiedEmail and TrustedEmail and define precisely how one becomes the other. This way your typesystem will already tell you what is valid and what is not and you can't accidental mix them up.
  - TeMPOraL 5y ago
    You can also further generalize this idea by noticing you can encode all kinds of life cycle information in your type system. As you transform some data in a sequence of steps, you can use types to document and enforce the steps are always executed in order. In this example, the user input validation step is f(String) -> ValidatedEmail, then the process of verifying it is f(ValidatedEmail) -> VerifiedEmail. But the same principle can apply to e.g. append() operation being f(List[T], T) -> NonEmptyList[T], and you can write code accepting NonEmptyList to save yourself an emptiness check. Or, take a multi-step algorithm that gets a list of users, filters them by some criterion, sorts the list, and sends these users e-mails. Type-wise, it's a flow of Users -> EligibleUsers -> SortedEligibleUsers -> ContactedEligibleUsers. And then, why should types be singular anyway? You should be able to tag data with properties, and then filter on or transform a subtag of them. This is the area of theory I'm not familiar with yet, but I imagine you should be able to do things like: List[User] -> List[User, NonEmpty] -> List[User[Eligible], NonEmpty] -> List[User[Eligible], NonEmpty, Sorted[Asc]] -> List[User[Contacted], Sorted[Asc]]. Or, Email -> Email[Validated] -> Email[Validated, Verified] -> Email[Validated, Verified, Trusted]. I'm sure there's a programming language that does that, and then there's probably lots of reasons that this doesn't work in practice. I'd love to know about them, as I haven't encountered anything like it in practice, except bits and pieces of compiler code that can sometimes propagate such information "in the background", for optimization and correctness checking.
cle 5y ago
There are lots of siblings explaining why “parse don’t validate”. But also, it’s not always wise to take this to an extreme. I’ve seen over the years many scenarios where dev teams were over-enthusiastic about this and parsed themselves into a corner by making system components over-strict and enforcing invariants that weren’t necessary to enforce, making them much harder to change later. The right answer is, of course, somewhere in the middle, and depends on your domain and situation.
- Iazel 5y ago
  hi, @cle! Curious to hear more about that, were they actually running validation/assertions in constructors?
  - cle 5y ago
    That can be a case of that yeah. Using your example, a lot of devs might use that email parsing logic in various independent components of the same system. Eg if you have a reporting component that sends you business reports, that component really shouldn’t be validating the structure of email addresses…if you need to refine the parsing logic now you’ve got to do coordinated deployments, possibly backfills, etc., whereas if you just treated it as an opaque string in that system you’d be better off. This isn’t really a criticism of the approach, it’s super useful, just that it needs to be applied judiciously. “Parse all the things” isn’t always the best advice.
    - Kranar 5y ago
      Your response isn't clear. >that component really shouldn’t be validating the structure of email addresses That's right, as the article says you no longer validate data, instead you parse data. Components don't take strings and validate them, they take an EMail address. >If you need to refine the parsing logic now you’ve got to do coordinated deployments If you need to refine the parsing logic, you modify the constructor of the EMail class to do whatever refinements you need. I have no doubt you may have seen codebases that did something wrong, and that wrong thing may have been related to parsing or validation, but nothing you've said indicates that there is something wrong with replacing validation with parsing, so that code always operates on semantically meaningful data types that are valid by construction, instead of opaque binary blobs that need to constantly be properly interpreted in ad-hoc ways.
mirekrusin 5y ago
Imagine you're writing typescript project. You type everything and have type safety. This type safety is an illusion on I/O boundaries – whenever ie. JSON.parse(...) from file/websocket/http happens. To preserve type safety, you want to use something like [0] to do runtime type assertions. Once i/o boundaries are parsing unknown types at runtime into what is defined as static types, your type safety is guaranteed. [0] https://github.com/appliedblockchain/assert-combinators https://github.com/appliedblockchain/assert-combinators
TheAceOfHearts 5y ago
Refining types so they encode all desired constraints before use. This is explained in the linked article: Parse, don’t validate [0]. It helps reduce the risk of using invalid inputs by representing constraints over the value as part of the type. For example: a common problem in web development security is that query parameters aren't properly validated which can lead to denial of service attacks. As a trivial example of this, consider a web server which paginates some data using "offset" and "limit" by passing those parameters directly to a database query; an attacker could set "limit" to some incredibly high value and cause the server to crash. If you're just doing validation on your inputs it's possible that some usage could end up being overlooked. [0] https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/ https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-va...
- gregors 5y ago
  So real question - in the "offset" "limit" example what makes it any more safe if at first the programmer sets those types to be integers? The same problem persists does it not? Does the explicit creation of a type add this introspection? I'm not convinced that it does. Now once you fix this bug, encoding it in a type prevents it from creeping into other parts of the the code. This seems more like DRY principles in action.
  - didibus 5y ago
    Yeah, it seems to be more about guarantees as a code base grows larger and more people touch it. If there's a Limit class whose constructor and setter all check that the range is between say 5 to 100, and all existing code that needs the limit uses the instance of Limit, it just becomes less likely a code change is made that uses the limit input as it was directly provided by the user (and thus possibly out of range). But you'd still need to have had someone be smart enough to make sure the Limit class does prevent limits that could cause DB crashes. In practice I'm thinking, ok, so someone must have thought... Hey we should validate this user input and put in some logic for it. So I think what this says is, validation works by having all external input validated as they are received. But it can be easy to make a code change at the boundary where you forget to add proper validation. If all existing functions in the lower layers, like in the data access layer, are designed to take a Limit object, the person who took a limit as external input and was about to pass it to the query function will get a compile error and realize... Oh I need to first parse my integer limit into a Limit, and thus reminds them to use the thing that enforces the valid range. If instead the code had a util function called assertValidLimit, and the query function took a limit as an integer, it be easy for that person to forget to add a call to assertValidLimit when getting the limit from the user and then pass that unvalidated to the query and possibly cause a vulnerability. And lastly, it seems they argue, if you were to validate instead in the query function itself, thus it wouldn't matter if others forget to validate since where it matters would, but then it is hard to fail at that layer, since you might have already made other changes and that can leave your state corrupted. So basically it seems the argument is: "It is best to validate external input at the boundary as soon as it is received, but it can be easy to forget to do so and that's dangerous. So to help you not forget, have all implementing functions take a different type then the type of the external input, which will remind people... Oh right I need to parse this thing first and in doing so assert it's valid as well.
Iazel 5y ago
Cool to see you perfectly got the point in the end! I wonder though, were you confused by the README? What made it clear for you?
- didibus 5y ago
  Hum, it was the people here who replied to my question, and also reading the linked article. I think my confusion was in trying to frame things as parsing VS validating. While I now appreciate that use of word, now that I understand, it also caused my biggest source of confusion. That's because I think most people think of parsing as conversion, like I turn a String to an Int. Where as in your case, you're simply wanting to tag a type as having been validated, but you don't really convert the type itself, so you simply wrap it in another type in order to tag it as having been validated simply because the language offers no other way to tag the type with meta-information for the compiler to assert statically. So because it seemed more like you're just wrapping the input, but still all code will be using the input value as it is, extracting it out of your wrapped type, the idea that you were "Parsing" and not "Validating" well just confused me.
  - sullyj3 5y ago
    It's true that here the data type was just a tag. There's always a decision to be made about how much structure you want to enforce by the construction of the data type. You'd still consider it to be "parsing" because you now statically know that an Email is valid according to the rules of the parser - in a loose sense, it's a "conversion" between a type you know nothing about, and one you know a lot about. If you wanted to go further you could start going the "correct by construction" route - having a data type that enforces more invariants. For example, you might store the recipient name, domain name, and tld of your email in separate fields. Then your parser would more obviously be a parser. I think of the "mere tag" type as a kind of degenerate case of this, rather than something totally separate.
  - Iazel 5y ago
    I see, thanks for sharing. I will improve the README by adding a clear definition for parsing, validation and deserialization, so that we can all be on the same page ;)
rdedev 5y ago
I find this approach combined with phantom data types really cool. Now you can easily introduce a semantic differentiation between two instances of the same data type but without much overhead
GordonS 5y ago
If it helps, here's a related blog post but with a C# slant: https://andrewlock.net/using-strongly-typed-entity-ids-to-avoid-primitive-obsession-part-1 https://andrewlock.net/using-strongly-typed-entity-ids-to-av... The author refers to using primitives everywhere as "primitive obsession", and proposed using types instead.
- dmux 5y ago
  Similar to the idea of "microtypes" (I've most often seen it used in Java circles): https://www.markhneedham.com/blog/2009/03/10/oo-micro-types/ https://www.markhneedham.com/blog/2009/03/10/oo-micro-types/
StreamBright 5y ago
Interesting naming. Strongly typed languages (especially in the ML family) have best practices that include using types instead of strings as function parameters. Email type itself is enough to skip validation in each function accepting that particular type. I think this is great first step using functional languages but you can go much much deeper than that. https://www.slideshare.net/ScottWlaschin/the-power-of-composition-ndc-oslo-2020 https://www.slideshare.net/ScottWlaschin/the-power-of-compos...
matheusmoreira 5y ago
This also has security implications. The input handling layer is critical. Bugs in parsing and validation code are responsible for a huge number of vulnerabilities. More details: http://langsec.org/ http://langsec.org/ > The Language-theoretic approach (LANGSEC) regards the Internet insecurity epidemic as a consequence of ad hoc programming of input handling at all layers of network stacks, and in other kinds of software stacks. > LANGSEC posits that the only path to trustworthy software that takes untrusted inputs is treating all valid or expected inputs as a formal language, and the respective input-handling routines as a recognizer for that language.

ledauphin 5y ago

I've been looking for a solid Typescript implementation of "parse don't validate" that performs runtime parsing using semantics attached to the defined Typescript types themselves. In other words, much like attrs for Python, I want to be able to define a low/no-boilerplate type, and then register parsers for those types that will work recursively to parse my data, resulting in the specified Typescript type. Has anyone seen or written something like this?

bmuon 5y ago
I've been using this small library inspired by Elm/Swift decoders [1]. It works, but it's not low boilerplate. I'm gravitating towards GraphQL now because strict parsing is built into it, so there is no need for all this boilerplate. https://www.npmjs.com/package/@mojotech/json-type-validation https://www.npmjs.com/package/@mojotech/json-type-validation
- ledauphin 5y ago
  we use GraphQL for this purpose as well, but I'd also like to be able to validate across other boundaries. However, as I'm saying this, I wonder if I've been looking at this problem wrong. Since we already generate types from GraphQL schemas, maybe I should figure out how to use the same client side parser that's already in my GraphQL client, define a GraphQL schema for the types I'm interested in, and then just generate and use those types. One thing that doesn't necessarily give me is the ability to define custom parsers corresponding to custom types. At least, I think most of that sort of thing is usually done server side with GraphQL. So, thank you for the link and also the inspiration for considering an alternative.
renke1 5y ago
Not exactly what you want, I think, but there is zod [0]. I really would like to see nominal typing support in TypeScript. Currently, it's hard to validate a piece of data (or parse for that matter) once and have other functions only operate on that validated data. There are (ugly?) workarounds though [1]. [0]: https://github.com/colinhacks/zod https://github.com/colinhacks/zod [1]: https://gist.github.com/dcolthorp/aa21cf87d847ae9942106435bf47565d https://gist.github.com/dcolthorp/aa21cf87d847ae9942106435bf...

billytetrud 5y ago

To me this just looks like they're arguing for using class types rather than raw strings. The parsing seems kind of orthogonal and a special case of the kinds of validation you might want to do. It's also misleading in that the code is still doing validation, just in a different place.

alserio 5y ago
I mean, yes, the point is that it is a better place to do the validation step. Also, parse is generic to mean from a representation to a more structured one.
Iazel 5y ago
Yes, it is basically a combination of proofing some data has been validated by encoding this proof in a specific type, like Email :) We want to popularize this idea and make it easier to work with it by offering some nice, type-safe abstraction.
didibus 5y ago
Yeah, but I think it's even more so, they're arguing that you should model the fact that something has been validated or not, and functions should indicate if they expect a validated form of input or not. In that sense, using types is only one way to do this, but you could model that in other ways. For example: var foo = "123" foo = validFoo(foo) print(foo) > {"value" : "123", "valid?" : true} And now you could have: function bar(validFoo) { if (!validFoo.get("valid?")) throw new InvalidInputException("Foo must be validated prior to calling bar.") ... } Now types are a convenient way to do this that also gives you static checking for it, but I believe the idea is more to model that things were validated and expects validated input or fail. That allows you to push all validation at the boundary, and make sure that no one ever forgets to validate the input, because if they do, the inner functions will fail reminding the caller: Please remember to validate this!
- billytetrud 5y ago
  Makes sense. It just seems like parsing is kind of a separate issue and shouldn't be entangled with the concept of input validation.

smnrchrds 5y ago

Fun fact: Parsix was the name of a Linux distro optimized for Persian speakers.

skybrian 5y ago

This library seems to be providing a framework and doesn’t include any interesting parsers. (There is no email address parser, despite the example.) It seems to allow for some composition of parsers, but the basic idea is a design pattern that’s simple enough that it doesn’t obviously require a framework. So it seems like most of the value comes from standardizing on domain types like Username, Email, and so on. Using a framework doesn’t get you there, and it adds a dependency on the framework.

Iazel 5y ago
Hi skybrian, would you mind explaining why do you see this as a framework? About missing interesting parsers, you are right, for now only the core part is done. Based on community interest, we will work on complementary packages, like more common parsers, easy integration with a web framework like ktor, effectful parsers based on coroutine, etc... Lots of work ahead :D
- skybrian 5y ago
  Well, it's a minimal framework. Parsers are supposed to implement a particular function definition [1] and use the "Parsed" type for their return value, which if widely adopted, will result in references to the core library's types appearing in a lot of API's. The advantage is that you can write code using generics that works with any parser, but I'm a little skeptical about the value of generic code. [1] https://github.com/parsix/parsix#build-your-own-parse https://github.com/parsix/parsix#build-your-own-parse
  - Iazel 5y ago
    I see, but isn't that true for any library? xD In the end, `Parsed` should only be handled whereever you receive the initial input, so the rest of your business logic will be free of it :) In the end, this is just a tool and it's fine if it isn't fitting your particular style or use case ;)
    - skybrian 5y ago

throwawayboise 5y ago

I do as much of this as I can with database constraints. Foreign key constraints, or check constraints, or even triggers if necessary (though I do try to avoid them). Databases tend to outlive application code, or may be fronted by different applications (internal vs external for example). Keeping the constraints with the data is the best way to ensure that your data remains consistent within itself.

Iazel 5y ago
I see, this is also an interesting approach and definitely have its usages. Thinking about it, though, it has its own limitations when it comes to scalability and business requirements naturally out from the database box, eg: how would you ensure an S3 file reference is actually valid and it does exist?
- mixedCase 5y ago
  You can't, since the file can be deleted from under you after verification anyway. But you can treat it as an "S3FileReference" or something arbitrarily generic like "RemoteFile" and have associated procedures/methods for verifying the properties you want at any point without confusing it with "just a string"
jhardy54 5y ago
I do this too, but I’m always frustrated by the mismatch between database constraints and application constraints. For example, when using Django you can declare a field as varchar(32) but that constraint isn’t checked until you actually insert the row into the database. I suppose maybe that’s not a problem in languages with more mature type safety ecosystems?
- Iazel 5y ago
  Yeah, I've also worked with weak type systems in the past too (PHP, Ruby, JS), so I can definitely share the pain! I learned the hard way how much easier it is to build complex systems when you have a compiler helping you ;)

alex_duf 5y ago

I think this can be summarised by "model your domain by using types, then let the compiler ensure you're not doing anything silly"

hinkley 5y ago
Also known as Stop Using Stringly Typed Data

Waterluvian 5y ago

I developed a pattern in typescript (I'm sure it's not original) where I have an interface describing an API entity and a class of the same name with only static methods, one of which is Foo.fromApi() that validates and parses. I haven't seen any need to bring a library in to handle this. Though it would be nice to marry the worlds of TS, API, and Json Schema.

lhnz 5y ago
It doesn't use json schema but you might be interested in something like this: https://gcanti.github.io/io-ts/ https://gcanti.github.io/io-ts/ (You can define runtime encoder/decoders which produce typed values.)
- brundolf 5y ago
  io-ts is fantastic (I linked it myself above). The killer feature is that it infers the static types of your runtime schemas for you, so you don't have to define them twice. You make a change to the schema, the rest of your code will typecheck against it.

twic 5y ago

Great, but why do you need a library for this? I just write classes with a falliable static parse method and a private constructor. It looks like this library was written by someone labouring under the mistaken belief that it's better to build and use a DSL to create the illusion of declarativity than to just write a line or two of normal code (eg the focusedParse stuff). Also, i demur somewhat at calling this parsing. It's tracking validation using typestate.

mixedCase 5y ago
You'll want to read on "parser combinator" as a concept. A parser combinator library can provide a good repertoire of primitives, standard interfaces for defining and using the parsers, and utilities for working with those interfaces. User-friendly error reporting comes to mind.
- slver 5y ago
  This only matters when you parse complex syntax. Most input isn’t.
  - mixedCase 5y ago
    Boring Line-of-Business Applications have their bread and butter by parsing JSON and treating it with more meaning than what `string | number | boolean | object` can express. Therefore, they can benefit from parsers. If you think JSON primitives are "enough to express most input" then you missed the point; because of course you can, but no person who has learnt to use types to their advantage would want to write error-prone code and an unnecessary amount of unit tests that deal with ambiguous behavior for no good reason.
    - slver 5y ago
      Parsing JSON with various slightly specialized string fields like "email" or "url" and so on is precisely what I had in mind by simple input. There's absolutely nothing there for "parser combinators" to do. JSON parsing is a standalone step, and the values retrieved from it can be passed to another EXISTING (mind you) validators to handle the rest. I live and breath this every day as I deal with forms and APIs for a living. And I never was like "oh man this is hard, if I only had parser combinators". > If you think JSON primitives are "enough to express most input" then you missed the point Unclear why you're trying to put words in my mouth, but JSON is enough to describe the composition of input (in terms of lists, dictionaries and leaf scalar values). Which as simple as it is, turns out to be a very significant part of the task. Of course you have additional constraints on top of that. But no need for "parser combinators". To JSON, a string is a string. Any additional operations performed on that string happen AFTER it's decoded from a JSON string format to an in-memory native string literal. Those are entirely independent steps. A simple function composition would do i.e. json => string => email. And that's more or less how I parse arbitrary data input. All of which is simple to parse. Relative to, you know, say parsing a full programming language.

brundolf 5y ago

Similar thing for TypeScript: https://github.com/gcanti/io-ts https://github.com/gcanti/io-ts

samatman 5y ago

As a minor point of order, the exact phrase "parse, don't validate" has been conventional wisdom in langsec circles since I got involved, so 2014 at the earliest. I asked around on the work Matrix as to who actually coined it, but it's the weekend. This is not to take anything away from @lexi_lambda, who cited her sources and documented an interesting type-theoretic approach to applying the principle. She did a great job! If anyone wants to do a deeper dive, look into langsec, language-theoretic security. There's a lot of prior art to explore.

matt-noonan 5y ago
The principle was certainly known, but I think Alexis really does deserve the credit for the catchy "parse, don't validate" wording. A Google search for that phrase, restricted to October 2019 and earlier, has no results (or rather, the results that do show up all are more recent additions such as comments, appended to previously-existing content)
- samatman 5y ago
  I assure you this isn't the case, I have personally heard it said as early as 2014. Meredith Patterson got back to me and attributes it to Sergey Bratus. We're a little vague on when, but it was quite some time ago. It's a great blog post, and it popularized the slogan, which is the important part. She was quite clear in the original post to cite langsec, everyone here is on a collegial basis. My point was not really about 'credit', it was about langsec. If the ideas in this library, and that post, are interesting, there's a lot more to discover in langsec. That's it.
Iazel 5y ago
Cool, didn't know that, thanks for sharing!

araknafobia 5y ago

What forces you to use validator function and not just do Email(input)? This is how you actually do it in Scala if anyone interested https://github.com/fthomas/refined https://github.com/fthomas/refined

agbell 5y ago
Yeah, refined types are a great solution to this problem, but can lead to slow compile times. Just having private constructor for a case class and a public apply method that is an Option or Either or something can go a long way in my experience.

andrenarchy 5y ago

For TypeScript there is the pure functional https://github.com/paperhive/fefe https://github.com/paperhive/fefe Less boilerplate than the other solutions I know... for example it allows you to define types easily just by writing a parser/validator ("transformation" in fefe terminology).

slver 5y ago

Wrapping primitives comes with its own set of problems. Like having to unbox it all the time for things like basic display. So you either validate it all the time or you unbox it all the time. Also with dedicated primitives you need to validate the email when you read it from the dB to hydrate it. So you removed couple of validations here but added a couple there. My solution is stick to basic DTO and validate in the end at the service receiving the data, and let the errors, if any, propagate up the chain in a way that preserves where the input came from. In most cases I validate once. So this issue we’re talking about. Doesn’t happen.

Iazel 5y ago
I guess it depends on how you structure your code. If you validate at constructor level, then you have this problem. However if you actually parse it, then this will have to be external to the actual class and you are left with a simple Value Object. This means that you can apply constraints only when it makes sense. Most of the time, there is little reason to validate something coming from the DB, so I would rather skip it. About unboxing, that depends. I would expect for the overall business logic to always use the more structured types, probably this will only be needed once you need to serialize them :) That's said, if you are happy with your current way, please keep doing it! I would still suggest to give this style a try though :)

jart 5y ago

Didn't really understand the linked blog posts, but "Parse Don't Validate" sounds like another way of stating Postel's Maxim, which is to be liberal in what you accept but conservative in what you send. I generally try to follow that principle with the way I've written the redbean web server. When it comes to things like URLs and HTTP messages, there's a whole bunch of characters in each field that the RFCs say aren't allowed, but the parser code is still able to work fine even if they're permitted. For example, the RFC says ` isn't allowed in a path, but why should the parser care? It's not like that's a syntactic element. There's no reason to consult a map of validate characters, which slows things down, until you want to re-encode the parsed URL. The re-encoded version will turn the parsed ` into %60 since that's what you might send back to the network, and that needs to be perfect, since you don't know if the other end has one of those strict as a matter of policy parsers, that ignore what's possible and enforces only what's legal.

Iazel 5y ago
Hey there, I wonder if this article I wrote about the same concept would make it more clear? It's on medium, but free :) https://pelligra-s.medium.com/parse-dont-validate-in-kotlin-afbac505a04f https://pelligra-s.medium.com/parse-dont-validate-in-kotlin-... I honestly think the two arguments are different though. We could say that Postel's Maxim is about "how much you should parse/validate", while the problem we are addressing is rather "how you should do it" :)
- jart 5y ago
  Oh I see. You define parsing more in the yacc sense. Carry on. I've been working with char arrays too long.
IshKebab 5y ago
I'm not sure if you're aware but Postel's Maxim is fairly universally viewed as a huge mistake. It leads to security issues and poor reliability.
- jart 5y ago

devit 5y ago

"Parse don't validate is suboptimal" What you ideally want is a first step that "validates", i.e. creates a representation from text that is easy to use but also succeeds for any input, and then a further "parsing" step that converts it to another representation such that the parsing only succeeds for valid input and is invertible for any value of the representation type, and then finally code that checks constraints that aren't captured by the type system. This way you can support IDEs that need to edit partially incorrect code.

Iazel 5y ago
Hey, I fail to see how this is suboptimal. Some points I don't understand: * What's the goal of a validation that always succeed? * What's the value of checking constraints after you parsed it and not capture this information? In the end, parsing is just a combination of what you described: ensure something fit a particular shape and constraints :)

robertlagrant 5y ago

Interesting stuff. Having read Parse, Don't Validate, it would be good to know how to face potential practical pitfalls, such as: 1) how to serialise these types into different formats (e.g. "how do I save you in a SQL database?) - inside the type, which requires one size fits all, or as separate mapping functions, which could result in significant code proliferation? 2) how to cope with partial validation, e.g. do I need a wrapper class for (say) some form inputs that have these types in its fields, but override nullable on if it's off? 3) sort of a combo of (1) and (2), but how do I serialise groups of these typed fields in different ways? E.g. I want to save a model made up of these typed fields via SQLAlchemy and I also want to publish it to RabbitMQ. How do I manage the potentially varied formats in a type-safe way? When everything was Just Strings™ it appeared to Just Work™. 4) how to compose these things with monads - e.g. if I have an optional field, should Maybe treat the value as nullable, or should the type itself contain that option for validation purposes elsewhere? Even as I'm writing these questions I'm coming up with a potential pattern for using this sort of thing, but I'm curious to hear thoughts.

Iazel 5y ago
Hey, thanks for starting this discussion, I think it will be an interesting one :) 1) Just to be clear, when I read "serialise" I understand that you want to parse some object into a format suitable for storage. I would honestly go with the common flow: use some library like kotlinx.serialization or whatever your database library support. If this isn't what you meant, please let me know. 2) I personally like well defined type. For example, if you have a case where Email must be like Email(local, domain, tld) and another where you just want a String out of your email, I would rather use two different types for the two different use cases and therefore different parsing logic (which will probably share most code, but still produce different results). On the other hand, if you want to model the fact that some piece of information could be provided or not by the external source, than yes, I would just go with nullable types, which in Kotlin is like `Something?`, but in other language are known as Maybe or Optional. 3) Just to be clear, I will strongly advice against having validation in object constructors. Once this is out of the picture, you are left with simple Value Objects: simple, immutable, bundle of data with a defined structure and semantic meaning. When you work with these simple objects, it's very easy to derive different views for whatever use case you need. I'm not familiar with SQLAlchemy and Python ecosystem in general, but I guess there will be some way of converting a specific type into one that is suitable for DB consumption. 4) Both Parse and Parsed are actually Monads ;) And we know something about Monads, is that composing different Monads together is usually not straightforward xD That's said, in your particular example of having an optional field, I would see the parser produce a Maybe<Something> (or Something? in Kotlin) Let me know if I got you right and what you think about it :)