9 ms·

Parse, Don't Validate (2019)

321 points by revskill 2y ago

brunooliv 2y ago
This post is bookmarked for me since this first time I read it and I occasionally come back to it, it’s a great one
PreInternet01 2y ago
(2019), but still good-ish advice. The pattern works like a charm in modern C# as well, and has nice space-saving effects too by allowing you to omit the explicit variable declaration: if(!Whatever.TryParse<Thingy>(input, out var output)) output = some-sane-default; or: if(!Whatever.TryParse<Thingy>(input, out var output)) throw new ApplicationException($"Not a valid Thingy: {input}"); Protip: don't do the latter in your kernel-mode driver.
- yakshaving_jgt 2y ago
  > (2019), but still good-ish advice Why only "good-ish"? And how does it relate to the year the article was published? Surely you are implying that the advice in the article would be more authoritative if it were published earlier than 2019, right?
  - HelloNurse 2y ago
    Aversion to truth seems more fashionable now than in 2019.
- zo1 2y ago
  Protip: Don't do either. And definitely don't do the first. Explicit is always better than implicit defaults that get used instead when you give it a wrong value that you think is correct. What you should do is throw your hands up early, fail to parse, and have a very clearly defined process and protocol to handle files that couldn't be loaded. It'll force you to ask yourself very difficult questions that aren't covered by either of the two options you posted. The real failure in the recent Crowdstrike kernel-mode driver failing to parse some def/config file is that the dev/product owner/BA didn't ask "what happens if we try load a file that's invalid?"
- Akronymus 2y ago
  if(!Whatever.TryParse<Thingy>(input, out var output)) output = some-sane-default; I absolutely hate that. IMO you should handle the error of an invalid input outside of the function to parse. F# makes that easy. type Whatever = static member create input = match input with | ValidWhatever x -> Some x | _ -> None match Whatever.create input with | Some x -> //process the parsed data | None -> //handle it not being parsed well Or you could also use Option.map/Option.bind to build a pipeline to handle chained operations in an ergonomic manner. With this, you can only instantiate any instances through the create method with parses the input. Altough, you probably want to use a result rather than option, but I digress.
- zeendo 2y ago
  Please don't do the first. Handle the bad cases. "sane default" fallback should be extremely rare. Explicit > Implicit
  - account42 2y ago
    As always, it depends. E.g. you don't want to refuse to open a document just because ten pages in there is a footnote that is somehow malformed. Depending on the possible consequences, GIGO is much better for usability. Ideally you'd warn the user that something unexpected has occured and he should manually check the result but whenever you are handling external data where malformed input cannot be avoided then erroring early will only make your software less useful to the user who more often than not cannot do anything about the error but can manually correct the fallback if you got it wrong.
- nucleardog 2y ago
  > if(!Whatever.TryParse<Thingy>(input, out var output)) output = some-sane-default; I can't think of many (probably any) situations where I'd want to find that. If _no_ input is provided (i.e., the parameter is optional), sure, using a sane default makes sense. If _invalid_ input is provided, for the love of god please don't pretend like nothing's wrong. If someone walks into a florist and asks for a coffee, the correct answer is not for them to be handed a rose. They're going to cut their mouth all up when they try and drink it. Your method/module/program does not have an output defined for that set of inputs. Make that obvious rather than just doing wrong or non-obvious things in a way that quickly makes your program almost impossible to reason about. Do yourself a favour and clearly raise the issue and leave yourself a stack trace pointing directly to the issue instead of setting yourself up for the vague bug about incorrect behaviour when someone catches this in a few months.
  - account42 2y ago
    If someone reserves a hotel room it makes sense to reserver any room if the desired selection is not valid. Of course you should notify the user of the error but preventing the process from continuing is often unhelpful. Similarly when converting a file you often should not abort the operation just because there is some minor detail in the original that you can't parse. The user is generally not in a position where they can do anything about such an error but they can often fix up the partially correct result. Of course you shouldn't do the fallback silently and at least notify the user that there was a problem.
teeheelol 2y ago
Forwarded to crowdstrike.
dgb23 2y ago
This is very good advice and a great article. It comes up on this site now and then because of it. For those who don't necessarily program in statically typed functional languages: The idea transcends paradigms. You'll find very similar notions in 80's/90's OO literature, for example in Design by Contract. I'm sure one can dig deeper and find papers, discussions and specifications that go further back. I think TypeScript is often written in such a way where you refine the types at runtime. I assume Design by Contract has influenced Clojure's spec (Clojure is a dynamic language). Fundamentally this is about assumptions and guarantees (or requiring and providing). Once an assumption is checked and guarantees can be made, then other parts of the program don't need to check overlapping assumptions again. In fact I think one of the most confusing things when you read code is seeing already guaranteed properties being checked again somewhere else. It makes code harder to reason about and improve.
- Sakos 2y ago
  I've been going through the comments on previous posts. I think one of the biggest issues with the article is the title. It seems to act as an anchor for a lot of people to the point where they'll argue against things that aren't in the article, just implied by the title (without context). So there'll be people arguing that she doesn't want to validate at all and just wants to parse, when really the article is about where you validate your data (and what you do with the result). It is not about getting rid of all validation.
  - blowski 2y ago
    This is true of all famous essays. People remove the nuance included in the body and over-apply the title without really understanding it. For example "Goto considered harmful". I remember working with a very good programmer who'd used a "goto", and a much less senior one[1] rejected their PR by linking to the article. [1] I'm ashamed to say it was me, a long time ago.
    - ffsm8 2y ago
      That's precisely why I love to only comment things with questions, i.e. "isn't go-to considered harmful? Is this really the best way?". It gives the person making the change the opportunity to give context for their decision - and either change or spell out their reasoning for it, potentially giving me an opportunity to learn from them. Another good benefit is that it doesn't "attack" the PR author in the "non-violent communication" way
WiSaGaN 2y ago
Utilize strong type system to make the error case unrepresentable. This is great advice to reduce bugs in software in general. It takes more time to think about the problem and to make a design following this. However, a lot of times it is worth the time.
- Smaug123 2y ago
  I'm going to make the bold claim that it doesn't take more time to do this, if your language supports algebraic data types. It just sort of naturally happens. Of course, if your language requires a great deal of ceremony (C++, Java, C#, Python, Golang, Javascript, …) to model data, then it will take more time.
zendist 2y ago
It seems that CloudStrike only parsed and didn't validate, to great effect :-) /s Not saying that this advice isn't solid, just thought it's funny given the news of this week.
- keybored 2y ago
  Parsing subsumes validation.
blowski 2y ago
Some good previous discussions on this: https://news.ycombinator.com/item?id=35053118 https://news.ycombinator.com/item?id=35053118 https://news.ycombinator.com/item?id=21476261 https://news.ycombinator.com/item?id=21476261
knallfrosch 2y ago
Great advice and works well in TypeScript as well. See https://www.typescriptlang.org/docs/handbook/advanced-types.html#using-type-predicates https://www.typescriptlang.org/docs/handbook/advanced-types.... More than merely catching compiler errors, this also makes writing code faster by offering better auto-completion for new code.
- tossandthrow 2y ago
  This is a super tool! But it still requires one to write good gates. For all projects of some. Size I would advice people to use Zod or the like (unless there are special circumstances such that external deps can not be used).
  - epolanski 2y ago
    And if you want more power at the type level, use effect/schema https://github.com/effect-ts/effect/tree/main/packages/schema https://github.com/effect-ts/effect/tree/main/packages/schem...
vladssw 2y ago
Related: "Making Impossible States Impossible" by Richard Feldman https://www.youtube.com/watch?v=IcgmSRJHu_8 https://www.youtube.com/watch?v=IcgmSRJHu_8
Vosporos 2y ago
A seminal text that has had a high cultural impact.
hatsuseno 2y ago
I feel like this idea is another form of, or at least related to, my own habit to process input in two phases, plan and execute. Run the input through a planner component that produces a sequential list of instruction that would 'do' whatever it is we're doing. I've changed this style to permit parallel execution or other more complicated structures than just a flat list, but I often come back to this base plan. Invalid input would be caught during the planning phase, and I need to ensure the planner can't make impossible plans up to an extent. Can't say I've avoided every type of problem or bug this way, but it sure as hell gives me a good base to work with.
sriram_malhar 2y ago
Can someone steeped in type theory explain the following: I never understood why the default for [a] means that the list could be empty. If ... foo: [a] -> a ... foo is supposed to get a list of a's, it should get a list of a's, with at least one a. If the list can be empty, then explicitly annotate it so: foo: [a*] -> a One way or the other one has to deal with the empty list explicitly (in the signature). If you allow empty lists, it will have to return a 'Maybe a'. It seems to me that it just makes processing the result easier in the common case if the input were to be constrained.
- noelwelsh 2y ago
  The definition of a list is that it contains zero or more elements. There's nothing deep about it; it's just the way the type is defined (and always has been, back to the early days of Lisp from which functional programming evolved.)
  - TeMPOraL 2y ago
    Conceptually, I see this as seeing a list as its own thing, independent of its contents. That is, a List<Int> is a value in itself, that can exist even if no instance of an Int exists anywhere in the program; it'll just be a List of Int that happens to have zero Ints in it. Probably makes more sense when you come from imperative programming background, where List<Int> is a piece of mutable state that you can construct and then fill, in two explicitly separate steps (and then possibly empty it again in yet another step). Then again, I believe even mathematicians are fine with ideas of an empty set, or of a one-element set being distinct from the element itself.
    - nyssos 2y ago
      > Then again, I believe even mathematicians are fine with ideas of an empty set, or of a one-element set being distinct from the element itself. Especially mathematicians. Distinguishing stuff and structure is a common theme throughout mathematics.
  - sriram_malhar 2y ago
mindesc 2y ago
yeah. use raw datatype url provided by the language and get hacked by some exotic xss you are not aware, because the specs have kitchen and sink included
- yakshaving_jgt 2y ago
  You can make your own types.
kgeist 2y ago
>Now I have a single, snappy slogan that encapsulates what type-driven design means to me, and better yet, it’s only three words long: >Parse, don’t validate. For me the slogan is rather "always validate only in the single constructor" (or constructor function, doesn't matter). That way, you cannot have invalid objects at all, and there's always a single source of truth. If you want to modify the object, implement it via constructing a new state by calling the same constructor again.
- epolanski 2y ago
  Not really the same thing. The point is that validation alone is then lost as information later on. E.g. validating an int to be positive has limited benefits if you don't parse it to be a positive int, because there's no such information at the type level later on, same could apply to a non empty array/list where following consumers would then need to check again if the list is really non empty. This kind of information cannot always be encoded in objects or constructors.
the_gipsy 2y ago
Sadly this doesn't work at all in go with the zero-values concept.
- Smaug123 2y ago
  I imagine it can be made to work, as long as you make sure the zero value is never actually valid; then you're back in the usual billion-dollar-mistake world where everything is implicitly optional, rather than in the hellscape where you can never know whether you've got something meaningful in your hand or not. (I realise that Golang strongly believes zero values are usually valid, but Golang is wrong about this.)
  - the_gipsy 2y ago
    The zero value is valid in many cases: false, 0, "", etc.
    - Smaug123 2y ago
      There's no such thing as a "valid" value. Values can only be valid in some context. The empty string is valid only in contexts where… it has meaning. (Quite hard to say this without being obviously circular!) Golang has chosen the point of view that "in most contexts, the zero value is valid"; it is wrong about this. There are certainly some contexts where for some particular type the zero value is valid, but it is rarely true in general. Booleans are harmful for other reasons, sometimes related to "parse, don't validate". Write a discriminated union that makes the true/false distinction meaningful, if you really do have a two-valued data type ("--dry-run" is `| Dry | Wet`, not a bool; this mistake is called "boolean blindness"). Write a type that actually contains the data you want, if the boolean is supposed to indicate the validity of some other data (`option<string>`, not `string * bool` where `something, false` implicitly means the `something` is meaningless).
      - the_gipsy 2y ago
        I agree, it seems I wasn't clear. The values false, zero, or empty string are "valid" in most contexts. But in go, using the stdlib json package, it is not possible to distinguish a missing value from the zero value. You can use pointers in some cases, but then everything becomes a full-blown pointer for the lack of an Option<T> type, and it still doesn't fully solve the problem. When parsing "enums" in go, which are really just a type alias and some loose constants, again it's not possible to prevent zero values sneaking in with the stdlib json package. E.g. you get a value of the correct type, but it's not one of your consts, but equivalent to a new value `YourType("")`. Thus, a lot of validation is necessary, between parsing and using values.
madduci 2y ago
I don't know why this is a good advice at all. If you have exposed APIs, you should prevent malicious payloads and what happens when the parser can be broken through invalid data, causing also Out of Memory exceptions? It might work only if you have some safe guardrails around the APIs, but just exposing naked endpoints, without a minimum of checks or a Web Application Firewall, this isn't a real good advice
- Smaug123 2y ago
  Obviously if you've written a buggy parser then all bets are off, just as all bets are off if you've written bugs into any other part of your program. But those bugs would probably still be there if you didn't explicitly parse the input - they'd just appear later on, mixed in with your business logic which is performing ad-hoc parses whenever it needs the data!
yakshaving_jgt 2y ago
One of my favourite articles published during my career. I've noticed that people often just read the title and assume parsing and validation are somehow mutually exclusive, but in practice that's not the case. Parsing often includes validation. This is addressed in the article, under Use abstract datatypes to make validators “look like” parsers. It's the same kind of ground as avoiding primitive obsession.
fragmede 2y ago
(2019)
keybored 2y ago
One of my favorite things that I’ve read via HN. It seems like this approach will often bottom out in smart constructors since type systems either are limited or make you work too hard to prove relatively simple thing.
valenterry 2y ago
> Is it possible to implement foo? Trivially, the answer is no, as Void is a type that contains no values, so it’s impossible for any function to produce a value of type Void That's actually not really correct. Or rather, it is technically correct but it will confuse the readers who work in languages like Java. While void in languages like Java means that the result of the function cannot be used or has no meaning, it is NOT equivalent to types like the bottom type of Haskell. Because that would mean that the function can never return. Rather, void is similar to the "unit type" (https://en.wikipedia.org/wiki/Unit_type https://en.wikipedia.org/wiki/Unit_type) which does have a value. It's like an empty tuple. It contains no information other then "the function call has finished". (and of course in languages with exceptions, this means that no exception was thrown) Otherwise, I like the article. More people should read and understand this way of thinking.
- thedataangel 2y ago
  `Void` is not equivalent to the unit type. Haskell _has_ a unit type, called `()`. A return value of `Void` represents a function which is unimplementable, or which never terminates. It can also represent an unreachable possibility, e.g. in the type `Either Void a`.
  - valenterry 2y ago
    Yeah - I just wanted to emphasize that this article will be confusing for most developers. Since most are used to void from languages like Java. And the meaning of that void is different from the one in Haskell.
deleted 2y ago
[deleted]
Garlef 2y ago
Hm... I quite like the idea but I think the initial example is not very good and also the remark about 'shotgun parsing' seems to blut some levels: The key idea seems to be that the border between the periphery/plumbing/deserialization code and the actual business logic should be as strict and direct and isolated as possible. Only pass objects/data/payloads to the business logic that have been fully ingested into the data model of the business logic. And keep the ingestion in one place. From this perspective, the section about "shotgun parsing" might give some people the wrong idea and derail some discussions: If it's an actual part of the business requirements that branching and validations need to happen (branching for example over the existence of an optional value), a superficial reading of the article might lead someone to incorrectly identify this as "shotgun parsing".
zigzag312 2y ago
Is this opposite to the following opinion? > "“required” keyword in Protocol Buffers turned out to be a horrible mistake" https://capnproto.org/faq.html#how-do-i-make-a-field-required-like-in-protocol-buffers https://capnproto.org/faq.html#how-do-i-make-a-field-require... Having both flexible, unvalidated parsing and validated parsing functions would probably be best IMHO.
- dgb23 2y ago
  The problem here as I understood it is more general and orthogonal to what the Parse, Don't Validate article is talking about. 1. A field being "required", is not a property of the field itself, but of the construct holding the field. JSON-Schema does this correctly, by letting you define an array of required fields on the schema of an object instead of it being a property of a specific field. 2. Consumers, not producers should decide what their assumptions are. Producers should decide what their guarantees are. 3. Everyone, including the parts in the middle (here it's a message bus) should only state assumptions that they actually need in order to function. Use different schemas for different assumptions (which is easier to do if the "required" assumption is a property of a construct and not a field). The example in the article you mentioned illustrates nicely how these three principles are broken by "required" (or maybe how Protocol Buffers are used in general).
  - zigzag312 2y ago
    > 2. Consumers, not producers should decide what their assumptions are. Producers should decide what their guarantees are. Issue with schema definition languages is that only one set classes is generated. Forcing consumers to create large amounts of mapping logic to transform input to objects with required assumptions. I like the idea of an array of required fields on the schema of an object. Using arrays of required fields, could be used to generate different set of classes. That would make writing near-duplicate classes of input objects and mapping logic unnecessary.
- lexicality 2y ago
  No, the idea is that your application level validation functions take in data, own it and return a validation error or new type that has invalid states removed. In this case you could make a wrapper function that accepts the raw binary data, passes it to Cap'n'Proto, validates the output (this field is actually required etc) and then returns it
swyx 2y ago
fwiw, Lexi works at Hasura, where the practical application of this principle is called the "PDV refactor" https://x.com/tanmaigo/status/1291710315223306243 https://x.com/tanmaigo/status/1291710315223306243 (from my own notes https://x.com/swyx/status/1548380295765733378 https://x.com/swyx/status/1548380295765733378)
TacticalCoder 2y ago
What is needed are canonical representation of data and then you must parse, re-encode, and verify that the re-encoded data matches bit for bit what was parsed. Not in unit tests but when the app is running: you take the data in, you parse it, you re-encode/re-serialize it/re-whatever it. If it's not matching the data that came in, the data that came in is rejected. And that should just be one of the steps taken to verify that the data looks legit.
railsguy2001 2y ago
[dead]
kayo_20211030 2y ago
Is Postel's law relevant here?
willsmith72 2y ago
this reminds me of a team i worked in which had very few unit tests, and compensated with in-depth complex type systems comprehensive test coverage meets the same goal, and TDD looks a lot like "type-driven design", only easier to read and maintain.
maw 2y ago
Whenever this comes up, I'm reminded of section 5 in https://cr.yp.to/qmail/guarantee.html https://cr.yp.to/qmail/guarantee.html which among other things says "Don't parse" and "there are two types of command interfaces in the world of computing: good interfaces and user interfaces". If I were were to teach a class about programming in the medium (as opposed to in the small or in the large), I think I'd assign my students an essay comparing and contrasting these suggestions. Each has something to teach us, and maybe they're not as contradictory as it may seem at first.
deleted 2y ago
[deleted]
andrewghull 2y ago
Here's the final example in TypeScript if you find that easier to read: type NonEmpty<T> = [T, ...T[]] const head = <T>(list: NonEmpty<T>) => list[0] function getConfigurationDirectories(): NonEmpty<string> { const configDirsString = process.env["CONFIG_DIRS"] const [firstDir, ...restDirs] = configDirsString.split(',') if (firstDir === undefined) throw Error("CONFIG_DIRS cannot be empty"); return [firstDir, ...restDirs]; } function main() { const configDirs = getConfigurationDirectories(); initializeCache(head(configDirs)) }
wormlord 2y ago
This idea + Domain modeling go together like peanut butter and chocolate. The idea that an invalid state in your Domain is unrepresentable takes so much work off the database and API, and makes programs so easy to test-- since an invalid Domain representation simply will not parse, and can never make it to your API or DB, taking so much logic out of the parts of a software stack that are hardest to test.
hintymad 2y ago
This reminds me of a comment someone made during the craze of XMLs in the mid 2000s. In the comment the author suspected that so many organizations chose XML to implement their domain-specific languages, configuration languages included, only because XML offers a parser, while most organizations didn't want to bother with writing their own parser. It beats me why people didn't want to write parsers, though. Writing parsers is not that hard, and is quite fun.

321 points by revskill 2y ago

This post is bookmarked for me since this first time I read it and I occasionally come back to it, it’s a great one

(2019), but still good-ish advice. The pattern works like a charm in modern C# as well, and has nice space-saving effects too by allowing you to omit the explicit variable declaration: if(!Whatever.TryParse<Thingy>(input, out var output)) output = some-sane-default; or: if(!Whatever.TryParse<Thingy>(input, out var output)) throw new ApplicationException($"Not a valid Thingy: {input}"); Protip: don't do the latter in your kernel-mode driver.

yakshaving_jgt 2y ago
> (2019), but still good-ish advice Why only "good-ish"? And how does it relate to the year the article was published? Surely you are implying that the advice in the article would be more authoritative if it were published earlier than 2019, right?
- HelloNurse 2y ago
  Aversion to truth seems more fashionable now than in 2019.
zo1 2y ago
Protip: Don't do either. And definitely don't do the first. Explicit is always better than implicit defaults that get used instead when you give it a wrong value that you think is correct. What you should do is throw your hands up early, fail to parse, and have a very clearly defined process and protocol to handle files that couldn't be loaded. It'll force you to ask yourself very difficult questions that aren't covered by either of the two options you posted. The real failure in the recent Crowdstrike kernel-mode driver failing to parse some def/config file is that the dev/product owner/BA didn't ask "what happens if we try load a file that's invalid?"
Akronymus 2y ago
if(!Whatever.TryParse<Thingy>(input, out var output)) output = some-sane-default; I absolutely hate that. IMO you should handle the error of an invalid input outside of the function to parse. F# makes that easy. type Whatever = static member create input = match input with | ValidWhatever x -> Some x | _ -> None match Whatever.create input with | Some x -> //process the parsed data | None -> //handle it not being parsed well Or you could also use Option.map/Option.bind to build a pipeline to handle chained operations in an ergonomic manner. With this, you can only instantiate any instances through the create method with parses the input. Altough, you probably want to use a result rather than option, but I digress.
zeendo 2y ago
Please don't do the first. Handle the bad cases. "sane default" fallback should be extremely rare. Explicit > Implicit
- account42 2y ago
  As always, it depends. E.g. you don't want to refuse to open a document just because ten pages in there is a footnote that is somehow malformed. Depending on the possible consequences, GIGO is much better for usability. Ideally you'd warn the user that something unexpected has occured and he should manually check the result but whenever you are handling external data where malformed input cannot be avoided then erroring early will only make your software less useful to the user who more often than not cannot do anything about the error but can manually correct the fallback if you got it wrong.
nucleardog 2y ago
> if(!Whatever.TryParse<Thingy>(input, out var output)) output = some-sane-default; I can't think of many (probably any) situations where I'd want to find that. If _no_ input is provided (i.e., the parameter is optional), sure, using a sane default makes sense. If _invalid_ input is provided, for the love of god please don't pretend like nothing's wrong. If someone walks into a florist and asks for a coffee, the correct answer is not for them to be handed a rose. They're going to cut their mouth all up when they try and drink it. Your method/module/program does not have an output defined for that set of inputs. Make that obvious rather than just doing wrong or non-obvious things in a way that quickly makes your program almost impossible to reason about. Do yourself a favour and clearly raise the issue and leave yourself a stack trace pointing directly to the issue instead of setting yourself up for the vague bug about incorrect behaviour when someone catches this in a few months.
- account42 2y ago
  If someone reserves a hotel room it makes sense to reserver any room if the desired selection is not valid. Of course you should notify the user of the error but preventing the process from continuing is often unhelpful. Similarly when converting a file you often should not abort the operation just because there is some minor detail in the original that you can't parse. The user is generally not in a position where they can do anything about such an error but they can often fix up the partially correct result. Of course you shouldn't do the fallback silently and at least notify the user that there was a problem.

teeheelol 2y ago

Forwarded to crowdstrike.

dgb23 2y ago

This is very good advice and a great article. It comes up on this site now and then because of it. For those who don't necessarily program in statically typed functional languages: The idea transcends paradigms. You'll find very similar notions in 80's/90's OO literature, for example in Design by Contract. I'm sure one can dig deeper and find papers, discussions and specifications that go further back. I think TypeScript is often written in such a way where you refine the types at runtime. I assume Design by Contract has influenced Clojure's spec (Clojure is a dynamic language). Fundamentally this is about assumptions and guarantees (or requiring and providing). Once an assumption is checked and guarantees can be made, then other parts of the program don't need to check overlapping assumptions again. In fact I think one of the most confusing things when you read code is seeing already guaranteed properties being checked again somewhere else. It makes code harder to reason about and improve.

Sakos 2y ago
I've been going through the comments on previous posts. I think one of the biggest issues with the article is the title. It seems to act as an anchor for a lot of people to the point where they'll argue against things that aren't in the article, just implied by the title (without context). So there'll be people arguing that she doesn't want to validate at all and just wants to parse, when really the article is about where you validate your data (and what you do with the result). It is not about getting rid of all validation.
- blowski 2y ago
  This is true of all famous essays. People remove the nuance included in the body and over-apply the title without really understanding it. For example "Goto considered harmful". I remember working with a very good programmer who'd used a "goto", and a much less senior one[1] rejected their PR by linking to the article. [1] I'm ashamed to say it was me, a long time ago.
  - ffsm8 2y ago
    That's precisely why I love to only comment things with questions, i.e. "isn't go-to considered harmful? Is this really the best way?". It gives the person making the change the opportunity to give context for their decision - and either change or spell out their reasoning for it, potentially giving me an opportunity to learn from them. Another good benefit is that it doesn't "attack" the PR author in the "non-violent communication" way

WiSaGaN 2y ago

Utilize strong type system to make the error case unrepresentable. This is great advice to reduce bugs in software in general. It takes more time to think about the problem and to make a design following this. However, a lot of times it is worth the time.

Smaug123 2y ago
I'm going to make the bold claim that it doesn't take more time to do this, if your language supports algebraic data types. It just sort of naturally happens. Of course, if your language requires a great deal of ceremony (C++, Java, C#, Python, Golang, Javascript, …) to model data, then it will take more time.

zendist 2y ago

It seems that CloudStrike only parsed and didn't validate, to great effect :-) /s Not saying that this advice isn't solid, just thought it's funny given the news of this week.

keybored 2y ago
Parsing subsumes validation.

blowski 2y ago

Some good previous discussions on this: https://news.ycombinator.com/item?id=35053118 https://news.ycombinator.com/item?id=35053118 https://news.ycombinator.com/item?id=21476261 https://news.ycombinator.com/item?id=21476261

knallfrosch 2y ago

Great advice and works well in TypeScript as well. See https://www.typescriptlang.org/docs/handbook/advanced-types.html#using-type-predicates https://www.typescriptlang.org/docs/handbook/advanced-types.... More than merely catching compiler errors, this also makes writing code faster by offering better auto-completion for new code.

tossandthrow 2y ago
This is a super tool! But it still requires one to write good gates. For all projects of some. Size I would advice people to use Zod or the like (unless there are special circumstances such that external deps can not be used).
- epolanski 2y ago
  And if you want more power at the type level, use effect/schema https://github.com/effect-ts/effect/tree/main/packages/schema https://github.com/effect-ts/effect/tree/main/packages/schem...

vladssw 2y ago

Related: "Making Impossible States Impossible" by Richard Feldman https://www.youtube.com/watch?v=IcgmSRJHu_8 https://www.youtube.com/watch?v=IcgmSRJHu_8

Vosporos 2y ago

A seminal text that has had a high cultural impact.

hatsuseno 2y ago

I feel like this idea is another form of, or at least related to, my own habit to process input in two phases, plan and execute. Run the input through a planner component that produces a sequential list of instruction that would 'do' whatever it is we're doing. I've changed this style to permit parallel execution or other more complicated structures than just a flat list, but I often come back to this base plan. Invalid input would be caught during the planning phase, and I need to ensure the planner can't make impossible plans up to an extent. Can't say I've avoided every type of problem or bug this way, but it sure as hell gives me a good base to work with.

sriram_malhar 2y ago

Can someone steeped in type theory explain the following: I never understood why the default for [a] means that the list could be empty. If ... foo: [a] -> a ... foo is supposed to get a list of a's, it should get a list of a's, with at least one a. If the list can be empty, then explicitly annotate it so: foo: [a*] -> a One way or the other one has to deal with the empty list explicitly (in the signature). If you allow empty lists, it will have to return a 'Maybe a'. It seems to me that it just makes processing the result easier in the common case if the input were to be constrained.

noelwelsh 2y ago
The definition of a list is that it contains zero or more elements. There's nothing deep about it; it's just the way the type is defined (and always has been, back to the early days of Lisp from which functional programming evolved.)
- TeMPOraL 2y ago
  Conceptually, I see this as seeing a list as its own thing, independent of its contents. That is, a List<Int> is a value in itself, that can exist even if no instance of an Int exists anywhere in the program; it'll just be a List of Int that happens to have zero Ints in it. Probably makes more sense when you come from imperative programming background, where List<Int> is a piece of mutable state that you can construct and then fill, in two explicitly separate steps (and then possibly empty it again in yet another step). Then again, I believe even mathematicians are fine with ideas of an empty set, or of a one-element set being distinct from the element itself.
  - nyssos 2y ago
    > Then again, I believe even mathematicians are fine with ideas of an empty set, or of a one-element set being distinct from the element itself. Especially mathematicians. Distinguishing stuff and structure is a common theme throughout mathematics.
- sriram_malhar 2y ago

mindesc 2y ago

yeah. use raw datatype url provided by the language and get hacked by some exotic xss you are not aware, because the specs have kitchen and sink included

yakshaving_jgt 2y ago
You can make your own types.

kgeist 2y ago

>Now I have a single, snappy slogan that encapsulates what type-driven design means to me, and better yet, it’s only three words long: >Parse, don’t validate. For me the slogan is rather "always validate only in the single constructor" (or constructor function, doesn't matter). That way, you cannot have invalid objects at all, and there's always a single source of truth. If you want to modify the object, implement it via constructing a new state by calling the same constructor again.

epolanski 2y ago
Not really the same thing. The point is that validation alone is then lost as information later on. E.g. validating an int to be positive has limited benefits if you don't parse it to be a positive int, because there's no such information at the type level later on, same could apply to a non empty array/list where following consumers would then need to check again if the list is really non empty. This kind of information cannot always be encoded in objects or constructors.

the_gipsy 2y ago

Sadly this doesn't work at all in go with the zero-values concept.

Smaug123 2y ago
I imagine it can be made to work, as long as you make sure the zero value is never actually valid; then you're back in the usual billion-dollar-mistake world where everything is implicitly optional, rather than in the hellscape where you can never know whether you've got something meaningful in your hand or not. (I realise that Golang strongly believes zero values are usually valid, but Golang is wrong about this.)
- the_gipsy 2y ago
  The zero value is valid in many cases: false, 0, "", etc.
  - Smaug123 2y ago
    There's no such thing as a "valid" value. Values can only be valid in some context. The empty string is valid only in contexts where… it has meaning. (Quite hard to say this without being obviously circular!) Golang has chosen the point of view that "in most contexts, the zero value is valid"; it is wrong about this. There are certainly some contexts where for some particular type the zero value is valid, but it is rarely true in general. Booleans are harmful for other reasons, sometimes related to "parse, don't validate". Write a discriminated union that makes the true/false distinction meaningful, if you really do have a two-valued data type ("--dry-run" is `| Dry | Wet`, not a bool; this mistake is called "boolean blindness"). Write a type that actually contains the data you want, if the boolean is supposed to indicate the validity of some other data (`option<string>`, not `string * bool` where `something, false` implicitly means the `something` is meaningless).
    - the_gipsy 2y ago
      I agree, it seems I wasn't clear. The values false, zero, or empty string are "valid" in most contexts. But in go, using the stdlib json package, it is not possible to distinguish a missing value from the zero value. You can use pointers in some cases, but then everything becomes a full-blown pointer for the lack of an Option<T> type, and it still doesn't fully solve the problem. When parsing "enums" in go, which are really just a type alias and some loose constants, again it's not possible to prevent zero values sneaking in with the stdlib json package. E.g. you get a value of the correct type, but it's not one of your consts, but equivalent to a new value `YourType("")`. Thus, a lot of validation is necessary, between parsing and using values.

madduci 2y ago

I don't know why this is a good advice at all. If you have exposed APIs, you should prevent malicious payloads and what happens when the parser can be broken through invalid data, causing also Out of Memory exceptions? It might work only if you have some safe guardrails around the APIs, but just exposing naked endpoints, without a minimum of checks or a Web Application Firewall, this isn't a real good advice

Smaug123 2y ago
Obviously if you've written a buggy parser then all bets are off, just as all bets are off if you've written bugs into any other part of your program. But those bugs would probably still be there if you didn't explicitly parse the input - they'd just appear later on, mixed in with your business logic which is performing ad-hoc parses whenever it needs the data!

yakshaving_jgt 2y ago

One of my favourite articles published during my career. I've noticed that people often just read the title and assume parsing and validation are somehow mutually exclusive, but in practice that's not the case. Parsing often includes validation. This is addressed in the article, under Use abstract datatypes to make validators “look like” parsers. It's the same kind of ground as avoiding primitive obsession.

fragmede 2y ago

(2019)

keybored 2y ago

One of my favorite things that I’ve read via HN. It seems like this approach will often bottom out in smart constructors since type systems either are limited or make you work too hard to prove relatively simple thing.

valenterry 2y ago

> Is it possible to implement foo? Trivially, the answer is no, as Void is a type that contains no values, so it’s impossible for any function to produce a value of type Void That's actually not really correct. Or rather, it is technically correct but it will confuse the readers who work in languages like Java. While void in languages like Java means that the result of the function cannot be used or has no meaning, it is NOT equivalent to types like the bottom type of Haskell. Because that would mean that the function can never return. Rather, void is similar to the "unit type" (https://en.wikipedia.org/wiki/Unit_type https://en.wikipedia.org/wiki/Unit_type) which does have a value. It's like an empty tuple. It contains no information other then "the function call has finished". (and of course in languages with exceptions, this means that no exception was thrown) Otherwise, I like the article. More people should read and understand this way of thinking.

thedataangel 2y ago
`Void` is not equivalent to the unit type. Haskell _has_ a unit type, called `()`. A return value of `Void` represents a function which is unimplementable, or which never terminates. It can also represent an unreachable possibility, e.g. in the type `Either Void a`.
- valenterry 2y ago
  Yeah - I just wanted to emphasize that this article will be confusing for most developers. Since most are used to void from languages like Java. And the meaning of that void is different from the one in Haskell.

deleted 2y ago

[deleted]

Garlef 2y ago

Hm... I quite like the idea but I think the initial example is not very good and also the remark about 'shotgun parsing' seems to blut some levels: The key idea seems to be that the border between the periphery/plumbing/deserialization code and the actual business logic should be as strict and direct and isolated as possible. Only pass objects/data/payloads to the business logic that have been fully ingested into the data model of the business logic. And keep the ingestion in one place. From this perspective, the section about "shotgun parsing" might give some people the wrong idea and derail some discussions: If it's an actual part of the business requirements that branching and validations need to happen (branching for example over the existence of an optional value), a superficial reading of the article might lead someone to incorrectly identify this as "shotgun parsing".

zigzag312 2y ago

Is this opposite to the following opinion? > "“required” keyword in Protocol Buffers turned out to be a horrible mistake" https://capnproto.org/faq.html#how-do-i-make-a-field-required-like-in-protocol-buffers https://capnproto.org/faq.html#how-do-i-make-a-field-require... Having both flexible, unvalidated parsing and validated parsing functions would probably be best IMHO.

dgb23 2y ago
The problem here as I understood it is more general and orthogonal to what the Parse, Don't Validate article is talking about. 1. A field being "required", is not a property of the field itself, but of the construct holding the field. JSON-Schema does this correctly, by letting you define an array of required fields on the schema of an object instead of it being a property of a specific field. 2. Consumers, not producers should decide what their assumptions are. Producers should decide what their guarantees are. 3. Everyone, including the parts in the middle (here it's a message bus) should only state assumptions that they actually need in order to function. Use different schemas for different assumptions (which is easier to do if the "required" assumption is a property of a construct and not a field). The example in the article you mentioned illustrates nicely how these three principles are broken by "required" (or maybe how Protocol Buffers are used in general).
- zigzag312 2y ago
  > 2. Consumers, not producers should decide what their assumptions are. Producers should decide what their guarantees are. Issue with schema definition languages is that only one set classes is generated. Forcing consumers to create large amounts of mapping logic to transform input to objects with required assumptions. I like the idea of an array of required fields on the schema of an object. Using arrays of required fields, could be used to generate different set of classes. That would make writing near-duplicate classes of input objects and mapping logic unnecessary.
lexicality 2y ago
No, the idea is that your application level validation functions take in data, own it and return a validation error or new type that has invalid states removed. In this case you could make a wrapper function that accepts the raw binary data, passes it to Cap'n'Proto, validates the output (this field is actually required etc) and then returns it

swyx 2y ago

fwiw, Lexi works at Hasura, where the practical application of this principle is called the "PDV refactor" https://x.com/tanmaigo/status/1291710315223306243 https://x.com/tanmaigo/status/1291710315223306243 (from my own notes https://x.com/swyx/status/1548380295765733378 https://x.com/swyx/status/1548380295765733378)

TacticalCoder 2y ago

What is needed are canonical representation of data and then you must parse, re-encode, and verify that the re-encoded data matches bit for bit what was parsed. Not in unit tests but when the app is running: you take the data in, you parse it, you re-encode/re-serialize it/re-whatever it. If it's not matching the data that came in, the data that came in is rejected. And that should just be one of the steps taken to verify that the data looks legit.

railsguy2001 2y ago

[dead]

kayo_20211030 2y ago

Is Postel's law relevant here?

willsmith72 2y ago

this reminds me of a team i worked in which had very few unit tests, and compensated with in-depth complex type systems comprehensive test coverage meets the same goal, and TDD looks a lot like "type-driven design", only easier to read and maintain.

maw 2y ago

Whenever this comes up, I'm reminded of section 5 in https://cr.yp.to/qmail/guarantee.html https://cr.yp.to/qmail/guarantee.html which among other things says "Don't parse" and "there are two types of command interfaces in the world of computing: good interfaces and user interfaces". If I were were to teach a class about programming in the medium (as opposed to in the small or in the large), I think I'd assign my students an essay comparing and contrasting these suggestions. Each has something to teach us, and maybe they're not as contradictory as it may seem at first.

deleted 2y ago

[deleted]

andrewghull 2y ago

Here's the final example in TypeScript if you find that easier to read: type NonEmpty<T> = [T, ...T[]] const head = <T>(list: NonEmpty<T>) => list[0] function getConfigurationDirectories(): NonEmpty<string> { const configDirsString = process.env["CONFIG_DIRS"] const [firstDir, ...restDirs] = configDirsString.split(',') if (firstDir === undefined) throw Error("CONFIG_DIRS cannot be empty"); return [firstDir, ...restDirs]; } function main() { const configDirs = getConfigurationDirectories(); initializeCache(head(configDirs)) }

wormlord 2y ago

This idea + Domain modeling go together like peanut butter and chocolate. The idea that an invalid state in your Domain is unrepresentable takes so much work off the database and API, and makes programs so easy to test-- since an invalid Domain representation simply will not parse, and can never make it to your API or DB, taking so much logic out of the parts of a software stack that are hardest to test.

hintymad 2y ago

This reminds me of a comment someone made during the craze of XMLs in the mid 2000s. In the comment the author suspected that so many organizations chose XML to implement their domain-specific languages, configuration languages included, only because XML offers a parser, while most organizations didn't want to bother with writing their own parser. It beats me why people didn't want to write parsers, though. Writing parsers is not that hard, and is quite fun.