5 ms·

Parse, Don’t Validate – Some C Safety Tips

130 points by lelanthran 1y ago

pianoben 1y ago
The trouble I have with this approach (which, conceptually, I agree with) is that it's damned hard to do anything with the parse results. Want to print that email_t? Then you're right back to char*, unless you somehow write your own I/O system that knows about your opaque conventions. So you say, okay, I'll make an `email_to_string` function. Does it return a copy or a reference? Who frees it? etc, etc, and you're back to square one again. The idea is to keep char* and friends at "the edge", but I've never found a way to really achieve that. Could just be my limitations as a C programmer, in which case I'd be thrilled to learn better.
- dwattttt 1y ago
  email_t doesn't have to be opaque; if it's just a visible wrapper around char* then you can still do everything with it as a char* (that is, everything you do with strings). The benefit is to avoid treating char*s as email_t, not avoiding treating email_t as char*.
  - bcrosby95 1y ago
    In the example code they explicitly put the struct in the c file so the char* is not available. If you're suggesting getting around this by casting an email_t* to char* then I wish you good luck on your adventures. There's some times you gotta do stuff like that but this ain't it.
    - dwattttt 1y ago
      You could probably get away with the typecast if you satisfy the "common struct prefix" requirement, that's nowhere near necessary. While the article does hide the internal char*, that's not strictly necessary to get the benefit of "parse, don't validate". Hide implementation details sure, but not everything is an implementation detail.
  - maxbond 1y ago
    (Using a thin wrapper like this to add safety is called the newtype pattern, if anyone wants to know.)
    - tetha 1y ago
      I was curious how this would look in C, and I found this article[1] how this could look in C, apparently with very little overhead. And as I just saw, Python 3.10 also introduced a NewType[2] wrapper. I'll have to see how that feels to handle. 1: https://blog.nelhage.com/2010/10/using-haskells-newtype-in-c/ https://blog.nelhage.com/2010/10/using-haskells-newtype-in-c... 2: https://typing.python.org/en/latest/spec/aliases.html#newtype https://typing.python.org/en/latest/spec/aliases.html#newtyp...
      - masklinn 1y ago
        Python’s NewType is, confusingly, a very different thing: it’s a compile-time-only subtype of the original, rather than a Haskell-style newtype (which is an entirely separate type from its source).
    - mrkeen 1y ago
      I've re-read the article again since getting a bunch of up and down votes across the comment section, and I think you've chosen a better name for this article than PdV. It really is just about using newtype wrappers.
- lelanthran 1y ago
  Firstly, `parsing` is just a way to say "serialise from a string". The reverse operation can be done for every type you are creating. If the reverse operation (serialise to a string) does not exist in the interface then adding it gives you a single place to catch all the bugs. I'm thinking of that recent git bug that occurred because the round-trip of `string -> type -> string` had an error (stripping out the CR character). Using a specific type for a value that is being round-tripped means that a bugfix needs to only be made in the parser function. Storing the value as simple strings would result in needing to put your fix everywhere. > The trouble I have with this approach (which, conceptually, I agree with) is that it's damned hard to do anything with the parse results. You're right - it is damn hard, but that is on purpose; if you're doing something with the email that boils down to "treat it like a `char *`" then the potential for error is large. If you're forced to add in a new use-case to the `email_t` interface then you have reduced the space of potential errors. For example: > Want to print that email_t? Then you're right back to char, unless you somehow write your own I/O system that knows about your opaque conventions. is a bug waiting to surface, because it's an email, not a string, and if you decide to print an email* that was read as a `char *` you might not get what you expect. It's all a trade-off - if you want more flexibility with the value stored in a variable, then sure, you can have it but it comes at a cost: some code somewhere almost certainly will eventually use that flexibility to mismatch the type! If you want to prevent type mismatches, then a lot of flexibility goes out the window.
  - jagged-chisel 1y ago
    Linguistic nit: deserialize from a string, serialize to a string “Serialization” is the act of taking an internal data structure (of whatever shape and depth) and outputting it for transmission or storage. The opposite is “deserialization,” restoring the original shape and depth.
  - KerrAvon 1y ago
    But that’s where TFA breaks down: the whole point of this is to claim “hey, C does too have typesafety” —if you use a modern language with actual typesafety, you can just make the underlying representation accessible (probably read-only) and you don’t need a reverse conversion step. You can have type safety, efficiency, and ease of use. Just rip the band-aid off and move to Rust or Swift already.
- 8organicbits 1y ago
  In the past I've taken inspiration from strncpy: the caller needs to allocate the memory. For the email example, you'd probably also want a function to tell you the length of the emailstring, but for other types there are clear size limits. This puts the caller in control of memory allocation, so they may be able to statically allocate, allocate in an arena, or use other methods which promote performance. The static approach is really nice when it works, because there's nothing to free.
- restalis 1y ago
  The main benefit for me with this approach is that the boundries are not transparent anymore. That content printing is such a boundry. Your data is about to exit through there and you're summoned to handle that. The inconvenience that comes with it is as any other when security enters the play. The same with the data management responsibilities - who handles what, for how long, and with whom. Without data type distinctions everything is (more or less) common, with vague or broadly defined ownership.
o11c 1y ago
`_t` should not be used for custom types since it's reserved for future standard types (and/or types declared in a header you might include someday). This does cause real-world problems (`key_t` anyone?). Gratuitous allocations are gratuitous. The whole "prevent double free" claim is completely bogus. Setting a variable to `NULL` only works for cases where there is one, obvious, owner, which is not the circumstance under which double free is prone to happening in the first place. Actually preventing double free requires determining ownership of every object, and C sucks at that.
- lelanthran 1y ago
  > The whole "prevent double free" claim is completely bogus. "Completely" means "for all". Are you seriously claiming that "for all instances of double-free, setting the pointer to NULL after freeing it would not help"?
  - lisper 1y ago
    > "Completely" means "for all". Not in the case of bogosity. Completely bogus things might occasionally work under some very particular circumstances, but unless those particular circumstances just happen to be the circumstances you actually care about, complete bogosity can still obtain. > setting the pointer to NULL There is no such thing as setting a pointer to null. You can set the value of a variable (whose current value is a pointer) to null, but you cannot guarantee that there isn't a copy of the pointer stored somewhere else except in a few very particular circumstances. This is what the GP meant by "setting a variable to `NULL` only works for cases where there is one, obvious, owner". And, as the GP also pointed out, this "is not the circumstance under which double free is prone to happening in the first place." Hence: complete bogosity.
  - taneq 1y ago
    Eeeeh, I don't think 'completely bogus' means 'exhaustively false for all situations'. It just means 'demonstrably false' (for some relatively sane example, we're talking about C after all which means there will always be bogus examples which break any given assumption). There's plenty of cases where zeroing a pointer immediately after freeing it will prevent any further issues. It's still bogus to claim that it categorically solves the problem of double frees. But it does help.
mrkeen 1y ago
This stuck out: email_t theEmail = parseEmail(untrustedInput); if (theEmail == PARSE_ERROR) { return error; } An email_t is not a parse error, and a parse error is not one of the emails, so this shouldn't compile (and I don't take 'pseudocode' as an excuse).
- parkcedar 1y ago
  > and I don't take 'pseudocode' as an excuse Weird hill to die on, since neither email_t nor PARSE_ERROR were defined in the sample snippets. How do you know PARSE_ERROR is not email_t?
  - VMG 1y ago
    Because an error is not an email?
    - howaboutno2312 1y ago
      [dead]
    - exe34 1y ago
      By that logic, a float couldn't store NaN.
      - mrkeen 1y ago
        Correct. You'll never see a raw float on the LHS of a PDV expression.
        exe34 1y ago
        Who puts their personal data vault on the left hand side of a raw float?
  - mrkeen 1y ago
    It's the parse-versus-validate hill in this case. This pseudocode is "Validate" for at least 3 reasons: Forgetting to check: this check is fragile: it’s extremely easy to forget. Because its return value is unused, it can always be omitted, and the code that needs it would still typecheck. Repeatable/redundant checks: First, it’s just annoying. We already checked that the list is non-empty, why do we have to clutter our code with another redundant check? Second, it has a potential performance cost. Although the cost of the redundant check is trivial in this particular example, one could imagine a more complex scenario where the redundant checks could add up, such as if they were happening in a tight loop. Not using the type system: Use a data structure that makes illegal states unrepresentable. Model your data using the most precise data structure you reasonably can. If ruling out a particular possibility is too hard using the encoding you are currently using, consider alternate encodings that can express the property you care about more easily. Don’t be afraid to refactor. > How do you know PARSE_ERROR is not email_t It has to be for it to compile, right? Which means that email_t is the type which represents both valid and invalid emails. How do you know if it's valid? You remember to write a check for it. Why not just save yourself some keystrokes and use char* instead. This is validate, not parse.
deleted 1y ago
[deleted]
anon-3988 1y ago
> Here, I’ll build on that by showing how this technique can be used outside of niche academic languages by demonstrating it in a language that is as practical as it is dangerous - C. The "practical" part really bugged me because the entire post is trying to explain exactly why it is not. The only way to make C reasonably safe is to encode information via newtype pattern. Wrap `char *` inside a struct that have proper names and include the size in there as well. Basically, there should be ZERO pointers except at creation and consumption by outside libraries (open, write, etc)
jkuli 1y ago
validating twice is safe. it will not error the second time. it is against the rules to call someone dumb on this server.
- jagged-chisel 1y ago
  Comments, however … ;-)
hackrmn 1y ago
I've been shouting this from "the rooftops" ever since I eventually independently had come to the same conclusion through sufficient volume of experience and empiry. I consider it a small tragedy, however, that my words tend to fall on deaf ears as I try to explain to e.g. co-workers why they should, in fact, parse their input and use the language's type system to their advantage, vs. all the `validateThis` and `validateThat` they seem to be copying from e.g. Stack Overflow. I see them stumbling and breaking their nose time and again passing URIs around as strings -- even in languages other than C (so this isn't obviously a problem C created and needs to solve alone) -- where it's never enforced that the URI is e.g. "absolute" or "relative" (nevermind the fact these people often don't seem to care for the difference). They do the same error with date and time (how many decades of advice do we need to accumulate and impart on our younger peers for them to irrevocably understand why e.g. date and time should never be passed around as numbers or strings, with very clearly documented and justified exceptions?). But the tragedy is never more evident when the person looks at you like you're suddenly talking Klingon to them and how bored they are with the notion and how it cannot be so important as to disturb their very important "coding project" (we don't write Knuth's "literal" programs any more -- we _code_).
pixelpoet 1y ago
Such a recognisable name, greetz after many years :D You might remember me from uni as lycium, a few lifetimes ago. Going to check the rest of your blog now...
- lelanthran 1y ago
  > Such a recognisable name, greetz after many years :D You might remember me from uni as lycium, a few lifetimes ago. Going to check the rest of your blog now... I do remember you? You had/joined a sartup in NZ, correct? Warm regards Lycium :-)
  - pixelpoet 1y ago
    Thanks :D Yeah, we had a good time for ~10 years: https://glaretechnologies.com https://glaretechnologies.com I saw you're doing consultancy last years, cool :) Lemme know if you wanna chat on Discord or so, cheers!
    - lelanthran 1y ago
      > Thanks :D Yeah, we had a good time for ~10 years: https://glaretechnologies.com https://glaretechnologies.com > I saw you're doing consultancy last years, cool :) Lemme know if you wanna chat on Discord or so, cheers! I do wanna chat and perhaps catchup :-) Although I have a discord account, I don't remember ever using it.
hansvm 1y ago
One flaw I've seen in "Parse, Don't Validate" as it pertains to real codebases is that you end up with a combinatorial prolieration of types. E.g., requiring that a string be base64, have a certain fixed length, and be provided by the user. E.g., requiring that a file have the correct MIME type, not be too large, and contain no EXIF metadata. If you really always need all n of those things then life isn't terrible (you can parse your data into some type representing the composition of all of them), but you often only need 1, 2, or 3 and simultaneously don't want to duplicate too much code or runtime work, leading to a combinatorial explosion of intermediate types and parsing code. As one possible solution, I put together a POC in Zig [0] with one idea, where you abuse comptime to add arbitrary tagging to types, treating a type as valid if it has the subset of tags you care about. I'm very curious what other people do to appropriately model that sort of thing though. [0] https://github.com/hmusgrave/pdv https://github.com/hmusgrave/pdv
- jeremyscanvic 1y ago
  You might be interested in Lean's way of doing things. They have normal types (e.g. numeric types) and subtypes (e.g. numbers less than zero). An element of the subtype "numbers less than zero" can be understood as a tuple containing the actual number (which has a normal numeric type) and a proof that this specific number is indeed less than zero. https://lean-lang.org/doc/reference/latest/Basic-Types/Subtypes/ https://lean-lang.org/doc/reference/latest/Basic-Types/Subty...
- tacitusarc 1y ago
  Structs are representations of combinatorial types! In your file case, you could parse the input into a struct, and then accept or reject further processing based on that contents of that struct. Of course, it would be reasonable to claim that the accept/reject step is validation, but I believe “Parse, don’t validate” is about handling input, not an admonition to never perform validation.
  - mrkeen 1y ago
    In pure C however, you still get the types-in-source-code explosion, for lack of parametric polymorphism. You need an email_or_error and a name_or_error, etc. The alternative is to fake PP with a void*, but that's so ugly I think I'd scrap the whole effort and just use char*. > I believe “Parse, don’t validate” is about handling input, not an admonition to never perform validation. It's about validation happening at exactly one place in the code base (during the "parse" - even though it's not limited to string-processing), so that callers can't do the validation themselves - because callers will validate 0 times or n>1 times.
myaccountonhn 1y ago
From experience, parsing input into data structures that fit the problem domain once at the "edge" is a good idea. The code becomes a lot more maintainable without a bunch of validation checks scattered all over the place, picking a data structure for the problem at hand usually leads to cleaner solutions, and errors usually show up much earlier and are easier to debug. From experience though I've found that wrapping all data in newtypes adds too much ceremony and boilerplate. If the data can reasonably be expressed as a primitive type, then you might as well express that way. I can't think of a time where newtype wrapping would have saved me from accidentally not validating or accidentally inputting the wrong data as a parameter. Especially the email example is quite weak, with ~30 lines of code just being ceremony due to wrapping a string, and most likely it's just going to be fed as is to various crud operations that will cast the data to a string immediately. Interacting with Haskell/elm libraries that have pervasive use of newtypes everywhere can be painful, especially if they don't give you a way to access the internal data. If a use-case comes up that the library developer didn't account for, then you might have no way of modifying the data and you end up needing to patch the library upstream.
- movpasd 1y ago
  I think it can be useful to think of the parsing and logic parts both as modules, with the parsing part interfacing with the outside world via unstructured data, and the parsing and logic parts interfacing via structured data, i.e.: the validated types. From that perspective, there is a clear trade-off on the size of the parsing–logic interface. Introducing more granular, safer validated types may give you better functionality, but it forces you to expand that interface and create coupling. I think there is a middle ground, which is that these safe types should be chunked into larger structures that enforce a range of related invariants and hopefully have some kind of domain meaning. That way, you shrink the conceptual surface area of the interface so that working with it is less painful.
belter 1y ago
This pattern seems to be shown almost as a comprehensive security solution when it's really just one layer of defense. This parse dont validate has to be combined with resource limits, and other protective measures during the parsing phase itself.
charcircuit 1y ago
I don't trust C developers to parse strings.
rorylaitila 1y ago
While this is the style I do in all of my (web) applications, it took a while to build my framework/tooling that overcomes the boilerplate to parse all incoming data into value objects. In a typical application of mine, there is usually well over dozens of entities and hundreds to thousands of fields. Creating value objects manually for all stringly types and numeric fields is a lot of work. Most value objects are some simple validation like limit to 255 characters or ints of some range. I solve for this by using reflection and auto generating all value objects (inheriting by default from base types) and auto generating all accessor/controller classes or methods into the domain model. Therefore I model in base types, override the generated value object constructors for validation (if required), and all of the boundaries are using value objects. The internal code generally works with the underlying base types, because boxing/unboxing the value objects can be non-negligible performance impact when serializing a lot of data (which tends to be common in web applications... SQL > JSON > HTML). I'm a huge fan, but I think ymmv. Web applications tend to have a wide interface (much of the domain model is user accessible). I think it's ideal for this case because of the number of fields a user can ultimately set and reused across many places.
OptionOfT 1y ago
In general I'm a huge favor of using the type system to its fullest extend to protect you from messing up. If the compiler can protect me, it should. Examples where I've used it in the past: ValidatedEmail, which is a special form of Email, one that has been validated by the user. We can have actions that require a `PriviligedUser`, which can be created from a `User`. That creation validates ONCE whether your user is privileged. This saves you from a whole bunch of .is_priviliged() calls in your admin panel. The post 'Boolean Blindness' [0] talks about much of the same issues. [0]: https://existentialtype.wordpress.com/2011/03/15/boolean-blindness/ https://existentialtype.wordpress.com/2011/03/15/boolean-bli...
- dkarl 1y ago
  I'm a big proponent of using types, but my experience with "Validated" types has not been positive. It's nice when it's unambiguous, but adding "validated" to the name of a type can mean almost anything. What has been validated? Validated according to what rules? You can add what has been validated to the type name, like I can tell that ValidatedNonnegativeInt is a nonnegative integer, but NonnegativeInt would tell me the same thing more concisely. I've also had a negative experience with using types to encode privileges. Types work well in simple situations, but they scale very badly with additional complexity. Something like PrivilegedUser works fine as long as privilege is binary and one-dimensional, but the need for new types will very quickly grow out of hand for only a modest increase in the complexity of requirements. Encoding privileges as data handles a combinatorial explosion of possibilities much more gracefully, and it is much more straightforward for checking rules that are stored outside the codebase.
deleted 1y ago
[deleted]
_zagj 1y ago
Never use "_t" suffixes for your own types in C or C++: they're reserved by POSIX.

130 points by lelanthran 1y ago

The trouble I have with this approach (which, conceptually, I agree with) is that it's damned hard to do anything with the parse results. Want to print that email_t? Then you're right back to char*, unless you somehow write your own I/O system that knows about your opaque conventions. So you say, okay, I'll make an `email_to_string` function. Does it return a copy or a reference? Who frees it? etc, etc, and you're back to square one again. The idea is to keep char* and friends at "the edge", but I've never found a way to really achieve that. Could just be my limitations as a C programmer, in which case I'd be thrilled to learn better.

dwattttt 1y ago
email_t doesn't have to be opaque; if it's just a visible wrapper around char* then you can still do everything with it as a char* (that is, everything you do with strings). The benefit is to avoid treating char*s as email_t, not avoiding treating email_t as char*.
- bcrosby95 1y ago
  In the example code they explicitly put the struct in the c file so the char* is not available. If you're suggesting getting around this by casting an email_t* to char* then I wish you good luck on your adventures. There's some times you gotta do stuff like that but this ain't it.
  - dwattttt 1y ago
    You could probably get away with the typecast if you satisfy the "common struct prefix" requirement, that's nowhere near necessary. While the article does hide the internal char*, that's not strictly necessary to get the benefit of "parse, don't validate". Hide implementation details sure, but not everything is an implementation detail.
- maxbond 1y ago
  (Using a thin wrapper like this to add safety is called the newtype pattern, if anyone wants to know.)
  - tetha 1y ago
    I was curious how this would look in C, and I found this article[1] how this could look in C, apparently with very little overhead. And as I just saw, Python 3.10 also introduced a NewType[2] wrapper. I'll have to see how that feels to handle. 1: https://blog.nelhage.com/2010/10/using-haskells-newtype-in-c/ https://blog.nelhage.com/2010/10/using-haskells-newtype-in-c... 2: https://typing.python.org/en/latest/spec/aliases.html#newtype https://typing.python.org/en/latest/spec/aliases.html#newtyp...
    - masklinn 1y ago
      Python’s NewType is, confusingly, a very different thing: it’s a compile-time-only subtype of the original, rather than a Haskell-style newtype (which is an entirely separate type from its source).
  - mrkeen 1y ago
    I've re-read the article again since getting a bunch of up and down votes across the comment section, and I think you've chosen a better name for this article than PdV. It really is just about using newtype wrappers.
lelanthran 1y ago
Firstly, `parsing` is just a way to say "serialise from a string". The reverse operation can be done for every type you are creating. If the reverse operation (serialise to a string) does not exist in the interface then adding it gives you a single place to catch all the bugs. I'm thinking of that recent git bug that occurred because the round-trip of `string -> type -> string` had an error (stripping out the CR character). Using a specific type for a value that is being round-tripped means that a bugfix needs to only be made in the parser function. Storing the value as simple strings would result in needing to put your fix everywhere. > The trouble I have with this approach (which, conceptually, I agree with) is that it's damned hard to do anything with the parse results. You're right - it is damn hard, but that is on purpose; if you're doing something with the email that boils down to "treat it like a `char *`" then the potential for error is large. If you're forced to add in a new use-case to the `email_t` interface then you have reduced the space of potential errors. For example: > Want to print that email_t? Then you're right back to char, unless you somehow write your own I/O system that knows about your opaque conventions. is a bug waiting to surface, because it's an email, not a string, and if you decide to print an email* that was read as a `char *` you might not get what you expect. It's all a trade-off - if you want more flexibility with the value stored in a variable, then sure, you can have it but it comes at a cost: some code somewhere almost certainly will eventually use that flexibility to mismatch the type! If you want to prevent type mismatches, then a lot of flexibility goes out the window.
- jagged-chisel 1y ago
  Linguistic nit: deserialize from a string, serialize to a string “Serialization” is the act of taking an internal data structure (of whatever shape and depth) and outputting it for transmission or storage. The opposite is “deserialization,” restoring the original shape and depth.
- KerrAvon 1y ago
  But that’s where TFA breaks down: the whole point of this is to claim “hey, C does too have typesafety” —if you use a modern language with actual typesafety, you can just make the underlying representation accessible (probably read-only) and you don’t need a reverse conversion step. You can have type safety, efficiency, and ease of use. Just rip the band-aid off and move to Rust or Swift already.
8organicbits 1y ago
In the past I've taken inspiration from strncpy: the caller needs to allocate the memory. For the email example, you'd probably also want a function to tell you the length of the emailstring, but for other types there are clear size limits. This puts the caller in control of memory allocation, so they may be able to statically allocate, allocate in an arena, or use other methods which promote performance. The static approach is really nice when it works, because there's nothing to free.
restalis 1y ago
The main benefit for me with this approach is that the boundries are not transparent anymore. That content printing is such a boundry. Your data is about to exit through there and you're summoned to handle that. The inconvenience that comes with it is as any other when security enters the play. The same with the data management responsibilities - who handles what, for how long, and with whom. Without data type distinctions everything is (more or less) common, with vague or broadly defined ownership.

o11c 1y ago

`_t` should not be used for custom types since it's reserved for future standard types (and/or types declared in a header you might include someday). This does cause real-world problems (`key_t` anyone?). Gratuitous allocations are gratuitous. The whole "prevent double free" claim is completely bogus. Setting a variable to `NULL` only works for cases where there is one, obvious, owner, which is not the circumstance under which double free is prone to happening in the first place. Actually preventing double free requires determining ownership of every object, and C sucks at that.

lelanthran 1y ago
> The whole "prevent double free" claim is completely bogus. "Completely" means "for all". Are you seriously claiming that "for all instances of double-free, setting the pointer to NULL after freeing it would not help"?
- lisper 1y ago
  > "Completely" means "for all". Not in the case of bogosity. Completely bogus things might occasionally work under some very particular circumstances, but unless those particular circumstances just happen to be the circumstances you actually care about, complete bogosity can still obtain. > setting the pointer to NULL There is no such thing as setting a pointer to null. You can set the value of a variable (whose current value is a pointer) to null, but you cannot guarantee that there isn't a copy of the pointer stored somewhere else except in a few very particular circumstances. This is what the GP meant by "setting a variable to `NULL` only works for cases where there is one, obvious, owner". And, as the GP also pointed out, this "is not the circumstance under which double free is prone to happening in the first place." Hence: complete bogosity.
- taneq 1y ago
  Eeeeh, I don't think 'completely bogus' means 'exhaustively false for all situations'. It just means 'demonstrably false' (for some relatively sane example, we're talking about C after all which means there will always be bogus examples which break any given assumption). There's plenty of cases where zeroing a pointer immediately after freeing it will prevent any further issues. It's still bogus to claim that it categorically solves the problem of double frees. But it does help.

mrkeen 1y ago

This stuck out: email_t theEmail = parseEmail(untrustedInput); if (theEmail == PARSE_ERROR) { return error; } An email_t is not a parse error, and a parse error is not one of the emails, so this shouldn't compile (and I don't take 'pseudocode' as an excuse).

parkcedar 1y ago
> and I don't take 'pseudocode' as an excuse Weird hill to die on, since neither email_t nor PARSE_ERROR were defined in the sample snippets. How do you know PARSE_ERROR is not email_t?
- VMG 1y ago
  Because an error is not an email?
  - howaboutno2312 1y ago
    [dead]
  - exe34 1y ago
    By that logic, a float couldn't store NaN.
    - mrkeen 1y ago
      Correct. You'll never see a raw float on the LHS of a PDV expression.
      - exe34 1y ago
        Who puts their personal data vault on the left hand side of a raw float?
- mrkeen 1y ago
  It's the parse-versus-validate hill in this case. This pseudocode is "Validate" for at least 3 reasons: Forgetting to check: this check is fragile: it’s extremely easy to forget. Because its return value is unused, it can always be omitted, and the code that needs it would still typecheck. Repeatable/redundant checks: First, it’s just annoying. We already checked that the list is non-empty, why do we have to clutter our code with another redundant check? Second, it has a potential performance cost. Although the cost of the redundant check is trivial in this particular example, one could imagine a more complex scenario where the redundant checks could add up, such as if they were happening in a tight loop. Not using the type system: Use a data structure that makes illegal states unrepresentable. Model your data using the most precise data structure you reasonably can. If ruling out a particular possibility is too hard using the encoding you are currently using, consider alternate encodings that can express the property you care about more easily. Don’t be afraid to refactor. > How do you know PARSE_ERROR is not email_t It has to be for it to compile, right? Which means that email_t is the type which represents both valid and invalid emails. How do you know if it's valid? You remember to write a check for it. Why not just save yourself some keystrokes and use char* instead. This is validate, not parse.

deleted 1y ago

[deleted]

anon-3988 1y ago

> Here, I’ll build on that by showing how this technique can be used outside of niche academic languages by demonstrating it in a language that is as practical as it is dangerous - C. The "practical" part really bugged me because the entire post is trying to explain exactly why it is not. The only way to make C reasonably safe is to encode information via newtype pattern. Wrap `char *` inside a struct that have proper names and include the size in there as well. Basically, there should be ZERO pointers except at creation and consumption by outside libraries (open, write, etc)

jkuli 1y ago

validating twice is safe. it will not error the second time. it is against the rules to call someone dumb on this server.

jagged-chisel 1y ago
Comments, however … ;-)

hackrmn 1y ago

I've been shouting this from "the rooftops" ever since I eventually independently had come to the same conclusion through sufficient volume of experience and empiry. I consider it a small tragedy, however, that my words tend to fall on deaf ears as I try to explain to e.g. co-workers why they should, in fact, parse their input and use the language's type system to their advantage, vs. all the `validateThis` and `validateThat` they seem to be copying from e.g. Stack Overflow. I see them stumbling and breaking their nose time and again passing URIs around as strings -- even in languages other than C (so this isn't obviously a problem C created and needs to solve alone) -- where it's never enforced that the URI is e.g. "absolute" or "relative" (nevermind the fact these people often don't seem to care for the difference). They do the same error with date and time (how many decades of advice do we need to accumulate and impart on our younger peers for them to irrevocably understand why e.g. date and time should never be passed around as numbers or strings, with very clearly documented and justified exceptions?). But the tragedy is never more evident when the person looks at you like you're suddenly talking Klingon to them and how bored they are with the notion and how it cannot be so important as to disturb their very important "coding project" (we don't write Knuth's "literal" programs any more -- we _code_).

pixelpoet 1y ago

Such a recognisable name, greetz after many years :D You might remember me from uni as lycium, a few lifetimes ago. Going to check the rest of your blog now...

lelanthran 1y ago
> Such a recognisable name, greetz after many years :D You might remember me from uni as lycium, a few lifetimes ago. Going to check the rest of your blog now... I do remember you? You had/joined a sartup in NZ, correct? Warm regards Lycium :-)
- pixelpoet 1y ago
  Thanks :D Yeah, we had a good time for ~10 years: https://glaretechnologies.com https://glaretechnologies.com I saw you're doing consultancy last years, cool :) Lemme know if you wanna chat on Discord or so, cheers!
  - lelanthran 1y ago
    > Thanks :D Yeah, we had a good time for ~10 years: https://glaretechnologies.com https://glaretechnologies.com > I saw you're doing consultancy last years, cool :) Lemme know if you wanna chat on Discord or so, cheers! I do wanna chat and perhaps catchup :-) Although I have a discord account, I don't remember ever using it.

hansvm 1y ago

One flaw I've seen in "Parse, Don't Validate" as it pertains to real codebases is that you end up with a combinatorial prolieration of types. E.g., requiring that a string be base64, have a certain fixed length, and be provided by the user. E.g., requiring that a file have the correct MIME type, not be too large, and contain no EXIF metadata. If you really always need all n of those things then life isn't terrible (you can parse your data into some type representing the composition of all of them), but you often only need 1, 2, or 3 and simultaneously don't want to duplicate too much code or runtime work, leading to a combinatorial explosion of intermediate types and parsing code. As one possible solution, I put together a POC in Zig [0] with one idea, where you abuse comptime to add arbitrary tagging to types, treating a type as valid if it has the subset of tags you care about. I'm very curious what other people do to appropriately model that sort of thing though. [0] https://github.com/hmusgrave/pdv https://github.com/hmusgrave/pdv

jeremyscanvic 1y ago
You might be interested in Lean's way of doing things. They have normal types (e.g. numeric types) and subtypes (e.g. numbers less than zero). An element of the subtype "numbers less than zero" can be understood as a tuple containing the actual number (which has a normal numeric type) and a proof that this specific number is indeed less than zero. https://lean-lang.org/doc/reference/latest/Basic-Types/Subtypes/ https://lean-lang.org/doc/reference/latest/Basic-Types/Subty...
tacitusarc 1y ago
Structs are representations of combinatorial types! In your file case, you could parse the input into a struct, and then accept or reject further processing based on that contents of that struct. Of course, it would be reasonable to claim that the accept/reject step is validation, but I believe “Parse, don’t validate” is about handling input, not an admonition to never perform validation.
- mrkeen 1y ago
  In pure C however, you still get the types-in-source-code explosion, for lack of parametric polymorphism. You need an email_or_error and a name_or_error, etc. The alternative is to fake PP with a void*, but that's so ugly I think I'd scrap the whole effort and just use char*. > I believe “Parse, don’t validate” is about handling input, not an admonition to never perform validation. It's about validation happening at exactly one place in the code base (during the "parse" - even though it's not limited to string-processing), so that callers can't do the validation themselves - because callers will validate 0 times or n>1 times.

myaccountonhn 1y ago

From experience, parsing input into data structures that fit the problem domain once at the "edge" is a good idea. The code becomes a lot more maintainable without a bunch of validation checks scattered all over the place, picking a data structure for the problem at hand usually leads to cleaner solutions, and errors usually show up much earlier and are easier to debug. From experience though I've found that wrapping all data in newtypes adds too much ceremony and boilerplate. If the data can reasonably be expressed as a primitive type, then you might as well express that way. I can't think of a time where newtype wrapping would have saved me from accidentally not validating or accidentally inputting the wrong data as a parameter. Especially the email example is quite weak, with ~30 lines of code just being ceremony due to wrapping a string, and most likely it's just going to be fed as is to various crud operations that will cast the data to a string immediately. Interacting with Haskell/elm libraries that have pervasive use of newtypes everywhere can be painful, especially if they don't give you a way to access the internal data. If a use-case comes up that the library developer didn't account for, then you might have no way of modifying the data and you end up needing to patch the library upstream.

movpasd 1y ago
I think it can be useful to think of the parsing and logic parts both as modules, with the parsing part interfacing with the outside world via unstructured data, and the parsing and logic parts interfacing via structured data, i.e.: the validated types. From that perspective, there is a clear trade-off on the size of the parsing–logic interface. Introducing more granular, safer validated types may give you better functionality, but it forces you to expand that interface and create coupling. I think there is a middle ground, which is that these safe types should be chunked into larger structures that enforce a range of related invariants and hopefully have some kind of domain meaning. That way, you shrink the conceptual surface area of the interface so that working with it is less painful.

belter 1y ago

This pattern seems to be shown almost as a comprehensive security solution when it's really just one layer of defense. This parse dont validate has to be combined with resource limits, and other protective measures during the parsing phase itself.

charcircuit 1y ago

I don't trust C developers to parse strings.

rorylaitila 1y ago

While this is the style I do in all of my (web) applications, it took a while to build my framework/tooling that overcomes the boilerplate to parse all incoming data into value objects. In a typical application of mine, there is usually well over dozens of entities and hundreds to thousands of fields. Creating value objects manually for all stringly types and numeric fields is a lot of work. Most value objects are some simple validation like limit to 255 characters or ints of some range. I solve for this by using reflection and auto generating all value objects (inheriting by default from base types) and auto generating all accessor/controller classes or methods into the domain model. Therefore I model in base types, override the generated value object constructors for validation (if required), and all of the boundaries are using value objects. The internal code generally works with the underlying base types, because boxing/unboxing the value objects can be non-negligible performance impact when serializing a lot of data (which tends to be common in web applications... SQL > JSON > HTML). I'm a huge fan, but I think ymmv. Web applications tend to have a wide interface (much of the domain model is user accessible). I think it's ideal for this case because of the number of fields a user can ultimately set and reused across many places.

OptionOfT 1y ago

In general I'm a huge favor of using the type system to its fullest extend to protect you from messing up. If the compiler can protect me, it should. Examples where I've used it in the past: ValidatedEmail, which is a special form of Email, one that has been validated by the user. We can have actions that require a `PriviligedUser`, which can be created from a `User`. That creation validates ONCE whether your user is privileged. This saves you from a whole bunch of .is_priviliged() calls in your admin panel. The post 'Boolean Blindness' [0] talks about much of the same issues. [0]: https://existentialtype.wordpress.com/2011/03/15/boolean-blindness/ https://existentialtype.wordpress.com/2011/03/15/boolean-bli...

dkarl 1y ago
I'm a big proponent of using types, but my experience with "Validated" types has not been positive. It's nice when it's unambiguous, but adding "validated" to the name of a type can mean almost anything. What has been validated? Validated according to what rules? You can add what has been validated to the type name, like I can tell that ValidatedNonnegativeInt is a nonnegative integer, but NonnegativeInt would tell me the same thing more concisely. I've also had a negative experience with using types to encode privileges. Types work well in simple situations, but they scale very badly with additional complexity. Something like PrivilegedUser works fine as long as privilege is binary and one-dimensional, but the need for new types will very quickly grow out of hand for only a modest increase in the complexity of requirements. Encoding privileges as data handles a combinatorial explosion of possibilities much more gracefully, and it is much more straightforward for checking rules that are stored outside the codebase.

deleted 1y ago

[deleted]

_zagj 1y ago

Never use "_t" suffixes for your own types in C or C++: they're reserved by POSIX.