6 ms·

Oh god, please just use primitive types. Don't make assumptions about things. Everyone thinks they are so smart validating emails, phone numbers, zip codes and

Oh god, please just use primitive types. Don't make assumptions about things. Everyone thinks they are so smart validating emails, phone numbers, zip codes and all until their great design goes live and they discover that users in the real world do not follow their assumptions. I have seen that happen again and again. No, if your idea of validating an email is more complicated than "should have an @ symbol", I guarantee you, there is counter example that will mess you pretty system up. Have fun scrambling to fix that ticket. Oh you think you know how an address and zip code should look like? No, you don't. Please people, just use strings and call it a day. Why do you like to suffer? I mean, sure, using the type system to protect you from mixing up units can be useful. Everything in moderation. Primitive types are good types.

revskill 4y ago
To me, type is for data abstraction, like we use generic for function abstraction. Abstraction in this case, mean, for future changes, i just need to change in one place, the Email type abstraction instead of searching and replacing every ussage of primitive email string.
chii 4y ago
> if your idea of validating an email is more complicated than "should have an @ symbol", I guarantee you, there is counter example that will mess you pretty system up. unless you are writing an email server. Then you would need to do this properly. In other words, validate the data that is in the domain of your application. If your app simply _sends_ email, then it's not your domain, and don't need to validate, as long as the receiving end of the email (aka, the email server) accepts it.
dwheeler 4y ago
Using specialized types can be useful, but if it provide s methods they need to be correct for ALL situations. Here's a subversion of GitHub's authentication (now fixed) where they assumed that "lowercasing domain name using English case rules is always fine and produces the same result" led to a vulnerability: https://dev.to/jagracey/hacking-github-s-auth-with-unicode-s-turkish-dotless-i-460n https://dev.to/jagracey/hacking-github-s-auth-with-unicode-s...
- afiori 4y ago
  TIL that punycode breaks email.split('@') to get the domain: Apparently John@Gıthub.com normalizes to xn--john@gthub-2ub.com but Gıthub.com normalizes to xn--gthub-n4a.com. For now I will go back to forgetting that emails not covered by /[A-Za-z0-9.-+_]+@[A-Za-z0-9.-_](.[A-Za-z0-9.-_])*/ exist to preserve my sanity but this does confuse me.
hansvm 4y ago
That sounds like an argument against excess validation rather than against types (maybe because of blog-driven development showing minimal examples of the power of types?). Picking on one of those points, suppose emails are just strings. What then prevents them from getting used as names, identifiers, and other unrelated data? Just your continued vigilance as a programmer and hoping that nobody ever carelessly names the field "address" so that mistakes can slip by over a series of devolution commits. You might not know much about what an email is, but you know _something_ about how you expect them to behave and be used, and if you like the computer to automate your work it's not totally unreasonable to rebind the string type as an email type and require explicit conversions at the point of use to treat it as anything other than just an email. There's zero runtime cost, it's not much more code, and that class of bugs is greatly reduced except for at the boundaries of the system. Maybe that effort isn't worth it, or maybe your domain changes fast enough you'd have a lot of churn, or maybe those bugs aren't too important, or whatever. Using types to help you reason about the things you do know and do care about can be a huge productivity boost though, so I wouldn't just write the whole technique off.
- indymike 4y ago
  > That sounds like an argument against excess validation rather than against types (maybe because of blog-driven development showing minimal examples of the power of types?). A lot of developers confuse types and validation.
RHSeeger 4y ago
Just because there are cases where over-validation (email address being the primary example) can be a problem doesn't mean that _no_ type validation is useful. There are many, many places where validation-beyond-primitives is useful, or even just types that need actions taken on them before they can be used in place of another type. One simple one is "non-negative integers", not commonly provided as a primitive type but _very_ common to need to be able enforce. Complex numbers are another. Normal strings vs HTML strings (need to be properly handled before they can be output to a front end). The list is endless, and custom types can provide a _lot_ of safety.
hiAndrewQuinn 4y ago
My favorite "easy win" find from learning Haskell last year was just the `newtype` keyword, which basically just let you alias primitive types with zero runtime impact. `newtype Email = Email String` and `newtype Username = Username String` are just `String`s with guard rails.
- nerdponx 4y ago
  Python's type annotation system has this feature too: from typing import NewType UserId = NewType('UserId', str) user_id: UserId user_id = "abc" # Type check error user_id = UserId("abc") # OK # At runtime, a NewType is the identity function s1: str = "abc" s2: UserId = UserId(s1) assert s1 is s2 Docs: https://docs.python.org/3/library/typing.html#newtype https://docs.python.org/3/library/typing.html#newtype Runtime: https://tio.run/##TU49D4JADN37K544CIkLupk4OBoTXXQ2CKdcDHeXtsTw6/GQoHZp@9r3ETqtvVv3/Z19A@2CdQ/YJnhWHM3r3AVDdBHD@wrbCUkXI7JYQpQzojauV1ttMOJfIFKS4lYm@NYcgwDK2pRPGGbPf78jO/1QsuH3dCCaY6fg1qltzBLFlAFWoHVslYkn7XBvXanWO5J8M8SavElWU66fheQZFRJnheSDlKyIAlunaXI6zJKs798 https://tio.run/##TU49D4JADN37K544CIkLupk4OBoTXXQ2CKdcDHeXts... Type checking: https://mypy-play.net/?mypy=latest&python=3.10&flags=show-error-codes%2Cstrict&gist=2f9be920806f100b1654770dca7d3639 https://mypy-play.net/?mypy=latest&python=3.10&flags=show-er...
- yakshaving_jgt 4y ago
  You probably want smart constructors, as you probably don't want the possibility for a 10,000 character username to float through your system.
kajaktum 4y ago
What? Would you rather have JSON as a string or as something like Map<string, JsonValue>? > No, if your idea of validating an email is more complicated than "should have an @ symbol" Then how is just using a string any better? Would you rather litter the entire codebase with validations that this is indeed a valid email or just do it once at the entry point? I think excessive type system hacks are bad too but they are more often than not the problem of the language where its unable to express certain concepts naturally (see C++ template hacks).
- jstimpfle 4y ago
  > Would you rather litter the entire codebase with validations that this is indeed a valid email Why would you? Just use it. For most of parts of the program it's not relevant to the computation whether the string is a "valid email", whatever that means. It's a string.
  - rowanG077 4y ago
    "Just use it" is what lead to the big SQL injection fallout and even today we pay the price as not even a year ago thousands of crucial service were vulnerable via log4j because of the "Just Use It" mantra.
    - jstimpfle 4y ago
      You don't understand what I said. I say, don't make assumptions unless you need them. Formatting an email address in an HTML document would work by wrapping it in the appropriate tag. (which implies html-quoting it correctly, but that is unrelated to email syntax). Sending an email using an API would work by passing the email as a string to the API. Looking up an email address from an address book using a pattern to match would be implemented with normal text search. It doesn't matter if the email is "valid" or not. Don't overthink it.
    - afiori 4y ago
      SQL injection, XSS, and similar attacks are about incorrectly encoding embedded fragments. The solution is either not to embed (SQL parameter bindings) or to always escape embedded fragments based on the embedding context For reference: The Last XSS Defense Talk - Jim Manico - NDC Porto 2022 | https://youtu.be/wRC7jyhTkEM https://youtu.be/wRC7jyhTkEM
tel 4y ago
There's a sharp distinction between validation and typing. I can cast a string into a domain-specific Email type without validating the string. I can also reject a string due to a validation rule without changing its type. Opaque type aliases are great because they impose semantics. Even if I don't know how to validate an email, it's nice, at times, to distinguish a string which I suspect to be an email from all other sorts of strings.
- persimmonster 4y ago
rowanG077 4y ago
You are talking about validation not types.
hgomersall 4y ago
The point is not about validation, it's about conveying semantic information through types. It's perfectly valid to have an email type that is just a wrapper around a string. The advantage is now you and all your functions unambiguously know that the type represents an email (whatever that means) and not a frobinator.
blackbrokkoli 4y ago
Thank you, I feel like I am going insane here. About 90% of stuff I ever worked does email things here or there, and I never had the desire for some hyper complex abstraction with 10 pages of documentation just to store email. I also never experienced a bug that would have been fixed by this. I mean if the user is putting in some bullshit there, the basic validation of type="email" or pendant if I am not working in HTML is gonna tell him that and after that he does or does not get his verification email. Boom, problem solved along with other one's. Need just the domain? Well I'll use a damn 1 liner function with split() or something for this hyper specific use case, don't encumber me with the probably faulty abstraction of email some library designer cooked up 8 years ago just for this?! I do use types when they are actually convenient, but these types-are-the-best-everywhere-all-the-time articles keep failing to convince me...
AnimalMuppet 4y ago
Ahem. Not all email addresses have an @ in them. If you're sending an email to another user on the same machine, just their username is enough (at least for some mail implementations). Note that this makes your overall point stronger, not weaker.
- draw_down 4y ago
noobermin 4y ago
People are getting lost about your example as they don't see the analogy as it sounds like validation of input. The point is still there if people are willing to look past the confusion, that making a type hierarchy requires you clearly know upfront what different subsets of the data you deal with actually look like and this will change at different stages of the project. This is the reason people ditched c++ and java in 2000 and started hacking in python and js, at least at the beginning although things are shifting. The thing is a lot of things are reversed here, the reality is the way evolution is more sensible is to "generalize" from the bottom up, unless you are very sure from the start that certain types categories make sense for your problem. That's how things should always be, you can only generalize once you've worked through a problem and realized certain aspects are actually common in someway and can be connected somehow, and thus you can take a subset of data you have in your program and organize it into a type. The thing is I feel like that is even harder than thinking top-down (which I feel like is where the pendulum is swinging now) because that takes time. The reason top-downers still do their thing is they feel their method is better is selective thinking (sorry, I know this is harsh) because they honestly discount the anger and fury people from the outside have dealing with their code and they ignore the amount of refactoring they need to do when their designs break. In fact, they love the churn really or at least merely accept it as a "part of development."
skybrian 4y ago
Unless you're really going to use textareas with scrollbars for every string and store everything in unlimited length database fields, you probably do want to distinguish between single-line and multiline strings and have some limit on their lengths.
dividedbyzero 4y ago
> No, if your idea of validating an email is more complicated than "should have an @ symbol", I guarantee you, there is counter example that will mess you pretty system up. Requiring b2b users to use their @company.com email is common and some b2b customers actually expect to be able to configure that. Another simple case is stripping out any "+whatever" from gmail addresses and to flag that sort of thing for other eomains so support can verify it's not a case of someone creating 72 trial accounts. Yes, rejecting users due to naive and incorrect validation is bad, but treating emails as entirely opaque strings isn't always an option either.
- xigoi 4y ago
  > Another simple case is stripping out any "+whatever" from gmail addresses and to flag that sort of thing for other eomains so support can verify it's not a case of someone creating 72 trial accounts. trial1@mypersonaldomain trial2@mypersonaldomain trial3@mypersonaldomain
  - dividedbyzero 4y ago
    But for that you need a personal domain and the ability to configure something like that, whereas everyone in a corporation that runs their email through Google can use the +abc thing out of the box, and I've seen it used a ton. The goal with measures like that is usually not to catch every single one who tries to game the free tier but to ensure it's more inconvenient to do so than just buy it for most people, and typically filtering +abc emails will be just one of several different measures taken to that end.
    - xigoi 4y ago
      There are sites where you can easily get all addresses of the format something@yourname.sitename.xxx.

revskill 4y ago

To me, type is for data abstraction, like we use generic for function abstraction. Abstraction in this case, mean, for future changes, i just need to change in one place, the Email type abstraction instead of searching and replacing every ussage of primitive email string.

chii 4y ago

> if your idea of validating an email is more complicated than "should have an @ symbol", I guarantee you, there is counter example that will mess you pretty system up. unless you are writing an email server. Then you would need to do this properly. In other words, validate the data that is in the domain of your application. If your app simply _sends_ email, then it's not your domain, and don't need to validate, as long as the receiving end of the email (aka, the email server) accepts it.

dwheeler 4y ago

Using specialized types can be useful, but if it provide s methods they need to be correct for ALL situations. Here's a subversion of GitHub's authentication (now fixed) where they assumed that "lowercasing domain name using English case rules is always fine and produces the same result" led to a vulnerability: https://dev.to/jagracey/hacking-github-s-auth-with-unicode-s-turkish-dotless-i-460n https://dev.to/jagracey/hacking-github-s-auth-with-unicode-s...

afiori 4y ago
TIL that punycode breaks email.split('@') to get the domain: Apparently John@Gıthub.com normalizes to xn--john@gthub-2ub.com but Gıthub.com normalizes to xn--gthub-n4a.com. For now I will go back to forgetting that emails not covered by /[A-Za-z0-9.-+_]+@[A-Za-z0-9.-_](.[A-Za-z0-9.-_])*/ exist to preserve my sanity but this does confuse me.

hansvm 4y ago

That sounds like an argument against excess validation rather than against types (maybe because of blog-driven development showing minimal examples of the power of types?). Picking on one of those points, suppose emails are just strings. What then prevents them from getting used as names, identifiers, and other unrelated data? Just your continued vigilance as a programmer and hoping that nobody ever carelessly names the field "address" so that mistakes can slip by over a series of devolution commits. You might not know much about what an email is, but you know _something_ about how you expect them to behave and be used, and if you like the computer to automate your work it's not totally unreasonable to rebind the string type as an email type and require explicit conversions at the point of use to treat it as anything other than just an email. There's zero runtime cost, it's not much more code, and that class of bugs is greatly reduced except for at the boundaries of the system. Maybe that effort isn't worth it, or maybe your domain changes fast enough you'd have a lot of churn, or maybe those bugs aren't too important, or whatever. Using types to help you reason about the things you do know and do care about can be a huge productivity boost though, so I wouldn't just write the whole technique off.

indymike 4y ago
> That sounds like an argument against excess validation rather than against types (maybe because of blog-driven development showing minimal examples of the power of types?). A lot of developers confuse types and validation.

RHSeeger 4y ago

Just because there are cases where over-validation (email address being the primary example) can be a problem doesn't mean that _no_ type validation is useful. There are many, many places where validation-beyond-primitives is useful, or even just types that need actions taken on them before they can be used in place of another type. One simple one is "non-negative integers", not commonly provided as a primitive type but _very_ common to need to be able enforce. Complex numbers are another. Normal strings vs HTML strings (need to be properly handled before they can be output to a front end). The list is endless, and custom types can provide a _lot_ of safety.

hiAndrewQuinn 4y ago

My favorite "easy win" find from learning Haskell last year was just the `newtype` keyword, which basically just let you alias primitive types with zero runtime impact. `newtype Email = Email String` and `newtype Username = Username String` are just `String`s with guard rails.

nerdponx 4y ago
Python's type annotation system has this feature too: from typing import NewType UserId = NewType('UserId', str) user_id: UserId user_id = "abc" # Type check error user_id = UserId("abc") # OK # At runtime, a NewType is the identity function s1: str = "abc" s2: UserId = UserId(s1) assert s1 is s2 Docs: https://docs.python.org/3/library/typing.html#newtype https://docs.python.org/3/library/typing.html#newtype Runtime: https://tio.run/##TU49D4JADN37K544CIkLupk4OBoTXXQ2CKdcDHeXtsTw6/GQoHZp@9r3ETqtvVv3/Z19A@2CdQ/YJnhWHM3r3AVDdBHD@wrbCUkXI7JYQpQzojauV1ttMOJfIFKS4lYm@NYcgwDK2pRPGGbPf78jO/1QsuH3dCCaY6fg1qltzBLFlAFWoHVslYkn7XBvXanWO5J8M8SavElWU66fheQZFRJnheSDlKyIAlunaXI6zJKs798 https://tio.run/##TU49D4JADN37K544CIkLupk4OBoTXXQ2CKdcDHeXts... Type checking: https://mypy-play.net/?mypy=latest&python=3.10&flags=show-error-codes%2Cstrict&gist=2f9be920806f100b1654770dca7d3639 https://mypy-play.net/?mypy=latest&python=3.10&flags=show-er...
yakshaving_jgt 4y ago
You probably want smart constructors, as you probably don't want the possibility for a 10,000 character username to float through your system.

kajaktum 4y ago

What? Would you rather have JSON as a string or as something like Map<string, JsonValue>? > No, if your idea of validating an email is more complicated than "should have an @ symbol" Then how is just using a string any better? Would you rather litter the entire codebase with validations that this is indeed a valid email or just do it once at the entry point? I think excessive type system hacks are bad too but they are more often than not the problem of the language where its unable to express certain concepts naturally (see C++ template hacks).

jstimpfle 4y ago
> Would you rather litter the entire codebase with validations that this is indeed a valid email Why would you? Just use it. For most of parts of the program it's not relevant to the computation whether the string is a "valid email", whatever that means. It's a string.
- rowanG077 4y ago
  "Just use it" is what lead to the big SQL injection fallout and even today we pay the price as not even a year ago thousands of crucial service were vulnerable via log4j because of the "Just Use It" mantra.
  - jstimpfle 4y ago
    You don't understand what I said. I say, don't make assumptions unless you need them. Formatting an email address in an HTML document would work by wrapping it in the appropriate tag. (which implies html-quoting it correctly, but that is unrelated to email syntax). Sending an email using an API would work by passing the email as a string to the API. Looking up an email address from an address book using a pattern to match would be implemented with normal text search. It doesn't matter if the email is "valid" or not. Don't overthink it.
  - afiori 4y ago
    SQL injection, XSS, and similar attacks are about incorrectly encoding embedded fragments. The solution is either not to embed (SQL parameter bindings) or to always escape embedded fragments based on the embedding context For reference: The Last XSS Defense Talk - Jim Manico - NDC Porto 2022 | https://youtu.be/wRC7jyhTkEM https://youtu.be/wRC7jyhTkEM

tel 4y ago

There's a sharp distinction between validation and typing. I can cast a string into a domain-specific Email type without validating the string. I can also reject a string due to a validation rule without changing its type. Opaque type aliases are great because they impose semantics. Even if I don't know how to validate an email, it's nice, at times, to distinguish a string which I suspect to be an email from all other sorts of strings.

persimmonster 4y ago

rowanG077 4y ago

You are talking about validation not types.

hgomersall 4y ago

The point is not about validation, it's about conveying semantic information through types. It's perfectly valid to have an email type that is just a wrapper around a string. The advantage is now you and all your functions unambiguously know that the type represents an email (whatever that means) and not a frobinator.

blackbrokkoli 4y ago

Thank you, I feel like I am going insane here. About 90% of stuff I ever worked does email things here or there, and I never had the desire for some hyper complex abstraction with 10 pages of documentation just to store email. I also never experienced a bug that would have been fixed by this. I mean if the user is putting in some bullshit there, the basic validation of type="email" or pendant if I am not working in HTML is gonna tell him that and after that he does or does not get his verification email. Boom, problem solved along with other one's. Need just the domain? Well I'll use a damn 1 liner function with split() or something for this hyper specific use case, don't encumber me with the probably faulty abstraction of email some library designer cooked up 8 years ago just for this?! I do use types when they are actually convenient, but these types-are-the-best-everywhere-all-the-time articles keep failing to convince me...

AnimalMuppet 4y ago

Ahem. Not all email addresses have an @ in them. If you're sending an email to another user on the same machine, just their username is enough (at least for some mail implementations). Note that this makes your overall point stronger, not weaker.

draw_down 4y ago

noobermin 4y ago

People are getting lost about your example as they don't see the analogy as it sounds like validation of input. The point is still there if people are willing to look past the confusion, that making a type hierarchy requires you clearly know upfront what different subsets of the data you deal with actually look like and this will change at different stages of the project. This is the reason people ditched c++ and java in 2000 and started hacking in python and js, at least at the beginning although things are shifting. The thing is a lot of things are reversed here, the reality is the way evolution is more sensible is to "generalize" from the bottom up, unless you are very sure from the start that certain types categories make sense for your problem. That's how things should always be, you can only generalize once you've worked through a problem and realized certain aspects are actually common in someway and can be connected somehow, and thus you can take a subset of data you have in your program and organize it into a type. The thing is I feel like that is even harder than thinking top-down (which I feel like is where the pendulum is swinging now) because that takes time. The reason top-downers still do their thing is they feel their method is better is selective thinking (sorry, I know this is harsh) because they honestly discount the anger and fury people from the outside have dealing with their code and they ignore the amount of refactoring they need to do when their designs break. In fact, they love the churn really or at least merely accept it as a "part of development."

skybrian 4y ago

Unless you're really going to use textareas with scrollbars for every string and store everything in unlimited length database fields, you probably do want to distinguish between single-line and multiline strings and have some limit on their lengths.

dividedbyzero 4y ago

> No, if your idea of validating an email is more complicated than "should have an @ symbol", I guarantee you, there is counter example that will mess you pretty system up. Requiring b2b users to use their @company.com email is common and some b2b customers actually expect to be able to configure that. Another simple case is stripping out any "+whatever" from gmail addresses and to flag that sort of thing for other eomains so support can verify it's not a case of someone creating 72 trial accounts. Yes, rejecting users due to naive and incorrect validation is bad, but treating emails as entirely opaque strings isn't always an option either.

xigoi 4y ago
> Another simple case is stripping out any "+whatever" from gmail addresses and to flag that sort of thing for other eomains so support can verify it's not a case of someone creating 72 trial accounts. trial1@mypersonaldomain trial2@mypersonaldomain trial3@mypersonaldomain
- dividedbyzero 4y ago
  But for that you need a personal domain and the ability to configure something like that, whereas everyone in a corporation that runs their email through Google can use the +abc thing out of the box, and I've seen it used a ton. The goal with measures like that is usually not to catch every single one who tries to game the free tier but to ensure it's more inconvenient to do so than just buy it for most people, and typically filtering +abc emails will be just one of several different measures taken to that end.
  - xigoi 4y ago
    There are sites where you can easily get all addresses of the format something@yourname.sitename.xxx.