8 ms·

Stop writing CLI validation. Parse it right the first time

204 points by dahlia 1y ago

yakshaving_jgt 1y ago
I've noticed that many programmers believe that parsing is some niche thing that the average programmer likely won't need to contend with, and that it's only applicable in a few specific low-level cases, in which you'll need to reach for a parser combinator library, etc. But this is wrong. Programmers should be writing parsers all the time!
- WJW 1y ago
  Last week my primary task was writing a github action that needed to log in to Heroku and push the current code on main and development branches to the production and staging environments respectively. The week before that, I wrote some code to make sure the type the object was included in the filters passed to an API call. Don't get me wrong, I actually love writing parsers. It's just not required all that often in my day-to-day work. 99% of the time when I need to write a parser myself it's for and Advent of Code problem, usually I just import whatever JSON or YAML parser is provided for the platform and go from there.
  - yakshaving_jgt 1y ago
    Do you not write validation? Or handle user input? Or handle server responses? Surely there’s some data processing somewhere.
- dkubb 1y ago
  The three most common things I think about when coding are DAGs, State Machines and parsing. The latter two come up all the time in regexps which I probably write at least once a day, and I’m always thinking about state transitions and dependencies.
- eska 1y ago
  I think most security issues are just due to people not parsing input at all/properly. Then security consultants give each one a new name as if it was something new. :-)
- nine_k 1y ago
  I'd say that engineers should use the highest-level tools that are adequate for the task. Sometimes it's going down to machine code, or rolling your own hash table, or writing your own recursive-descent parser from first principles. But most of the time you don't have to reach that low, and things like parsing are but a minor detail in the grand scheme. The engineer should not spend time on building them, but should be able to competently choose a ready-made part. I mean, creating your own bolts and nuts may be fun, but mot of the time, if you want to build something, you just pick a few from an appropriate box, and this is exactly right.
  - yakshaving_jgt 1y ago
    I don’t understand. Every mainstream language has libraries for parsing into general types, but none of them will have libraries for parsing values specific to your application. TFA links to Alexis King’s Parse, Don’t Validate article, which explains this well. Did you not read it?
dvdkon 1y ago
I, for one, do think the world needs more CLI argument parsers :) This project looks neat, I've never thought to use parser combinators for something other than left-to-right string/token stream parsing. And I like how it uses Typescript's metaprogramming to generate types from the parser code. I think that would be much harder (or impossible) in other languages, making the idiomatic design of a similar similar library very different.
HL33tibCe7 1y ago
Stopped reading after realising this is written by ChatGPT
- bfung 1y ago
  Looked human-ish to me, what signs did you see?
  - bobbiechen 1y ago
    I thought the style was like ChatGPT in a "clever, casual, snarky" prompt flavor as well. I see it a lot on LinkedIn especially in sentence structures like these: "Invalid data? The parser rejects it. Done." "That validation logic that used to be 30% of my CLI code? Gone." "Mutually exclusive groups? Sure. Context-dependent options? Why not." For me this really piled on at the end of the blog post. But maybe it's just personal style too.
- cazum 1y ago
  What makes you think that and not that it's just an average auto-translate job from the author's native language (Korean)?
  - urxvtcd 1y ago
    I’ll go one step further: what makes you think it’s an average auto-translate job? I didn’t notice anything weird, felt like your average, slightly ranty HN post. I’m not a native speaker though.
- akoboldfrying 1y ago
  I found the content novel and helpful (applying a known but underappreciated technique (Parse, Don't Validate) to a common problem where I hadn't thought to use it before) and the tone very enjoyable. In fact, it's so idiomatically written that I can't even believe it's just a machine translation of something written in another language. In short, a great article.
thealistra 1y ago
Isn’t this like argparse from Python for typescript?
- whilenot-dev 1y ago
  What OP calls an "combinatorial parser" I'd call object schema validation and that's more similar to pydantic[0] than argparse in python land. [0]: https://docs.pydantic.dev/latest/ https://docs.pydantic.dev/latest/
  - nhumrich 1y ago
    So, typer than
    - mrugge 1y ago
      Or click
jmull 1y ago
> Think about it. When you get JSON from an API, you don't just parse it as any and then write a bunch of if-statements. You use something like Zod to parse it directly into the shape you want. Invalid data? The parser rejects it. Done. Isn’t writing code and using zod the same thing? The difference being who wrote the code. Of course, you hope zod is robust, tested, supported, extensible, and has docs so you can understand how to express your domain in terms it can help you with. And you hope you don’t have to spend too much time migrating as zod’s api changes.
- akoboldfrying 1y ago
  Yes, both are writing code. But nearly all the time, the constraints you want to express can be expressed with zod, and in that case using zod means you write less code, and the code you do write is more correct. > Of course, you hope zod is robust, tested, supported, extensible, and has docs so you can understand how to express your domain in terms it can help you with. And you hope you don’t have to spend too much time migrating as zod’s api changes. Yes, judgement is required to make depending on zod (or any library) worthwhile. This is not different in principle from trusting those same things hold for TypeScript, or Node, or V8, or the C++ compiler V8 was compiled with, or the x86_64 chip it's running on, or the laws of physics.
  - jmull 1y ago
    Sure... the laws of physics last broke backwards compatibility at the Big Bang, Zod last broke backwards compatibility a few months ago.
- bigstrat2003 1y ago
  Yeah, the "parse, don't validate" advice seems vacuous to me because of this. Someone is doing that validation. I think the advice would perhaps be phrased better as "try to not reimplement popular libraries when you could just use them".
  - remexre 1y ago
    The difference between parse and validate is function parse(x: Foo): Bar { ... } const y = parse(x); and function validate(x: Foo): void { ... } validate(x); const y = x as Bar; Zod has a parser API, not a validator API.
parhamn 1y ago
> Try to access it and TypeScript yells at you. No runtime validation needed. I was recently thinking about type safety and validation strategies are particularly thorny in languages where the typings are just annotations. E.g. the Typescript/Zod or Python/Pydantic universes. Especially in IO cases where the data doesn't originate in the same type system. In a language like Go (just an example, not endorsing) if you parse something into say a struct you know worst case you're getting that struct with all the fields set to zero, and you just have to handle the zero values. In typescript-likes you can get a totally different structure and run into all sorts of errors. All that is to say, the runtime validation is always somewhere (perhaps in the library, as they often are?), and the feature here isn't no runtime validation but typed cli arguments. Which is cool and great.
- metaltyphoon 1y ago
  > worst case you're getting that struct with all the fields set to zero, and you just have to handle the zero values In the field I work, zero values are valid and doing it in Go would be a nightmare
  - parhamn 1y ago
    Agreed, the pointer or "<field>_empty: bool" patterns are annoying. Point still stands though, you always get the structure you ask for.
  - mjevans 1y ago
    Database NULL is a valid pattern that any parser SHOULD support and I do consider that a design bug in every parser Go has. Offhand most of them effectively 'update' an object, but make it difficult or impossible to tell if something was __set__ with a value, or merely inherited a default.
nine_k 1y ago
This is a recurring idea: "Parse, don't validate". Previously: https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/ https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-va... (2019, using Haskell) https://www.lelanthran.com/chap13/content.html https://www.lelanthran.com/chap13/content.html (April 2025, using C)
- jetrink 1y ago
  The author credits Alexis King at the beginning and links to that post.
ThinkBeat 1y ago
And that is why there are plenty of parser generators so you dont have to write the parser yourself every time.
bsoles 1y ago
>> // This is a parser >> const port = option("--port", integer()); I don't understand. Why is this a parser? Isn't it just way of enforcing a type in a language that doesn't have types? I was expecting something like a state machine that takes the command line text and parses it to validate the syntax and values.
- hansvm 1y ago
  The heavy lifting happens in the definitions of `option` and `integer`. Those will take in whatever arguments they take in and output some sort of `Stream -> Result<Tuple<T, Stream>>` function. That might sound messy but to the author's point about parser combinators not being complicated, they really don't take much time to get used to, and they're quite simple if you wanted to build such a library yourself. There's not much code (and certainly no magic) going on under the hood. The advantage of that parsing approach: It's reasonably declarative. This seems like the author's core point. Parser-combinator code largely looks like just writing out the object you want as a parse result, using your favorite combinator library as the building blocks, and everything automagically works, with amazing type-checking if your language has such features. The disadvantages: 1. Like any parsing approach, you have to actually consider all the nuances of what you really want parsed (e.g., conditional rules around whitespace handling). It looks a little to me (just from the blog post, not having examined the inner workings yet) like this project side-stepped that by working with the `Stream` type as just the `argv` list, allowing you to be able to say things like "parse the next blob as a string" without also having to encode whitespace and blob boundaries. 2. It's definitely slower (and more memory-intensive) than a hand-rolled parser, and usually also worse in that regard than other sorts of "auto-generated" parsing code. For CLI arguments, especially if they picked argv as their base stream type, those disadvantages mostly don't exist. I could see it performing poorly for argv parsing for something like `cp` though (maybe not -- maybe something like `git cp`, which has more potential parse failures from delimiters like `--`?), which has both options and potentially ginormous lists of files; if you're not very careful in your argument specification then you might have exponential backtracking issues, and where that would be blatantly obvious in a hand-rolled parser it'll probably get swept under the rug with parser combinators.
lihaoyi 1y ago
That's basically what my MainArgs Scala library does: take either a method definition or class structure and use it's structure to parse your command line arguments. You get the final fields you want immediately without needing to imperatively walk to args array (and probably getting it wrong!) https://github.com/com-lihaoyi/mainargs https://github.com/com-lihaoyi/mainargs
SloopJon 1y ago
I don't see anything in the post or the linked tutorial that gives a flavor of the user experience when you supply an invalid option. I tried running the example, but I've forgotten too much about Node and TypeScript to make it work. (It can't resolve the @optique references.) What happens when you pass --foo, --target bar, or --port 3.14?
- macintux 1y ago
  I had a similar question: to me, the output format “or” statement looks like it might deterministically pick one winner instead of alerting the user that they erred. A good parser is terrific, but it needs to give useful feedback.
  - Dragging-Syrup 1y ago
    Absolutely; I think calling the function xor would be more appropriate.
SoftTalker 1y ago
I like just writing functions for each valid combination of flags and parameters. Anything that isn’t handled is default rejected. Languages like Erlang with pattern matching and guards make this a breeze.
AfterHIA 1y ago
You've got to be careful; if you validate the CLI too much you might get URA in your validator. #chugalug #house
12_throw_away 1y ago
I like this advice, and yeah, I always try to make illegal states unrepresentable, possibly even to a fault. The problem I run into here is - how do you create good error messages when you do this? If the user has passed you input with multiple problems, how do you build a list of everything that's wrong with it if the parser crashes out halfway through?
- ambicapter 1y ago
  Most validation libraries worth their salt give you options to deal with this sort of thing? They'll hand you an aggregate error with an 'errors' array, or they'll let you write an error message "prettify-er" to make a particular validation error easier to read.
  - Thaxll 1y ago
    This work if all errors are self contained, stoping at the first one is fine too.
  - pmarreck 1y ago
    Right, but that's validation, and this article is talking about parsing (not validating) into an already-correct structure by making invalid inputs unrepresentable. So maybe the reason why they were able to reduce the code is because they lost the ability to do good error reporting.
    - jpc0 1y ago
      How is getting an error array not making invalid input unrepresentable. You either get the correctly parsed data or you get an error array. The incorrect input was never represented in code, vs a 0 value being returned or even worse random gibberish. A trivial example: 1/0 should return DivisionByZero not 0 or infinity or NaN or whatever else. You can then decide in your UI whether that is a case you want to handle as an error or as an edge case but the parser knows that is not possible to represent.
    - lmm 1y ago
suff 1y ago
[dead]
andrewguy9 1y ago
Docopt! http://docopt.org/ http://docopt.org/ Make use of the usage string be the specification! A criminally underused library.
- fragmede 1y ago
  My favorite. A bit too much magic for some, but it seems well specified to me.
- tomjakubowski 1y ago
  A great example of "declaration follows use" outside of C syntax.
sudahtigabulan 1y ago
Is there no getopt implementation for Typescript? The input this library tries to handle better looks to me like bad design. "options that depend on options" should not be a thing. Every option should be optional. Even if you have working code that can handle some complex situation, this doesn't make the situation any less unintuitive for the users. If you need more complex relationships, consider using arguments as well. Top level, or under an option. Yes, they are not named, but since they are mandatory anyway, you are likely to remember their meaning (spaced repetition and all that). They can still be optional (if they come last). Sometimes an argument may need to have multiple parts, like user@host:port You can still parse it instead of validating, if you want. > mutually exclusive --json, --xml, --yaml. Use something like -t TYPE instead, where TYPE can be one of json, xml, or yaml. (Make illegal states unrepresentable.) > debug: optional(option("--debug")), Again, I believe it's called "option" because it's meant to be optional already. optional(optional(option("--common-sense"))) EOR
- dwattttt 1y ago
  > options that depend on options What would you do for "top level option, which can be modified in two other ways"? (--option | --option-with-flag1 | --option-with-flag2 | --option-with-flag1-and-flag2) would solve invalid representation, but is unwieldy. Something that results in the usage string [--option [--flag1 --flag2]] doesn't seem so bad at that point.
  - sudahtigabulan 1y ago
    I think I've seen it done like that --option flag1,flag2 (Maybe with another separator, as long as it doesn't need to be escaped.) Another possibility is to make the main option an argument, like the subcommands in git, systemctl, and others: command option --flag1 --flag2 This depends on the specifics, though.
    - dwattttt 1y ago
      > --option flag1,flag2 Embedding a second parse step that the first parser doesn't deal with is done, but it's a rough compromise. It feels like the difficulty in dealing with [--option [--flag1 --flag2]] Is more to do with its expression in the language parsed to, than CLI elegance.
dcre 1y ago
Some other libraries I’ve been enjoying building CLIs with in TS that do more or less the same thing, though perhaps with slightly worse composability than Optique: https://cliffy.io/ https://cliffy.io/ https://github.com/tj/commander.js https://github.com/tj/commander.js
curtisszmania 1y ago
[dead]
esafak 1y ago
The "problem" is that some languages don't have rich enough type systems to encode all the constraints that people want to support with CLI options. And many programmers aren't that great at wielding the type systems at their disposal.
m463 1y ago
This kind of stuff is what makes me appreciate python's argparse. It's a genuine pleasure to use, and I use it often. If you dig a little deeper into it, it does all the type and value validation, file validation, it does required and mutually exclusive args, it does subargs. And it lets you do special cases of just about anything. And of course it does the "normal" stuff like short + long args, boolean args, args that are lists, default values, and help strings.
- MrJohz 1y ago
  Actually, I think argparse falls into the same trap that the author is talking about. You can define lots of invariants in the parser, and say that these two arguments can't be passed together, or that this argument, if specified, requires these arguments to also be specified, etc. But the end result is a namespace with a bunch of key-value pairs on it, and argparse doesn't play well with typing systems like mypy or pyright. So the rest of the tool has to assume that the invariants were correctly specified up-front. The result is that you often still this kind of defensive programming, where argparse ensures that an invariant holds, but other functions still check the same invariant later on because they might have been called a different way or just because the developer isn't sure whether everything was checked where they are in the program. What I think the author is looking for is a combination of argparse and Pydantic, such that when you define a parser using argparse, it automatically creates the relevant Pydantic classes that define the type of the parsed arguments.
  - sgarland 1y ago
    Precisely my thought. I love argparse, but you can really back yourself into a corner if you aren’t careful.
  - hahn-kev 1y ago
    It's almost like you want compile time type safety
    - MrJohz 1y ago
      You can have that with Mypy and friends in Python, and Typescript in the JS world. The problem is that older libraries often don't utilise that type safety very well because their API wasn't designed for it. The library in the original post is essentially a Javascript library, but it's one designed so that if you use it with Typescript, it provides that type safety.
jiggawatts 1y ago
This is one of the many reasons I like PowerShell: it parses strongly typed parameters for you and outputs human readable error messages for every kind of validation failure.
adamddev1 1y ago
Yay for parser combinators in the JS/TS wild!
- brabel 1y ago
  Exactly, the author's library is just a parser combinator [1] that specializes in providing constructs mirrorring CLI options. [1] https://en.wikipedia.org/wiki/Parser_combinator https://en.wikipedia.org/wiki/Parser_combinator
throwaway984393 1y ago
[dead]
bvrmn 1y ago
A valid type for server and port should be a single value. Stop parse it separately please. ":3000" -> use port 3000 with a default host. "some-host" -> use host with a default port. "some-host:3000" -> you guess it. It also allows to extend it to other sources/destinations like unix domain sockets and other stuff without cluttering your CLI options. Also please consider to use DSN or URI to define database configurations. Host, port, dbname, credentials as separate options or environment variables are quite painful to use.
slifin 1y ago
So use Clojure Spec or better yet Malli to parse your input data at the edges of your program Makes sense, I think a lot of developers would want to complect this problem with their runtime type system of choice without considering the set of downsides for the users
bschwindHN 1y ago
Rust with Clap solved this forever ago. Also - don't write CLI programs in languages that don't compile to native binaries. I don't want to have to drag around your runtime just to execute a command line tool.
- majorbugger 1y ago
  I will keep writing my CLI programs in the languages I want, thanks. Have it crossed your mind that these programs might be for yourself or for internal consumption? When you know runtime will be installed anyway?
  - dcminter 1y ago
    You do you, obviously, but "now let npm work its wicked way" is an offputting step for some of us when narrowing down which tool to use. My most comfortable tool is Java, but I'm not going to persuade most of the HN crowd to install a JVM unless the software I'm offering is unbearably compelling. Internal to work? Yeah, Java's going to be an easy sell. I don't think OP necessarily meant it as a political statement.
    - goku12 1y ago
      There should be some way to define the CLI argument format and its constraints in some sort of DSL that can be compiled into the target language before the final compilation of the application. This way, it can be language agnostic (though I don't know why you would need this) without the need for another runtime. The same interface specification should be able to represent a customizable help/usage message with sane defaults, generate dynamic tab completions code for multiple shells, generate code for good quality customizable error messages in case of CLI argument errors and generate a neatly formatted man page with provisions for additional content, etc. In fact, I think something like this already exists. I just can't recollect the project.
      - craftkiller 1y ago
        docopt: https://github.com/docopt/docopt https://github.com/docopt/docopt
deleted 1y ago
[deleted]
panzi 1y ago
No mention of yargs?
globular-toast 1y ago
Not all of this validation belongs in the same layer. A lot of the problems people seem to have is due to people thinking it all has to be done in the I/O layer. A CLI and an API should indeed occupy the same layer of a program architecture, namely they are entry points that live on the periphery. But really all you should be doing there is lifting the low byte stream you are getting from users to something higher level you can use to call your internals. So "CLI validation" should be limited to just "I need an int here, one of these strings here, optionally" etc. Stuff like "is this port out of range" or "if you give me this I need this too" should be handled by your internals by e.g. throwing an exception. Your CLI can then display that as an error message in a nice way.
AndrewDucker 1y ago
This is one of the things that makes me glad that PowerShell does all of this intrinsically. I define the parameters, it makes sure that the arguments make sense and match them (and their validation).
einpoklum 1y ago
Exactly the opposite of this. We should parse the command-line using _no_ strict types. Not even integers. Nothing beyond parsing its structure, e.g. which option names get which (string) values, and which flags are enabled. This can be done without knowing _anything_ about the application domain, and provide a generic options structure which is no longer a sequence of characters. This approach IMNSHO is much cleaner than the intrication of cmdline parser libraries with application logic and application-domain-related types. Then one can specify validation logic declaratively, and apply it generically. This has the added benefit - for compiled rather than interpreted library - of not having to recompile the CLI parsing library for each different app and each different definition of options.
- MrJohz 1y ago
  Can you give some examples of this working well? It certainly goes against all of my experience working with CLIs and with parsing inputs in general (e.g. web APIs etc). In general, I've found that the quicker I can convert strings into rich types, the easier that code is to work with and the less likely I am to have troubles with invalid data.
  - einpoklum 1y ago
    Think of it this way: Your code for quickly converting things into rich types - just run it on a map of argument-to-string value map rather than on a sequence of characters. It's still "quick": This is just like we current get an array-of-strings instead of a single-string command-line; it's initial domain-agnostic parsing, which can't even fail.
- bakkoting 1y ago
  This is the approach taken by node's built-in argument parser util.parseArgs.
jappgar 1y ago
I really think parse don't validate gives people a false sense of security (particularly false in dynamic languages like javascript and python). "Well, I already know this is a valid uuid, so I don't really need to worry about sql injection at this point." Sure, this is a dumb thing to do in any case, but I've seen this exact thing happen. Typesafety isn't safety.
- yakshaving_jgt 1y ago
  Type safety is absolutely some degree of safety. And I don’t know why anyone would think parsing a value into a type that has fewer inhabitants would absolve them of having to prevent SQL injection — these are orthogonal things. The quote here — which I suspect is a straw man — is such a weird non sequitur. What would logically follow from “I already know this is a valid UUID” is “so I don’t need to worry about this not being a UUID at this point”.
  - jappgar 1y ago
    In python or typescript, the most popular languages in the world, it offers no runtime safety. Even in languages like Haskell, "safety" is an illusion. You might create a NumberGreaterThanFive type with smart constructors but that doesn't stop another dev from exporting and abusing the plain constructor somewhere else. For the most part it's fine to assume the names of types are accurate, but for safety critical operations it absolutely makes sense to revalidate inputs.
    - yakshaving_jgt 1y ago
      > that doesn't stop another dev from exporting and abusing the plain constructor somewhere else. That seems like a pretty unfair constraint. Yes, you can deliberately circumvent safeguards and you can deliberately write bad code. That doesn't mean those language features are bad.
foundart 1y ago
The author of the article also wrote a CLI parser library for Typescript, called Optique. I really appreciate them including a "When Optique makes sense" section in the docs. It would be great if more projects did that. https://optique.dev/why#when-optique-makes-sense https://optique.dev/why#when-optique-makes-sense
nickdothutton 1y ago
It’s been about 30 years but I seem to remember the compiler taking care of this for me (in Ada) with types.
kiliancs 1y ago
Great project. Clear goal, well executed, very nice API (safe, terse, clear). I use Effect CLI https://github.com/Effect-TS/effect/tree/main/packages/cli https://github.com/Effect-TS/effect/tree/main/packages/cli for the same reasons. It has the advantage of fitting within the ecosystem. For example, I can reuse existing schemas.
baroninthetrees 1y ago
I too got tired of dealing with cli arg parsing and am experimenting with passing a natural language description of the program and its args to a tiny LLM to sort out, offer suggestions (did you mean?), types conversions, etc. So far, it’s working great and given enough detail is deterministic.
AnimalMuppet 1y ago
Well, they're dictating that if you want them to use it, do it this way. Some people want others to use the programs they write; for such people, the GP actually has been given the right to have some valid say in the matter. Why CLIs in particular? Because they usually are smaller tools. For a big, important tool, you might be willing to jump through more hoops (installing the right runtime), but for a smaller, less important tool, it's just not worth it.
geon 1y ago
I just recently implemented my own parser combinator lib in typescript too. It was surprisingly simple in the end. This function parses a number in 6502 asm. So `255` in dec or `$ff` in hex: https://github.com/geon/dumbasm/blob/main/src/parsers/parseNumber.ts https://github.com/geon/dumbasm/blob/main/src/parsers/parseN... I looked at several typescript libraries but they all felt off. Writing my own at least ensured I know how it works.
amterp 1y ago
Very much agree with the article, this is one of the reasons why I wrote Rad [0], which people here might find interesting. The idea is you write CLI scripts with a declarative approach to script arguments, including all the constraints on them, including relational ones. So you don't write your own CLI validation - you declare the shape that args should take, let Rad check user input for you, and you can focus your script on the interesting stuff. For example args: username str # Required string password str? # Optional string token str? # Optional auth token age int # Required integer status str # Required string username requires password // If username is provided, password must also be provided token excludes password // Token and password cannot be used together age range [18, 99] // Inclusive range from 18 to 99 status enum ["active", "inactive", "pending"] Rad will handle all the validation for you, you can just write the rest of your script assuming the constraints you declared are met. [0]: https://github.com/amterp/rad https://github.com/amterp/rad

204 points by dahlia 1y ago

I've noticed that many programmers believe that parsing is some niche thing that the average programmer likely won't need to contend with, and that it's only applicable in a few specific low-level cases, in which you'll need to reach for a parser combinator library, etc. But this is wrong. Programmers should be writing parsers all the time!

WJW 1y ago
Last week my primary task was writing a github action that needed to log in to Heroku and push the current code on main and development branches to the production and staging environments respectively. The week before that, I wrote some code to make sure the type the object was included in the filters passed to an API call. Don't get me wrong, I actually love writing parsers. It's just not required all that often in my day-to-day work. 99% of the time when I need to write a parser myself it's for and Advent of Code problem, usually I just import whatever JSON or YAML parser is provided for the platform and go from there.
- yakshaving_jgt 1y ago
  Do you not write validation? Or handle user input? Or handle server responses? Surely there’s some data processing somewhere.
dkubb 1y ago
The three most common things I think about when coding are DAGs, State Machines and parsing. The latter two come up all the time in regexps which I probably write at least once a day, and I’m always thinking about state transitions and dependencies.
eska 1y ago
I think most security issues are just due to people not parsing input at all/properly. Then security consultants give each one a new name as if it was something new. :-)
nine_k 1y ago
I'd say that engineers should use the highest-level tools that are adequate for the task. Sometimes it's going down to machine code, or rolling your own hash table, or writing your own recursive-descent parser from first principles. But most of the time you don't have to reach that low, and things like parsing are but a minor detail in the grand scheme. The engineer should not spend time on building them, but should be able to competently choose a ready-made part. I mean, creating your own bolts and nuts may be fun, but mot of the time, if you want to build something, you just pick a few from an appropriate box, and this is exactly right.
- yakshaving_jgt 1y ago
  I don’t understand. Every mainstream language has libraries for parsing into general types, but none of them will have libraries for parsing values specific to your application. TFA links to Alexis King’s Parse, Don’t Validate article, which explains this well. Did you not read it?

dvdkon 1y ago

I, for one, do think the world needs more CLI argument parsers :) This project looks neat, I've never thought to use parser combinators for something other than left-to-right string/token stream parsing. And I like how it uses Typescript's metaprogramming to generate types from the parser code. I think that would be much harder (or impossible) in other languages, making the idiomatic design of a similar similar library very different.

HL33tibCe7 1y ago

Stopped reading after realising this is written by ChatGPT

bfung 1y ago
Looked human-ish to me, what signs did you see?
- bobbiechen 1y ago
  I thought the style was like ChatGPT in a "clever, casual, snarky" prompt flavor as well. I see it a lot on LinkedIn especially in sentence structures like these: "Invalid data? The parser rejects it. Done." "That validation logic that used to be 30% of my CLI code? Gone." "Mutually exclusive groups? Sure. Context-dependent options? Why not." For me this really piled on at the end of the blog post. But maybe it's just personal style too.
cazum 1y ago
What makes you think that and not that it's just an average auto-translate job from the author's native language (Korean)?
- urxvtcd 1y ago
  I’ll go one step further: what makes you think it’s an average auto-translate job? I didn’t notice anything weird, felt like your average, slightly ranty HN post. I’m not a native speaker though.
akoboldfrying 1y ago
I found the content novel and helpful (applying a known but underappreciated technique (Parse, Don't Validate) to a common problem where I hadn't thought to use it before) and the tone very enjoyable. In fact, it's so idiomatically written that I can't even believe it's just a machine translation of something written in another language. In short, a great article.

thealistra 1y ago

Isn’t this like argparse from Python for typescript?

whilenot-dev 1y ago
What OP calls an "combinatorial parser" I'd call object schema validation and that's more similar to pydantic[0] than argparse in python land. [0]: https://docs.pydantic.dev/latest/ https://docs.pydantic.dev/latest/
- nhumrich 1y ago
  So, typer than
  - mrugge 1y ago
    Or click

jmull 1y ago

> Think about it. When you get JSON from an API, you don't just parse it as any and then write a bunch of if-statements. You use something like Zod to parse it directly into the shape you want. Invalid data? The parser rejects it. Done. Isn’t writing code and using zod the same thing? The difference being who wrote the code. Of course, you hope zod is robust, tested, supported, extensible, and has docs so you can understand how to express your domain in terms it can help you with. And you hope you don’t have to spend too much time migrating as zod’s api changes.

akoboldfrying 1y ago
Yes, both are writing code. But nearly all the time, the constraints you want to express can be expressed with zod, and in that case using zod means you write less code, and the code you do write is more correct. > Of course, you hope zod is robust, tested, supported, extensible, and has docs so you can understand how to express your domain in terms it can help you with. And you hope you don’t have to spend too much time migrating as zod’s api changes. Yes, judgement is required to make depending on zod (or any library) worthwhile. This is not different in principle from trusting those same things hold for TypeScript, or Node, or V8, or the C++ compiler V8 was compiled with, or the x86_64 chip it's running on, or the laws of physics.
- jmull 1y ago
  Sure... the laws of physics last broke backwards compatibility at the Big Bang, Zod last broke backwards compatibility a few months ago.
bigstrat2003 1y ago
Yeah, the "parse, don't validate" advice seems vacuous to me because of this. Someone is doing that validation. I think the advice would perhaps be phrased better as "try to not reimplement popular libraries when you could just use them".
- remexre 1y ago
  The difference between parse and validate is function parse(x: Foo): Bar { ... } const y = parse(x); and function validate(x: Foo): void { ... } validate(x); const y = x as Bar; Zod has a parser API, not a validator API.

parhamn 1y ago

> Try to access it and TypeScript yells at you. No runtime validation needed. I was recently thinking about type safety and validation strategies are particularly thorny in languages where the typings are just annotations. E.g. the Typescript/Zod or Python/Pydantic universes. Especially in IO cases where the data doesn't originate in the same type system. In a language like Go (just an example, not endorsing) if you parse something into say a struct you know worst case you're getting that struct with all the fields set to zero, and you just have to handle the zero values. In typescript-likes you can get a totally different structure and run into all sorts of errors. All that is to say, the runtime validation is always somewhere (perhaps in the library, as they often are?), and the feature here isn't no runtime validation but typed cli arguments. Which is cool and great.

metaltyphoon 1y ago
> worst case you're getting that struct with all the fields set to zero, and you just have to handle the zero values In the field I work, zero values are valid and doing it in Go would be a nightmare
- parhamn 1y ago
  Agreed, the pointer or "<field>_empty: bool" patterns are annoying. Point still stands though, you always get the structure you ask for.
- mjevans 1y ago
  Database NULL is a valid pattern that any parser SHOULD support and I do consider that a design bug in every parser Go has. Offhand most of them effectively 'update' an object, but make it difficult or impossible to tell if something was __set__ with a value, or merely inherited a default.

nine_k 1y ago

This is a recurring idea: "Parse, don't validate". Previously: https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/ https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-va... (2019, using Haskell) https://www.lelanthran.com/chap13/content.html https://www.lelanthran.com/chap13/content.html (April 2025, using C)

jetrink 1y ago
The author credits Alexis King at the beginning and links to that post.

ThinkBeat 1y ago

And that is why there are plenty of parser generators so you dont have to write the parser yourself every time.

bsoles 1y ago

>> // This is a parser >> const port = option("--port", integer()); I don't understand. Why is this a parser? Isn't it just way of enforcing a type in a language that doesn't have types? I was expecting something like a state machine that takes the command line text and parses it to validate the syntax and values.

hansvm 1y ago
The heavy lifting happens in the definitions of `option` and `integer`. Those will take in whatever arguments they take in and output some sort of `Stream -> Result<Tuple<T, Stream>>` function. That might sound messy but to the author's point about parser combinators not being complicated, they really don't take much time to get used to, and they're quite simple if you wanted to build such a library yourself. There's not much code (and certainly no magic) going on under the hood. The advantage of that parsing approach: It's reasonably declarative. This seems like the author's core point. Parser-combinator code largely looks like just writing out the object you want as a parse result, using your favorite combinator library as the building blocks, and everything automagically works, with amazing type-checking if your language has such features. The disadvantages: 1. Like any parsing approach, you have to actually consider all the nuances of what you really want parsed (e.g., conditional rules around whitespace handling). It looks a little to me (just from the blog post, not having examined the inner workings yet) like this project side-stepped that by working with the `Stream` type as just the `argv` list, allowing you to be able to say things like "parse the next blob as a string" without also having to encode whitespace and blob boundaries. 2. It's definitely slower (and more memory-intensive) than a hand-rolled parser, and usually also worse in that regard than other sorts of "auto-generated" parsing code. For CLI arguments, especially if they picked argv as their base stream type, those disadvantages mostly don't exist. I could see it performing poorly for argv parsing for something like `cp` though (maybe not -- maybe something like `git cp`, which has more potential parse failures from delimiters like `--`?), which has both options and potentially ginormous lists of files; if you're not very careful in your argument specification then you might have exponential backtracking issues, and where that would be blatantly obvious in a hand-rolled parser it'll probably get swept under the rug with parser combinators.

lihaoyi 1y ago

That's basically what my MainArgs Scala library does: take either a method definition or class structure and use it's structure to parse your command line arguments. You get the final fields you want immediately without needing to imperatively walk to args array (and probably getting it wrong!) https://github.com/com-lihaoyi/mainargs https://github.com/com-lihaoyi/mainargs

SloopJon 1y ago

I don't see anything in the post or the linked tutorial that gives a flavor of the user experience when you supply an invalid option. I tried running the example, but I've forgotten too much about Node and TypeScript to make it work. (It can't resolve the @optique references.) What happens when you pass --foo, --target bar, or --port 3.14?

macintux 1y ago
I had a similar question: to me, the output format “or” statement looks like it might deterministically pick one winner instead of alerting the user that they erred. A good parser is terrific, but it needs to give useful feedback.
- Dragging-Syrup 1y ago
  Absolutely; I think calling the function xor would be more appropriate.

SoftTalker 1y ago

I like just writing functions for each valid combination of flags and parameters. Anything that isn’t handled is default rejected. Languages like Erlang with pattern matching and guards make this a breeze.

AfterHIA 1y ago

You've got to be careful; if you validate the CLI too much you might get URA in your validator. #chugalug #house

12_throw_away 1y ago

I like this advice, and yeah, I always try to make illegal states unrepresentable, possibly even to a fault. The problem I run into here is - how do you create good error messages when you do this? If the user has passed you input with multiple problems, how do you build a list of everything that's wrong with it if the parser crashes out halfway through?

ambicapter 1y ago
Most validation libraries worth their salt give you options to deal with this sort of thing? They'll hand you an aggregate error with an 'errors' array, or they'll let you write an error message "prettify-er" to make a particular validation error easier to read.
- Thaxll 1y ago
  This work if all errors are self contained, stoping at the first one is fine too.
- pmarreck 1y ago
  Right, but that's validation, and this article is talking about parsing (not validating) into an already-correct structure by making invalid inputs unrepresentable. So maybe the reason why they were able to reduce the code is because they lost the ability to do good error reporting.
  - jpc0 1y ago
    How is getting an error array not making invalid input unrepresentable. You either get the correctly parsed data or you get an error array. The incorrect input was never represented in code, vs a 0 value being returned or even worse random gibberish. A trivial example: 1/0 should return DivisionByZero not 0 or infinity or NaN or whatever else. You can then decide in your UI whether that is a case you want to handle as an error or as an edge case but the parser knows that is not possible to represent.
  - lmm 1y ago

suff 1y ago

[dead]

andrewguy9 1y ago

Docopt! http://docopt.org/ http://docopt.org/ Make use of the usage string be the specification! A criminally underused library.

fragmede 1y ago
My favorite. A bit too much magic for some, but it seems well specified to me.
tomjakubowski 1y ago
A great example of "declaration follows use" outside of C syntax.

sudahtigabulan 1y ago

Is there no getopt implementation for Typescript? The input this library tries to handle better looks to me like bad design. "options that depend on options" should not be a thing. Every option should be optional. Even if you have working code that can handle some complex situation, this doesn't make the situation any less unintuitive for the users. If you need more complex relationships, consider using arguments as well. Top level, or under an option. Yes, they are not named, but since they are mandatory anyway, you are likely to remember their meaning (spaced repetition and all that). They can still be optional (if they come last). Sometimes an argument may need to have multiple parts, like user@host:port You can still parse it instead of validating, if you want. > mutually exclusive --json, --xml, --yaml. Use something like -t TYPE instead, where TYPE can be one of json, xml, or yaml. (Make illegal states unrepresentable.) > debug: optional(option("--debug")), Again, I believe it's called "option" because it's meant to be optional already. optional(optional(option("--common-sense"))) EOR

dwattttt 1y ago
> options that depend on options What would you do for "top level option, which can be modified in two other ways"? (--option | --option-with-flag1 | --option-with-flag2 | --option-with-flag1-and-flag2) would solve invalid representation, but is unwieldy. Something that results in the usage string [--option [--flag1 --flag2]] doesn't seem so bad at that point.
- sudahtigabulan 1y ago
  I think I've seen it done like that --option flag1,flag2 (Maybe with another separator, as long as it doesn't need to be escaped.) Another possibility is to make the main option an argument, like the subcommands in git, systemctl, and others: command option --flag1 --flag2 This depends on the specifics, though.
  - dwattttt 1y ago
    > --option flag1,flag2 Embedding a second parse step that the first parser doesn't deal with is done, but it's a rough compromise. It feels like the difficulty in dealing with [--option [--flag1 --flag2]] Is more to do with its expression in the language parsed to, than CLI elegance.

dcre 1y ago

Some other libraries I’ve been enjoying building CLIs with in TS that do more or less the same thing, though perhaps with slightly worse composability than Optique: https://cliffy.io/ https://cliffy.io/ https://github.com/tj/commander.js https://github.com/tj/commander.js

curtisszmania 1y ago

[dead]

esafak 1y ago

The "problem" is that some languages don't have rich enough type systems to encode all the constraints that people want to support with CLI options. And many programmers aren't that great at wielding the type systems at their disposal.

m463 1y ago

This kind of stuff is what makes me appreciate python's argparse. It's a genuine pleasure to use, and I use it often. If you dig a little deeper into it, it does all the type and value validation, file validation, it does required and mutually exclusive args, it does subargs. And it lets you do special cases of just about anything. And of course it does the "normal" stuff like short + long args, boolean args, args that are lists, default values, and help strings.

MrJohz 1y ago
Actually, I think argparse falls into the same trap that the author is talking about. You can define lots of invariants in the parser, and say that these two arguments can't be passed together, or that this argument, if specified, requires these arguments to also be specified, etc. But the end result is a namespace with a bunch of key-value pairs on it, and argparse doesn't play well with typing systems like mypy or pyright. So the rest of the tool has to assume that the invariants were correctly specified up-front. The result is that you often still this kind of defensive programming, where argparse ensures that an invariant holds, but other functions still check the same invariant later on because they might have been called a different way or just because the developer isn't sure whether everything was checked where they are in the program. What I think the author is looking for is a combination of argparse and Pydantic, such that when you define a parser using argparse, it automatically creates the relevant Pydantic classes that define the type of the parsed arguments.
- sgarland 1y ago
  Precisely my thought. I love argparse, but you can really back yourself into a corner if you aren’t careful.
- hahn-kev 1y ago
  It's almost like you want compile time type safety
  - MrJohz 1y ago
    You can have that with Mypy and friends in Python, and Typescript in the JS world. The problem is that older libraries often don't utilise that type safety very well because their API wasn't designed for it. The library in the original post is essentially a Javascript library, but it's one designed so that if you use it with Typescript, it provides that type safety.

jiggawatts 1y ago

This is one of the many reasons I like PowerShell: it parses strongly typed parameters for you and outputs human readable error messages for every kind of validation failure.

adamddev1 1y ago

Yay for parser combinators in the JS/TS wild!

brabel 1y ago
Exactly, the author's library is just a parser combinator [1] that specializes in providing constructs mirrorring CLI options. [1] https://en.wikipedia.org/wiki/Parser_combinator https://en.wikipedia.org/wiki/Parser_combinator

throwaway984393 1y ago

[dead]

bvrmn 1y ago

A valid type for server and port should be a single value. Stop parse it separately please. ":3000" -> use port 3000 with a default host. "some-host" -> use host with a default port. "some-host:3000" -> you guess it. It also allows to extend it to other sources/destinations like unix domain sockets and other stuff without cluttering your CLI options. Also please consider to use DSN or URI to define database configurations. Host, port, dbname, credentials as separate options or environment variables are quite painful to use.

slifin 1y ago

So use Clojure Spec or better yet Malli to parse your input data at the edges of your program Makes sense, I think a lot of developers would want to complect this problem with their runtime type system of choice without considering the set of downsides for the users

bschwindHN 1y ago

Rust with Clap solved this forever ago. Also - don't write CLI programs in languages that don't compile to native binaries. I don't want to have to drag around your runtime just to execute a command line tool.

majorbugger 1y ago
I will keep writing my CLI programs in the languages I want, thanks. Have it crossed your mind that these programs might be for yourself or for internal consumption? When you know runtime will be installed anyway?
- dcminter 1y ago
  You do you, obviously, but "now let npm work its wicked way" is an offputting step for some of us when narrowing down which tool to use. My most comfortable tool is Java, but I'm not going to persuade most of the HN crowd to install a JVM unless the software I'm offering is unbearably compelling. Internal to work? Yeah, Java's going to be an easy sell. I don't think OP necessarily meant it as a political statement.
  - goku12 1y ago
    There should be some way to define the CLI argument format and its constraints in some sort of DSL that can be compiled into the target language before the final compilation of the application. This way, it can be language agnostic (though I don't know why you would need this) without the need for another runtime. The same interface specification should be able to represent a customizable help/usage message with sane defaults, generate dynamic tab completions code for multiple shells, generate code for good quality customizable error messages in case of CLI argument errors and generate a neatly formatted man page with provisions for additional content, etc. In fact, I think something like this already exists. I just can't recollect the project.
    - craftkiller 1y ago
      docopt: https://github.com/docopt/docopt https://github.com/docopt/docopt

deleted 1y ago

[deleted]

panzi 1y ago

No mention of yargs?

globular-toast 1y ago

Not all of this validation belongs in the same layer. A lot of the problems people seem to have is due to people thinking it all has to be done in the I/O layer. A CLI and an API should indeed occupy the same layer of a program architecture, namely they are entry points that live on the periphery. But really all you should be doing there is lifting the low byte stream you are getting from users to something higher level you can use to call your internals. So "CLI validation" should be limited to just "I need an int here, one of these strings here, optionally" etc. Stuff like "is this port out of range" or "if you give me this I need this too" should be handled by your internals by e.g. throwing an exception. Your CLI can then display that as an error message in a nice way.

AndrewDucker 1y ago

This is one of the things that makes me glad that PowerShell does all of this intrinsically. I define the parameters, it makes sure that the arguments make sense and match them (and their validation).

einpoklum 1y ago

Exactly the opposite of this. We should parse the command-line using _no_ strict types. Not even integers. Nothing beyond parsing its structure, e.g. which option names get which (string) values, and which flags are enabled. This can be done without knowing _anything_ about the application domain, and provide a generic options structure which is no longer a sequence of characters. This approach IMNSHO is much cleaner than the intrication of cmdline parser libraries with application logic and application-domain-related types. Then one can specify validation logic declaratively, and apply it generically. This has the added benefit - for compiled rather than interpreted library - of not having to recompile the CLI parsing library for each different app and each different definition of options.

MrJohz 1y ago
Can you give some examples of this working well? It certainly goes against all of my experience working with CLIs and with parsing inputs in general (e.g. web APIs etc). In general, I've found that the quicker I can convert strings into rich types, the easier that code is to work with and the less likely I am to have troubles with invalid data.
- einpoklum 1y ago
  Think of it this way: Your code for quickly converting things into rich types - just run it on a map of argument-to-string value map rather than on a sequence of characters. It's still "quick": This is just like we current get an array-of-strings instead of a single-string command-line; it's initial domain-agnostic parsing, which can't even fail.
bakkoting 1y ago
This is the approach taken by node's built-in argument parser util.parseArgs.

jappgar 1y ago

I really think parse don't validate gives people a false sense of security (particularly false in dynamic languages like javascript and python). "Well, I already know this is a valid uuid, so I don't really need to worry about sql injection at this point." Sure, this is a dumb thing to do in any case, but I've seen this exact thing happen. Typesafety isn't safety.

yakshaving_jgt 1y ago
Type safety is absolutely some degree of safety. And I don’t know why anyone would think parsing a value into a type that has fewer inhabitants would absolve them of having to prevent SQL injection — these are orthogonal things. The quote here — which I suspect is a straw man — is such a weird non sequitur. What would logically follow from “I already know this is a valid UUID” is “so I don’t need to worry about this not being a UUID at this point”.
- jappgar 1y ago
  In python or typescript, the most popular languages in the world, it offers no runtime safety. Even in languages like Haskell, "safety" is an illusion. You might create a NumberGreaterThanFive type with smart constructors but that doesn't stop another dev from exporting and abusing the plain constructor somewhere else. For the most part it's fine to assume the names of types are accurate, but for safety critical operations it absolutely makes sense to revalidate inputs.
  - yakshaving_jgt 1y ago
    > that doesn't stop another dev from exporting and abusing the plain constructor somewhere else. That seems like a pretty unfair constraint. Yes, you can deliberately circumvent safeguards and you can deliberately write bad code. That doesn't mean those language features are bad.

foundart 1y ago

The author of the article also wrote a CLI parser library for Typescript, called Optique. I really appreciate them including a "When Optique makes sense" section in the docs. It would be great if more projects did that. https://optique.dev/why#when-optique-makes-sense https://optique.dev/why#when-optique-makes-sense

nickdothutton 1y ago

It’s been about 30 years but I seem to remember the compiler taking care of this for me (in Ada) with types.

kiliancs 1y ago

Great project. Clear goal, well executed, very nice API (safe, terse, clear). I use Effect CLI https://github.com/Effect-TS/effect/tree/main/packages/cli https://github.com/Effect-TS/effect/tree/main/packages/cli for the same reasons. It has the advantage of fitting within the ecosystem. For example, I can reuse existing schemas.

baroninthetrees 1y ago

I too got tired of dealing with cli arg parsing and am experimenting with passing a natural language description of the program and its args to a tiny LLM to sort out, offer suggestions (did you mean?), types conversions, etc. So far, it’s working great and given enough detail is deterministic.

AnimalMuppet 1y ago

Well, they're dictating that if you want them to use it, do it this way. Some people want others to use the programs they write; for such people, the GP actually has been given the right to have some valid say in the matter. Why CLIs in particular? Because they usually are smaller tools. For a big, important tool, you might be willing to jump through more hoops (installing the right runtime), but for a smaller, less important tool, it's just not worth it.

geon 1y ago

I just recently implemented my own parser combinator lib in typescript too. It was surprisingly simple in the end. This function parses a number in 6502 asm. So `255` in dec or `$ff` in hex: https://github.com/geon/dumbasm/blob/main/src/parsers/parseNumber.ts https://github.com/geon/dumbasm/blob/main/src/parsers/parseN... I looked at several typescript libraries but they all felt off. Writing my own at least ensured I know how it works.

amterp 1y ago

Very much agree with the article, this is one of the reasons why I wrote Rad [0], which people here might find interesting. The idea is you write CLI scripts with a declarative approach to script arguments, including all the constraints on them, including relational ones. So you don't write your own CLI validation - you declare the shape that args should take, let Rad check user input for you, and you can focus your script on the interesting stuff. For example args: username str # Required string password str? # Optional string token str? # Optional auth token age int # Required integer status str # Required string username requires password // If username is provided, password must also be provided token excludes password // Token and password cannot be used together age range [18, 99] // Inclusive range from 18 to 99 status enum ["active", "inactive", "pending"] Rad will handle all the validation for you, you can just write the rest of your script assuming the constraints you declared are met. [0]: https://github.com/amterp/rad https://github.com/amterp/rad