10 ms·
Show HN: Verify JSON using minimal schema
- Eikon 7y agoIt's crazy the amount of work that is spent to essentially implement static typing into dynamically / duck typed languages. Projects like this, mypy and others makes the whole thing bizarre to say the least.
- BiteCode_dev 6y agoI used to dislike type hints for Python because: - I was afraid it would change the language culture - It was not comfortable to use - The idea and design was, as you said, bizarre - Use use a dynamic language, why the hell would I want that? Years after the fact, I'm really happy about it. First, most of my Python code don't use them, so Python stays python. But if I ever need them, there are here. Of course, you could argue I should just use a language that has been designed for that since I need type. But that's missing the point: I don't usually start using hints. I code regular Python, because it's awesome. I just sometimes add type hints after the fact as a bonus. The ability to do that is fantastic, even if the type system is far from perfect. I want to code with Python most of the time anyway. I personally have very few use cases for another language, doing mostly web stuff. Type hints are still very verbose, and convoluted, although it will get better with 3.9 thanks to 2 PEP targeting type hints ergonomics. So sure, it's never going to fit perfectly, but I'm glad it's here. Now for a JSON schema, I guess it's the same deal. You use JSON because it fits your use case, and one day you realize your situation could now benefit from more robustness, and you add it to the mix. Again, you could argue you should have though of that at the beginning, but projects don't have frozen requirements, they evolve. And often, you want to start lean, because it will most probably stays that way, and otherwise would be so costly the project would never grow to a point you would need robustness. Now in this particular case, this lib is not just checking type, but also arbitrary logic. And making sure data is consistent is necessary no matter how dynamic or static your language is. There is a limit to the constraints your type system can represent.
- thrwaway69 6y agoHave you ever checked dart? Type inference can get you so so far
- BiteCode_dev 6y agoYes. Flutter is a killer app. But then again, Django as well.
- dehrmann 6y agoI led a project for validating customer-provided data against a json schema. The structure of the data was moderately complex, and it was getting bound to pojos by a pretty lenient binder. This caused issues when making additions to the API since customers were sending arbitrary fields, and duck typing the data wasn't that reliable. The easiest thing to validate before binding with a json schema. What's different this and Python type hints is that this was a public-facing API, so you have far less control over what data you'll see, and the server code was Java, so getting things strongly typed and validated early on makes your life easier.
- BiteCode_dev 6y agoSchema validation is more than type checking, and has been very well solved by libraries like marshmallow and pydantic for a long time. I don't think the comparison holds.
- dehrmann 6y agoI've worked with Python professionally for 3-5 years, but only type-hinted Python for ~5 months. This is coming from someone who prefers Java, but I don't like Python type hints because they were bolted onto the language worse than classes, type hierarchies aren't mature, and if I wanted types, I'd be using Java. That said, you do need strong typing for a project once it hits a certain line count or developer count.
- BiteCode_dev 6y ago
- mhd 6y agoIt's not like bringing features you get for free from dynamic languages into static languages doesn't involve an crazy amount of work, either.
- junke 6y agoDeclaring and checking the type of values is still dynamic typing, I see no static types here.
- catlifeonmars 6y agoThis is orthogonal to type safety. In fact, schema-based validation is an excellent way to perform type narrowing even in statically typed languages. An extremely prolific example of this is parsing incoming JSON from network requests :)
- ken 6y agoWhat statically typed language lets me define a type as “number between -180 and +180”, or “string which contains only alphanumeric chars”? I think that would be a great feature but from what I see static typing fails here, too.
- JoeAltmaier 6y agoBeen missing this for 20 years.
- wry_discontent 6y agoIt's not statically typed, but you can do this with clojure.spec (s/def ::number #(<= -180 % 180)) (s/def ::my-string (s/and string? #(re-matches #"[a-zA-Z0-9]*" %)))
- arethuza 6y agoI think Pascal's subranges would cover both those examples? NB Last time I used Pascal was probably 35 years ago.... Edit: I would suspect Ada would have something like this.
- metalrain 6y agoI think Idris does, see http://docs.idris-lang.org/en/latest/tutorial/typesfuns.html#the-finite-sets http://docs.idris-lang.org/en/latest/tutorial/typesfuns.html... But probably there is some horrible trade-off or limits that seem arbitrary.
- rraval 6y ago> What statically typed language lets me define a type as “number between -180 and +180”, or “string which contains only alphanumeric chars”? Pretty much all of them. Any simple predicate like this can be encoded with witness types. Here's an example in Java, which is hardly the paragon of static typing (i.e. it's no Haskell/Idris/Agda/Rust/Typescript): class AlphaNumericString { private final String str; // use a fallible factory with a `private` constructor if you're // morally opposed to exceptions public AlphaNumericString(String str) throws AlphaNumericException { if (!str.matches("^[a-zA-Z0-9]*$")) { throw new AlphaNumericException(); } this.str = str; } private static class AlphaNumericException extends Exception { } } Now code can freely use `AlphaNumericString` and be guaranteed that it has been validated. You may object and say that newtype wrapping is cumbersome but: 1. That's an argument about sugar and ergonomics, not about the semantics that the static type system enforces 2. Some languages make it easier to generate forwarding methods to the underlying type (a la https://kotlinlang.org/docs/reference/delegation.html https://kotlinlang.org/docs/reference/delegation.html) 3. The `AlphaNumericString` is describing a smaller set of values than `String`. In general, you should be strongly considering the methods you allow and make sure that all paths continue to enforce the semantics you intend.
- lewiscollard 6y agoThis has little to do with dynamically typed languages, "JS" in the name notwithstanding. JSON is a serialisation format that is used in both statically & dynamically-typed languages. It makes as much sense to call XML validators bizarre as it does to pin this project as a weird byproduct of dynamically typed languages. If your program produces or consumes JSON _and_ you think it would be a good idea to ensure said JSON is what you are/a client is expecting, then you need something like this in any implementation language.
- baq 6y agopreach. what I completely don't understand is the current fad on everything-should-be-yaml where yaml is marginally easier to write than XML, a tiny bit more than marginally easier to read (caveat emptor - Norway has a country code) and otherwise exactly as broken. (i'm talking about the 'safe' variant.)
- globular-toast 6y agoI don't think you understand static typing and you might be confusing dynamic typing with duck typing or other concepts. The presence of types doesn't mean static typing. Python, for example, is strongly typed, but it's dynamic because the type of any object can't be determined until run time. But you can still check types!
- tylerchr 7y agoVery neat. I had a similar need recently which led to a similar solution[^1]. Notably, I used a syntax nearly identical to yours, albeit without the very handy pluggable validators yours has. Nice job. What led you to choose “!” as the “optional” modifier? My intuition would have guessed that character to have the opposite meaning. [1]: project: https://github.com/tylerchr/jstn/ https://github.com/tylerchr/jstn/ — playground: https://tylerchr.github.io/jstn/ https://tylerchr.github.io/jstn/
- willvarfar 6y agoA long time ago, before MyPy, I did a python schema checker for JSON. It grew into argument checking too: https://pypi.org/project/obiwan/ https://pypi.org/project/obiwan/ Simple stuff like: Person = { "name": str, "age": int } The basic idea was to use a file full of obiwan types to document the rest api. The schema was both human-readable and valid python. The api could then trivially check the schema. Once the schema is checked, of course, its safe to go walk the json knowing that things aren't going to crash on you etc. It worked great. Still works great. I still prefer the syntax to that of MyPy. No real adoption, of course :) But fun to remember what could have been :)
- wryun 6y agoMy vaguely similar thing - https://wryun.github.io/yajsonschema/ https://wryun.github.io/yajsonschema/
- philliphaydon 7y agoWow this is very clean!
- eventreduce 6y agoI do not think so. If you check the example schema, it is very hard to understand it. What does ':lat' mean? is it a string or a number? What does '!b' mean, can it only be false?
- philliphaydon 6y agoIf you read the whole example: lat = custom validator b = shorthand for boolean (as is s for string) ! = optional So for me reading the whole example its very easy to understand and digest.
- eventreduce 6y agoYes, reading the docs explains what these keywords mean. But by just looking at the schema it is impossible to understand what that is. And if you check the definition of clean code which is something like "intuitive understandable" then it comes clear that this is not clean.
- philliphaydon 6y agoAh I see where you’re coming from. Good point.
- catlifeonmars 6y agoThe “!” for optional is the only thing that confused me. Why not “?” for optional?
- yusufnb 6y agoUpdated to ? in 1.0.3
- crtlaltdel 7y agoi saw this and was thinking “hrm...this is the sorts thing i use lodash for...if i even need it”. and now i see lodash is a dependency.
- ddoolin 6y agoIt also exposes itself as a Lodash mixin, which I have never used nor seen, so that's something learned.
- gunn 7y agoWhy are schemas strings? It would seem much more natural to make them JSON structures since they're almost that anyway.
- ZenPsycho 6y agothen you might as well use jsonschema.
- mikl 6y agoIndeed, there seems to be no reason to use this instead of JSON Schema.
- yusufnb 6y agoStrings make it easy to define as configs and can be implemented uniformly across different languages. In world of microservices, a Python and a Go service can easily share the schema definitions via some config files. Also strings provided the best minimalist syntax.
- therufa 6y agoThere has been a schema specification for json already. it's called conveniently `json-schema` https://json-schema.org/ https://json-schema.org/
- eventreduce 6y agoI think the author knew this but wanted a more minimal type of validation.
- arethuza 6y agoJson schema does scale down pretty well (indeed as the link above shows the simplest possible Json schema is just "{}") - it also has pretty good tool support (e.g. VS Code) - there is a Json schema for Json schemas which helps a lot. The main advantage of the approach in the article appears to be that its easy to extend validation with code - which is a nice touch. Pretty much any complex Json validation I have written has to combine both schema based validation and code based checks - usually done completely separately.
- eventreduce 6y agoJson-schema itself scales down pretty well. But the libraries for validating it do not. Even if your schema is only {}, the build size increment because of pulling in a jsonschema validator is enormous because of all these supported features.
- arethuza 6y agoFair enough I guess - don't think I've ever had a need to do client side JSON validation against a 'schema' so I've never had that concern.
- xg15 6y ago> Json schema does scale down pretty well (indeed as the link above shows the simplest possible Json schema is just "{}") I've not used JSON Schema so far, but I think showing the simplest possible definition does not tell much about how complex or easy to use a language is. E.g., it's easy to write C code for a working program that does nothing or just prints "Hello World". The resulting code is very simple and can be written without much experience in C. However that doesn't make C a simple language - and indeed writing programs in C that perform useful tasks is a lot more complex and requires more knowledge about the language. In the same way, it's nice that Json Schema has a compact syntax for the "do nothing" schema, but I think a comparison between practically used examples would be better. (Also the "do nothing" schema would have to compare against not doing schema validation at all - and compared to that, even the two brackets are still pretty costly)
- suref 6y agoI built something similar for my latest webapp to validate json-requests in python, the syntax is like this: { "content": Required(str), "username": Required(str, validate=username_is_ok, transform=lambda x: x.lower(), fail_message="Username isn't valid"), "message": Optional(str), "some_list": Required({ "name": Required(str), "date": Required(str) }, is_list=True) } Then I can provide this in a hook to the request method.
- 1f97 6y agocan you tell me a bit more about this? i wanted something like this for a small api i have but ended up using marshmallow to validate jsons against defined schemas which i think is a bit overkill.
- kingosticks 6y agoIf you haven't seen, these alternatives might also be helpful: * https://github.com/keleshev/schema * https://github.com/alecthomas/voluptuous * https://github.com/Pylons/colander
- 1f97 6y agothanks, i will take a look!
- oefrha 6y agoExclamation mark for optional is just weird. Question mark would be a lot more intuitive, unless there’s an origin story here I’m not aware of.
- angry_cactus 6y agoNot sure of the exact motivation, but Swift uses exclamation mark to unwrap optionals.
- oefrha 6y agoIn Swift optional types are marked by a question mark (exactly as I suggested). Unwrapping an optional makes it non-optional, so the exclamation point has the opposite meaning.
- yusufnb 6y agoIts updated now to work with ? syntax
- deleted 6y ago[deleted]
- tobyhinloopen 6y agoCool another syntax to learn! -nobody
- tckr 6y agoJSON schema and ajv are my go to tools for this: https://github.com/epoberezkin/ajv https://github.com/epoberezkin/ajv
- jacekm 6y agoKarate[1] has its own dsl for describing both json and xml. The output is short, but not very readable. For people new to the project it's rather cryptic. It is easy to maintain though once your team becomes fluent in Karate. [1] https://github.com/intuit/karate https://github.com/intuit/karate
- catlifeonmars 6y agoThis has already been said in the nested comments, but why not “?” for optional? Coming from Typescript, Swift, etc “!” feels more like an assertion of non-nullity (is that a word?). It’s also easy to read as “not <key>”.
- ivanhoe 6y agoThis is OK for simple structures, but in a deeply nested structures and larger APIs you'd probably also need some equivalent of json-schema $refs and ability to reuse blocks in your api schema.
- verdverm 6y agoEnter https://cuelang.org https://cuelang.org
- smoyer 6y agoMaybe I'm in the minority but I don't want light-weight at the expense of compatible/standard. It's being recognized that validation is more important than we'd expected - the recent ACM guidelines for moderation on Postel's Law really resonated with my experiences with RESTful/JSON micro-services (https://queue.acm.org/detail.cfm?id=1999945 https://queue.acm.org/detail.cfm?id=1999945). But if we're going to the effort of validating, let's do it in a language/framework agnostic way ... jsonschema.org is mentioned in other comments but I'm a fan of the OpenAPI initiative (https://www.openapis.org/ https://www.openapis.org/). Then we can all share the validation rules!
- yusufnb 6y agoI understand. In my experience, the usability of those standards is pretty bad. Which in effect leads to projects not using anything at all. One of the benefit of JSON over XML is that it is concise and fast to work with. The standard should reflect that part as well. With this lib, as a developer to verify a JSON for a simple REST request is as simple as - `verify(json, "{a,b,c}")`
- smoyer 6y ago> leads to projects not using anything at all We're counting that against products when we're analyzing what services to adopt/purchase. You doing nothing at all ... or something non-standard results in everyone else having to attempt to do it for you with the resulting inconsistencies. I'm not sure why you think OpenAPI is tailored for XML ... Here's the specification for the sample application (petstore) with service and model definitions in YAML - https://github.com/OAI/OpenAPI-Specification/blob/master/examples/v3.0/petstore.yaml https://github.com/OAI/OpenAPI-Specification/blob/master/exa....
- yusufnb 6y agoI think we are talking about different things here. I love OpenAPI and wouldn't do a project without it. verify-json is a lightweight way to validate JSON schema and does not intend to change/improve anything about OpenAPI. This is more for client-server JSON schema validations, comes handy when different teams are writing client and server side code. Especially in startups where iterations are rapid.
- andrenarchy 6y agoYou can also use fefe (https://github.com/paperhive/fefe/ https://github.com/paperhive/fefe/) to validate any data with its pure functional and minimalist approach. Bonus: it's 100% type-safe automatically if you use Typescript.
- Zinggi 6y agoYou shouldn't validate json. You should parse it, e.g. transform it into your desired data structure if it comes in in a valid shape. https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/ https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-va... I really like Elms way of Json decoding for this.
- guitarbill 6y agoI read the post. It seems to be mainly applicable to statically-typed languages. Apparently, the core problem with validation is "shotgun parsing": > Shotgun parsing is a programming antipattern whereby parsing and input-validating code is mixed with and spread across processing code—throwing a cloud of checks at the input [...] > The problem is that validation-based approaches make it extremely difficult or impossible to determine if everything was actually validated up front [...] Err, what? So validating against a well-defined schema won't necessarily cause this. But okay, again, I can buy the benefits for statically typed languages. There's more though: > Don’t be afraid to parse data in multiple passes. Avoiding shotgun parsing just means you shouldn’t act on the input data before it’s fully parsed [...] > Use abstract datatypes to make validators “look like” parsers. Sometimes, making an illegal state truly unrepresentable is just plain impractical given the tools Haskell provides, such as ensuring an integer is in a particular range. I've experienced this in Java/Jackson (which btw, proves this is not exactly new, sexy, or rare in the statically typed world). What is the suggestion for a dynamic language? In e.g. Javascript, classes will only get you so far, and seems needlessly heavyweight if you aren't going to get other benefits of type-safety. I really don't see how this is helpful, even after putting in the time to investigate this.
- Zinggi 6y agoIn JS, my preferred way of handling json is heavily inspired by the elm json decoding way. I want my decoders to essentially do 2 things: 1. Transform received data into a data structure that is best suited for my app. This might involve converting lists to objects, objects to sets, parse dates etc. 2. Only succeed on valid data This way, my application never has to deal with bad data. Also, I get to design the data structures I use, not the APIs I use. By only validating and not transforming, you are pushing more advanced validation further away from where the data was received (since you'll need to transform the data at some point anyway). As for libraries, I want composable parsers. For TS I'd probably use: https://github.com/paperhive/fefe/ https://github.com/paperhive/fefe/