17 ms·
Using JSON Schema to Document, Test, and Debug APIs
- mcheshier 8y agoWell written and informative. Thanks for posting/writing this!
- kemitchell 8y agoFor JavaScript, I highly recommend the ajv and tv4 npm packages.
- z3t4 8y agoIm a fan of running these tests live in production. A type check is cheap so it doesnt affect performence that much. You should not force correctness on others, meaning their code will fail because of your strictness. But you can force correctness on yourself so that your code never causes someone else's to fail. So make the schema validation or typecheck on outgoing responses, not only in testing, but also in production.
- brandur 8y agoChecking responses in production is also a good idea because it reveals all the edges that may not be exercised in an artificial test suite. There is still some performance cost for checking a response against a schema though, so a nice compromise is just to check a small % of outgoing responses — the vast majority of requests stay as fast as possible, but given a reasonable traffic load, it's still enough to eventually reveal any places the schema doesn't match reality. (This is an approach we use at Stripe for checking our OpenAPI specification.)
- sbr464 8y agoSide note @brandur, I always look forward to your blog posts, time for a new one!
- brandur 8y agoHah, thanks! :) I'll got a few queued up that I hope to get finished up pretty soon.
- foxtacles 8y agoWe are also validating responses against JSON schemas in our Ruby production apps. I can confirm it's a very helpful practice and I'm a big fan of it. Currently we are checking 100% of the responses. I even wrote this gem which uses native code to perform the schema validation to minimize the overhead in our endpoints: https://github.com/foxtacles/rj_schema https://github.com/foxtacles/rj_schema Validating our largest and most complex responses is taking <10ms, on average no more than 2-5ms which is quite affordable.
- theK 8y agoThe article describes some very good engineering practices. Nevertheless I’d suggest the json Schema itself is not the root of the documentation but a higher level integrated system like API Blueprints MSON. You still can produce Schema from it but it also can tie into a much better API design lifecycle.
- handrews 8y agoYes, JSON Schema describes documents/resources, not entire APIs on its own. Even Hyper-Schema (to the extent that it's implemented at all yet) is a resource-by-resource system, not an API-scope system.
- pingec 8y agoNot directly related but I like writing a json schema for my json configs. In a pinch I can get an interface for editing and validating it very cheaply with https://github.com/json-editor/json-editor https://github.com/json-editor/json-editor It's not perfect but a great time saver for me and users.
- pjmlp 8y agoAnd now the circle is complete, with the difference that XML Schema allows for proper comments.
- Pxtl 8y agoI never saw XML schema as a bad idea, just overdesigned and verbose, same as all things XML.
- pjmlp 8y agoI happen to think otherwise regarding XML, and rather enjoy using all high level tooling for XML processing.
- FigmentEngine 8y agorelax ng?
- edejong 8y agoAs opposed to underspecified, and ill thought out? To me it seems the XML era had better engineers making the standards.
- ivanhoe 8y agoWell, JSON Scheme has "title", "description", and "examples" keywords that are meant to be used for additional info, so not having comments is not really that big deal as with plain JSON. For XML Schema comments are essential as it's super verbose and hard to read. Of course, if you use it a lot you'll with time train your brain to scan through the forrest of XML tags without effort, just like in 90s we were able to easily make sense of multi-level nested tables - but to untrained eyes it's still a mess of tags with million options and it's the main reason why XML lost the popularity - too much noise and verbosity.
- handrews 8y agoJSON Schema has "$comment" as of draft-06 or -07 (I forget, which is funny b/c I was the one who added it).
- mendelk 8y agoThis is precisely the purpose of the OpenAPI Specification[0], an extended subset of the JSON Schema specification. Plus there a whole family of supporting tools, such as Swagger UI[1], and lots more[2]. [0] https://swagger.io/docs/specification/about/ https://swagger.io/docs/specification/about/ [1] https://swagger.io/tools/swagger-ui/ https://swagger.io/tools/swagger-ui/ [2] https://openapi.tools/ https://openapi.tools/
- why-el 8y agoWe went with this exact approach in my previous job, using OpenAPI to do everything the blog post mentions, including code generation, diffs for API consumers (published to Slack, for instance), and more. I believe this has a very good shot at success, but it's not complete yet. For instance we had to do a lot of in house work to make code generation work.
- wing328hk 8y agoNot sure which code generation tool you used but if you've time, please try OpenAPI Generator (https://openapi-generator.tech https://openapi-generator.tech) and let us know if you've any feedback.
- nathan_f77 8y agoI agree, I've also invested a huge amount of time and effort into my openapi-generator setup [1], and there's tons of room for improvement. We've probably even done some of the same work for code generation (custom Java subclasses, etc.) [1] https://github.com/OpenAPITools/openapi-generator https://github.com/OpenAPITools/openapi-generator
- handrews 8y agoHi- I'm one of the main editors of the JSON Schema specification, and one thing we are doing with the next draft is making it easier to build extensions for things like code generation. We expect to work with the OpenAPI folks on this as it is very relevant to their interests :-) Also, we and the OpenAPI Technical Steering Committee are actively working together to re-converge the specifications. OpenAPI 3 was developed while JSON Schema progress was stalled due to the prior editors leaving and a new group of us (eventually) picking it up. OpenAPI 3.1 will most likely include a keyword to allow using standard JSON Schema as an alternative to their customized syntax, and hopefully we can achieve full integration on OpenAPI 4. There are also some other ideas being explored for improved compatibility in 3.x. Standards work is hard, but the relationship between the OpenAPI TSC and the JSON Schema editors is quite healthy and we are making good progress.
- Existenceblinks 8y agoI was building an app using JSON Schema last year (until Feb this year) and gave up. I had read JSON Schema specification like 10 times and there are some undefined behaviors. I had read a lot of JSON Schema issues on Github, there is basically one man show and the others come to review mostly. After the point they added all logics to the spec, everything becomes a mess. Example 1: { "oneOf": [ {"type": "number"}, {"type": "number"} ] } 1 or 2 or 3 is an invalid input based on above schema. It doesn't make sense to me at first glance. (hint: it's XOR) --------- Example 2: { "oneOf": [ {"minimum": 0, "maximum": 10}, {"minimum": 5, "maximum": 20} ] } These ranges will pass: [0,4] and [11,20] But you would be surprise it doesn't reject string, bool, null. Basically it doesn't reject anything except [5,10] range. --------- Example 3: { "type": ["object", "array", "null"], "not": {} } Easy but confusing. This rejects every thing because comma means AND, and `"not": {}` means false. --------- Example 4: { "allOf": [ { "type": "object", "properties": { "a": {"type": "string"}, "b": {"type": "integer"} } }], "additionalProperties": false } A well-known problem. This rejects (all - {}) This is because the "allOf" is object AND outside it is an empty object. "properties" at level 0 is empty if not specified. So this schema only accept {}; an object AND is empty. They add a new keyword called "unevaluatedProperties" to solve this and I won't explain it to you! ----- If you just use basic stuff like draft-04, you will be fine. But I will NEVER touch this spec again! ----- EDIT: My app was sort of static analysis on schema, and this spec doesn't suppose to help doing thing like that.
- Existenceblinks 8y agoI also read this paper: https://martinugarte.com/media/pdfs/p263.pdf https://martinugarte.com/media/pdfs/p263.pdf (Foundations of JSON Schema) Also this: https://www.genivia.com/sjot.html#SJOT_versus_JSON_schema https://www.genivia.com/sjot.html#SJOT_versus_JSON_schema (JSON Schema problems) GOOD LUCK!
- awwright 8y agoIt seems like there's two slight misunderstandings: > "oneOf": [ {"type": "number"}, {"type": "number"} ] > { "oneOf": [ {"minimum": 0, "maximum": 10}, {"minimum": 5, "maximum": 20} ] } "oneOf" requires exactly one match (hence the name), typically you want to use "anyOf" or "allOf". And there's no benefit in putting two identical values inside them. > { "type": ["object", "array", "null"], "not": {} } > { "allOf": [ { "type": "object", "properties": { "a": {"type": "string"}, "b": {"type": "integer"} } }], "additionalProperties": false } JSON Schema is merely a list of assertions. Some test the type of the value (that's the "type" keyword), others test the value ranges within a single type. This way, you can allow values to be one of multiple types, e.g.: {type:["string","object"], minLength:1} means "Value must be a string or object; and if it's a string, it must have at least one character." Some of the assertions are spread across multiple keywords, "additionalProperties" depends on "properties", for example. So, {additionalProperties:false} means: if value is an object, then only an empty object is permitted. Some implementations can tell you if you're trying to do nonsensical things (like test the maximum length of a value that's only allowed to be a boolean), but that's up to the implementation to test for.
- dilyevsky 8y agoWhy not just use something with a natively supported schema at this point like protobuf/grpc or graphql?
- smt88 8y agoOpen API Spec (formerly known as Swagger) is purpose-built on top of (or slightly forked from) JSON Schema and has much wider support among API-related tools and libraries. I tried using JSON Schema for some internal (non-HTTP) APIs and found bugs/inconsistencies in tooling that made me abandon it.
- tnolet 8y agoI'm really torn whether to adopt JSON schema validation for a Saas that I run. pro: It validates API JSON responses based on an open spec. cons: It's absolutely terrible to work with. XML all over again. Documentation is terrible. Finding examples is, well you get it...
- handrews 8y agoNote that "Understanding JSON Schema" has been updated for draft-07 and is now under the aegis of the main JSON Schema project: https://json-schema.org/understanding-json-schema/ https://json-schema.org/understanding-json-schema/
- kiliancs 8y agoThis is a very powerful approach because there are json schema libraries available in all major languages. TypeScript is a better source for truth because it can represent things json schema can't. It is possible to generate a json schema from a TypeScript definition (see https://github.com/YousefED/typescript-json-schema https://github.com/YousefED/typescript-json-schema), which means TypeScript can be the single source of truth for API definitions, be used directly in the frontend and, in json schema form, used to test directly against API responses.
- atombender 8y agoThese days all my needs for "structured data" start with JSON Schema. I refuse to work with schemaless apps and databases anymore. The workflow which I use for one app is: * Define a JSON Schema for all the app's models * Generate static Go types from these definitions * Generate corresponding GraphQL types and inputs in the Go app for serving the API * Generate TypeScript types in JavaScript front ends against the same schema * Same model structs in Go are used to shuffle data in and out of data stores Since GraphQL is more limited than JSON Schema (for example, very limited union ("oneof") support), there are some features I'm not able to fully make use of, but the common denominator covers most use cases. I love having end-to-end static typing, validation and consistency.
- sergiotapia 8y agoThis seems really verbose? Very XML-y. What's wrong with simple https://github.com/omniti-labs/jsend https://github.com/omniti-labs/jsend ``` { status : "success", data : { "post" : { "id" : 1, "title" : "A blog post", "body" : "Some useful content" } } } ```
- awwright 8y agoHow is that simple? What does "status" mean? I'm personally familiar with the word "status"—but whatever that property is, I'd have to re-implement it in my user agent, which seems like a waste because my user agent already understands HTTP status codes. How is it simple to have to look at and interpret two different "status" fields? (And that's just the first property!) What part of the article is verbose? What's wrong with verbose?
- echelon 8y agoComing from protobuf land, this tool seems gross. It's XML schema all over again. Why are we trying to shoehorn a schema design language into a format that isn't good for human authoring? It's annoying to write JSON, yet the language masquerades as being human-readable. Frontend engineers should check out protobuf. It's an amazing data definition language that generates bindings in every language under the sun and has a compact binary serialization that is much more efficient than JSON both to encode/decode and transmit over the wire. JSON should die the same death XML did. It's so bad.
- cageface 8y agoBeing able to see JSON responses in browser dev tools is extremely helpful for development and debugging. Is there something similar for protobuf?
- carlmr 8y agoI concur, JSON is easy enough to read/write for humans, easier to parse than XML, too, it kind of hits the sweet spot in between protobuf and XML.
- hasperdi 8y agoProtobuf is a lower level interface. You can't rely on it for validation and use it for complex model / schema. For example oneof customer (individual / organisation) with different distinct fields. In protobuf, all these fields need to be defined.