14 ms·

JSON Schema bundling formalised

140 points by relequestual 5y ago

RustyRussell 5y ago
I use jsonschema to validate my projects' output, and also generate the documentation from it. The python jsonschema support makes this possible: our testsuite is in Python already. But it's an awkward fit, and I despair of having anyone else in my team write schemas: the default of allowing additional unspecified fields must be continually overridden, otherwise your schema has no teeth, and things like "if this field is this value, these additional fields exist" must then always have an "else" indicating that no additional fields exist. In summary, it's better than nothing, but it's not easy. I'm not sure that JSON is a great language to specify schemas in, sorry.
- tored 5y ago
  Yes, the unspecified fields is a nuisance. It is hard to manually look at JSON data and compare it visually to a JSON schema because the schema has a depth to every property and lots of other cruft, which makes it somewhat cumbersome to retrofit a JSON schema to existing JSON data. I once developed an alternative JSON schema internally for a company where the structure was the same as the JSON data (and default strict of course). We implemented multiple implementations for every language we used. It sort of worked, not as feature complete as the official JSON schema of course, but my conclusion is that JSON doesn't fit well for this.
- paulddraper 5y ago
  > otherwise your schema has no teeth I don't disagree, but FWIW, this is common deliberate choice. E.g. Protobufs work exactly the same way. Except unlike JSONSchema, there's no way to disable it. The reasoning is to permit future extensibility.
  - RustyRussell 5y ago
    For JSON itself, ignoring extra fields it makes sense. For validating my own output, it does not. And validating someone else's output seems counterproductive?
endisneigh 5y ago
I’m curious - what’s an argument against this that’s logically consistent with a position that supports databases that have schemas?
csmpltn 5y ago
We've come full circle back to XML SOAP. We should consider restricting the registration of .org domains to actual non-profit organizations, and restricting the use of words like "schema" and "standard" to things that have been fully certified as such by internationally accredited engineering bodies. This doesn't add anything on-top of the tools we've had 40 years ago. We're stuck with this everywhere now, all thanks to pure marketing.
- hyperpallium2 5y ago
  back to CORBA
  - ChrisMarshallNY 5y ago
    > CORBA Get off my lawn!
  - johnnycerberus 5y ago
    > CORBA "Well, it's high noon somewhere in the world."
  - DonHopkins 5y ago
    The World's Second Fully Modular Software Disaster! It's just riddled with features.
    - dnautics 5y ago
      what was the first fully modular software disaster?
      - DonHopkins 5y ago
        Glad you asked! ;) https://donhopkins.medium.com/the-x-windows-disaster-128d398ebd47 https://donhopkins.medium.com/the-x-windows-disaster-128d398... [...] X-Windows is the Iran-Contra of graphical user interfaces: a tragedy of political compromises, entangled alliances, marketing hype, and just plain greed. X-Windows is to memory as Ronald Reagan was to money. Years of “Voodoo Ergonomics” have resulted in an unprecedented memory deficit of gargantuan proportions. Divisive dependencies, distributed deadlocks, and partisan protocols have tightened gridlocks, aggravated race conditions, and promulgated double standards. [...] X: The First Fully Modular Software Disaster >X-Windows started out as one man’s project in an office on the fifth floor of MIT’s Laboratory for Computer Science. A wizardly hacker, who was familiar with W, a window system written at Stanford University as part of the V project, decided to write a distributed graphical display server. The idea was to allow a program, called a client, to run on one computer and allow it to display on another computer that was running a special program called a window server. The two computers might be VAXes or Suns, or one of each, as long as the computers were networked together and each implemented the X protocol. [...] X-Windows: …A mistake carried out to perfection. X-Windows: …Dissatisfaction guaranteed. X-Windows: …Don’t get frustrated without it. X-Windows: …Even your dog won’t like it. X-Windows: …Flaky and built to stay that way. X-Windows: …Complex non-solutions to simple non-problems. X-Windows: …Flawed beyond belief. X-Windows: …Form follows malfunction. X-Windows: …Garbage at your fingertips. X-Windows: …Ignorance is our most important resource. X-Windows: …It could be worse, but it’ll take time. X-Windows: …It could happen to you. X-Windows: …Japan’s secret weapon. X-Windows: …Let it get in your way. X-Windows: …Live the nightmare. X-Windows: …More than enough rope. X-Windows: …Never had it, never will. X-Windows: …No hardware is safe. X-Windows: …Power tools for power fools. X-Windows: …Putting new limits on productivity. X-Windows: …Simplicity made complex. X-Windows: …The cutting edge of obsolescence. X-Windows: …The art of incompetence. X-Windows: …The defacto substandard. X-Windows: …The first fully modular software disaster. X-Windows: …The joke that kills. X-Windows: …The problem for your problem. X-Windows: …There’s got to be a better way. X-Windows: …Warn your friends about it. X-Windows: …You’d better sit down. X-Windows: …You’ll envy the dead.
akie 5y ago
I might be exceptionally dense, but it's hard for me to see practical applications for something like this. In the end, if you implement this in your application, you will have a mechanism to say "this input document is invalid". AND THEN WHAT? Your only option is to discard it. I'd rather live by the old maxim "be liberal in what you accept, and strict in what you produce". But perhaps I'm overlooking an important use case here, in which I'd happily stand corrected.
- sorokod 5y ago
  liberal or not you can express your expectation in a well defined manner. As to "then what" its up to you, i'd go with some 4XX ( or other technology specific error) that implies that no processing has occured. All arguments for and against schemas in databases translate well to this case.
- 66fm472tjy7 5y ago
  There are things you cannot accept. What do you do if an order has a negative quantity, if you get a string instead of a number, if a value does not fit in your DB column, etc ? Having this as part of your interface definition, along with generated representations of the possible data in the language your choice[0] and automatic (so you can't forget it) validation that rejects invalid input with HTTP 400 bad request instead of 500 internal server error when the DB constraint fails (or even worse, you persist invalid data and something breaks later) is definitively useful. That said, OpenAPI fails at this somewhat as the people writing the spec don't consider the (de)serialization and validation libraries. Thus, OpenAPI keeps adding features/constraints that are not supported by the libraries and are thus cannot be automatically validated or even produce nonsense output on generation (e.g. in Java you might end up with raw Map or even Object fields) [0] https://openapi.tools/#sdk https://openapi.tools/#sdk
  - reaperducer 5y ago
    What do you do if an order has a negative quantity, if you get a string instead of a number, if a value does not fit in your DB column, etc ? Isn't that all just basic data validation and sanitation that we all do anyway? And wouldn't it still have to be done, even if the JSON came with all the schemabloat? I don't think I'm suddenly going to accept that the data presented to my function is valid and fit for purposes just because it goes through someone's JSON schema library. That's just passing the responsibility on to a third party, which feels lazy and dangerous to me. Maybe it'll be OK for systems that don't get unusual data. But mine are always ingesting strange things, and the only person who knows what's truly valid and what's not is me, not a third party. That's my responsibility as the programmer of the system. I looked through the web page, and it's probably great for someone. Somewhere. But I'd like to see that SKU example that was presented fully fleshed out with real-world type data. It seems OK if your data only consists of "productID," "productName," and "productTags." But in real life, SKUs are often incredibly complex, and this just seems to invite disaster by making complex systems even more complex. So much so that they require even more complex tools to manage them.
gone35 5y ago
History has not been kind to efforts like these.
- josteink 5y ago
  History has been repeating efforts like this.
- sorokod 5y ago
  Worked well for relational databases.
- speedbird 5y ago
  Software "engineering" repeatedly goes through the same loop: - something simple, easy to learn, easy to use - doesn't cover edge case X - lacks rigour - lets add a bunch of features - and committees and processes to manage that - this is really complicated and hard to learn and use, we need something simpler ...
sorokod 5y ago
Would be good if there was a clear statement about who the authors are and what makes their content authorative.
- oefrha 5y ago
  > what makes their content authorative. Nothing, it’s just relatively widely known and used (e.g. for VS Code and Windows Terminal config files, which I expect a lot of people to have experience with), with relatively good library support in a wide selection of languages. Although I should mention that most of the libraries are stuck on old versions, and the versions they’re stuck on are all over the place: https://json-schema.org/implementations.html https://json-schema.org/implementations.html
  - brabel 5y ago
    It's a horrible mess, made worse by the several years in which OpenAPI defined their own "dialect" of json-schema for their data schemas which was incompatible with the non-OpenAPI variant of json-schema, so you get libraries that support both, or support only a certain version of each, and because different versions tend to be completely incompatible with each other, you might run into lots of problems if you have more than one language consuming schemas, or even the same language but using different libraries due to transitive dependencies... it's just terrible, just use XML if you need a schema.
    - oefrha 5y ago
      The language is a godforsaken monstrosity, but if you only have to write it once for some non-critical validation task it’s acceptable. For instance, I use JSON schema to (optionally) validate some generated JSON data which is fed into a web app as a JSON module. I also generated the TS spec for it from the JSON schema (although writing the TS spec by hand would have been 10–100x easier than writing the JSON schema). XML is obviously not suitable here — TypeScript doesn’t have XML modules.
- relequestual 5y ago
  Do you mean the article or the specification?
chmod775 5y ago
A yes. Because things like "integer" totally need to be abstracted away into another schema. Couldn't have given this DSL built-ins for the things that are... you know... built into JSON already. Also bonus points for the apparent lack of shorthands, turning this language into a verbose word salad. I hope there's at least a line of reasoning that explains why some keys are prefixed with '$' and others aren't. Great way to turn something as beautifully simple as JSON into something abhorrent. Hey. You know what is better than JSON at being XML? XML.
- conceptme 5y ago
  afaik there is no integer type in json but a numeric type, and most languages do have different numeric types.
  - chmod775 5y ago
    Neither is 'non negative' - but that should also be a built-in, considering how verbose the language becomes otherwise. Now you're adding ~5 lines of boilerplate every time you want to have an integer, another 5 for positive integers, and so on.
    - michaelmior 5y ago
      To support non-negative integers, all you need is the following {"type": "integer", "minimum": 0} I think the format used in the example was used to demonstrate bundling, but if what you really want is a non-negative integer, the above is the simplest way to do it.
      - relequestual 5y ago
        This is correct. If I had used a more complex example, it would have been much harder to follow. This was the simplest. We factor this out for our meta-schema construction due to reuse.
- Someone 5y ago
jonny383 5y ago
If someone tried to push this on my engineering team, they would be laughed out of my office and ridiculed for the idiocy of even suggesting such garbage. The best factor of JSON is being a schema-less, human readable format. Imagine having someone present this monstrosity to you with a straight face, thinking it's a good use for JSON.
- sorokod 5y ago
  Would a library that validates conformance to the schema so that you don't have to handcraft error handling sweeten the proposition?
- NicoJuicy 5y ago
  To be honest, i used json schema for auto generating layouts. Eg. https://github.com/rjsf-team/react-jsonschema-form https://github.com/rjsf-team/react-jsonschema-form
- endisneigh 5y ago
  You’d laugh at someone for merely suggesting a technology? Seems like a toxic team If you think it’s bad surely an explanation would win them over and stop their efforts?
- Jenk 5y ago
  Do you prohibit the use of VSCode in your workplace? Anyone using that is using jsonschema. It's built into the applicatoin for its settings and configuration files. Plugins and extensions use it to manage their own config _and_ as validation for the editor's api.
  - vultour 5y ago
    Your editor has nothing to do with your product. People use PyCharm to write Python yet would "laugh out" anyone who suggests to start writing Java.
    - Jenk 5y ago
      So vscode product team should laugh at themselves? Do you get how ridiculous GGP is now?
jacobmischka 5y ago
I'd like to offer a contrasting opinion to all of the other currently negative comments: I've used JSON schema in the past to validate outside input and it was a pleasant and straightforward experience. There are unfortunately no standard type declarations that I'm aware of, which is a pain, but tools exist to translate schema.org schemas[1] (I have not used this myself). [1]: https://github.com/charlestati/schema-org-json-schemas https://github.com/charlestati/schema-org-json-schemas
- tored 5y ago
  When I used JSON schema last time a few years ago the different implementations of it in PHP and nodejs where either incorrect of feature incomplete. Seems like it is hard to get right.
  - relequestual 5y ago
    If only they had used used the official test suite... which is provided, free of charge, in JSON.
  - dolmen 5y ago
    The main problem is that JSON Schema is still an evolving design. All tools may not yet implement the latest draft.
- solids 5y ago
  I agree with you, I didn’t expect the amount of negative comments. Used in the past and it was a great tool to “avoid representing invalid states” and also as documentation to share between teams.
  - chrisjc 5y ago
    > I didn’t expect the amount of negative comments I did! It's been made clear time and time again on HN that "everyone" HATES XML and anything to do with it... especially XSD, WSDL, etc! I really don't see any difference between XML and JSON except the semantics (I lie, I see a lot of reasons why XML is more powerful). But I do get why people prefer JSON over XML for visual reasons alone. Since they're very similar logically/structurally, it only makes sense that the same things that are possible with XML are possible with JSON. And here we are, discussing JSON schemas! So is it really that they hate XML and love JSON for any other reason besides the visual ones? If not, what is it that they hate? I think it has to do with the responsibility and effort that goes into creating a contract, the "schema". All the anxiety and headbanging that goes into setting up all the meetings and discussions. All the convincing that happens over and over again just to move forward. All of the back-and-forth that goes into whether it's this type or that bc the requirements are ambiguous to begin with and now it's the developer's responsibility. "I could just throw this API together in a day if it wasn't for all these useless meetings. Why do I have to wait for Bob to update the schema when I can just cast this field to a boolean in my code?". "I don't want to have to deal with validation exceptions at runtime, just fix your damn code and read the wiki I put together." etc. (I do understand that not all APIs, data exchange files, tightly coupled project, etc... need schemas) In my experience, while these schema-less approaches lead to projects being leaner and completed sooner, the price is payed continuously there after. Something changes on the server-side, some edge case appears on the UI several weeks after release to prod. Enhance the structure, breaks ETLs. And so on. A perceptual state of keeping everything in tranquility emerges.
ChrisMarshallNY 5y ago
I generally prefer using JSON in my interactions, where possible. That's because it is lightweight, and 99.9% of the data I'm transferring is scalar. JSON is basically "implied" for scalar types. The good thing (if you want to call it "good") about XML, is Schema. Schema is a "rock hard" contract. It is definite, empirical, unambiguous. When I am looking at an API, and it has a Schema, then I know that I can figure out exactly what shape the data will take/emit. If possible, I still use JSON for the exchange. I just use the XML to figure out the specifics in the exchange. I've been working in XML forever, it seems. I've done ONVIF stuff, which is SOAP/WSDL-based, and even XSLT. I still like JSON. Mostly because I don't have to deal with Schema. Over the years, I've gotten fairly good at developing XML Schemas, but I have never gotten "used" to it, and still have to look everything up. If I develop APIs, I will often do an XML variant, alongside the JSON, because that forces me to write a Schema. Doing this, helps me to "code review" my schema, and also gives me a very convenient automated test hook. But, boy, I still hate XML Schema...
- tored 5y ago
  One thing I'm guilty of is designing a document format as JSON, thus the need to formally verify it. Passing data works well as JSON but documents works better as XML where you can use XML Schema.
  - ChrisMarshallNY 5y ago
    This is where I would also develop an XML variant, even if it is never to be used, simply so that I can have a Schema to go along with it. I'll start with a data structure/class, in the code, then use a built-in transformer, to turn it into JSON or XML. That way, I know that the "kernel" of the data is the same, between them. Even though JSON is a "natural" for scalar types, it can get weird, there. For example, let's say we have a float: { "kids": 1.0 } It needs to be parsed as a float, because we could have: { "kids": 2.3 } But I often see these sent as: { "kids": 1 } or { "kids": "1.0" } or even { "kids": "1" } And I have differing results, based on the parser.
    - 5y ago
deleted 5y ago
[deleted]
enriquto 5y ago
Great, now we can finally add the next step in "The Ascent of Ward". EDIT (for the lucky 10.000 of today): http://harmful.cat-v.org/software/xml/ http://harmful.cat-v.org/software/xml/
tasogare 5y ago
Still missing: namespaces, attributes and comments (only-half ironical comment).
- deleted 5y ago
  [deleted]
- ether_at_cpan 5y ago
  Comments are supported. There is a "$comment" keyword, and you are free to invent any other keyword you like, as they are generally ignored.
diegoperini 5y ago
I haven't stumbled upon a better schema for JSON than the Typescript definition files. If only I could use them in non Typescript contexts. https://www.typescriptlang.org/docs/handbook/declaration-files/deep-dive.html https://www.typescriptlang.org/docs/handbook/declaration-fil...
- YousefED 5y ago
  This is why I originally started a project to convert ts to json schema; https://github.com/YousefED/typescript-json-schema https://github.com/YousefED/typescript-json-schema (and also check out a good alternative https://github.com/vega/ts-json-schema-generator https://github.com/vega/ts-json-schema-generator)
- anentropic 5y ago
  you might like https://jsontypedef.com/ https://jsontypedef.com/ if you haven't seen it
- panzerklein 5y ago
  You can write validation schemas using zod library [1]. They end up looking pretty similar to typescript definitions (same vocabulary, same methods for combining/intersecting/filtering). And if you migrate to typescript you'll get type definitions for free out of those schemas. [1] https://github.com/colinhacks/zod https://github.com/colinhacks/zod
  - IggleSniggle 5y ago
    If you like zod, you should check out myzod: https://github.com/davidmdm/myzod https://github.com/davidmdm/myzod
theteapot 5y ago
Article: > "Developers of platforms and libraries that use OpenAPI haven't had such a shake up before, and my feeling is it may take more than a few releases to correctly implement all the new shiny features full JSON Schema has to offer." Dear OpenAPI, please avoid the shiny features.
- ether_at_cpan 5y ago
  Like which? Or are you just knee-jerking at the term "shiny" and presuming it means "over-complicated and unnecessary"?
rincewind 5y ago
I use json schema. I used to understand what it is and what it does. Now I don't.
theteapot 5y ago
> "There are several libraries which offer bundling solutions, however they all have caveats, and I haven't seen any to date which are fully JSON Schema aware." But what's wrong with just stuffing the schemas in a flat array or object exactly?
- relequestual 5y ago
  Expectation of a single schema by existing tooling.
  - theteapot 5y ago
    That's pretty vague.
netfl0 5y ago
Reinventing JSON-LD.
- aeberhart 5y ago
  There certainly are some similarities, but I think JSON-LD and and JSON Schema solve different problems. You might find this article helpful (I ended up writing this after being confused by the various standards in this space): https://dashjoin.medium.com/json-schema-schema-org-json-ld-whats-the-difference-e30d7315686a https://dashjoin.medium.com/json-schema-schema-org-json-ld-w...
segphault 5y ago
The way $ref resolution works is hideously complicated, to the point where many JSON Schema implementations just don't even bother supporting external $refs. It really should be a totally separate concern from validation, especially since how you store and compose schemas may end up being specific to a given use case.
- beardyw 5y ago
  Much as I value JSON Schema, building a composite schema, to use and reuse parts, can easily hit a brick wall. I often end up building each schema programmatically and exporting to JSON. Could do better.
atombender 5y ago
I'll be one of the few positive voices here, I guess. JSON Schema is pretty good. It's a relatively simple, extensible, pragmatic specification that supports validating the kinds of data that JSON can express. XML was killed by complexity, same as lots of technologies that preceded it, such as CORBA and SOAP. Developers don't like complexity. W3C tried to build an enormously complicated ecosystem of tools on top of XML. The specifications for XML Schema, WSDL, XSLT, etc. were gigantic. XML Schema and WSDL are both so big they are split into multiple parts. XSLT 3.0 is maybe 500 pages when printed. Many of these originated in the wrong sort of place, designed by committee and from inside big enterprises like IBM, rather than being adopted from evolving practices. With XML out of the way, we still need a way to represent structured data, and it turns out JSON is pretty good for that. JSON has problems, but its simplicity is what lead to it becoming so prevalent in the first place. Now, we also need a way to define the structure and format of that data, and JSON Schema is pretty good for that. I think JSON Schema could have been a bit simpler, but it's still nowhere near to making the same mistakes as XML Schema.
- arethuza 5y ago
  Strongly agree - I've worked with ONC RPC, CORBA, SOAP, XML & XSD, XSLT etc. and I much prefer the relatively straightforward nature of JSON and JSON Schema. Yes it's not perfect - but for me it is more than good enough. Edit: Of course, there is no direct equivalent of XSLT in the JSON world - which pretty much counts as a feature to me.
  - lolive 5y ago
    Nor XPath.
    - arethuza 5y ago
      I think XPath is actually one of the nicer bits of the XML universe - there is a JSON equivalent in JSONPath that I have used a bit.
  - espadrine 5y ago
    One thing I am wondering is whether the protobuf folks have a better development environment overall, or a worse one. Is the schema-to-self-validating-serialization-code approach going to win just as JSON won over XML?
dolmen 5y ago
Bundling for OpenAPI specification has long been a need for authors to allow to reduce duplication, and to allow to split a big specification in multiples files, but publish a single one. A few years ago I've written a tool to fit that niche: https://github.com/dolmen-go/openapi-preprocessor https://github.com/dolmen-go/openapi-preprocessor https://github.com/dolmen-go/openapi-preprocessor https://github.com/dolmen-go/openapi-preprocessor I have now to tweak it (well, it will be a major rewrite) to handle $ref relative to $id instead of the file location.
9814715843 5y ago
amit Ma ndal
run-types 5y ago
For those looking to sanitize input from Typescript, I highly recommend the runtypes library: https://github.com/pelotom/runtypes https://github.com/pelotom/runtypes It let's you specify type definitions a DSL in Typescript using syntax very similar to Typescript's type definitions. Once you define your types in the DSL, you get Typescript types and parsing / verification for free. Not as general purpose as JSON Schema, but 1000x cleaner and easier to use.
- dolmen 5y ago
  The point of JSON Schema is to have a common format independant of the programming language to ease interoperability.
- move-on-by 5y ago
  I've never seen runtypes, but I've seen io-ts [1]. Are you aware of the pros/cons between them? [1] https://gcanti.github.io/io-ts/ https://gcanti.github.io/io-ts/
  - run-types 5y ago
    They look roughly the same
corentin88 5y ago
JSON format is simple and easily readable. It’s now so common and probably one of most used language on earth. That’s great! Now, anyone can create a decent API with basic programming tool available. And everyone will be able to use it. Why does some people want to over-engineer it? > { "$id": "http://… http://xn--rvg" } Oh no. Please don’t do that.
streamofdigits 5y ago
Never had to learn the intimidating XML stack in depth but it seems clear that the (superficially definetely more digestible) JSON way of notating data must slowly and painfully reinvent the wheel. Reaching the same level of logical complexity (if it solves the same set of problems) seems unavoidable, no? So if the main advantage of JSON is human readability (not a machine oriented attribute btw :-) might be possible to JSON-ify the XML stack, essentially focusing on appearances and preserving the substance (a bit like the JSON-LD approach) In the end of the day these are frameworks for public data / metadata exchange and removing these frictions would be enormously beneficial to everybody...
- spookthesunset 5y ago
  What makes JSON so nice to use is it is basically how you already write complex data structures in most loosely typed languages anyway. JSON documents look a hell of a lot like how you'd write the same structure in PERL, python, javascript, and more. It provides a very natural mapping into the types of native data structures these languages offer. Dumping native data structures in these languages out to JSON is trivial. Dumping these same data structures into XML was always a pain in the ass because you'd have to manually map every field into elements and attributes.
- aeberhart 5y ago
  I think so too. JSON Schema enables very useful tooling such as validation, Swagger-style UIs for interacting with services, or declarative web forms that people would have to build again and again.
bitfield 5y ago
A slightly different approach: https://bitfieldconsulting.com/golang/cuelang-exciting https://bitfieldconsulting.com/golang/cuelang-exciting
ironman1478 5y ago
I started using jsonschema at work and can't go back. We have an extremely large configuration file to configure our system and there are lots of 'optional' blocks that can be configured. Jsonschema really cleanly allows you to express that logic without having to write a line of code. It's been great. Protip: if you are using jsonschema in python, it takes in a dict() not a file, so you can use yaml (or something else) as your configuration file and still validate it with jsonschema.
deepakarora3 5y ago
Not at all to be critical of JSON schema, but my experience has been that for most use cases it is an overkill. As simple as it may be, it still is relatively complex. As someone mentioned in a prior post, if the validation fails then what? Even though the JSON schema document is human readable, just by looking at the JSON schema document, its not intuitive to visualize where the particular path resides in the actual document. And one would need tools written on top of JSON schema to be able to do operations like merge one JSON document into another while validating at the same time. To make things simple, I had created JDocs. What this allows is to write the JSON schema in exactly the same structure as the JSON data document, just that the value field contains the validation specifications. Of course it does not have all the advanced features of JSON Schema (like some properties and cross referencing abilities), but then as I said, it meets our requirements in a real straightforward and simple way and opens up multiple possibilities in manipulating JSON data. You can read about it here. I would welcome feedback and apologies if you feel it is off track. Thanks. https://github.com/americanexpress/unify-jdocs https://github.com/americanexpress/unify-jdocs
- ether_at_cpan 5y ago
  > ...if the validation fails then what? Even though the JSON schema document is human readable, just by looking at the JSON schema document, its not intuitive to visualize where the particular path resides in the actual document. The JSON Schema specification actually does say that errors should indicate both the location within the data that the error occurred, and the path in the schema itself. So if you are having difficulty deciphering validation errors, that's a failure of your particular implementation failing to follow the spec, not the spec itself.
Waterluvian 5y ago
A possibly silly question about JSON Schema: What’s with the emphasis on full URLs to describe where to find related schema? Are developers really assembling a bunch of related schema over the Internet instead of just coalescing them in one place for local use? Maybe I just don’t understand the use case. Why would I ever want to make a client do a bunch of calls to URLs I don’t own rather than serving them up reliably and consistently at one time? It feels to me like the idea is some utopia where independent resources everywhere provide schemas and you can start stitching them together. But it just feels… unrealistic and not what I’d actually want. Is this actively realized today? Or does everyone just reference local schemas by relative file path like I do?
- cinaboniver 5y ago
  To be clear, I don’t have a lot of experience with JSON schema, but recently I found myself needing to validate Azure ARM templates using json schema, and my god is the Microsoft provided schema insane. Go look for yourself: https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json# https://schema.management.azure.com/schemas/2019-04-01/deplo... That’s the root schema definition, but there are many, many references within, and they go deep. I tried using the bundling tool mentioned in the article to bundle all the references schemas and it came out to 25MB, minified. But if you don’t bundle, then you are right, your tool has to crawl the document and make many more http calls to deref everything. It’s so frustrating to work with.
  - Waterluvian 5y ago
    That thing is ridiculous. Surely they build it in a different language and render the result as json.
- ether_at_cpan 5y ago
  They don't have to be real URLs, but just URIs. That is -- the files don't have to be accessable on the network at those URIs -- you can just use these strings as identifiers, and load the files up manually as you need them. In fact, the JSON Schema specification even says that identifiers are just URIs and the evaluator implementation need not be expected to have to load the documents from the network (ref. https://json-schema.org/draft/2020-12/json-schema-core.html#rfc.section.8.2.1 https://json-schema.org/draft/2020-12/json-schema-core.html#... and https://json-schema.org/draft/2020-12/json-schema-core.html#rfc.section.8.2.3 https://json-schema.org/draft/2020-12/json-schema-core.html#...)

140 points by relequestual 5y ago

I use jsonschema to validate my projects' output, and also generate the documentation from it. The python jsonschema support makes this possible: our testsuite is in Python already. But it's an awkward fit, and I despair of having anyone else in my team write schemas: the default of allowing additional unspecified fields must be continually overridden, otherwise your schema has no teeth, and things like "if this field is this value, these additional fields exist" must then always have an "else" indicating that no additional fields exist. In summary, it's better than nothing, but it's not easy. I'm not sure that JSON is a great language to specify schemas in, sorry.

tored 5y ago
Yes, the unspecified fields is a nuisance. It is hard to manually look at JSON data and compare it visually to a JSON schema because the schema has a depth to every property and lots of other cruft, which makes it somewhat cumbersome to retrofit a JSON schema to existing JSON data. I once developed an alternative JSON schema internally for a company where the structure was the same as the JSON data (and default strict of course). We implemented multiple implementations for every language we used. It sort of worked, not as feature complete as the official JSON schema of course, but my conclusion is that JSON doesn't fit well for this.
paulddraper 5y ago
> otherwise your schema has no teeth I don't disagree, but FWIW, this is common deliberate choice. E.g. Protobufs work exactly the same way. Except unlike JSONSchema, there's no way to disable it. The reasoning is to permit future extensibility.
- RustyRussell 5y ago
  For JSON itself, ignoring extra fields it makes sense. For validating my own output, it does not. And validating someone else's output seems counterproductive?

endisneigh 5y ago

I’m curious - what’s an argument against this that’s logically consistent with a position that supports databases that have schemas?

csmpltn 5y ago

We've come full circle back to XML SOAP. We should consider restricting the registration of .org domains to actual non-profit organizations, and restricting the use of words like "schema" and "standard" to things that have been fully certified as such by internationally accredited engineering bodies. This doesn't add anything on-top of the tools we've had 40 years ago. We're stuck with this everywhere now, all thanks to pure marketing.

hyperpallium2 5y ago
back to CORBA
- ChrisMarshallNY 5y ago
  > CORBA Get off my lawn!
- johnnycerberus 5y ago
  > CORBA "Well, it's high noon somewhere in the world."
- DonHopkins 5y ago
  The World's Second Fully Modular Software Disaster! It's just riddled with features.
  - dnautics 5y ago
    what was the first fully modular software disaster?
    - DonHopkins 5y ago
      Glad you asked! ;) https://donhopkins.medium.com/the-x-windows-disaster-128d398ebd47 https://donhopkins.medium.com/the-x-windows-disaster-128d398... [...] X-Windows is the Iran-Contra of graphical user interfaces: a tragedy of political compromises, entangled alliances, marketing hype, and just plain greed. X-Windows is to memory as Ronald Reagan was to money. Years of “Voodoo Ergonomics” have resulted in an unprecedented memory deficit of gargantuan proportions. Divisive dependencies, distributed deadlocks, and partisan protocols have tightened gridlocks, aggravated race conditions, and promulgated double standards. [...] X: The First Fully Modular Software Disaster >X-Windows started out as one man’s project in an office on the fifth floor of MIT’s Laboratory for Computer Science. A wizardly hacker, who was familiar with W, a window system written at Stanford University as part of the V project, decided to write a distributed graphical display server. The idea was to allow a program, called a client, to run on one computer and allow it to display on another computer that was running a special program called a window server. The two computers might be VAXes or Suns, or one of each, as long as the computers were networked together and each implemented the X protocol. [...] X-Windows: …A mistake carried out to perfection. X-Windows: …Dissatisfaction guaranteed. X-Windows: …Don’t get frustrated without it. X-Windows: …Even your dog won’t like it. X-Windows: …Flaky and built to stay that way. X-Windows: …Complex non-solutions to simple non-problems. X-Windows: …Flawed beyond belief. X-Windows: …Form follows malfunction. X-Windows: …Garbage at your fingertips. X-Windows: …Ignorance is our most important resource. X-Windows: …It could be worse, but it’ll take time. X-Windows: …It could happen to you. X-Windows: …Japan’s secret weapon. X-Windows: …Let it get in your way. X-Windows: …Live the nightmare. X-Windows: …More than enough rope. X-Windows: …Never had it, never will. X-Windows: …No hardware is safe. X-Windows: …Power tools for power fools. X-Windows: …Putting new limits on productivity. X-Windows: …Simplicity made complex. X-Windows: …The cutting edge of obsolescence. X-Windows: …The art of incompetence. X-Windows: …The defacto substandard. X-Windows: …The first fully modular software disaster. X-Windows: …The joke that kills. X-Windows: …The problem for your problem. X-Windows: …There’s got to be a better way. X-Windows: …Warn your friends about it. X-Windows: …You’d better sit down. X-Windows: …You’ll envy the dead.

akie 5y ago

I might be exceptionally dense, but it's hard for me to see practical applications for something like this. In the end, if you implement this in your application, you will have a mechanism to say "this input document is invalid". AND THEN WHAT? Your only option is to discard it. I'd rather live by the old maxim "be liberal in what you accept, and strict in what you produce". But perhaps I'm overlooking an important use case here, in which I'd happily stand corrected.

sorokod 5y ago
liberal or not you can express your expectation in a well defined manner. As to "then what" its up to you, i'd go with some 4XX ( or other technology specific error) that implies that no processing has occured. All arguments for and against schemas in databases translate well to this case.
66fm472tjy7 5y ago
There are things you cannot accept. What do you do if an order has a negative quantity, if you get a string instead of a number, if a value does not fit in your DB column, etc ? Having this as part of your interface definition, along with generated representations of the possible data in the language your choice[0] and automatic (so you can't forget it) validation that rejects invalid input with HTTP 400 bad request instead of 500 internal server error when the DB constraint fails (or even worse, you persist invalid data and something breaks later) is definitively useful. That said, OpenAPI fails at this somewhat as the people writing the spec don't consider the (de)serialization and validation libraries. Thus, OpenAPI keeps adding features/constraints that are not supported by the libraries and are thus cannot be automatically validated or even produce nonsense output on generation (e.g. in Java you might end up with raw Map or even Object fields) [0] https://openapi.tools/#sdk https://openapi.tools/#sdk
- reaperducer 5y ago
  What do you do if an order has a negative quantity, if you get a string instead of a number, if a value does not fit in your DB column, etc ? Isn't that all just basic data validation and sanitation that we all do anyway? And wouldn't it still have to be done, even if the JSON came with all the schemabloat? I don't think I'm suddenly going to accept that the data presented to my function is valid and fit for purposes just because it goes through someone's JSON schema library. That's just passing the responsibility on to a third party, which feels lazy and dangerous to me. Maybe it'll be OK for systems that don't get unusual data. But mine are always ingesting strange things, and the only person who knows what's truly valid and what's not is me, not a third party. That's my responsibility as the programmer of the system. I looked through the web page, and it's probably great for someone. Somewhere. But I'd like to see that SKU example that was presented fully fleshed out with real-world type data. It seems OK if your data only consists of "productID," "productName," and "productTags." But in real life, SKUs are often incredibly complex, and this just seems to invite disaster by making complex systems even more complex. So much so that they require even more complex tools to manage them.

gone35 5y ago

History has not been kind to efforts like these.

josteink 5y ago
History has been repeating efforts like this.
sorokod 5y ago
Worked well for relational databases.
speedbird 5y ago
Software "engineering" repeatedly goes through the same loop: - something simple, easy to learn, easy to use - doesn't cover edge case X - lacks rigour - lets add a bunch of features - and committees and processes to manage that - this is really complicated and hard to learn and use, we need something simpler ...

sorokod 5y ago

Would be good if there was a clear statement about who the authors are and what makes their content authorative.

oefrha 5y ago
> what makes their content authorative. Nothing, it’s just relatively widely known and used (e.g. for VS Code and Windows Terminal config files, which I expect a lot of people to have experience with), with relatively good library support in a wide selection of languages. Although I should mention that most of the libraries are stuck on old versions, and the versions they’re stuck on are all over the place: https://json-schema.org/implementations.html https://json-schema.org/implementations.html
- brabel 5y ago
  It's a horrible mess, made worse by the several years in which OpenAPI defined their own "dialect" of json-schema for their data schemas which was incompatible with the non-OpenAPI variant of json-schema, so you get libraries that support both, or support only a certain version of each, and because different versions tend to be completely incompatible with each other, you might run into lots of problems if you have more than one language consuming schemas, or even the same language but using different libraries due to transitive dependencies... it's just terrible, just use XML if you need a schema.
  - oefrha 5y ago
    The language is a godforsaken monstrosity, but if you only have to write it once for some non-critical validation task it’s acceptable. For instance, I use JSON schema to (optionally) validate some generated JSON data which is fed into a web app as a JSON module. I also generated the TS spec for it from the JSON schema (although writing the TS spec by hand would have been 10–100x easier than writing the JSON schema). XML is obviously not suitable here — TypeScript doesn’t have XML modules.
relequestual 5y ago
Do you mean the article or the specification?

chmod775 5y ago

A yes. Because things like "integer" totally need to be abstracted away into another schema. Couldn't have given this DSL built-ins for the things that are... you know... built into JSON already. Also bonus points for the apparent lack of shorthands, turning this language into a verbose word salad. I hope there's at least a line of reasoning that explains why some keys are prefixed with '$' and others aren't. Great way to turn something as beautifully simple as JSON into something abhorrent. Hey. You know what is better than JSON at being XML? XML.

conceptme 5y ago
afaik there is no integer type in json but a numeric type, and most languages do have different numeric types.
- chmod775 5y ago
  Neither is 'non negative' - but that should also be a built-in, considering how verbose the language becomes otherwise. Now you're adding ~5 lines of boilerplate every time you want to have an integer, another 5 for positive integers, and so on.
  - michaelmior 5y ago
    To support non-negative integers, all you need is the following {"type": "integer", "minimum": 0} I think the format used in the example was used to demonstrate bundling, but if what you really want is a non-negative integer, the above is the simplest way to do it.
    - relequestual 5y ago
      This is correct. If I had used a more complex example, it would have been much harder to follow. This was the simplest. We factor this out for our meta-schema construction due to reuse.
Someone 5y ago

jonny383 5y ago

If someone tried to push this on my engineering team, they would be laughed out of my office and ridiculed for the idiocy of even suggesting such garbage. The best factor of JSON is being a schema-less, human readable format. Imagine having someone present this monstrosity to you with a straight face, thinking it's a good use for JSON.

sorokod 5y ago
Would a library that validates conformance to the schema so that you don't have to handcraft error handling sweeten the proposition?
NicoJuicy 5y ago
To be honest, i used json schema for auto generating layouts. Eg. https://github.com/rjsf-team/react-jsonschema-form https://github.com/rjsf-team/react-jsonschema-form
endisneigh 5y ago
You’d laugh at someone for merely suggesting a technology? Seems like a toxic team If you think it’s bad surely an explanation would win them over and stop their efforts?
Jenk 5y ago
Do you prohibit the use of VSCode in your workplace? Anyone using that is using jsonschema. It's built into the applicatoin for its settings and configuration files. Plugins and extensions use it to manage their own config _and_ as validation for the editor's api.
- vultour 5y ago
  Your editor has nothing to do with your product. People use PyCharm to write Python yet would "laugh out" anyone who suggests to start writing Java.
  - Jenk 5y ago
    So vscode product team should laugh at themselves? Do you get how ridiculous GGP is now?

jacobmischka 5y ago

I'd like to offer a contrasting opinion to all of the other currently negative comments: I've used JSON schema in the past to validate outside input and it was a pleasant and straightforward experience. There are unfortunately no standard type declarations that I'm aware of, which is a pain, but tools exist to translate schema.org schemas[1] (I have not used this myself). [1]: https://github.com/charlestati/schema-org-json-schemas https://github.com/charlestati/schema-org-json-schemas

tored 5y ago
When I used JSON schema last time a few years ago the different implementations of it in PHP and nodejs where either incorrect of feature incomplete. Seems like it is hard to get right.
- relequestual 5y ago
  If only they had used used the official test suite... which is provided, free of charge, in JSON.
- dolmen 5y ago
  The main problem is that JSON Schema is still an evolving design. All tools may not yet implement the latest draft.
solids 5y ago
I agree with you, I didn’t expect the amount of negative comments. Used in the past and it was a great tool to “avoid representing invalid states” and also as documentation to share between teams.
- chrisjc 5y ago
  > I didn’t expect the amount of negative comments I did! It's been made clear time and time again on HN that "everyone" HATES XML and anything to do with it... especially XSD, WSDL, etc! I really don't see any difference between XML and JSON except the semantics (I lie, I see a lot of reasons why XML is more powerful). But I do get why people prefer JSON over XML for visual reasons alone. Since they're very similar logically/structurally, it only makes sense that the same things that are possible with XML are possible with JSON. And here we are, discussing JSON schemas! So is it really that they hate XML and love JSON for any other reason besides the visual ones? If not, what is it that they hate? I think it has to do with the responsibility and effort that goes into creating a contract, the "schema". All the anxiety and headbanging that goes into setting up all the meetings and discussions. All the convincing that happens over and over again just to move forward. All of the back-and-forth that goes into whether it's this type or that bc the requirements are ambiguous to begin with and now it's the developer's responsibility. "I could just throw this API together in a day if it wasn't for all these useless meetings. Why do I have to wait for Bob to update the schema when I can just cast this field to a boolean in my code?". "I don't want to have to deal with validation exceptions at runtime, just fix your damn code and read the wiki I put together." etc. (I do understand that not all APIs, data exchange files, tightly coupled project, etc... need schemas) In my experience, while these schema-less approaches lead to projects being leaner and completed sooner, the price is payed continuously there after. Something changes on the server-side, some edge case appears on the UI several weeks after release to prod. Enhance the structure, breaks ETLs. And so on. A perceptual state of keeping everything in tranquility emerges.

ChrisMarshallNY 5y ago

I generally prefer using JSON in my interactions, where possible. That's because it is lightweight, and 99.9% of the data I'm transferring is scalar. JSON is basically "implied" for scalar types. The good thing (if you want to call it "good") about XML, is Schema. Schema is a "rock hard" contract. It is definite, empirical, unambiguous. When I am looking at an API, and it has a Schema, then I know that I can figure out exactly what shape the data will take/emit. If possible, I still use JSON for the exchange. I just use the XML to figure out the specifics in the exchange. I've been working in XML forever, it seems. I've done ONVIF stuff, which is SOAP/WSDL-based, and even XSLT. I still like JSON. Mostly because I don't have to deal with Schema. Over the years, I've gotten fairly good at developing XML Schemas, but I have never gotten "used" to it, and still have to look everything up. If I develop APIs, I will often do an XML variant, alongside the JSON, because that forces me to write a Schema. Doing this, helps me to "code review" my schema, and also gives me a very convenient automated test hook. But, boy, I still hate XML Schema...

tored 5y ago
One thing I'm guilty of is designing a document format as JSON, thus the need to formally verify it. Passing data works well as JSON but documents works better as XML where you can use XML Schema.
- ChrisMarshallNY 5y ago
  This is where I would also develop an XML variant, even if it is never to be used, simply so that I can have a Schema to go along with it. I'll start with a data structure/class, in the code, then use a built-in transformer, to turn it into JSON or XML. That way, I know that the "kernel" of the data is the same, between them. Even though JSON is a "natural" for scalar types, it can get weird, there. For example, let's say we have a float: { "kids": 1.0 } It needs to be parsed as a float, because we could have: { "kids": 2.3 } But I often see these sent as: { "kids": 1 } or { "kids": "1.0" } or even { "kids": "1" } And I have differing results, based on the parser.
  - 5y ago

deleted 5y ago

[deleted]

enriquto 5y ago

Great, now we can finally add the next step in "The Ascent of Ward". EDIT (for the lucky 10.000 of today): http://harmful.cat-v.org/software/xml/ http://harmful.cat-v.org/software/xml/

tasogare 5y ago

Still missing: namespaces, attributes and comments (only-half ironical comment).

deleted 5y ago
[deleted]
ether_at_cpan 5y ago
Comments are supported. There is a "$comment" keyword, and you are free to invent any other keyword you like, as they are generally ignored.

diegoperini 5y ago

I haven't stumbled upon a better schema for JSON than the Typescript definition files. If only I could use them in non Typescript contexts. https://www.typescriptlang.org/docs/handbook/declaration-files/deep-dive.html https://www.typescriptlang.org/docs/handbook/declaration-fil...

YousefED 5y ago
This is why I originally started a project to convert ts to json schema; https://github.com/YousefED/typescript-json-schema https://github.com/YousefED/typescript-json-schema (and also check out a good alternative https://github.com/vega/ts-json-schema-generator https://github.com/vega/ts-json-schema-generator)
anentropic 5y ago
you might like https://jsontypedef.com/ https://jsontypedef.com/ if you haven't seen it
panzerklein 5y ago
You can write validation schemas using zod library [1]. They end up looking pretty similar to typescript definitions (same vocabulary, same methods for combining/intersecting/filtering). And if you migrate to typescript you'll get type definitions for free out of those schemas. [1] https://github.com/colinhacks/zod https://github.com/colinhacks/zod
- IggleSniggle 5y ago
  If you like zod, you should check out myzod: https://github.com/davidmdm/myzod https://github.com/davidmdm/myzod

theteapot 5y ago

Article: > "Developers of platforms and libraries that use OpenAPI haven't had such a shake up before, and my feeling is it may take more than a few releases to correctly implement all the new shiny features full JSON Schema has to offer." Dear OpenAPI, please avoid the shiny features.

ether_at_cpan 5y ago
Like which? Or are you just knee-jerking at the term "shiny" and presuming it means "over-complicated and unnecessary"?

rincewind 5y ago

I use json schema. I used to understand what it is and what it does. Now I don't.

theteapot 5y ago

> "There are several libraries which offer bundling solutions, however they all have caveats, and I haven't seen any to date which are fully JSON Schema aware." But what's wrong with just stuffing the schemas in a flat array or object exactly?

relequestual 5y ago
Expectation of a single schema by existing tooling.
- theteapot 5y ago
  That's pretty vague.

netfl0 5y ago

Reinventing JSON-LD.

aeberhart 5y ago
There certainly are some similarities, but I think JSON-LD and and JSON Schema solve different problems. You might find this article helpful (I ended up writing this after being confused by the various standards in this space): https://dashjoin.medium.com/json-schema-schema-org-json-ld-whats-the-difference-e30d7315686a https://dashjoin.medium.com/json-schema-schema-org-json-ld-w...

segphault 5y ago

The way $ref resolution works is hideously complicated, to the point where many JSON Schema implementations just don't even bother supporting external $refs. It really should be a totally separate concern from validation, especially since how you store and compose schemas may end up being specific to a given use case.

beardyw 5y ago
Much as I value JSON Schema, building a composite schema, to use and reuse parts, can easily hit a brick wall. I often end up building each schema programmatically and exporting to JSON. Could do better.

atombender 5y ago

I'll be one of the few positive voices here, I guess. JSON Schema is pretty good. It's a relatively simple, extensible, pragmatic specification that supports validating the kinds of data that JSON can express. XML was killed by complexity, same as lots of technologies that preceded it, such as CORBA and SOAP. Developers don't like complexity. W3C tried to build an enormously complicated ecosystem of tools on top of XML. The specifications for XML Schema, WSDL, XSLT, etc. were gigantic. XML Schema and WSDL are both so big they are split into multiple parts. XSLT 3.0 is maybe 500 pages when printed. Many of these originated in the wrong sort of place, designed by committee and from inside big enterprises like IBM, rather than being adopted from evolving practices. With XML out of the way, we still need a way to represent structured data, and it turns out JSON is pretty good for that. JSON has problems, but its simplicity is what lead to it becoming so prevalent in the first place. Now, we also need a way to define the structure and format of that data, and JSON Schema is pretty good for that. I think JSON Schema could have been a bit simpler, but it's still nowhere near to making the same mistakes as XML Schema.

arethuza 5y ago
Strongly agree - I've worked with ONC RPC, CORBA, SOAP, XML & XSD, XSLT etc. and I much prefer the relatively straightforward nature of JSON and JSON Schema. Yes it's not perfect - but for me it is more than good enough. Edit: Of course, there is no direct equivalent of XSLT in the JSON world - which pretty much counts as a feature to me.
- lolive 5y ago
  Nor XPath.
  - arethuza 5y ago
    I think XPath is actually one of the nicer bits of the XML universe - there is a JSON equivalent in JSONPath that I have used a bit.
- espadrine 5y ago
  One thing I am wondering is whether the protobuf folks have a better development environment overall, or a worse one. Is the schema-to-self-validating-serialization-code approach going to win just as JSON won over XML?

dolmen 5y ago

Bundling for OpenAPI specification has long been a need for authors to allow to reduce duplication, and to allow to split a big specification in multiples files, but publish a single one. A few years ago I've written a tool to fit that niche: https://github.com/dolmen-go/openapi-preprocessor https://github.com/dolmen-go/openapi-preprocessor https://github.com/dolmen-go/openapi-preprocessor https://github.com/dolmen-go/openapi-preprocessor I have now to tweak it (well, it will be a major rewrite) to handle $ref relative to $id instead of the file location.

9814715843 5y ago

amit Ma ndal

run-types 5y ago

For those looking to sanitize input from Typescript, I highly recommend the runtypes library: https://github.com/pelotom/runtypes https://github.com/pelotom/runtypes It let's you specify type definitions a DSL in Typescript using syntax very similar to Typescript's type definitions. Once you define your types in the DSL, you get Typescript types and parsing / verification for free. Not as general purpose as JSON Schema, but 1000x cleaner and easier to use.

dolmen 5y ago
The point of JSON Schema is to have a common format independant of the programming language to ease interoperability.
move-on-by 5y ago
I've never seen runtypes, but I've seen io-ts [1]. Are you aware of the pros/cons between them? [1] https://gcanti.github.io/io-ts/ https://gcanti.github.io/io-ts/
- run-types 5y ago
  They look roughly the same

corentin88 5y ago

JSON format is simple and easily readable. It’s now so common and probably one of most used language on earth. That’s great! Now, anyone can create a decent API with basic programming tool available. And everyone will be able to use it. Why does some people want to over-engineer it? > { "$id": "http://… http://xn--rvg" } Oh no. Please don’t do that.

streamofdigits 5y ago

Never had to learn the intimidating XML stack in depth but it seems clear that the (superficially definetely more digestible) JSON way of notating data must slowly and painfully reinvent the wheel. Reaching the same level of logical complexity (if it solves the same set of problems) seems unavoidable, no? So if the main advantage of JSON is human readability (not a machine oriented attribute btw :-) might be possible to JSON-ify the XML stack, essentially focusing on appearances and preserving the substance (a bit like the JSON-LD approach) In the end of the day these are frameworks for public data / metadata exchange and removing these frictions would be enormously beneficial to everybody...

spookthesunset 5y ago
What makes JSON so nice to use is it is basically how you already write complex data structures in most loosely typed languages anyway. JSON documents look a hell of a lot like how you'd write the same structure in PERL, python, javascript, and more. It provides a very natural mapping into the types of native data structures these languages offer. Dumping native data structures in these languages out to JSON is trivial. Dumping these same data structures into XML was always a pain in the ass because you'd have to manually map every field into elements and attributes.
aeberhart 5y ago
I think so too. JSON Schema enables very useful tooling such as validation, Swagger-style UIs for interacting with services, or declarative web forms that people would have to build again and again.

bitfield 5y ago

A slightly different approach: https://bitfieldconsulting.com/golang/cuelang-exciting https://bitfieldconsulting.com/golang/cuelang-exciting

ironman1478 5y ago

I started using jsonschema at work and can't go back. We have an extremely large configuration file to configure our system and there are lots of 'optional' blocks that can be configured. Jsonschema really cleanly allows you to express that logic without having to write a line of code. It's been great. Protip: if you are using jsonschema in python, it takes in a dict() not a file, so you can use yaml (or something else) as your configuration file and still validate it with jsonschema.

deepakarora3 5y ago

Not at all to be critical of JSON schema, but my experience has been that for most use cases it is an overkill. As simple as it may be, it still is relatively complex. As someone mentioned in a prior post, if the validation fails then what? Even though the JSON schema document is human readable, just by looking at the JSON schema document, its not intuitive to visualize where the particular path resides in the actual document. And one would need tools written on top of JSON schema to be able to do operations like merge one JSON document into another while validating at the same time. To make things simple, I had created JDocs. What this allows is to write the JSON schema in exactly the same structure as the JSON data document, just that the value field contains the validation specifications. Of course it does not have all the advanced features of JSON Schema (like some properties and cross referencing abilities), but then as I said, it meets our requirements in a real straightforward and simple way and opens up multiple possibilities in manipulating JSON data. You can read about it here. I would welcome feedback and apologies if you feel it is off track. Thanks. https://github.com/americanexpress/unify-jdocs https://github.com/americanexpress/unify-jdocs

ether_at_cpan 5y ago
> ...if the validation fails then what? Even though the JSON schema document is human readable, just by looking at the JSON schema document, its not intuitive to visualize where the particular path resides in the actual document. The JSON Schema specification actually does say that errors should indicate both the location within the data that the error occurred, and the path in the schema itself. So if you are having difficulty deciphering validation errors, that's a failure of your particular implementation failing to follow the spec, not the spec itself.

Waterluvian 5y ago

A possibly silly question about JSON Schema: What’s with the emphasis on full URLs to describe where to find related schema? Are developers really assembling a bunch of related schema over the Internet instead of just coalescing them in one place for local use? Maybe I just don’t understand the use case. Why would I ever want to make a client do a bunch of calls to URLs I don’t own rather than serving them up reliably and consistently at one time? It feels to me like the idea is some utopia where independent resources everywhere provide schemas and you can start stitching them together. But it just feels… unrealistic and not what I’d actually want. Is this actively realized today? Or does everyone just reference local schemas by relative file path like I do?

cinaboniver 5y ago
To be clear, I don’t have a lot of experience with JSON schema, but recently I found myself needing to validate Azure ARM templates using json schema, and my god is the Microsoft provided schema insane. Go look for yourself: https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json# https://schema.management.azure.com/schemas/2019-04-01/deplo... That’s the root schema definition, but there are many, many references within, and they go deep. I tried using the bundling tool mentioned in the article to bundle all the references schemas and it came out to 25MB, minified. But if you don’t bundle, then you are right, your tool has to crawl the document and make many more http calls to deref everything. It’s so frustrating to work with.
- Waterluvian 5y ago
  That thing is ridiculous. Surely they build it in a different language and render the result as json.
ether_at_cpan 5y ago
They don't have to be real URLs, but just URIs. That is -- the files don't have to be accessable on the network at those URIs -- you can just use these strings as identifiers, and load the files up manually as you need them. In fact, the JSON Schema specification even says that identifiers are just URIs and the evaluator implementation need not be expected to have to load the documents from the network (ref. https://json-schema.org/draft/2020-12/json-schema-core.html#rfc.section.8.2.1 https://json-schema.org/draft/2020-12/json-schema-core.html#... and https://json-schema.org/draft/2020-12/json-schema-core.html#rfc.section.8.2.3 https://json-schema.org/draft/2020-12/json-schema-core.html#...)