8 ms·
Select Transform: JSON template over JSON
- skocznymroczny 9y agoXSLT for JSON?
- athenot 9y agoThat's the first thing that came to my mind too. XSLT is very powerful but also incredibly messy because it has to be expressed in XML. Doing the same in JSON seems like it would result in the same problem.
- gcb0 9y agoall code generators that are not code end up being like that. and just yesterday there was an article about that magazine reporting the end of programming because of some BASIC code generator. heh. dbaseIIIplus was the one that actually almost managed this feat.
- masklinn 9y ago> XSLT is very powerful but also incredibly messy because it has to be expressed in XML. Being expressed in XML does not help, but the language is just bad with implicit data flowing through and very odd constructs, there are XML template languages which I found much more readable than XSLT e.g. I remember my experience with genshi much more fondly than the years I spent with XLST.
- elcapitan 9y agoI could imagine use cases for this for example for REST apis that consume json and then have to transform it generically to some other json for db storage in a document database. The internal storage could change over time with the outside spec staying the same, and such transforms could be used to do the mapping per version. This particular implementation looks a bit messy though - XSLT is valid XML and Xpath, while this solution uses templating syntax. That's more human readable, but also slightly inconsequent - you wouldn't be able to validate such a template with json schema for example.
- gliechtenstein 9y ago> That's more human readable, but also slightly inconsequent - you wouldn't be able to validate such a template with json schema for example. Could you elaborate?
- elcapitan 9y agoJust to use the json-schema example - because the transform uses templating syntax inside strings, I wouldn't be able to write a json-schema to check if a given transform object is valid or not, which can quickly become an issue for growing sizes of transforms.
- fokinsean 9y agoOh man I have nightmares of XSLT from an internship I had a few years ago. Thankfully there's an XSLT guru on stack overflow that would answer every question I posted. This is the guy: https://stackoverflow.com/users/317052/daniel-haley https://stackoverflow.com/users/317052/daniel-haley
- Latty 9y agoDo people not remember how terrible XSLT was? I mean, sure, it could do the job, but did anyone ever sit down at a file of XSLT and think "yes, this is the best way this information about transforming this XML could be represented"? JSON is even less suitable for this. It's horrible to read this stuff, horrible to write.
- eponeponepon 9y agoXSLT was, and still is (imho) a great language - at least after 2.0. There's a steep learning curve full of false summits, for sure, but once you grasp it, there is always an elegant way of solving a problem using it. Sometimes there's even a fast way! Use of it in the wild, though, leaves a lot to be desired. The false summits lead people to write bad code, the bad code gives it a bad rep, the bad rep leads people to dislike using it, people's dislike leads to then writing bad code... and so on.
- Latty 9y agoI just don't buy it - sure it can be used to do good things, but it's not about learning curves or "grasping it" - it's about "is this a good way for humans to write and read this transformation", and the answer is clearly no. XML is not a good form for a transformation language. It's not the quality of the code written in it or anything like that. Take XSLT and change the syntax, and sure, it might be great. You can't ignore the terrible, terrible choice of making the transformation itself XML. It's completely unreadable and horrific to write.
- eponeponepon 9y agoConversely, I personally find it very readable, and enjoyable to write; don't forget, when people talk about using the right tool for a job, it's not all about the language and the task - the dynamic between the person and the language is just as important :-)
- Latty 9y ago
- xfour 9y agoCurrently I'm having issues trying to figure out a new stack where we can't figure out where to put the backend for front-end logic in a React App over the top of GraphQL. I've been thinking of implementing something like this as the transforms or the formatters don't make a ton of sense to be on the client or the API or in GraphQL and I'm loathe to make another API call in the chain as it feels like I there's enough of those already? Would that be a good use case for this?
- gliechtenstein 9y agoCreator here. Yup, in fact you can check out something like that here https://github.com/SelectTransform/JSONQL https://github.com/SelectTransform/JSONQL Basically you can send the JSON template over JSON to the server, and let the server transform the raw data using the template before returning. The most important benefit here is that, since everything can be expressed in JSON, it can be stored or transmitted over the Internet, which enables interesting cases like this.
- fiatjaf 9y agoI like the fact that this exists, but I prefer to use jq. I also would like to know in which situation this is preferable to jq.
- astral303 9y agoThere's also http://jmespath.org http://jmespath.org -- the syntax for which is supported in AWS CLIs. Not sure which one is better at this point. Using jq still often feels like trying to decode a code golf contest entry.
- fiatjaf 9y agoReally? I find jq syntax rather pleasant to read and write.
- mtrn 9y agoThe jq tool is great. Start using it today as a JSON pretty printer and learn it's more powerful features (selection, object creation) as you need them. The only thing I would like to add to jq would be the ability to use multiple cores.
- turtlebits 9y agoLooks cool, but I can't help but think that you're just reproducing what map() and filter() do, without any real benefits?
- XaspR8d 9y agoI perceive it being middleware as the benefit. For when your client isn't set up to manipulate the incoming data and you can't edit the backend resource that generates it. Or vice versa.
- vvkmnn 9y agoNow that's a goddamn demo. No fussing about. I can't wait to play around with this.
- Edmond 9y agoHave you considered reaching out to DevOps folks to consider something like this for complex configurations?...There are related solutions such as jsonnet/ksonette but I think something like this might be preferable. I work on a similar problem though using a UI based approach, I gave a demo to the Kubernetes SIG App group a while back which be found here: https://www.youtube.com/watch?v=alEzE8MSNaI&t=30m https://www.youtube.com/watch?v=alEzE8MSNaI&t=30m Btw Ethan, maybe my email is landing in your Spam folder? I've tried reaching out :)
- gliechtenstein 9y agoThanks! I actually built this library for my own usage (Jasonette) and decided to open source it because I realized it can be powerful for all kinds of other purposes. In essence ST.js is a finite state machine implemented purely in JSON. So I can imagine it being used for things I haven't thought of, which is why I open sourced it. Would be really cool. Please feel free to reach out at ethan.gliechtenstein[at]gmail thanks!
- rtpg 9y agoI think there's a lot of useful unexplored territory in this sort of tool. So much work nowadays is happening in maps/dictionaries/what-have-you, with a lot of work being, essentially, data shape transformation. Despite this, many languages aren't great at this! we have tools that work alright on first-order operations but fall apart on higher level things. Even really simple things like "conditionally include a key" are rarely supported in a way that leads to concise code. How many times have we all written foo = {...} if bar: foo['a'] = x instead of something like foo = { ... , 'a': x if bar } and then having bugs because of branches? Tools that solve this sort of problem will be to modern programming what things like the Unix shell was to data mungers in the past. Haskell lenses kinda goes into this stuff, though Haskell itself is a bit hard to use in a lightweight fashion. Clojure has Specter. But still looking for some more stuff to fill this hole, especially in the "transitionally typed" stuff like Typescript or Python + Mypy
- rraval 9y agoSince you're using Python for your example, it's worth mentioning that 3.5 extended the unpacking syntax [1] to sort of do what you want. foo = { 'a': 1, 'b': 2, **({ 'c': 3, 'd': 4, } if bar else {}), } This adds `c: 3` and `d: 4` into `foo` conditional on `bar`. [1] https://docs.python.org/3/whatsnew/3.5.html#pep-448-additional-unpacking-generalizations https://docs.python.org/3/whatsnew/3.5.html#pep-448-addition...
- rtpg 9y agooh, this is most definitely a thing I want. Not sure if my coworkers would appreciate this however. Would still have liked a bit of a simpler syntax for inclusion but this could be worth the ugliness Nice that this works on lists as well.
- rraval 9y agoYeah, I personally find this sort of syntax convolution ugly and hard to read but I thought it relevant. Enough rope to hang yourself with and whatnot.
- arkh 9y agoGotta love JSON. It's XML without the tools so you get to reinvent everything to pad your resume. XML -> JSON WSDL -> JSON schema XPath -> JSON path XSLT -> JSON template? SOAP -> Swagger and friends One or two decades from now there will be too many tools and things to learn about JSON so another generation will reinvent a new "perfect format for everything". And a new cycle will have started.
- dualogy 9y agoThe fact that all these use-cases pop up again tells us that they're real. I was dubious the first-time-around =) Well in green-field projects it's certainly advisable to minimize what is "done in"/"expressed in"/achieved-via the current-interchange-format de-jour --- but in messy real-world projects such rabbit holes can unavoidably open. And then you sooner or later end up thinking, as I did back then, "XML is kinda sucky but XPath and XSLT make it actually pretty nifty & fun to deal with and somewhat escape its own very suckiness".
- DLA 9y agoYeah, makes total sense to traffic in XML then convert back and forth to/from JSON for web layer. Gotta love XML.
- rtpg 9y agoI kinda get this criticism, but this is also a bit like lamenting that we write the same libraries for different programming languages. I mean... yeah. Some ideas are good across protocols. That's why they're copied. On a merits-based thing, if you're just looking at XML, it has a lot of things going against it .If someone sends me something "in JSON", I can have a good guess what it'll look like. Not as much luck with XML. XML punishes you for using attributes (because you can't place composed types within them), but the awkward alternative is placing attributes in sub-elements. I have a list of products with prices. I store the price as a quantity attribute on a <product>. I decide later on to store currencies for all my "money" objects. Do I now double the amount of attributes? It's all super awkward, and JSON's mental model is more straightforward, even if you end up with approximations.
- 9y ago
- gliechtenstein 9y agoHi, my name is Ethan. I'm the creator. I thought I would provide some context on why I wrote this library, and how I'm using this right now. So here it goes: Other than ST.js, I also work on another open source project called Jasonette (https://www.jasonette.com https://www.jasonette.com), which lets you write an entire native iOS/Android app in nothing but JSON markup. And when you can express the entire app logic--from model to view to controller--in JSON, you can split them up whichever way you want and load them from anywhere (from a file, from cache, from a remote server, or from local memory). But the biggest benefit of all is: you can load an entire ios/android native app from the server in realtime, just like how web browsers load HTML/JS/CSS in realtime. When working on Jasonette, implementing model and view was relatively simple. For model it's natural since JSON is all about describing data. For view i just needed to come up with a syntax to describe layouts and all the standard mobile UI components in JSON. However the biggest challenge was how do i actually describe functions in JSON. Without a function, it's just a mockup and won't really do anything meaningful. Which brings us to ST.js. When you think about what a function is, it takes one value and turns it into another. So basically what I needed to do was build something that will take one JSON, and turn it into another JSON, but most importantly I would have to do it using JSON. Basically I needed to implement a finite state machine in purely JSON. And this is what templates do. So I set out to build a JSON template engine that turns one JSON into another using a template which itself is written in JSON. What's really cool about this is, since the template is JSON (As far as I know there doesn't exist any prior art that uses JSON as a template, otherwise I would have used it instead), it has all the benefits of the JSON format itself: 1. You can store it anywhere (Most DBMS nowadays support JSON natively) 2. You can send it over the Internet 3. You can compose them easily 4. You can validate them using JSON schema 5. etc. To use a more specific example, I use ST.js in both Android and iOS versions of Jasonette as the template engine. And a JSON template is absolutely necessary in this case. For example, if I want to take a piece of device-generated data and render it, I need to be able to somehow parse it client-side (can't send it back to server to re-generate a JSON) http://docs.jasonette.com/templates/#3-device-api-generated-data http://docs.jasonette.com/templates/#3-device-api-generated-... This also applies to many cases where the client-side data contains privacy-sensitive data. The only way to dynamically parse something that happens on the client side is by writing the template itself in JSON and sending it over as data. Anyway, I hope you guys take a look at the examples on the homepage to get a glimpse of what makes ST.js powerful. Each example is almost its own library, except that you don't need a separate library for those purposes since all you're dealing with is JSON.
- xer 9y agoJust saying learn ramda
- dschuett 9y agoThis is game changing.
- mishoo 9y agoWhat's wrong with: return { labels: data.items.map(function(value){ return { type: "label", value: value > 10 ? value : value * 100 }; }) }; I hope your project does not get too popular.
- addicted 9y agoEven cleaner with ES6. return { labels: data.items.map(value => { type: "label", value: value > 10 ? value : value * 100 }) };
- DEinspanjer 9y agoThis seems pretty interesting, and I will be happy to play around with it. I am a bit concerned that there is no discussion in the docs of the potential security risks of allowing direct native JS execution of arbitrary instructions passed in by an untrusted source. I use a project called JSONLogic (jsonlogic.com) which bears some similarity to ST in terms of being able to select and transform values. The biggest advantage I see with it is, unless you explicitly plug in a rule that parses and executes user data, there is no way for the data to "escape the sandbox". This means you can safely build a query syntax on top of it where you can directly consume the arbitrarily complex query from an untrusted source and execute it in a secure manner.