21 ms·
jq is awesome, a real achievement. At the same time it demonstrates everything that is wrong with traditional shells and what PowerShell gets so right. jq is
by useerup 11y ago
jq is awesome, a real achievement.
At the same time it demonstrates everything that is wrong with traditional shells and what PowerShell gets so right.
jq is not a "Unixy" tool in the sense that it should do one thing and do it right. jq implements its own expression language, command processor complete with internal "pipelines". Why would a tool need to do that? find is another utility that does many, many other things than to "find" items: It executes commands, deletes objects etc.
Consider this challenge that included parsing json, filtering, projecting and csv-formatting output: https://news.ycombinator.com/item?id=9438109 https://news.ycombinator.com/item?id=9438109
Several solutions uses jq - to good effect. But the PowerShell solution uses PowerShell expressions to filter, sort and project items.
The problem is - at the core - that the traditional command tools are severely restricted by only being able to rely on a text pipeline convention. You cannot parse json and send the "objects" along to another tool. Well, you can, if the json tree is extremely basic - like 2 levels.
PowerShell also has a json parser tool: https://technet.microsoft.com/en-us/library/hh849898.aspx https://technet.microsoft.com/en-us/library/hh849898.aspx. It is called ConvertFrom-Json (follows the verb-noun convention) and is distributed as part of PowerShell (built-in if you want - but that's a misnomer for PowerShell modules).
It is extremely simple - it doesn't even take any parameters - it just converts a string (parses it as json) and outputs the object/objects which can be arbitrarily complex. It truly does only one thing. If you want to select specific properties, sort, group etc you do it with other PS tools like select (alias for Select-Object), sort (alias for Sort-Object), group (alias for Group-Object) etc. If you want to convert back to json you use ConvertTo-Json.
- kbenson 11y agoWhile jarring when I first used PowerShell, once I realized what it was doing, I immediately understood it as a powerful concept. It allows shell pipeline processing to work in much the same way you can filter lists of hashes/dicts/arrays in more traditional scripting languages. I haven't used it much, and not at all in the last few years as I'm not often on Windows boxes, but I've always remained impressed.
- fragmede 11y agoDepends how you look at it. jq is a "Unixy" tool in the sense that it should do one thing and do it right - and that 'one thing' is parse json data to output it in the desired format. Awk, arguably one of the oldest unix tools, also has its own language. Would you consider it non-Unixy or is it grandfathered in due to age?
- josephkern 11y agoAwk does one thing well, it runs Awk.
- kbenson 11y agoAwk can be used as a unix pipeline tool, or it can be used more as a stand-alone that you make scripts/programs in to do larger tasks. Similarly, Perl falls under the same category, being an awk/sed replacement. But considering Perl just a unix pipeline tool is also severely underselling it.
- smorrow 11y agoI'd like it better if it was more like csvkit, with a separate jqgrep, jqfmt, jqsort... that you combine using the shell. Because of course everybody already knows the shell.
- useerup 11y agolet's see: jq does JSON parsing. And does it really well. Check. jq does filtering jq does projection (. operator, .[] operator, select etc) jq builds objects (the {} operator) jq does arithmetic (addition, subtraction, division, multiplication) jq does database-like "lookups" in collections (keys, has, length etc) jq does pipelining of the in-tool "objects". jq does object generation of new objects like range operators etc. jq does data type conversions jq does sorting, grouping, extracts min/max etc. jq does string interpolation and string formatting jq does equals and relational comparisons, tests and other boolean tests. awk does many of these too. bash has some boolean tests, but typically very filesystem or envvar centric. In general, when using shells and tools: Sorting should be handled by sort. Except when it is handled by jq or by ls or by find or by ps or .... Filtering should be handled by grep. Except when it is handled by ls or by find or by ps or by jq or by awk or ... Formatting is always handled by each and every command. Which is understandable given that the format of each line is so important. Now consider PowerShell where ConvertFrom-Json does one thing: Converts text from JSON to objects. Where-Object (aliases ?, where) does filtering. On any type of object. Sort-Object (alias sort) does sorting. On any type of object. Select-Object does projections. On any type of object. Format-Table, Format-List and other Format- cmdlets formats objects to human-readable output. The text protocol does not scale well with structural complexity. It is really only good for lines of text. It has been somewhat enhanced to be able to delimit fields which then makes for an informal tabular-like protocol. The object protocol scales with structural complexity. That's the difference.
- Sir_Cmpwn 11y agoI don't entirely agree. With traditional shells, you can still use objects, you just have to serialize them to text. jq is perfect for this, you can use it to store a serialized object in a variable and then pull things out of it for use with other commands. Example: example="{}" example=$(echo $example | jq .foobar=10) Later on you can pass example.foobar into another program like so: grep $(echo $example | jq .foobar) ... You can write simple helper functions that make this more concise. The beauty of it is that this works with any data format - plaintext, json, csv, etc - you just need a tool to serialize and deserialize it.
- kbenson 11y agoMy understanding is that PowerShell is passing around specialized objects that have a few required methods, and some source specific ones. The beauty of this is that you can pass massive amounts of data without serializing and unserializing it at every step, and also that they can contain links to other information, such as links to parent objects which may or may not be part of the same list (I have no idea if this is allowed, but it seems a logical extension of the concept). Additionally, dates passed around can be complex types that don't have to lose relevant info such as time zone, regardless of what the default display format is. The downside is that tools need to buy into this and work with this data stream rather than text, which makes the set of available tools necessarily much smaller.
- zobzu 11y agopowershell is a nice concept and a pretty nice shell, but its also complex due to this. serializing to text just means "everyone must speak the same language" its also complex due to this. And in fact jq does serialize. i'd argue jq is a "one tool, one job" and thus unixy. Its the way the "one job" is seen that differs. After using powershell an bash/etc. for years, i'm still unconvinced which one is the most powerful in the real world. sometimes (most often) its faster/easier to do things with bash, sometimes the convenience of powershell objects is nice (that's where i'd basically use python on linux because of bash limitations). As long as i can parse, grep, etc with jq, and pass it to another tool via a text stream output, im happy. as a human, i find it easy to deal with unstructured or very lightly structured text. Hard to dig through complex structures.
- bch 11y agowell, JSON is a serialization format -- so I'm genuinely curious what you mean. Can you better describe the dichotomy you're talking about (text vs objects) and give an sample problem that demonstrates a weakness of text ?
- useerup 11y agoThe http protocol is just text when you look at it. But I cannot just send "text" to a web browser or server. I have to follow a protocol. When I do so, I can anticipate how the receiver will understand the "text" - and plan for it. The same way with shell tools - they input/output byte streams - but by convention we expect those to be (mostly) text. But there is no *protocol there. One tool cannot build a text stream in anticipation of how the next tool will understand it. Hence you can only standardize on a very basic level. In traditional shells that basic level is text lines and sometimes space or tab delimited fields of a line. There is no common way do describe a tree of nodes or a graph of objects that you can expect all of the tools to understand. Sure, you can use xml - that's just text. Or you can use json - that's also just text. At least they can describe compound values and trees. But other common tools simply do not understand those formats. PowerShell outputs objects - i.e. the pipelines are delimited so that the elements are objects. Each object can be arbitrarily complex. Each tool understands that it may receive complex objects - but can carry out their task regardless. Example: select (projection akin to the map function), group, sort operates on objects. Even if you sort the objects - the output will be the same objects. If you select properties using the select cmdlet, the objects that are the properties are passed on - and they can be arbitrarily complex as well. To sum up: PowerShell has a higher-level protocol for pipelines and standardizes an object model (duck typing) so that tools can produce and consume trees, graphs, dates, customers, services etc.
- xaduha 11y agoPowerShell is cool, I wish it was more popular. I also wish that XProc was more popular too, but XML is a big no-no on the Web nowadays.
- andrewd18 11y ago> This cmdlet is introduced in Windows PowerShell 3.0. So... I can only guarantee it's included on my customers' computers if they have Windows 8 or higher. I'm happy most of them moved from XP to 7. Maybe I can start shipping PowerShell instead of batch files in 2020.
- masukomi 11y agoI don't think that's really the point. The point is that the unix command line has some serious shortcomings in its fundamental design. By extension he may be suggesting that maybe we should get off our asses and do something about it because windows has actually managed to ship something that's not only good, but better than any of the comparable options on *nix.
- yarrel 11y agoI don't see how it's better. You can send objects over pipes if you want to (in, for example, json format). If PowerShell wants to be the bloated Word-equivalent of shells that's certainly a design objective. I'm sure someone will eventually do a Mono or systemd of it, saving UNIX from the tragedy of not working like Windows. But it smells of Taligent.
- DonHopkins 11y agoDoes it smell like people, places or things? ;)
- andrewd18 11y agoAgreed, GP's post was about the technical merits of PS. PowerShell may be the greatest thing since virtual desktops, but that doesn't help me until PS 3.0 is included in the OS by default. Since my customer base is decidedly conservative on OS upgrades, it's going to be a while until I can try out any of those great PS features.
- clinta 11y agoPowerShell 4 for Windows 7: http://www.microsoft.com/en-us/download/details.aspx?id=40855 http://www.microsoft.com/en-us/download/details.aspx?id=4085...
- skybrian 11y agoI'm wondering how PowerShell's object model compares to other scripting languages. What's a good introduction to PowerShell's object model?
- useerup 11y agotype help about_objects (or go to the online version at https://technet.microsoft.com/en-us/library/hh847810.aspx https://technet.microsoft.com/en-us/library/hh847810.aspx) Follow the pointers to the other topics from there.
- Touche 11y agoBut Powershell builds on top of .NET, which means you can't write cmdlets in non-.NET languages. And writing them even in C# is quite painful. Which is why there is virtually no ecosystem for Powershell cmdlets and there is a gigantic, ever expanding, ecosystem of Unix shell tools. At the end of the day you have to choose a primitive and history has shown that text is the right one.
- Arnavion 11y agoYou can write them in PS itself, and it is quite simple and minimal.
- useerup 11y agoPowerShell cmdlets can be written in PowerShell. PowerShell modules (the unit of distribution) can be written in entirely in PowerShell. You can also write them in a .NET language, like C#, F# or VB.NET. I disagree that it is painful to write cmdlets in C#. A cmdlet is just a class, the parameters are just properties annotated with the [Parameter] attribute. The shell does all of the parameter parsing, matching and type conversion/coercion. If I wrote a traditional commandline tool I'd have to do that parsing myself. Yes, I could use a library function like getopt for that - but the code and complexity would still be there. But my point was not to get anyone to use PowerShell on Unix/Linux. PowerShell draws much of it strength on Windows from the fact that there are already well-established object models on that platform (COM, .NET and WMI/CIM). The point was that the age-old text-only view in command shells is beginning to show some shortcomings. XML and JSON (among other formats) challenge the tabular view of the world. Sometimes you have to experience the alternative to realize that you have been living in a restricted world. Like swallowing the red pill. Who know, maybe an object-oriented shell that mixes well with the Unix tradition will emerge.
- Touche 11y ago> Who know, maybe an object-oriented shell that mixes well with the Unix tradition will emerge. This doesn't make any sense though. Unix has a tradition of language diversity where Windows has a tradition of using whatever Microsoft says to use. Unix has language-specific shells but like Powershell they haven't really caught on.
- keenerd 11y agoIf you want to see another approach, jshon is much more unixy. (disclaimer, I am the author) homepage: http://kmkeen.com/jshon/ http://kmkeen.com/jshon/ Jshon is also a year older. I've never been able to figure out why/how Jq got so much larger so much faster.
- kstrauser 11y ago> I've never been able to figure out why/how Jq got so much larger so much faster. I offer this in the sense of constructive criticism: for me, it's because your examples are scary. Here's the command you give for reading a URL of a photo uploaded to reddit (in file foo.json): $ jshon -e data -e children -e 0 -e data -e url < foo.json Here's the jq equivalent: $ jq -r '.data.children[0].data.url' < foo.json which looks an awful lot like every other tree parser I've used. Now suppose I want to get all the URLs with jq? That's just: $ jq -r '.data.children[].data.url' < foo.json In Jshon, I have to rewrite the query as: $ jshon -e data -e children -a -e data -e url -u < foo.json That is, replacing the "-e 0" (get item at index 0) with "-a" (do the rest for all items). The jq equivalent changes ".children[0]" to ".children[]", which to my poor muddled brain more clearly represents what I'm asking for. For me, jq's query language maps well onto how I think about the problem. Jshon's does not. That's why I use one and not the other. Again, I offer this sincerely. Even if I don't personally use it, I love when cool ideas cross-pollinate from one project to another and I'm glad there's more than one player in the space. Thanks for your work!
- 1_player 11y agoHey, thanks for jshon. I've been using it a lot lately!
- mkonecny 11y agoI have only recently realized the effect a well designed project website can have on the users impression of your project. I'm sure the code/design quality of jshon is great, but the website design/layout doesn't reflect this. The current look of your website looks like a website from the late 90's, and may as well implicate that the project is no longer maintained.
- integraton 11y agoPowerShell is a .NET CLI and is more comparable to a JVM CLI (or the Python, Ruby, or Node REPLs) than it is to POSIX shells. Unix pipelines are for streaming data between processes using standard input and output. PowerShell pipelines are for sharing data between .NET classes called cmdlets and are more like chaining methods within a programming language. There are multiple programming languages with this functionality available in the *nix ecosystems. Working with the PowerShell "pipeline" is like working in a CLI/REPL for Python, Ruby, Node, not like working with Unix pipelines.