7 ms·

I find that to be the issue. You are considering it just RAW text when it is actually formatted text that has been parsable for years with common unix command l

I find that to be the issue. You are considering it just RAW text when it is actually formatted text that has been parsable for years with common unix command line tools. It not being in the format you consider a structured object does not mean it's not a object or even parsable. If you are using ad hoc regex I suspect you are not using all the tools available to you. I feel like Kernighan and Pike do a much better job of explaining than I could ever. https://www.amazon.com/Unix-Programming-Environment-Prentice-Hall-Software/dp/013937681X/ref=sr_1_1?ie=UTF8&qid=1471543239&sr=8-1&keywords=The+UNIX+Programming+Environment https://www.amazon.com/Unix-Programming-Environment-Prentice...

madmax96 10y ago
So much this. What the M$ community fails to see is that text streams can be consumed by __everyone__. We've basically all accepted the fact that programs are used longer and in different ways than we'd expect. A brittle object model can only lead to vendor lock in and indigestible output.
- ygra 10y ago
  Nothing prevents me from changing those objects into a text stream. In fact, it's infinitely easier than turning a text stream into objects.
  - mbrumlow 10y ago
    I feel you missed my point. It's not just a stream of raw data. It's a stream of formatted text... There is no magic or hand waving involved.
- temac 10y ago
  It's even more than that: text formatting allows the use of generic filtering and text processing tools. Whereas if you are using objects, you will tend to use less on-the-fly command line composition, and write or reuse more tools dedicated to one particular object or the other. In the end I'm not sure keeping the structuring as the default use case yields an usage so much improved in the real world, because if you work on a broad number of types you will tend to need to know more specialised tools, instead of generic ones, and have less higher-order composition tools at your disposal. Now of course you can always format to text from your structured object, but this does not matter. What matters is what is convenient in the proposed UI, and what is mainly practiced in the real world because of such convenience and the amplification loop it creates between tools authors and users. Also some objects are originally handled in text form, and their structured decomposition is extremely complex compared to an informal description and the naturally refinable heuristic which comes with it. For example you can grep into a source code in any language (with more or less efficiency depending on the language, but at least if there is a unique symbol you will find it), whereas trying to get and handle it in a structured way basically means you would need half of a (modular) compiler, and a huge manual to describe the structure, and possibly a non trivial amount of code to actually do the lookup. The PowerShell approach is not all-bad, though, and obviously there are some usages where it superior to text based shells. But for a day to day usage as a general purpose shell, and a programmer shell, I'll stick to the Unix approach.
  - Firadeoclus 10y ago
    Structured data allows the use of generic filtering and structured data processing tools. The basic requirement is reflection, objects being able to tell you about their structural composition. If code was stored in a structured representation you could still search for a string object containing a symbol name in the structured representation. You can match a structure pattern just like you match a regex to text. Typical shells can be thought of as REPL for a programming language that makes the file system easily accessible and uses said file system or pipes to pass around data between functions/commands, sometimes as human-readable text. Most programming languages don't encourage passing data around as strings.
- blakeyrat 10y ago
  The dollar sign is charming. There's a few points here: 1) Not all data is text. In fact, very little of the data people see/work with day-to-day is raw text. It's silly to transform a PNG image into text to be able to pipe it around. (Or to pipe around its filename instead and have a dozen tools all having to open and re-parse it each time.) 2) There's nothing on PowerShell preventing you from serializing a piece of data to text if you want to. The key is: you don't have to. 3) Systems that depend on 50,000 CLI tools all having their own ad-hoc text parsers are cemented, mummified, cannot change. You can't change the output format of ps (to use an example in this thread) without breaking an unknown number of CLI tools. Even if you come up with a great way to improve the output, doesn't matter, you've still broken everything. This is less (but not none!) of an issue with PowerShell. I like computers to evolve to become better over time, and text-based CLIs are a huge anchor preventing that.
  - dragonwriter 10y ago
    Unfortunately PowerShell relies on everything running on .NET (well, I think COM works, too); the idea of a shell that can expose live objects is useful, but PowerShell's platform limitations in reality doing that make it a far from ideal implementation of that concept. Something built on a protocol that is platform agnostic would be better.
    - Const-me 10y ago
      Live objects don’t usually expose any protocols at all. They only expose an ABI. Do you know any platform-agnostic OO ABI, besides what’s in .NET? If you’ll wrap your objects into some platform-agnostic protocol like JSON, you gonna waste enormous amount of CPU time parsing/formatting those streams at the object’s boundaries.
      - tracker1 10y ago
        You can run streams of many millions of JSON objects pretty much as fast as the IO can feed it... most of the time, in situations like this, you're constrained by IO speed, not CPU... assuming you are working with a stream that has flow control. I tend to dump out data structures to line terminated JSON, and it works out really well for streams, can even gz almost transparently. Parse/stringify has never been the bottleneck... it's usually memory (to hold all the objects being processed, unless you block/pushback on the stream), or IO (the feed/source of said stream can't keep up).
- MaulingMonkey 10y ago
  > What the M$ community fails to see is that text streams can be consumed by __everyone__. Text can be poorly parsed by everyone, yes. I especially love it when the default tools settings mean two different computers will give different text results, because the installation defaults changed at some point. What's not to love about trying to properly escape and unescape strings via the command line, while simultaneously keeping in mind your own shell's escaping, and having scripts which are neither backwards nor forwards compatible? And this is to say nothing of different distros. It's as if the text was built for humans instead of my tools half the time, or something. I usually try to centralize my parsing of text in one place so when it invariably breaks, I don't have to rewrite my entire shell script. Some basic structuring - I hesitate to call it a brittle object model, when most of the time I'm dealing with something more akin to C structs than invariant and abstraction laden Smalltalk or somesuch, or Java's sprawl of factories and inheritance - makes things a bit easier. New fields can be added without breaking my regular expressions. I don't need to worry about escapes to handle special characters. I can trivially dump to text for display (by simply running the command as this is the default behavior of the shell), or to feed into text consuming tools.
- NetStrikeForce 10y ago
  I guess that's why HTTP2 is now binary, because text is awesome. https://http2.github.io/faq/#why-is-http2-binary https://http2.github.io/faq/#why-is-http2-binary
haberman 10y ago
> You are considering it just RAW text when it is actually formatted text that has been parsable for years with common unix command line tools. Parsing command output with sed/awk/etc (ie. "common unix command line tools") is absolutely an ad hoc parser. Let me give you an example that I recently ran into. I have a tool that parses the output of "readelf -sW", which dumps symbols and their sizes. The output normally looks like this: 885: 000000000043f0a0 249 FUNC WEAK DEFAULT 13 _ZNSt6vectorIPN3re23DFA5StateESaIS3_EE19_M_emplace_back_auxIJRKS3_EEEvDpOT_ 886: 000000000041c380 64 FUNC GLOBAL DEFAULT 13 _dwarf_get_return_address_reg 887: 0000000000424e60 122 FUNC GLOBAL DEFAULT 13 _dwarf_decode_s_leb128 888: 000000000043dca0 157 FUNC GLOBAL DEFAULT 13 _ZN3re23DFA10StateSaverC2EPS0_PNS0_5StateE So I wrote a regex to parse this. Seems pretty straightforward, right? But then I noticed a bug where some symbols were not showing up. And it turns out those symbols look like this: 5898: 00000000001a4d80 0x801058 OBJECT GLOBAL DEFAULT 33 _ZN8tcmalloc6Static9pageheap_E Notice the difference? Because it's a large symbol, readelf decided to print it starting with "0x" and in hex instead of decimal. I had to update my regex to accommodate this. That is what makes a parser "ad hoc". You write a parser based on the examples you have seen, but other examples might break your parser. Parsing text robustly is non-trivial. Worse, it is an unnecessary cognitive burden. Readelf already had this data in a structured format, why does it have to go to text and back? Why do I have to spend mental cycles figuring out which "common unix command-line tools" (and what options) can parse it back into a data structure?
- mbrumlow 10y ago
  I happen to find your example of why to use a object a bit hilarious. You are right, readelf has the object in a structure -- because that is what elf is... typedef struct { uint32_t st_name; unsigned char st_info; unsigned char st_other; uint16_t st_shndx; Elf64_Addr st_value; uint64_t st_size; } Elf64_Sym; If you wanted the object why did you need readelf in the first place? Why not just read the elf format directly and bypass readelf all together? That seems to be what you are advocating by having readelf passing a object instead of what it does today.

madmax96 10y ago

So much this. What the M$ community fails to see is that text streams can be consumed by __everyone__. We've basically all accepted the fact that programs are used longer and in different ways than we'd expect. A brittle object model can only lead to vendor lock in and indigestible output.

ygra 10y ago
Nothing prevents me from changing those objects into a text stream. In fact, it's infinitely easier than turning a text stream into objects.
- mbrumlow 10y ago
  I feel you missed my point. It's not just a stream of raw data. It's a stream of formatted text... There is no magic or hand waving involved.
temac 10y ago
It's even more than that: text formatting allows the use of generic filtering and text processing tools. Whereas if you are using objects, you will tend to use less on-the-fly command line composition, and write or reuse more tools dedicated to one particular object or the other. In the end I'm not sure keeping the structuring as the default use case yields an usage so much improved in the real world, because if you work on a broad number of types you will tend to need to know more specialised tools, instead of generic ones, and have less higher-order composition tools at your disposal. Now of course you can always format to text from your structured object, but this does not matter. What matters is what is convenient in the proposed UI, and what is mainly practiced in the real world because of such convenience and the amplification loop it creates between tools authors and users. Also some objects are originally handled in text form, and their structured decomposition is extremely complex compared to an informal description and the naturally refinable heuristic which comes with it. For example you can grep into a source code in any language (with more or less efficiency depending on the language, but at least if there is a unique symbol you will find it), whereas trying to get and handle it in a structured way basically means you would need half of a (modular) compiler, and a huge manual to describe the structure, and possibly a non trivial amount of code to actually do the lookup. The PowerShell approach is not all-bad, though, and obviously there are some usages where it superior to text based shells. But for a day to day usage as a general purpose shell, and a programmer shell, I'll stick to the Unix approach.
- Firadeoclus 10y ago
  Structured data allows the use of generic filtering and structured data processing tools. The basic requirement is reflection, objects being able to tell you about their structural composition. If code was stored in a structured representation you could still search for a string object containing a symbol name in the structured representation. You can match a structure pattern just like you match a regex to text. Typical shells can be thought of as REPL for a programming language that makes the file system easily accessible and uses said file system or pipes to pass around data between functions/commands, sometimes as human-readable text. Most programming languages don't encourage passing data around as strings.
blakeyrat 10y ago
The dollar sign is charming. There's a few points here: 1) Not all data is text. In fact, very little of the data people see/work with day-to-day is raw text. It's silly to transform a PNG image into text to be able to pipe it around. (Or to pipe around its filename instead and have a dozen tools all having to open and re-parse it each time.) 2) There's nothing on PowerShell preventing you from serializing a piece of data to text if you want to. The key is: you don't have to. 3) Systems that depend on 50,000 CLI tools all having their own ad-hoc text parsers are cemented, mummified, cannot change. You can't change the output format of ps (to use an example in this thread) without breaking an unknown number of CLI tools. Even if you come up with a great way to improve the output, doesn't matter, you've still broken everything. This is less (but not none!) of an issue with PowerShell. I like computers to evolve to become better over time, and text-based CLIs are a huge anchor preventing that.
- dragonwriter 10y ago
  Unfortunately PowerShell relies on everything running on .NET (well, I think COM works, too); the idea of a shell that can expose live objects is useful, but PowerShell's platform limitations in reality doing that make it a far from ideal implementation of that concept. Something built on a protocol that is platform agnostic would be better.
  - Const-me 10y ago
    Live objects don’t usually expose any protocols at all. They only expose an ABI. Do you know any platform-agnostic OO ABI, besides what’s in .NET? If you’ll wrap your objects into some platform-agnostic protocol like JSON, you gonna waste enormous amount of CPU time parsing/formatting those streams at the object’s boundaries.
    - tracker1 10y ago
      You can run streams of many millions of JSON objects pretty much as fast as the IO can feed it... most of the time, in situations like this, you're constrained by IO speed, not CPU... assuming you are working with a stream that has flow control. I tend to dump out data structures to line terminated JSON, and it works out really well for streams, can even gz almost transparently. Parse/stringify has never been the bottleneck... it's usually memory (to hold all the objects being processed, unless you block/pushback on the stream), or IO (the feed/source of said stream can't keep up).
MaulingMonkey 10y ago
> What the M$ community fails to see is that text streams can be consumed by __everyone__. Text can be poorly parsed by everyone, yes. I especially love it when the default tools settings mean two different computers will give different text results, because the installation defaults changed at some point. What's not to love about trying to properly escape and unescape strings via the command line, while simultaneously keeping in mind your own shell's escaping, and having scripts which are neither backwards nor forwards compatible? And this is to say nothing of different distros. It's as if the text was built for humans instead of my tools half the time, or something. I usually try to centralize my parsing of text in one place so when it invariably breaks, I don't have to rewrite my entire shell script. Some basic structuring - I hesitate to call it a brittle object model, when most of the time I'm dealing with something more akin to C structs than invariant and abstraction laden Smalltalk or somesuch, or Java's sprawl of factories and inheritance - makes things a bit easier. New fields can be added without breaking my regular expressions. I don't need to worry about escapes to handle special characters. I can trivially dump to text for display (by simply running the command as this is the default behavior of the shell), or to feed into text consuming tools.
NetStrikeForce 10y ago
I guess that's why HTTP2 is now binary, because text is awesome. https://http2.github.io/faq/#why-is-http2-binary https://http2.github.io/faq/#why-is-http2-binary

haberman 10y ago

> You are considering it just RAW text when it is actually formatted text that has been parsable for years with common unix command line tools. Parsing command output with sed/awk/etc (ie. "common unix command line tools") is absolutely an ad hoc parser. Let me give you an example that I recently ran into. I have a tool that parses the output of "readelf -sW", which dumps symbols and their sizes. The output normally looks like this: 885: 000000000043f0a0 249 FUNC WEAK DEFAULT 13 _ZNSt6vectorIPN3re23DFA5StateESaIS3_EE19_M_emplace_back_auxIJRKS3_EEEvDpOT_ 886: 000000000041c380 64 FUNC GLOBAL DEFAULT 13 _dwarf_get_return_address_reg 887: 0000000000424e60 122 FUNC GLOBAL DEFAULT 13 _dwarf_decode_s_leb128 888: 000000000043dca0 157 FUNC GLOBAL DEFAULT 13 _ZN3re23DFA10StateSaverC2EPS0_PNS0_5StateE So I wrote a regex to parse this. Seems pretty straightforward, right? But then I noticed a bug where some symbols were not showing up. And it turns out those symbols look like this: 5898: 00000000001a4d80 0x801058 OBJECT GLOBAL DEFAULT 33 _ZN8tcmalloc6Static9pageheap_E Notice the difference? Because it's a large symbol, readelf decided to print it starting with "0x" and in hex instead of decimal. I had to update my regex to accommodate this. That is what makes a parser "ad hoc". You write a parser based on the examples you have seen, but other examples might break your parser. Parsing text robustly is non-trivial. Worse, it is an unnecessary cognitive burden. Readelf already had this data in a structured format, why does it have to go to text and back? Why do I have to spend mental cycles figuring out which "common unix command-line tools" (and what options) can parse it back into a data structure?

mbrumlow 10y ago
I happen to find your example of why to use a object a bit hilarious. You are right, readelf has the object in a structure -- because that is what elf is... typedef struct { uint32_t st_name; unsigned char st_info; unsigned char st_other; uint16_t st_shndx; Elf64_Addr st_value; uint64_t st_size; } Elf64_Sym; If you wanted the object why did you need readelf in the first place? Why not just read the elf format directly and bypass readelf all together? That seems to be what you are advocating by having readelf passing a object instead of what it does today.