4 ms·

> You are considering it just RAW text when it is actually formatted text that has been parsable for years with common unix command line tools. Parsing command

> You are considering it just RAW text when it is actually formatted text that has been parsable for years with common unix command line tools. Parsing command output with sed/awk/etc (ie. "common unix command line tools") is absolutely an ad hoc parser. Let me give you an example that I recently ran into. I have a tool that parses the output of "readelf -sW", which dumps symbols and their sizes. The output normally looks like this: 885: 000000000043f0a0 249 FUNC WEAK DEFAULT 13 _ZNSt6vectorIPN3re23DFA5StateESaIS3_EE19_M_emplace_back_auxIJRKS3_EEEvDpOT_ 886: 000000000041c380 64 FUNC GLOBAL DEFAULT 13 _dwarf_get_return_address_reg 887: 0000000000424e60 122 FUNC GLOBAL DEFAULT 13 _dwarf_decode_s_leb128 888: 000000000043dca0 157 FUNC GLOBAL DEFAULT 13 _ZN3re23DFA10StateSaverC2EPS0_PNS0_5StateE So I wrote a regex to parse this. Seems pretty straightforward, right? But then I noticed a bug where some symbols were not showing up. And it turns out those symbols look like this: 5898: 00000000001a4d80 0x801058 OBJECT GLOBAL DEFAULT 33 _ZN8tcmalloc6Static9pageheap_E Notice the difference? Because it's a large symbol, readelf decided to print it starting with "0x" and in hex instead of decimal. I had to update my regex to accommodate this. That is what makes a parser "ad hoc". You write a parser based on the examples you have seen, but other examples might break your parser. Parsing text robustly is non-trivial. Worse, it is an unnecessary cognitive burden. Readelf already had this data in a structured format, why does it have to go to text and back? Why do I have to spend mental cycles figuring out which "common unix command-line tools" (and what options) can parse it back into a data structure?

mbrumlow 10y ago
I happen to find your example of why to use a object a bit hilarious. You are right, readelf has the object in a structure -- because that is what elf is... typedef struct { uint32_t st_name; unsigned char st_info; unsigned char st_other; uint16_t st_shndx; Elf64_Addr st_value; uint64_t st_size; } Elf64_Sym; If you wanted the object why did you need readelf in the first place? Why not just read the elf format directly and bypass readelf all together? That seems to be what you are advocating by having readelf passing a object instead of what it does today.
- haberman 10y ago
  You're asking me why I use a tool instead of parsing a binary format manually? Does that really need explanation? If that is your attitude, why use any command-line tools ever? Why use "ls" when you can call readdir()? Why use "ps" when you can parse /proc? You just pointed me to Kernighan and Pike a second ago. I didn't expect I would need to justify why piping standard tools together is better than programming everything manually.
  - mbrumlow 10y ago
    I never said anything about not liking command line tools. In fact I love them and think they do a awesome job! In any case you just proved my point. You think its insane to parse binary data while scripting and I do too. That is why I think the passing binary objects is insane on the shell. Now if you were talking about text base objects (not binary ones) then that is an entirely different story and I feel that is what we do today. In your example you have rows which could be called objects, and members which would be separated out in columns. To argue a different text base format is better than another is not something I am interested in doing -- mostly because there are a million different ways one could format the output. If you were to do "objects" I think they would have to be in binary to get any of the benefits one could perceived. To be honest I feel the output you posted is a bug in readlef. I would expect all data from that column to be in the same base. I will level with you I can see some benefits of having binary passed between command line programs but I think the harm it would do would outweigh the benefit. But if you you really wanted to do that you could. There is nothing stopping command line utility makers from outputting a binary or any other formats of text. You don't need shell to make that happen. What I think everybody is asking for is for command line developers to standardize their output to something parsable -- which I feel that most command line utilities already do that. They give you many different ways to format the data as it is. Some do this better than others, and I think that would hold true even if somebody forced all programs to only produce binary, or json text format when pipped.
    - haberman 10y ago
      This isn't about binary vs text, it is about structured vs. unstructured. The legacy of UNIX is flat text. Yes it may be expressing some underlying structure, but you can't access that structure unless you write a parser. Writing that parser is error-prone and unnecessary cognitive burden. PowerShell makes it so the structure of objects is automatically propagated between processes. This is undeniably an improvement. I'm not saying PowerShell is perfect. From what I understand the objects are some kind of COM or .NET thing, which seems unnecessary to me. JSON or some other structured format would suffice. What matters is that it has structure. I still don't think you appreciate how fragile your ad hoc parsers are. When things go wrong, you say you "feel" readelf has a bug. What if they disagree with you and they "feel" it is correct? There is no document that says what readelf output promises to do. You're writing parsers based on your expectations, but no one ever promised to meet your expectations. But if the data was in JSON, then there would be a promise that the data follows the JSON spec.
vasilia 10y ago
I would like to see how it will look in PowerShell. $(readelf).ShowMeTheSecondStringFromTheEnd or $(readelf).PrintTheLastColumnInHeX?
- ygra 10y ago
  A comparable PowerShell cmdlet would give you one object per line with properties corresponding to the columns. And no, those properties usually have sensible names, instead of "LastColumn". Of course, for wrapping the native command you'd still have to do text parsing if you want objects. This was more as a comparison of the different worlds here not so much as "if I ran readelf in PowerShell it would get magically different output".
tracker1 10y ago
I've actually become a pretty big fan of line terminated json (all string linefeeds, etc are escaped). Each line is a separate JSON object... In this way, it's very easy to parse, handle your use case, and even pipe-through as JSON some more. In this case, you can have objects as text, and still get plain text streaming with the ability to use just about any language you like for processing.
qwertyuiop924 10y ago
First off, readelf shouldn't switch between hex and base10. Secondly, that's DSV, so you shouldn't have written a regex for it. You should have either cut, or awk, both tools SPECIFICALLY DESIGNED to do what you want.
- Rapidwire 10y ago
  What's DSV?
  - qwertyuiop924 10y ago
    DSV: Delimiter Separated values. readelf uses a delimiter matching the regex /\w+/. In AWK, this is $FS by default, so AWK will parse this by default. Or you can pipe tr -s [:blank:] To cut, which will give you the row you want.
inimino 10y ago
The Unix answer is why are you using a regex on tab-separated values? Wrong tool for the job. Of course the problem with Unix is that there are a thousand different semi-structured text formats, edge cases, and tools that must be mastered before you can make any sense of it all. Any time you point out the pain a Unix fan can just respond by pointing out your ignorance.
- haberman 10y ago
  > The Unix answer is why are you using a regex on tab-separated values? They aren't tab-separated. There are no tabs in readelf output. Also if you assume they are space-separated, note that the first few lines look like this: Symbol table '.dynsym' contains 164 entries: Num: Value Size Type Bind Vis Ndx Name 0: 0000000000000000 0 NOTYPE LOCAL DEFAULT UND > respond by pointing out your ignorance. Indeed.

mbrumlow 10y ago

I happen to find your example of why to use a object a bit hilarious. You are right, readelf has the object in a structure -- because that is what elf is... typedef struct { uint32_t st_name; unsigned char st_info; unsigned char st_other; uint16_t st_shndx; Elf64_Addr st_value; uint64_t st_size; } Elf64_Sym; If you wanted the object why did you need readelf in the first place? Why not just read the elf format directly and bypass readelf all together? That seems to be what you are advocating by having readelf passing a object instead of what it does today.

haberman 10y ago
You're asking me why I use a tool instead of parsing a binary format manually? Does that really need explanation? If that is your attitude, why use any command-line tools ever? Why use "ls" when you can call readdir()? Why use "ps" when you can parse /proc? You just pointed me to Kernighan and Pike a second ago. I didn't expect I would need to justify why piping standard tools together is better than programming everything manually.
- mbrumlow 10y ago
  I never said anything about not liking command line tools. In fact I love them and think they do a awesome job! In any case you just proved my point. You think its insane to parse binary data while scripting and I do too. That is why I think the passing binary objects is insane on the shell. Now if you were talking about text base objects (not binary ones) then that is an entirely different story and I feel that is what we do today. In your example you have rows which could be called objects, and members which would be separated out in columns. To argue a different text base format is better than another is not something I am interested in doing -- mostly because there are a million different ways one could format the output. If you were to do "objects" I think they would have to be in binary to get any of the benefits one could perceived. To be honest I feel the output you posted is a bug in readlef. I would expect all data from that column to be in the same base. I will level with you I can see some benefits of having binary passed between command line programs but I think the harm it would do would outweigh the benefit. But if you you really wanted to do that you could. There is nothing stopping command line utility makers from outputting a binary or any other formats of text. You don't need shell to make that happen. What I think everybody is asking for is for command line developers to standardize their output to something parsable -- which I feel that most command line utilities already do that. They give you many different ways to format the data as it is. Some do this better than others, and I think that would hold true even if somebody forced all programs to only produce binary, or json text format when pipped.
  - haberman 10y ago
    This isn't about binary vs text, it is about structured vs. unstructured. The legacy of UNIX is flat text. Yes it may be expressing some underlying structure, but you can't access that structure unless you write a parser. Writing that parser is error-prone and unnecessary cognitive burden. PowerShell makes it so the structure of objects is automatically propagated between processes. This is undeniably an improvement. I'm not saying PowerShell is perfect. From what I understand the objects are some kind of COM or .NET thing, which seems unnecessary to me. JSON or some other structured format would suffice. What matters is that it has structure. I still don't think you appreciate how fragile your ad hoc parsers are. When things go wrong, you say you "feel" readelf has a bug. What if they disagree with you and they "feel" it is correct? There is no document that says what readelf output promises to do. You're writing parsers based on your expectations, but no one ever promised to meet your expectations. But if the data was in JSON, then there would be a promise that the data follows the JSON spec.

vasilia 10y ago

I would like to see how it will look in PowerShell. $(readelf).ShowMeTheSecondStringFromTheEnd or $(readelf).PrintTheLastColumnInHeX?

ygra 10y ago
A comparable PowerShell cmdlet would give you one object per line with properties corresponding to the columns. And no, those properties usually have sensible names, instead of "LastColumn". Of course, for wrapping the native command you'd still have to do text parsing if you want objects. This was more as a comparison of the different worlds here not so much as "if I ran readelf in PowerShell it would get magically different output".

tracker1 10y ago

I've actually become a pretty big fan of line terminated json (all string linefeeds, etc are escaped). Each line is a separate JSON object... In this way, it's very easy to parse, handle your use case, and even pipe-through as JSON some more. In this case, you can have objects as text, and still get plain text streaming with the ability to use just about any language you like for processing.

qwertyuiop924 10y ago

First off, readelf shouldn't switch between hex and base10. Secondly, that's DSV, so you shouldn't have written a regex for it. You should have either cut, or awk, both tools SPECIFICALLY DESIGNED to do what you want.

Rapidwire 10y ago
What's DSV?
- qwertyuiop924 10y ago
  DSV: Delimiter Separated values. readelf uses a delimiter matching the regex /\w+/. In AWK, this is $FS by default, so AWK will parse this by default. Or you can pipe tr -s [:blank:] To cut, which will give you the row you want.

inimino 10y ago

The Unix answer is why are you using a regex on tab-separated values? Wrong tool for the job. Of course the problem with Unix is that there are a thousand different semi-structured text formats, edge cases, and tools that must be mastered before you can make any sense of it all. Any time you point out the pain a Unix fan can just respond by pointing out your ignorance.

haberman 10y ago
> The Unix answer is why are you using a regex on tab-separated values? They aren't tab-separated. There are no tabs in readelf output. Also if you assume they are space-separated, note that the first few lines look like this: Symbol table '.dynsym' contains 164 entries: Num: Value Size Type Bind Vis Ndx Name 0: 0000000000000000 0 NOTYPE LOCAL DEFAULT UND > respond by pointing out your ignorance. Indeed.