5 ms·
The real problem is that Unix commands produce flat text output without any information about how to parse that text back into structured data. Any user who wan
by stcredzero 14y ago
The real problem is that Unix commands produce flat text output without any information about how to parse that text back into structured data. Any user who wants the structured version of the data has to parse it themselves, but these parsers are ad hoc and incomplete by their very nature.
There are a variety of serialization schemes that are quite easy to parse and would be suitable for the output of most Unix command-line tools. BEncode from bittorrent would do nicely, and it's quite easy to parse. JSON would do nicely as well.
Better yet, unify the shell with a virtual machine that is used to implement the OS, and have everything available as 1st class Objects.
- haberman 14y ago> JSON would do nicely as well. Yep, some friends of mine did this with JSON, but didn't make the schema explicit like I mean to: https://github.com/benbernard/RecordStream https://github.com/benbernard/RecordStream > Better yet, unify the shell with a virtual machine that is used to implement the OS, and have everything available as 1st class Objects. Please no. This is the Microsoft PowerShell approach, where everything is a .NET object. Once you start dictating representations of objects, you are dictating far too much about the implementation of individual pipeline nodes.
- nwmcsween 14y agoWhy does the protocol define the objects? Shouldn't the object metadata should have enough information to transform the output? E.g. object is a URL vs protocol is a URL
- neutronicus 14y agoHow can your tools all accept the same kind of structured data without dictating its representation? I don't get it.
- jcmhn 14y agoAllow a shell variable to control output record and field separators. Default to space and newline if nothing is specified. ORS=: OFS=, ls -l # rw-r--r--:1:uname:gname:1025:Jun 1:somefile.txt,...,... Add a specifier for dates, usernames, groupnames, etc. DSF="%Y-%m-%d" ls -l
- haberman 14y agoWhen I talk about a "representation," I mean an in-memory format. For example, the "representation" of an HTML tree is the DOM. Yes, you have to agree on a serialization format (JSON, Protocol Buffers, etc), but that's not the same thing. From a serialization format you can represent the data however you see fit in your process. For example, a C++ user might represent a string as a std::string object whereas a Python user would represent it as a native Python string. The VM-based approach (like PowerShell) defines an in-memory tree representation, namely .NET objects. This means that you can't really interoperate with this stack unless you use .NET too, since you don't have an easy way of converting .NET objects to your own objects.
- stcredzero 14y agoYes, but you could also just use IronPython and script in Python.
- haberman 14y agoThe more decisions you force on your users, the more reasons you give them to choose some other technology, and the less future-proof you are. For example, .NET is only really usable on Windows.
- akkartik 14y agoJust a nit: I think protocol buffers include representations, not just serialization formats. You need to have the schema of the proto to parse it correctly, know which fields are required, repeated, etc. Am I understanding you correctly?
- Evbn 14y agoSchema isn't representation the way haberman is using it. He means something like implementation, which protobuf has in many languages in various VMs and can be easily ported to more.
- haberman 14y agoIt's true that many Protocol Buffer libraries include representations, but these are for convenience; Protocol Buffers are defined in terms of their serialization format and schema.
- griswald 14y agoAs it is not a perfect world, sometimes you will be faced with data that is unstructured and there is no existing library that does everything for you. No JSON, no bencode. The UNIX uitilities have the flexibility to handle these situations. I've seen developers who could not get anywhere with a project because they could not see how to get from unstructured to structured data -- the project was literally a nonstarter in their mind -- and this was because the data did not fit their programming paradigm -- it was not something their libraries accounted for. Meanwhile it was a trivial task to reformat the data using a series of steps (in some cases it was only one step!) with the old UNIX utilties so that they could then work with it using their language of choice. But surprisingly they did not know how to use the UNIX utilities enough to do this themselves.
- darkarmani 14y ago> JSON would do nicely as well. Until you need to incrementally process 2GB of data.
- stcredzero 14y agoBSON in that case?