9 ms·
This is exceedingly cool. It could be a modern-day Lisp Machine that takes advantage of the Linux kernel's great hardware support. Since Python is such a dynami
by snazz 6y ago
This is exceedingly cool. It could be a modern-day Lisp Machine that takes advantage of the Linux kernel's great hardware support. Since Python is such a dynamic language, I'd be interested in how they plan to allow the Python processes to communicate. This would be a great opportunity to avoid the classic Unix mistake of using text files as the lowest common denominator everywhere; a structured data format like S-expressions could be passed around instead.
Edit: Other languages I'm thinking of for this idea:
* Lua or LuaJIT (even more dynamic and really great performance; smaller and somewhat fragmented ecosystem)
* Scheme (potentially also fast and good S-expression support, also very dynamic; small and very fragmented ecosystem)
- 0xdeadbeefbabe 6y ago> a structured data format like S-expressions could be passed around instead. Why would a structured format be a better lowest common denominator than an unstructured one like text files? Or is it files that are the issue?
- snazz 6y agoUnix pipes are great, but having to parse plain-text streams is inefficient and error-prone. Imagine how much more powerful it would be if commands returned data in structures that could be used by other applications without parsing an ad-hoc text format. The filesystem can be used to approximate this, like it is in Plan 9, but I think that using text as the lowest common denominator leaves a lot of potential power on the table.
- zozbot234 6y agoUnix-like pipes are not restricted to sending text, they may send general bytestreams. Modern data serialization formats (flatbuffers, capnproto, msgpack, etc.) are highly efficient, and using serialization makes it easier to deploy applications in a distributed setting.
- snazz 6y agoSure, but the standard GNU or BSD userland doesn't do that, does it? If you're going to the trouble of implementing your own userspace, then you might as well use a consistent, proper data serialization format (either text or binary, but text is more accessible).
- throwaway894345 6y agoThe value is in having everything speak a common standard. In unix, that 'standard' is lines of text, which is inherently linear and lacks the ability to idiomatically represent hierarchy. Conceivably, one could use modern serialization formats or we could do something a bit more interesting, like supporting some notion of "objects" directly in the operating system or runtime (in the case of powershell, the "runtime" is the .net vm; in unix it's the OS).
- brutt 6y agoFormat: * Field type: S - 16 bytes, D - 32, L - 64, Q - 128, B - 256, P - 512, T - 1024. * Field ID - 3 bytes ASCII Letters, numbers, '_'; * Field-value separator - 2 bytes: ':' and field value terminator, e.g. ': ' or ':"' or ':[' or ':{' or ':(' ; * Field value 9 bytes ASCII for S, 27 for D, and so on, padded with ' '; * Field separator: 1 byte '\n'; SOBJ: X23456789\n SFN_:"Length" \n SFV_: 23 \n SEND: \n Such text can be parsed by both human and computer, because computer can just jump from field to field without parsing everything in between, like humans do.
- d0mine 6y agoYou don't need to send text (but you should by default for 99%+ cases when the performance is good enough and use the advantages of a human-readable format) The simplicity of a pipe is a feature.
- zozbot234 6y agoUltimately, these "text" formats are simply a matter of convention that makes some binary-formatted streams easier to document for humans' benefit. There's nothing wrong with a binary format if it's properly documented in some way, and modern serialization formats can be augmented with schemas and other features that help provide this.
- laumars 6y agoBeing able to easily read and edit them matters too (pipes are just files so one might choose that file to be a persistent file on disk). Having a none-text format would mean all your file editing tools would have to then understand this new binary format. So you’d have to rewrite every tool to manipulate that data as if it’s text. By which point you’ve effectively just reinvented ASCII.
- travbrack 6y agoPowershell has been growing on me lately. It uses objects to pass data around and it's quite nice.
- AnIdiotOnTheNet 6y agoUsing powershell for a month or so and then going back to doing things with text-parsing UNIX commands really highlights how much better things could be if only we moved past doing things the same way they were done in the 70s.
- chacha2 6y agoHave you checked out NuShell?
- strbean 6y agoI'm really sad I missed the boat so hard on Powershell. I grabbed it at the first public release (beta? alpha?), and the thing was absurdly slow. Trying to `ls` took around 10 seconds. I nope'd out and didn't touch it again. Now I have a coworker who uses it religiously (on linux, who would have thought!) and when I'm watching over his shoulder I'm astounded by how great it is.
- throwaway894345 6y agoHow do people use environment variables with powershell? I often want to do `FOO=bar ./program`, but this doesn't seem to be supported. I googled briefly, but there weren't many satisfactory answers.
- majkinetor 6y ago$Env:FOO='bar'; ./program
- brutt 6y agoIt's equivalent of `export FOO='bar'; ./program` , which is not same as `FOO='bar' ./program`, which, in turn, is equivalent to `OLD_FOO="$FOO"; export FOO='bar'; ./program; export FOO="$OLD_FOO";` .
- riazrizvi 6y agoInteresting idea, might be important to brush up on the implementation rationale of Unix. To that end there is The Design of The Unix Operating System, Bach [1]. Or A Fast File System for Unix [2]. Or The Design and Implementation of a Log-Structured File System [3]. It seems to me that files solve for the problem of how to deliver an economic memory management system, and they do it by paging cheap disk space so that it's an effective proxy for fast-but-expensive memory. Also files, with their file pointers, can certainly be used as data structures, but this 'understanding' of what each section means is clearly variable, so is best left to specific applications and it's not good to build into the system as a default. Different applications will want different structures for optimal performance... [1] https://www.amazon.com/Design-UNIX-Operating-System/dp/0132017997 https://www.amazon.com/Design-UNIX-Operating-System/dp/01320... [2] https://people.eecs.berkeley.edu/~brewer/cs262/FFS.pdf https://people.eecs.berkeley.edu/~brewer/cs262/FFS.pdf [3] https://people.eecs.berkeley.edu/~brewer/cs262/LFS.pdf https://people.eecs.berkeley.edu/~brewer/cs262/LFS.pdf
- bityard 6y ago> This would be a great opportunity to avoid the classic Unix mistake of using text files as the lowest common denominator everywhere There are a great many who do not consider this to be a mistake. The fact that after half a century, Unix-like operating systems and programming environments are de rigueur is plain enough evidence of this. I can't think of much else in the whole field of computing that has had this kind of staying power over such a long period of time, excepting perhaps the idea of binary digital computers themselves. Fun history note: Part of the reason the Unix userspace and development chain lean so heavily on text processing is because its inventors were tasked with designing a system for more powerful tools for the writing and production of technical documentation. But they were dyed-in-the-wool systems and programming nerds so stand-alone documentation tools were not very interesting. So their solution was to invent a new OS, borrowing the features they liked from other OSes they had been exposed to, under the guise of delivering a documentation production system. If it turned out that this Unix thing they came up with happened to be an excellent programming environment as well, that was just a nice bonus as far as management was to be informed. One of the greatest hacks of computing history, if you ask me. Edit to add: As others have pointed out, you are hardly restricted to piping around text. And if you do, the power of text is that you can impose arbitrary structure on top of it, e.g. tabular data, shell scripts, XML, YAML, JSON, etc. Saying it's too bad Unix abstracts everything as text files is like saying it's too bad microprocessors abstract everything down to either a one or a zero when far more interesting number exist.
- deleted 6y ago[deleted]
- pwdisswordfish2 6y agoWasn't UNIX originally used to format patent applications at Bell Labs/AT&T? That was purportedly the selling point of UNIX to the business people. When someone suggests using something other than "text" (usually it's some binary format, I consider S-Expressions to be text), I always wonder, why is this person writing their suggestion using text? Then we have the suggestions that "everything is a file" is also a "mistake". That's great but if a "file" is not a good enough abstraction then what is?
- matheusmoreira 6y agoLinux is the best operating system for a project like this. The system call binary interface is stable at the processor architecture level: https://www.kernel.org/doc/Documentation/ABI/stable/syscalls https://www.kernel.org/doc/Documentation/ABI/stable/syscalls http://man7.org/linux/man-pages/man2/syscalls.2.html http://man7.org/linux/man-pages/man2/syscalls.2.html http://man7.org/linux/man-pages/man2/syscall.2.html http://man7.org/linux/man-pages/man2/syscall.2.html Any Linux process can perform Linux system calls, no matter what language the software is written in. Instead of relying on C libraries to do everything, a compiler could emit system call code when necessary. The entire Linux user space could be rewritten in a higher level language that way. Rust would be the likely candidate but there's no reason a Lisp or Scheme couldn't do it. The language would need full support for features of binary interfaces such as function calling conventions, pointers, structures with padding and alignment, etc.
- heavyset_go 6y agoDo you know of any projects exploring this space?
- anton96 6y agoIn matter of scheme you have the guile project. https://www.gnu.org/software/guile/guile.html https://www.gnu.org/software/guile/guile.html
- slim 6y agoso maybe guixsd ?
- yencabulator 6y agoGo does syscalls directly on Linux. Libc is only needed if you use a thing that only exists in libc (e.g. NSS).
- AnIdiotOnTheNet 6y agoIn that vein, Zig also uses syscalls directly on Linux in its standard library so as to avoid linking with C unless desired. This is not possible on the various BSDs.
- lifeisstillgood 6y ago> This would be a great opportunity to avoid the classic Unix mistake of using text files as the lowest common denominator everywhere Text files are just a binary format that happens to be easily readable by humans, thus making the hurdle of parsing or generating something to use that format much lower. It usually is this hurdle that is most important to longevity and wide spread success than close to metal sympathies- cf JSON and Protocol Buffers
- ken 6y agoText files are easily readable on a Teletype or a VT100. That was a solid design decision back in the day when you had to say "screen editor" to distinguish them from the ordinary line editors. We didn't have interactive debuggers and inspectors and loggers, so we did everything we could to make systems naturally debuggable. It's like how the Model T engine cover just lifted off with a big door handle. When a technology is new, people using it are expected to be mechanics, too. But today, even amateur wrote-it-over-the-weekend programs have realtime visual editing of rich data. There are countless free libraries for serialization of every format, and GUI (and TUI) libraries for every toolkit/language/style. Developers enjoy using JSON because it lowers the "hurdle of parsing or generating". It's not 1970 any more.
- traverseda 6y agoI'd love to see something like RPYC used for interprocess-communication.
- kbr2000 6y agoIt's been done using Tcl [0] in the nineties, in the EtLinux [1] project for embedded systems. One of the goals was to shave off memory usage by implementing tools under a single scripting interpreter (instead of each of them having their own binary). Kind of overlaps with BusyBox [2] philosophy... [0] https://www.tcl.tk/ https://www.tcl.tk/ [1] http://www.etlinux.org/ http://www.etlinux.org/ [2] https://busybox.net/FAQ.html#goals https://busybox.net/FAQ.html#goals
- amyjess 6y agoThinking about structured data at the OS level makes me want to see a revival of Classic Mac OS's resource fork. Structured metadata in every file is a dream of mine.
- Animats 6y agoThe resource fork idea was OK, but it suffered from a implementation optimized for a floppy disk. Writes were deferred as long as possible, and if anything went wrong, the whole data structure was corrupted. "Text Edit is not a text editor, and the resource fork is not a database" - early Apple publication. Both could have been. Programs tend to need little databases, for preferences and such, and the resource fork was way ahead of doing everything with a text file. But the implementation was never toughened up to database standards after the Macintosh got a hard drive and more memory. And Text Edit was limited to 32K for a long time. Meanwhile, Unix struggled along with text files for everything. And lock files for the text files. And daemons to remove dead lock files. And a lack of file locking. And a lack of interprocess communication. Eventually the Unix/Linux world got that all fixed, but way too late.
- AnIdiotOnTheNet 6y agoThis could be handled by convention of treating directories as though they were a single structured "file" under certain circumstances. There is precedent in the form of Application Directories from NeXT, RiscOS, and ROX Desktop. Modern MS Office files are also basically just a zipped up directory of files and metadata.
- joyj2nd 6y ago"This would be a great opportunity to avoid the classic Unix mistake of using text files" Don't get it. Where is the mistake? I think it was its biggest strength. In the successor, Plan 9, everything is a file...
- linguae 6y agoI have a similar idea. I've been working on a design for a Common Lisp-based desktop environment that can run on top of Linux and the BSDs. It's inspired by the Lisp machines of old and OpenDoc, an API for building component-based software that was championed by Apple and IBM back in the 1990s before Steve Jobs returned. I haven't started writing any code yet, partly because I've been learning X programming and other parts of the graphics stack like Cairo and Pango. I have some more ideas at http://mmcthrow-musings.blogspot.com/2020/04/a-proposal-for-flexible-composable.html http://mmcthrow-musings.blogspot.com/2020/04/a-proposal-for-..., though I wrote this before I committed to using Common Lisp (I talk about using GNUstep near the end of the document, but I've since ruled it out due to the difficulty of finding modern Objective-C bindings for SBCL that work with GNUstep).
- tlavoie 6y agoThere is Mezzano, a Common Lisp operating system. Currently designed to be run within Virtualbox. https://github.com/froggey/Mezzano https://github.com/froggey/Mezzano