18 ms·

I often wonder what if we decide to build a completely new, well thought out computing system (including network) that is totally separate from any legacy syste

I often wonder what if we decide to build a completely new, well thought out computing system (including network) that is totally separate from any legacy system and 100% backwards incompatible. Not any bit less, 100% backwards incompatible and everything would be fresh. There would be no jpeg support (it would have its own image format), it would not have standard TCP/IP stack - a new protocol and network infrastructure, new display format, new IO ports, etc. Why am I imagining this? I feel like computing systems as they've evolved, they've built stuff on layers upon layers and just like biological evolution, it is impossible (difficult) to undo because the marginal cost is too high to unwind and break backwards compatibility. We get stuck in local maxima at each layer and the critical energy required to escape out of it would be too large. I am not saying backwards compatibility is bad, far from that, I am saying that if we force ourselves to cut the cord and completely build computing systems from scratch - I feel like we would have a hindsight to design it better, faster, and cheaper.

jamesmishra 6y ago
I think it is possible to build a better system from scratch, and there are likely space aliens that have such better systems. But ultimately a computing system should do what its users want, and the market has favored backwards-compatibility for a long time. If anything, it's a little alarming how old software and hardware can quickly become useless or obsolete. For example, any web browser that doesn't support any encryption better than, say, SSLv3, is cut off from a significant fraction of the Internet.
- squibbles 6y ago
  > I think it is possible to build a better system from scratch, and there are likely space aliens that have such better systems. I respectfully disagree. Alien systems can be thwarted by inverting some bits. [0] I would not have believed this had I not seen it for myself. [1] After all, seeing is believing. True, the alien code seems very insecure. Perhaps aliens write their code based on the historical documents. [2] [0] https://www.reddit.com/r/movies/comments/2oi7cz/i_am_dean_devlin_writer_producer_of_independence/cmo0r1y/?utm_source=share&utm_medium=web2x&context=3 https://www.reddit.com/r/movies/comments/2oi7cz/i_am_dean_de... [1] https://www.youtube.com/watch?v=cgqswujkxPE https://www.youtube.com/watch?v=cgqswujkxPE [2] https://www.youtube.com/watch?v=lLZQz26-kms https://www.youtube.com/watch?v=lLZQz26-kms
throwaway9870 6y ago
I think you underestimate how many smart people proceeded you. Yes, in some cases there are new technologies that transform how we build things and using those from the start might put you in a better place. But in-between those inventions is a lot of hard work by really smart people that would be incredibly hard to match let alone exceed. To build a system, you would have to start at the bottom layers and then build up. To think you could design any of those layers (protocols, languages, tools, etc.) without making mistakes is naive and I think you would end up with a system that is layered just like current systems, but each layer being much less thought out. Yes, that is even with complete hindsight. I have watched many experienced teams fail on gen 2 of systems. Don't get me wrong, there are many things I dislike in computers, starting with autotools :) But things like autotools are a lot easier to replace in Hacker News comments than in real life.
- systemvoltage 6y ago
  I had no intention to belittle the complexity, monumental effort and almost impossible to imagine amount of man hours that would go into this - it was purely a thought experiment. Do you really think that it would be less thought out if we have lots of resources and smart people? I feel optimistic that we can bake in the lessons from the past and improve on existing layers/stack/protocols.
  - oh_sigh 6y ago
    I think the problem is that, even if you end up getting a proof of concept that attracts others, and they end up being 100x as productive as people using the Old Way, you will still be outpaced because the Old Way has 100,000x as many people working on it as there are on the New Way.
    - divbzero 6y ago
      The odds are stacked against any specific New Way from taking hold. But we can also invert our perspective and ask: Will there a dramatic shift to a New Way in the next half century? Since 1970 there have been at least 3 major transitions in how people work with technology, it’s easy to imagine more will come before 2070.
stunami 6y ago
Somewhat reminded me of the constructive look at TempleOS http://www.codersnotes.com/notes/a-constructive-look-at-templeos/ http://www.codersnotes.com/notes/a-constructive-look-at-temp...
jcrawfordor 6y ago
One of the overriding themes which I attempt to convey in my teaching, blog, etc is that the modern technology landscape is something that we arrived at largely by chance. Nearly every decision made in the development of computing had a wide variety of alternatives at their time, but once things become ossified as a norm it is difficult to imagine another way. For example, I have recently mentioned a few times how it is hard now to see TCP/IP as not being the "obvious design," but in the early '90s there was healthy competition of network protocols offering various feature sets. I think it is extremely important from a teaching and engineering perspective to view the modern computing environment not as some elegant design (which it most certainly is not) but instead as an concretion of "best option at the time" solutions to real problems. It is a bit intoxicating to think about how things could be if computing really were approached as a single, monolothic, elegant design. But essentially every attempt to do so has failed. Much like how we think of computing as layers of abstraction, the computing industry is functionally constructed of layers of abstraction, and so it is difficult to really change anything at lower levels.
- shakezula 6y ago
  On one hand, it’s fun to imagine what a truly full-stack rewrite of the internet would allow us to do, and what it would look like. On the other hand, the current internet feels like a good approximation of the best trade offs for performance vs. usability in most areas, and I doubt a full rewrite would fundamentally change much about how we use the internet.
  - notduncansmith 6y ago
    > I doubt a full rewrite would fundamentally change much about how we use the internet. I imagine that it would in fact have to support people using the internet exactly the way they do now, if it hoped to get traction.
  - egypturnash 6y ago
    An Internet with a payment system built in from the ground up might be one that did not end up throwing its hands in the air and saying “I guess everything will be funded by ads, I dunno”. An Internet where “keep engagement high so we can serve shitloads of ads” was not the default path to financial success would be an Internet without social media that happily pushes people into ever-more controversial positions, because that’s what keeps them around longer and lets you serve more ads. Decide for yourself what social changes that would create.
yjftsjthsd-h 6y ago
We could, sure, but nobody would use it, and at some level they'd be right, since the point of computers is to do stuff for us, and a computer that (intentionally!) doesn't work with anything else is an island, which is to say, not very useful.
KMag 6y ago
It's possible to remove layers of cruft without throwing everything out. QNX and BeOS both implemented mostly POSIX compliance on top of radically different foundations. We've gotten rid of the ISA bus and that old pre-PS/2 mouse port, and mostly gotten rid of PS/2. In terms of display technologies, we have DisplayPDF from Apple and Wayland on the Linux side. You take huge risks by just throwing out everything at once. You want a modular system anyway, so run your revolution one module at a time. People don't notice your little revolutions, but that's kind of the point.
- matheusmoreira 6y ago
  > mostly gotten rid of PS/2 Isn't that port required if users want keyboards with true N-key rollover?
  - baybal2 6y ago
    No, USB HID API can have all 104 key rollover if you want. The urban myth has something truthful behind itself, as at some point Windows refused to work past some number of key rollovers, and then keyboard makers started to limit it to accommodate Windows.
    - matheusmoreira 6y ago
      > then keyboard makers started to limit it to accommodate Windows Wow, I really had no idea. That's really frustrating... Any way to fix this? Maybe we can somehow hack the keyboards. They have an USB controller with firmware on it, right?
      - baybal2 6y ago
        Have the keyboard maker order a controller chip that does HID in a way compliant to spec
      - analognoise 6y ago
        USB capable microcontrollers are a buck or two in unit quantity; most of them with a USB drivers fron the manufacturer have a HID device as a standard example. It's like "Hello, World!" but for USB. Hack? Nah. Engineer? Easily. But the other commenter who noted that it was a BIOS limitation was right - it's not a Windows thing.
dpc_pw 6y ago
> I often wonder what if we decide to build a completely new, well thought out computing system Check out Urbit.
throwawaywrench 6y ago
>I feel like we would have a hindsight to design it better, faster, and cheaper. Maybe I'm just pessimistic but I suspect the opposite would occur. It would be buggier, slower and more expensive once you started trying to operate at any kind of scale. The reason being that hidden within all the accumulated layers of cruft are a million fixes for specific security, reliability and performance issues discovered through real world usage.
- ithkuil 6y ago
  I think that's the interesting word: "hidden" That's a pity. Wouldn't it be nice if we somehow could separate the cruft from the craft? What if some computational deity sifted through our collective codebase and distilled the essence, the minimal, purest kernel? Such minimangel's task would be daunting and probably useless, since the moment we'd throw some new use cases and requirements, NIH and patents, POCs and MVPs and we're back to the simmering mess in no time.
  - croo 6y ago
    Those hidden snippets are not the cruft. Every ugly hack we would find is likely a required use case and the whole mess is the craft. Maybe we should stop calling it a mess and accept that non-trivial software with on-demand requirements always take this form... So lets call it detail instead. You cannot separate detail from the craft.
    - ithkuil 6y ago
      I meant that the hidden things were the "craft" hidden among tons of cruft
      - tremon 6y ago
        I think you'll rather find that there's tons of craft, and relatively little cruft.
        6y ago
alexmingoia 6y ago
That’s what Urbit is: https://www.urbit.org https://www.urbit.org
- nineteen999 6y ago
  > totally separate from any legacy system It most certainly is not. It's a macOS and Linux userspace application, no more no less[1]. How on earth do you consider that totally separate from any legacy system? [1] https://urbit.org/using/install/ https://urbit.org/using/install/
  - alexmingoia 6y ago
    Urbit runs as a VM with a translation layer for things like networking, file system, etc. Using a host OS is expedient to build a working OS that the user actually wants to use, since hardware support is an uninteresting implementation detail. Eventually, one could write a native hardware kernel in Nock/Hoon, and do networking without TCP/IP/DNS between Urbit computers.
    - nineteen999 6y ago
      > that the user actually wants to use Only a handful of people would rather use something like Urbit than established systems. > Eventually, one could write a native hardware kernel When it does, you can call it an OS. For now, it's a user space application.
      - dfischer 6y ago
        Definitely agree with you. The practical implementation that it is, vs what it advertises itself as visionary-wise.
        nineteen999 6y ago
        > what it advertises itself as visionary-wise All puff. The amateurs over at /r/osdev are much closer to running directly on hypervisors or bare metal than Urbit will ever be.
lsllc 6y ago
Its been tried, for example Taos: https://sites.google.com/site/dicknewsite/home/computing/byte-articles/the-taos-operating-system-1991 https://sites.google.com/site/dicknewsite/home/computing/byt... Very cool and groundbreaking technology! NOTE: Discussed previously on HN: https://news.ycombinator.com/item?id=9802379 https://news.ycombinator.com/item?id=9802379 https://news.ycombinator.com/item?id=9806607 https://news.ycombinator.com/item?id=9806607 https://news.ycombinator.com/item?id=15470338 https://news.ycombinator.com/item?id=15470338 https://news.ycombinator.com/item?id=15527936 https://news.ycombinator.com/item?id=15527936
AnimalMuppet 6y ago
It would be 100% user free, as well. When we started moving to cell phones, you could still call a landline phone from one. When we created web browsers, they could hit gopher or ftp URLs. When Stroustrup created C++, he let C code freely interoperate, which let people migrate at their own pace, or even flip back and forth. When Windows was created, it let DOS programs run. When IBM introduces a new mainframe, they always have a way to run programs from the previous generation (back to the 1960s, at a minimum). In all of these cases, interoperability with the old system was essential for the growth of users for the new system. How many computer users are there now? Three billion? Break backward compatibility and interoperability, and you lock half the planet out - the half that uses computers. And the other half... if they decide to use a computer, are they going to use what everyone else uses, or are they going to decide that your completely incompatible new thing is the right answer? Even if (in some platonic sense) your new thing is the right answer, how are they going to know that? So, like I said... zero users. 100% user free.
xkcd-sucks 6y ago
Aha, I have deduced you have a lucrative-enough career in B2B/"enterprise" software
- browserface 6y ago
  :) haha. Underrated comment. ;p ;)x xx
visarga 6y ago
> We get stuck in local maxima at each layer and the critical energy required to escape out of it would be too large. ML solution - many restarts, add momentum and in the end maybe all local maxima are equally good after all.
deepsun 6y ago
One fun exercise I had is to disable IPv4 in my network settings and go with IPv6 only, see what's left working.
matheusmoreira 6y ago
Hardware is much more complex than software and also requires incredible amounts of money in order to fabricate. Not sure if that will ever change. We'll probably remain stuck with whatever is available on the market. Starting from scratch is actually feasible on the software side though. The OSDev wiki has many examples. The most recent one I know is Serenity. Still a monumental task. It's probably a lot of fun if you have the time and motivation to stick with it. You can actually throw POSIX in the trash and create your own abstractions. Hopefully people aren't reinventing signals... I think Linux is the best option for those who want to create something from scratch. The Linux kernel is unique in that it is completely separate from user space. There is a stable, documented kernel interface in the form of system calls: https://man7.org/linux/man-pages/man2/syscall.2.html https://man7.org/linux/man-pages/man2/syscall.2.html https://man7.org/linux/man-pages/man2/syscalls.2.html https://man7.org/linux/man-pages/man2/syscalls.2.html https://www.kernel.org/doc/Documentation/ABI/stable/syscalls https://www.kernel.org/doc/Documentation/ABI/stable/syscalls https://www.kernel.org/doc/Documentation/ABI/stable/vdso https://www.kernel.org/doc/Documentation/ABI/stable/vdso So people can actually do things like trash all of GNU and come up with something entirely new. It's possible to rewrite the entire Linux user space in Rust or Lisp or whatever. It doesn't have to be POSIX compliant. The kernel even has this amazing header which lets people build freestanding Linux programs: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/include/nolibc/nolibc.h https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
- eindiran 6y ago
  That header file is amazing. Thank you for the pointer to that. For anyone wondering, it lets you build programs without a libc: > This file is designed to be used as a libc alternative for minimal programs with very limited requirements. It consists of a small number of syscall and type definitions, and the minimal startup code needed to call main(). All syscalls are declared as static functions so that they can be optimized away by the compiler when not used.
  - matheusmoreira 6y ago
    > That header file is amazing. Thank you for the pointer to that. Yeah, it's an awesome header! It comes straight from the kernel developers too. They use it for their own tools, apparently. It might be missing some of the newer or niche system calls but it has generic mechanisms which can be used to access everything. I found it during my research for my own Linux system calls library project: https://github.com/matheusmoreira/liblinux https://github.com/matheusmoreira/liblinux There's some additional details implemented by my project such as process start up functions which handle argument, environment and auxiliary vectors. I don't see the need to maintain it anymore though since nolibc.h is so amazing and practical. I even asked Greg Kroah-Hartman about nolibc.h on Reddit: https://old.reddit.com/r/linux/comments/fx5e4v/im_greg_kroahhartman_linux_kernel_developer_ama/fn3rpht/ https://old.reddit.com/r/linux/comments/fx5e4v/im_greg_kroah...
gabcoh 6y ago
I feel like IPFS is taking an interesting approach to this. They aren't trying to rebuild modern computing from scratch, however they are designing a tool which, as the name suggests, has the ability to scale basically unbounded. I'm talking in particular about their focus on defining specs that are themselves parametrized by other specs. At the base of this is multiformats which are a suite of specs for things like network addresses which allow you to wrap any current network protocol like tcp over ip over ethernet or dns over https over quic over udp over whatever. By using these paremetrizable protocols they are lowering the marginal cost you mention and making it much easier to migrate to improved technology in the future. It's been a while since I seriously looked into IPFS, but this is what I remember about it.
Razengan 6y ago
> I often wonder what if we decide to build a completely new, well thought out computing system (including network) that is totally separate from any legacy system and 100% backwards incompatible. Not any bit less, 100% backwards compatible and everything would be fresh. There would be no jpeg support (it would have its own image format), it would not have standard TCP/IP stack - a new protocol and network infrastructure, new display format, new IO ports, etc. The iPad is probably the biggest such attempt to do most of those things. :)
jacobwilliamroy 6y ago
Okay, but you should still run ZFS because I really don't think anyone is going to make a better file system.
- aethertron 6y ago
  If you don't have files, you don't need a file system http://web.archive.org/web/20081120092935/http://www.memetech.com/nofiles.html http://web.archive.org/web/20081120092935/http://www.memetec...
  - jacobwilliamroy 6y ago
    Yes. ZFS has a couple layers of caching in RAM and it actually reads from those caches first, before trying to read from disk. And then it caches whatever it has to read from disk, in RAM. That's why you need lots of ECC RAM to run ZFS. Because errors which propogate from RAM to disk cannot be corrected and such will destroy the whole file system.
- zokula 6y ago
  Ze cult of ZFS strikes again. Comment is funny in assuming ZFS is objectively better than NTFS or APFS or FAT16.
  - jacobwilliamroy 6y ago
    I actually use openzfs. I like it. Why are you giving me shit for that?
tomc1985 6y ago
You'll have a lot more success, I think, if you curate the things you keep and the things you throw out. All of the successful (relatively speaking) 'whole cloth' rethinks did this.
crististm 6y ago
You are on to something but I think this would mean throwing the baby with the bathwater. "From scratch" is a loose definition. And there will necessary be reinventing the wheel. Making the new wheel incompatible on purpose with the old ones is wasteful. For one thing, you are correct. You don't build a car on a carriage framework. But the principles are the same and there are some technologies that will be reused: bushings for example. What should we do about them?
janee 6y ago
Isn't TCP a good example of these two strategies, e.g. where imperfect natural evolution won over it's competitors which I would say is an example of what your describing https://en.m.wikipedia.org/wiki/Protocol_Wars https://en.m.wikipedia.org/wiki/Protocol_Wars I think in general doing a grass roots, ground up, all encompassing design is inferior to a more battle tested natural evolution based strategy where you build on top of each other. Might be wrong, but that's my gut feel. I'm sure there some good examples of the two strategies and how they played out?
ROARosen 6y ago
Interesting, I was thinking the exact same thing but in regard to real-life things. Like imagine cars / roads were invented today do you thing they would have all have this same ubiquitous elongated design, with the grill in front and two headlights in the side, or maybe cars would be completely round maybe they would be made generally with standup room, one big head light across the bottom etc. While some of these design decisions are practical, I believe a lot would be different if we would reimagine it from the bottom up. The reason - for instance - cars cannot be wider is because the wouldn't fit on today's roads roads. Existing safety regulations also prevent certain design possibilities.
- systemvoltage 6y ago
  Holyshit, I have thought about the exact same thing. Like totally questioning everything about cars and why do they look the way they look. Cheers, made my day. Have you also though about how we see 3 holes on a face (2 eyes and 1 mouth, ignoring the downward facing holes of the nostrils) as normal. Just add one more and it’s horror. We take these things granted and then slight changes to it is completely terrifying. I also wonder - if we rewind evolutionary time and restart with slightly different initial conditions, not sure what species including humans would even exist in the same form. It would likely be like that game Spore.
gigatexal 6y ago
What’s wrong with JPEG?
simonebrunozzi 6y ago
Absolutely agree with you. Same, with larger impact, would be if we apply this thinking to how we build cities. Think about it for a moment. How much "crap" and "stupidity" is there just because of legacy.
browserface 6y ago
Soviet computers are an interesting historical riff on this. But I think this would be impossible to cut the cord completely, because a computer is basically a communication system (internally and networked) and by Conway's law groups zero in on comms systems that reflect their own dynamics, so as a civilizations we've built devices and networks that have zeroed in on our own internal dynamics. Therefore as a member of this civilization, you could not create something different. You'd need to go hang with some folks completely different until you absorbed that, then make it to reflect their dynamics. Uncontacted Amazon tribes? Dolphins? Elephants? Plants?
acrossthepond10 6y ago
I've always wondered if its possible to write software that will run on computers with arbitrary number of memory heirarchy eg. x amount of l1/l2 cache -> main memory (RAM) -> SSD -> HDD -> Archive storage where you can plug in a new type of memory as it becomes available (eg. 3dxpoint) and it will adapt to run optimally on that hardware without rewrites.
- acrossthepond10 6y ago
  Or designing software for a theoretical system like an ideal reconfigurable computer with reconfiguring latency that is as fast as processing latency and then have hardware engineers design towards that as opposed to having software engineers target hardware like we do know. Feels like that would make for an interesting experiment.
- Reelin 6y ago
  > I've always wondered if its possible to write software that will run on computers with arbitrary number of memory heirarchy I think that's already possible today - the Legion programming system claims to do exactly that (among other things). I have absolutely no experience with it though so I might be completely wrong. (https://legion.stanford.edu/ https://legion.stanford.edu/) > By making the Legion programming system aware of the structure of program data, it can automate many of the tedious tasks programmers currently face, including correctly extracting task- and data-level parallelism and moving data around complex memory hierarchies.
despair3435 6y ago
With regard to networking, please have a look at the Ouroboros project [0][1]. [0] https://ouroboros.rocks https://ouroboros.rocks [1] https://arxiv.org/abs/2001.09707 https://arxiv.org/abs/2001.09707
Cthulhu_ 6y ago
What came close was the smartphone (and a bit later, tablets); one big one is that they did not support Flash, which (at least in my neck of the woods) was an important factor in the shift towards webapps / SPA's. At the same time, browsers / JS became faster, NodeJS became a thing, and because of both apps, webapps and REST interfaces, the inefficiencies and insecurity of HTTP came to the forefront, leading to (more, easier, accessibe) HTTPS and HTTP/2, and now a push to UDP-based protocols like quic / HTTP/3. Anyway, smartphones / tablets have made us rethink hardware, its developments have led to more compact laptops as well, and the push to more compact laptops have caused their manufacturers (mostly thinking of Apple throughout this comment btw) to eschew legacy systems - disk drives, USB ports, even 3.5mm jacks all went out the window. But that is exactly what you're proposing here; what did you think when Apple got rid of a lot of that legacy? Personally I have mixed feelings; I do like USB-C. I didn't like Lightning because it was nonstandard, but it was an improvement over their broad connectors and USB-C wasn't a thing yet. I can see why they're pushing to wireless connections for audio and input. Anyway final point, despite the cut in legacy connectors and the like, Apple's hardware hasn't gotten cheaper.
- gmueckl 6y ago
  But smartphones - both iOS and Android - take their core operating systems essentially from the 70s. The kernels themselves are few decades newer than that. The concepts and interfaces implemented by them, however, are not. Similar things can be said about Windows Phone and NT. Arguably, using an existing OS as a base was kind of the only way to get these products done in a reasonable timeframe. They run complex and powerful software and writing that stack alone is hugely demanding. Rewriting the OS would easily have lost the market to the competition because of cost and time overruns.
tremon 6y ago
Why wonder? That's basically what IPv6 did for networking, or what Intel tried with Itanium, but with even larger gap to existing usage. The cost of switching would be so astronomically high, your product would fail before it even started. "Throw out the old" happens in local pieces of the stack all the time. AMD removed a lot of cruft from the Intel ISA when they designed AMD64, Wayland removes a lot of complexity from the display protocol stack, and every I/O port before USB was designed to be incompatible with existing ports. I feel that what you are describing will look like the SBC situation in the ARM world: the processor instruction set may be identical, but every vendor uses a slightly different incompatible way of connecting and designing the surrounding chipset, which results in a myriad of almost-identical drivers and subtle bugs that are impossible to fix, because the cost/benefit ratio would be prohibitively high. Standardization isn't just for backwards compatibility, nor does it mandate it. But it does allow for reuse of existing components in an interoperable way. Imagine if every application came as a unikernel microservice instead of an OS framework library: there would be no UI consistency, copy-pasting between different applications would be hard, and you would have no OS-level update facility -- every application would have its own set of vulnerabilities.
- pjmlp 6y ago
  Intel only failed with Itanium, because AMD exists and had access to Intel's IP. Without AMD64 Itanium would have prevailed.
jiggawatts 6y ago
There are some contexts where this makes sense. As far as I know, currently there are no "good" real time operating systems available. There are some ancient, crufty ones... and some modern soft-real-time operating systems, but that's it. This would be a fun exercise for someone wanting to write something uniquely useful in Rust, for example. It might make sense to write a pure web browser engine designed for embedded contexts. Think infotainment systems where the web content being served is fully controlled. This would be an opportunity to write something like a HTML5-only engine with UTF8 only, TLS 1.3 only, HTTP 3 only, etc... Relational database engines need a massive rethink. All of the current popular ones are 20+ years old, and based on 50+ year old architectures. Why do schema operations take time? Why does index rebuilds lock tables? Why are there column size limits? Why is SQL injection still a thing? Why, why, why... because legacy, that's why. Because hard-coding, that's why. Because 1980s C-style programming with fixed-size buffers, that's why. All of this could be just washed away with a rewrite using Rust, Modern C++, and some fresh ideas.
dfischer 6y ago
I 100% agree with this and was just talking about this in another thread on one of my earlier comments. I love what Plan 9 and Oberon was working on. I feel many are feeling similar at this point given the frequency I'm running into this and also I myself wanting to solve for it. Awesome! I think the networking layer should be built off of http://PJON-technologies.com http://PJON-technologies.com - it's perfect for standardizing the next layer of networking and without depending on any ISP or centralized authority that also has support for multiple "packet/connection" protocols (LoraWAN, Light, Ethernet, and more). I am going through a history lesson right now on studying older OS's and why they didn't take off while also getting my feet wet with Forth and lower-level hardware experiments. I still have no idea how video truly works, that's something that's been bugging me from a first principles perspective and I want to understand that more intuitively. Something I keep asking myself is: "How do I poke at the video card and make it display stuff without having to use a OS level API that wraps openGL or similar?" - I know some will be shocked at that question and some may go "woah, actually I don't know that" - we're in an interesting generational leap with where knowledge has been in the zeitgeist of computation and networks. The OG's are in a particular precious bucket where they've done their part and not that active or have been yelling the current stack is wrong but it's hard to find their voices. Then you have the hackers and hardware folk who learned a lot pre-internet that may or may not be active on the typical net's hanging out only in Plan 9 esoteric rooms saying how good the internet was before marketing took over, and then the dot com boom generation which either knows the earlier internet or thinks the internet is only social media and search engines. I want to make the internet feel adventurous and fun again like the 90's. I want it to be focused on individuals and not organizations. It was exciting because you connected to people, today you don't do that anymore - you interface with some centralized entity that exposes you to products. I have more to say here that is a lot more nuanced but you get my gist. Urbit is doing something similar to what you're saying, however I personally don't like their networking model and a few other decisions that just feel contrived to me. Plus I want to get my feet wet and understand the layers myself from a first principles perspective so I can add to it as I am imagining it. Sounds like a new dawn of innovation is bubbling. My previous comment on a similar tone/opportunity from yesterday: https://news.ycombinator.com/item?id=24467534 https://news.ycombinator.com/item?id=24467534
asplake 6y ago
Gall’s law though. For that complex system to work, it would have to be based on something simpler that worked. Yes you might well come up with something radically different, but still you’d have gone through some of the same evolutionary process, with some of the same results.
Joeri 6y ago
Many of the accidental decisions of the 70’s and 80’s ended up becoming de facto standards, and then actual standards, and then requirements in purchases, and then table stakes for market entry. Why must bytes have 8 bits, why do we count time as seconds since 1970, why must text be encoded as unicode code points, etc... There is now such strong external pressure on projects to conform to these table stakes that I think the only thing that will overturn these decisions at this point is the singularity.
magicalhippo 6y ago
Lets say you did. You launch it, people are excited, they start using it. But stuff will evolve, people will want to innovate. So they make new hardware, and they make new libraries because lets face it, chances you get everything right the first time is slim to none. And after a few years you're back to having backwards compatibility issues, and some years after that you're starting to feel the mismatch between hardware and software again etc. As for cheaper, consider this. You can get an ATX motherboard with lots of functionality most people won't use, for like $50 or so. In low quantities just the PCB alone, just empty tracks with no components would be much more than that.
slx26 6y ago
I think an interesting compromise would be to have a collaborative project for the design of such a system. Even if it was never implemented, it could highlight many of the problems current architectures have, and provide some ideas going forwards. And if it became very obvious that the new system is so much better, then we could always try to do an actual implementation. Nowadays in comparison everything leaks so much. Another partial approach would be this: imagine you had a completely new console which had a single programming language, all tools needed to develop integrated, dumb simple, and you could draw on the screen easily, manipulate the audio buffers easily, and had trivial access to button inputs. Even without networking, I bet we would see amazing things happen. A closed, small, understandable and yet practical and performant system? The dream of every programmer. I know some people already think we have libraries that do this, but they can't provide such a platform.
mikelevins 6y ago
So I've actually been paid to work on two projects of this kind: Apple's Newton, and Peter Neumann's Clean Slate project (https://www.nytimes.com/2012/10/30/science/rethinking-the-computer-at-80.html https://www.nytimes.com/2012/10/30/science/rethinking-the-co...). In both cases, for different reasons, and in different contexts, the projects set out to reinvent computing from the ground up. The Newton failed in ways that make it a punchline, but the punchline obscures useful things about how it succeeded and how it failed. Clean Slate seems to have morphed into another research program called CHERI (http://www.csl.sri.com/users/neumann/20191213-ctsrd-ftr-final.pdf http://www.csl.sri.com/users/neumann/20191213-ctsrd-ftr-fina...), but it hasn't taken the world by storm, either. Both projects embodied some fairly radical rethinking of how computing should work. Both were fun and inspiring to work on--among the best things I've done in my career. Neither has conquered the world, though some of Newton's novel features have become commonplace, at least in some form. On the one hand, the fate of these projects is most likely the fate of any similar attempt to reinvent computing. The scale of the existing computing ecosystem is planetary. No small group of visionaries can exert the force needed to move the planet, and no group big enough to move the planet can be cohesive enough to embody a coherent vision. On the other hand, I think it's worthwhile to invest in pipe dreams like these. They sometimes invent compelling new things that go on to make positive contributions. They're also instructive. They teach you what is possible when you ignore inconvenient realities. Smart people given a green field and an inspiring vision can invent some truly wonderful things. They also teach you what is not possible. You aren't going to simply wipe away the existing infrastructure and replace it wholesale. Its gravitational pull is inexorable. At some stage you have to come to terms with what has gone before, unless you're content for your effort to become a footnote. Both things are useful things to learn: yes, you can do something novel and inspiring, and that's well worth doing for its own sake. No, you can't, simply through the charisma of a beautiful idea, undo all that has gone before. Nor, if you could, would that necessarily be a good thing. Finding workable compromises with your surroundings is, after all, a constructive endeavor.
- buran77 6y ago
  The problem with starting from scratch and doing it completely differently is that inevitably you'll end up researching only to get to the same point (for all intents and purposes). Some things just converge towards the same solution given the set of conditions. So between spending all that (partially duplicated) effort to rethink everything and the fact that the end result would have little buy in due to the high threshold of effort involved with adopting and using it, this is rarely a tempting proposition for many. Either to develop, or to use.
simonh 6y ago
At the heart of this idea is that we can avoid the compromises imposed by the systems we use. But all system designs involve making compromises. The best you can do is design a new system that makes different compromises, but it's not clear that necessarily makes it better. The existing systems we have were often not the first of their kind, often there were many competing systems that could have won. These ones we have did win for reasons, many of them really solid ones and often because they made the right compromises. Take JPEG for example, Six different companies helped develop it, there are six different patented ideas in the arithmetic encoding system and a slew of papers were influential in it's development over decades. You're really going to come up with better ways to do all of that over again from scratch? OK maybe you come up with a better file system or kernel architecture, but why does that mean you need a new graphics compression format? You're also piling up all the risks together. Coupling together a huge swath of unrelated concerns arbitrarily means if some of them sink they'll bring all the other stuff down with them.
vaccinator 6y ago
re-inventing the wheel brings along with it another set of problems
amelius 6y ago
I often wonder why, in open source, we don't see the logical divide between architects and developers. Why aren't there architects writing down the requirements of operating systems and defining their architecture, and, separated, developers working from these designs? I also wonder why we see so few new ideas in the OS space. From a user's viewpoint, it seems little has changed since the first Unixes were born.
- Kototama 6y ago
  Because software development is closer to developing a prototype object than a production object. There is so much knowledge to gain from implementing the problem that need to reflow back to the conception for a (hopefully possible) second implementation. Also who was saying that architecture is the wrong metaphor for software architects and the good metaphor is city planners? They put some frames and infrastructures but mostly enable the creation.
- hencoappel 6y ago
  Google has been experimenting with Fuscia and I believe doing some interesting stuff, but I'm by no means an expert.
vbezhenar 6y ago
IMO there won't be any ground-breaking results. Computing is pretty efficient as it is. May be you'll run modern workloads using 256 MB RAM and 10GB HDD. So what? RAM and disks are cheap. Where it matters, we use hardware acceleration and very thin layers over that acceleration. For example games. JPEG have good compression and few hundreds of kilobytes to implement its support (and those kilobytes are battle-tested and probably have little to no exploitable bugs). I just don't understand why would you want to drop it. Modern web adopts new image formats, it's not like you must use JPEG today.
Stierlitz 6y ago
@systemvoltage: “I often wonder what if we decide to build a completely new, well thought out computing system (including network) that is totally separate from any legacy system and 100% backwards incompatible.” Good idea, and then run legacy apps under an emulator. Each instance created and destroyed after use. That way there will be virtually no chance of malware propagating through the system.
nsm 6y ago
There is this entertaining series called Houyhnhnm Computing that imagines computing invented by a different species. https://ngnghm.github.io/index.html https://ngnghm.github.io/index.html

jamesmishra 6y ago

I think it is possible to build a better system from scratch, and there are likely space aliens that have such better systems. But ultimately a computing system should do what its users want, and the market has favored backwards-compatibility for a long time. If anything, it's a little alarming how old software and hardware can quickly become useless or obsolete. For example, any web browser that doesn't support any encryption better than, say, SSLv3, is cut off from a significant fraction of the Internet.

squibbles 6y ago
> I think it is possible to build a better system from scratch, and there are likely space aliens that have such better systems. I respectfully disagree. Alien systems can be thwarted by inverting some bits. [0] I would not have believed this had I not seen it for myself. [1] After all, seeing is believing. True, the alien code seems very insecure. Perhaps aliens write their code based on the historical documents. [2] [0] https://www.reddit.com/r/movies/comments/2oi7cz/i_am_dean_devlin_writer_producer_of_independence/cmo0r1y/?utm_source=share&utm_medium=web2x&context=3 https://www.reddit.com/r/movies/comments/2oi7cz/i_am_dean_de... [1] https://www.youtube.com/watch?v=cgqswujkxPE https://www.youtube.com/watch?v=cgqswujkxPE [2] https://www.youtube.com/watch?v=lLZQz26-kms https://www.youtube.com/watch?v=lLZQz26-kms

throwaway9870 6y ago

I think you underestimate how many smart people proceeded you. Yes, in some cases there are new technologies that transform how we build things and using those from the start might put you in a better place. But in-between those inventions is a lot of hard work by really smart people that would be incredibly hard to match let alone exceed. To build a system, you would have to start at the bottom layers and then build up. To think you could design any of those layers (protocols, languages, tools, etc.) without making mistakes is naive and I think you would end up with a system that is layered just like current systems, but each layer being much less thought out. Yes, that is even with complete hindsight. I have watched many experienced teams fail on gen 2 of systems. Don't get me wrong, there are many things I dislike in computers, starting with autotools :) But things like autotools are a lot easier to replace in Hacker News comments than in real life.

systemvoltage 6y ago
I had no intention to belittle the complexity, monumental effort and almost impossible to imagine amount of man hours that would go into this - it was purely a thought experiment. Do you really think that it would be less thought out if we have lots of resources and smart people? I feel optimistic that we can bake in the lessons from the past and improve on existing layers/stack/protocols.
- oh_sigh 6y ago
  I think the problem is that, even if you end up getting a proof of concept that attracts others, and they end up being 100x as productive as people using the Old Way, you will still be outpaced because the Old Way has 100,000x as many people working on it as there are on the New Way.
  - divbzero 6y ago
    The odds are stacked against any specific New Way from taking hold. But we can also invert our perspective and ask: Will there a dramatic shift to a New Way in the next half century? Since 1970 there have been at least 3 major transitions in how people work with technology, it’s easy to imagine more will come before 2070.

stunami 6y ago

Somewhat reminded me of the constructive look at TempleOS http://www.codersnotes.com/notes/a-constructive-look-at-templeos/ http://www.codersnotes.com/notes/a-constructive-look-at-temp...

jcrawfordor 6y ago

One of the overriding themes which I attempt to convey in my teaching, blog, etc is that the modern technology landscape is something that we arrived at largely by chance. Nearly every decision made in the development of computing had a wide variety of alternatives at their time, but once things become ossified as a norm it is difficult to imagine another way. For example, I have recently mentioned a few times how it is hard now to see TCP/IP as not being the "obvious design," but in the early '90s there was healthy competition of network protocols offering various feature sets. I think it is extremely important from a teaching and engineering perspective to view the modern computing environment not as some elegant design (which it most certainly is not) but instead as an concretion of "best option at the time" solutions to real problems. It is a bit intoxicating to think about how things could be if computing really were approached as a single, monolothic, elegant design. But essentially every attempt to do so has failed. Much like how we think of computing as layers of abstraction, the computing industry is functionally constructed of layers of abstraction, and so it is difficult to really change anything at lower levels.

shakezula 6y ago
On one hand, it’s fun to imagine what a truly full-stack rewrite of the internet would allow us to do, and what it would look like. On the other hand, the current internet feels like a good approximation of the best trade offs for performance vs. usability in most areas, and I doubt a full rewrite would fundamentally change much about how we use the internet.
- notduncansmith 6y ago
  > I doubt a full rewrite would fundamentally change much about how we use the internet. I imagine that it would in fact have to support people using the internet exactly the way they do now, if it hoped to get traction.
- egypturnash 6y ago
  An Internet with a payment system built in from the ground up might be one that did not end up throwing its hands in the air and saying “I guess everything will be funded by ads, I dunno”. An Internet where “keep engagement high so we can serve shitloads of ads” was not the default path to financial success would be an Internet without social media that happily pushes people into ever-more controversial positions, because that’s what keeps them around longer and lets you serve more ads. Decide for yourself what social changes that would create.

yjftsjthsd-h 6y ago

We could, sure, but nobody would use it, and at some level they'd be right, since the point of computers is to do stuff for us, and a computer that (intentionally!) doesn't work with anything else is an island, which is to say, not very useful.

KMag 6y ago

It's possible to remove layers of cruft without throwing everything out. QNX and BeOS both implemented mostly POSIX compliance on top of radically different foundations. We've gotten rid of the ISA bus and that old pre-PS/2 mouse port, and mostly gotten rid of PS/2. In terms of display technologies, we have DisplayPDF from Apple and Wayland on the Linux side. You take huge risks by just throwing out everything at once. You want a modular system anyway, so run your revolution one module at a time. People don't notice your little revolutions, but that's kind of the point.

matheusmoreira 6y ago
> mostly gotten rid of PS/2 Isn't that port required if users want keyboards with true N-key rollover?
- baybal2 6y ago
  No, USB HID API can have all 104 key rollover if you want. The urban myth has something truthful behind itself, as at some point Windows refused to work past some number of key rollovers, and then keyboard makers started to limit it to accommodate Windows.
  - matheusmoreira 6y ago
    > then keyboard makers started to limit it to accommodate Windows Wow, I really had no idea. That's really frustrating... Any way to fix this? Maybe we can somehow hack the keyboards. They have an USB controller with firmware on it, right?
    - baybal2 6y ago
      Have the keyboard maker order a controller chip that does HID in a way compliant to spec
    - analognoise 6y ago
      USB capable microcontrollers are a buck or two in unit quantity; most of them with a USB drivers fron the manufacturer have a HID device as a standard example. It's like "Hello, World!" but for USB. Hack? Nah. Engineer? Easily. But the other commenter who noted that it was a BIOS limitation was right - it's not a Windows thing.

dpc_pw 6y ago

> I often wonder what if we decide to build a completely new, well thought out computing system Check out Urbit.

throwawaywrench 6y ago

>I feel like we would have a hindsight to design it better, faster, and cheaper. Maybe I'm just pessimistic but I suspect the opposite would occur. It would be buggier, slower and more expensive once you started trying to operate at any kind of scale. The reason being that hidden within all the accumulated layers of cruft are a million fixes for specific security, reliability and performance issues discovered through real world usage.

ithkuil 6y ago
I think that's the interesting word: "hidden" That's a pity. Wouldn't it be nice if we somehow could separate the cruft from the craft? What if some computational deity sifted through our collective codebase and distilled the essence, the minimal, purest kernel? Such minimangel's task would be daunting and probably useless, since the moment we'd throw some new use cases and requirements, NIH and patents, POCs and MVPs and we're back to the simmering mess in no time.
- croo 6y ago
  Those hidden snippets are not the cruft. Every ugly hack we would find is likely a required use case and the whole mess is the craft. Maybe we should stop calling it a mess and accept that non-trivial software with on-demand requirements always take this form... So lets call it detail instead. You cannot separate detail from the craft.
  - ithkuil 6y ago
    I meant that the hidden things were the "craft" hidden among tons of cruft
    - tremon 6y ago
      I think you'll rather find that there's tons of craft, and relatively little cruft.
      - 6y ago

alexmingoia 6y ago

That’s what Urbit is: https://www.urbit.org https://www.urbit.org

nineteen999 6y ago
> totally separate from any legacy system It most certainly is not. It's a macOS and Linux userspace application, no more no less[1]. How on earth do you consider that totally separate from any legacy system? [1] https://urbit.org/using/install/ https://urbit.org/using/install/
- alexmingoia 6y ago
  Urbit runs as a VM with a translation layer for things like networking, file system, etc. Using a host OS is expedient to build a working OS that the user actually wants to use, since hardware support is an uninteresting implementation detail. Eventually, one could write a native hardware kernel in Nock/Hoon, and do networking without TCP/IP/DNS between Urbit computers.
  - nineteen999 6y ago
    > that the user actually wants to use Only a handful of people would rather use something like Urbit than established systems. > Eventually, one could write a native hardware kernel When it does, you can call it an OS. For now, it's a user space application.
    - dfischer 6y ago
      Definitely agree with you. The practical implementation that it is, vs what it advertises itself as visionary-wise.
      - nineteen999 6y ago
        > what it advertises itself as visionary-wise All puff. The amateurs over at /r/osdev are much closer to running directly on hypervisors or bare metal than Urbit will ever be.

lsllc 6y ago

Its been tried, for example Taos: https://sites.google.com/site/dicknewsite/home/computing/byte-articles/the-taos-operating-system-1991 https://sites.google.com/site/dicknewsite/home/computing/byt... Very cool and groundbreaking technology! NOTE: Discussed previously on HN: https://news.ycombinator.com/item?id=9802379 https://news.ycombinator.com/item?id=9802379 https://news.ycombinator.com/item?id=9806607 https://news.ycombinator.com/item?id=9806607 https://news.ycombinator.com/item?id=15470338 https://news.ycombinator.com/item?id=15470338 https://news.ycombinator.com/item?id=15527936 https://news.ycombinator.com/item?id=15527936

AnimalMuppet 6y ago

It would be 100% user free, as well. When we started moving to cell phones, you could still call a landline phone from one. When we created web browsers, they could hit gopher or ftp URLs. When Stroustrup created C++, he let C code freely interoperate, which let people migrate at their own pace, or even flip back and forth. When Windows was created, it let DOS programs run. When IBM introduces a new mainframe, they always have a way to run programs from the previous generation (back to the 1960s, at a minimum). In all of these cases, interoperability with the old system was essential for the growth of users for the new system. How many computer users are there now? Three billion? Break backward compatibility and interoperability, and you lock half the planet out - the half that uses computers. And the other half... if they decide to use a computer, are they going to use what everyone else uses, or are they going to decide that your completely incompatible new thing is the right answer? Even if (in some platonic sense) your new thing is the right answer, how are they going to know that? So, like I said... zero users. 100% user free.

xkcd-sucks 6y ago

Aha, I have deduced you have a lucrative-enough career in B2B/"enterprise" software

browserface 6y ago
:) haha. Underrated comment. ;p ;)x xx

visarga 6y ago

> We get stuck in local maxima at each layer and the critical energy required to escape out of it would be too large. ML solution - many restarts, add momentum and in the end maybe all local maxima are equally good after all.

deepsun 6y ago

One fun exercise I had is to disable IPv4 in my network settings and go with IPv6 only, see what's left working.

matheusmoreira 6y ago

Hardware is much more complex than software and also requires incredible amounts of money in order to fabricate. Not sure if that will ever change. We'll probably remain stuck with whatever is available on the market. Starting from scratch is actually feasible on the software side though. The OSDev wiki has many examples. The most recent one I know is Serenity. Still a monumental task. It's probably a lot of fun if you have the time and motivation to stick with it. You can actually throw POSIX in the trash and create your own abstractions. Hopefully people aren't reinventing signals... I think Linux is the best option for those who want to create something from scratch. The Linux kernel is unique in that it is completely separate from user space. There is a stable, documented kernel interface in the form of system calls: https://man7.org/linux/man-pages/man2/syscall.2.html https://man7.org/linux/man-pages/man2/syscall.2.html https://man7.org/linux/man-pages/man2/syscalls.2.html https://man7.org/linux/man-pages/man2/syscalls.2.html https://www.kernel.org/doc/Documentation/ABI/stable/syscalls https://www.kernel.org/doc/Documentation/ABI/stable/syscalls https://www.kernel.org/doc/Documentation/ABI/stable/vdso https://www.kernel.org/doc/Documentation/ABI/stable/vdso So people can actually do things like trash all of GNU and come up with something entirely new. It's possible to rewrite the entire Linux user space in Rust or Lisp or whatever. It doesn't have to be POSIX compliant. The kernel even has this amazing header which lets people build freestanding Linux programs: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/include/nolibc/nolibc.h https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...

eindiran 6y ago
That header file is amazing. Thank you for the pointer to that. For anyone wondering, it lets you build programs without a libc: > This file is designed to be used as a libc alternative for minimal programs with very limited requirements. It consists of a small number of syscall and type definitions, and the minimal startup code needed to call main(). All syscalls are declared as static functions so that they can be optimized away by the compiler when not used.
- matheusmoreira 6y ago
  > That header file is amazing. Thank you for the pointer to that. Yeah, it's an awesome header! It comes straight from the kernel developers too. They use it for their own tools, apparently. It might be missing some of the newer or niche system calls but it has generic mechanisms which can be used to access everything. I found it during my research for my own Linux system calls library project: https://github.com/matheusmoreira/liblinux https://github.com/matheusmoreira/liblinux There's some additional details implemented by my project such as process start up functions which handle argument, environment and auxiliary vectors. I don't see the need to maintain it anymore though since nolibc.h is so amazing and practical. I even asked Greg Kroah-Hartman about nolibc.h on Reddit: https://old.reddit.com/r/linux/comments/fx5e4v/im_greg_kroahhartman_linux_kernel_developer_ama/fn3rpht/ https://old.reddit.com/r/linux/comments/fx5e4v/im_greg_kroah...

gabcoh 6y ago

I feel like IPFS is taking an interesting approach to this. They aren't trying to rebuild modern computing from scratch, however they are designing a tool which, as the name suggests, has the ability to scale basically unbounded. I'm talking in particular about their focus on defining specs that are themselves parametrized by other specs. At the base of this is multiformats which are a suite of specs for things like network addresses which allow you to wrap any current network protocol like tcp over ip over ethernet or dns over https over quic over udp over whatever. By using these paremetrizable protocols they are lowering the marginal cost you mention and making it much easier to migrate to improved technology in the future. It's been a while since I seriously looked into IPFS, but this is what I remember about it.

Razengan 6y ago

> I often wonder what if we decide to build a completely new, well thought out computing system (including network) that is totally separate from any legacy system and 100% backwards incompatible. Not any bit less, 100% backwards compatible and everything would be fresh. There would be no jpeg support (it would have its own image format), it would not have standard TCP/IP stack - a new protocol and network infrastructure, new display format, new IO ports, etc. The iPad is probably the biggest such attempt to do most of those things. :)

jacobwilliamroy 6y ago

Okay, but you should still run ZFS because I really don't think anyone is going to make a better file system.

aethertron 6y ago
If you don't have files, you don't need a file system http://web.archive.org/web/20081120092935/http://www.memetech.com/nofiles.html http://web.archive.org/web/20081120092935/http://www.memetec...
- jacobwilliamroy 6y ago
  Yes. ZFS has a couple layers of caching in RAM and it actually reads from those caches first, before trying to read from disk. And then it caches whatever it has to read from disk, in RAM. That's why you need lots of ECC RAM to run ZFS. Because errors which propogate from RAM to disk cannot be corrected and such will destroy the whole file system.
zokula 6y ago
Ze cult of ZFS strikes again. Comment is funny in assuming ZFS is objectively better than NTFS or APFS or FAT16.
- jacobwilliamroy 6y ago
  I actually use openzfs. I like it. Why are you giving me shit for that?

tomc1985 6y ago

You'll have a lot more success, I think, if you curate the things you keep and the things you throw out. All of the successful (relatively speaking) 'whole cloth' rethinks did this.

crististm 6y ago

You are on to something but I think this would mean throwing the baby with the bathwater. "From scratch" is a loose definition. And there will necessary be reinventing the wheel. Making the new wheel incompatible on purpose with the old ones is wasteful. For one thing, you are correct. You don't build a car on a carriage framework. But the principles are the same and there are some technologies that will be reused: bushings for example. What should we do about them?

janee 6y ago

Isn't TCP a good example of these two strategies, e.g. where imperfect natural evolution won over it's competitors which I would say is an example of what your describing https://en.m.wikipedia.org/wiki/Protocol_Wars https://en.m.wikipedia.org/wiki/Protocol_Wars I think in general doing a grass roots, ground up, all encompassing design is inferior to a more battle tested natural evolution based strategy where you build on top of each other. Might be wrong, but that's my gut feel. I'm sure there some good examples of the two strategies and how they played out?

ROARosen 6y ago

Interesting, I was thinking the exact same thing but in regard to real-life things. Like imagine cars / roads were invented today do you thing they would have all have this same ubiquitous elongated design, with the grill in front and two headlights in the side, or maybe cars would be completely round maybe they would be made generally with standup room, one big head light across the bottom etc. While some of these design decisions are practical, I believe a lot would be different if we would reimagine it from the bottom up. The reason - for instance - cars cannot be wider is because the wouldn't fit on today's roads roads. Existing safety regulations also prevent certain design possibilities.

systemvoltage 6y ago
Holyshit, I have thought about the exact same thing. Like totally questioning everything about cars and why do they look the way they look. Cheers, made my day. Have you also though about how we see 3 holes on a face (2 eyes and 1 mouth, ignoring the downward facing holes of the nostrils) as normal. Just add one more and it’s horror. We take these things granted and then slight changes to it is completely terrifying. I also wonder - if we rewind evolutionary time and restart with slightly different initial conditions, not sure what species including humans would even exist in the same form. It would likely be like that game Spore.

gigatexal 6y ago

What’s wrong with JPEG?

simonebrunozzi 6y ago

Absolutely agree with you. Same, with larger impact, would be if we apply this thinking to how we build cities. Think about it for a moment. How much "crap" and "stupidity" is there just because of legacy.

browserface 6y ago

Soviet computers are an interesting historical riff on this. But I think this would be impossible to cut the cord completely, because a computer is basically a communication system (internally and networked) and by Conway's law groups zero in on comms systems that reflect their own dynamics, so as a civilizations we've built devices and networks that have zeroed in on our own internal dynamics. Therefore as a member of this civilization, you could not create something different. You'd need to go hang with some folks completely different until you absorbed that, then make it to reflect their dynamics. Uncontacted Amazon tribes? Dolphins? Elephants? Plants?

acrossthepond10 6y ago

I've always wondered if its possible to write software that will run on computers with arbitrary number of memory heirarchy eg. x amount of l1/l2 cache -> main memory (RAM) -> SSD -> HDD -> Archive storage where you can plug in a new type of memory as it becomes available (eg. 3dxpoint) and it will adapt to run optimally on that hardware without rewrites.

acrossthepond10 6y ago
Or designing software for a theoretical system like an ideal reconfigurable computer with reconfiguring latency that is as fast as processing latency and then have hardware engineers design towards that as opposed to having software engineers target hardware like we do know. Feels like that would make for an interesting experiment.
Reelin 6y ago
> I've always wondered if its possible to write software that will run on computers with arbitrary number of memory heirarchy I think that's already possible today - the Legion programming system claims to do exactly that (among other things). I have absolutely no experience with it though so I might be completely wrong. (https://legion.stanford.edu/ https://legion.stanford.edu/) > By making the Legion programming system aware of the structure of program data, it can automate many of the tedious tasks programmers currently face, including correctly extracting task- and data-level parallelism and moving data around complex memory hierarchies.

despair3435 6y ago

With regard to networking, please have a look at the Ouroboros project [0][1]. [0] https://ouroboros.rocks https://ouroboros.rocks [1] https://arxiv.org/abs/2001.09707 https://arxiv.org/abs/2001.09707

Cthulhu_ 6y ago

What came close was the smartphone (and a bit later, tablets); one big one is that they did not support Flash, which (at least in my neck of the woods) was an important factor in the shift towards webapps / SPA's. At the same time, browsers / JS became faster, NodeJS became a thing, and because of both apps, webapps and REST interfaces, the inefficiencies and insecurity of HTTP came to the forefront, leading to (more, easier, accessibe) HTTPS and HTTP/2, and now a push to UDP-based protocols like quic / HTTP/3. Anyway, smartphones / tablets have made us rethink hardware, its developments have led to more compact laptops as well, and the push to more compact laptops have caused their manufacturers (mostly thinking of Apple throughout this comment btw) to eschew legacy systems - disk drives, USB ports, even 3.5mm jacks all went out the window. But that is exactly what you're proposing here; what did you think when Apple got rid of a lot of that legacy? Personally I have mixed feelings; I do like USB-C. I didn't like Lightning because it was nonstandard, but it was an improvement over their broad connectors and USB-C wasn't a thing yet. I can see why they're pushing to wireless connections for audio and input. Anyway final point, despite the cut in legacy connectors and the like, Apple's hardware hasn't gotten cheaper.

gmueckl 6y ago
But smartphones - both iOS and Android - take their core operating systems essentially from the 70s. The kernels themselves are few decades newer than that. The concepts and interfaces implemented by them, however, are not. Similar things can be said about Windows Phone and NT. Arguably, using an existing OS as a base was kind of the only way to get these products done in a reasonable timeframe. They run complex and powerful software and writing that stack alone is hugely demanding. Rewriting the OS would easily have lost the market to the competition because of cost and time overruns.

tremon 6y ago

Why wonder? That's basically what IPv6 did for networking, or what Intel tried with Itanium, but with even larger gap to existing usage. The cost of switching would be so astronomically high, your product would fail before it even started. "Throw out the old" happens in local pieces of the stack all the time. AMD removed a lot of cruft from the Intel ISA when they designed AMD64, Wayland removes a lot of complexity from the display protocol stack, and every I/O port before USB was designed to be incompatible with existing ports. I feel that what you are describing will look like the SBC situation in the ARM world: the processor instruction set may be identical, but every vendor uses a slightly different incompatible way of connecting and designing the surrounding chipset, which results in a myriad of almost-identical drivers and subtle bugs that are impossible to fix, because the cost/benefit ratio would be prohibitively high. Standardization isn't just for backwards compatibility, nor does it mandate it. But it does allow for reuse of existing components in an interoperable way. Imagine if every application came as a unikernel microservice instead of an OS framework library: there would be no UI consistency, copy-pasting between different applications would be hard, and you would have no OS-level update facility -- every application would have its own set of vulnerabilities.

pjmlp 6y ago
Intel only failed with Itanium, because AMD exists and had access to Intel's IP. Without AMD64 Itanium would have prevailed.

jiggawatts 6y ago

There are some contexts where this makes sense. As far as I know, currently there are no "good" real time operating systems available. There are some ancient, crufty ones... and some modern soft-real-time operating systems, but that's it. This would be a fun exercise for someone wanting to write something uniquely useful in Rust, for example. It might make sense to write a pure web browser engine designed for embedded contexts. Think infotainment systems where the web content being served is fully controlled. This would be an opportunity to write something like a HTML5-only engine with UTF8 only, TLS 1.3 only, HTTP 3 only, etc... Relational database engines need a massive rethink. All of the current popular ones are 20+ years old, and based on 50+ year old architectures. Why do schema operations take time? Why does index rebuilds lock tables? Why are there column size limits? Why is SQL injection still a thing? Why, why, why... because legacy, that's why. Because hard-coding, that's why. Because 1980s C-style programming with fixed-size buffers, that's why. All of this could be just washed away with a rewrite using Rust, Modern C++, and some fresh ideas.

dfischer 6y ago

I 100% agree with this and was just talking about this in another thread on one of my earlier comments. I love what Plan 9 and Oberon was working on. I feel many are feeling similar at this point given the frequency I'm running into this and also I myself wanting to solve for it. Awesome! I think the networking layer should be built off of http://PJON-technologies.com http://PJON-technologies.com - it's perfect for standardizing the next layer of networking and without depending on any ISP or centralized authority that also has support for multiple "packet/connection" protocols (LoraWAN, Light, Ethernet, and more). I am going through a history lesson right now on studying older OS's and why they didn't take off while also getting my feet wet with Forth and lower-level hardware experiments. I still have no idea how video truly works, that's something that's been bugging me from a first principles perspective and I want to understand that more intuitively. Something I keep asking myself is: "How do I poke at the video card and make it display stuff without having to use a OS level API that wraps openGL or similar?" - I know some will be shocked at that question and some may go "woah, actually I don't know that" - we're in an interesting generational leap with where knowledge has been in the zeitgeist of computation and networks. The OG's are in a particular precious bucket where they've done their part and not that active or have been yelling the current stack is wrong but it's hard to find their voices. Then you have the hackers and hardware folk who learned a lot pre-internet that may or may not be active on the typical net's hanging out only in Plan 9 esoteric rooms saying how good the internet was before marketing took over, and then the dot com boom generation which either knows the earlier internet or thinks the internet is only social media and search engines. I want to make the internet feel adventurous and fun again like the 90's. I want it to be focused on individuals and not organizations. It was exciting because you connected to people, today you don't do that anymore - you interface with some centralized entity that exposes you to products. I have more to say here that is a lot more nuanced but you get my gist. Urbit is doing something similar to what you're saying, however I personally don't like their networking model and a few other decisions that just feel contrived to me. Plus I want to get my feet wet and understand the layers myself from a first principles perspective so I can add to it as I am imagining it. Sounds like a new dawn of innovation is bubbling. My previous comment on a similar tone/opportunity from yesterday: https://news.ycombinator.com/item?id=24467534 https://news.ycombinator.com/item?id=24467534

asplake 6y ago

Gall’s law though. For that complex system to work, it would have to be based on something simpler that worked. Yes you might well come up with something radically different, but still you’d have gone through some of the same evolutionary process, with some of the same results.

Joeri 6y ago

Many of the accidental decisions of the 70’s and 80’s ended up becoming de facto standards, and then actual standards, and then requirements in purchases, and then table stakes for market entry. Why must bytes have 8 bits, why do we count time as seconds since 1970, why must text be encoded as unicode code points, etc... There is now such strong external pressure on projects to conform to these table stakes that I think the only thing that will overturn these decisions at this point is the singularity.

magicalhippo 6y ago

Lets say you did. You launch it, people are excited, they start using it. But stuff will evolve, people will want to innovate. So they make new hardware, and they make new libraries because lets face it, chances you get everything right the first time is slim to none. And after a few years you're back to having backwards compatibility issues, and some years after that you're starting to feel the mismatch between hardware and software again etc. As for cheaper, consider this. You can get an ATX motherboard with lots of functionality most people won't use, for like $50 or so. In low quantities just the PCB alone, just empty tracks with no components would be much more than that.

slx26 6y ago

I think an interesting compromise would be to have a collaborative project for the design of such a system. Even if it was never implemented, it could highlight many of the problems current architectures have, and provide some ideas going forwards. And if it became very obvious that the new system is so much better, then we could always try to do an actual implementation. Nowadays in comparison everything leaks so much. Another partial approach would be this: imagine you had a completely new console which had a single programming language, all tools needed to develop integrated, dumb simple, and you could draw on the screen easily, manipulate the audio buffers easily, and had trivial access to button inputs. Even without networking, I bet we would see amazing things happen. A closed, small, understandable and yet practical and performant system? The dream of every programmer. I know some people already think we have libraries that do this, but they can't provide such a platform.

mikelevins 6y ago

So I've actually been paid to work on two projects of this kind: Apple's Newton, and Peter Neumann's Clean Slate project (https://www.nytimes.com/2012/10/30/science/rethinking-the-computer-at-80.html https://www.nytimes.com/2012/10/30/science/rethinking-the-co...). In both cases, for different reasons, and in different contexts, the projects set out to reinvent computing from the ground up. The Newton failed in ways that make it a punchline, but the punchline obscures useful things about how it succeeded and how it failed. Clean Slate seems to have morphed into another research program called CHERI (http://www.csl.sri.com/users/neumann/20191213-ctsrd-ftr-final.pdf http://www.csl.sri.com/users/neumann/20191213-ctsrd-ftr-fina...), but it hasn't taken the world by storm, either. Both projects embodied some fairly radical rethinking of how computing should work. Both were fun and inspiring to work on--among the best things I've done in my career. Neither has conquered the world, though some of Newton's novel features have become commonplace, at least in some form. On the one hand, the fate of these projects is most likely the fate of any similar attempt to reinvent computing. The scale of the existing computing ecosystem is planetary. No small group of visionaries can exert the force needed to move the planet, and no group big enough to move the planet can be cohesive enough to embody a coherent vision. On the other hand, I think it's worthwhile to invest in pipe dreams like these. They sometimes invent compelling new things that go on to make positive contributions. They're also instructive. They teach you what is possible when you ignore inconvenient realities. Smart people given a green field and an inspiring vision can invent some truly wonderful things. They also teach you what is not possible. You aren't going to simply wipe away the existing infrastructure and replace it wholesale. Its gravitational pull is inexorable. At some stage you have to come to terms with what has gone before, unless you're content for your effort to become a footnote. Both things are useful things to learn: yes, you can do something novel and inspiring, and that's well worth doing for its own sake. No, you can't, simply through the charisma of a beautiful idea, undo all that has gone before. Nor, if you could, would that necessarily be a good thing. Finding workable compromises with your surroundings is, after all, a constructive endeavor.

buran77 6y ago
The problem with starting from scratch and doing it completely differently is that inevitably you'll end up researching only to get to the same point (for all intents and purposes). Some things just converge towards the same solution given the set of conditions. So between spending all that (partially duplicated) effort to rethink everything and the fact that the end result would have little buy in due to the high threshold of effort involved with adopting and using it, this is rarely a tempting proposition for many. Either to develop, or to use.

simonh 6y ago

At the heart of this idea is that we can avoid the compromises imposed by the systems we use. But all system designs involve making compromises. The best you can do is design a new system that makes different compromises, but it's not clear that necessarily makes it better. The existing systems we have were often not the first of their kind, often there were many competing systems that could have won. These ones we have did win for reasons, many of them really solid ones and often because they made the right compromises. Take JPEG for example, Six different companies helped develop it, there are six different patented ideas in the arithmetic encoding system and a slew of papers were influential in it's development over decades. You're really going to come up with better ways to do all of that over again from scratch? OK maybe you come up with a better file system or kernel architecture, but why does that mean you need a new graphics compression format? You're also piling up all the risks together. Coupling together a huge swath of unrelated concerns arbitrarily means if some of them sink they'll bring all the other stuff down with them.

vaccinator 6y ago

re-inventing the wheel brings along with it another set of problems

amelius 6y ago

I often wonder why, in open source, we don't see the logical divide between architects and developers. Why aren't there architects writing down the requirements of operating systems and defining their architecture, and, separated, developers working from these designs? I also wonder why we see so few new ideas in the OS space. From a user's viewpoint, it seems little has changed since the first Unixes were born.

Kototama 6y ago
Because software development is closer to developing a prototype object than a production object. There is so much knowledge to gain from implementing the problem that need to reflow back to the conception for a (hopefully possible) second implementation. Also who was saying that architecture is the wrong metaphor for software architects and the good metaphor is city planners? They put some frames and infrastructures but mostly enable the creation.
hencoappel 6y ago
Google has been experimenting with Fuscia and I believe doing some interesting stuff, but I'm by no means an expert.

vbezhenar 6y ago

IMO there won't be any ground-breaking results. Computing is pretty efficient as it is. May be you'll run modern workloads using 256 MB RAM and 10GB HDD. So what? RAM and disks are cheap. Where it matters, we use hardware acceleration and very thin layers over that acceleration. For example games. JPEG have good compression and few hundreds of kilobytes to implement its support (and those kilobytes are battle-tested and probably have little to no exploitable bugs). I just don't understand why would you want to drop it. Modern web adopts new image formats, it's not like you must use JPEG today.

Stierlitz 6y ago

@systemvoltage: “I often wonder what if we decide to build a completely new, well thought out computing system (including network) that is totally separate from any legacy system and 100% backwards incompatible.” Good idea, and then run legacy apps under an emulator. Each instance created and destroyed after use. That way there will be virtually no chance of malware propagating through the system.

nsm 6y ago

There is this entertaining series called Houyhnhnm Computing that imagines computing invented by a different species. https://ngnghm.github.io/index.html https://ngnghm.github.io/index.html