6 ms·
When a computer's permanent storage is all right there in the processors' memory map, there is no need for disk controllers or filesystems. It's all just RAM.
by Animats 5y ago
When a computer's permanent storage is all right there in the processors' memory map, there is no need for disk controllers or filesystems. It's all just RAM.
And errors are forever.
There were once LISP, Smalltalk, and Forth environments where you saved the entire state of the system, rather than using files. This was not a good thing. Only one person could really work on something. If you messed up the image, undoing the mess was hard. Revision control? What's that?
Progress has been made by learning how to minimize and discard state. What makes the web backend application industry go is that almost all the state is in the database. Database systems are well debugged and reliable today. This allows for mediocre quality in web apps.
Containers are another example of discarding state. Don't sysadmin, just flush and reload. So is the transition from object-oriented to functional programming.
No, making memory persistent is not the answer.
We do need to be thinking about new OS designs, because the hardware world is changing.
- Lots of RAM. Swapping out to disk, even SSD, is so last-cen. The moment that happens performance degrades so badly you may as well restart.
- Lots of CPUs. There are now 128-core CPUs. NVidia just announced a 144-core ARM CPU. The OS and hardware need to be really serious about interprocess communication. It's an afterthought in Unix/Linux. Needs to be at least as good as QNX. Probably better, and with hardware support.
- Different kinds of compute units. GPUs need to be first-class players in the OS. There are going to be more special purpose devices, for machine learning.
- SSD is fast enough that buffering SSD in RAM is mostly unnecessary.
- kragen 5y agoMultiuser systems with orthogonal persistence, like the single-user Smalltalk and image-based Lisps you mention, include EUMEL, L3, KeyKOS, OS/400 (iSeries), and I think Multics. (The Forths I know of were never image-based, so I can't comment on them.) It is definitely not the case that "only one person could really work on something [at a time]" on EUMEL, L3, KeyKOS, and OS/400. Maybe only one person could hack on the TCB at a time, but EUMEL, L3, KeyKOS, and OS/400 had extremely tiny TCBs, for that reason among others. Having a single-level store doesn't save you from minimizing and discarding state. It just (mostly) decouples discarding state from power failures, making it intentional rather than accidental. In today's battery-powered environments, this is probably a better idea than ever. KeyKOS and EUMEL still had named files, containing sequences of bytes, in directories. They just weren't coupled to the disk device drivers. OS/400 has files, too. Smalltalk and Lisp environments were more radical in this sense. Also, Smalltalk has had version control within its images for decades. Stephen White and Pavel Curtis's MOO also had transparent persistence, didn't have files, and supported massively multiuser access; possibly a MOO server has supported more concurrent users than any AS/400, and certainly more than any EUMEL, L3, or KeyKOS system ever did. (LambdaMOO has 48 users online right now according to http://mudstats.com/Browse http://mudstats.com/Browse but had many more at its peak.) But, while MOO was programmable, it never supported high-performance or high-reliability computing as those other systems did. You could think of it as being written in a groupware scripting DSL, like a centralized Lotus Notes. The most complex MOO software I know of was a Gopher client, a "Gopher slate" that multiple people could use simultaneously. It used cooperative multitasking with timeouts that would abort long-running event handlers without rolling back their incomplete effects. I agree, though, that a single-level store on its own doesn't solve the problems OSes are facing. But it might be useful. We have many layers of software on top of everything that accesses persistent data being built on the assumption that anything persistent is necessarily slow enough to cover millions of nanoseconds of processing. That greatly reduces the benefits of the enormous improvement SSDs represent over spinning rust. I'd say that buffering SSD in RAM goes beyond "mostly unnecessary": it needs to be done in hardware (as with NVDIMMs) to not hurt performance in many cases. FRAM is even more extreme on this axis. Another interesting question to me is migration. If I'm editing a Jupyter notebook on my laptop, it'd be nice to move it onto my cellphone and keep editing it when I travel, and it'd be nice to move the computations onto heavy servers when I'm connected to the internet. Lacking this, we seem to be moving to warehouse-scale computing: everybody runs all their notebooks on Google App Engine so they can use TPUs and access them from anywhere. Related to migration is UI remoting. Why can't I use two cellphones together to access the same app, or two cellphones and a mouse as I/O devices to the same music synthesizer (running on my laptop)? Accelerometers and multitouch could be useful for a lot of things. Maybe ROS (the Willow Garage one, not the BBS software) has the right approach here? Also, why is everything so goddamn slow and unreliable? Ivan Sutherland had a VR wireframe cube running on a head-mounted display more than 50 years ago. Why are these supercomputers in our pockets still too slow to run VR with low enough latency that people don't throw up? Why does it take multiple minutes to start up again if its battery runs down and I plug it in? Why does the alarm app on my cellphone refuse to launch when its Flash fills up? Why does my laptop become unresponsive and require rebooting when a web page uses too much RAM? shakes fist at cloud Anyway maybe I'll have a running prototype of a capability-based single-level-store operating system using optimistically-synchronized software transactional memory to guarantee real-time responsivity without an MMU or lots of concurrency bugs, I don't know, sometime next week. Or maybe not.
- zozbot234 5y ago> It just (mostly) decouples discarding state from power failures, making it intentional rather than accidental. Just wondering, but how is an OS supposed to achieve this without fsync()ing all writes to memory and being slow as dog$h!t? Having a "single level" of storage means it becomes harder to tell what might be purely ephemeral, and can thus be disregarded altogether if RAM or disk caches are lost in a failure. You might think you can do this via new "volatile segments" API's but then you're just reintroducing the volatile storage that you were trying to get rid of.
- kragen 5y agoBoth KeyKOS and EUMEL used periodic globally consistent checkpoints supplemented by an explicit fsync() call for use on things that required durability like database transaction journals. So when the system crashed you would lose your last few minutes of work; EUMEL ran on floppies, so the interval could be as long as 15 minutes, but I think even KeyKOS only took a checkpoint every minute or so, perhaps because it would pause the whole system for close to a second on the IBM 370s it ran on. (Many years later, Norm Hardy showed me a SPARC workstation with I think 32 or 64 MiB of RAM; he said it was the biggest machine KeyKOS had ever run on, twice the size of the mainframe that was the previous record holder. Physically it was small enough to hold in one hand.) I assume L3 used the same approach as EUMEL. I don't know enough about OS/400 or Multics to comment intelligently, but at least OS/400 is evidently reasonably performant and reliable, so I'd be surprised if it isn't a similar strategy. There's a latency/bandwidth tradeoff: if you take checkpoints less often, more of the data you would have checkpointed gets overwritten, so your checkpoint bandwidth gets smaller. NetApp's WAFL (not a single-level store, but a filesystem using a similar globally-consistent-snapshot strategy) originally (01996) took a checkpoint ("snapshot") once a second and stored its journals in battery-backed RAM, but with modern SSDs a snapshot every 10 milliseconds or so seems feasible, depending on write bandwidth. Most of these systems did have "volatile segments" actually, but not for the reason you suggest, which is performance; rather, it was for device drivers. You can't swap out your device drivers (in most cases) and they don't have state that would be useful across power failures anyway. So they would be, as EUMEL called it, "resident": exempted from both the snapshot mechanism and from paging.
- zasdffaa 5y ago> SSD is fast enough that buffering SSD in RAM is mostly unnecessary. A very strong claim, which I'd contest depending on what you're caching. If it's application code then probably but if you've lots of data which you're expecting to re-read such as with a database with many gigs, I really disagree with that. If an SSD had say a 1 gig/sec bandwidth for reasing that's about 22 times slower than my arthritic 7-year dual-channel RAM bandwidth[1]. A more modern higher spec server will have more channels (and hopefully more modern, faster ram). Can you elaborate a bit on your thoughts? [1] and the path to ram may not have as many page faults -> kernel switches as reading from an SSD, but I may be wrong and it may not matter
- kragen 5y agohttps://www.club386.com/silicon-motion-pcie-5-0-nvme-ssds-promise-14gb-s-bandwidth/ https://www.club386.com/silicon-motion-pcie-5-0-nvme-ssds-pr... says 14 GB/s. And of course the sum of bandwidths within a set of NVDIMMs can easily be much higher than that.
- a9h74j 5y ago> GPUs need to be first-class players in the OS I've been starting to wonder, taking a crazy-stupid extreme, whether a GPU unit should be in charge of booting. Perhaps you boot into a graphical interpeter (like a boot into BASIC of old) which can then implement a menu (or have "autoexec.bas") to select a heavyweight OS to boot into.
- GauntletWizard 5y agoYour Raspberry Pi already does :) https://thekandyancode.wordpress.com/2013/09/21/how-the-raspberry-pi-boots-up/ https://thekandyancode.wordpress.com/2013/09/21/how-the-rasp... More seriously, there's probably something to this from a motherboard design perspective, there's enough general purpose compute on a modern graphics card to do the initial bootloader stuff. But graphics cards are not standardized as much as CPUs are, so it's unclear if it would work as we've currently designed systems.
- a9h74j 5y agoGreat link for this, thanks. I had suspected some different ordering might already be called for given the need for e.g. RAM training. That [version of?] pi booting is much more filename-oriented than all of the partitioning I have seen documented for a Rockchip [ http://www.loper-os.org/?p=2295 http://www.loper-os.org/?p=2295 ].
- jandrewrogers 5y ago> Lots of RAM. Swapping out to disk, even SSD, is so last-cen. The moment that happens performance degrades so badly you may as well restart. RAM is the new disk. These days we try to make codes live out of CPU cache as much as possible, scheduling the RAM access the way we used to schedule disk access, which is effective -- plus ça change, plus c'est la même chose. > SSD is fast enough that buffering SSD in RAM is mostly unnecessary. This is not even approximately true in practice for many applications. Striping an array of NVMe storage devices is still about an order of magnitude less bandwidth than RAM. Storage I/O is also high latency and highly concurrent; if your storage I/O is zero-copy then it implies locking up considerable amounts of RAM just to schedule the I/O transactions safely, never mind caching. Modern database kernels require clever I/O schedulers and large amounts of RAM to keep throughput from degrading to the throughput of even very fast storage hardware. And this assumes your storage code is aware of the peculiarities of SSD (inconvenient mix of block sizes, no overwrite, garbage collection, etc). That said, you obliquely raise a valid point. Buffering storage in a cache usually doesn't work well once the storage:cache ratios become high enough, and the 1000:1 ratios seen in modern servers are well past that threshold. In these cases, you still need a lot of RAM but you end up redeploying it to other parts of the kernel architecture where it can be more effective than a buffer cache for storage.
- solarengineer 5y ago“ If you messed up the image, undoing the mess was hard. Revision control? What's that?” VisualAge for Smalltalk provided built-in version control.
- imtringued 5y ago>And errors are forever. Have you tried turning it on and off again? Oh, you can't. >- Lots of CPUs. There are now 128-core CPUs. NVidia just announced a 144-core ARM CPU. The OS and hardware need to be really serious about interprocess communication. It's an afterthought in Unix/Linux. Needs to be at least as good as QNX. Probably better, and with hardware support. Ah yes, Minecraft, the biggest traitor of all. A world bigger than earth but even getting to 250 players on a single server requires absurd levels of optimizations.
- rurban 5y ago> Have you tried turning it on and off again? Oh, you can't. Actually you can. That was one of their biggest advantages, to save and restore running images, and debug into running images. But diffing a serialized binary image (what changed?) was hard. you had to diff the commands leading to your image, a bit like in docker.