5 ms·

> Like it being trivial to accidentally introduce potentially very expensive cloning or accidentally using a ton of unneeded memory. You're describing things C

> Like it being trivial to accidentally introduce potentially very expensive cloning or accidentally using a ton of unneeded memory. You're describing things COW is designed to prevent as things COW causes. There is NEVER a very expensive cloning happening at once with COW. That's the whole point of it. Everything is shared until the last moment when it can't be, and even then only small parts are copied. Makes me think you're missing something here. Maybe not familiar with how COW is used in practice. COW is used in your operating systems' memory management, and in your file system, as I already said. Do you assume they're like "oh yes, we want to have randomly triggered very expensive cloning and use a ton of unneeded memory!"? You may call COW a "magic trick" but this magic trick is fundamental to how your computer works. Not only is it in your file system and OS. It's also essential to virtual machines and containerization. It's essential to virtually all backup & recovery systems. Do you understand how integral COW is to the world of computing? This scary magic trick you're so afraid of? You're fine defending a "very useful mess" and rejecting clear improvements with inadequate or outright misleading conclusions. Most people just react superficially like this to change and I'm used to it. But I'll never stop pointing it out when I see it. As for C compatibility without copying the memory model flaws, it's possible, but it requires more thought than Rust put into it.

dathinab 3y ago
> You're describing things COW is designed to prevent as things COW causes. not quite see below > You may call COW a "magic trick" I didn't call it a magic trick, I stated rust decided against using "magic tricks" of internally implicitly not visible doing things below the hood. Which might involve using a Cow or similar. Cow is just the tool the trick could use not the trick by itself. > Maybe not familiar with how COW I'm very familiar with it. But there is a difference between using CoW manually carefully in some places and using it everywhere all the time implicitly as part of the language design. COW prevents you from always copying everything everywhere at the cost of on-the-fly copying things when needed if everything is COW all the time this will mean most of the copies are not expensive because most parts of the thing you need to copy will not need to be copied because they are also cow but there are some gotchas first if you have it implicitly everywhere it is very easily to overlook how many copies you make, or that you make way to many and should change your code and even with "everything COW" the cost still can be very expensive for all kinds of reasons e.g. in some hypothetical code like `a[10] = 4` you might now have accidentally cloned a 4GiB byte buffer, COW doesn't prevent this because that 4GiB byte buffer is a single thing. Now you might come up with ways to internally split the buffer into chunks or only use lists etc. But that isn't really working for many low level system language use-cases (but totally working for many other cases, including many server programming cases). Similar it means if you have nested types they always need an indirection to the sub-types so that you can just copy the GC or RC references (or whatever you use for memory management) when needing to make a copy of the outer type (except where an optimizer can optimize it away) which is fine, except that for a system language you often need C layout compatible types. So you would need to severely special case C-layout types. And as far I can tell you either special case a lot of thing in a way which isolates them away from your language (things like C-layout, fundamental non-clonable types, etc.) or you need to have a implementation of the nested CoW which no longer guarantees that it's efficient at all. Isolating things away which doesn't fit well with your language is totally an option for a general purpose language but can become very unpleasant and/or hard to use for a low level system language. > rejecting clear improvements I'm not rejecting anything, I'm saying that many of the reasons design decisions you called defects are fundamentally the reason why rust has such a huge success. And calling this aspects defects isn't right. It's like saying city bike without a suspension is defect or a poor decisions. They are not. But I still would prefer other approaches (like a suspension you can switch off). Theoretical improvements are nice, but turning them into practical improvements outside of pet projects is hard. Which is why rust mostly went with decisions where there had been a bunch of successful research and prototyping done before. And even in that aspect with rather conservative choices. I mean that is partially where the name "rust" comes from, it mostly used stuff which was so old (in research) it could have been rusted. Because what rust pursed wasn't being a perfect language which overhauls everything. Instead it targeted a solid improvement over C/C++ for some specific set of use-cases which very intentionally does not do anything too unusual. E.g. they directly support the C-API/ABI so that it reasonably easy to include in C/C++ projects and include C/C++ projects into it, so that you can replace some components of larger C++ projects with rust. > As for C compatibility without copying the memory model flaws, it's possible, but it requires more thought than Rust put into it. If you combine it with higher performance requirements and requirements like wide spread platform support etc. it's very very hard. There had been research on this for years mostly for GC languages and for a long time success was rather limited. I think in "recent" years there had been some nice progress on this topic. But then this design decisions where made for rust nearly 9 years ago... I mean the first stable release of rust was 8 years ago. So not including research which didn't yet exist isn't really a poor decision. Furthermore even that success might not be practically appalling for all use-cases. Not for technical reasons. But sadly programming language design isn't just about technical aspects. Anyway I hop you all success with creating such a language or motivating people to create such a language. IMHO most software today doesn't need to be written in any system language at all. This even includes most "system" components. I mean a huge part of systemd could probably be written in WASM and it still would work fine for a lot of use cases as crazy as it sounds. So good luck.
- BulgarianIdiot 3y ago
  You said splitting entities into small chunks doesn't work for low-level. But as you already know (exposition monologue follows :D) memory, all of it, is allocated in small pages. Typically 4kb of size. That's what the low level actually is. On some hardware the page is 2MB which is called, literally "huge pages" (or "large pages"). But that's about it, bigger pages are essentially not used. If your language can't help but see 4GB as one thing, that's a flaw in your language that it even permits that. I'm not talking about how it appears on the surface, but how it's allocated and chunked. In data structures designed for COW a single 4GB chunk would be anomalous occurrence, basically a bug, everything is split up in structures roughly the size of a page max, or so. So we had some truth here at the hardware and OS level, which is "never allocate things in huge chunks" but we lost it at the higher level and then we think this is just how things are and design all our languages around it. An OS may permit you to allocate more memory, but that's with the idea you'll manage this yourself, say as an arena. If you instead just store a monolithic piece of state in it, that's just ignorance, or a special case, like say large buffer for DMA, or the like. Regarding nested types, you can still inline content, you don't NEED to add indirection. The behavior of inlining vs referencing can be determined based on content structure and should be transparent to your code. It's trivial and many runtimes do this (I suppose your next objection will be how come we have a runtime, but ostensibly even C has part of its library acting as a runtime for basic things like mem alloc). > If you combine it with higher performance requirements and requirements like wide spread platform support etc. it's very very hard. If you have the right foundation, writing optimizations on top of it is a lot more trivial than when your foundation is 70 years of ad-hoc cruft. This was demonstrated by Microsoft experimenting with C# written OS (I think called Singularity) that bypassed a lot of OS and even hardware APIs in order to manage most memory operations & safety on its own. And that turned out faster than C code. Unfortunately, not compatible with the status quo. C is a nightmare.
kaba0 3y ago
Your filesystem and OS has absolutely different constraints, they don’t modify their objects to such a degree as most programs modify memory. So honestly, you should provide more evidence why would that work (for a low-level, systems programming language). And don’t get me wrong, FP languages en large can be very competitive with a more imperative approach, especially when multithreading is on the table, but they achieve it through extensive compile time transformations, and on the data structure level by much smarter algorithms than plain COW.

dathinab 3y ago

> You're describing things COW is designed to prevent as things COW causes. not quite see below > You may call COW a "magic trick" I didn't call it a magic trick, I stated rust decided against using "magic tricks" of internally implicitly not visible doing things below the hood. Which might involve using a Cow or similar. Cow is just the tool the trick could use not the trick by itself. > Maybe not familiar with how COW I'm very familiar with it. But there is a difference between using CoW manually carefully in some places and using it everywhere all the time implicitly as part of the language design. COW prevents you from always copying everything everywhere at the cost of on-the-fly copying things when needed if everything is COW all the time this will mean most of the copies are not expensive because most parts of the thing you need to copy will not need to be copied because they are also cow but there are some gotchas first if you have it implicitly everywhere it is very easily to overlook how many copies you make, or that you make way to many and should change your code and even with "everything COW" the cost still can be very expensive for all kinds of reasons e.g. in some hypothetical code like `a[10] = 4` you might now have accidentally cloned a 4GiB byte buffer, COW doesn't prevent this because that 4GiB byte buffer is a single thing. Now you might come up with ways to internally split the buffer into chunks or only use lists etc. But that isn't really working for many low level system language use-cases (but totally working for many other cases, including many server programming cases). Similar it means if you have nested types they always need an indirection to the sub-types so that you can just copy the GC or RC references (or whatever you use for memory management) when needing to make a copy of the outer type (except where an optimizer can optimize it away) which is fine, except that for a system language you often need C layout compatible types. So you would need to severely special case C-layout types. And as far I can tell you either special case a lot of thing in a way which isolates them away from your language (things like C-layout, fundamental non-clonable types, etc.) or you need to have a implementation of the nested CoW which no longer guarantees that it's efficient at all. Isolating things away which doesn't fit well with your language is totally an option for a general purpose language but can become very unpleasant and/or hard to use for a low level system language. > rejecting clear improvements I'm not rejecting anything, I'm saying that many of the reasons design decisions you called defects are fundamentally the reason why rust has such a huge success. And calling this aspects defects isn't right. It's like saying city bike without a suspension is defect or a poor decisions. They are not. But I still would prefer other approaches (like a suspension you can switch off). Theoretical improvements are nice, but turning them into practical improvements outside of pet projects is hard. Which is why rust mostly went with decisions where there had been a bunch of successful research and prototyping done before. And even in that aspect with rather conservative choices. I mean that is partially where the name "rust" comes from, it mostly used stuff which was so old (in research) it could have been rusted. Because what rust pursed wasn't being a perfect language which overhauls everything. Instead it targeted a solid improvement over C/C++ for some specific set of use-cases which very intentionally does not do anything too unusual. E.g. they directly support the C-API/ABI so that it reasonably easy to include in C/C++ projects and include C/C++ projects into it, so that you can replace some components of larger C++ projects with rust. > As for C compatibility without copying the memory model flaws, it's possible, but it requires more thought than Rust put into it. If you combine it with higher performance requirements and requirements like wide spread platform support etc. it's very very hard. There had been research on this for years mostly for GC languages and for a long time success was rather limited. I think in "recent" years there had been some nice progress on this topic. But then this design decisions where made for rust nearly 9 years ago... I mean the first stable release of rust was 8 years ago. So not including research which didn't yet exist isn't really a poor decision. Furthermore even that success might not be practically appalling for all use-cases. Not for technical reasons. But sadly programming language design isn't just about technical aspects. Anyway I hop you all success with creating such a language or motivating people to create such a language. IMHO most software today doesn't need to be written in any system language at all. This even includes most "system" components. I mean a huge part of systemd could probably be written in WASM and it still would work fine for a lot of use cases as crazy as it sounds. So good luck.

BulgarianIdiot 3y ago
You said splitting entities into small chunks doesn't work for low-level. But as you already know (exposition monologue follows :D) memory, all of it, is allocated in small pages. Typically 4kb of size. That's what the low level actually is. On some hardware the page is 2MB which is called, literally "huge pages" (or "large pages"). But that's about it, bigger pages are essentially not used. If your language can't help but see 4GB as one thing, that's a flaw in your language that it even permits that. I'm not talking about how it appears on the surface, but how it's allocated and chunked. In data structures designed for COW a single 4GB chunk would be anomalous occurrence, basically a bug, everything is split up in structures roughly the size of a page max, or so. So we had some truth here at the hardware and OS level, which is "never allocate things in huge chunks" but we lost it at the higher level and then we think this is just how things are and design all our languages around it. An OS may permit you to allocate more memory, but that's with the idea you'll manage this yourself, say as an arena. If you instead just store a monolithic piece of state in it, that's just ignorance, or a special case, like say large buffer for DMA, or the like. Regarding nested types, you can still inline content, you don't NEED to add indirection. The behavior of inlining vs referencing can be determined based on content structure and should be transparent to your code. It's trivial and many runtimes do this (I suppose your next objection will be how come we have a runtime, but ostensibly even C has part of its library acting as a runtime for basic things like mem alloc). > If you combine it with higher performance requirements and requirements like wide spread platform support etc. it's very very hard. If you have the right foundation, writing optimizations on top of it is a lot more trivial than when your foundation is 70 years of ad-hoc cruft. This was demonstrated by Microsoft experimenting with C# written OS (I think called Singularity) that bypassed a lot of OS and even hardware APIs in order to manage most memory operations & safety on its own. And that turned out faster than C code. Unfortunately, not compatible with the status quo. C is a nightmare.

kaba0 3y ago

Your filesystem and OS has absolutely different constraints, they don’t modify their objects to such a degree as most programs modify memory. So honestly, you should provide more evidence why would that work (for a low-level, systems programming language). And don’t get me wrong, FP languages en large can be very competitive with a more imperative approach, especially when multithreading is on the table, but they achieve it through extensive compile time transformations, and on the data structure level by much smarter algorithms than plain COW.