3 ms·

I can appreciate your "kernel driver and embedded programming" background, and I'm working on my attitude. I think about programming a lot. I don't do much pr

by geocar 11y ago

I can appreciate your "kernel driver and embedded programming" background, and I'm working on my attitude. I think about programming a lot. I don't do much programming since my programs usually work correctly the first time, but when I do I write fast web servers and operating systems and text editors and language bindings and compilers and interpreters and compressors/decompressors and image editing tools, backup tools, system administration tools, reverse engineering network protocols, reverse engineering file formats, invoice generation tools, billing and accounting packages, mobile apps, high volume mail servers, mp4 decoders, 1wire (dallas) drivers, ad servers, ad players, web applications like email clients and web stores, windows device drivers, linux device drivers, and other things that I can't remember right now. I have observed that the bugs I tend to make have more to do with my failing memory (i.e. what I can remember), than overflowing buffers that I allocated. Maybe memory buffer errors happen to other programmers often enough that it's worth worrying about for other people, but I'd argue this says more about those other programmers, and I propose that the techniques I use to avoid making those mistakes are more valuable than memory protection since it clearly allows me to have my cake and eat it too. I've observed a great deal of performance can be gained by controlling how my structure is laid out in memory, but in order to do that, the compiler needs to trust me. Maybe. Some languages are experimenting with algebraic types and what they're doing might be a good-enough middle ground such that they can actually get zero-cost buffer protection, but I haven't played with them much to know for certain. I'm willing to be convinced, but I haven't yet, so I continue to maintain that enforced memory protection is not ideal. Re 2) I think the original point is that the implementation should be free to change it however (and whenever) they like. That's why they wanted opaque strings, such that you access them from the interface itself. Re 3) It depends. If it fails, then we've lost however long it takes to scan 300GB (a few minutes), but just because we copied it once from the disk, doesn't mean we need to copy it again and again: That turns minutes into hours. I agree that most strings don't need to be mutated, and that it's reasonable to default to it, however order-of-magnitude performance gains are worth spending a little time thinking about, and they often require mutating-in-place. Traversing a tree is another good example, and the Schorr-Deutsch-Waite link-inversion algorithm (Knuth, TAOCP vol.1 § 2.3.5) is essential for constrained-memory devices when you need a tree-walker. Sometimes I need to mutate, so I think enforcing immutability is simply not valuable.

vardump 11y ago
> I do I write fast web servers and operating systems and text editors and language bindings and compilers... I didn't want to invoke any contest, but to remind C is used a lot in those contexts and zero termination seems to be one of the bug magnets. A very common source for security vulnerabilities as well, pretty large portion of them are related to string processing. > Maybe memory buffer errors happen to other programmers often enough that it's worth worrying about for other people Yup. Maybe a large part of how I think of this to protect the other people. It's not rare someone messes up on an embedded system and writes a bit somewhere else in memory. Often it won't crash, but the bugs you get take ages to hunt down. There's just no way around it. I do want to delegate things to other people. Defensive coding tends to catch and prevent a lot of mistakes at that point. Many of them have pretty cowboy attitudes towards details such as parameter validation or error checking. Some of them will need to maintain it in the distant future. > I've observed a great deal of performance can be gained by controlling how my structure is laid out in memory, but in order to do that, the compiler needs to trust me. Yeah, performance is often about cache, locality of reference. Pointer chasing and other random access destroys performance. Applying SIMD without penalties requires at least 16 byte alignment. In 2016, 64 byte align is even better. SIMDs are getting pretty wide. Re 3)... I still think that 300GB string example is esoteric, even off topic. You're not going to put 300GB string in any standard string abstraction. There are probably a lot of other worries, such as running out of disk space, undetected data corruption on disk (happens pretty often), failure in the middle of the modification process, etc. > Sometimes I need to mutate, so I think enforcing immutability is simply not valuable. I don't think anyone wanted to enforce immutability. Just to have it default.
- geocar 11y ago
  > I didn't want to invoke any contest, but to remind C is used a lot in those contexts and zero termination seems to be one of the bug magnets. A very common source for security vulnerabilities as well, pretty large portion of them are related to string processing. I understand. I think the gross majority of this comes from the C standard library though, and not from the C language. qmail for example had no buffer overflow problems -- the first security vulnerabilities found 10 years after release wouldn't affect any system that qmail was developed to run on simply because nobody gave their mail server 4GB of ram! Surely the author should have predicted this problem, but it demonstrates methodology can protect against this issue. KDB takes an interesting approach of reference counting in all operators. Operators can then be optimised for situations where one argument (or both) have a reference-count of exactly one. This means writing C code with KDB's memory management library can make things very simple. Antirez also has an interesting string library that uses a combination of length+null-termination specifically for the purpose of finding bugs. > There's just no way around it. I do want to delegate things to other people. Defensive coding tends to catch and prevent a lot of mistakes at that point. Many of them have pretty cowboy attitudes towards details such as parameter validation or error checking. Some of them will need to maintain it in the distant future. I understand what you're saying, but I think technique can help a lot more than we think: Most of the software engineering industry wants to move to smarter and better tooling, and I'm just proposing we upgrade our brains some too. I don't have all the answers yet. > You're not going to put 300GB string in any standard string abstraction. In KDB this is actually pretty common, but it doesn't have a "standard string abstraction" because while it meaningfully supports byte-arrays that are in the 300GB range, the string-atoms (symbols) which are used similar to strings in python et al, never approach 1MB let alone 300GB. However in C I just did mmap() and updated the file. It took about 15 minutes to write and debug, and a few minutes to run on the server. > I don't think anyone wanted to enforce immutability. Just to have it default. I don't know. I parsed I don't know any good reason to deviate from these rules. as "enforce". I might be wrong, you'd have to ask "cousin_it" what he/she meant at the time. I know at least later they either were convinced, or clarified[1] this position. However what I meant was that I don't want to enforce those rules because I can think of an exception for each, and they understood that. Sorry I was unclear though. [1]: https://news.ycombinator.com/item?id=10842858 https://news.ycombinator.com/item?id=10842858

by geocar 11y ago

vardump 11y ago

> I do I write fast web servers and operating systems and text editors and language bindings and compilers... I didn't want to invoke any contest, but to remind C is used a lot in those contexts and zero termination seems to be one of the bug magnets. A very common source for security vulnerabilities as well, pretty large portion of them are related to string processing. > Maybe memory buffer errors happen to other programmers often enough that it's worth worrying about for other people Yup. Maybe a large part of how I think of this to protect the other people. It's not rare someone messes up on an embedded system and writes a bit somewhere else in memory. Often it won't crash, but the bugs you get take ages to hunt down. There's just no way around it. I do want to delegate things to other people. Defensive coding tends to catch and prevent a lot of mistakes at that point. Many of them have pretty cowboy attitudes towards details such as parameter validation or error checking. Some of them will need to maintain it in the distant future. > I've observed a great deal of performance can be gained by controlling how my structure is laid out in memory, but in order to do that, the compiler needs to trust me. Yeah, performance is often about cache, locality of reference. Pointer chasing and other random access destroys performance. Applying SIMD without penalties requires at least 16 byte alignment. In 2016, 64 byte align is even better. SIMDs are getting pretty wide. Re 3)... I still think that 300GB string example is esoteric, even off topic. You're not going to put 300GB string in any standard string abstraction. There are probably a lot of other worries, such as running out of disk space, undetected data corruption on disk (happens pretty often), failure in the middle of the modification process, etc. > Sometimes I need to mutate, so I think enforcing immutability is simply not valuable. I don't think anyone wanted to enforce immutability. Just to have it default.

geocar 11y ago
> I didn't want to invoke any contest, but to remind C is used a lot in those contexts and zero termination seems to be one of the bug magnets. A very common source for security vulnerabilities as well, pretty large portion of them are related to string processing. I understand. I think the gross majority of this comes from the C standard library though, and not from the C language. qmail for example had no buffer overflow problems -- the first security vulnerabilities found 10 years after release wouldn't affect any system that qmail was developed to run on simply because nobody gave their mail server 4GB of ram! Surely the author should have predicted this problem, but it demonstrates methodology can protect against this issue. KDB takes an interesting approach of reference counting in all operators. Operators can then be optimised for situations where one argument (or both) have a reference-count of exactly one. This means writing C code with KDB's memory management library can make things very simple. Antirez also has an interesting string library that uses a combination of length+null-termination specifically for the purpose of finding bugs. > There's just no way around it. I do want to delegate things to other people. Defensive coding tends to catch and prevent a lot of mistakes at that point. Many of them have pretty cowboy attitudes towards details such as parameter validation or error checking. Some of them will need to maintain it in the distant future. I understand what you're saying, but I think technique can help a lot more than we think: Most of the software engineering industry wants to move to smarter and better tooling, and I'm just proposing we upgrade our brains some too. I don't have all the answers yet. > You're not going to put 300GB string in any standard string abstraction. In KDB this is actually pretty common, but it doesn't have a "standard string abstraction" because while it meaningfully supports byte-arrays that are in the 300GB range, the string-atoms (symbols) which are used similar to strings in python et al, never approach 1MB let alone 300GB. However in C I just did mmap() and updated the file. It took about 15 minutes to write and debug, and a few minutes to run on the server. > I don't think anyone wanted to enforce immutability. Just to have it default. I don't know. I parsed I don't know any good reason to deviate from these rules. as "enforce". I might be wrong, you'd have to ask "cousin_it" what he/she meant at the time. I know at least later they either were convinced, or clarified[1] this position. However what I meant was that I don't want to enforce those rules because I can think of an exception for each, and they understood that. Sorry I was unclear though. [1]: https://news.ycombinator.com/item?id=10842858 https://news.ycombinator.com/item?id=10842858