10 ms·

Point 3 reminds me of Haskell trying to prove certain grammars correct at compile-time or C++ calculating factorials using template meta-programming. Very cool

Point 3 reminds me of Haskell trying to prove certain grammars correct at compile-time or C++ calculating factorials using template meta-programming. Very cool to show off to your geek programmer friends and impress them, but I really question the day-to-day usefulness when considering the added complexity. I have never ever had bugs in my code from somehow having invalid Unicode code-points in my paths. 99% of the time, replacing '/' with '\' and handling drive letters gets you there. I realize certain classes of bugs could happen, but I'd rather take that risk, have a bug, and fix it, than burden the type system and libraries. In a way, I'm a little sad that modern computer languages are going from C's extreme of "always trust the programmer" to the other of "never trust the programmer". Bugs are bad, but language usability, ease-of-use and simplicity are also worth fighting for. Having said all that, I am really happy to see Rust being awesome in the general case and appreciate all the work you and the rest of the team are putting forth.

oldmanjay 11y ago
I definitely fall on the other side of the fence, philosophically. Perhaps I've just fixed too many of those "trust me" bugs that programmers are constantly adding into code I have to maintain. I'm especially over the thinking that sufficient vigilance is something we should rely on humans to achieve when the computer is way, way better at it.
vessenes 11y ago
> 99% of the time, replacing '/' with '\' and handling drive letters gets you there. I think that's precisely the point; at scale, and with today's modular software culture (like last week's left-pad news cycle demonstrates), 99% correct software can mean millions of installations of bad code, all vulnerable to attack, or even just 1%-type bugs. One way to think about the last 10 years of language design is an ongoing dialogue on how much or little to trust engineer intuition about which tools are safe for which jobs. The pendulum is swinging towards the belief that in general engineers should be given much more safety-oriented tools.
pcwalton 11y ago
> Point 3 reminds me of Haskell trying to prove certain grammars correct at compile-time or C++ calculating factorials using template meta-programming. They don't seem analogous in the slightest. Grammars can be correct without being proven correct. Calculating factorials at runtime instead of at compile time doesn't change the result of the calculation. But treating all paths as though they were UTF-8 breaks programs when certain paths are fed to them. > I have never ever had bugs in my code from somehow having invalid Unicode code-points in my paths. 99% of the time, replacing '/' with '\' and handling drive letters gets you there. I realize certain classes of bugs could happen, but I'd rather take that risk, have a bug, and fix it, than burden the type system and libraries. So this philosophy is what brought us the fun of feeding paths that contain spaces to shell scripts. Except this would be worse, because you can write space-correct Bash scripts—but if paths in the language are UTF-8, and you have to work with a path that contains non-UTF-8 characters, there is literally nothing you can do, and your program will be broken no matter what†. I'm glad that Rust took the time to do things right and figured out how to make the right thing ergonomic instead of giving up, deciding that correct paths are impossible, and forcing everyone to write broken programs. By the way, this is not a theoretical problem: VCS's have been bitten by this in the real world. https://www.redmine.org/issues/2664 https://www.redmine.org/issues/2664 † Note that Golang has a different solution to this problem: strings in Golang don't have to be UTF-8, and they're only UTF-8 by convention. This is a totally valid solution, but Rust didn't go with it because removing the invariant that strings are UTF-8 has downstream consequences that affect every API that takes strings.
Dylan16807 11y ago
Fixing a handful of programs that way is already more expensive than putting it in the language.
dllthomas 11y ago
This is not "never trust the programmer", it's "do we hand the programmer a broken library?"
catnaroek 11y ago
It's not like we embrace complexity for its own sake. The harsh reality is that things like file paths, dates and times, phone numbers and even people's names are already intrinsically complex; and you can either acknowledge this complexity in your code, or ignore it at your own peril. If you want to blame anyone anyone for the complexity, blame those who design file systems, or ISO-standardize date and time formats, or even humanity as a whole for coming up with all sorts of naming customs! A programmer (or any other given single person) has only so much control over the complexity of their environment. Also, you seem to think typeful programming is just for those who either want to show off (“to your geek programmer friends”) or don't trust other programmers (“to the other [extreme] of "never trust the programmer"”). This plainly untrue. Typeful programming is for those who embrace code as a communication medium (“my data structure has these complex invariants, which I can communicate either with three lines of code, or with two long English paragraphs in comments than might even get out of sync”) and want to include the type checker in the conversation (“this code is rather tricky, and I made lots of changes to it, could you check for me if I missed anything?”). Of course, the effectiveness of code as a communication medium is contingent on your ability to understand code.
- bjz_ 11y ago
  Exactly. The complexity is inherent in the domain, and making it seem simpler by sweeping it under the carpet is not solving the problem, and it will result in hard to find bugs when working at scale.
  - dllthomas 11y ago
    There's abstraction, leaky abstraction, and brokenness. Making things seem simpler by finding an abstraction over them is great! Making things seem simpler by finding an abstraction that leaks some details so you sometimes need to handle them at a high level is... less great. But sometimes still a win. Making things seem simpler but neither handling the complexity nor allowing your users to handle the complexity is technical debt at best.
    - catnaroek 11y ago
sgift 11y ago
> I have never ever had bugs in my code from somehow having invalid Unicode code-points in my paths. 99% of the time, replacing '/' with '\' and handling drive letters gets you there. I realize certain classes of bugs could happen, but I'd rather take that risk, have a bug, and fix it, than burden the type system and libraries. Lucky for you. I will think about you next time I have to fix one of these 1% pathname bugs you do not seem to be concerned about, which tend to happen all the time when your program is used by many people with many files (i.e. the case every developer hopes for)
- spotman 11y ago
  Yea the last 1% is the most fun and difficult. One of the projects I am working on lately is a data integrity and syncing platform with multiple PB of data in multiple data centers, and the dataset has been around for 10-12 years. Some of the file names and path names I see have the most unnatural and unbelievable characters in them. It can absolutely be quite a challenge. This point about rusts string system sounds like maybe could give me some very precise control.
Retra 11y ago
You shouldn't have to roll your own (likely incorrect) path validation code every time you handle a file. This is something just about every program does. There are standard ways of doing this, and those standards can be done in a standard library.
rkangel 11y ago
The problem is that C's case of "always trust the programmer" has proven over time not to be sufficient. This is particularly true for security bugs. The major driver behind Rust's design is memory safety - make it impossible (or very difficult) for certain classes of bugs to exist. You can write C software that is reliable without that safety, but it is very time consuming and never perfect. This is a similar situation - give the programmer tools to write "obviously" correct code.
lmm 11y ago
> I have never ever had bugs in my code from somehow having invalid Unicode code-points in my paths. Do you use any computers with non-unicode filesystems? I hit such a bug every month or two. (Most recently a couple of days ago in Deluge, for a concrete example, which seems to not open .torrent files with non-unicode filenames). If you're actually using a computer with a non-unicode filesystem at all regularly then I'm honestly amazed you haven't seen such bugs. > I realize certain classes of bugs could happen, but I'd rather take that risk, have a bug, and fix it, than burden the type system and libraries. It's a question of costs and benefits, no? A particular class of bugs has severity x and happens at rate y. If the language is such that the type system and libraries can handle it at a cost of less than xy (in terms of complicating or slowing development) then it's worth handling it there; if not, not. > In a way, I'm a little sad that modern computer languages are going from C's extreme of "always trust the programmer" to the other of "never trust the programmer". Bugs are bad, but language usability, ease-of-use and simplicity are also worth fighting for. Language usability, ease-of-use and simplicity are absolutely valuable. A good language is one that allows you to write correct programs without sacrificing those things. I find a language that will keep track of things for me is a actually huge help in writing correct code. E.g. if I'm writing Python 2 then I still have to worry about whether a particular object is a string or a byte sequence - worse, I have to do all the work of keeping track of it in my head, because the language will just silently corrupt data if I get it wrong. I end up writing comments on my functions that say what a particular value is, and carefully reading my code to verify I've made encode()/decode() calls in the right places. I don't want my language to "trust me", any more than I'd want to e.g. try to cut a piece of wood to the right length without measuring or marking it first.
- douche 11y ago
  Out of curiosity, what would be some examples of computers with non-unicode filesystems? I'm scratching my head trying to come up with anything other than old, old legacy systems that used EBCDIC.
  - lmm 11y ago
    Windows. (Many of the western codepages are either subsets of UTF-8, or not actually UTF-8 but can roundtrip through UTF-8 so you don't actually notice the difference as long as the program is just using the filename as an opaque "handle" and not actually manipulating it. In my case my filesystem is Shift-JIS for compatibility with some old Japanese programs, and that doesn't roundtrip cleanly through unicode)

oldmanjay 11y ago

I definitely fall on the other side of the fence, philosophically. Perhaps I've just fixed too many of those "trust me" bugs that programmers are constantly adding into code I have to maintain. I'm especially over the thinking that sufficient vigilance is something we should rely on humans to achieve when the computer is way, way better at it.

vessenes 11y ago

> 99% of the time, replacing '/' with '\' and handling drive letters gets you there. I think that's precisely the point; at scale, and with today's modular software culture (like last week's left-pad news cycle demonstrates), 99% correct software can mean millions of installations of bad code, all vulnerable to attack, or even just 1%-type bugs. One way to think about the last 10 years of language design is an ongoing dialogue on how much or little to trust engineer intuition about which tools are safe for which jobs. The pendulum is swinging towards the belief that in general engineers should be given much more safety-oriented tools.

pcwalton 11y ago

> Point 3 reminds me of Haskell trying to prove certain grammars correct at compile-time or C++ calculating factorials using template meta-programming. They don't seem analogous in the slightest. Grammars can be correct without being proven correct. Calculating factorials at runtime instead of at compile time doesn't change the result of the calculation. But treating all paths as though they were UTF-8 breaks programs when certain paths are fed to them. > I have never ever had bugs in my code from somehow having invalid Unicode code-points in my paths. 99% of the time, replacing '/' with '\' and handling drive letters gets you there. I realize certain classes of bugs could happen, but I'd rather take that risk, have a bug, and fix it, than burden the type system and libraries. So this philosophy is what brought us the fun of feeding paths that contain spaces to shell scripts. Except this would be worse, because you can write space-correct Bash scripts—but if paths in the language are UTF-8, and you have to work with a path that contains non-UTF-8 characters, there is literally nothing you can do, and your program will be broken no matter what†. I'm glad that Rust took the time to do things right and figured out how to make the right thing ergonomic instead of giving up, deciding that correct paths are impossible, and forcing everyone to write broken programs. By the way, this is not a theoretical problem: VCS's have been bitten by this in the real world. https://www.redmine.org/issues/2664 https://www.redmine.org/issues/2664 † Note that Golang has a different solution to this problem: strings in Golang don't have to be UTF-8, and they're only UTF-8 by convention. This is a totally valid solution, but Rust didn't go with it because removing the invariant that strings are UTF-8 has downstream consequences that affect every API that takes strings.

Dylan16807 11y ago

Fixing a handful of programs that way is already more expensive than putting it in the language.

dllthomas 11y ago

This is not "never trust the programmer", it's "do we hand the programmer a broken library?"

catnaroek 11y ago

It's not like we embrace complexity for its own sake. The harsh reality is that things like file paths, dates and times, phone numbers and even people's names are already intrinsically complex; and you can either acknowledge this complexity in your code, or ignore it at your own peril. If you want to blame anyone anyone for the complexity, blame those who design file systems, or ISO-standardize date and time formats, or even humanity as a whole for coming up with all sorts of naming customs! A programmer (or any other given single person) has only so much control over the complexity of their environment. Also, you seem to think typeful programming is just for those who either want to show off (“to your geek programmer friends”) or don't trust other programmers (“to the other [extreme] of "never trust the programmer"”). This plainly untrue. Typeful programming is for those who embrace code as a communication medium (“my data structure has these complex invariants, which I can communicate either with three lines of code, or with two long English paragraphs in comments than might even get out of sync”) and want to include the type checker in the conversation (“this code is rather tricky, and I made lots of changes to it, could you check for me if I missed anything?”). Of course, the effectiveness of code as a communication medium is contingent on your ability to understand code.

bjz_ 11y ago
Exactly. The complexity is inherent in the domain, and making it seem simpler by sweeping it under the carpet is not solving the problem, and it will result in hard to find bugs when working at scale.
- dllthomas 11y ago
  There's abstraction, leaky abstraction, and brokenness. Making things seem simpler by finding an abstraction over them is great! Making things seem simpler by finding an abstraction that leaks some details so you sometimes need to handle them at a high level is... less great. But sometimes still a win. Making things seem simpler but neither handling the complexity nor allowing your users to handle the complexity is technical debt at best.
  - catnaroek 11y ago

sgift 11y ago

> I have never ever had bugs in my code from somehow having invalid Unicode code-points in my paths. 99% of the time, replacing '/' with '\' and handling drive letters gets you there. I realize certain classes of bugs could happen, but I'd rather take that risk, have a bug, and fix it, than burden the type system and libraries. Lucky for you. I will think about you next time I have to fix one of these 1% pathname bugs you do not seem to be concerned about, which tend to happen all the time when your program is used by many people with many files (i.e. the case every developer hopes for)

spotman 11y ago
Yea the last 1% is the most fun and difficult. One of the projects I am working on lately is a data integrity and syncing platform with multiple PB of data in multiple data centers, and the dataset has been around for 10-12 years. Some of the file names and path names I see have the most unnatural and unbelievable characters in them. It can absolutely be quite a challenge. This point about rusts string system sounds like maybe could give me some very precise control.

Retra 11y ago

You shouldn't have to roll your own (likely incorrect) path validation code every time you handle a file. This is something just about every program does. There are standard ways of doing this, and those standards can be done in a standard library.

rkangel 11y ago

The problem is that C's case of "always trust the programmer" has proven over time not to be sufficient. This is particularly true for security bugs. The major driver behind Rust's design is memory safety - make it impossible (or very difficult) for certain classes of bugs to exist. You can write C software that is reliable without that safety, but it is very time consuming and never perfect. This is a similar situation - give the programmer tools to write "obviously" correct code.

lmm 11y ago

> I have never ever had bugs in my code from somehow having invalid Unicode code-points in my paths. Do you use any computers with non-unicode filesystems? I hit such a bug every month or two. (Most recently a couple of days ago in Deluge, for a concrete example, which seems to not open .torrent files with non-unicode filenames). If you're actually using a computer with a non-unicode filesystem at all regularly then I'm honestly amazed you haven't seen such bugs. > I realize certain classes of bugs could happen, but I'd rather take that risk, have a bug, and fix it, than burden the type system and libraries. It's a question of costs and benefits, no? A particular class of bugs has severity x and happens at rate y. If the language is such that the type system and libraries can handle it at a cost of less than xy (in terms of complicating or slowing development) then it's worth handling it there; if not, not. > In a way, I'm a little sad that modern computer languages are going from C's extreme of "always trust the programmer" to the other of "never trust the programmer". Bugs are bad, but language usability, ease-of-use and simplicity are also worth fighting for. Language usability, ease-of-use and simplicity are absolutely valuable. A good language is one that allows you to write correct programs without sacrificing those things. I find a language that will keep track of things for me is a actually huge help in writing correct code. E.g. if I'm writing Python 2 then I still have to worry about whether a particular object is a string or a byte sequence - worse, I have to do all the work of keeping track of it in my head, because the language will just silently corrupt data if I get it wrong. I end up writing comments on my functions that say what a particular value is, and carefully reading my code to verify I've made encode()/decode() calls in the right places. I don't want my language to "trust me", any more than I'd want to e.g. try to cut a piece of wood to the right length without measuring or marking it first.

douche 11y ago
Out of curiosity, what would be some examples of computers with non-unicode filesystems? I'm scratching my head trying to come up with anything other than old, old legacy systems that used EBCDIC.
- lmm 11y ago
  Windows. (Many of the western codepages are either subsets of UTF-8, or not actually UTF-8 but can roundtrip through UTF-8 so you don't actually notice the difference as long as the program is just using the filename as an opaque "handle" and not actually manipulating it. In my case my filesystem is Shift-JIS for compatibility with some old Japanese programs, and that doesn't roundtrip cleanly through unicode)