6 ms·

They should be byte (or 16-bit code unit, or whatever) strings. There is no ambiguity here, only incompatibility and delusion. Rust gets this almost right. Pyt

They should be byte (or 16-bit code unit, or whatever) strings. There is no ambiguity here, only incompatibility and delusion. Rust gets this almost right. Python gets this very wrong.

jefftk 4y ago
> Rust gets this almost right. Python gets this very wrong. Having worked in both, I'd say they both chose ideomatic solutions: Rust: I can't prove this is utf-8, so if you want to use it as utf-8 you'll need to tell me what to do if it isn't. Python: if you're in the common situation where everything is utf-8 and you want to just work with strings, go do the simple thing. Or you can be explicit about wanting to work with bytes, and that's good too. (Though I think Python should throw an error instead of silently omitting non-utf8 files.)
- jefftk 4y ago
  Actually, I think I'm wrong about python's behavior, at least now; it doesn't omit the files, and instead does something with surrogates. On Linux: >>> b'\xc0'.decode('utf-8') Traceback (most recent call last): File "<stdin>", line 1, in <module> UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc0 in position 0: invalid start byte >>> open(b'\xc0', 'w').write('foo') 3 >>> os.listdir() ['\udcc0'] >>> open(b'\xc0').read() 'foo' >>> open('\udcc0').read() 'foo'
  - naniwaduni 4y ago
    PEP 383[1] is a workaround, but it took a while to get there[2]. [1]: https://peps.python.org/pep-0383/ https://peps.python.org/pep-0383/ [2]: cf. https://github.com/bup/bup/blob/master/DESIGN#L667-L729 https://github.com/bup/bup/blob/master/DESIGN#L667-L729
- nerdponx 4y ago
  I just did a bit of research into this here https://news.ycombinator.com/item?id=32472087 https://news.ycombinator.com/item?id=32472087 You can actually reconfigure Python to throw an error instead of using a surrogate escape, but only (I think) changing something at compile time: https://docs.python.org/3/library/sys.html#sys.getfilesystemencodeerrors https://docs.python.org/3/library/sys.html#sys.getfilesystem...

They should be byte (or 16-bit code unit, or whatever) strings. There is no ambiguity here, only incompatibility and delusion. Rust gets this almost right. Python gets this very wrong.

jefftk 4y ago

> Rust gets this almost right. Python gets this very wrong. Having worked in both, I'd say they both chose ideomatic solutions: Rust: I can't prove this is utf-8, so if you want to use it as utf-8 you'll need to tell me what to do if it isn't. Python: if you're in the common situation where everything is utf-8 and you want to just work with strings, go do the simple thing. Or you can be explicit about wanting to work with bytes, and that's good too. (Though I think Python should throw an error instead of silently omitting non-utf8 files.)

jefftk 4y ago
Actually, I think I'm wrong about python's behavior, at least now; it doesn't omit the files, and instead does something with surrogates. On Linux: >>> b'\xc0'.decode('utf-8') Traceback (most recent call last): File "<stdin>", line 1, in <module> UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc0 in position 0: invalid start byte >>> open(b'\xc0', 'w').write('foo') 3 >>> os.listdir() ['\udcc0'] >>> open(b'\xc0').read() 'foo' >>> open('\udcc0').read() 'foo'
- naniwaduni 4y ago
  PEP 383[1] is a workaround, but it took a while to get there[2]. [1]: https://peps.python.org/pep-0383/ https://peps.python.org/pep-0383/ [2]: cf. https://github.com/bup/bup/blob/master/DESIGN#L667-L729 https://github.com/bup/bup/blob/master/DESIGN#L667-L729
nerdponx 4y ago
I just did a bit of research into this here https://news.ycombinator.com/item?id=32472087 https://news.ycombinator.com/item?id=32472087 You can actually reconfigure Python to throw an error instead of using a surrogate escape, but only (I think) changing something at compile time: https://docs.python.org/3/library/sys.html#sys.getfilesystemencodeerrors https://docs.python.org/3/library/sys.html#sys.getfilesystem...