5 ms·

" so if you want good performance, you need to keep all your filesys references in wstirngs." Why? For path manipulation, you can still use whatever format int

by roel_v 12y ago

" so if you want good performance, you need to keep all your filesys references in wstirngs." Why? For path manipulation, you can still use whatever format internally that you want. Only just before the native API calls you'll need to convert. I'm open to many arguments on performance, but I have a hard time believing that this conversion would matter (or even be measurable) relative to the time the actual I/O will take. Filesystem is a great addition to the C++ standard library. If you're seriously claiming that filesystem will not be suitable for production code, I can't help but put you in the (hypothetical) group of people who say 'the standard allocator is only a great prototyping aid, you'd typically want to put in your own allocator before shipping production version.'. Uh no, millions of application ship just fine with the standard allocator, just like thousands (or maybe even hundreds of thousands) of applications are already shipping just fine with boost filesystem.

MaulingMonkey 12y ago
>> " so if you want good performance, you need to keep all your filesys references in wstirngs." > Why? I'd take this even further: The less (re)interpretation the better. Will my UTF16 <-> UTF8 conversion roundtrip my malformed UTF16 codepoints? If not, I've just lost compatibility with (admittedly buggy) software. But I've seen bugs from far tamer, such as from the simple act of converting from a fixed width wchar_t[] that was observed to be a NUL terminated UTF16 string to std::wstring. It turned out that under certain circumstances that couldn't be tested, it was an opaque binary blob with NULs in the middle... and the bit after the NULs being important. Whoops! Less relevant here though: std::experimental::filesystem::v1::path::value_type looks to match the native OS character type. But then again, is that unichar (16-bit) or wchar_t (32-bit) on iOS? Fufufufu~ It's also just plain easier when you're dealing with native filesystem APIs for whatever reason to work in the native character type, which is the main reason I'd prefer it. > but I have a hard time believing that this conversion would matter (or even be measurable) relative to the time the actual I/O will take. RE: Performance: If you're doing actual I/O, agreed. If you're dealing with OS or application cached filesystem entries, it becomes more believable. If you're enumerating over and only conditionally performing I/O over hundreds of thousands of paths - you'll want to be quite cautious of conversions if you care about performance (potentially see: directory enumeration via backups, stale file scans during builds, or comparing file lists between VCS revisions...) It drives me absolutely insane when a 'nothing changed' build takes 10, 20 seconds just from filesystem scanning. And I've seen and fixed far worse (think O(n*n) scans -_-;;). As others have noted though, if you keep your conversions 'reasonable' you should be able to get away with UTF8 paths from a performance perspective. > Filesystem is a great addition to the C++ standard library. Agreed.
- kuschku 12y ago
  Also another problem: UCS-2, commonly used as implementation for UTF-16, doesn’t support all the characters that UTF-8 does, which can lead to other nice problems during conversion.
  - vardump 12y ago
    That's not true. Both UTF-16 and UTF-8 can encode any valid unicode code point.
  - deleted 12y ago
    [deleted]
  - MaulingMonkey 12y ago
    I suspect you're thinking of UCS-2 (UTF-16's precursor), not UTF-16 proper. Both UTF-8 and UTF-16 are variable length encodings capable of storing the entire unicode character set.
  - kevin_thibedeau 12y ago
    UTF-16 covers the full Unicode space just like UTF-8. You're thinking of UCS-2 (limited to plane 0) as implemented by Windows.
    - kuschku 12y ago
      Which can create nice issues, especially because stuff like Java is limited to UCS-2...
      - vardump 12y ago
        > Which can create nice issues, especially because stuff like Java is limited to UCS-2... Not true. Java uses UTF-16, not UCS-2. There are no encoding representation issues.
        heinrich5991 12y ago

by roel_v 12y ago

MaulingMonkey 12y ago

>> " so if you want good performance, you need to keep all your filesys references in wstirngs." > Why? I'd take this even further: The less (re)interpretation the better. Will my UTF16 <-> UTF8 conversion roundtrip my malformed UTF16 codepoints? If not, I've just lost compatibility with (admittedly buggy) software. But I've seen bugs from far tamer, such as from the simple act of converting from a fixed width wchar_t[] that was observed to be a NUL terminated UTF16 string to std::wstring. It turned out that under certain circumstances that couldn't be tested, it was an opaque binary blob with NULs in the middle... and the bit after the NULs being important. Whoops! Less relevant here though: std::experimental::filesystem::v1::path::value_type looks to match the native OS character type. But then again, is that unichar (16-bit) or wchar_t (32-bit) on iOS? Fufufufu~ It's also just plain easier when you're dealing with native filesystem APIs for whatever reason to work in the native character type, which is the main reason I'd prefer it. > but I have a hard time believing that this conversion would matter (or even be measurable) relative to the time the actual I/O will take. RE: Performance: If you're doing actual I/O, agreed. If you're dealing with OS or application cached filesystem entries, it becomes more believable. If you're enumerating over and only conditionally performing I/O over hundreds of thousands of paths - you'll want to be quite cautious of conversions if you care about performance (potentially see: directory enumeration via backups, stale file scans during builds, or comparing file lists between VCS revisions...) It drives me absolutely insane when a 'nothing changed' build takes 10, 20 seconds just from filesystem scanning. And I've seen and fixed far worse (think O(n*n) scans -_-;;). As others have noted though, if you keep your conversions 'reasonable' you should be able to get away with UTF8 paths from a performance perspective. > Filesystem is a great addition to the C++ standard library. Agreed.

kuschku 12y ago
Also another problem: UCS-2, commonly used as implementation for UTF-16, doesn’t support all the characters that UTF-8 does, which can lead to other nice problems during conversion.
- vardump 12y ago
  That's not true. Both UTF-16 and UTF-8 can encode any valid unicode code point.
- deleted 12y ago
  [deleted]
- MaulingMonkey 12y ago
  I suspect you're thinking of UCS-2 (UTF-16's precursor), not UTF-16 proper. Both UTF-8 and UTF-16 are variable length encodings capable of storing the entire unicode character set.
- kevin_thibedeau 12y ago
  UTF-16 covers the full Unicode space just like UTF-8. You're thinking of UCS-2 (limited to plane 0) as implemented by Windows.
  - kuschku 12y ago
    Which can create nice issues, especially because stuff like Java is limited to UCS-2...
    - vardump 12y ago
      > Which can create nice issues, especially because stuff like Java is limited to UCS-2... Not true. Java uses UTF-16, not UCS-2. There are no encoding representation issues.
      - heinrich5991 12y ago