3 ms·

Yeah but that's not the same. UTF-8 is vastly more efficient for european alphabets (and imo, ends up being a wash for non-european ones), while not pandering t

Yeah but that's not the same. UTF-8 is vastly more efficient for european alphabets (and imo, ends up being a wash for non-european ones), while not pandering the seductive lie of UTF-16 that 1 character/rune == 2 bytes. It really shows up in memory usage of large pieces of text, speed of Regex etc. Unfortunately the NT codebase is old enough to have made the wrong choice regarding this. I wonder if such a a transition would be possible for .NET as well, because while technically you don't get access to the raw strings easily, there's tons of APIs that allow you to grab a pointer to the underlying memory in strings, which would immediately break if such a transition was made.

pjmlp 5y ago
This is one of the scenarios where Java being more high level than .NET wins out. Additionally to what is being discussed, string compression and deduplication exists since several releases.
- recursive 5y ago
  It's certainly possible in .net, and has been seriously proposed. https://github.com/dotnet/runtime/issues/933 https://github.com/dotnet/runtime/issues/933
  - pjmlp 5y ago
    This is not the same as Java, because there the change is transparent, no need for additional types.
  - torginus 5y ago
    Yeah, but this is talking about introducing a separate, UTF-8 string type. .NET already has some support for UTF-8, for example ASP.NET and the new Json API uses UTF-8 directly, however the API is just gnarly.
  - WorldMaker 5y ago
    https://github.com/dotnet/runtime/issues/6612 https://github.com/dotnet/runtime/issues/6612 is the more directly comparable proposal.
- torginus 5y ago
  I'm not sure about compression, but deduplication has been in .NET since forever.
- WorldMaker 5y ago
  .NET hasn't been so "low level" that it can't consider a different string implementation since "Core". There have been multiple explorations and discussions and considerations of moving .NET to using UTF-8 string representations internally (rather than UTF-16): https://github.com/dotnet/runtime/issues/6612 https://github.com/dotnet/runtime/issues/6612 It would potentially slow down platform invokes (C/C++ DLLs) and COM calls, especially on Windows, but that's not a "low level" concern just as JNI's existence doesn't imply that Java is "low level".
  - pjmlp 5y ago
    People tend to forget CLR was developed to support all languages, including C and C++. As such, it is possible to access details in unsafe code via MSIL, than on the JVM side are not accessible. So naturally while nothing prevents them to eventually introduce such change, it might come with breaking changes to code that although legal, is playing with such low level details. As for the JNI, that is outside of what I am discussing of what is possible only with bytecodes.
    - WorldMaker 5y ago
      It doesn't have anything to do with C/C++ support: .NET strings have never been null terminated C-style strings, they've always been Length encoded, garbage-collected strings and have always needed to be marshalled/converted to/from C-style strings no matter how "raw" bytecode you get. It is possible to do unsafe pointer manipulation in the CLR, but using that for string manipulation was always a bad idea (too easy to fall into bad UCS-2 assumptions that don't apply to UTF-16), and in modern .NET not only is it the wrong approach but Span<T> and Rune APIs provide a much friendlier and better way to do it without unsafe pointer manipulation. You are correct that there probably is more bad code using unsafe CLR pointer access for string manipulation in the wild than should probably exist in .NET and ever existed in the JVM, but .NET has made breaking changes to unsafe code before (.NET Framework 1 to .NET Framework 2 involved quite a few; it may be long ago in memory, but modern .NET transitions have been closer to that 1 to 2 transition than not; already gone are the days of .NET Framework 2 to 4.x where they were afraid to make low level breaking changes because the Framework was bundled with Windows).

pjmlp 5y ago

This is one of the scenarios where Java being more high level than .NET wins out. Additionally to what is being discussed, string compression and deduplication exists since several releases.

recursive 5y ago
It's certainly possible in .net, and has been seriously proposed. https://github.com/dotnet/runtime/issues/933 https://github.com/dotnet/runtime/issues/933
- pjmlp 5y ago
  This is not the same as Java, because there the change is transparent, no need for additional types.
- torginus 5y ago
  Yeah, but this is talking about introducing a separate, UTF-8 string type. .NET already has some support for UTF-8, for example ASP.NET and the new Json API uses UTF-8 directly, however the API is just gnarly.
- WorldMaker 5y ago
  https://github.com/dotnet/runtime/issues/6612 https://github.com/dotnet/runtime/issues/6612 is the more directly comparable proposal.
torginus 5y ago
I'm not sure about compression, but deduplication has been in .NET since forever.
WorldMaker 5y ago
.NET hasn't been so "low level" that it can't consider a different string implementation since "Core". There have been multiple explorations and discussions and considerations of moving .NET to using UTF-8 string representations internally (rather than UTF-16): https://github.com/dotnet/runtime/issues/6612 https://github.com/dotnet/runtime/issues/6612 It would potentially slow down platform invokes (C/C++ DLLs) and COM calls, especially on Windows, but that's not a "low level" concern just as JNI's existence doesn't imply that Java is "low level".
- pjmlp 5y ago
  People tend to forget CLR was developed to support all languages, including C and C++. As such, it is possible to access details in unsafe code via MSIL, than on the JVM side are not accessible. So naturally while nothing prevents them to eventually introduce such change, it might come with breaking changes to code that although legal, is playing with such low level details. As for the JNI, that is outside of what I am discussing of what is possible only with bytecodes.
  - WorldMaker 5y ago
    It doesn't have anything to do with C/C++ support: .NET strings have never been null terminated C-style strings, they've always been Length encoded, garbage-collected strings and have always needed to be marshalled/converted to/from C-style strings no matter how "raw" bytecode you get. It is possible to do unsafe pointer manipulation in the CLR, but using that for string manipulation was always a bad idea (too easy to fall into bad UCS-2 assumptions that don't apply to UTF-16), and in modern .NET not only is it the wrong approach but Span<T> and Rune APIs provide a much friendlier and better way to do it without unsafe pointer manipulation. You are correct that there probably is more bad code using unsafe CLR pointer access for string manipulation in the wild than should probably exist in .NET and ever existed in the JVM, but .NET has made breaking changes to unsafe code before (.NET Framework 1 to .NET Framework 2 involved quite a few; it may be long ago in memory, but modern .NET transitions have been closer to that 1 to 2 transition than not; already gone are the days of .NET Framework 2 to 4.x where they were afraid to make low level breaking changes because the Framework was bundled with Windows).