5 ms·

> rustdoc, being a part of the Rust distribution, > has the same stability guarantees that Rust does: > once it gains a feature, it will never go away. We

> rustdoc, being a part of the Rust distribution, > has the same stability guarantees that Rust does: > once it gains a feature, it will never go away. We > (well, not me I don't really work on rustdoc, to be > clear) need the capacity to try out new features > without committing to them, same as features in the > language. I guess I expect rustdoc and rustc to be much less tightly coupled, so that I could use (for example) stable rustc and nightly rustdoc on the same code. In the current model, I can't use nightly rustdoc features without also using nightly rustc, because the annotations are processed by both systems. You could imagine an alternate model where rustc stabilizes `#[doc(...)]` and leaves everything in there up to rustdoc's interpretation, and nightly rustdoc might enable new parameters to the #[doc] annotation. > Neither of these support namespacing though, right? That's correct. I meant that mostly in terms of "package registries are not a new and untested idea". It's like using strings for file paths: not obviously a bad idea, turns out to be unworkable for obscure reasons, and as a new language Rust got to avoid that particular mistake. It feels like crates.io didn't (/ doesn't) have a similar process of avoiding known mistakes. > This shouldn't change based on if you're using Cargo > or not; if it's re-building dependencies every time > you touch a file, that is a bug. Given a crate with a hundred files, each one fairly independent of the other (i.e. a broad shallow build graph), changing one requires (if I understand rustc correctly) recompiling them all. Otherwise the `crate::` references can't be type-checked.

migueloller 6y ago
Would you be willing to expand on why using strings for file paths turns out to be unworkable? Very curious what the obscure reasons are!
- jmgao 6y ago
  If your String type must be valid Unicode (which is true for pretty much every language that isn't C++), it cannot represent all paths. On most unixy filesystems, filenames are arbitrary byte sequences which aren't valid UTF-8, and on windows, NTFS stores filenames as UTF-16, but it allows unpaired surrogates.
  - erik_seaberg 6y ago
    Is that technically UCS-2 rather than UTF-16?
    - ChrisSD 6y ago
      Technically not since ~2000. Windows uses UTF-16 throughout the OS. That is, Windows strings will be encoded and decoded as UTF-16 not UCS-2. However the tricky bit is that the kernel doesn't enforce this so it's possible for programmers to intentionally make broken UTF-16 strings. A broken UTF-16 string shouldn't be considered UCS-2 just because it happens to be a valid UCS-2 string (otherwise all bit patterns could be called "UCS-2" so long as they are an even number of bytes in length).
  - throwaway894345 6y ago
    What languages require string types to be valid Unicode? Go for example let’s you put anything in a string; it’s only Unicode if you put Unicode into the string including via creating string literals.
    - ChrisSD 6y ago
      Rust, Java, Swift...
    - cptroot 6y ago
      I think python also takes the approach where strings must be valid unicode. Both Rust and Python use their String types to represent valid unicode, and something else ([] and byte strings, respectively) to represent other encodings and invalid unicode.
    - rkangel 6y ago
      Requiring strings to be valid unicode simplifies the internal workings of (and use of) a language greatly. Dealing with unknown input becomes a single operation at the boundary after which you can safely assume everything is validly printable or processable as text.
- steveklabnik 6y ago
  (To add on to the sibling comment's great answer, here's Rust's docs on this https://doc.rust-lang.org/stable/std/path/index.html https://doc.rust-lang.org/stable/std/path/index.html and https://doc.rust-lang.org/stable/std/ffi/struct.OsString.html https://doc.rust-lang.org/stable/std/ffi/struct.OsString.htm... )
- jmillikin 6y ago
  Short version: file paths are an OS-specific construct, and not all OSes have paths that can be round-tripped through Unicode. The first bad option is to say "paths are bytes", which is true on traditional POSIX platforms. Then you run your code on macOS and discover it's processing your bytes as UTF-8 and applying Unicode normalization. After fixing that you try porting to Windows and get to learn about `wchar_t`. The other bad option is to say "paths are (Unicode) strings", which works fine on Windows and macOS. It works 90% of the time on POSIX, but you'll eventually find users with non-UTF8 file paths and then you're in trouble. My first hand experience with this going wrong is GHC, which started off with raw bytes, then moved to strings using a bad conversion function, then they tried to work around the issue by placing non-UTF8 bytes into a Unicode reserved area. Java also has problems because it represents paths as strings and tries to use the libc locale for decoding. https://github.com/bazelbuild/bazel/pull/10111 https://github.com/bazelbuild/bazel/pull/10111 is an example of how this breaks software in ways that are difficult to work around.
  - migueloller 6y ago
    Thanks for the response!
- rstuart4133 6y ago
  More interesting is what you do use for file paths, if you accept that both strings and bytes don't work. The answer will be I guess a dedicated FilePath type that knows paths are lists of blobs, and the blobs can further be divided up into prefixes and an extensions. But that doesn't really solve the problem. You still have to convert file paths to strings to print them, and strings to file paths to accept them as from the terminal and databases. For that to work the round trip had better be safe on any given platform otherwise the user can't copy and paste. Not addressing this problem has been python3's undoing when it is used as a shell scripting replacement. Actually, it's an undoing any time you want to treat "text" as interesting ascii swimming in an unknown encoding, particularly when you want to modify that ascii. HTML with an unknown encoding is a prime example.
steveklabnik 6y ago
> I guess I expect rustdoc and rustc to be much less tightly coupled Yeah, and that could work in theory. The thing is, rustdoc uses the compiler as a library, so they actually are pretty tightly coupled. This causes a lot of pain but also has a bunch of upsides. I really want to see a rustdoc that is not, but it's gonna take a long time. > That's correct. Cool, I thought so, I just wasn't 100% sure! I think it's interesting how different people take away different things here, to me, this demonstrates that you don't have to have namespaces; many of the largest ecosystems do not and have not had them. We did learn a lesson there, just didn't come to the same conclusion that you did. This one is more controversial within the community though, many people do agree with you. > Given a crate with a hundred files Yes, but that doesn't change with rustc vs cargo though; it's still a single rustc invocation to compile all those files. (And, it doesn't quite do so, there is some degree of incremental compilation going on, but it's not as fine-grained as it can be. Working on it!)
- jmillikin 6y ago
  > Yes, but that doesn't change with rustc vs cargo > though; it's still a single rustc invocation to > compile all those files. When using Bazel instead of Cargo the developer can split the build graph into much smaller crates (down to one per module if desired), and still treat the entire assembly as one package. I should emphasize that this is a limitation of Cargo, not rustc. rustc does just fine with big libraries, it's only Cargo that makes it difficult to write hundred-crate packages.
  - steveklabnik 6y ago
    Ah, so you're not saying that you take the same project and port it from Cargo to Bazel and it somehow does less work, you're saying that it's easier to make projects with a larger number of smaller crates with Bazel than Cargo? That would be the bit I'm missing, Bazel and Cargo should invoke rustc the same amount of times per crate: once. Cargo doesn't change the compilation model of the language. (Sorry, I feel a bit dense here, but I am very curious to understand exactly. I haven't used Bazel a ton yet, but am very interested in it.)
eddyb 6y ago
You can use #[cfg(doc)] and #[cfg_attr(doc, ...)], but it looks like that still requires #![feature(doc_cfg)], so I guess it wouldn't be compatible with stable rustc anyway, until it's stabilized itself.

migueloller 6y ago

Would you be willing to expand on why using strings for file paths turns out to be unworkable? Very curious what the obscure reasons are!

jmgao 6y ago
If your String type must be valid Unicode (which is true for pretty much every language that isn't C++), it cannot represent all paths. On most unixy filesystems, filenames are arbitrary byte sequences which aren't valid UTF-8, and on windows, NTFS stores filenames as UTF-16, but it allows unpaired surrogates.
- erik_seaberg 6y ago
  Is that technically UCS-2 rather than UTF-16?
  - ChrisSD 6y ago
    Technically not since ~2000. Windows uses UTF-16 throughout the OS. That is, Windows strings will be encoded and decoded as UTF-16 not UCS-2. However the tricky bit is that the kernel doesn't enforce this so it's possible for programmers to intentionally make broken UTF-16 strings. A broken UTF-16 string shouldn't be considered UCS-2 just because it happens to be a valid UCS-2 string (otherwise all bit patterns could be called "UCS-2" so long as they are an even number of bytes in length).
- throwaway894345 6y ago
  What languages require string types to be valid Unicode? Go for example let’s you put anything in a string; it’s only Unicode if you put Unicode into the string including via creating string literals.
  - ChrisSD 6y ago
    Rust, Java, Swift...
  - cptroot 6y ago
    I think python also takes the approach where strings must be valid unicode. Both Rust and Python use their String types to represent valid unicode, and something else ([] and byte strings, respectively) to represent other encodings and invalid unicode.
  - rkangel 6y ago
    Requiring strings to be valid unicode simplifies the internal workings of (and use of) a language greatly. Dealing with unknown input becomes a single operation at the boundary after which you can safely assume everything is validly printable or processable as text.
steveklabnik 6y ago
(To add on to the sibling comment's great answer, here's Rust's docs on this https://doc.rust-lang.org/stable/std/path/index.html https://doc.rust-lang.org/stable/std/path/index.html and https://doc.rust-lang.org/stable/std/ffi/struct.OsString.html https://doc.rust-lang.org/stable/std/ffi/struct.OsString.htm... )
jmillikin 6y ago
Short version: file paths are an OS-specific construct, and not all OSes have paths that can be round-tripped through Unicode. The first bad option is to say "paths are bytes", which is true on traditional POSIX platforms. Then you run your code on macOS and discover it's processing your bytes as UTF-8 and applying Unicode normalization. After fixing that you try porting to Windows and get to learn about `wchar_t`. The other bad option is to say "paths are (Unicode) strings", which works fine on Windows and macOS. It works 90% of the time on POSIX, but you'll eventually find users with non-UTF8 file paths and then you're in trouble. My first hand experience with this going wrong is GHC, which started off with raw bytes, then moved to strings using a bad conversion function, then they tried to work around the issue by placing non-UTF8 bytes into a Unicode reserved area. Java also has problems because it represents paths as strings and tries to use the libc locale for decoding. https://github.com/bazelbuild/bazel/pull/10111 https://github.com/bazelbuild/bazel/pull/10111 is an example of how this breaks software in ways that are difficult to work around.
- migueloller 6y ago
  Thanks for the response!
rstuart4133 6y ago
More interesting is what you do use for file paths, if you accept that both strings and bytes don't work. The answer will be I guess a dedicated FilePath type that knows paths are lists of blobs, and the blobs can further be divided up into prefixes and an extensions. But that doesn't really solve the problem. You still have to convert file paths to strings to print them, and strings to file paths to accept them as from the terminal and databases. For that to work the round trip had better be safe on any given platform otherwise the user can't copy and paste. Not addressing this problem has been python3's undoing when it is used as a shell scripting replacement. Actually, it's an undoing any time you want to treat "text" as interesting ascii swimming in an unknown encoding, particularly when you want to modify that ascii. HTML with an unknown encoding is a prime example.

steveklabnik 6y ago

> I guess I expect rustdoc and rustc to be much less tightly coupled Yeah, and that could work in theory. The thing is, rustdoc uses the compiler as a library, so they actually are pretty tightly coupled. This causes a lot of pain but also has a bunch of upsides. I really want to see a rustdoc that is not, but it's gonna take a long time. > That's correct. Cool, I thought so, I just wasn't 100% sure! I think it's interesting how different people take away different things here, to me, this demonstrates that you don't have to have namespaces; many of the largest ecosystems do not and have not had them. We did learn a lesson there, just didn't come to the same conclusion that you did. This one is more controversial within the community though, many people do agree with you. > Given a crate with a hundred files Yes, but that doesn't change with rustc vs cargo though; it's still a single rustc invocation to compile all those files. (And, it doesn't quite do so, there is some degree of incremental compilation going on, but it's not as fine-grained as it can be. Working on it!)

jmillikin 6y ago
> Yes, but that doesn't change with rustc vs cargo > though; it's still a single rustc invocation to > compile all those files. When using Bazel instead of Cargo the developer can split the build graph into much smaller crates (down to one per module if desired), and still treat the entire assembly as one package. I should emphasize that this is a limitation of Cargo, not rustc. rustc does just fine with big libraries, it's only Cargo that makes it difficult to write hundred-crate packages.
- steveklabnik 6y ago
  Ah, so you're not saying that you take the same project and port it from Cargo to Bazel and it somehow does less work, you're saying that it's easier to make projects with a larger number of smaller crates with Bazel than Cargo? That would be the bit I'm missing, Bazel and Cargo should invoke rustc the same amount of times per crate: once. Cargo doesn't change the compilation model of the language. (Sorry, I feel a bit dense here, but I am very curious to understand exactly. I haven't used Bazel a ton yet, but am very interested in it.)

eddyb 6y ago

You can use #[cfg(doc)] and #[cfg_attr(doc, ...)], but it looks like that still requires #![feature(doc_cfg)], so I guess it wouldn't be compatible with stable rustc anyway, until it's stabilized itself.