4 ms·

I like that direction better but it requires the ability to declare data-flow based contracts whereas Rust’s tools are only lifetime and type contracts. Is ther

I like that direction better but it requires the ability to declare data-flow based contracts whereas Rust’s tools are only lifetime and type contracts. Is there a language that has data-flow based contracts?

ijustlovemath 2y ago
what do you mean by this?
- vlovich123 2y ago
  There would need to be contractual declarations on the read method that the compiler is able to enforce that tells it that the input &mut slice has N elements clobbered based on the returned length. That’s basically what BorrowedBuf is accomplishing via the type system and runtime enforcement of the contract. Using a non-existent syntax: fn read<T, N: size_t>(&mut self, buf: &mut [MaybeUninit<T>] becomes &[T; N] after call) -> N { … enforces the body initializes N elements out of buf } and then rules that &mut [T] can also be supplied to such functions that today could only accept a &mut [MaybeUninit<T>] transparently. A more likely interface you could write today would look like: fn read_uninit<T>(&mut self, buf: &mut [MaybeUninit<T>]) -> (&[T], &[MaybeUninit<T>]) { … enforces the body initializes N elements out of buf } You still have to cast &[T] into &[MaybeUninit<T>] somehow.
  - Someone 2y ago
    I think an ergonomic way to do that would to have read return not an integer, but a slice of that integer’s length. Problem would be: how do you express “you can only access the buffer you sent me through the read-only slice I returned, but you have to free that same buffer when you’re done calling me? I think that can be done using a function creating a read buffer for a given input stream that - during calls to read is ‘owned for writing’ by that stream (so, it has to borrow a capability that the creator of the buffer doesn’t have. I don’t think Rust currently supports that) - where stream.read returns a read only slice whose lifetime is bound to that of the buffer So, the creator of the buffer can only pass it to read to get a slice back that contains precisely the data read. The stream can write to the entire buffer.
  - gpm 2y ago
    > You still have to cast &[T] into &[MaybeUninit<T>] somehow. unsafe{ std::mem::transmute(slice) } This is probably the only way that will ever exist, because let slice: &mut [NonZeroU8] = ...; let slice_uninit: &mut [MaybeUninit<NonZeroU8>] = ...; let nonzero_uninit: &mut MaybeUninit<NonZeroU8> = &mut slice_uninit[0]; *nonzero_uninit = MaybeUninit::zeroed(); slice[0]; // Undefined behavior for sure by now. Is all safe except for the cast. I.e. MaybeUninit<T> allows you to write invalid bit-patterns to T, so you can't safely cast a reference to T to it (and if you do unsafely cast a reference to T to it you can't soundly write an invalid bit pattern). All current forms of safely making a MaybeUninit take ownership of the value they are declaring to be MaybeUninit for this reason. I guess at some point we might get methods for this on types that can take on all bit patterns - if/when that's encoded as a trait.
queuebert 2y ago
That would be easier but is not required. There are no compiler hints these days to unroll loops or hoist invariants, even though if done incorrectly it could change the result. It would take some complicated analysis, but I think it could be done safely in some cases.
- gpm 2y ago
  I was going to make this argument, but I actually don't think it's true in almost any case. Most functions could be inferred, but the ultimate source of basically all of these write only APIs is FFI functions, which in turn call systemcalls. You're at least going to need a way to annotate the FFI calls and systemcalls to describe to the compiler how they access data.
  - queuebert 2y ago
    If you're calling FFIs in an inner loop, you have bigger issues than the time it takes to clear the buffer, right?
    - gpm 2y ago
      No? It depends on your definition of inner loop I guess. If you're doing some sort of zero-copy IO, the time to clear the buffer might be non-trivial (not huge, but non-trivial). It's true that you need a large enough buffer that syscall/ffi overhead doesn't dominate, but that's not unrealistic. It's rare that we care about this, that's true, that's why generally rust has been fine with "just zero buffers". There are definitely domains that care though.
    - thayne 2y ago
      In some languages, like Java, go and probably Javascript, this is probably true, depending on how much memory needs to to be initialized. But in rust FFI isn't any more expensive than any other non-inlined function call.
- vlovich123 2y ago
  The loop unrolling & invariant hoisting is a static transformation. What the “read” function does semantically isn’t captured today within that and the compiler wouldn’t be able to automatically infer it. It would have to be told that information and there would need to be unsafe annotations for things like syscalls and FFI boundaries. The other approach is to change the API which is what BorrowedBuf is. If you can think of a different approach of how the compiler can figure out automatically what memory has become initialized by a random function call I’m all ears.

ijustlovemath 2y ago

what do you mean by this?

vlovich123 2y ago
There would need to be contractual declarations on the read method that the compiler is able to enforce that tells it that the input &mut slice has N elements clobbered based on the returned length. That’s basically what BorrowedBuf is accomplishing via the type system and runtime enforcement of the contract. Using a non-existent syntax: fn read<T, N: size_t>(&mut self, buf: &mut [MaybeUninit<T>] becomes &[T; N] after call) -> N { … enforces the body initializes N elements out of buf } and then rules that &mut [T] can also be supplied to such functions that today could only accept a &mut [MaybeUninit<T>] transparently. A more likely interface you could write today would look like: fn read_uninit<T>(&mut self, buf: &mut [MaybeUninit<T>]) -> (&[T], &[MaybeUninit<T>]) { … enforces the body initializes N elements out of buf } You still have to cast &[T] into &[MaybeUninit<T>] somehow.
- Someone 2y ago
  I think an ergonomic way to do that would to have read return not an integer, but a slice of that integer’s length. Problem would be: how do you express “you can only access the buffer you sent me through the read-only slice I returned, but you have to free that same buffer when you’re done calling me? I think that can be done using a function creating a read buffer for a given input stream that - during calls to read is ‘owned for writing’ by that stream (so, it has to borrow a capability that the creator of the buffer doesn’t have. I don’t think Rust currently supports that) - where stream.read returns a read only slice whose lifetime is bound to that of the buffer So, the creator of the buffer can only pass it to read to get a slice back that contains precisely the data read. The stream can write to the entire buffer.
- gpm 2y ago
  > You still have to cast &[T] into &[MaybeUninit<T>] somehow. unsafe{ std::mem::transmute(slice) } This is probably the only way that will ever exist, because let slice: &mut [NonZeroU8] = ...; let slice_uninit: &mut [MaybeUninit<NonZeroU8>] = ...; let nonzero_uninit: &mut MaybeUninit<NonZeroU8> = &mut slice_uninit[0]; *nonzero_uninit = MaybeUninit::zeroed(); slice[0]; // Undefined behavior for sure by now. Is all safe except for the cast. I.e. MaybeUninit<T> allows you to write invalid bit-patterns to T, so you can't safely cast a reference to T to it (and if you do unsafely cast a reference to T to it you can't soundly write an invalid bit pattern). All current forms of safely making a MaybeUninit take ownership of the value they are declaring to be MaybeUninit for this reason. I guess at some point we might get methods for this on types that can take on all bit patterns - if/when that's encoded as a trait.

queuebert 2y ago

That would be easier but is not required. There are no compiler hints these days to unroll loops or hoist invariants, even though if done incorrectly it could change the result. It would take some complicated analysis, but I think it could be done safely in some cases.

gpm 2y ago
I was going to make this argument, but I actually don't think it's true in almost any case. Most functions could be inferred, but the ultimate source of basically all of these write only APIs is FFI functions, which in turn call systemcalls. You're at least going to need a way to annotate the FFI calls and systemcalls to describe to the compiler how they access data.
- queuebert 2y ago
  If you're calling FFIs in an inner loop, you have bigger issues than the time it takes to clear the buffer, right?
  - gpm 2y ago
    No? It depends on your definition of inner loop I guess. If you're doing some sort of zero-copy IO, the time to clear the buffer might be non-trivial (not huge, but non-trivial). It's true that you need a large enough buffer that syscall/ffi overhead doesn't dominate, but that's not unrealistic. It's rare that we care about this, that's true, that's why generally rust has been fine with "just zero buffers". There are definitely domains that care though.
  - thayne 2y ago
    In some languages, like Java, go and probably Javascript, this is probably true, depending on how much memory needs to to be initialized. But in rust FFI isn't any more expensive than any other non-inlined function call.
vlovich123 2y ago
The loop unrolling & invariant hoisting is a static transformation. What the “read” function does semantically isn’t captured today within that and the compiler wouldn’t be able to automatically infer it. It would have to be told that information and there would need to be unsafe annotations for things like syscalls and FFI boundaries. The other approach is to change the API which is what BorrowedBuf is. If you can think of a different approach of how the compiler can figure out automatically what memory has become initialized by a random function call I’m all ears.