4 ms·

Are you asking for what is roughly Python generator expressions but in a compiled language?

10000truths 3y ago
Yes, something like Python generators which have yield and send, but typed, and which compiles down to an efficiently sized struct with a resume function. Here's a crude illustration of what I mean, using pseudo-code describing a TCP session flow: coroutine tcp_session() { var syn_packet = @consume(); if(!check_syn_flag(syn_packet)) { @produce(ERROR_BAD_INPUT); return; } var syn_ack_packet = build_tcp_segment(...); @produce(syn_ack_packet); var ack_packet = @consume(); if(!check_ack_flag(ack_packet)) { @produce(ERROR_BAD_INPUT); } // ... proceed with tcp session } Such a coroutine would then be usable elsewhere in code like so: var tcp_connections = map<(ip_address, u16), tcp_session>::new(); // ... var incoming_ip_packet = read_ip_packet_from_raw_socket(); var result = tcp_connections[get_dst(incoming_ip_packet)].resume(incoming_ip_packet); if(result == ERROR_BAD_INPUT) { // ... } And the compiler would be able to determine upfront what the size of the tcp_session struct is, so there'd be no need for implicit boxing/heap allocations.
- tekacs 3y ago
  You kinda can do this in Rust at the moment with async, in the sense of: https://docs.rs/async-stream/latest/async_stream/ https://docs.rs/async-stream/latest/async_stream/ So something like this, off the top of my head. It's not the most Rust-native code because I tried to match your code block as closely as possible to make the parallels more obvious. struct TCPHandle { sender: Sender<TCPPacket>, // I'm taking liberties here, you can't use impl like this // so you'd have to engage in some shenanigans which I don't // want to type right now... stream_to_client: impl Stream<Item = Result<TCPPacket, TCPError>>, } fn tcp_session() -> TCPHandle { let (sender, receiver) = channel(); let stream_to_client = stream! { let syn_packet = receiver.recv().await; if (!check_syn_flag(syn_packet)) { yield Err(TCPError::BadInputError); return; } let syn_ack_packet = build_tcp_segment(...); yield Ok(syn_ack_packet); let ack_packet = receiver.recv().await; if (!check_ack_flag(ack_packet)) { yield Err(TCPError::BadInputError); return; } // ... proceed with TCP session } TCPHandle { sender, stream_to_client } } let tcp_connections = HashMap<(Ipv4Addr, u16), TCPHandle>; loop { let incoming_ip_packet = read_ip_packet_from_raw_socket().await; let handle = tcp_connections.get_mut(get_dst(&incoming_ip_packet)); handle.sender.send(incoming_ip_packet).await; let result = handle.stream_to_client.next().await; match result { Ok(packet) => send_ip_packet_to_raw_socket(packet).await, Err(e) => { ... handle errors }, } } I'm curious how something like this would work for your purposes... you could probably wrap the whole thing up a little more and make it behave even more like your example?
  - 10000truths 3y ago
    I'm aware that encapsulated control flow like this exists at a syntactic level, but the implementation deficiencies remain unaddressed: 1. No access to the actual concrete type, as you mentioned, which means you can't do Vec<Stream<...>> - you have to resort to indirections like Vec<Box<dyn Stream<...>>>. You might be able to work around this somehow if you were determined enough (DWARF debug info and std::mem::transmute come to mind), but any such approach would result in an ugly and exceedingly fragile abomination. 2. Brittle, if any, optimization. `stream!` is basically async/await with proc macro syntax dressing, and Rust's async/await implementation still suffers the same sort of bloated frame size issues as generators do, under certain circumstances. To provide some context, suppose I want to write a custom event loop that handles millions of concurrent sessions (could be filesystem transactions, or TCP sessions, or some other I/O). At that level, every kilobyte of per-session data I add equates to additional gigabytes of memory usage. Every byte of space used by the compiler-generated state machine has to be carefully accounted for. I can't afford to have my per-session context blowing up in size because the compiler naively duplicates stack slots for arguments across yield points [1], or because one of my local variable types implements Copy [2], or because the compiler will simply fail to optimize local variables that are never live across a yield point [3]. I suppose if I care so much about memory usage, I could just draw up my own state machine, mentally track the liveness of each state variable and lay out a hand-written size-optimized struct accordingly, but that's going to be a painful exercise and a maintainability nightmare. [1] https://github.com/rust-lang/rust/issues/62958 https://github.com/rust-lang/rust/issues/62958 [2] https://github.com/rust-lang/rust/issues/62952 https://github.com/rust-lang/rust/issues/62952 [3] https://github.com/rust-lang/rust/issues/59087 https://github.com/rust-lang/rust/issues/59087
  - nicoburns 3y ago
    Rust also has actual generators, which are used to implement async/await. They're unstable, but they are accessible from user code in nightly Rust https://doc.rust-lang.org/beta/unstable-book/language-features/generators.html https://doc.rust-lang.org/beta/unstable-book/language-featur...
- di4na 3y ago
  Isn't that basically what typed effect handlers are?

Yes, something like Python generators which have yield and send, but typed, and which compiles down to an efficiently sized struct with a resume function. Here's a crude illustration of what I mean, using pseudo-code describing a TCP session flow: coroutine tcp_session() { var syn_packet = @consume(); if(!check_syn_flag(syn_packet)) { @produce(ERROR_BAD_INPUT); return; } var syn_ack_packet = build_tcp_segment(...); @produce(syn_ack_packet); var ack_packet = @consume(); if(!check_ack_flag(ack_packet)) { @produce(ERROR_BAD_INPUT); } // ... proceed with tcp session } Such a coroutine would then be usable elsewhere in code like so: var tcp_connections = map<(ip_address, u16), tcp_session>::new(); // ... var incoming_ip_packet = read_ip_packet_from_raw_socket(); var result = tcp_connections[get_dst(incoming_ip_packet)].resume(incoming_ip_packet); if(result == ERROR_BAD_INPUT) { // ... } And the compiler would be able to determine upfront what the size of the tcp_session struct is, so there'd be no need for implicit boxing/heap allocations.

tekacs 3y ago
You kinda can do this in Rust at the moment with async, in the sense of: https://docs.rs/async-stream/latest/async_stream/ https://docs.rs/async-stream/latest/async_stream/ So something like this, off the top of my head. It's not the most Rust-native code because I tried to match your code block as closely as possible to make the parallels more obvious. struct TCPHandle { sender: Sender<TCPPacket>, // I'm taking liberties here, you can't use impl like this // so you'd have to engage in some shenanigans which I don't // want to type right now... stream_to_client: impl Stream<Item = Result<TCPPacket, TCPError>>, } fn tcp_session() -> TCPHandle { let (sender, receiver) = channel(); let stream_to_client = stream! { let syn_packet = receiver.recv().await; if (!check_syn_flag(syn_packet)) { yield Err(TCPError::BadInputError); return; } let syn_ack_packet = build_tcp_segment(...); yield Ok(syn_ack_packet); let ack_packet = receiver.recv().await; if (!check_ack_flag(ack_packet)) { yield Err(TCPError::BadInputError); return; } // ... proceed with TCP session } TCPHandle { sender, stream_to_client } } let tcp_connections = HashMap<(Ipv4Addr, u16), TCPHandle>; loop { let incoming_ip_packet = read_ip_packet_from_raw_socket().await; let handle = tcp_connections.get_mut(get_dst(&incoming_ip_packet)); handle.sender.send(incoming_ip_packet).await; let result = handle.stream_to_client.next().await; match result { Ok(packet) => send_ip_packet_to_raw_socket(packet).await, Err(e) => { ... handle errors }, } } I'm curious how something like this would work for your purposes... you could probably wrap the whole thing up a little more and make it behave even more like your example?
- 10000truths 3y ago
  I'm aware that encapsulated control flow like this exists at a syntactic level, but the implementation deficiencies remain unaddressed: 1. No access to the actual concrete type, as you mentioned, which means you can't do Vec<Stream<...>> - you have to resort to indirections like Vec<Box<dyn Stream<...>>>. You might be able to work around this somehow if you were determined enough (DWARF debug info and std::mem::transmute come to mind), but any such approach would result in an ugly and exceedingly fragile abomination. 2. Brittle, if any, optimization. `stream!` is basically async/await with proc macro syntax dressing, and Rust's async/await implementation still suffers the same sort of bloated frame size issues as generators do, under certain circumstances. To provide some context, suppose I want to write a custom event loop that handles millions of concurrent sessions (could be filesystem transactions, or TCP sessions, or some other I/O). At that level, every kilobyte of per-session data I add equates to additional gigabytes of memory usage. Every byte of space used by the compiler-generated state machine has to be carefully accounted for. I can't afford to have my per-session context blowing up in size because the compiler naively duplicates stack slots for arguments across yield points [1], or because one of my local variable types implements Copy [2], or because the compiler will simply fail to optimize local variables that are never live across a yield point [3]. I suppose if I care so much about memory usage, I could just draw up my own state machine, mentally track the liveness of each state variable and lay out a hand-written size-optimized struct accordingly, but that's going to be a painful exercise and a maintainability nightmare. [1] https://github.com/rust-lang/rust/issues/62958 https://github.com/rust-lang/rust/issues/62958 [2] https://github.com/rust-lang/rust/issues/62952 https://github.com/rust-lang/rust/issues/62952 [3] https://github.com/rust-lang/rust/issues/59087 https://github.com/rust-lang/rust/issues/59087
- nicoburns 3y ago
  Rust also has actual generators, which are used to implement async/await. They're unstable, but they are accessible from user code in nightly Rust https://doc.rust-lang.org/beta/unstable-book/language-features/generators.html https://doc.rust-lang.org/beta/unstable-book/language-featur...
di4na 3y ago
Isn't that basically what typed effect handlers are?