6 ms·

IMO a stronger and more flexible concept is a stream. A stream takes a buffer reader/writer of some kind (mem, socket, file, etc..) and does the usual I/O & enc

IMO a stronger and more flexible concept is a stream. A stream takes a buffer reader/writer of some kind (mem, socket, file, etc..) and does the usual I/O & encoding/decoding. It's similar to read()/write() vs fread()/fwrite() in stdio.

saagarjha 8y ago
But this doesn’t seem to solve the issue the author had to provide typed access to remote memory.
- halayli 8y ago
  How so? If the stream's buf is HTTP and you do a seek(x); read(y); that can be converted to a an HTTP POST + Range: x-y; and the stream can decode/cast to the type required.
  - saagarjha 8y ago
    This doesn't really fit the random-access primitives that remote memory provides, and you still don't really get type safety.
xiii1408 8y ago
For parallel programs, streams are problematic because they enforce a strict order to communication. (I assume you're talking about creating distributed istream/ostream classes.) Also, stream communication is explicitly two-sided; if you want to pull in a value on an istream, there needs to be a corresponding process on the other side inserting something into an ostream. I've found pointers to be a much better abstraction for data structures, which often require irregular access that two-sided communication can't satisfy easily. Streams also have the problem that, while the underlying interface (a socket, file, etc.) is untyped, the interface is typed. This leads to the unfortunate situation where you can attempt to pull an std::string in on an istream and get junk because you're using the wrong type. This does depend a lot on the kinds of programs you're writing and what your assumptions are--in the HPC world, we don't generally care about fault tolerance, and we actually have hardware that gives us fast RDMA read/write. In a distributed systems setting where processes are communicating with messages and need to be fault tolerant and reach consensus using algorithms like Paxos or something, I could see abstractions like streams being really useful.
- shakna 8y ago
  > Also, stream communication is explicitly two-sided; if you want to pull in a value on an istream, there needs to be a corresponding process on the other side inserting something into an ostream. That's not always the case is it, though? stdin on a standard computer for one, and in embedded world you might have an ostream to speak UART and give yourself a nice printing function, but you probably won't have a corresponding istream. So why can't this be similarly exceptional?
  - xiii1408 8y ago
    That's true--you can add the ability to seek, in which case the stream becomes like a file. Whether or not that's desirable depends on what kind of programs you're writing. For programs like databases that traditionally store things using the filesystem, that's probably a very useful abstraction, and some people at VMware have actually implemented an "RDMA over the filesystem" abstraction, which is super cool: https://256fd102-a-62cb3a1a-s-sites.googlegroups.com/site/mkaguilera/remote-regions-atc2018.pdf https://256fd102-a-62cb3a1a-s-sites.googlegroups.com/site/mk.... If you're looking for an abstraction with which to directly implement programs that use data structures, however, I still think pointers are much more valuable than files because files are not strongly typed, and it's easy to read junk due to type errors and lack of structure.

saagarjha 8y ago

But this doesn’t seem to solve the issue the author had to provide typed access to remote memory.

halayli 8y ago
How so? If the stream's buf is HTTP and you do a seek(x); read(y); that can be converted to a an HTTP POST + Range: x-y; and the stream can decode/cast to the type required.
- saagarjha 8y ago
  This doesn't really fit the random-access primitives that remote memory provides, and you still don't really get type safety.

xiii1408 8y ago

For parallel programs, streams are problematic because they enforce a strict order to communication. (I assume you're talking about creating distributed istream/ostream classes.) Also, stream communication is explicitly two-sided; if you want to pull in a value on an istream, there needs to be a corresponding process on the other side inserting something into an ostream. I've found pointers to be a much better abstraction for data structures, which often require irregular access that two-sided communication can't satisfy easily. Streams also have the problem that, while the underlying interface (a socket, file, etc.) is untyped, the interface is typed. This leads to the unfortunate situation where you can attempt to pull an std::string in on an istream and get junk because you're using the wrong type. This does depend a lot on the kinds of programs you're writing and what your assumptions are--in the HPC world, we don't generally care about fault tolerance, and we actually have hardware that gives us fast RDMA read/write. In a distributed systems setting where processes are communicating with messages and need to be fault tolerant and reach consensus using algorithms like Paxos or something, I could see abstractions like streams being really useful.

shakna 8y ago
> Also, stream communication is explicitly two-sided; if you want to pull in a value on an istream, there needs to be a corresponding process on the other side inserting something into an ostream. That's not always the case is it, though? stdin on a standard computer for one, and in embedded world you might have an ostream to speak UART and give yourself a nice printing function, but you probably won't have a corresponding istream. So why can't this be similarly exceptional?
- xiii1408 8y ago
  That's true--you can add the ability to seek, in which case the stream becomes like a file. Whether or not that's desirable depends on what kind of programs you're writing. For programs like databases that traditionally store things using the filesystem, that's probably a very useful abstraction, and some people at VMware have actually implemented an "RDMA over the filesystem" abstraction, which is super cool: https://256fd102-a-62cb3a1a-s-sites.googlegroups.com/site/mkaguilera/remote-regions-atc2018.pdf https://256fd102-a-62cb3a1a-s-sites.googlegroups.com/site/mk.... If you're looking for an abstraction with which to directly implement programs that use data structures, however, I still think pointers are much more valuable than files because files are not strongly typed, and it's easy to read junk due to type errors and lack of structure.