4 ms·

Very interesting question! I don't think there is any language/environment that does that natively. What you seem to be asking for is an abstraction of proces

by btrask 6y ago

Very interesting question! I don't think there is any language/environment that does that natively. What you seem to be asking for is an abstraction of processes. However the process itself is ultimately an abstraction that is made-up by operating systems. For example, let's say you call getpid(2) in your program. Then you do some "async offline" work, during which the process exits and is restarted. Now you have a new PID, which is not what you expected. That means you need to introduce "virtual PIDs" that don't change across async calls. Before long you've reimplemented all of libc (or equivalent). Golang has had a lot of problems just making green (non-OS) threads work correctly. Any time you stray from the OS, you will have problems interfacing with it. As an OS feature, I can imagine a kernel that aggressively pages out programs while they're blocking, etc. Before long you get into philosophical questions like "what does it mean for a process to be running?" Usually it's something that the OS defines, and we leave it at that. But you don't have to, if you don't want to.

llimos 6y ago
An easier solution would be to add getpid to the list of things the compiler won't allow to be on both sides of the await offline, along with file handles and open sockets. Perhaps one approach would be to have the compiler split the program into separate programs, which pass each other command lines to run when they are complete (a bit like Javascript callback arguments in the old days). Then the OS doesn't need to know anything, by the time it runs they're all genuinely separate processes, but the developer can still write the code in that expressive way. (Going back to Javascript, a bit like how Babel rewrote async/await before the browsers supported it.)
- btrask 6y ago
  It's not just getpid. It's everything that interacts with processes. fork(2), wait(2), kill(1/2)... Any external management tools (supervisors, debuggers, etc)... Imagine trying to administer such a program, where even tools like top(1) couldn't track it over time. Here's one that I think you'll find compelling: threads. What do you do if you have multiple background threads running when one thread wants to exit the process? You could just exit the thread, but I think that would defeat the point. Imagine you're using a language, and you find this feature in the docs. Then it lists all the caveats and things you have to give up. Would you still use it, or would you make do without it? Maybe I haven't convinced you but I hope you can agree it's a major tradeoff. I think that explains why a general purpose language hasn't tried to make it a standard feature.
  - llimos 6y ago
    You have at least convinced me that anyone using it would need to know very well what they are doing and what is going to happen behind the scenes. And that it's better suited to the higher-level languages (node, ruby family) rather than those that give you more fine-grained control over processes. You've also convinced me that it's better as syntactic sugar for splitting into true separate processes behind the scenes, rather than a first-class language construct. This would solve many of the issues you raised. But I do still think there's value in such a concept. Perhaps it could be a good fit for serverless functions. You write it as one function, but it happens as two, and the developer doesn't have to worry about the glue code. In terms of jumping to the right place in the code, several languages have the concept of continuations, it doesn't seem like a huge leap to something like this.
    - btrask 6y ago
      > Perhaps it could be a good fit for serverless functions. I was thinking of that too. :) If you're interested in the technical mechanism, it's possible to load a core dump and resume execution. I found https://web.archive.org/web/20091026234235/http://geocities.com/asimshankar/checkpointing/ https://web.archive.org/web/20091026234235/http://geocities.... which explains the (surprisingly complex) process of resuming from a core dump. Also see https://criu.org/ https://criu.org/ and https://checkpointing.org/ https://checkpointing.org/
      - llimos 6y ago
        Cool, very interesting! That just convinces me even more that it's better to leave the OS out of it and let each language's runtime figure out its own way of serializing its state and restoring it for a second invocation.

by btrask 6y ago

llimos 6y ago

An easier solution would be to add getpid to the list of things the compiler won't allow to be on both sides of the await offline, along with file handles and open sockets. Perhaps one approach would be to have the compiler split the program into separate programs, which pass each other command lines to run when they are complete (a bit like Javascript callback arguments in the old days). Then the OS doesn't need to know anything, by the time it runs they're all genuinely separate processes, but the developer can still write the code in that expressive way. (Going back to Javascript, a bit like how Babel rewrote async/await before the browsers supported it.)

btrask 6y ago
It's not just getpid. It's everything that interacts with processes. fork(2), wait(2), kill(1/2)... Any external management tools (supervisors, debuggers, etc)... Imagine trying to administer such a program, where even tools like top(1) couldn't track it over time. Here's one that I think you'll find compelling: threads. What do you do if you have multiple background threads running when one thread wants to exit the process? You could just exit the thread, but I think that would defeat the point. Imagine you're using a language, and you find this feature in the docs. Then it lists all the caveats and things you have to give up. Would you still use it, or would you make do without it? Maybe I haven't convinced you but I hope you can agree it's a major tradeoff. I think that explains why a general purpose language hasn't tried to make it a standard feature.
- llimos 6y ago
  You have at least convinced me that anyone using it would need to know very well what they are doing and what is going to happen behind the scenes. And that it's better suited to the higher-level languages (node, ruby family) rather than those that give you more fine-grained control over processes. You've also convinced me that it's better as syntactic sugar for splitting into true separate processes behind the scenes, rather than a first-class language construct. This would solve many of the issues you raised. But I do still think there's value in such a concept. Perhaps it could be a good fit for serverless functions. You write it as one function, but it happens as two, and the developer doesn't have to worry about the glue code. In terms of jumping to the right place in the code, several languages have the concept of continuations, it doesn't seem like a huge leap to something like this.
  - btrask 6y ago
    > Perhaps it could be a good fit for serverless functions. I was thinking of that too. :) If you're interested in the technical mechanism, it's possible to load a core dump and resume execution. I found https://web.archive.org/web/20091026234235/http://geocities.com/asimshankar/checkpointing/ https://web.archive.org/web/20091026234235/http://geocities.... which explains the (surprisingly complex) process of resuming from a core dump. Also see https://criu.org/ https://criu.org/ and https://checkpointing.org/ https://checkpointing.org/
    - llimos 6y ago
      Cool, very interesting! That just convinces me even more that it's better to leave the OS out of it and let each language's runtime figure out its own way of serializing its state and restoring it for a second invocation.