4 ms·

Write signatures, not schemas

8 points by catern 5y ago

theamk 5y ago
Counterpoints: - Libraries often CAN'T be written in a separate language without an additional effort -- and often writing with interop in mind means you have to give up nice features like C++ STL containers - Libraries often CAN'T have separate build environment -- want to switch to C++14 but you have a third-party library which is stuck in C++03? Tough. - Libraries often CAN'T isolate failures. A segfault or OOM in library will kill your app. Not so with external process. - For many languages, libraries CAN'T have access to resources that the rest of the program cannot access. Other that few exceptions (browser Javascript, Java with non-default security settings, or exotic execution models), most languages have "escape hatches" which allow underlying OS access, thus bypassing any language-based restrictions there are. - Libraries often CAN'T have effective resource management. Trying to find out how much memory does library use, or changing library IO priority is very hard. (putting all library in a separate thread might help with CPU usage but not with other things) There are definitely times when you want a library -- your "priority queue" class should not be out of process. But for things which can tolerate extra latency / interface limitations, consider making them a separate process.
- catern 5y ago
  Great comment! I've added points about most of these to the post. But just to respond quickly to these directly: - Writing in a single language, not multiple languages which requires you to speak IPC protocols, is the whole point. - Added a note to the article - Added a note to the article - Several of the techniques I listed apply to C - raw, unsafe native code. If they work for C, they work for all languages. - You can change the IO priority of threads, and track their IO usage separately. The question of separating memory usage is, I think, sufficiently addressed by the other points in the article (if you're using SFI, for example, you're already going to separate memory usage)
  - theamk 5y ago
    Re "Writing in a single language, not multiple languages": I think this contradicts your article. Your first paragraph ends with: > You can express far more sophisticated abstractions [...] without losing any functionality. but "support multiple languages" is definitely a functionality and a very important one, too. Did you want to say: "You can express far more sophisticated abstractions with a type signature than you can with a protocol schema, _as long as you are willing to limit yourself to a single language and give up on resource isolation" ---- Re "Software fault isolation", I think you are simply wrong there. The SFI paper you cited is about NaCL sandbox, and with NaCL you are not going to be passing arbitrary structures to the app -- with NaCL programming (and all other call gates), you have manually serialize/deserialize messages -- and once you start writing serdes function, you might as well go full remote protocol to gain support of other languages etc.. And related to this topic, you want to clarify: are we talking about production languages which people write programs on, or are you talking about rewriting existing code in a whole new language in order to get rid of separate processes? Because if my choices are either "put it in a separate process" or "rewrite whole thing in NaCl or Java" I am not going to be choosing the latter option.
    - catern 5y ago
      Sorry, I responded too quickly to this comment: >- Libraries often CAN'T be written in a separate language without an additional effort -- and often writing with interop in mind means you have to give up nice features like C++ STL containers And I said something that wasn't quite right. What I should have said is: Yes, writing a language-independent library can be harder, because you need to use a language-independent type system to describe the signature of the library. But it's harder still to write a standalone process, because protocol schemas are even less capable than language-independent type systems. And no, you don't need to avoid STL or anything. >Re "Software fault isolation", ... you have manually serialize/deserialize messages No you don't. For further reference, also look at the other library privilege-separation techniques I linked. >are we talking about production languages which people write programs on Yes. Such as C.
      - 5y ago

8 points by catern 5y ago

Counterpoints: - Libraries often CAN'T be written in a separate language without an additional effort -- and often writing with interop in mind means you have to give up nice features like C++ STL containers - Libraries often CAN'T have separate build environment -- want to switch to C++14 but you have a third-party library which is stuck in C++03? Tough. - Libraries often CAN'T isolate failures. A segfault or OOM in library will kill your app. Not so with external process. - For many languages, libraries CAN'T have access to resources that the rest of the program cannot access. Other that few exceptions (browser Javascript, Java with non-default security settings, or exotic execution models), most languages have "escape hatches" which allow underlying OS access, thus bypassing any language-based restrictions there are. - Libraries often CAN'T have effective resource management. Trying to find out how much memory does library use, or changing library IO priority is very hard. (putting all library in a separate thread might help with CPU usage but not with other things) There are definitely times when you want a library -- your "priority queue" class should not be out of process. But for things which can tolerate extra latency / interface limitations, consider making them a separate process.

catern 5y ago
Great comment! I've added points about most of these to the post. But just to respond quickly to these directly: - Writing in a single language, not multiple languages which requires you to speak IPC protocols, is the whole point. - Added a note to the article - Added a note to the article - Several of the techniques I listed apply to C - raw, unsafe native code. If they work for C, they work for all languages. - You can change the IO priority of threads, and track their IO usage separately. The question of separating memory usage is, I think, sufficiently addressed by the other points in the article (if you're using SFI, for example, you're already going to separate memory usage)
- theamk 5y ago
  Re "Writing in a single language, not multiple languages": I think this contradicts your article. Your first paragraph ends with: > You can express far more sophisticated abstractions [...] without losing any functionality. but "support multiple languages" is definitely a functionality and a very important one, too. Did you want to say: "You can express far more sophisticated abstractions with a type signature than you can with a protocol schema, _as long as you are willing to limit yourself to a single language and give up on resource isolation" ---- Re "Software fault isolation", I think you are simply wrong there. The SFI paper you cited is about NaCL sandbox, and with NaCL you are not going to be passing arbitrary structures to the app -- with NaCL programming (and all other call gates), you have manually serialize/deserialize messages -- and once you start writing serdes function, you might as well go full remote protocol to gain support of other languages etc.. And related to this topic, you want to clarify: are we talking about production languages which people write programs on, or are you talking about rewriting existing code in a whole new language in order to get rid of separate processes? Because if my choices are either "put it in a separate process" or "rewrite whole thing in NaCl or Java" I am not going to be choosing the latter option.
  - catern 5y ago
    Sorry, I responded too quickly to this comment: >- Libraries often CAN'T be written in a separate language without an additional effort -- and often writing with interop in mind means you have to give up nice features like C++ STL containers And I said something that wasn't quite right. What I should have said is: Yes, writing a language-independent library can be harder, because you need to use a language-independent type system to describe the signature of the library. But it's harder still to write a standalone process, because protocol schemas are even less capable than language-independent type systems. And no, you don't need to avoid STL or anything. >Re "Software fault isolation", ... you have manually serialize/deserialize messages No you don't. For further reference, also look at the other library privilege-separation techniques I linked. >are we talking about production languages which people write programs on Yes. Such as C.
    - 5y ago