12 ms·

A Badass Way to Connect Programs Together

351 points by norswap 11y ago

jessaustin 11y ago
Title probably should be edited to remove "Joe Armstrong: ". He might be badass, but he isn't a way to connect programs together.
- dmix 11y ago
  I read the article because it said Joe Armstrong. That way I knew it was someone who knows what he's talking about. Especially given the topic.
  - auvi 11y ago
    Same here. If I have not seen "Joe Armstrong" in the title I wouldn't have clicked on it.
    - deleted 11y ago
      [deleted]
  - mooreds 11y ago
    Just curious, wasn't the hostname a clue?
  - recurser 11y ago
    It was a play on words with the original title, reading ':' as 'is' > Joe Armstring: A Badass Way to Connect Programs Together
- lostcolony 11y ago
  He isn't a way to connect programs together, but he -is- a generator for connecting programs together. You just have to show him two programs and a reason to connect them, wait a bit, and there'll be a github repo. I hope I'm that inquisitive and playfully experimental when I reach his age.
- dwc 11y ago
  How sure are you about that?
ChuckMcM 11y ago
Ah OSC, my first encounter with it was with the Make Controller[1] (basically an ARM chip with an OSC service endpoint). OSC always struck me as one of those protocols that did something really useful in the wrong way. It just "feels" awkward to me for some reason and I never quite put my finger on it. That said it was great fun building a robot with the controller but hard to get asynchronous feedback. If I were doing it again I'd try to figure out a full duplex OSC channel for something like that. [1] http://makezine.com/2006/05/11/make-controller-kit/ http://makezine.com/2006/05/11/make-controller-kit/
cyphar 11y ago
OP hasn't addressed the fact that UDP will willfully drop packets and send them out of order. What if I need to stream some data? What if I care about the order my IPC messages arrive in?
- deleted 11y ago
  [deleted]
- notthetup 11y ago
  The article doesn't mention OSC Bundles, which is a group of OSC messages which are designed to be 'interpreted' together. A bundle usually has a timestamp associated with it. I agree this might not be the best method of dealing with out of order reception of packages, but it does exist in the protocol.
  - crucialfelix 11y ago
    a bundle always has a timestamp
- brazzledazzle 11y ago
  Then UDP won't work for your requirements, right?
  - tyingq 11y ago
    UDP, without some layer on top of it for ack or retry, or ordering, isn't useful for many requirements. Maybe things like displaying stock quotes, where stale data doesn't matter after a few seconds? I think the point is just to say that the article shouldn't tout plain UDP as a fix for TCP being hard in certain languages without calling out the downside.
    - klodolph 11y ago
      TCP and UDP are different solutions for different problems. You also shouldn't tout TCP as a fix for UDP being hard without calling out the downside--TCP doesn't let you drop packets and can suffer from bad latency problems.
dmix 11y ago
> This is wonderful news for systems like Node.js whose concurrency model is non-existent and whose idea of having a fun time is throwing promises into the future and hoping that nobody breaks them. Oh so true. I wish I learned Erlang (and to a lesser extent Go) much earlier in my career after wasting so much time with Node. I now laugh when I see people on Twitter/Reddit talk about Node's concurrency as a reason to learn it over other languages.
- sotojuan 11y ago
  I started programming with Node a year ago and currently diving into Erlang/Elixir... I am hopeful/afraid that I'll reach the same conclusion!
- toast0 11y ago
  To be fair, Erlang escaped from the lab in 1988, and only supported SMP in 2006. Node.js was released in 2009, it's only had 6 years, maybe it's still a bit early for SMP?
  - btilly 11y ago
    You can use SMP just fine in Node.js today with https://nodejs.org/api/cluster.html https://nodejs.org/api/cluster.html. The problem with Node.js and concurrency is that everything depends on trusting code to be perfectly written, and that perfectly written code must be written in a naturally confusing callback style that is really easy to screw up. If you do it perfectly, then you get great scalability. But one bonehead mistake will ruin your concurrency. By contrast with a pre-emptive model you get decent scalability really easily, but now you've got a million and one possible race conditions that are hard to reason about. This is not a new design tradeoff. Go back to the days of Windows 3.1 or the old MacOS versions below OS X. They all used cooperative multi-tasking, just like Node.js. Today what do we have? Pre-emptive multi-tasking in Windows, OS X, *nix and iOS. Web development has actually gone back to a model that operating systems abandoned long ago. As long as your app is small, there is a chance that it will work. But as your app grows? Good luck! (That is why operating systems uniformly wound up choosing hard to debug race conditions over predictably impossible to solve latency problems from cooperative multi-tasking.)
watmough 11y ago
Wow, odd that this comes up as I was casting around in another domain, but my working assumption is that I would implement my GPU processing library using a 'tags data' style binary protocol. However, I was thinking of it more from a Amiga COPPER instruction list perspective, but it amounts to the same thing. The really cool thing is that approaches like this tend to be language-agnostic, easy to program to, and extremely fast. All points raised in the article.
- vidarh 11y ago
  Tag lists more generally appears all over AmigaOS as well - especially as they started running into limitations with the existing APIs that depended heavily on fixed structs with AmigaOS 2.0, they more and more opted for adding taglist parameters to avoid running into the same situation again. And while it's request-response based over message ports, personally I see AREXX ports in most larger Amiga applications as a good example of the overall approach of cooperating processes to build larger systems. Dbus and similar systems today has lots of great functionality, but what they miss is what ensured AREXX was "everywhere": It was trivially simple to support. Pointing to the simplicity of OSC in the article really resonated with me, because a lot of these systems are far less useful than they could be because it's extra work. Simple systems on the other hand end up shaping application structure: AmigaOS was massively message based from the outset: Message ports and messages were "built in" and used all over the place. E.g. when communicating with filesystems or drivers or the GUI you do it via messages. Then came AREXX and there was suddenly a standard message format for RPC - both application-to-application and user-to-application. If Amiga-applications were message oriented before, a lot of "post-AREXX" Amiga applications took it to the next level by being shaped around a dispatch loop that included not only e.g. GUI updates, but defining internal APIs that exposed pretty much every higher level action in the application, and used that as an explicit implementation strategy.
dayjah 11y ago
The basis of the protocol is similar to that of protocol buffers [1]. Though the tag field containing the map/list of segments in data is pretty novel IME. [1] https://developers.google.com/protocol-buffers/docs/encoding https://developers.google.com/protocol-buffers/docs/encoding
- codemac 11y ago
  Message Pack[0] is another protocol that follows this very successful simplicity. A leading thought in a lot of this is the "unset and a zero value are equivalent". It simplifies a lot of what the serializer and deserializers have to understand, and places the burden on code generation or libraries rather than complex serialization. It was unnatural for me to think this way, as I take the opposite for memory (record whether something is set or not, don't just assume it equals zero), but on the wire this makes for very compact and backwards compatible messaging. Cap'n Proto[1] I have not played with, but would love to learn more about to see the advantages and disadvantages between protobuf, msgpack, and capn. [0]: http://msgpack.org/ http://msgpack.org/ [1]: https://capnproto.org/ https://capnproto.org/
  - cypher543 11y ago
    For a standardized version of Message Pack, see CBOR (RFC 7049): http://cbor.io/ http://cbor.io/
    - codemac 11y ago
      What makes it a standardized version of Message Pack? Message Pack has a specification that is well documented[0] and even the rfc you refer to specifically states it has different goals (though, eliding specifics with pretty sad generalizations)[1] [0]: https://github.com/msgpack/msgpack/blob/master/spec.md https://github.com/msgpack/msgpack/blob/master/spec.md [1]: https://tools.ietf.org/html/rfc7049#section-9 https://tools.ietf.org/html/rfc7049#section-9
      - spc476 11y ago
gourneau 11y ago
The best thing about OSC IMO are the many wonderful configurable controllers. When I was messing with OSC a few years ago TouchOSC http://hexler.net/software/touchosc http://hexler.net/software/touchosc was my favorite It is really fun to use these beautiful Star Trek looking interfaces to make your robots move around for example ;)
- thirdsun 11y ago
  TouchOSC has been some kind of model app for me - was available on the iPad very early and is still going strong to this day while countless other controllers came and went.
steeve 11y ago
Reminds me a lot of bencoding, which is not as popular as should (outside of bittorrent of course)
- nyan4 11y ago
  At least bencoding has dictionaries.
- klodolph 11y ago
  It's unfortunate that bencoding doesn't support text strings.
yarrel 11y ago
| ?
vshan 11y ago
Woah, funny I see this. I've been working on a distributed shell, basically trying to combine sockets and pipes to create remote pipes, aka inter-host piping. Something like: $ echo foo | host2:grep bar | host3:grep baz
- RickHull 11y ago
  > basically trying to combine sockets and pipes to create remote pipes Sockets, or ssh? To get a remote shell, you're going to want ssh.
paperkettle 11y ago
Major supporter of this thinking, and did several sketches in this direction a few years ago: http://www.illucia.com/ http://www.illucia.com/ Made of: 1) a suite of interconnectable videogames & apps and 2) physical patchbays to connect them Came from thinking about software like modules in a modular synth. Playgrounds of interconnectable control structures. Was patching games into samplers, slaving text editors to drum machines, etc. related writeups: http://www.illucia.com/faq/ http://www.illucia.com/faq/ another from 2010 or so when I was just starting coding (so a lot of it feels silly now.. but still some related nuggets): http://www.paperkettle.com/codebending/ http://www.paperkettle.com/codebending/
- detaro 11y ago
  I stumbled over your projects a few years back. Awesome work!
- lbotos 11y ago
  Side Q: can I ask what typeface you used for illucia?
  - hjek 11y ago
    It's 'Square Serif Medium' (Look at the css, or do Right Click => Inspect Element in your browser)
  - vijaykiran 11y ago
    Side A: I use http://chengyinliu.com/whatfont.html http://chengyinliu.com/whatfont.html to quickly check the fonts on a page with Safari, other browsers may have similar tools too.
- mej10 11y ago
  That is really cool!
ggreer 11y ago
I think OSC is interesting, but Armstrong goes too far in his disparagement of XML and JSON: > I think it is totally crazy to send JSON or XML “over the air” since this will degrade the performance of the applications giving a bad user experience and higher bills - since ultimately we pay for every bit of data. If every JSON and XML API switched to OSC, the difference in data usage would scarcely be measurable. First: The vast majority of mobile data is video, audio, images, and application updates. Second: When sent over the wire, JSON and XML are typically compressed with gzip or deflate. And let's not forget: JSON and XML have some significant advantages over most other serialization formats. They're ubiquitous. Practically every programming language has libraries to parse and generate them. Practically every programmer knows how to work with them. They're human-readable. No special programs are needed to view their contents. In other words, JSON and XML are only "wasteful" if you don't count developer time. Once you do, it suddenly makes a lot of sense to "waste" hardware resources. After all, programmers are expensive. Hardware is cheap.
- colordrops 11y ago
  > After all, programmers are expensive. Hardware is cheap. Except in China. I worked at a US company that did almost zero optimization on their code, deploying to 5000 AWS instances with a team of about six. In China, bandwidth closely followed by CPU were the cost centers, and the Chinese team of 30 engineers optimized the same code to run on around 50 (admittedly beefy) servers.
  - lostcolony 11y ago
    Doesn't invalidate the point; another EC2 instance is a few dozen bucks a month to run full time; another developer, even overseas, is going to cost you more, at just 40 hours a week. But that said, it's rather a good thing to point out for the OP, as Joe Armstrong himself tends to value programmer time over a machine's, and that micro-optimizations are rarely ever worth it (as opposed to yours, which sounds like there were huge, glaring algorithmic issues to fix with macro optimizations, complete replacements of algorithms rather than tightening up code, careful profiling and tweaking to get a 30% speedup in one critical section, etc)
jph 11y ago
> nobody does this properly for purely local applications. You may be interested in qmail and its sub-applications, which does it properly IMHO. It's a great example of the article's key point, which is the value of simple messaging.
bjacks 11y ago
Can anyone explain what a sequential programming language is? Context here: "I guess I’d underestimated the difficulty of implementing Anything-over-TCP in a sequential language. Just because it’s really really easy in Erlang doesn’t mean to say it’s easy in sequential languages."
- Gracana 11y ago
  Imperative programming. Lists of statements, vs Erlang's declarative programming, where you describe the program without specifying control flow.
  - white-flame 11y ago
    Erlang is absolutely a fixed in-order control flow programming language, on message receipt. It is (generally) referentially transparent, but not declarative.
    - Gracana 11y ago
      Oh, I don't know much about Erlang so I thought for sure that's what he meant. What was that supposed to mean, then?
- xrstf 11y ago
  Given that he is contrasting to Erlang, maybe he's talking about imperative languages instead of (pure) functional ones?
- perishabledave 11y ago
  I think he's referring to how Erlang uses the Actor Model. An Erlang program consists of (possibly) thousands of individual lightweight-processes. In this model a typical task may be spread out over many asynchronous processes. You can think of it as a program thats made up of thousands of micro-services. Instead of a task taking place over a sequential piece of code, it takes places over many individual processes that coordinate by communicating with each other.
- 11y ago
ww520 11y ago
I don't understand his emphasis on session on TCP and UDP. A session requires nothing more than a "unique" number to identify it. You can certainly have session in UDP; just attach a session number on all the messages belonging to the session. And you can certainly be sessionless in TCP; just close the connection after each message. HTTP request has no session by definition and it sits on top of TCP.
- rdtsc 11y ago
  > just close the connection after each message. That is if you don't expect to handle more than one connection at a time in a reasonable way. You can certainly do this loop: server_socket = create_and_listen_on_socket(...) while True: csock = server_sock.accept() csock.send(process(csock.recv())) csock.close() TCP setup and teardown is not too cheap (especially compared to sending UDP packets).
  - ww520 11y ago
    UDP is certainly faster than TCP, no question about it, because of different levels of quality of service. However, coding for both is similarly simple, given the same requirement for short receive-only message. TCP session has nothing to do with the complexity. The complexity comes in how to do process() in parallel. A receive-only message can be handled in TCP: server_socket = create_and_listen_on_socket(..., MAXCONN) while True: csock = server_sock.accept() msg = csock.recv() csock.close() process(msg) A UDP loop has similar steps, minus the accept and close. The complexity to use threads or async to run process() is the same for UDP and TCP. For a normal data volume remote command protocol, the simple TCP loop is more than adequate. Just set a reasonable MAXCONN to queue up client connection requests, which can drop connections if there are too many requests, just like UDP dropping packets. Edit: I don't believe lock can be avoided in Sonic Pi server when handling concurrent incoming commands, whether it's written in Erlang or not. Sonic Pi I believe has a single audio device, which makes it a shared resource. Concurrent access to a shared resource has to be managed with lock somewhere along the call path. In that case a single thread TCP server is perfectly fine, serving as a lock as well.
jacobolus 11y ago
So what happens if you want to transmit some data in a nested hierarchical format? Just pack it into some (ad-hoc?) higher level protocol inside one of the flat pieces? Seems like a recipe for inevitable incompatibilities when two languages no longer have codecs for the new underspecified layered format.
- wallacoloo 11y ago
  Although the article may not have made it clear, OSC is designed primarily to pass around control signals, rather than arbitrary, non-contextualized data. Control signals have inherent meaning, like "fast-forward <n> seconds", "set color to <x>", etc. The important distinction is that "set color to <x>" (a command) is distinct from "{ 'color': 'x'}" (non-contextualized data). The layout of controls is hierarchical in the protocol; each control has an associated endpoint, which looks very much like a unix file path. For example, to unpause an audio player, you would send the message "/player/is_playing/set t" (or f, to pause). If you wanted to control the bands of a 10-band equalizer, you could implement that as "/player/eq/band/4/amp/set 0.5" (there are certainly other ways to approach it too). If you want to send control signals to multiple parts of the audio player at the same time, you could use a "bundle" composed of a separate message for each parameter you're altering. All the messages will then be handled simultaneously. Because of this, data does not need to be hierarchical. If you really want to, you can consider the endpoints to be "keys" and the control data to be "values" in a hierachical key -> value map, just to prove that everything you could do with hierarchical data is still possible using OSC. But I would encourage you to not think of it in that abstraction because the fact that OSC is built to pass messages, and not non-contextualized data is a large part of what differentiates it from other protocols like json streaming.
- crucialfelix 11y ago
  Its a low level protocol. There are blob and string types, so you can pack and send anything you like. I send JSON via OSC all the time.
nosuchthing 11y ago
OSC is in use for a lot of DIY hardware interfaces fed into custom modular (software) synths, DSP, or weird art programs using Pure Data, Max/MSP, Super Collider, Ableton Live and MIDI bridges to other pieces of sound plugins or external music hardware. Nice to see a good write up on this project!! http://en.flossmanuals.net/pure-data/ch065_osc/ http://en.flossmanuals.net/pure-data/ch065_osc/ https://puredata.info/community/pdwiki/OpenSoundControl https://puredata.info/community/pdwiki/OpenSoundControl http://livecontrol.q3f.org/ableton-liveapi/liveosc/ http://livecontrol.q3f.org/ableton-liveapi/liveosc/
deleted 11y ago
[deleted]
echochar 11y ago
Is there any similarity between "tag-length-value" and netstrings (1997)?
gtirloni 11y ago
gRPC seems like a good alternative.
dschiptsov 11y ago
My 5¢ is that 9P2000 protocol would be much better and canonical example. Also I hope some day one would show how to implement most of functionality of Hadoop orders of magnitude more effeciently in bunch of Plan9 shell scripts. Or in Erlang with 9P protocol driver.
yomritoyj 11y ago
From the title I thought that the ultimate monad tutorial had finally arrived.
chrisdew 11y ago
The problem with flat messages is that optional composite types can be partially present.
qwertyuiop924 11y ago
Binary protocols are only cheap if your time is free. If you can't connect to your service via telnet, you're going to spend a lot more time debugging. The only way to fix this is to develop a textual representation of your format, and write your own telnet, which is feasible, but nobody seems to have done that to OSC. Also, I find Armstrong's fanaticism grating, and am inherently distrustful of anybody who claims that their community and language has all the good ideas. Because odds are, they're wrong.
- 59nadir 11y ago
  > Also, I find Armstrong's fanaticism grating, and am inherently distrustful of anybody who claims that their community and language has all the good ideas. Because odds are, they're wrong. You might find this[0] blog post by Armstrong interesting, wherein he comments on some things that were/are done wrong in Erlang. I would also argue that this 'fanaticism' is made to look more severe as you can find at least 4 different talks where Armstrong argues for the same things over and over. In any case, Armstrong is actually very humble, which I think seems fairly obvious if you actually watch a few talks and read more than a couple of blog posts by him. Could you point to something specific where you feel Armstrong misrepresents a strong side of Erlang, or is it just a general thing? (Edit: Just to add, I think it should be said that the Erlang team created their favorite language and that it probably represents the most useful language in use today to them, so I don't think it's surprising that they'd see more good ideas there than in any other language. On top of that Armstrong is a very straight shooter, it seems, so I think this might be something to take into account.) 0 - http://joearms.github.io/2013/05/31/a-week-with-elixir.html http://joearms.github.io/2013/05/31/a-week-with-elixir.html
  - qwertyuiop924 11y ago
    It isn't so much that Joe doesn't think that erlang did anything wrong, it just feels like he believes that erlang's way of doing things, and erlang's ideas are the only good way of doing things, and are the only right way of doing things. This makes me suspicious, because that is almost never true of any language or community.
AndrewUnmuted 11y ago
For an especially bad-ass implementation of OSC, check out the SuperCollider project at https://supercollider.github.io/ https://supercollider.github.io/. SuperCollider is a programming language for audio synthesis/analysis, and algorithmic music composition. It works using a client-server paradigm, and they communicate with each other via OSC. Shameless plug time! I've been using SuperCollider to design a machine-learning solo electroacoustic music performance platform, called Sonic Multiplicities. You can hear it at http://www.sonicmultiplicities.net/ http://www.sonicmultiplicities.net/.

351 points by norswap 11y ago

Title probably should be edited to remove "Joe Armstrong: ". He might be badass, but he isn't a way to connect programs together.

dmix 11y ago
I read the article because it said Joe Armstrong. That way I knew it was someone who knows what he's talking about. Especially given the topic.
- auvi 11y ago
  Same here. If I have not seen "Joe Armstrong" in the title I wouldn't have clicked on it.
  - deleted 11y ago
    [deleted]
- mooreds 11y ago
  Just curious, wasn't the hostname a clue?
- recurser 11y ago
  It was a play on words with the original title, reading ':' as 'is' > Joe Armstring: A Badass Way to Connect Programs Together
lostcolony 11y ago
He isn't a way to connect programs together, but he -is- a generator for connecting programs together. You just have to show him two programs and a reason to connect them, wait a bit, and there'll be a github repo. I hope I'm that inquisitive and playfully experimental when I reach his age.
dwc 11y ago
How sure are you about that?

ChuckMcM 11y ago

Ah OSC, my first encounter with it was with the Make Controller[1] (basically an ARM chip with an OSC service endpoint). OSC always struck me as one of those protocols that did something really useful in the wrong way. It just "feels" awkward to me for some reason and I never quite put my finger on it. That said it was great fun building a robot with the controller but hard to get asynchronous feedback. If I were doing it again I'd try to figure out a full duplex OSC channel for something like that. [1] http://makezine.com/2006/05/11/make-controller-kit/ http://makezine.com/2006/05/11/make-controller-kit/

cyphar 11y ago

OP hasn't addressed the fact that UDP will willfully drop packets and send them out of order. What if I need to stream some data? What if I care about the order my IPC messages arrive in?

deleted 11y ago
[deleted]
notthetup 11y ago
The article doesn't mention OSC Bundles, which is a group of OSC messages which are designed to be 'interpreted' together. A bundle usually has a timestamp associated with it. I agree this might not be the best method of dealing with out of order reception of packages, but it does exist in the protocol.
- crucialfelix 11y ago
  a bundle always has a timestamp
brazzledazzle 11y ago
Then UDP won't work for your requirements, right?
- tyingq 11y ago
  UDP, without some layer on top of it for ack or retry, or ordering, isn't useful for many requirements. Maybe things like displaying stock quotes, where stale data doesn't matter after a few seconds? I think the point is just to say that the article shouldn't tout plain UDP as a fix for TCP being hard in certain languages without calling out the downside.
  - klodolph 11y ago
    TCP and UDP are different solutions for different problems. You also shouldn't tout TCP as a fix for UDP being hard without calling out the downside--TCP doesn't let you drop packets and can suffer from bad latency problems.

dmix 11y ago

> This is wonderful news for systems like Node.js whose concurrency model is non-existent and whose idea of having a fun time is throwing promises into the future and hoping that nobody breaks them. Oh so true. I wish I learned Erlang (and to a lesser extent Go) much earlier in my career after wasting so much time with Node. I now laugh when I see people on Twitter/Reddit talk about Node's concurrency as a reason to learn it over other languages.

sotojuan 11y ago
I started programming with Node a year ago and currently diving into Erlang/Elixir... I am hopeful/afraid that I'll reach the same conclusion!
toast0 11y ago
To be fair, Erlang escaped from the lab in 1988, and only supported SMP in 2006. Node.js was released in 2009, it's only had 6 years, maybe it's still a bit early for SMP?
- btilly 11y ago
  You can use SMP just fine in Node.js today with https://nodejs.org/api/cluster.html https://nodejs.org/api/cluster.html. The problem with Node.js and concurrency is that everything depends on trusting code to be perfectly written, and that perfectly written code must be written in a naturally confusing callback style that is really easy to screw up. If you do it perfectly, then you get great scalability. But one bonehead mistake will ruin your concurrency. By contrast with a pre-emptive model you get decent scalability really easily, but now you've got a million and one possible race conditions that are hard to reason about. This is not a new design tradeoff. Go back to the days of Windows 3.1 or the old MacOS versions below OS X. They all used cooperative multi-tasking, just like Node.js. Today what do we have? Pre-emptive multi-tasking in Windows, OS X, *nix and iOS. Web development has actually gone back to a model that operating systems abandoned long ago. As long as your app is small, there is a chance that it will work. But as your app grows? Good luck! (That is why operating systems uniformly wound up choosing hard to debug race conditions over predictably impossible to solve latency problems from cooperative multi-tasking.)

watmough 11y ago

Wow, odd that this comes up as I was casting around in another domain, but my working assumption is that I would implement my GPU processing library using a 'tags data' style binary protocol. However, I was thinking of it more from a Amiga COPPER instruction list perspective, but it amounts to the same thing. The really cool thing is that approaches like this tend to be language-agnostic, easy to program to, and extremely fast. All points raised in the article.

vidarh 11y ago
Tag lists more generally appears all over AmigaOS as well - especially as they started running into limitations with the existing APIs that depended heavily on fixed structs with AmigaOS 2.0, they more and more opted for adding taglist parameters to avoid running into the same situation again. And while it's request-response based over message ports, personally I see AREXX ports in most larger Amiga applications as a good example of the overall approach of cooperating processes to build larger systems. Dbus and similar systems today has lots of great functionality, but what they miss is what ensured AREXX was "everywhere": It was trivially simple to support. Pointing to the simplicity of OSC in the article really resonated with me, because a lot of these systems are far less useful than they could be because it's extra work. Simple systems on the other hand end up shaping application structure: AmigaOS was massively message based from the outset: Message ports and messages were "built in" and used all over the place. E.g. when communicating with filesystems or drivers or the GUI you do it via messages. Then came AREXX and there was suddenly a standard message format for RPC - both application-to-application and user-to-application. If Amiga-applications were message oriented before, a lot of "post-AREXX" Amiga applications took it to the next level by being shaped around a dispatch loop that included not only e.g. GUI updates, but defining internal APIs that exposed pretty much every higher level action in the application, and used that as an explicit implementation strategy.

dayjah 11y ago

The basis of the protocol is similar to that of protocol buffers [1]. Though the tag field containing the map/list of segments in data is pretty novel IME. [1] https://developers.google.com/protocol-buffers/docs/encoding https://developers.google.com/protocol-buffers/docs/encoding

codemac 11y ago
Message Pack[0] is another protocol that follows this very successful simplicity. A leading thought in a lot of this is the "unset and a zero value are equivalent". It simplifies a lot of what the serializer and deserializers have to understand, and places the burden on code generation or libraries rather than complex serialization. It was unnatural for me to think this way, as I take the opposite for memory (record whether something is set or not, don't just assume it equals zero), but on the wire this makes for very compact and backwards compatible messaging. Cap'n Proto[1] I have not played with, but would love to learn more about to see the advantages and disadvantages between protobuf, msgpack, and capn. [0]: http://msgpack.org/ http://msgpack.org/ [1]: https://capnproto.org/ https://capnproto.org/
- cypher543 11y ago
  For a standardized version of Message Pack, see CBOR (RFC 7049): http://cbor.io/ http://cbor.io/
  - codemac 11y ago
    What makes it a standardized version of Message Pack? Message Pack has a specification that is well documented[0] and even the rfc you refer to specifically states it has different goals (though, eliding specifics with pretty sad generalizations)[1] [0]: https://github.com/msgpack/msgpack/blob/master/spec.md https://github.com/msgpack/msgpack/blob/master/spec.md [1]: https://tools.ietf.org/html/rfc7049#section-9 https://tools.ietf.org/html/rfc7049#section-9
    - spc476 11y ago

gourneau 11y ago

The best thing about OSC IMO are the many wonderful configurable controllers. When I was messing with OSC a few years ago TouchOSC http://hexler.net/software/touchosc http://hexler.net/software/touchosc was my favorite It is really fun to use these beautiful Star Trek looking interfaces to make your robots move around for example ;)

thirdsun 11y ago
TouchOSC has been some kind of model app for me - was available on the iPad very early and is still going strong to this day while countless other controllers came and went.

steeve 11y ago

Reminds me a lot of bencoding, which is not as popular as should (outside of bittorrent of course)

nyan4 11y ago
At least bencoding has dictionaries.
klodolph 11y ago
It's unfortunate that bencoding doesn't support text strings.

yarrel 11y ago

| ?

vshan 11y ago

Woah, funny I see this. I've been working on a distributed shell, basically trying to combine sockets and pipes to create remote pipes, aka inter-host piping. Something like: $ echo foo | host2:grep bar | host3:grep baz

RickHull 11y ago
> basically trying to combine sockets and pipes to create remote pipes Sockets, or ssh? To get a remote shell, you're going to want ssh.

paperkettle 11y ago

Major supporter of this thinking, and did several sketches in this direction a few years ago: http://www.illucia.com/ http://www.illucia.com/ Made of: 1) a suite of interconnectable videogames & apps and 2) physical patchbays to connect them Came from thinking about software like modules in a modular synth. Playgrounds of interconnectable control structures. Was patching games into samplers, slaving text editors to drum machines, etc. related writeups: http://www.illucia.com/faq/ http://www.illucia.com/faq/ another from 2010 or so when I was just starting coding (so a lot of it feels silly now.. but still some related nuggets): http://www.paperkettle.com/codebending/ http://www.paperkettle.com/codebending/

detaro 11y ago
I stumbled over your projects a few years back. Awesome work!
lbotos 11y ago
Side Q: can I ask what typeface you used for illucia?
- hjek 11y ago
  It's 'Square Serif Medium' (Look at the css, or do Right Click => Inspect Element in your browser)
- vijaykiran 11y ago
  Side A: I use http://chengyinliu.com/whatfont.html http://chengyinliu.com/whatfont.html to quickly check the fonts on a page with Safari, other browsers may have similar tools too.
mej10 11y ago
That is really cool!

ggreer 11y ago

I think OSC is interesting, but Armstrong goes too far in his disparagement of XML and JSON: > I think it is totally crazy to send JSON or XML “over the air” since this will degrade the performance of the applications giving a bad user experience and higher bills - since ultimately we pay for every bit of data. If every JSON and XML API switched to OSC, the difference in data usage would scarcely be measurable. First: The vast majority of mobile data is video, audio, images, and application updates. Second: When sent over the wire, JSON and XML are typically compressed with gzip or deflate. And let's not forget: JSON and XML have some significant advantages over most other serialization formats. They're ubiquitous. Practically every programming language has libraries to parse and generate them. Practically every programmer knows how to work with them. They're human-readable. No special programs are needed to view their contents. In other words, JSON and XML are only "wasteful" if you don't count developer time. Once you do, it suddenly makes a lot of sense to "waste" hardware resources. After all, programmers are expensive. Hardware is cheap.

colordrops 11y ago
> After all, programmers are expensive. Hardware is cheap. Except in China. I worked at a US company that did almost zero optimization on their code, deploying to 5000 AWS instances with a team of about six. In China, bandwidth closely followed by CPU were the cost centers, and the Chinese team of 30 engineers optimized the same code to run on around 50 (admittedly beefy) servers.
- lostcolony 11y ago
  Doesn't invalidate the point; another EC2 instance is a few dozen bucks a month to run full time; another developer, even overseas, is going to cost you more, at just 40 hours a week. But that said, it's rather a good thing to point out for the OP, as Joe Armstrong himself tends to value programmer time over a machine's, and that micro-optimizations are rarely ever worth it (as opposed to yours, which sounds like there were huge, glaring algorithmic issues to fix with macro optimizations, complete replacements of algorithms rather than tightening up code, careful profiling and tweaking to get a 30% speedup in one critical section, etc)

jph 11y ago

> nobody does this properly for purely local applications. You may be interested in qmail and its sub-applications, which does it properly IMHO. It's a great example of the article's key point, which is the value of simple messaging.

bjacks 11y ago

Can anyone explain what a sequential programming language is? Context here: "I guess I’d underestimated the difficulty of implementing Anything-over-TCP in a sequential language. Just because it’s really really easy in Erlang doesn’t mean to say it’s easy in sequential languages."

Gracana 11y ago
Imperative programming. Lists of statements, vs Erlang's declarative programming, where you describe the program without specifying control flow.
- white-flame 11y ago
  Erlang is absolutely a fixed in-order control flow programming language, on message receipt. It is (generally) referentially transparent, but not declarative.
  - Gracana 11y ago
    Oh, I don't know much about Erlang so I thought for sure that's what he meant. What was that supposed to mean, then?
xrstf 11y ago
Given that he is contrasting to Erlang, maybe he's talking about imperative languages instead of (pure) functional ones?
perishabledave 11y ago
I think he's referring to how Erlang uses the Actor Model. An Erlang program consists of (possibly) thousands of individual lightweight-processes. In this model a typical task may be spread out over many asynchronous processes. You can think of it as a program thats made up of thousands of micro-services. Instead of a task taking place over a sequential piece of code, it takes places over many individual processes that coordinate by communicating with each other.
11y ago

ww520 11y ago

I don't understand his emphasis on session on TCP and UDP. A session requires nothing more than a "unique" number to identify it. You can certainly have session in UDP; just attach a session number on all the messages belonging to the session. And you can certainly be sessionless in TCP; just close the connection after each message. HTTP request has no session by definition and it sits on top of TCP.

rdtsc 11y ago
> just close the connection after each message. That is if you don't expect to handle more than one connection at a time in a reasonable way. You can certainly do this loop: server_socket = create_and_listen_on_socket(...) while True: csock = server_sock.accept() csock.send(process(csock.recv())) csock.close() TCP setup and teardown is not too cheap (especially compared to sending UDP packets).
- ww520 11y ago
  UDP is certainly faster than TCP, no question about it, because of different levels of quality of service. However, coding for both is similarly simple, given the same requirement for short receive-only message. TCP session has nothing to do with the complexity. The complexity comes in how to do process() in parallel. A receive-only message can be handled in TCP: server_socket = create_and_listen_on_socket(..., MAXCONN) while True: csock = server_sock.accept() msg = csock.recv() csock.close() process(msg) A UDP loop has similar steps, minus the accept and close. The complexity to use threads or async to run process() is the same for UDP and TCP. For a normal data volume remote command protocol, the simple TCP loop is more than adequate. Just set a reasonable MAXCONN to queue up client connection requests, which can drop connections if there are too many requests, just like UDP dropping packets. Edit: I don't believe lock can be avoided in Sonic Pi server when handling concurrent incoming commands, whether it's written in Erlang or not. Sonic Pi I believe has a single audio device, which makes it a shared resource. Concurrent access to a shared resource has to be managed with lock somewhere along the call path. In that case a single thread TCP server is perfectly fine, serving as a lock as well.

jacobolus 11y ago

So what happens if you want to transmit some data in a nested hierarchical format? Just pack it into some (ad-hoc?) higher level protocol inside one of the flat pieces? Seems like a recipe for inevitable incompatibilities when two languages no longer have codecs for the new underspecified layered format.

wallacoloo 11y ago
Although the article may not have made it clear, OSC is designed primarily to pass around control signals, rather than arbitrary, non-contextualized data. Control signals have inherent meaning, like "fast-forward <n> seconds", "set color to <x>", etc. The important distinction is that "set color to <x>" (a command) is distinct from "{ 'color': 'x'}" (non-contextualized data). The layout of controls is hierarchical in the protocol; each control has an associated endpoint, which looks very much like a unix file path. For example, to unpause an audio player, you would send the message "/player/is_playing/set t" (or f, to pause). If you wanted to control the bands of a 10-band equalizer, you could implement that as "/player/eq/band/4/amp/set 0.5" (there are certainly other ways to approach it too). If you want to send control signals to multiple parts of the audio player at the same time, you could use a "bundle" composed of a separate message for each parameter you're altering. All the messages will then be handled simultaneously. Because of this, data does not need to be hierarchical. If you really want to, you can consider the endpoints to be "keys" and the control data to be "values" in a hierachical key -> value map, just to prove that everything you could do with hierarchical data is still possible using OSC. But I would encourage you to not think of it in that abstraction because the fact that OSC is built to pass messages, and not non-contextualized data is a large part of what differentiates it from other protocols like json streaming.
crucialfelix 11y ago
Its a low level protocol. There are blob and string types, so you can pack and send anything you like. I send JSON via OSC all the time.

nosuchthing 11y ago

OSC is in use for a lot of DIY hardware interfaces fed into custom modular (software) synths, DSP, or weird art programs using Pure Data, Max/MSP, Super Collider, Ableton Live and MIDI bridges to other pieces of sound plugins or external music hardware. Nice to see a good write up on this project!! http://en.flossmanuals.net/pure-data/ch065_osc/ http://en.flossmanuals.net/pure-data/ch065_osc/ https://puredata.info/community/pdwiki/OpenSoundControl https://puredata.info/community/pdwiki/OpenSoundControl http://livecontrol.q3f.org/ableton-liveapi/liveosc/ http://livecontrol.q3f.org/ableton-liveapi/liveosc/

deleted 11y ago

[deleted]

echochar 11y ago

Is there any similarity between "tag-length-value" and netstrings (1997)?

gtirloni 11y ago

gRPC seems like a good alternative.

dschiptsov 11y ago

My 5¢ is that 9P2000 protocol would be much better and canonical example. Also I hope some day one would show how to implement most of functionality of Hadoop orders of magnitude more effeciently in bunch of Plan9 shell scripts. Or in Erlang with 9P protocol driver.

yomritoyj 11y ago

From the title I thought that the ultimate monad tutorial had finally arrived.

chrisdew 11y ago

The problem with flat messages is that optional composite types can be partially present.

qwertyuiop924 11y ago

Binary protocols are only cheap if your time is free. If you can't connect to your service via telnet, you're going to spend a lot more time debugging. The only way to fix this is to develop a textual representation of your format, and write your own telnet, which is feasible, but nobody seems to have done that to OSC. Also, I find Armstrong's fanaticism grating, and am inherently distrustful of anybody who claims that their community and language has all the good ideas. Because odds are, they're wrong.

59nadir 11y ago
> Also, I find Armstrong's fanaticism grating, and am inherently distrustful of anybody who claims that their community and language has all the good ideas. Because odds are, they're wrong. You might find this[0] blog post by Armstrong interesting, wherein he comments on some things that were/are done wrong in Erlang. I would also argue that this 'fanaticism' is made to look more severe as you can find at least 4 different talks where Armstrong argues for the same things over and over. In any case, Armstrong is actually very humble, which I think seems fairly obvious if you actually watch a few talks and read more than a couple of blog posts by him. Could you point to something specific where you feel Armstrong misrepresents a strong side of Erlang, or is it just a general thing? (Edit: Just to add, I think it should be said that the Erlang team created their favorite language and that it probably represents the most useful language in use today to them, so I don't think it's surprising that they'd see more good ideas there than in any other language. On top of that Armstrong is a very straight shooter, it seems, so I think this might be something to take into account.) 0 - http://joearms.github.io/2013/05/31/a-week-with-elixir.html http://joearms.github.io/2013/05/31/a-week-with-elixir.html
- qwertyuiop924 11y ago
  It isn't so much that Joe doesn't think that erlang did anything wrong, it just feels like he believes that erlang's way of doing things, and erlang's ideas are the only good way of doing things, and are the only right way of doing things. This makes me suspicious, because that is almost never true of any language or community.

AndrewUnmuted 11y ago

For an especially bad-ass implementation of OSC, check out the SuperCollider project at https://supercollider.github.io/ https://supercollider.github.io/. SuperCollider is a programming language for audio synthesis/analysis, and algorithmic music composition. It works using a client-server paradigm, and they communicate with each other via OSC. Shameless plug time! I've been using SuperCollider to design a machine-learning solo electroacoustic music performance platform, called Sonic Multiplicities. You can hear it at http://www.sonicmultiplicities.net/ http://www.sonicmultiplicities.net/.