4 ms·

The isolated faults thing: how is this different than every other managed language? Especially if shared state isn't used?

gamache 11y ago
One way it's different is that there is rich support for specifying what should happen when subsystems crash. You make what's called a "supervision tree" where supervisors are notified when a child process crashes, and can choose what to do -- restart that child, restart all similar children, halt execution, do something completely custom, etc. Scala's Akka library is another place you can find this (although a bit harder to configure) but I don't know many other languages that offer this.
deleted 11y ago
[deleted]
rdtsc 11y ago
Start multiple threads. They write some data from them. One of them crashes. Is it safe to just restart that thread and continue? Not always, because that thread might have written to so some shared data structure and left it in an inconsistent state. Restarting that one thread might seem to work but because there is no guarantee, the safest way is to restart the OS process. It might not even be your code. Maybe your RPC library or some other package did it internally. You might restart and 99% of time it will work, but it will not be a guaranteed thing. Now, of course the equivalent counterpart to this is not threads but you can do this with OS processes. And I've done it in Python. You can use pipes, file system writes, sockets to create a reliable system were some processes can crash without bringing down the whole service. But it is cumbersome. But imagine the sever handles a million connections. Certainly doable in Erlang. Each one handled by one lightweight process. Connection does something unusual, maybe uses a new feature which was added last night. It crashes, that's fine. Other 999999 are ok. If this was a C++ program, it might have segfault-ed and caused the other 999999 connections to be dropped. I've seen systems which had a periodically crashing subsystem that was being restarted behind the scenes for a while most of the system stayed up without noticeable issues. In business terms that means a smaller ops team, it means not having to wake up at 4am to answer pages (if subsystem heals itself, can just fix it in the morning) and so on. Moreover, live code hot-patching is often touted as a gimmick. But I've used on a large production clusters while it was serving tens of thousands of requests per second without stopping. Sure it was scary and by that time something has already gone wrong (to need that fix), and it was nice not having to to shut down everything just to add an extra trace or log statement.
- illumen 11y ago
  I don't think it's so cumbersome with modern python. There's first class coroutines, futures, process pools and such now. Also there are a number of queues systems in python which easily allow running failed tasks again. One project I use is basically task(func, args). That's it. Then if it fails we can have them automatically restart, or wait for review etc. This is over multiple machines too. It's only about 60 lines of code, including nice gui interfaces to manage the tasks, and deploy the whole lot to various machines. Erlang is amazing of course. Just wanted to say that some of those patterns are now available in python and easily usable. Not common to all python code, which is where I think erlang wins out. It puts this stuff front and center, and has first class support. Looking at a random erlang code base, you'll probably see it there. Random python code bases... not so much (ok, queue systems are quite common in Django/Flask projects). It's very rare to see python greelets/eventlets in the wild for example, but generators and async stuff are becoming quite commonplace. Also python has single dispatch now (built in, not in a third party library). Another thing which Erlang does well (pattern matching). Again, not so commonly used except in modern python shops. These combined with quickcheck for python (hypothesis), and gradual typing really have made modern python a much more happy place. Erlang deserves some big respect for spreading good ideas.
  - rdtsc 11y ago
    Sure. Individual aspects of Erlang/OTP are all present in other systems. OS processes, queuing backends, light weight co-routines, Java has some code reloading too. But it makes a difference if it is in one language/framework and built-in. It makes tracing/debugging/developing easier. Like with Python, yeah can use a queuing subsystems and submit jobs. But that is another service to configure and manage. Can use multiprocessing (and I've done that), but can't launch 1M of sub-processes. Which now changes how you develop. It has a green-thread co-routine support via greenlet (eventlet & gevent) but those share memory and if you do any CPU intensive work will block each other and will also share the heap.
    - vvanders 11y ago
      I'd say it's a stretch that Java has code reloading. Sure you can reload the bytecode, but unless you're marshaling the data to the new version like Erlang does you're just introducing more errors than fixing things.
jlouis 11y ago
It is only half the story. In reality it is isolated resources: memory, CPU usage, files, sockets, ... The key benefit is that a single process has a harder time monopolizing the system and taking over the system as a whole. This allows you to write code where the code is known to be suboptimal because it cannot take over the machine. In contrast, many other systems relies on a proactive model where each line of code has to be scrutinized for utmost correctness, or your system as a whole will fall. As for the faults, there is a subtle but important difference compared to exceptions: you handle any error, not just the ones you figured could happen. And because resources are bound to processes, when the process goes away, so does its resource. Usually you would have to handle this in other languages. The final point is that a fault is made transient: the system recovers and reestablishes the last known good state/invariant of the system. This allows you to try again which often handles the fault. And if it keeps failing, the fault is escalated to gradually larger and larger parts of the program. There is nothing magical about the model. It is just that other languages have next to no convenient support for it.

gamache 11y ago

One way it's different is that there is rich support for specifying what should happen when subsystems crash. You make what's called a "supervision tree" where supervisors are notified when a child process crashes, and can choose what to do -- restart that child, restart all similar children, halt execution, do something completely custom, etc. Scala's Akka library is another place you can find this (although a bit harder to configure) but I don't know many other languages that offer this.

deleted 11y ago

[deleted]

rdtsc 11y ago

Start multiple threads. They write some data from them. One of them crashes. Is it safe to just restart that thread and continue? Not always, because that thread might have written to so some shared data structure and left it in an inconsistent state. Restarting that one thread might seem to work but because there is no guarantee, the safest way is to restart the OS process. It might not even be your code. Maybe your RPC library or some other package did it internally. You might restart and 99% of time it will work, but it will not be a guaranteed thing. Now, of course the equivalent counterpart to this is not threads but you can do this with OS processes. And I've done it in Python. You can use pipes, file system writes, sockets to create a reliable system were some processes can crash without bringing down the whole service. But it is cumbersome. But imagine the sever handles a million connections. Certainly doable in Erlang. Each one handled by one lightweight process. Connection does something unusual, maybe uses a new feature which was added last night. It crashes, that's fine. Other 999999 are ok. If this was a C++ program, it might have segfault-ed and caused the other 999999 connections to be dropped. I've seen systems which had a periodically crashing subsystem that was being restarted behind the scenes for a while most of the system stayed up without noticeable issues. In business terms that means a smaller ops team, it means not having to wake up at 4am to answer pages (if subsystem heals itself, can just fix it in the morning) and so on. Moreover, live code hot-patching is often touted as a gimmick. But I've used on a large production clusters while it was serving tens of thousands of requests per second without stopping. Sure it was scary and by that time something has already gone wrong (to need that fix), and it was nice not having to to shut down everything just to add an extra trace or log statement.

illumen 11y ago
I don't think it's so cumbersome with modern python. There's first class coroutines, futures, process pools and such now. Also there are a number of queues systems in python which easily allow running failed tasks again. One project I use is basically task(func, args). That's it. Then if it fails we can have them automatically restart, or wait for review etc. This is over multiple machines too. It's only about 60 lines of code, including nice gui interfaces to manage the tasks, and deploy the whole lot to various machines. Erlang is amazing of course. Just wanted to say that some of those patterns are now available in python and easily usable. Not common to all python code, which is where I think erlang wins out. It puts this stuff front and center, and has first class support. Looking at a random erlang code base, you'll probably see it there. Random python code bases... not so much (ok, queue systems are quite common in Django/Flask projects). It's very rare to see python greelets/eventlets in the wild for example, but generators and async stuff are becoming quite commonplace. Also python has single dispatch now (built in, not in a third party library). Another thing which Erlang does well (pattern matching). Again, not so commonly used except in modern python shops. These combined with quickcheck for python (hypothesis), and gradual typing really have made modern python a much more happy place. Erlang deserves some big respect for spreading good ideas.
- rdtsc 11y ago
  Sure. Individual aspects of Erlang/OTP are all present in other systems. OS processes, queuing backends, light weight co-routines, Java has some code reloading too. But it makes a difference if it is in one language/framework and built-in. It makes tracing/debugging/developing easier. Like with Python, yeah can use a queuing subsystems and submit jobs. But that is another service to configure and manage. Can use multiprocessing (and I've done that), but can't launch 1M of sub-processes. Which now changes how you develop. It has a green-thread co-routine support via greenlet (eventlet & gevent) but those share memory and if you do any CPU intensive work will block each other and will also share the heap.
  - vvanders 11y ago
    I'd say it's a stretch that Java has code reloading. Sure you can reload the bytecode, but unless you're marshaling the data to the new version like Erlang does you're just introducing more errors than fixing things.

jlouis 11y ago

It is only half the story. In reality it is isolated resources: memory, CPU usage, files, sockets, ... The key benefit is that a single process has a harder time monopolizing the system and taking over the system as a whole. This allows you to write code where the code is known to be suboptimal because it cannot take over the machine. In contrast, many other systems relies on a proactive model where each line of code has to be scrutinized for utmost correctness, or your system as a whole will fall. As for the faults, there is a subtle but important difference compared to exceptions: you handle any error, not just the ones you figured could happen. And because resources are bound to processes, when the process goes away, so does its resource. Usually you would have to handle this in other languages. The final point is that a fault is made transient: the system recovers and reestablishes the last known good state/invariant of the system. This allows you to try again which often handles the fault. And if it keeps failing, the fault is escalated to gradually larger and larger parts of the program. There is nothing magical about the model. It is just that other languages have next to no convenient support for it.