6 ms·

Does anyone know of erlang being used for scientfic computing? The ideas and "let it crash" scheme outlined in the article are really cool, and it made me think

Does anyone know of erlang being used for scientfic computing? The ideas and "let it crash" scheme outlined in the article are really cool, and it made me think how it would actually be very useful for making robust data pipelines in scientific contexts. For example, in astronomy you would have a telescope spitting out raw data, and a pipeline processing the data to create more inspectable intermediate products. Typically these are brittle C/C++ programs, with lots of state and branching logic, and further they become extremely hard to refactor because their function is so integral to running the facility. Erlang seems like it would be very useful for a control and pipeline system like this that needs to stay alive and not crash, but it's hard to tell (for me) if it's well suited to numeric/scientific computing tasks. Do people know of libraries or resources that would be useful for that? I did find this neat lsci[1] package that aims to call python modules like numpy/matplotlib (which would be perfect) but is sadly in a nascent state. [1] https://github.com/lfex/lsci https://github.com/lfex/lsci

pg_bot 8y ago
Erlang's main weakness is computationally intensive mathematics. You could always write a NIF in C if you needed pure speed for a specific function. If you're not interested in writing safe c code it looks like there is a bridge from rust to erlang. https://github.com/hansihe/rustler https://github.com/hansihe/rustler
- digitalzombie 8y ago
  Even if you write NIF C or in Rust. They can take down Erlang's BEAM VM. > http://erlang.org/doc/tutorial/nif.html http://erlang.org/doc/tutorial/nif.html Erlang itself shield you from actually doing stupid thing and follow the assumptions that BEAM have while running programs. You're better off choosing a different language. Have Erlang deal with concurrency but don't have it do number crunching.
  - xtagon 8y ago
    Rustler in particular boasts safety and an inability to crash the Erlang VM. Whether that holds true in practice, I don't know, but it's the first thing they mention in their README.
  - tov_objorkin 8y ago
    If you need fast small function use NIF, for large logic blocks better use C node its safe and isolated.
deleted 8y ago
[deleted]
Ceezy 8y ago
You should probably use erlang to start/monitor/kill process but not for doing the actual computing
andy_ppp 8y ago
I have thought about this too, even doing things like modelling climate change through distributed message passing (each process could act like a cube of the earth) but sadly I think the performance is going to be too far from C to be useful. HiPE has seen some very large performance increases in maths performance but still C is going to alway be a substantial amount faster.
xtrapolate 8y ago
> "Typically these are brittle C/C++ programs, with lots of state and branching logic, and further they become extremely hard to refactor because their function is so integral to running the facility" Switching to Erlang won't inherently make those issues disappear. Erlang could definitely save you some C/C++ boilerplate code, but if your business logic is still stateful, it's not necessarily going to be easier to implement any of that with Erlang. Erlang is probably more forgiving on the developers than C/C++ are, but that can be said of many other languages (ie. Python, Java). If you're looking to pass messages around between processes/threads, there are plenty of libraries you can use today to start refactoring that C/C++ code: http://nanomsg.org/ http://nanomsg.org/ http://zeromq.org/ http://zeromq.org/ https://www.rabbitmq.com/ https://www.rabbitmq.com/ (most likely an overkill for the specific use-case you described, will be more relevant in a distributed computing scenario)
jiangplus 8y ago
What you need should be a language/framework to better model the complex state of your system, and handle distributing processing, monitoring, fault tolerance, recovery and retries. Consider a functional programming language with immutability and pattern matching, which should help a lot.
- codetrotter 8y ago
  > Consider a functional programming language with immutability and pattern matching, which should help a lot. So Erlang then
  - farresito 8y ago
    Erlang is not particularly performant, and it seems like what the top comment is describing requires something faster.
    - tormeh 8y ago
      You only use Erlang for the orchestration part. Probably Fortran or something for the actual number crunching.
  - IronBacon 8y ago
    I'm suggesting Ocaml. ^__^ It doesn't have something like OTP (there are libraries like LWT for concurrency and IPC tho) but allegedly should have good math performances, from what I recall the last time I've played with the "big languages shootout" game...
pjmlp 8y ago
Chapel and Julia have much better acceptance in HPC, than Erlang ever will. The language is not designed for number crunching.
jlouis 8y ago
I think the typical experienced Erlang user would keep the C/C++ programs, but manage them through a fabric of Erlang code. That is, each part is cut up into a small program, and these are controlled by the Erlang system so it acts like a supervisor of the programs. They are run under some kind of "visor" so their resources are limited to sane values etc. When the C/C++ program does something bad, the Erlang system is notified of this. It can then take action in order to keep the processing pipeline running. You could argue that in the current context of containers, this is easy to achieve with them. But I claim writing all of this in a programming language is far more manageable than trying to handle the rate at which the whole container-context is evolving.
di4na 8y ago
There are some examples. See https://youtu.be/xj3smNjGLaE https://youtu.be/xj3smNjGLaE

pg_bot 8y ago

Erlang's main weakness is computationally intensive mathematics. You could always write a NIF in C if you needed pure speed for a specific function. If you're not interested in writing safe c code it looks like there is a bridge from rust to erlang. https://github.com/hansihe/rustler https://github.com/hansihe/rustler

digitalzombie 8y ago
Even if you write NIF C or in Rust. They can take down Erlang's BEAM VM. > http://erlang.org/doc/tutorial/nif.html http://erlang.org/doc/tutorial/nif.html Erlang itself shield you from actually doing stupid thing and follow the assumptions that BEAM have while running programs. You're better off choosing a different language. Have Erlang deal with concurrency but don't have it do number crunching.
- xtagon 8y ago
  Rustler in particular boasts safety and an inability to crash the Erlang VM. Whether that holds true in practice, I don't know, but it's the first thing they mention in their README.
- tov_objorkin 8y ago
  If you need fast small function use NIF, for large logic blocks better use C node its safe and isolated.

deleted 8y ago

[deleted]

Ceezy 8y ago

You should probably use erlang to start/monitor/kill process but not for doing the actual computing

andy_ppp 8y ago

I have thought about this too, even doing things like modelling climate change through distributed message passing (each process could act like a cube of the earth) but sadly I think the performance is going to be too far from C to be useful. HiPE has seen some very large performance increases in maths performance but still C is going to alway be a substantial amount faster.

xtrapolate 8y ago

> "Typically these are brittle C/C++ programs, with lots of state and branching logic, and further they become extremely hard to refactor because their function is so integral to running the facility" Switching to Erlang won't inherently make those issues disappear. Erlang could definitely save you some C/C++ boilerplate code, but if your business logic is still stateful, it's not necessarily going to be easier to implement any of that with Erlang. Erlang is probably more forgiving on the developers than C/C++ are, but that can be said of many other languages (ie. Python, Java). If you're looking to pass messages around between processes/threads, there are plenty of libraries you can use today to start refactoring that C/C++ code: http://nanomsg.org/ http://nanomsg.org/ http://zeromq.org/ http://zeromq.org/ https://www.rabbitmq.com/ https://www.rabbitmq.com/ (most likely an overkill for the specific use-case you described, will be more relevant in a distributed computing scenario)

jiangplus 8y ago

What you need should be a language/framework to better model the complex state of your system, and handle distributing processing, monitoring, fault tolerance, recovery and retries. Consider a functional programming language with immutability and pattern matching, which should help a lot.

codetrotter 8y ago
> Consider a functional programming language with immutability and pattern matching, which should help a lot. So Erlang then
- farresito 8y ago
  Erlang is not particularly performant, and it seems like what the top comment is describing requires something faster.
  - tormeh 8y ago
    You only use Erlang for the orchestration part. Probably Fortran or something for the actual number crunching.
- IronBacon 8y ago
  I'm suggesting Ocaml. ^__^ It doesn't have something like OTP (there are libraries like LWT for concurrency and IPC tho) but allegedly should have good math performances, from what I recall the last time I've played with the "big languages shootout" game...

pjmlp 8y ago

Chapel and Julia have much better acceptance in HPC, than Erlang ever will. The language is not designed for number crunching.

jlouis 8y ago

I think the typical experienced Erlang user would keep the C/C++ programs, but manage them through a fabric of Erlang code. That is, each part is cut up into a small program, and these are controlled by the Erlang system so it acts like a supervisor of the programs. They are run under some kind of "visor" so their resources are limited to sane values etc. When the C/C++ program does something bad, the Erlang system is notified of this. It can then take action in order to keep the processing pipeline running. You could argue that in the current context of containers, this is easy to achieve with them. But I claim writing all of this in a programming language is far more manageable than trying to handle the rate at which the whole container-context is evolving.

di4na 8y ago

There are some examples. See https://youtu.be/xj3smNjGLaE https://youtu.be/xj3smNjGLaE