3 ms·

My OTP 21 Highlights

172 points by mischov 8y ago

rdtsc 8y ago
7.5% speed gain from BEAMJIT. 2.8x faster file ops because of dirty nifs support. Multiple poll sets. This is good stuff, I can't wait to benchmark it and play with it!
- delta1 8y ago
  I was unsure what nifs referred to, is it Native Implemented Functions? [0] [0] http://erlang.org/doc/tutorial/nif.html http://erlang.org/doc/tutorial/nif.html
  - Ndymium 8y ago
    Yes. The problem earlier was that NIFs had a maximum runtime of 1 millisecond before they needed to store their work and return control to BEAM, to avoid blocking the BEAM schedulers. Dirty NIFs refer to NIFs that can run for longer, on special dirty schedulers, without affecting the stability of the BEAM system.
fasteo 8y ago
>>>> While working on the BEAMJIT development I’ve been looking a lot at the luajit project and what Mike Pall has done both in the JIT but also in the interpreter. Inspired by this and some other ideas that we got from the BEAMJIT project we decided it was time to do a major overhaul of the way that the BEAM interpreter is created As a long time Openresty (luajit) user, I have always feel a deep admiration for Mike Pall's work. After reading this, more so.
dnautics 8y ago
> Also it is now possible to open device files using file:open Anyone want to write a high-performance, highly distributed object store in a BEAM language?
- macintux 8y ago
  Based on my time at Basho observing and assisting with the development of Riak[1], I don't think you'll get the performance you want from built-in I/O, even with the new dirty scheduler support. We (well, primarily Matthew Von-Maszewski) spent countless hours optimizing LevelDB as the most performant backend for most use cases. You'd be much better off building atop Riak than starting your own object store from scratch. [1]: Most active Riak development today is courtesy the NHS - https://github.com/nhs-riak/riak https://github.com/nhs-riak/riak
  - skrebbel 8y ago
    Ha! I wonder whether all them Brexiteers knew that they weren't just saving the NHS but also Riak!
  - macintux 8y ago
    Having said that, Martin Sumner with the NHS created a Leveldb replacement in Erlang with some impressive performance characteristics before the latest BEAM I/O enhancements. https://github.com/martinsumner/leveled https://github.com/martinsumner/leveled
  - macintux 8y ago
    Correction: Basho's repository is still canonical. https://github.com/basho/riak/ https://github.com/basho/riak/
  - dnautics 8y ago
    I doubt for most distributed object stores (with consistency or availability guarantees), performance is bottlenecked by the underlying filesystem access. Probably, network latency runs the show.
    - macintux 8y ago
      To what degree that's true depends significantly on use case and architectural choices. One design error we made with Riak in its early days was shuffling data around the servers via distributed Erlang, which led to some serious performance bottlenecks. Distributed Erlang is better used for control messages; large blocks of data should be distributed out of band. Nonetheless, our customers regularly needed assistance with disk tuning, because disk access does matter quite a bit.
ghayes 8y ago
It's great to see the team continuing to work toward better performance. Each improvement can help every application built on BEAM.
alberth 8y ago
So is JIT coming to OTP 21 or not? Because reading that post it’s confusing if they are releasing a JIT or simply making BEAM runtime improvements. I ask because a JIT was scheduled for OTP 21 as noted here https://mobile.twitter.com/FrancescoC/status/872736686006046721 https://mobile.twitter.com/FrancescoC/status/872736686006046...
- mischov 8y ago
  I think it's part of the "and beyond" in that picture, and not coming in OTP 21. This video also supports that guess: https://youtu.be/hHhm0bfdj-4?t=10m14s https://youtu.be/hHhm0bfdj-4?t=10m14s

172 points by mischov 8y ago

7.5% speed gain from BEAMJIT. 2.8x faster file ops because of dirty nifs support. Multiple poll sets. This is good stuff, I can't wait to benchmark it and play with it!

delta1 8y ago
I was unsure what nifs referred to, is it Native Implemented Functions? [0] [0] http://erlang.org/doc/tutorial/nif.html http://erlang.org/doc/tutorial/nif.html
- Ndymium 8y ago
  Yes. The problem earlier was that NIFs had a maximum runtime of 1 millisecond before they needed to store their work and return control to BEAM, to avoid blocking the BEAM schedulers. Dirty NIFs refer to NIFs that can run for longer, on special dirty schedulers, without affecting the stability of the BEAM system.

fasteo 8y ago

>>>> While working on the BEAMJIT development I’ve been looking a lot at the luajit project and what Mike Pall has done both in the JIT but also in the interpreter. Inspired by this and some other ideas that we got from the BEAMJIT project we decided it was time to do a major overhaul of the way that the BEAM interpreter is created As a long time Openresty (luajit) user, I have always feel a deep admiration for Mike Pall's work. After reading this, more so.

dnautics 8y ago

> Also it is now possible to open device files using file:open Anyone want to write a high-performance, highly distributed object store in a BEAM language?

macintux 8y ago
Based on my time at Basho observing and assisting with the development of Riak[1], I don't think you'll get the performance you want from built-in I/O, even with the new dirty scheduler support. We (well, primarily Matthew Von-Maszewski) spent countless hours optimizing LevelDB as the most performant backend for most use cases. You'd be much better off building atop Riak than starting your own object store from scratch. [1]: Most active Riak development today is courtesy the NHS - https://github.com/nhs-riak/riak https://github.com/nhs-riak/riak
- skrebbel 8y ago
  Ha! I wonder whether all them Brexiteers knew that they weren't just saving the NHS but also Riak!
- macintux 8y ago
  Having said that, Martin Sumner with the NHS created a Leveldb replacement in Erlang with some impressive performance characteristics before the latest BEAM I/O enhancements. https://github.com/martinsumner/leveled https://github.com/martinsumner/leveled
- macintux 8y ago
  Correction: Basho's repository is still canonical. https://github.com/basho/riak/ https://github.com/basho/riak/
- dnautics 8y ago
  I doubt for most distributed object stores (with consistency or availability guarantees), performance is bottlenecked by the underlying filesystem access. Probably, network latency runs the show.
  - macintux 8y ago
    To what degree that's true depends significantly on use case and architectural choices. One design error we made with Riak in its early days was shuffling data around the servers via distributed Erlang, which led to some serious performance bottlenecks. Distributed Erlang is better used for control messages; large blocks of data should be distributed out of band. Nonetheless, our customers regularly needed assistance with disk tuning, because disk access does matter quite a bit.

ghayes 8y ago

It's great to see the team continuing to work toward better performance. Each improvement can help every application built on BEAM.

alberth 8y ago

So is JIT coming to OTP 21 or not? Because reading that post it’s confusing if they are releasing a JIT or simply making BEAM runtime improvements. I ask because a JIT was scheduled for OTP 21 as noted here https://mobile.twitter.com/FrancescoC/status/872736686006046721 https://mobile.twitter.com/FrancescoC/status/872736686006046...

mischov 8y ago
I think it's part of the "and beyond" in that picture, and not coming in OTP 21. This video also supports that guess: https://youtu.be/hHhm0bfdj-4?t=10m14s https://youtu.be/hHhm0bfdj-4?t=10m14s