4 ms·
> Also it is now possible to open device files using file:open Anyone want to write a high-performance, highly distributed object store in a BEAM language?
by dnautics 9y ago
> Also it is now possible to open device files using file:open
Anyone want to write a high-performance, highly distributed object store in a BEAM language?
- macintux 9y agoBased on my time at Basho observing and assisting with the development of Riak[1], I don't think you'll get the performance you want from built-in I/O, even with the new dirty scheduler support. We (well, primarily Matthew Von-Maszewski) spent countless hours optimizing LevelDB as the most performant backend for most use cases. You'd be much better off building atop Riak than starting your own object store from scratch. [1]: Most active Riak development today is courtesy the NHS - https://github.com/nhs-riak/riak https://github.com/nhs-riak/riak
- skrebbel 9y agoHa! I wonder whether all them Brexiteers knew that they weren't just saving the NHS but also Riak!
- macintux 9y agoHaving said that, Martin Sumner with the NHS created a Leveldb replacement in Erlang with some impressive performance characteristics before the latest BEAM I/O enhancements. https://github.com/martinsumner/leveled https://github.com/martinsumner/leveled
- macintux 9y agoCorrection: Basho's repository is still canonical. https://github.com/basho/riak/ https://github.com/basho/riak/
- dnautics 9y agoI doubt for most distributed object stores (with consistency or availability guarantees), performance is bottlenecked by the underlying filesystem access. Probably, network latency runs the show.
- macintux 9y agoTo what degree that's true depends significantly on use case and architectural choices. One design error we made with Riak in its early days was shuffling data around the servers via distributed Erlang, which led to some serious performance bottlenecks. Distributed Erlang is better used for control messages; large blocks of data should be distributed out of band. Nonetheless, our customers regularly needed assistance with disk tuning, because disk access does matter quite a bit.
- misterbowfinger 9y agoi got u
- lliamander 9y agoMaybe something based on this paper: https://blog.acolyer.org/2018/03/27/anna-a-kvs-for-any-scale/ https://blog.acolyer.org/2018/03/27/anna-a-kvs-for-any-scale... ?
- Immortalin 9y agoCouchDB?
- dnautics 9y agobinaries, not structured objects. Redundancy via erasure coding, etc.