6 ms·
People tend to forget that scalability is not a binary property. You always scale up to some users, up to some architecture, up to some amount of nodes. There i
by josevalim 9y ago
People tend to forget that scalability is not a binary property. You always scale up to some users, up to some architecture, up to some amount of nodes. There is no system that will scale to infinity without requiring developer intervention once business needs and application patterns start to settle in.
Distributed Erlang/Elixir has known limitations. For example, the network is fully meshed, which gives you about 60 to 200 nodes in a cluster. Or don't send large data over distribution, as that delays the other messages, etc. Some of those are easily solvable. For example, you can rely on your orchestration tools to break your clusters in groups. Or you can setup an out of band tcp/udp socket for large data. Others may be more complex.
The important question, however, is how far you can go without having to tweak, and, once you reach those roadblocks, how well you can address them. In many platforms, writing a distributed system is a no-no or, at best, they require you to assemble and tweak from day one. In this case, the ability to start with Erlang/Elixir and tweak as you grow is a feature.
And if you never run into those roadblocks, then you can happily continue running on the default stack. Just look at the many companies using Phoenix PubSub and Phoenix Presence, both distributed, without having to worry about fine-tuning the distribution.
- whitepoplar 9y agoThis is a great point! What would you say is the best book to really learn OTP?
- RexM 9y agoDesigning for Scalability with Erlang/OTP is pretty good so far. http://shop.oreilly.com/product/0636920024149.do http://shop.oreilly.com/product/0636920024149.do
- Abekkus 9y agoI'd read up on the elixir language itself from the official website, then look into the "Little Elixir and OTP Guidebook" https://www.manning.com/books/the-little-elixir-and-otp-guidebook https://www.manning.com/books/the-little-elixir-and-otp-guid...
- oldpond 9y agoPragprog has a good selection. https://pragprog.com/titles/category/elixir?f[include_mine]=yes&f[sort_by]=pubdate&f[category]=elixir&f[skill_level]=All&f[title_contains]= https://pragprog.com/titles/category/elixir?f[include_mine]=...
- dtzur 9y agoShut up and take my money! I'm sold. Thank you for this explanation.
- majidazimi 9y agoWhat difference does it make? Erlang/OTP distribution doesn't have pluggable architecture. Sooner or later you will reach a point that you have to modify it. Then you are diverging from the original branch which makes it even more difficult to maintain it. You have to merge your additions into every release (minor or major) and test it thoroughly. A better architecture for a distributed system has a strong composability property. It should be possible to modify every possible aspect of it on a running cluster without introducing downtime. Write your own standard well documented distribution layer and become independent of underlying technologies.
- andy_ppp 9y agoI think, 1) you are misunderstanding the grand parents points. Better to have primitives that scale you to X rather than having to get to X on your own. 2) The grand parent wrote the Elixir language. 3) Making sweeping statements like "A better architecture for a distributed system has a strong composability property" is very easy.
- majidazimi 9y ago> 1) Better to have primitives that scale you to X rather than having to get to X on your own. For a fast time to market, I do agree with you. But if you are well established company in the market, then in most cases "do it yourself" is the best idea. Unless you have the money to call for experts to fix the problem for you. > 2) The grand parent wrote the Elixir language. I didn't know him, nor I care. > 3) Making sweeping like "A better architecture for a distributed system has a strong composability property" is very easy. I'm considering this as an excuse, than a technical argument.
- josevalim 9y ago> But if you are well established company in the market, then in most cases "do it yourself" is the best idea. Unless you have the money to call for experts to fix the problem for you. This requires having distributed system experts on your team or having the money to call them from day one, before you even reach the market and before you are even sure you will have custom needs. If you have the need, the expertize and the time, then surely. But writing a distributed system from scratch should certainly not be the first choice (IMO).
- janekm 9y agoThanks a lot for the very insightful reply. I'm learning a lot about Erlang from the responses, and a lot of respect for the community as well. Not bad for an on reflection somewhat inflammatory comment :)