6 ms·
The beam just garbage collects each actor separately, and it so happens that much of the time your actor has finished before a gc happened so you never see the
by fidotron 3y ago
The beam just garbage collects each actor separately, and it so happens that much of the time your actor has finished before a gc happened so you never see the cleanup.
The beam also has a spectacular failure mode: OOM whenever messages come in at a higher rate than they are processed. The lack of backpressure mechanisms mean a huge amount of beam language developers spend way too much time recreating their own way for dealing with this or pretend it is not a problem at all. This means too many libraries in the ecosystem behave totally differently under load.
- pdimitar 3y agoI can see how you would think that, yes. In practice I haven't noticed it except in super rare cases where processes (actors) hold on to huge binaries / strings -- which is one of the weak points of BEAM's GC.
- fidotron 3y agoMy experience of this is not theoretical.
- pdimitar 3y agoMaybe you should write a blog post about it, many in the Elixir's ecosystem are very serious devs and are always looking at ways to improve it.
- fidotron 3y agoThere are entire github repos dealing with it. Frankly the Elixir ecosystem, which is not without merit, is more interested in perpetuating the myth of magic scalability by virtue of the beam.
- pdimitar 3y agoFrankly I am seeing that as a myth, you seem to have made up your mind some time ago or judged by 1-2 occasions. I am on ElixirForum every day and worked with Elixir for 7 years and have never seen anyone "perpetuate myths". I've seen some people willing to "increase adoption" which was always met with resistance by the wider community -- we believe growth should be organic. Pretty sad stance from you though, I have no idea why people get so ticked off when another programmer wants to tell them about a secret weapon. If you are not willing to try it, that's fair. Say that. Claiming you know stuff about the ecosystem while a guy who is there every day is not seeing that at all comes across as... strange. Biased. And not arguing in good faith. :(
- fidotron 3y ago> If you are not willing to try it, that's fair. Say that. Claiming you know stuff about the ecosystem while a guy who is there every day comes across as... strange. Biased. And not arguing in good faith. :( I am not willing to drag others, such as those that wrote the repos, into a technical discussion with people out to act as you are.
- pdimitar 3y agoI really don't get your replies and why their tone has to be the way it is but, have it your way. I tried.
- DoesntMatter22 3y agoThe guy you are responding to was completely calm and reasonable. Didn't say anything attacking or otherwise. I'm not sure why you are seemingly trying to cast him (and the Beam) in such a bad light, with seemingly no reason to back it up.
- fidotron 3y agoHe is adopting the missionary tactic of feigning a desire for discussion when it is just veiled evangelism.
- throwawaymaths 3y agoThe big offenders is serializing huge jsons and chucking small binaries slices from that into an ets table. Also concatenating binaries. Don't do that, use an iolist
- pdimitar 3y agoYep. Has bitten me in the bottom before.
- fud101 3y agoI've been bit by this. In reality you need to know your shit when it comes to tuning the Beam and GC to achieve decent performance under load without triggering OOM.
- pdimitar 3y agoNot denying it, yeah, it's a blind spot and it's one of the more advanced topics when you start having actually loaded nodes in production.
- throwawaymaths 3y agoI think that's developers using GenServer.cast when they should be using call. Call gives you back pressure mechanism.
- fidotron 3y agoTBH at the load we had we got substantial savings by eventually replacing usage of gen_server as well, though that probably isn’t a good idea much of the time. The OOMs were largely being caused by calls to and from other services (i.e. kafka) so the answer proved to be in controlling the rate at which things come in and out at the very edge. From what I saw I got the impression the Beam devs assumed memory and CPU usage go together so a system that is under load memory wise would also be CPU wise, but this isn’t the case if your fan out and gather involves holding large* values on which the response is based, even if for tiny amounts of time. EDIT: *large meaning "surprisingly small" if you're coming from other universes.
- throwawaymaths 3y agoThis sounds like ets tables holding on to binaries that were extracted from JSON. This is something that iirc pager duty ran into.
- fidotron 3y agoThe underlying problem we had* was the rate work was being completed was lower than the rate requests were coming in which causes the mailboxes on the actors to grow indefinitely. In golang the approximate equivalent is a buffered channel that would start blocking because it has run out, but the beam will just keep those messages piling on to the queue as fast as possible until OOM. This is obviously a philosophical tradeoff. * I should qualify that each request here had a life in low double digit ms, but there were millions of them per second, and these were very big machines.
- throwawaymaths 3y agoWhy weren't your processes dropping messages? Also I think you can tell the VM to not allow the process to exceed a certain message size and trigger some sort of rate limiting or scaling out Edit: huh. I could swear the VM had memory limit options. Guess not. Time to rewrite it in zig!