3 ms·

This sounds like ets tables holding on to binaries that were extracted from JSON. This is something that iirc pager duty ran into.

• fidotron 3y ago
  The underlying problem we had* was the rate work was being completed was lower than the rate requests were coming in which causes the mailboxes on the actors to grow indefinitely. In golang the approximate equivalent is a buffered channel that would start blocking because it has run out, but the beam will just keep those messages piling on to the queue as fast as possible until OOM. This is obviously a philosophical tradeoff. * I should qualify that each request here had a life in low double digit ms, but there were millions of them per second, and these were very big machines.
  • throwawaymaths 3y ago
    Why weren't your processes dropping messages? Also I think you can tell the VM to not allow the process to exceed a certain message size and trigger some sort of rate limiting or scaling out Edit: huh. I could swear the VM had memory limit options. Guess not. Time to rewrite it in zig!
    • fidotron 3y ago
      Yeah, I think that's the assumption people had been operating under. That team would have thoroughly endorsed a zig rewrite! It was a very odd situation where most of us liked erlang the language but found the beam to be an annoying beast, whereas most of the world seems to be the opposite.