5 ms·
We used Elixir's Observer to hunt down bottlenecks
- losvedir 4y agoThis is really cool. We use Elixir at work, but we mostly use it in a "traditional web app" (i.e. non-Elixir) way, of Docker containers deployed to independent AWS instances. So I'm always intrigued by some of the more BEAM-specific things that folks do, like using `observer` on a remote (production??) node here, or distributed Elixir where the nodes communicate with each other, or "hot" code updates. How do companies deploy Elixir in such a way to take advantage of all those things? Does Sequin talk anywhere about their deploy process and how their infrastructure looks?
- lycos 4y agoDistributed Elixir can be done with Docker containers too, see https://github.com/bitwalker/libcluster https://github.com/bitwalker/libcluster which by default has some Kubernetes support but you can also have third party (or custom) clustering strategies. I've not done this myself but I've seen articles about this a lot during the past years. Hot code updates for most applications aren't really worth it in my opinion, assuming you do something like blue/green rollover deployments. It's cool that it's possible though. But it requires appup files and afaik Distillery is one of the release tools that has support for it built-in.
- ranyefet 4y agoIf you deploy to fly.io it should be very easy to create a cluster of elixir nodes.
- cpursley 4y agoDistributed Elixir on Render is crazy easy. Fly.io also looks neat.
- mattbaker 4y agoFor us we have our app deployed to $N containers with a load balancer in front (pretty standard stuff I think?) In Erlang/Elixir you can actually override how instances of the BEAM find each other (instead of the standard EPMD daemon), so we have a module that does some DNS queries, finds the IPs of the other containers and says “hi, here’s your cluster, discovery done.” (Your setup may preclude all that, I know this all depends on how a system’s architected.) After doing that we were free to use all of Erlang’s cool cluster stuff! In our case we have in-memory caches for a few things, and if a given instance does a lookup because of a cache miss it broadcasts a message to all the other nodes saying “I just looked up $expensive_thing, here’s its value” so they don’t have to do the lookup themselves, they just cache that value, so you end up with a little distributed cache with a few lines of code. In our case, btw, these cache entries are short lived and a little inconsistency does us no harm if one of our instances misses the message, networks are networks, but it’s been great! Anyway, I think it’s super cool and I’d encourage you to play around if you get the chance. Also the observer is just amazing. We’ve debugged some pretty weird memory and cpu usage issues with it, I have some internal blog posts, maybe I should see if I could make them public.
- JohnCurran 4y agoCan you speak more to how you bypass EPMD and send the IPs of the containers to each other? That would be great for a problem we’re seeing where I work
- throwawaymaths 4y agoSame. I'm not clustered yet, but I plan on it before EOY and that would be amazing. I think route53 has some internal routing capabilities, but some of the setup looks scary, or am I just being silly?
- sprite 4y agoNot sure if it's helpful but I got clustering working on ECS mostly following this guide: https://towardsaws.com/an-elixir-migration-to-microservices-in-aws-as-ecs-fargate-using-service-discovery-to-interconnect-91386a48dfa1 https://towardsaws.com/an-elixir-migration-to-microservices-...
- benmmurphy 4y agojust beware service discovery via DNS in AWS will return up to 8 addresses. so if you have more than 8 nodes you will get a random subsample for each request. depending on how your clustering works this may or may not be a problem. you can use the web api if you need to handle more than 8 hosts.
- toast0 4y agoI didn't write this blog, but it looks like it's got the right info: https://www.erlang-solutions.com/blog/erlang-and-elixir-distribution-without-epmd/ https://www.erlang-solutions.com/blog/erlang-and-elixir-dist... You can skip to "Let’s use something else" if you've already got a good grasp of the knobs to tune with epmd.
- mattbaker 4y agoYeah, so we followed the link toast0 shared to get started: https://www.erlang-solutions.com/blog/erlang-and-elixir-distribution-without-epmd/ https://www.erlang-solutions.com/blog/erlang-and-elixir-dist... It took some work to piece together but wasn't too complicated in the end. It's internal code that I can't quickly sanitize or I'd just dump it in a gist :/ Someone on the Elixir Forum might have a template or library handy though.
- conradfr 4y agoI think the screenshot under the "Memory" section is not the correct one.
- _acco 4y agoFixed, thanks!
- ananthakumaran 4y agorecon and observer_cli are the tools I reach out first to debug any issues in production. In any other language, I usually think about how to reproduce the issue locally. With Elixir, I just get into a remote shell in the affected machine and live debug the issue, and there are cases where we applied hotfix by using eval right there from the shell. The idea of the remote shell itself is alien to most languages.
- busterarm 4y agoAnd unfortunately the kind of thing that compliance flags as a big no-no once you've got any kind of filing or privacy requirements.
- jon-wood 4y agoThis sort of thing doesn’t have to be a compliance breach, but you will likely need some way of ensuring there’s a second person in the loop, typically that would take the form of having someone in a separate production infrastructure team actually driving a while you talk them through what needs to happen.
- busterarm 4y agoYes and with the added benefit of having to explain that control to your rotating bunch of compliance people every single year. I'm not criticizing the methodology as much as the useless performative nature of compliance work.
- d4mi3n 4y agoCompliance is performative until it isn't. If you've ever been party to a breach, the role of compliance and an audit trail to the security narrative becomes _very_ important. Consider: 1. We had a breach. A factor in this was insufficient oversight on a process that granted privileged access to customer data. We fixed the problem, promise that your data is safe, and don't believe this will happen again. 2. We had a breach. A factor in this was due to a gap in an existing control around customer data that had a problem we had not anticipated. These were the people involved. This is exactly how this problem occurred. This is the data that was exposed. This is documentation of our response to this incident. This is our existing policy around how we handle data and how we respond to breaches. Customers, partners, regulators, and law enforcement respond a lot better when you can demonstrate good intent and at least imply that you have some kind of process. Of the two scenarios I outlined, the latter provides those assurances. Compliance isn't the only way to do this, but it's often the easiest.
- dminor 4y agoSounds like there are some very nice observability features built into BEAM. I wish NodeJS had something similar!
- lliamander 4y agoThe BEAM is really cool, and was actually originally intended to be a bare-metal operating system. That's why it has so many features that are useful for operations: they couldn't assume you'd have any other tooling available, and often didn't even have physical access to the machines that were running it.
- austinjp 4y agoSo, the graphic at the top of the article (on mobile) is AI-generated, right? The character's fingers are smooshed. Interesting to see this approach to article graphics after I first read about it on HN recently.
- _acco 4y agoIt is. Dall-e did the heavy lifting, I tweaked with Photoshop > Painting of a detective from the 1800s, portrait, looking at a magnifying glass at a computer monitor, digital art
- conradfr 4y agoA related anecdote: some months ago I had a memory leak inside a (greatly duplicated) genserver while repeatedly calling a lib[0] function inside it, that would result in the server basically crashing after a while. I never understood what in that lib was causing the leak but I fixed it (or more accurately mitigated it) by wrapping the call in a Task.async/1 Maybe that will help someone else one day. [0] https://hexdocs.pm/shoutcast/Shoutcast.html#read_meta/1 https://hexdocs.pm/shoutcast/Shoutcast.html#read_meta/1
- filmor 4y agoIt was probably leaking refc binaries, see for example https://ferd.github.io/recon/recon.html#bin_leak-1 https://ferd.github.io/recon/recon.html#bin_leak-1. Running the function (which probably parses large binaries) in a separate process ensures that it's properly garbage collected in time.
- conradfr 4y agoInteresting thanks. Yes that could be it.
- toast0 4y agoRefC binaries should be taken care of by the virtual binary heap, which seems to be from r13 (although I thought it was newer than that... maybe there's another change I'm thinking of, but can't find). A related, but different refc binary hazard is a Process that obtains large refc binaries somehow, and makes a subbinary that it sends to another process (or ets!). The large binary is still referenced from the subbinary so there's a significant amount of excess memory. You can also run into this when binary creation is optimized to allow for appending [1], because that makes a binary of much larger than the required size (either double or 256 bytes, whichever is more). Either way, if you have a use case that naturally results in long term storage of binaries or subbinaries that allocate much more space than is really required, binary:copy/1 can be used to make a clean copy that's the exact size and isn't (yet) shared. I've seen mnesia (ets) nodes where due to the code structure, the memory use ended up at 4x what was needed, and binary:copy added before storing with ets fixed things up with no other code change. [1] https://www.erlang.org/doc/efficiency_guide/binaryhandling.html#constructing-binaries https://www.erlang.org/doc/efficiency_guide/binaryhandling.h...
- davidw 4y ago> Second, we passed one particularly large data structure from a manager to a pool of dedicated worker processes. This meant we were reincurring the memory cost of this data structure for each worker process. We couldn't eliminate the repetition, but reducing the data to its bare essentials before passing it down to the workers minimizes that cost. Hard to say without knowing much about the data in question, but my recollection is that large Erlang/Elixir/BEAM "binaries" are actually not copied around. That might be a strategy for sharing larger things in some cases. Marshalling data is pretty easy in Erlang: 2> Bin = erlang:term_to_binary([1, 2, 3]). <<131,107,0,3,1,2,3>> 3> erlang:binary_to_term(Bin). [1,2,3]
- realcorvus 4y agoIf the data does not change, persistent_term is useful as well
- lvass 4y agoJust confirmed it works. iex>term = for _ <- 0..1000000, into: [], do: :rand.uniform(0x1111111) iex>bin = :erlang.term_to_binary(term) iex>for _ <- 0..1000, do: spawn(fn -> x = bin; :timer.sleep(1000000) end) # Memory usage exploded in line below iex>for _ <- 0..1000, do: spawn(fn -> x = term; :timer.sleep(1000000) end)
- depr 4y ago"large binaries" is >64 bytes. https://www.erlang.org/doc/efficiency_guide/binaryhandling.html#refc-binaries https://www.erlang.org/doc/efficiency_guide/binaryhandling.h...
- deleted 4y ago[deleted]
- cpursley 4y agoSequin is really cool! Are y'all listening postgres WAL?
- _acco 4y agoThanks! We considered using Postgres' WAL but decided not to for the time being. Our solution now uses trigger functions. These trigger functions fire whenever a create/update/delete happens on a Sequin table. They insert a row into a log table. That log table is processed by our workers to send changes to the upstream API. The advantage of using trigger functions + a log table are all about ease of use and compatibility: our customers don't have to do anything fancy to setup Sequin, we just need a role with `create` privileges in the database. The log table also makes it easy for both them and us to debug issues, as the stream of changes that we captured is right there in the database.
- cpursley 4y agoVery cool. I'm using Elixir to listen to change events via https://github.com/cpursley/walex https://github.com/cpursley/walex (which I basically ripped off from Supabase).