3 ms·

Simplifying Systems with Elixir • Sasa Juric • YOW! 2020 https://www.youtube.com/watch?v=EDfm2fVS4Bo https://www.youtube.com/watch?v=EDfm2fVS4Bo A noob questi

Simplifying Systems with Elixir • Sasa Juric • YOW! 2020 https://www.youtube.com/watch?v=EDfm2fVS4Bo https://www.youtube.com/watch?v=EDfm2fVS4Bo A noob question, is Elixir good at crawling tasks in the process simplying the system architecture.

marginalia_nu 2y ago
I'm not well versed enough in Elixir to give a good answer. The hard problems in crawling is mostly dealing with a very large and potentially highly mutable state. It's a concurrent problem, but a fairly easy one and a good use case for a traditional thread pool since there is latency is mostly a non-issue.
- deleted 2y ago
  [deleted]
- muscomposter 2y ago
  based on what you said, and my assumption that Elixir (thanks to Erlang’s VM) is pretty good at concurrency, then yes Elixir would be good for crawling
jerf 2y ago
Elixir is one of a reasonably large number of acceptable languages. A crawler, and especially a very large one like Marginalia, is going to implement its own queuing logic, its own retry logic, its own management of same, and so on and so forth. As a result, things like "BEAM has supervisor trees" are actually not that useful; as they are nowhere near enough for this use case and need to be augmented anyhow, they don't really help a lot, and can even get in the way. At most the BEAM OTP infrastructure might help you bootstrap something up somewhat more quickly but I'd expect single-digit weeks into the dev process before it isn't really that helpful anymore. None of the BEAM-specific strengths strike me as hugely helpful here, mostly for similar reasons; they don't quite match what a crawler wants per se and the crawler is going to reimplement them anyhow. For a project of this scale, you also don't want to be doing the raw indexing in Elixir, as it is not a very fast language, and at this scale that adds up quickly, so you're going to pull in another language anyhow. All in all, while I might call it "acceptable", there's a solid half-dozen languages (plus runtimes, as appropriate) that I'd put solidly in front of Elixir for this use case and another several I'd rate as roughly ties. It certainly is not the case that it offers some sort of amazing, blow-me-away advantage that makes it the only sensible answer or anything.
- marginalia_nu 2y ago
  I think the demands on concurrency are relatively basic. It's nice to have something a bit more robust than raw pthreads, but the main thing that makes or breaks a crawler is access to a robust HTTP client library and HTML parser, solid I/O performance, that sort of thing. Because the problem domain looks the way it looks, concurrency is one of the easier parts.

marginalia_nu 2y ago

I'm not well versed enough in Elixir to give a good answer. The hard problems in crawling is mostly dealing with a very large and potentially highly mutable state. It's a concurrent problem, but a fairly easy one and a good use case for a traditional thread pool since there is latency is mostly a non-issue.

deleted 2y ago
[deleted]
muscomposter 2y ago
based on what you said, and my assumption that Elixir (thanks to Erlang’s VM) is pretty good at concurrency, then yes Elixir would be good for crawling

jerf 2y ago

Elixir is one of a reasonably large number of acceptable languages. A crawler, and especially a very large one like Marginalia, is going to implement its own queuing logic, its own retry logic, its own management of same, and so on and so forth. As a result, things like "BEAM has supervisor trees" are actually not that useful; as they are nowhere near enough for this use case and need to be augmented anyhow, they don't really help a lot, and can even get in the way. At most the BEAM OTP infrastructure might help you bootstrap something up somewhat more quickly but I'd expect single-digit weeks into the dev process before it isn't really that helpful anymore. None of the BEAM-specific strengths strike me as hugely helpful here, mostly for similar reasons; they don't quite match what a crawler wants per se and the crawler is going to reimplement them anyhow. For a project of this scale, you also don't want to be doing the raw indexing in Elixir, as it is not a very fast language, and at this scale that adds up quickly, so you're going to pull in another language anyhow. All in all, while I might call it "acceptable", there's a solid half-dozen languages (plus runtimes, as appropriate) that I'd put solidly in front of Elixir for this use case and another several I'd rate as roughly ties. It certainly is not the case that it offers some sort of amazing, blow-me-away advantage that makes it the only sensible answer or anything.

marginalia_nu 2y ago
I think the demands on concurrency are relatively basic. It's nice to have something a bit more robust than raw pthreads, but the main thing that makes or breaks a crawler is access to a robust HTTP client library and HTML parser, solid I/O performance, that sort of thing. Because the problem domain looks the way it looks, concurrency is one of the easier parts.