10 ms·

Elixir GenServer Explained

272 points by cheeseblubber 5y ago

FrancoisBosun 5y ago
Well explained article. Good job OP!
intergalplan 5y ago
Maybe a dumb complaint, but I hate the name. I have a really hard time not reading it as the active name "Generate Server". Like, I'd expect it to be a function that generates servers (whatever that means, since what it's dealing with isn't something I'd normally call "a server" either). It bugs the hell out of me every time I read Elixir code. One of those things I have to keep reminding myself: "OK, it's not what it looks like it is, at all... what did these misleading terms actually mean, again?"
- pessimizer 5y ago
  Is "general" or "generic" more misleading than "generate"?
  - deleted 5y ago
    [deleted]
- brobinson 5y ago
  It's "generic" server: http://erlang.org/doc/man/gen_server.html http://erlang.org/doc/man/gen_server.html
  - hinkley 5y ago
    The sooner you start smacking people with tribal knowledge, the sooner many people start to back away slowly. I used to interview ex-coworkers. I don't do it as much anymore because it gets old/depressing and the answers tend not to change that much. If, for the purposes of thinking about turnover, you look at a former colleague as an 'aggrieved party', we are generally somewhere in the neighborhood of mediocre at agreeing that certain things were the 'final straw', and work backward through that to actions to increase retention. One of the things that interested me about these conversations was that people tend to work chronologically backward through many of their complaints. But what was surprising was that some people would go all the way back to their first weeks. To the first straws. The warning signs they ignored. In a way this is not unlike talking to someone who just broke up with a romantic partner. And while their ex may realize that his/her problems started back with some early inconsiderate behavior that snowballed, and resolve to 'do better next time', I rarely see companies do this, unless I instigate it. Point is, these petty slights stack up, and can become a big part of someone's narrative for abandoning you (often in a huff). First impressions are important, and you dismiss them at your own peril.
    - dasil003 5y ago
      What? GP's response is terse but not impolite, and explaining what an abbreviation means is a far cry from "smacking people with tribal knowledge", it's just table stakes explanation in a field which requires precision. The only reason modern computing got where it is today is by standing on the shoulders of giants. If we go back and re-litigate every naming decision of the past several decades just to make newcomers feel welcome, all progress will slow as we collectively devolve into continuous bikeshedding.
      - hinkley 5y ago
        I'm talking about the name GenServer, not brobinson's response. There is the corollary: If we have not seen farther, it is because giants were standing on our toes. There's a huge degree of cognitive dissonance on these issues. Just because something had a reason in the past doesn't mean we have to keep doing it forever. Also "nobody understands" why people keep re-inventing wheels. It's not all hubris, at least not all the time. It's also declaring open season on those past compromises, especially the ones people bump into immediately. Especially the ones the current maintainers immediately get defensive about. Technology dies for a lot of reasons, but the apologists never seem to grasp that the apologies are a stopgap. They explain the pain, they don't cure it. Believe it or not, I'm pro-Elixir. It's just that as I learn any new technology, I start filling out proverbial bingo card boxes for the things I can predict the next person will complain about. Is that a bit cynical? It could be, but it also serves as my todo list for when someone makes an offhand comment about how you really don't have to do this dumb thing, you could use this library that does it for you. Now my coworkers either don't have to worry about that or I have a suggestion when they do. Turns out when people are in pain, they appreciate sympathy a lot more than they appreciate being gaslit about how it's just their poor sense of history. Their blatantly implied selfishness. GenServer could use a new name. I think we forget sometimes that you can rename old things by giving them a second, better name and phasing out the old one. I don't think there's anything seditious in that statement.
        pmontra 5y ago
        I've been using Elixir for 4 years, it pays my bills and I generally like it. GenServer could be a misleading name (it never occurred to me) but the real mess is the bag of handle this / handle that function names. They follow the conventions of Erlang [1] but Elixir's developers could have cast some syntactic sugar on top of it. The real name of those functions is in their first argument. handle_* is mostly noise. [1] https://erlang.org/doc/man/gen_server.html https://erlang.org/doc/man/gen_server.html
- sbuttgereit 5y ago
  When I first came to Elixir & Erlang, their naming convention gave me no trouble at all. Now of course, I haven't done much for decades with languages that would have given me the mental model that would bias me to automatically be thinking of the things you name. I do know that there are certain well-worn development environments where you might gain such a bias. Remember that Erlang, too, and some of these technologies, names, and conventions, are pretty old and that they may not have been as evocative then as they might be today with some of their conventions. "Process" almost certainly would have been confusing to the neophyte, but appropriately descriptive nonetheless, but "Gen" for Generic.... eh. Anyway I guess the point is that I have to look inward to see if something like this rubbing me the wrong way is the substandard choice of the project I'm diving into, or if it's me bringing unwarranted biases and assumptions to the table.
- elsurudo 5y ago
  You got downvoted, but I agree. "Gen" is very typically used as a shorthand for "generate", whereas I've _only_ seen it used as a shorthand for "generic" in the Erlang/Elixir world.
  - dnautics 5y ago
    I have sympathy for gp. But also, naming things is hard. Wait till you dig deeper into the erlang rabbit hole and discover the undocumented gen module.
    - hazn 5y ago
      And in recent computer science (~30 years) we often pick unintuitive names for things. Examples: WebAssembly(neither web, nor assembly), Serverless (has a server, actually), JavaScript.
  - intergalplan 5y ago
    Upvoted, then downvoted to hell, haha. IDK, I don't live in Elixir and that name trips me up every damn time I need to read/write some. Everywhere else, "gen" is typically a shortening of "generate" (which I don't love either—just write the word—but it's fairly common), and "generic" rarely occurs in code at all (elsewhere related to programming, yes—in code, no) [EDIT] incidentally, as long as I'm complaining about Elixir, I've done a lot of Ruby and have no clue whatsoever why people act like Elixir is similar to it, yet constantly see "oh yeah, it's so easy for Rubyists because it's so similar". Then again, I haven't done any Phoenix with Elixir, so maybe they just mean Phoenix is Rails-like.
- enraged_camel 5y ago
  GenServer is a (behavior) module name. IMO, it doesn’t make a lot of sense to expect it to mean “generate server”. If it was a gen_server() function then sure. Or a ServerGen module - that generates servers.
  - intergalplan 5y ago
    The name makes me read it as "generate server". Nothing else about it fits with that. That's the problem.
- dudul 5y ago
  I think it is accepted in the Erlang/Elixir community that some names are pretty poor yeah. GenServer, OTP, Application. Even Supervisor to a degree, I've read a post making the case that it's more a lifecycle manager than a supervisor. Anyway, it's just about getting used to the terms and what they truly mean.
- argvargc 5y ago
  It's a terrible name, anything starting with the abbreviated prefix "Gen" is hopelessly ambiguous. Is it "Generate"? "General"? "Generic"? "Genetic"? "Genesis"? While some would certainly be less frequently encountered, all can easily be found in programming context. When seeing "GenXxxxx" I could well understand someone feeling immediate brain-fog. That said, this is one of very few criticisms I can level at Elixir, and it's a tiny part of the eco-system - literally, one single name. I wonder would it be easy to alias?
  - OkayPhysicist 5y ago
    Yup. alias GenServer, as: GenericServer Done.
jimbokun 5y ago
Since these insert events are being buffered, what happens if the process dies? Are all of those inserts lost? I feel like Erlang/Elixir is designed to handle cases like this robustly, but it's not clear to me how this code avoids losing data when a process crashes, potentially up to 5s worth of updates!
- nrmitchi 5y ago
  I'd like a solid answer to this too; it's my biggest concern with any sort of "batch online requests" functionality in any language; unsafe shutdowns can/will always happen at some point, and it feels like this is always a data loss risk.
  - dnautics 5y ago
    To some degree there's always data loss risk... A backhoe could cut a line, you could lose power to a PSU, rack, data center, a meteor could destroy the willamette valley, etc. What you want your system to provide you with is the framework to understand the risks and what the recovery looks like and when recovery is a lost cause.
- tomjakubowski 5y ago
  I think you might solve this by writing a "reliable" supervisor process which actually receives the messages, and preserves message or state history to replay for its supervised process in case it crashes. Of course this really just shoves the problem up the stack, but at least you can stick to "let it crash" when writing the supervised process, and take more care when writing the supervisor which has a smaller scope of responsibility.
- paultannenbaum 5y ago
  You can add retry logic in your genServer, but for anything but the most simple use cases, you would want to add a Supervisor and define a retry strategy. https://elixir-lang.org/getting-started/mix-otp/supervisor-and-application.html https://elixir-lang.org/getting-started/mix-otp/supervisor-a...
- 5y ago
davidw 5y ago
This is also a good reference. Even if it's Erlang, what's going on under the hood is the same: https://learnyousomeerlang.com/clients-and-servers https://learnyousomeerlang.com/clients-and-servers
JohnCurran 5y ago
This line in the opening paragraph really rubs me the wrong way: > “I'm ashamed to say most of my Elixir education has been through trial and error, figuring things out as I go along“ This attitude is so prevalent in software as if we are all supposed to be divined with programming knowledge the moment our IDE spins up. In every other industry that’s exactly how you learn: Get your hands dirty, make mistakes, and fix them. There’s a lot of things wrong with software engineering - harboring this attitude that self-taught learning is bad or shameful makes it unnecessarily worse Great write-up otherwise
- vaer-k 5y ago
  Self-learning is one thing, but uninformed trial and error based on a lack of research is another beast entirely.
- StreamBright 5y ago
  Do you think that you can install a gas turbine the wrong way start it up and fix it later?
  - selykg 5y ago
    Like most things, you build up to it. You start software development with "Hello, world!" and then you do more, like getting input from the user, storing it in a variable, then eventually you're using classes and working with objects, then you're tying together a bunch of classes and working with APIs. No one just "studied" how to create a gas turbine and wah-la, it was made. The entire process of literally everything was one learning exercise after another. We started by using rocks as tools. Here we are, having built better tools from experience and a lot of trial and error and fooling around with things. Hell, Lego are the same exact idea.
- Jtsummers 5y ago
  There's a middle ground between (relatively) blind trial and error and being divinely granted insight. Deliberate reading of documentation (be it API, language standard, books, tutorials (better ones), etc.) lets you learn without just trying things or piecing a theory together from examples (underdocumented ones that don't explain the why of their choices).
Zarathu 5y ago
> In particular, I was itching to learn more about handling concurrency in Elixir. This, of course, led me to GenServers. Might be a nitpicking here, but GenServers aren't useful for concurrency. They just manage state, and only process one message at a time. If you're using this as a cache, your reads will be bottlenecked by however quickly the GenServer can handle the read requests.
- supernintendo 5y ago
  > GenServers aren't useful for concurrency. Sure, maybe not by themselves but typically you would run many GenServers concurrently in your application as part of a supervision tree. Libraries like Broadway (and the underlying GenStage) are essentially just leveraging GenServer to make it easier to orchestrate concurrency and state synchronization across multiple processes in your application. But you could build a comparable system on your own just using GenServer and a dynamic supervisor.
- Philip-J-Fry 5y ago
  If you've got 5 gen_servers and you cast each one a message then that is concurrency. The whole point of a gen_server is that it's a separate process doing it's own thing. I can dump 1000s of things into its message queue and then do something else. It'll keep working away. It's like saying threads aren't useful for concurrency because they can only do one thing at a time.
  - nesarkvechnep 5y ago
    I would advise to not rely on casting too much because it lacks backpressure.
areichert 5y ago
In case anyone was curious what I used for the diagrams in this article, it was an awesome open source [0] tool called Excalidraw! [1] [0] https://github.com/excalidraw/excalidraw https://github.com/excalidraw/excalidraw [1] https://excalidraw.com/ https://excalidraw.com/
- muxator 5y ago
  Excalidraw is a gem. I discovered it weeks ago, and cannot but praise it. The online instance is free and works really well, plus you have the possibility of hosting your own copy and be completely independent. I also saw that there is the possibility of doing collaborative real time drawing, but still did not try it. I have seen that the exported SVGs could be simplified (lots of repeated markup), and I am thinking about giving a try to do a PR. Excalidraw is superb.
  - dang 5y ago
    This and the GP comment got me curious to look at past threads because I remember previous discussion of that project. In case anyone else is curious: Show HN: Eraser — Excalidraw-based visual meeting canvas - https://news.ycombinator.com/item?id=26330288 https://news.ycombinator.com/item?id=26330288 - March 2021 (8 comments) One Year of Excalidraw - https://news.ycombinator.com/item?id=25608336 https://news.ycombinator.com/item?id=25608336 - Jan 2021 (19 comments) Show HN: Create a Slideshow with Excalidraw - https://news.ycombinator.com/item?id=25081914 https://news.ycombinator.com/item?id=25081914 - Nov 2020 (1 comment) Excalidraw whiteboard – easily sketch diagrams with a hand-drawn feel - https://news.ycombinator.com/item?id=23525648 https://news.ycombinator.com/item?id=23525648 - June 2020 (54 comments) Building Excalidraw's P2P Collaboration Feature - https://news.ycombinator.com/item?id=22719576 https://news.ycombinator.com/item?id=22719576 - March 2020 (1 comment) End-to-end encryption in the browser - https://news.ycombinator.com/item?id=22663435 https://news.ycombinator.com/item?id=22663435 - March 2020 (107 comments) End-to-End Encryption in the Browser and how we did it in Excalidraw - https://news.ycombinator.com/item?id=22655009 https://news.ycombinator.com/item?id=22655009 - March 2020 (4 comments) Show HN: Excalidraw – Sketch Hand-Drawn Like Diagrams - https://news.ycombinator.com/item?id=22146973 https://news.ycombinator.com/item?id=22146973 - Jan 2020 (6 comments) Excalidraw – a whiteboard tool to sketch hand-drawn diagrams (excalidraw.com) - https://news.ycombinator.com/item?id=22104502 https://news.ycombinator.com/item?id=22104502 - Jan 2020 (3 comments) Excalidraw – a whiteboard tool that lets you sketch hand-drawn diagrams - https://news.ycombinator.com/item?id=22101381 https://news.ycombinator.com/item?id=22101381 - Jan 2020 (21 comments)
xutopia 5y ago
GenServer is just one of the numerous things in the Elixir ecosystem that just bring joy. Once you I got the concepts behind Ecto and GenServers everything just felt easy to model in my mind. I find it so nice.
ammanley 5y ago
Thank you so much for sharing this. I've not yet finished a decade in the industry, nor ever laid an eye on Elixir before, and this made sense pretty quickly. Great job explaining it all! On a side note, its pretty amazing to see the impact on readability and understanding that native language features have on something like async. I've worked with similar systems to the example in the past in JS/Go/etc, and while you can definitely do the same stuff presented here, it tends to be a far messier callback-hell ime. For lack of better explanation, this Elixir code just "flows" better to my eye, which I know is a completely subjective and non-helpful unsolicited piece of info. Thanks again.
- dnautics 5y ago
  Oh that's so funny, because I'm an elixir user and we all claim that genserver (internal) organization is terrible. But yes, it is way better than callback hell... It just could be better still.
  - HH_GU 5y ago
    not genserver, what will you use?
    - deleted 5y ago
      [deleted]
    - dnautics 5y ago
      There isn't anything. Dave thomas proposed a better way of organizing genservers, but I can't get behind it because his implementation is too magical and clever, but his idea for what the api should look like in the big picture is reasonable... I'm also generally in favor having everyone speak the same idioms, so it would take a very good implementation to get me off of genserver. That said you shouldn't write genservers if you can help it. Using the frameworks genservers Is the best choice, Task is a better choice for most cases.
- lostcolony 5y ago
  It flows better because it is writing synchronous code. The code inside of every process in Erlang is synchronous. You achieve concurrency by spinning up processes (spawning functions). This isn't that different to firing an async function in Javascript, or starting a goroutine, etc. And you communicate between the processes by sending async messages. Those are the primitives. Asynchronously send messages between processes that are synchronously doing stuff. Go is actually not -that- different, except sending messages is synchronous, too, unless you set a buffer on the channel (and then you have to size it). Though I agree, because Erlang/Elixir keep it simpler than that, and are dynamically typed, it's a lot cleaner to write.
namelosw 5y ago
It is very interesting to me that components in OTP like GenServer, Register, and Supervisor are surprisingly intuitive if one tries to learn and play Erlang first, but not so obvious and possibly looks artificial if people don't learn Erlang in the first place. At first, I was surprised that the Erlang processes resemble the Actor model in such a minimalistic interface. Then I play with them for a while, I realized I wanted to extract something similar to GenServers. Then when I played with GenServers, soon I was bothered by a bunch of Pids, and yearning for Registry. In the end, Supervisor is also pretty natural because things in the universe generally cannot revive themselves. In a retrospective, OTP resembles a lot of concepts we are familiar with, the Registry just looks like how the web works because we need to resolve the address problem. And vice versa: Docker, Service Discovery, and Kubernetes look like GenServer, Registry, Supervisor respectively just at different scales. It seems to me to let our systems fulfill some properties, some recurring themes are required. We'll reuse them or rediscover them one way or another.
- OkayPhysicist 5y ago
  In BEAM circles, a modification go Greenspun's 10th rule is pretty popular: "Any sufficiently complicated concurrent program in another language contains an ad hoc informally-specified bug-ridden slow implementation of half of Erlang." Erlang's been around long enough at this point that for pretty much any problem in Erlang's area of expertice you find yourself solving twice, there's a damn good existing solution somewhere in the OTP.
mulmen 5y ago
> I'm ashamed to say most of my Elixir education has been through trial and error, figuring things out as I go along. Oh, gosh. This is how I learned... everything? I feel like I just walked into homeroom without my pants on.
dmitriid 5y ago
Erlang's OTP and Elixir's wrappers on top of that is what is severely lacking from all languages claiming to provide tools for better concurrency.

272 points by cheeseblubber 5y ago

Well explained article. Good job OP!

Maybe a dumb complaint, but I hate the name. I have a really hard time not reading it as the active name "Generate Server". Like, I'd expect it to be a function that generates servers (whatever that means, since what it's dealing with isn't something I'd normally call "a server" either). It bugs the hell out of me every time I read Elixir code. One of those things I have to keep reminding myself: "OK, it's not what it looks like it is, at all... what did these misleading terms actually mean, again?"

pessimizer 5y ago
Is "general" or "generic" more misleading than "generate"?
- deleted 5y ago
  [deleted]
brobinson 5y ago
It's "generic" server: http://erlang.org/doc/man/gen_server.html http://erlang.org/doc/man/gen_server.html
- hinkley 5y ago
  The sooner you start smacking people with tribal knowledge, the sooner many people start to back away slowly. I used to interview ex-coworkers. I don't do it as much anymore because it gets old/depressing and the answers tend not to change that much. If, for the purposes of thinking about turnover, you look at a former colleague as an 'aggrieved party', we are generally somewhere in the neighborhood of mediocre at agreeing that certain things were the 'final straw', and work backward through that to actions to increase retention. One of the things that interested me about these conversations was that people tend to work chronologically backward through many of their complaints. But what was surprising was that some people would go all the way back to their first weeks. To the first straws. The warning signs they ignored. In a way this is not unlike talking to someone who just broke up with a romantic partner. And while their ex may realize that his/her problems started back with some early inconsiderate behavior that snowballed, and resolve to 'do better next time', I rarely see companies do this, unless I instigate it. Point is, these petty slights stack up, and can become a big part of someone's narrative for abandoning you (often in a huff). First impressions are important, and you dismiss them at your own peril.
  - dasil003 5y ago
    What? GP's response is terse but not impolite, and explaining what an abbreviation means is a far cry from "smacking people with tribal knowledge", it's just table stakes explanation in a field which requires precision. The only reason modern computing got where it is today is by standing on the shoulders of giants. If we go back and re-litigate every naming decision of the past several decades just to make newcomers feel welcome, all progress will slow as we collectively devolve into continuous bikeshedding.
    - hinkley 5y ago
      I'm talking about the name GenServer, not brobinson's response. There is the corollary: If we have not seen farther, it is because giants were standing on our toes. There's a huge degree of cognitive dissonance on these issues. Just because something had a reason in the past doesn't mean we have to keep doing it forever. Also "nobody understands" why people keep re-inventing wheels. It's not all hubris, at least not all the time. It's also declaring open season on those past compromises, especially the ones people bump into immediately. Especially the ones the current maintainers immediately get defensive about. Technology dies for a lot of reasons, but the apologists never seem to grasp that the apologies are a stopgap. They explain the pain, they don't cure it. Believe it or not, I'm pro-Elixir. It's just that as I learn any new technology, I start filling out proverbial bingo card boxes for the things I can predict the next person will complain about. Is that a bit cynical? It could be, but it also serves as my todo list for when someone makes an offhand comment about how you really don't have to do this dumb thing, you could use this library that does it for you. Now my coworkers either don't have to worry about that or I have a suggestion when they do. Turns out when people are in pain, they appreciate sympathy a lot more than they appreciate being gaslit about how it's just their poor sense of history. Their blatantly implied selfishness. GenServer could use a new name. I think we forget sometimes that you can rename old things by giving them a second, better name and phasing out the old one. I don't think there's anything seditious in that statement.
      - pmontra 5y ago
        I've been using Elixir for 4 years, it pays my bills and I generally like it. GenServer could be a misleading name (it never occurred to me) but the real mess is the bag of handle this / handle that function names. They follow the conventions of Erlang [1] but Elixir's developers could have cast some syntactic sugar on top of it. The real name of those functions is in their first argument. handle_* is mostly noise. [1] https://erlang.org/doc/man/gen_server.html https://erlang.org/doc/man/gen_server.html
sbuttgereit 5y ago
When I first came to Elixir & Erlang, their naming convention gave me no trouble at all. Now of course, I haven't done much for decades with languages that would have given me the mental model that would bias me to automatically be thinking of the things you name. I do know that there are certain well-worn development environments where you might gain such a bias. Remember that Erlang, too, and some of these technologies, names, and conventions, are pretty old and that they may not have been as evocative then as they might be today with some of their conventions. "Process" almost certainly would have been confusing to the neophyte, but appropriately descriptive nonetheless, but "Gen" for Generic.... eh. Anyway I guess the point is that I have to look inward to see if something like this rubbing me the wrong way is the substandard choice of the project I'm diving into, or if it's me bringing unwarranted biases and assumptions to the table.
elsurudo 5y ago
You got downvoted, but I agree. "Gen" is very typically used as a shorthand for "generate", whereas I've _only_ seen it used as a shorthand for "generic" in the Erlang/Elixir world.
- dnautics 5y ago
  I have sympathy for gp. But also, naming things is hard. Wait till you dig deeper into the erlang rabbit hole and discover the undocumented gen module.
  - hazn 5y ago
    And in recent computer science (~30 years) we often pick unintuitive names for things. Examples: WebAssembly(neither web, nor assembly), Serverless (has a server, actually), JavaScript.
- intergalplan 5y ago
  Upvoted, then downvoted to hell, haha. IDK, I don't live in Elixir and that name trips me up every damn time I need to read/write some. Everywhere else, "gen" is typically a shortening of "generate" (which I don't love either—just write the word—but it's fairly common), and "generic" rarely occurs in code at all (elsewhere related to programming, yes—in code, no) [EDIT] incidentally, as long as I'm complaining about Elixir, I've done a lot of Ruby and have no clue whatsoever why people act like Elixir is similar to it, yet constantly see "oh yeah, it's so easy for Rubyists because it's so similar". Then again, I haven't done any Phoenix with Elixir, so maybe they just mean Phoenix is Rails-like.
enraged_camel 5y ago
GenServer is a (behavior) module name. IMO, it doesn’t make a lot of sense to expect it to mean “generate server”. If it was a gen_server() function then sure. Or a ServerGen module - that generates servers.
- intergalplan 5y ago
  The name makes me read it as "generate server". Nothing else about it fits with that. That's the problem.
dudul 5y ago
I think it is accepted in the Erlang/Elixir community that some names are pretty poor yeah. GenServer, OTP, Application. Even Supervisor to a degree, I've read a post making the case that it's more a lifecycle manager than a supervisor. Anyway, it's just about getting used to the terms and what they truly mean.
argvargc 5y ago
It's a terrible name, anything starting with the abbreviated prefix "Gen" is hopelessly ambiguous. Is it "Generate"? "General"? "Generic"? "Genetic"? "Genesis"? While some would certainly be less frequently encountered, all can easily be found in programming context. When seeing "GenXxxxx" I could well understand someone feeling immediate brain-fog. That said, this is one of very few criticisms I can level at Elixir, and it's a tiny part of the eco-system - literally, one single name. I wonder would it be easy to alias?
- OkayPhysicist 5y ago
  Yup. alias GenServer, as: GenericServer Done.

jimbokun 5y ago

Since these insert events are being buffered, what happens if the process dies? Are all of those inserts lost? I feel like Erlang/Elixir is designed to handle cases like this robustly, but it's not clear to me how this code avoids losing data when a process crashes, potentially up to 5s worth of updates!

nrmitchi 5y ago
I'd like a solid answer to this too; it's my biggest concern with any sort of "batch online requests" functionality in any language; unsafe shutdowns can/will always happen at some point, and it feels like this is always a data loss risk.
- dnautics 5y ago
  To some degree there's always data loss risk... A backhoe could cut a line, you could lose power to a PSU, rack, data center, a meteor could destroy the willamette valley, etc. What you want your system to provide you with is the framework to understand the risks and what the recovery looks like and when recovery is a lost cause.
tomjakubowski 5y ago
I think you might solve this by writing a "reliable" supervisor process which actually receives the messages, and preserves message or state history to replay for its supervised process in case it crashes. Of course this really just shoves the problem up the stack, but at least you can stick to "let it crash" when writing the supervised process, and take more care when writing the supervisor which has a smaller scope of responsibility.
paultannenbaum 5y ago
You can add retry logic in your genServer, but for anything but the most simple use cases, you would want to add a Supervisor and define a retry strategy. https://elixir-lang.org/getting-started/mix-otp/supervisor-and-application.html https://elixir-lang.org/getting-started/mix-otp/supervisor-a...
5y ago

davidw 5y ago

This is also a good reference. Even if it's Erlang, what's going on under the hood is the same: https://learnyousomeerlang.com/clients-and-servers https://learnyousomeerlang.com/clients-and-servers

JohnCurran 5y ago

This line in the opening paragraph really rubs me the wrong way: > “I'm ashamed to say most of my Elixir education has been through trial and error, figuring things out as I go along“ This attitude is so prevalent in software as if we are all supposed to be divined with programming knowledge the moment our IDE spins up. In every other industry that’s exactly how you learn: Get your hands dirty, make mistakes, and fix them. There’s a lot of things wrong with software engineering - harboring this attitude that self-taught learning is bad or shameful makes it unnecessarily worse Great write-up otherwise

vaer-k 5y ago
Self-learning is one thing, but uninformed trial and error based on a lack of research is another beast entirely.
StreamBright 5y ago
Do you think that you can install a gas turbine the wrong way start it up and fix it later?
- selykg 5y ago
  Like most things, you build up to it. You start software development with "Hello, world!" and then you do more, like getting input from the user, storing it in a variable, then eventually you're using classes and working with objects, then you're tying together a bunch of classes and working with APIs. No one just "studied" how to create a gas turbine and wah-la, it was made. The entire process of literally everything was one learning exercise after another. We started by using rocks as tools. Here we are, having built better tools from experience and a lot of trial and error and fooling around with things. Hell, Lego are the same exact idea.
Jtsummers 5y ago
There's a middle ground between (relatively) blind trial and error and being divinely granted insight. Deliberate reading of documentation (be it API, language standard, books, tutorials (better ones), etc.) lets you learn without just trying things or piecing a theory together from examples (underdocumented ones that don't explain the why of their choices).

Zarathu 5y ago

> In particular, I was itching to learn more about handling concurrency in Elixir. This, of course, led me to GenServers. Might be a nitpicking here, but GenServers aren't useful for concurrency. They just manage state, and only process one message at a time. If you're using this as a cache, your reads will be bottlenecked by however quickly the GenServer can handle the read requests.

supernintendo 5y ago
> GenServers aren't useful for concurrency. Sure, maybe not by themselves but typically you would run many GenServers concurrently in your application as part of a supervision tree. Libraries like Broadway (and the underlying GenStage) are essentially just leveraging GenServer to make it easier to orchestrate concurrency and state synchronization across multiple processes in your application. But you could build a comparable system on your own just using GenServer and a dynamic supervisor.
Philip-J-Fry 5y ago
If you've got 5 gen_servers and you cast each one a message then that is concurrency. The whole point of a gen_server is that it's a separate process doing it's own thing. I can dump 1000s of things into its message queue and then do something else. It'll keep working away. It's like saying threads aren't useful for concurrency because they can only do one thing at a time.
- nesarkvechnep 5y ago
  I would advise to not rely on casting too much because it lacks backpressure.

areichert 5y ago

In case anyone was curious what I used for the diagrams in this article, it was an awesome open source [0] tool called Excalidraw! [1] [0] https://github.com/excalidraw/excalidraw https://github.com/excalidraw/excalidraw [1] https://excalidraw.com/ https://excalidraw.com/

muxator 5y ago
Excalidraw is a gem. I discovered it weeks ago, and cannot but praise it. The online instance is free and works really well, plus you have the possibility of hosting your own copy and be completely independent. I also saw that there is the possibility of doing collaborative real time drawing, but still did not try it. I have seen that the exported SVGs could be simplified (lots of repeated markup), and I am thinking about giving a try to do a PR. Excalidraw is superb.
- dang 5y ago
  This and the GP comment got me curious to look at past threads because I remember previous discussion of that project. In case anyone else is curious: Show HN: Eraser — Excalidraw-based visual meeting canvas - https://news.ycombinator.com/item?id=26330288 https://news.ycombinator.com/item?id=26330288 - March 2021 (8 comments) One Year of Excalidraw - https://news.ycombinator.com/item?id=25608336 https://news.ycombinator.com/item?id=25608336 - Jan 2021 (19 comments) Show HN: Create a Slideshow with Excalidraw - https://news.ycombinator.com/item?id=25081914 https://news.ycombinator.com/item?id=25081914 - Nov 2020 (1 comment) Excalidraw whiteboard – easily sketch diagrams with a hand-drawn feel - https://news.ycombinator.com/item?id=23525648 https://news.ycombinator.com/item?id=23525648 - June 2020 (54 comments) Building Excalidraw's P2P Collaboration Feature - https://news.ycombinator.com/item?id=22719576 https://news.ycombinator.com/item?id=22719576 - March 2020 (1 comment) End-to-end encryption in the browser - https://news.ycombinator.com/item?id=22663435 https://news.ycombinator.com/item?id=22663435 - March 2020 (107 comments) End-to-End Encryption in the Browser and how we did it in Excalidraw - https://news.ycombinator.com/item?id=22655009 https://news.ycombinator.com/item?id=22655009 - March 2020 (4 comments) Show HN: Excalidraw – Sketch Hand-Drawn Like Diagrams - https://news.ycombinator.com/item?id=22146973 https://news.ycombinator.com/item?id=22146973 - Jan 2020 (6 comments) Excalidraw – a whiteboard tool to sketch hand-drawn diagrams (excalidraw.com) - https://news.ycombinator.com/item?id=22104502 https://news.ycombinator.com/item?id=22104502 - Jan 2020 (3 comments) Excalidraw – a whiteboard tool that lets you sketch hand-drawn diagrams - https://news.ycombinator.com/item?id=22101381 https://news.ycombinator.com/item?id=22101381 - Jan 2020 (21 comments)

xutopia 5y ago

GenServer is just one of the numerous things in the Elixir ecosystem that just bring joy. Once you I got the concepts behind Ecto and GenServers everything just felt easy to model in my mind. I find it so nice.

ammanley 5y ago

Thank you so much for sharing this. I've not yet finished a decade in the industry, nor ever laid an eye on Elixir before, and this made sense pretty quickly. Great job explaining it all! On a side note, its pretty amazing to see the impact on readability and understanding that native language features have on something like async. I've worked with similar systems to the example in the past in JS/Go/etc, and while you can definitely do the same stuff presented here, it tends to be a far messier callback-hell ime. For lack of better explanation, this Elixir code just "flows" better to my eye, which I know is a completely subjective and non-helpful unsolicited piece of info. Thanks again.

dnautics 5y ago
Oh that's so funny, because I'm an elixir user and we all claim that genserver (internal) organization is terrible. But yes, it is way better than callback hell... It just could be better still.
- HH_GU 5y ago
  not genserver, what will you use?
  - deleted 5y ago
    [deleted]
  - dnautics 5y ago
    There isn't anything. Dave thomas proposed a better way of organizing genservers, but I can't get behind it because his implementation is too magical and clever, but his idea for what the api should look like in the big picture is reasonable... I'm also generally in favor having everyone speak the same idioms, so it would take a very good implementation to get me off of genserver. That said you shouldn't write genservers if you can help it. Using the frameworks genservers Is the best choice, Task is a better choice for most cases.
lostcolony 5y ago
It flows better because it is writing synchronous code. The code inside of every process in Erlang is synchronous. You achieve concurrency by spinning up processes (spawning functions). This isn't that different to firing an async function in Javascript, or starting a goroutine, etc. And you communicate between the processes by sending async messages. Those are the primitives. Asynchronously send messages between processes that are synchronously doing stuff. Go is actually not -that- different, except sending messages is synchronous, too, unless you set a buffer on the channel (and then you have to size it). Though I agree, because Erlang/Elixir keep it simpler than that, and are dynamically typed, it's a lot cleaner to write.

namelosw 5y ago

It is very interesting to me that components in OTP like GenServer, Register, and Supervisor are surprisingly intuitive if one tries to learn and play Erlang first, but not so obvious and possibly looks artificial if people don't learn Erlang in the first place. At first, I was surprised that the Erlang processes resemble the Actor model in such a minimalistic interface. Then I play with them for a while, I realized I wanted to extract something similar to GenServers. Then when I played with GenServers, soon I was bothered by a bunch of Pids, and yearning for Registry. In the end, Supervisor is also pretty natural because things in the universe generally cannot revive themselves. In a retrospective, OTP resembles a lot of concepts we are familiar with, the Registry just looks like how the web works because we need to resolve the address problem. And vice versa: Docker, Service Discovery, and Kubernetes look like GenServer, Registry, Supervisor respectively just at different scales. It seems to me to let our systems fulfill some properties, some recurring themes are required. We'll reuse them or rediscover them one way or another.

OkayPhysicist 5y ago
In BEAM circles, a modification go Greenspun's 10th rule is pretty popular: "Any sufficiently complicated concurrent program in another language contains an ad hoc informally-specified bug-ridden slow implementation of half of Erlang." Erlang's been around long enough at this point that for pretty much any problem in Erlang's area of expertice you find yourself solving twice, there's a damn good existing solution somewhere in the OTP.

mulmen 5y ago

> I'm ashamed to say most of my Elixir education has been through trial and error, figuring things out as I go along. Oh, gosh. This is how I learned... everything? I feel like I just walked into homeroom without my pants on.

dmitriid 5y ago

Erlang's OTP and Elixir's wrappers on top of that is what is severely lacking from all languages claiming to provide tools for better concurrency.