8 ms·

Phoenix is practically built with this idea in mind. My startup is built out of a single phoenix monolith. Only external dependency is postgres. Despite this,

Phoenix is practically built with this idea in mind. My startup is built out of a single phoenix monolith. Only external dependency is postgres. Despite this, I've managed to completely avoid all the typical scaling issues of a monolith. Need a service or background worker? I juts make a Genserver or Oban worker and mount it in the supervision tree. The Beam takes care of maintaining the process, pubsub to said service and error handling in the event of a crash. I'm basicly getting all the best advantages of a monolith * one codebase to navigate, our team is TINY * all functionality is in libraries And all the advantages of microservices without the associated costs. (orchestration, management etc) * bottleneck use cases can live on their own nodes * publish subscribe to dispatch background jobs * services that each focus on doing one thing really well We've accomplished this all through just libraries and a little bit of configuration. At no point so far have we had to.. * setup a message broker (phoenix pubsub has been fine for us. we can send messages between different processes on the server and to the clients themselves over channels) * setup external clusters for background workers. (Oban was drop in and jsut works. I'm able to add a new worker without having to tell devops anything. the supervision tree allocates the memory for the process and takes care of fault tolerance for me. That leaves me to focus on solving the business problems) * setup a second github repo. Its a monorepo and at out scale, its fine. We have one library for communicating to the database that every system just uses. Eventually we'll probably have to start rethinking and building out separate services. But I'm happy that we're already able to get some of teh benefits of a microservice architecture while sticking to what makes monoliths great for mvps. It will be awhile before we need to think about scaling out web service. It just works. Leaves me more time to work on tuning out database to keep up!

fiddlerwoaroof 6y ago
BEAM will automatically distribute processes across a cluster, right?
- cultofmetatron 6y ago
  not automatically but its pretty easy to configure in your supervision tree file. I don't know the details because whatever happens by default has taken care of our needs so far. IT does automatically distribute processes across all the cores on a cpu though.
  - fiddlerwoaroof 6y ago
    What I like about the Erlang platform is that it seems like it has the most sensible “microservice” story: deploy your language runtime to all the nodes of a cluster and then configure distribution in code. Lambdas, containers, etc. all push this stuff outside your code into deployment tooling that is, inevitably, less pleasant to manage than your codebase.
- jorgeavaldez 6y ago
  No, typically you register a node and instruct it on what processes to run. But there are libraries to help instrument this kind of behavior. For elixir: - https://github.com/derekkraan/horde https://github.com/derekkraan/horde - https://github.com/bitwalker/swarm https://github.com/bitwalker/swarm
  - tonyarkles 6y ago
    A zero-cost alternative that has worked well for me so far is to use a front-end load balancer to distribute requests to multiple Phoenix instances (in k8s), and then just let those requests' background tasks run on the node that starts them. The whole app is approximately a websocket-based chat app (with some other stuff), and the beauty of OTP + libcluster is that the websocket processes can communicate with each other, whether or not they're running on the same OTP node.
wcarss 6y ago
this sounds really neat and I'm going to go read about it! I also wanted to call out that this ~sentence has just... a bunch of things that seem like jargon/names within the community? As an outsider, I have no idea what they mean: > make a Genserver or Oban worker and mount it in the supervision tree. The Beam takes care of maintaining the process it sounds very sci-fi :)
- jorgeavaldez 6y ago
  They do sound very sci-fi. I would say it made me more or less inclined to dig a little deeper myself hahahaha. Genservers are an abstraction encapsulating the typical request/response lifecycle of what we would consider a "server", but applied to a BEAM-specific process. Like "general server". Oban allows for job processing, instrumented much like you would any other process in the BEAM. This is an external library while genserver is built in.
- cultofmetatron 6y ago
  sorry! so I'll clarify Elixir inherits a library called OTP from erlang which is a set of primitives for building massively concurrent systems. A Genserver is sort of like a Base Class for a object that runs as an independant process that plugs into OTP. By inheriting/implementing the Genserver behavior, you create an independant process that can be mounted in an OTP supervision tree which runs at the top of your application and monitors everything below it. Out of the box, that means your process can be sent messages by other processes and can send messages itself. If a crash happens, the supervisor will kill it, resurrect it and redirect messages to the new process. Creating a Genserver is as easy as adding an annotation and implementing a few callbacks. Genservers are the base on which a lot of other systems build on. Oban, a job worker essentially builds on Genserver to use a postgres table as a job processor. Since its just a Genserver with some added behavior, adding a background worker is as simple as adding a file that inherits from Oban and specifying how many workers for it should be allocated in a config file. The result is that adding a background worker is about as much work for me as adding a controller. No additional work for devops either. And yes, it is very sci fi. Honestly I'm shocked elixir isn't more widespread. there's very little hype behind it but the engineering is pretty solid. Every scaling bottleneck we've had so far has been in the database (only because we particularly make heavy use of stored procedures)
travisjungroth 6y ago
Anyone like me first hearing about Phoenix and had trouble finding it, it's an Elixir framework: https://www.phoenixframework.org/ https://www.phoenixframework.org/
- PoignardAzur 6y ago
  Yeah, my first thought was "If this is called Phoenix and it's Cloud-related, I'm gonna have to swim through a lot of FFVII results before I find it".
  - hajhatten 6y ago
    I read your comment and halfway through i thought you were gonna bring up the devops book: The Phoenix Project https://www.amazon.com/Phoenix-Project-DevOps-Helping-Business/dp/0988262592 https://www.amazon.com/Phoenix-Project-DevOps-Helping-Busine...
bitexploder 6y ago
We have a similar philosophy with our Python code base. Redis and Postgres are our only real dependencies. We use celery but it’s basically just Python. Maybe takes a little more work setting it all up, but once done provides similar benefits.
whycombagator 6y ago
> Leaves me more time to work on tuning out database to keep up! As your using postgres have you looked at citus[0] at all? [0] https://github.com/citusdata/citus https://github.com/citusdata/citus
- cultofmetatron 6y ago
  I hadn't heard of this but it looks cool. Looks like something that will do the job when we do need it. so far we just store the parameters from certain heavily used mutations into a job table and run the actual insertion in a background worker. We're no where near needing this yet but it'll be good to have it on hand when we (hopefully) get to that point.
dgellow 6y ago
> And all the advantages of microservices without the associated costs. (orchestration, management etc) Really curious about this! What’s the deployment experience? What environment do you use for your production? How do you run/maintain the erlang VM and deploy your service?
- cultofmetatron 6y ago
  We use eks with some customizations specific to elixir and use a autodiscovery service for new nodes to connect to the mesh. One of the big differences we had to make was allocating one node per machine as the beam likes to have access to all resources. in practice this isn't a problem because its internal scheduler is way more efficient and performant than anything managing OS level processes. That said, a more complex deployemnt story is defiantly one of the downsides. But the good news is that once setup, its pretty damn resilient. Now of course, our deployment setup is more complex specifically to take advantage of beam such as distributed pubsub and shared memory between the cluster. If you don't need that, you could use dokku or heroku.
- valzam 6y ago
  To add to this, we use Phoenix in a typical dockerized, stateless Loadbalancer- X Webserver - Postgres/Redis setup and it works great. Deployment is exactly the same as any other dockerized webapp. What OP is using is the "next level" that allows you to really leverage the BEAM but you don't have to.
acjohnson55 6y ago
In my experience, this is also a major benefit of running Akka on Scala or Java. I had the realization that it's basically a single-language Kubernetes, with some really nice abstractions built on top.
- linkdd 6y ago
  > it's basically a single-language Kubernetes Well, yes and no. By using Kubernetes, you get a scalable infrastructure. By using OTP/Akka, you get a scalable application. While there is common problems to both domains, they are still 2 different domains. For example, using only Kubernetes, you won't have the ability to react to a Pod restart within your application (unless your application is aware of Kubernetes). Using only OTP/Akka, you still need a workflow for deployment and infrastructure management, and you still need to implement node discover for clustering. NB: For Elixir, you have libcluster[1] that can use different strategies for node discovery, including using a Kubernetes Service to discover the nodes (as Pods). EDIT: Using Kubernetes, libcluster, and Horde[2], you get the best of both worlds IMHO. [1] - https://hexdocs.pm/libcluster/2.5.0/readme.html https://hexdocs.pm/libcluster/2.5.0/readme.html [2] - https://hexdocs.pm/horde/0.1.4/api-reference.html https://hexdocs.pm/horde/0.1.4/api-reference.html
  - tonyarkles 6y ago
    Thanks! I was just going to point out that Elixir/Phoenix + Libcluster + K8s is like a match made in heaven. I haven't tried Horde yet but I'm quite intrigued now.
    - linkdd 6y ago
      Horde uses a CRDT[1] (Conflict-Free Replicated Data Type) to provide a distributed Supervisor/Registry, this allows you to run an OTP supervision tree only once in your cluster, with automatic takeover. Basically, you run your supervisor on each nodes, but only one will run it (thanks to the CRDT). I find it very useful because "Distributed OTP Application" are a pain IMHO (they must be started during the boot of the BEAM). [1] - https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type https://en.wikipedia.org/wiki/Conflict-free_replicated_data_...
machiaweliczny 6y ago
I can recommend watching this[0] to understand BEAM design better. [0] - https://www.youtube.com/watch?v=JvBT4XBdoUE https://www.youtube.com/watch?v=JvBT4XBdoUE
- whiskeytuesday 6y ago
  +1, probably my favourite talk on the BEAM too, even allowing for the fact that listening to Joe Armstrong himself was always a distinct pleasure.
  - mrlucax 6y ago
    Really great talk! He talks about some problems about the BEAM Distribution, but didn't get into details about it. Do you have any idea about those problems?
ngrilly 6y ago
How do you keep track of background jobs? Is the queue persisted on disk somewhere? If it's not and a background worker crashes for some reason, then is the job lost?
- valzam 6y ago
  Oban uses Postgres for persistence https://hexdocs.pm/oban/Oban.html https://hexdocs.pm/oban/Oban.html
  - ngrilly 6y ago
    That’s a very nice library. BEAM + Elixir + OTP supervision trees + Obama for background jobs + PostgreSQL for persistent storage seems a killer combination. And like you said you can scale horizontally naturally with the Erlang plateform without even using k8s. Really interesting.
qertoip 6y ago
Agree, we did similar with Elixir/Phoenix in my past company. Obviously, the services will still be necessary where Elixir simply isn't the right tool for the job like machine learning or cryptocurrency.
- cultofmetatron 6y ago
  the new numbat library https://dashbit.co/blog/nx-numerical-elixir-is-now-publicly-available https://dashbit.co/blog/nx-numerical-elixir-is-now-publicly-... exposes primatives for gpu acccellerated math powered by google's xla library. the same one that powers tensorflow.
frobisher 6y ago
Great stuff! We moved to microservices, despite my love for a monolith with libraries: - to enable different teams to deploy their smaller microservice more easily (without QA, database migrations etc affecting the whole app); - to solve the human temptation of crossing abstraction boundaries. Not black and white, and both approaches can solve these problems. Open to any feedback!
- doctor_eval 6y ago
  Yeah I moved to microservices for the exact same reasons. People are undisciplined, and making it harder (and much more obvious) for them to do the wrong thing is more important than having all your code in one place. Plus, if you need to upgrade some dependency or if you want to try a new language or library or idiom, you can do so without the risk, effort and sunk cost of upgrading the entire application.
twic 6y ago
I have done much the same in plain old Java, at a couple of jobs, going back ten years or so. No OTP, GenServer, etc. Just a webserver and some framework for scheduled jobs, third party or write your own. Config enables individual routes and jobs at the top level. You can deploy one instance with everything enabled, or a hundred instances with the customer-facing web routes enabled in 60, API routes in 20, admin routes in 5, report generation jobs in 10, maintenance jobs in 5, or anywhere in between. The only discipline you have to stick to is that you must pass information between components in a way which will work when the app is distributed. A shared database is the most obvious, and may be enough. We also used message queues, and at one point one of those distributed caches that were all the rage (Hazelcast / Terracotta / Infinispan / EHCache / etc - anyone remember JavaSpaces and Jini?).
damethos 6y ago
This was possible due to the fact that BEAM does allow you to do things like the ones you describe and enjoy advantages of both worlds (monolithic and microservices). If someone does not use Elixir/Erlang however or if the the product consists of parts written in multiple languages (for whatever reasons) then it's simply not possible to have the advantages of microservices in a monolithic approach.
- OkayPhysicist 6y ago
  Sounds like an argument for using Elixir.

fiddlerwoaroof 6y ago

BEAM will automatically distribute processes across a cluster, right?

cultofmetatron 6y ago
not automatically but its pretty easy to configure in your supervision tree file. I don't know the details because whatever happens by default has taken care of our needs so far. IT does automatically distribute processes across all the cores on a cpu though.
- fiddlerwoaroof 6y ago
  What I like about the Erlang platform is that it seems like it has the most sensible “microservice” story: deploy your language runtime to all the nodes of a cluster and then configure distribution in code. Lambdas, containers, etc. all push this stuff outside your code into deployment tooling that is, inevitably, less pleasant to manage than your codebase.
jorgeavaldez 6y ago
No, typically you register a node and instruct it on what processes to run. But there are libraries to help instrument this kind of behavior. For elixir: - https://github.com/derekkraan/horde https://github.com/derekkraan/horde - https://github.com/bitwalker/swarm https://github.com/bitwalker/swarm
- tonyarkles 6y ago
  A zero-cost alternative that has worked well for me so far is to use a front-end load balancer to distribute requests to multiple Phoenix instances (in k8s), and then just let those requests' background tasks run on the node that starts them. The whole app is approximately a websocket-based chat app (with some other stuff), and the beauty of OTP + libcluster is that the websocket processes can communicate with each other, whether or not they're running on the same OTP node.

wcarss 6y ago

this sounds really neat and I'm going to go read about it! I also wanted to call out that this ~sentence has just... a bunch of things that seem like jargon/names within the community? As an outsider, I have no idea what they mean: > make a Genserver or Oban worker and mount it in the supervision tree. The Beam takes care of maintaining the process it sounds very sci-fi :)

jorgeavaldez 6y ago
They do sound very sci-fi. I would say it made me more or less inclined to dig a little deeper myself hahahaha. Genservers are an abstraction encapsulating the typical request/response lifecycle of what we would consider a "server", but applied to a BEAM-specific process. Like "general server". Oban allows for job processing, instrumented much like you would any other process in the BEAM. This is an external library while genserver is built in.
cultofmetatron 6y ago
sorry! so I'll clarify Elixir inherits a library called OTP from erlang which is a set of primitives for building massively concurrent systems. A Genserver is sort of like a Base Class for a object that runs as an independant process that plugs into OTP. By inheriting/implementing the Genserver behavior, you create an independant process that can be mounted in an OTP supervision tree which runs at the top of your application and monitors everything below it. Out of the box, that means your process can be sent messages by other processes and can send messages itself. If a crash happens, the supervisor will kill it, resurrect it and redirect messages to the new process. Creating a Genserver is as easy as adding an annotation and implementing a few callbacks. Genservers are the base on which a lot of other systems build on. Oban, a job worker essentially builds on Genserver to use a postgres table as a job processor. Since its just a Genserver with some added behavior, adding a background worker is as simple as adding a file that inherits from Oban and specifying how many workers for it should be allocated in a config file. The result is that adding a background worker is about as much work for me as adding a controller. No additional work for devops either. And yes, it is very sci fi. Honestly I'm shocked elixir isn't more widespread. there's very little hype behind it but the engineering is pretty solid. Every scaling bottleneck we've had so far has been in the database (only because we particularly make heavy use of stored procedures)

travisjungroth 6y ago

Anyone like me first hearing about Phoenix and had trouble finding it, it's an Elixir framework: https://www.phoenixframework.org/ https://www.phoenixframework.org/

PoignardAzur 6y ago
Yeah, my first thought was "If this is called Phoenix and it's Cloud-related, I'm gonna have to swim through a lot of FFVII results before I find it".
- hajhatten 6y ago
  I read your comment and halfway through i thought you were gonna bring up the devops book: The Phoenix Project https://www.amazon.com/Phoenix-Project-DevOps-Helping-Business/dp/0988262592 https://www.amazon.com/Phoenix-Project-DevOps-Helping-Busine...

bitexploder 6y ago

We have a similar philosophy with our Python code base. Redis and Postgres are our only real dependencies. We use celery but it’s basically just Python. Maybe takes a little more work setting it all up, but once done provides similar benefits.

whycombagator 6y ago

> Leaves me more time to work on tuning out database to keep up! As your using postgres have you looked at citus[0] at all? [0] https://github.com/citusdata/citus https://github.com/citusdata/citus

cultofmetatron 6y ago
I hadn't heard of this but it looks cool. Looks like something that will do the job when we do need it. so far we just store the parameters from certain heavily used mutations into a job table and run the actual insertion in a background worker. We're no where near needing this yet but it'll be good to have it on hand when we (hopefully) get to that point.

dgellow 6y ago

> And all the advantages of microservices without the associated costs. (orchestration, management etc) Really curious about this! What’s the deployment experience? What environment do you use for your production? How do you run/maintain the erlang VM and deploy your service?

cultofmetatron 6y ago
We use eks with some customizations specific to elixir and use a autodiscovery service for new nodes to connect to the mesh. One of the big differences we had to make was allocating one node per machine as the beam likes to have access to all resources. in practice this isn't a problem because its internal scheduler is way more efficient and performant than anything managing OS level processes. That said, a more complex deployemnt story is defiantly one of the downsides. But the good news is that once setup, its pretty damn resilient. Now of course, our deployment setup is more complex specifically to take advantage of beam such as distributed pubsub and shared memory between the cluster. If you don't need that, you could use dokku or heroku.
valzam 6y ago
To add to this, we use Phoenix in a typical dockerized, stateless Loadbalancer- X Webserver - Postgres/Redis setup and it works great. Deployment is exactly the same as any other dockerized webapp. What OP is using is the "next level" that allows you to really leverage the BEAM but you don't have to.

acjohnson55 6y ago

In my experience, this is also a major benefit of running Akka on Scala or Java. I had the realization that it's basically a single-language Kubernetes, with some really nice abstractions built on top.

linkdd 6y ago
> it's basically a single-language Kubernetes Well, yes and no. By using Kubernetes, you get a scalable infrastructure. By using OTP/Akka, you get a scalable application. While there is common problems to both domains, they are still 2 different domains. For example, using only Kubernetes, you won't have the ability to react to a Pod restart within your application (unless your application is aware of Kubernetes). Using only OTP/Akka, you still need a workflow for deployment and infrastructure management, and you still need to implement node discover for clustering. NB: For Elixir, you have libcluster[1] that can use different strategies for node discovery, including using a Kubernetes Service to discover the nodes (as Pods). EDIT: Using Kubernetes, libcluster, and Horde[2], you get the best of both worlds IMHO. [1] - https://hexdocs.pm/libcluster/2.5.0/readme.html https://hexdocs.pm/libcluster/2.5.0/readme.html [2] - https://hexdocs.pm/horde/0.1.4/api-reference.html https://hexdocs.pm/horde/0.1.4/api-reference.html
- tonyarkles 6y ago
  Thanks! I was just going to point out that Elixir/Phoenix + Libcluster + K8s is like a match made in heaven. I haven't tried Horde yet but I'm quite intrigued now.
  - linkdd 6y ago
    Horde uses a CRDT[1] (Conflict-Free Replicated Data Type) to provide a distributed Supervisor/Registry, this allows you to run an OTP supervision tree only once in your cluster, with automatic takeover. Basically, you run your supervisor on each nodes, but only one will run it (thanks to the CRDT). I find it very useful because "Distributed OTP Application" are a pain IMHO (they must be started during the boot of the BEAM). [1] - https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type https://en.wikipedia.org/wiki/Conflict-free_replicated_data_...

machiaweliczny 6y ago

I can recommend watching this[0] to understand BEAM design better. [0] - https://www.youtube.com/watch?v=JvBT4XBdoUE https://www.youtube.com/watch?v=JvBT4XBdoUE

whiskeytuesday 6y ago
+1, probably my favourite talk on the BEAM too, even allowing for the fact that listening to Joe Armstrong himself was always a distinct pleasure.
- mrlucax 6y ago
  Really great talk! He talks about some problems about the BEAM Distribution, but didn't get into details about it. Do you have any idea about those problems?

ngrilly 6y ago

How do you keep track of background jobs? Is the queue persisted on disk somewhere? If it's not and a background worker crashes for some reason, then is the job lost?

valzam 6y ago
Oban uses Postgres for persistence https://hexdocs.pm/oban/Oban.html https://hexdocs.pm/oban/Oban.html
- ngrilly 6y ago
  That’s a very nice library. BEAM + Elixir + OTP supervision trees + Obama for background jobs + PostgreSQL for persistent storage seems a killer combination. And like you said you can scale horizontally naturally with the Erlang plateform without even using k8s. Really interesting.

qertoip 6y ago

Agree, we did similar with Elixir/Phoenix in my past company. Obviously, the services will still be necessary where Elixir simply isn't the right tool for the job like machine learning or cryptocurrency.

cultofmetatron 6y ago
the new numbat library https://dashbit.co/blog/nx-numerical-elixir-is-now-publicly-available https://dashbit.co/blog/nx-numerical-elixir-is-now-publicly-... exposes primatives for gpu acccellerated math powered by google's xla library. the same one that powers tensorflow.

frobisher 6y ago

Great stuff! We moved to microservices, despite my love for a monolith with libraries: - to enable different teams to deploy their smaller microservice more easily (without QA, database migrations etc affecting the whole app); - to solve the human temptation of crossing abstraction boundaries. Not black and white, and both approaches can solve these problems. Open to any feedback!

doctor_eval 6y ago
Yeah I moved to microservices for the exact same reasons. People are undisciplined, and making it harder (and much more obvious) for them to do the wrong thing is more important than having all your code in one place. Plus, if you need to upgrade some dependency or if you want to try a new language or library or idiom, you can do so without the risk, effort and sunk cost of upgrading the entire application.

twic 6y ago

I have done much the same in plain old Java, at a couple of jobs, going back ten years or so. No OTP, GenServer, etc. Just a webserver and some framework for scheduled jobs, third party or write your own. Config enables individual routes and jobs at the top level. You can deploy one instance with everything enabled, or a hundred instances with the customer-facing web routes enabled in 60, API routes in 20, admin routes in 5, report generation jobs in 10, maintenance jobs in 5, or anywhere in between. The only discipline you have to stick to is that you must pass information between components in a way which will work when the app is distributed. A shared database is the most obvious, and may be enough. We also used message queues, and at one point one of those distributed caches that were all the rage (Hazelcast / Terracotta / Infinispan / EHCache / etc - anyone remember JavaSpaces and Jini?).

damethos 6y ago

This was possible due to the fact that BEAM does allow you to do things like the ones you describe and enjoy advantages of both worlds (monolithic and microservices). If someone does not use Elixir/Erlang however or if the the product consists of parts written in multiple languages (for whatever reasons) then it's simply not possible to have the advantages of microservices in a monolithic approach.

OkayPhysicist 6y ago
Sounds like an argument for using Elixir.