11 ms·
Elixir on Google Cloud Platform and App Engine
- Exuma 9y agoWow this is cool. Does it take advantage of Erlang's distributed networking/nodes feature?
- dazuma 9y agoWe're not doing anything special to help set up distributed Erlang at the moment. It would probably be difficult with App Engine because it's designed to manage stateless containers. But I could imagine setting up such a network with Kubernetes Engine, or definitely with Compute Engine VMs.
- sb8244 9y agoAt elixir ATL meetup we setup an elixir app on GKE and connected the nodes via peerage in under 2 hours. Definitely possible!
- brightball 9y agoHow often do y’all meet up? I’m in Greenville SC but I could see a road trip happening periodically with the Upstate Elixir group.
- sb8244 9y agoIt's once per month. https://www.meetup.com/atlantaelixir/ https://www.meetup.com/atlantaelixir/ It's a pretty small group right now. I'm hoping that a new venue and push will help bring in more people.
- chingor13 9y agoIt currently doesn't do automated clustering of nodes out of the box. It should be possible to build a clustering strategy for libcluster (https://github.com/bitwalker/libcluster https://github.com/bitwalker/libcluster) for App Engine. Note that there's already a kubernetes strategy which could run on the Google Kubernetes Engine.
- etxm 9y agoI’ve used peerage (https://github.com/mrluc/peerage https://github.com/mrluc/peerage) on kubernetes for SD. It’s simple to set up and works in dev/test using its local adapters. Libcluster looks good too. Everything bitwalker touches seems to be solid AF.
- jesses 9y agoI can't speak to GCP much, but gigalixir.com is a platform-as-a-service that takes advantage of Erlang's distributed network/nodes. I'm the founder.
- dazuma 9y agoHey all! I'm one of the engineers who worked on this. If anyone has questions or wants to get involved, feel free to open an issue on the github repos. * Elixir clients for Google APIs: https://github.com/GoogleCloudPlatform/elixir-google-api https://github.com/GoogleCloudPlatform/elixir-google-api * Elixir runtime for App Engine: https://github.com/GoogleCloudPlatform/elixir-runtime https://github.com/GoogleCloudPlatform/elixir-runtime * Elixir examples: https://github.com/GoogleCloudPlatform/elixir-samples https://github.com/GoogleCloudPlatform/elixir-samples
- brightball 9y agoJust want to say thank you and this is great news! On a related note, there goes my weekend.
- willyk 9y agoAgreed, also just want to say thanks, this is awesome news. Much appreciated!
- burntcaramel 9y agoA question: what are the pluses/minuses of deploying to App Engine vs Compute Engine. Specifically, what are the tradeoffs of Elixir on App Engine? Thanks in advance.
- dazuma 9y agoThat's a good question. It comes down to how much you want to control the devops vs how much you want someone else to handle it for you. App Engine is a platform-as-a-service (PaaS) which generally means a lot of things are done for you. You give App Engine the code, and App Engine builds it, deploys it, scales it, monitors it, applies OS patches, and generally provides a bunch of related services for you. Compute Engine is more infrastructure-as-a-service (IaaS) which means the service gives you VMs, and you handle installation, build, deployment, and setting up load balancing, scaling, monitoring, etc. the way you want. There are also hybrid products that fall somewhere in the middle. Kubernetes Engine is an example. Indeed, you can locate most hosting platforms somewhere on this spectrum. Note also that there are trade-offs specific to Elixir as well. PaaSes generally have to be opinionated about application architecture. Most PaaSes assume a stateless web application well suited for container-based deployment, which runs somewhat counter to Erlang's stateful services and hot code swapping. If you're writing a typical web app (say, using Phoenix) then a PaaS like App Engine will often work very well. But if you are doing something different and managing persistent nodes and processes, you may need to control deployment yourself, which makes a lower level service like Compute Engine more useful.
- bpicolo 9y agoThat’s a really interesting addition to app engine. Great idea too - try to be first to an up and coming platform rather than try to move applications that have deployed in one way for a decade.
- mijoharas 9y agoCould someone explain what would make the Google cloud platform a better deploy target for my elixir application than any other?
- djsumdog 9y agoI'm curious about this as well. I hate vendor lock-in, so I'm for anything that's a minimal deployment that doesn't tie you to one provider.
- freedomben 9y agoI hate the idea of vendor lock-in, but years now of reaping the benefits has convinced me that it's just better to accept the lock-in and take advantage of your chosen platform's features. Remaining platform agnostic is difficult, complicates your code, and forces you to miss out on great services. these days I pick a provider and stick with them. It used to be AWS, but Google has been getting really good. I've been using Google cloud a lot.
- brightball 9y agoYou’ll need to evaluate GCP’s peripheral offerings for database, APIs, pricing, etc. The biggest thing that this does is give you first class access to the ecosystem APIs and with native setups for AppEngine (Heroku-ish), Compute Engine (do it yourself on a VM) and a provider in Gigalixir which runs on GCP that gives a Heroku-like experience but is tailored specifically to everything Elixir needs (hot loading, clustering out of the box, etc). Between those two things, GCP is pretty strong.
- mijoharas 9y agoThanks for the nice answer! I was reading up on Gigalixir thanks to this thread, is it basically a more "batteries included" version of AppEngine specifically targetted for Elixir?
- brightball 9y ago
- pselbert 9y agoSurprising development. I wonder if this will complement Gigalixir or compete with it. The ability to connect between nodes, avoid mandatory restart, avoid connection limits and keep websocket connections alive easily gives compute engine a significant advantage over platforms like Heroku.
- brightball 9y agoWill be interesting to watch. Between Gigalixir and now this, it’s starting to look like GCP is positioning itself as the place to go for Elixir apps.
- holografix 9y agoDisclaimer: I just got a job with Heroku As far as I understand Heroku lets you connect between nodes (Dynos within a private space with dns discovery [0]) and also supports websockets [1]... for the mandatory restart, wouldn't you want to design your process as stateless anyway? [0] https://devcenter.heroku.com/articles/dyno-dns-service-discovery https://devcenter.heroku.com/articles/dyno-dns-service-disco... [1] https://devcenter.heroku.com/articles/websockets https://devcenter.heroku.com/articles/websockets Sorry if I'm missing something obvious!
- enraged_camel 9y agoMore info here: https://hexdocs.pm/phoenix/heroku.html https://hexdocs.pm/phoenix/heroku.html Maybe you can convince the powers that be to do something about these. :)
- bostonvaulter2 9y agoThe problem with private spaces is the pricing is out of reach of the vast majority of early-mid-stage startups since it is only available under the enterprise model.
- brightball 9y agoIf I recall correctly, just turning on private spaces is not within most people’s price range. With most things you’d want stateless. Erlang/Elixir is designed with 3 built in databases, the ability to hold states on isolated processes and per process supervision and recovery. It’s a different animal than most.
- tabeth 9y agoExcellent! I recall someone on here (HN) saying that many of Elixir's strengths are weakened significantly because K8 does similar things (auto distribution, hot reloading, etc.) Any truth to this? Also, can you distribute nodes on VMs and deploy them to Google Cloud? --- As a side note, is there a front-end equivalent to Elixir? Elm is functional but is non intuitive. Elixirscript is a thing but seems a bit too early.
- willyk 9y agoThe two options you mention are the ones I see used most often (with React & Vue being 3rd/4th), but I wouldn't say any of those are "frontend equivalents".
- jrs95 9y agoThat’s just crazy to me. The amount of effort you have to put in to achieve this with Kubernetes is way beyond what you would have to do to write you application in Elixir. If you already have multiple services in another language, it may be worth the effort, but the BEAM gives you nearly for free what is relatively expensive to get elsewhere.
- hosh 9y agoElixir is great for stateless apps. So are a bunch of other platforms. Elixir is exceptional at stateful apps. And it is true, many of those things are now handled through K8S. However, K8S doesn't handle everything and requires a lot more ceremony and infrastructure than what you can get within the Elixir/Erlang ecosystem. I think there might be a way to work with Elixir + K8S in a way specific to the domain you are working with. Not everyone needs to be able to hot-reload code without dropping live connections, but if if you do, and you want to deploy on K8S, then you can try writing a K8S operator that encapsulates that idea. However: https://gravitational.com/blog/running-postgresql-on-kubernetes/ https://gravitational.com/blog/running-postgresql-on-kuberne... (which is more about how every distributed system requires deep domain knowledge, not just specifically about PostgreSQL) I have not deployed Elixir on GCP. I have deployed other things though, and I can tell you the private networking is much better set up than AWS. As far as distributing nodes, you can do that with any of the cloud providers.
- 9y ago
- qaq 9y agoNice Elixir is becoming more mainstream
- sergiotapia 9y agoReally excited to see Elixir going more mainstream. It's a hidden gem that solves so many painpoints of the day to day programming. It's POWERFUL!
- nstj 9y ago> Hidden gem :)
- richjdsmith 9y agoLikewise. I havn't come across a case of someone getting to know the language and disliking it. It's truly a great tool.
- beckler 9y agoOkay. I spent a lot of time last week trying to get elixir setup on gcloud, but variables were the one thing I could never figure out and I eventually gave up. Is there some magical way to get variables and secrets in gcloud without having to commit them to a repo?
- derefr 9y agoDeploy to Google Kubernetes Engine; `kubectl create secret`; set the env-vars of your Deployment to be derived from the secret.
- skrebbel 9y agoWait, you can't put secrets in GCP without first learning Kubernetes? That's pretty messed up :/
- derefr 9y agoMore like, Kubernetes (and Google Kubernetes Engine) is the right level of abstraction to actually have a first-class object-type of "secret." Compute Engine is just raw IaaS, like EC2. You "store secrets" on GCE by writing them to a volume and attaching it across your cluster, or putting them in a common database. Or just pushing data onto your instances after they're up using Fabric or something. You know, just like if they were real computers. App Engine is made to be a complete and self-contained PaaS (much moreso than anything like Heroku), to the point that it doesn't really use external credentials for anything. For example, you can access BigQuery tables from App Engine without supplying any "BigQuery credentials." Oddly enough, if you do need to connect to third-party services (or, say, Google Cloud SQL) from App Engine, they recommend that you store the credentials to do so in... a BigQuery table. GAE is proprietary and weird. It's basically its own "abstract machine." Kubernetes Engine, in comparison, is exactly for the use-case of "deploying workloads" of regular potentially-stateful, potentially-long-lived POSIX apps that need things like DB connection strings passed to them. If you're a modern backend developer who burns Docker images like people 10 years ago burned CDs, Kubernetes is the service you should be using on any compute cloud you interact with. Everything else is an impedance mismatch.
- skybrian 9y agoThis seems to be using App Engine's "flexible" environment. It's less appealing to me than "real" App Engine since as far as I know there's no free tier.
- ofek 9y agoDiscord (a heavy Erlang/Elixir user) runs on GCP, right?
- theDoug 9y agoYep. As of 24 days ago, there’s this public quote[1] "We are on GCP now and have no plans to migrate off of it, or go 'multi-cloud'" [1]: https://www.reddit.com/r/discordapp/comments/7mv84u/does_discord_still_run_on_gcp_google_cloud/ https://www.reddit.com/r/discordapp/comments/7mv84u/does_dis... (Disclosure: I work on Google Cloud)
- jhgg 9y agoYes we do! We’ve been able to shake out a fair amount of platform bugs that the Beam VM had on GCP too! Thanks to the google team for taking them all seriously and fixing them. Some were particularly nasty :)
- brightball 9y agoThanks for doing the hard work so we don’t have too. :)
- jshen 9y agoThis is really cool and all, but I really wish Google Cloud would move things out of beta faster. Cloud Functions are still in beta despite them being GA at both AWS and Azure for a while now. Enterprises typically won't use beta software which makes it nearly impossible to pick Google Cloud for an enterprise. But hey, they have Elixir support !?!?
- theshank 9y agoThis is super exciting!
- ninjakeyboard 9y agoI wrote a book on Akka with many years of experience with it and I have been using Elixir professionally for 7 months or so on greenfield/replatforming efforts and I have to say that the BEAM VM is really an incredible piece of technology that leaves Akka on the JVM in the dust in several aspects because the VM is actor-model native. I wish there was a statically typed language that ran on BEAM but Elixir and Erlang are great technologies to run in production if you can accept some of the associated risk of newer eco-systems in Elixir (but erlang has your back!).
- ihumanable 9y agoYou may already know about it, but if you are looking for a statically typed language that runs on BEAM, you might want to check out Elchemy. https://github.com/wende/elchemy https://github.com/wende/elchemy It allows you to write elm-like code that transpiles into easy to read and idiomatic (for the most part) elixir code.
- smt88 9y agoTHANK YOU. I didn't know about this, myself. I love the philosophy and robustness of the BEAM VM, but I can't go back to the stone ages of a language with a poor (or non-existent) static type system.
- bostonvaulter2 9y agoOut of curiosity, what languages do you prefer to work in?
- smt88 9y agoNothing is a slam dunk. I love and hate even my favorite languages (TypeScript, F#, Elm). Hoping Rust will be the silver bullet once I have time to try it.
- ninjakeyboard 9y ago
- keithnoizu 9y agoI'll have to look this over, I'm running a server cluster on GCE + Appengine currently aggregating and triggering alerts against about 300k device readings per minute. Huge pain point keeping things synchronized between GCE on java and the elixir backend.