5 ms·

I've been working on Encore for the past 3 years. I was at Spotify for the past 8 years as a Staff Software Engineer. We grew frustrated with the disproportion

by eandre 5y ago

I've been working on Encore for the past 3 years. I was at Spotify for the past 8 years as a Staff Software Engineer. We grew frustrated with the disproportionally large effort needed to build even simple backend applications, and created Encore to solve this problem. Encore uses static analysis and code generation to reduce the boilerplate you have to write, resulting in an extremely productive developer experience. It's a combination of a framework that combines with a cloud platform in order to simplify all aspects of building distributed systems, from how you write code to shipping it to production. I shared some more background on Encore here [1]. Feedback very much appreciated :) [1] https://twitter.com/_eandre/status/1381668476662734849 https://twitter.com/_eandre/status/1381668476662734849

datavinci 5y ago
Hey thanks a lot for sharing this, Could you recommend a couple of resources for someone to get into distributed systems? I read Data Intensive Applications, and have been finding such sources since.
- eandre 5y ago
  A lot of the seminal research papers are surprisingly readable! I really recommend reading them to get a good feel for the sort of problems you face at scale and good ways of handling them. Off the top of my head: a lot of stuff from Google (Dapper, Stubby, Zanzibar). Dynamo from Amazon. Leslie Lamport's guide on TLA+. The two generals problem, distributed consensus, and so on. Most distributed systems don't need all that fancy stuff, but it's useful to know regardless. Even the simplest systems will get weird and interesting bottlenecks with scale. And then a lot of just doing it in practice! Find a job at a startup with some scale, and learn on the job.
Philip-J-Fry 5y ago
I've built my fair share of Go microservices and have never felt the need to develop an entire framework like this. We have everything like CI/CD, distributed tracing and one line project generation. It's just based on some foundation code and some simple bash scripts to find and replace some service specific things. And then it's just the basic routing and HTTP handlers to unmarshal and marshal responses. When all your services are built off the same foundation then creating a new one is pretty much and copy and paste job in my experience. It's a cool project but it seems like unnecessary complexity, something I strive to avoid.
- eandre 5y ago
  If that works for you, all the better! There are quite a few things that Encore does beyond what you mentioned: - Automatically sets up dedicated Preview Environments for your Pull Requests - Provisions all the infrastructure you need, both for local development as well as in production (and other environments like staging or testing) - Provides auto-completion and compile-time type safety for making cross-service API calls, with a single line of code - Integrates secrets management, API documentation, frontend client generation, etc :)
- skrtskrt 5y ago
  What is your setup for distributed tracing?
  - wdb 5y ago
    Personally, I am using Opentelemetry to create and export my distributed traces. At first sight it looks like they have reinvented the wheel for this project.
  - Philip-J-Fry 5y ago
    Elastic APM. As far as I am aware it's an implementation of the OpenTracing specification. We just use the Elastic client libraries. Tell it where our APM server is, do the little bit of application set up and then view all our application traces in Kibana. All applications follow the same basic setup. Probably less than 20 lines of code and then wrapping some HTTP handlers and HTTP clients. Then we have some application specific spans.
    - wdb 5y ago
      Any plans to switch to Opentelemetry?
      - Philip-J-Fry 5y ago
        Not that I know of
- 5y ago
latch 5y ago
Did you look at whether Go was part of the problem or were you using something else and you found that Go was part of the solution?
nvarsj 5y ago
Hi André. Encore looks really cool! It must have taken a huge amount of work to put this together. I think streamlining from IDE to production is the future of tooling and frameworks, and it looks like you are going in that direction. I feel like that is the entire hype behind "serverless", but no one has really been able to pull it off yet imo. I wonder what you think about K8s integration? Spotify is all K8s right? I'm guessing that is tougher to monetize. I tried at a prior startup to work on something like this, offering a PaaS like experience on K8s with development libraries (it's actually still in progress - check out https://cloudstate.io/ https://cloudstate.io/). It is a large effort to build such a thing in a nice way. Anyways, I'm looking forward to seeing how Encore develops. Good luck!
MuffinFlavored 5y ago
> We grew frustrated with the disproportionally large effort needed to build even simple backend applications, and created Encore to solve this problem. I'm going to sound naive but where did running Docker daemon on a VPS and orchestrating deployment with `docker-compose.yml` or `Terraform` fall short?
Gys 5y ago
This seems very similar to https://goa.design/ https://goa.design/? Did you know it exists?

by eandre 5y ago

datavinci 5y ago

Hey thanks a lot for sharing this, Could you recommend a couple of resources for someone to get into distributed systems? I read Data Intensive Applications, and have been finding such sources since.

eandre 5y ago
A lot of the seminal research papers are surprisingly readable! I really recommend reading them to get a good feel for the sort of problems you face at scale and good ways of handling them. Off the top of my head: a lot of stuff from Google (Dapper, Stubby, Zanzibar). Dynamo from Amazon. Leslie Lamport's guide on TLA+. The two generals problem, distributed consensus, and so on. Most distributed systems don't need all that fancy stuff, but it's useful to know regardless. Even the simplest systems will get weird and interesting bottlenecks with scale. And then a lot of just doing it in practice! Find a job at a startup with some scale, and learn on the job.

Philip-J-Fry 5y ago

I've built my fair share of Go microservices and have never felt the need to develop an entire framework like this. We have everything like CI/CD, distributed tracing and one line project generation. It's just based on some foundation code and some simple bash scripts to find and replace some service specific things. And then it's just the basic routing and HTTP handlers to unmarshal and marshal responses. When all your services are built off the same foundation then creating a new one is pretty much and copy and paste job in my experience. It's a cool project but it seems like unnecessary complexity, something I strive to avoid.

eandre 5y ago
If that works for you, all the better! There are quite a few things that Encore does beyond what you mentioned: - Automatically sets up dedicated Preview Environments for your Pull Requests - Provisions all the infrastructure you need, both for local development as well as in production (and other environments like staging or testing) - Provides auto-completion and compile-time type safety for making cross-service API calls, with a single line of code - Integrates secrets management, API documentation, frontend client generation, etc :)
skrtskrt 5y ago
What is your setup for distributed tracing?
- wdb 5y ago
  Personally, I am using Opentelemetry to create and export my distributed traces. At first sight it looks like they have reinvented the wheel for this project.
- Philip-J-Fry 5y ago
  Elastic APM. As far as I am aware it's an implementation of the OpenTracing specification. We just use the Elastic client libraries. Tell it where our APM server is, do the little bit of application set up and then view all our application traces in Kibana. All applications follow the same basic setup. Probably less than 20 lines of code and then wrapping some HTTP handlers and HTTP clients. Then we have some application specific spans.
  - wdb 5y ago
    Any plans to switch to Opentelemetry?
    - Philip-J-Fry 5y ago
      Not that I know of
5y ago

latch 5y ago

Did you look at whether Go was part of the problem or were you using something else and you found that Go was part of the solution?

nvarsj 5y ago

Hi André. Encore looks really cool! It must have taken a huge amount of work to put this together. I think streamlining from IDE to production is the future of tooling and frameworks, and it looks like you are going in that direction. I feel like that is the entire hype behind "serverless", but no one has really been able to pull it off yet imo. I wonder what you think about K8s integration? Spotify is all K8s right? I'm guessing that is tougher to monetize. I tried at a prior startup to work on something like this, offering a PaaS like experience on K8s with development libraries (it's actually still in progress - check out https://cloudstate.io/ https://cloudstate.io/). It is a large effort to build such a thing in a nice way. Anyways, I'm looking forward to seeing how Encore develops. Good luck!

MuffinFlavored 5y ago

> We grew frustrated with the disproportionally large effort needed to build even simple backend applications, and created Encore to solve this problem. I'm going to sound naive but where did running Docker daemon on a VPS and orchestrating deployment with `docker-compose.yml` or `Terraform` fall short?

Gys 5y ago

This seems very similar to https://goa.design/ https://goa.design/? Did you know it exists?