5 ms·

Our approach to services at Khan Academy is likely a bit different from most. We're sticking with a monorepo (the code for all services lives in one repository)

Our approach to services at Khan Academy is likely a bit different from most. We're sticking with a monorepo (the code for all services lives in one repository). We have a single go.mod file at the top of the repo, so all services use the _same versions_ of dependencies. We're still building out our deployment system to better support multiple services, but we're planning to redeploy all of the services when library code changes (which is something we're trying to minimize). All of this ensures that we don't have trouble with services lagging behind on critical updates.

solidasparagus 7y ago
I don't really understand moving to microservices if you aren't going to give the service teams the autonomy to make their own decisions and move at their own pace. Microservices always seemed to me to be more of an organizational strategy than a technical one - if you have services, but they can't operate independently, it feels a bit like you are just creating a distributed monolith.
- gtirloni 7y ago
  Choosing the dependencies' version is not all there's to decide. Teams have a lot of control of basically everything else.
  - andyroid 7y ago
    So, no choice of language and no choice of the libraries used. What’s “everything else” exactly? Sounds like missing out on the more interesting parts of a micro service architecture.
    - sangnoir 7y ago
      Yeah, I'll never work for a company[1] where service teams are free to choose any language; 2 or 3 options at most is fine, but more than that is a hard no. I'll have to read/work on that code sooner or later, and I have no time to be dealing with a hodgepodge of languages 1. In the 10-5000 employee range: tech giants are a different beast when it comes to team accountability.
      - solidasparagus 7y ago
        > I'll have to read/work on that code sooner or later In the microservices organizations I've seen, this isn't true. The other service teams provide an API and, like any other SaaS you use, you do not need to be able to read the implementation. You would only work on that code if you switch service teams.
    - deleted 7y ago
      [deleted]
- dangoor 7y ago
  We're making a certain set of tradeoffs. For example, we're not adopting the "write code in whatever language you want" form of microservices that some folks adopt, because we don't feel like we're large enough to support that. Like I said, though, we do want to minimize the library footprint. The vast majority of deploys in this new world will be single service deploys, with the benefits that come with that. We already deploy our monolith several times a day. These services will speed that further.
  - vips7L 7y ago
    Didnt you guys recently decide to write some services in kotlin? Are you rewriting those in go now since everything is going to be one language?
    - dangoor 7y ago
      We have some dataflow jobs written in Kotlin (we blogged about that in June 2018[1]). We also have an internal service written in Kotlin. We ideally want one language. But Apache Beam (which is behind Google Dataflow) doesn't yet have production support for Go. More importantly, though, we have no time pressure on switching the Kotlin code over, so that's a long way out. [1]: https://engineering.khanacademy.org/posts/kotlin-adoption.htm https://engineering.khanacademy.org/posts/kotlin-adoption.ht...
      - vips7L 7y ago
        Thanks for answering! That was the specific article I remember reading.
  - solidasparagus 7y ago
    It'll be interesting to see how it works. I would probably fight to not have shared dependencies because the relatively small benefit does not seem to merit the increased coupling between teams - if all teams needs to agree before a library can be upgraded, I can't imagine it will be very easy to keep libraries up-to-date. To a certain extent it depends on how large your engineering org is, which I don't know. I've been in companies that moved from monolith to microservices and I saw it work well except when teams had such tight cross-service dependencies that they had to get other teams' ok before making changes to internal details of their service. Then developer velocity was slower than before because it took time to make the cross-team discussion happen and political capital to make other team care when they have other priorities.
phumbe 7y ago
Sounds similar to some work I’ve been doing, so thanks for unknowingly validating my design! At my employer, I’m spearheading a wholesale reimplementation of outdated process automation software, turning everything into Django web apps. I’ve been working with a monorepo and monolithic deployments to maintain development velocity but recently started transitioning the CI/CD pipeline to deploy each application/service in the monorepo independently. The pipeline packages common assets (including, e.g., manage.py, common HTML templates, and the dependency spec...all housed in the same monorepo) into each app directory before the deploy stage. Meanwhile, local developers clone the entire monorepo, and when they launch localhost, all of the services come online simultaneously. (That’s the goal, at least!) I was already excited to see my work come to fruition, and now I’ll be keeping an eye on Khan Academy, too!
- dangoor 7y ago
  That does sound pretty similar (though our services all have the luxury of serving nothing other than GraphQL!). Our current plan for local development is to continue cloning the monorepo and firing up all of the services. Go services don't take a whole lot of resources, so we think this plan will work fine for quite a while.
sethammons 7y ago
I'm in an org that runs multiple services in Go. The dependencies on library stuff has been a very minimal need. I think you are optimizing for an imaginary problem. With microservises, a team should have non breaking API versions running and work with teams to transition to a new API version when needed. If the underlying uuid lib or kaftka lib changes, teams may or may not need to update, but they can do so on their own time.

solidasparagus 7y ago

I don't really understand moving to microservices if you aren't going to give the service teams the autonomy to make their own decisions and move at their own pace. Microservices always seemed to me to be more of an organizational strategy than a technical one - if you have services, but they can't operate independently, it feels a bit like you are just creating a distributed monolith.

gtirloni 7y ago
Choosing the dependencies' version is not all there's to decide. Teams have a lot of control of basically everything else.
- andyroid 7y ago
  So, no choice of language and no choice of the libraries used. What’s “everything else” exactly? Sounds like missing out on the more interesting parts of a micro service architecture.
  - sangnoir 7y ago
    Yeah, I'll never work for a company[1] where service teams are free to choose any language; 2 or 3 options at most is fine, but more than that is a hard no. I'll have to read/work on that code sooner or later, and I have no time to be dealing with a hodgepodge of languages 1. In the 10-5000 employee range: tech giants are a different beast when it comes to team accountability.
    - solidasparagus 7y ago
      > I'll have to read/work on that code sooner or later In the microservices organizations I've seen, this isn't true. The other service teams provide an API and, like any other SaaS you use, you do not need to be able to read the implementation. You would only work on that code if you switch service teams.
  - deleted 7y ago
    [deleted]
dangoor 7y ago
We're making a certain set of tradeoffs. For example, we're not adopting the "write code in whatever language you want" form of microservices that some folks adopt, because we don't feel like we're large enough to support that. Like I said, though, we do want to minimize the library footprint. The vast majority of deploys in this new world will be single service deploys, with the benefits that come with that. We already deploy our monolith several times a day. These services will speed that further.
- vips7L 7y ago
  Didnt you guys recently decide to write some services in kotlin? Are you rewriting those in go now since everything is going to be one language?
  - dangoor 7y ago
    We have some dataflow jobs written in Kotlin (we blogged about that in June 2018[1]). We also have an internal service written in Kotlin. We ideally want one language. But Apache Beam (which is behind Google Dataflow) doesn't yet have production support for Go. More importantly, though, we have no time pressure on switching the Kotlin code over, so that's a long way out. [1]: https://engineering.khanacademy.org/posts/kotlin-adoption.htm https://engineering.khanacademy.org/posts/kotlin-adoption.ht...
    - vips7L 7y ago
      Thanks for answering! That was the specific article I remember reading.
- solidasparagus 7y ago
  It'll be interesting to see how it works. I would probably fight to not have shared dependencies because the relatively small benefit does not seem to merit the increased coupling between teams - if all teams needs to agree before a library can be upgraded, I can't imagine it will be very easy to keep libraries up-to-date. To a certain extent it depends on how large your engineering org is, which I don't know. I've been in companies that moved from monolith to microservices and I saw it work well except when teams had such tight cross-service dependencies that they had to get other teams' ok before making changes to internal details of their service. Then developer velocity was slower than before because it took time to make the cross-team discussion happen and political capital to make other team care when they have other priorities.

phumbe 7y ago

Sounds similar to some work I’ve been doing, so thanks for unknowingly validating my design! At my employer, I’m spearheading a wholesale reimplementation of outdated process automation software, turning everything into Django web apps. I’ve been working with a monorepo and monolithic deployments to maintain development velocity but recently started transitioning the CI/CD pipeline to deploy each application/service in the monorepo independently. The pipeline packages common assets (including, e.g., manage.py, common HTML templates, and the dependency spec...all housed in the same monorepo) into each app directory before the deploy stage. Meanwhile, local developers clone the entire monorepo, and when they launch localhost, all of the services come online simultaneously. (That’s the goal, at least!) I was already excited to see my work come to fruition, and now I’ll be keeping an eye on Khan Academy, too!

dangoor 7y ago
That does sound pretty similar (though our services all have the luxury of serving nothing other than GraphQL!). Our current plan for local development is to continue cloning the monorepo and firing up all of the services. Go services don't take a whole lot of resources, so we think this plan will work fine for quite a while.

sethammons 7y ago

I'm in an org that runs multiple services in Go. The dependencies on library stuff has been a very minimal need. I think you are optimizing for an imaginary problem. With microservises, a team should have non breaking API versions running and work with teams to transition to a new API version when needed. If the underlying uuid lib or kaftka lib changes, teams may or may not need to update, but they can do so on their own time.