4 ms·

We used services to great effect at the company I previously worked at. There was not a single engineer that ever voiced a concern over how things worked. A cou

We used services to great effect at the company I previously worked at. There was not a single engineer that ever voiced a concern over how things worked. A couple important notes: * We used a mono repo and services only had to talk to other services from the same commit (or branch). We didn't version service APIs, I think that would have been a huge waste of time for us. * Our tests did unit tests and integration tests. * We could use different languages as the problems required. We started one data processing service in Python and migrated it to Go when it became more CPU bound than we were comfortable with. We have an old but extremely reliable C++ service that had been running for almost a decade. * Each service ran in its own container. When a container/vm would run out of memory, we all didn't have to scramble to figure out who broke something. The person who owned the code for that service dealt with it. If a service started kicking out errors nobody but the owner had to be distracted. * We had a good team of architects that understood how to build things. We didn't split services across atomic boundaries. We knew how to build interfaces that handled errors robustly. If you don't know how to do these things, you service APIs can quickly become a mess. * The world is services. Just think about AWS and how it works. There is no such thing as a monolith, eventually you interface to services. Be it a weather API, a CI server, deployment system, a central logger, etc. Just because you didn't write all those services doesn't mean you don't have a service architecture. * We had really disjoint services that would have made zero sense to put together. For example, the web server that handled the front-end vs a back-end data system that refreshed a product database and built PDF assets based on the data in overnight batch runs. We actually had many of the later for different customers. We had services for central logging, for monitoring, for metric storage, etc. No way would I ever want to push all that into a single service and debug a GC issue or segfault. * If we had to hotfix a service, we could deploy just that service with zero concern the other services would have an issue. For us, services were a key part of building a very reliable system that we could update rapidly with minimal risk. We did this for almost a decade. We had a very good group of engineers and never did I hear one of them say services were holding us back. After this experience, I would say everyone one of them is an advocate for the appropriate use services.

gizzlon 6y ago
> * We used a mono repo and services only had to talk to other services from the same commit (or branch) Huh, that was a new one, can you expand? How did that work in practice? Did all the services update to the latest commit automatically?
- throwaway9870 6y ago
  We had a weekly deploy (plus any hotfixes) that built the images for all the new VMs (we used KVM) and then when done, stopped all the services and restarted them with their new images. Total down time was about 15 sec which for our application was acceptable. The build time was a lot longer than that (3-4 minutes), but that happened before the existing services were stopped so it didn't affect downtime. If any of the services failed to build, the deploy was aborted and the existing services were left running uninterrupted. The deploy tool was home grown and was part of the repo. Having everything in a mono repo was really important because it dramatically reduced the number of permutations of interfaces that had to be tested down to one. These trade-offs might not be right for everyone, but for us they were fine and allowed us to create a highly efficient development environment.
  - gizzlon 6y ago
    Thanks. So by deplyoing them all at once, as one unit, you avoid having to deal with the huge pain of backwards/forwards compatibility. But what would happen if 1 of them failed to start? Or start, but immediately fail its tasks? Then you would have to revert all of them, at the same time, after some has already run for a short while, potentially writing data on the new format (for example). So I guess you can't avoid the problem with backwards compatibility completely.. Or, I guess, more realistically, you wouldl try to fix the failing service fast, without rolling back anything :)
    - throwaway9870 6y ago
      Depends. I ran into this myself a couple times over the years and dealt with it depending on what changed. What was really nice was the weekly deploy. All commits were fresh in my head so there were not any changes from long ago that I forgot about. That was actually really important. On some holidays we would not do a deploy and even two weeks of changes was a lot more to remember than one week. So, if I knew the changes well, I could roll back if nothing in the API changed. We didn't like that, but sometimes you just need to be pragmatic. Other times an update was made that forced you to get it working. In this case services helped because rather than the whole system being down, only part of it was. So many of our services were background type work that in most cases, customers would never even know if they were down for even 1-2hrs. We had good staging environments with a lot of reporting, monitoring and metrics. We never deployed to production code that had not run reliably in staging for a least a couple of days.

gizzlon 6y ago

> * We used a mono repo and services only had to talk to other services from the same commit (or branch) Huh, that was a new one, can you expand? How did that work in practice? Did all the services update to the latest commit automatically?

throwaway9870 6y ago
We had a weekly deploy (plus any hotfixes) that built the images for all the new VMs (we used KVM) and then when done, stopped all the services and restarted them with their new images. Total down time was about 15 sec which for our application was acceptable. The build time was a lot longer than that (3-4 minutes), but that happened before the existing services were stopped so it didn't affect downtime. If any of the services failed to build, the deploy was aborted and the existing services were left running uninterrupted. The deploy tool was home grown and was part of the repo. Having everything in a mono repo was really important because it dramatically reduced the number of permutations of interfaces that had to be tested down to one. These trade-offs might not be right for everyone, but for us they were fine and allowed us to create a highly efficient development environment.
- gizzlon 6y ago
  Thanks. So by deplyoing them all at once, as one unit, you avoid having to deal with the huge pain of backwards/forwards compatibility. But what would happen if 1 of them failed to start? Or start, but immediately fail its tasks? Then you would have to revert all of them, at the same time, after some has already run for a short while, potentially writing data on the new format (for example). So I guess you can't avoid the problem with backwards compatibility completely.. Or, I guess, more realistically, you wouldl try to fix the failing service fast, without rolling back anything :)
  - throwaway9870 6y ago
    Depends. I ran into this myself a couple times over the years and dealt with it depending on what changed. What was really nice was the weekly deploy. All commits were fresh in my head so there were not any changes from long ago that I forgot about. That was actually really important. On some holidays we would not do a deploy and even two weeks of changes was a lot more to remember than one week. So, if I knew the changes well, I could roll back if nothing in the API changed. We didn't like that, but sometimes you just need to be pragmatic. Other times an update was made that forced you to get it working. In this case services helped because rather than the whole system being down, only part of it was. So many of our services were background type work that in most cases, customers would never even know if they were down for even 1-2hrs. We had good staging environments with a lot of reporting, monitoring and metrics. We never deployed to production code that had not run reliably in staging for a least a couple of days.