3 ms·

30 teams' services fit on 2-3 servers? We don't have anywhere near 30 teams, but at my last count end of last year there were ~250 repos, and everyday I get au

30 teams' services fit on 2-3 servers? We don't have anywhere near 30 teams, but at my last count end of last year there were ~250 repos, and everyday I get auto-emailed telling me I've been subscribed to a new repo. No way could we have only 2-3 servers.

tonyarkles 8y ago
I'm perpetually curious about what kind of resource usage you'd expect given those 250 repos, and which runtime you're deploying. I've recently been playing with Go and looking at existing JVM-based services (Spring Boot + starters + our microservices). We've seen rewrites of small services change from ~500MB of RAM to ~10MB of RAM. 250 Go-based services would probably fit very comfortably on a small cluster of cheap-ish servers, depending on what they're doing and how much traffic they're handling.
- chairmanwow 8y ago
  Can you go into detail about what things changed in the rewrites that enabled this reduction in RAM usage?
  - jacques_chester 8y ago
    The JVM is traditionally memory-hungry. There are lines of engineering underway which will change that quite a bit (GraalVM looks particularly promising), but for now seeing a JVM process merrily consuming hundreds of Mb of RAM as the base case is not unusual.
    - tonyarkles 8y ago
      That's exactly it. Plus add in all of the Spring Boot stuff (autodiscovery/autowiring, behaviour that gets added simply because a .jar is on the classpath, etc etc) and you end up with a pretty large footprint for a pretty tiny app. For reference too, the deployment artifacts with Go are ~5MB vs. a 70MB .war file. We've also just used really simple packages (the built-in net/http server for Go vs. embedded Tomcat, gorilla/mux vs. the Spring class annotations for routing), and I suspect that dramatically cuts down on the footprint too.
ajross 8y ago
> 30 teams' services fit on 2-3 servers? Why would system load scale with the number of engineers? Yeah. 30 "teams" worth of work can totally fit on one system. The sum total of all software engineering ever done before, I dunno, 1979 can fit on one box. Really, that's my point. People are far too enthused with the "feeling" of working on a "big" project and not being sufficiently reasonable or conservative about the technologies they try to use. Cluster deployment for an application this size just isn't needed or appropriate. It's cargo cult engineering, because all the cool kids are using k8s and we want to read about cool kids.
- antod 8y ago
  You can't see other reasons for running more servers than just system load? We are a bit smaller than uSwitch (no Kubernetes though), but we maybe a thousand VMs spread a couple of dozen physical servers and some cloud stuff. We have multiple environments across multiple sites for each of multiple enterprise customers with multiple server roles. There are no single points of failure in any server role and contractural/legal/privacy requirements to keep data (and staff access) separate between different 'zones'. There are integration points with 3rd party partners, logging systems, monitoring systems, bastion hosts, internal CI build clusters, internal business tools etc etc. Beyond a certain size and complexity, you need to separate stuff out to stop people treading all over each other. You need to be able to contain breakage by ideally running a single service on each VM. None of this duplication was due to people wanting a big project - it grew over time out of necessity even with conservative 'enterprisey' attitudes to technology.

tonyarkles 8y ago

I'm perpetually curious about what kind of resource usage you'd expect given those 250 repos, and which runtime you're deploying. I've recently been playing with Go and looking at existing JVM-based services (Spring Boot + starters + our microservices). We've seen rewrites of small services change from ~500MB of RAM to ~10MB of RAM. 250 Go-based services would probably fit very comfortably on a small cluster of cheap-ish servers, depending on what they're doing and how much traffic they're handling.

chairmanwow 8y ago
Can you go into detail about what things changed in the rewrites that enabled this reduction in RAM usage?
- jacques_chester 8y ago
  The JVM is traditionally memory-hungry. There are lines of engineering underway which will change that quite a bit (GraalVM looks particularly promising), but for now seeing a JVM process merrily consuming hundreds of Mb of RAM as the base case is not unusual.
  - tonyarkles 8y ago
    That's exactly it. Plus add in all of the Spring Boot stuff (autodiscovery/autowiring, behaviour that gets added simply because a .jar is on the classpath, etc etc) and you end up with a pretty large footprint for a pretty tiny app. For reference too, the deployment artifacts with Go are ~5MB vs. a 70MB .war file. We've also just used really simple packages (the built-in net/http server for Go vs. embedded Tomcat, gorilla/mux vs. the Spring class annotations for routing), and I suspect that dramatically cuts down on the footprint too.

ajross 8y ago

> 30 teams' services fit on 2-3 servers? Why would system load scale with the number of engineers? Yeah. 30 "teams" worth of work can totally fit on one system. The sum total of all software engineering ever done before, I dunno, 1979 can fit on one box. Really, that's my point. People are far too enthused with the "feeling" of working on a "big" project and not being sufficiently reasonable or conservative about the technologies they try to use. Cluster deployment for an application this size just isn't needed or appropriate. It's cargo cult engineering, because all the cool kids are using k8s and we want to read about cool kids.

antod 8y ago
You can't see other reasons for running more servers than just system load? We are a bit smaller than uSwitch (no Kubernetes though), but we maybe a thousand VMs spread a couple of dozen physical servers and some cloud stuff. We have multiple environments across multiple sites for each of multiple enterprise customers with multiple server roles. There are no single points of failure in any server role and contractural/legal/privacy requirements to keep data (and staff access) separate between different 'zones'. There are integration points with 3rd party partners, logging systems, monitoring systems, bastion hosts, internal CI build clusters, internal business tools etc etc. Beyond a certain size and complexity, you need to separate stuff out to stop people treading all over each other. You need to be able to contain breakage by ideally running a single service on each VM. None of this duplication was due to people wanting a big project - it grew over time out of necessity even with conservative 'enterprisey' attitudes to technology.