7 ms·

We tried this approach at a former company with ~600 engineers at the time. Trying to boot the full service on a single machine required every single developer

• rileymat2 2y ago
  I may be misunderstanding, but wouldn't you want the particular microservice you are working on independent enough to develop locally, then deploy into the remote environment to test the integration? (I don't work at this scale)
  • animal_spirits 2y ago
    Invariably this is an ideal and does not match up in reality. I work at ~50 ish employee company and we have layers of dependencies between at least 6 or 7 various microservices. I can see this adding up in complexity as the product scales
    • rileymat2 2y ago
      Yes, the real world trumps theory, hence my question.
      • jrs235 2y ago
        Reminds me of a favorite quote: "What's the difference between between theory and practice (reality)? In theory, they're the same."
    • flimsypremise 2y ago
      If the microservice has dependencies on other services it is not a microservice.
      • blitzar 2y ago
        Connecting to a messaging queue or database count as a dependency? Why not break a microservice into a series of microservices, its microservices all the way down.
        • Cthulhu_ 2y ago
          Only if you cannot change one service without changing the other simultaneously. It's fine to have evolving messages on the queue but they have to be backwards compatible with any existing subscribers, because you cannot expect all subscribers to update at the same time. Unless you have a distributed monolith in a monorepo, but at least be honest about it. Multiple services connecting to the same database has been considered a bad idea for a long time. I don't necessarily agree, but I have no experience in that department. It does mean more of your business logic lives in the database (rules, triggers, etc).
          • chipdart 2y ago
            > Only if you cannot change one service without changing the other simultaneously. Not true at all. You're conflating the need for distributed transactions with the definition of microservices. That's not it. > Multiple services connecting to the same database has been considered a bad idea for a long time. Not the same thing at all. Microservices do have the database per service pattern, and even the database instance per service instance pattern, but shared database pattern is also something that exists in the real world. That's not what makes a microservice a microservice.
      • dns_snek 2y ago
        According to whom? How do those microservices get anything done if they just live in their own isolated world where they can't depend on (call out to) any other microservice?
    • nightski 2y ago
      So basically a distributed monolith :P
      • Tainnor 2y ago
        I'm sure companies with well designed and properly isolated services exist but... in my time spent at several companies, "microservices" invariably degenerate to distributed monoliths.
    • chipdart 2y ago
      > Invariably this is an ideal and does not match up in reality. No, this does indeed match reality. At least for those who work with microservices. This is microservices 101. It's baffling how this is even being argued. We have industry behemoths building their whole development experience around this fact. Look at Microsoft. They even went to the extents of supporting Connected Services in Visual Studio 2022. Why on earth do you believe one of the most basic traits of backend development is unreal? > I work at ~50 ish employee company and we have layers of dependencies between at least 6 or 7 various microservices. Irrelevant. Each service has dependencies and consumers. When you need to run an instance of one of those services locally, you point it to it's dependencies and you unplug it from it's consumers. Done. This is not rocket science.
      • Tainnor 2y ago
        You can't compare Microsoft to your run-of-the-mill small (or even large) software shop, though. Maybe on HN, most people work on these amazingly designed systems, but in my experience most tech out there is shit and has no proper design or architecture beyond "we're doing microservices because everyone is".
        • chipdart 2y ago
          > You can't compare Microsoft to your run-of-the-mill small (or even large) software shop, though. I'm talking about how Microsoft added support for connected services to Visual Studio. It's literally a tool that anyone in the world can use. They added the feature to address existing customer needs.
          • Tainnor 2y ago
            Apart from the fact that not everyone uses Visual Studio, "connected services" appears to be something by which you can connect to existing cloud-based services. How does that solve the problem of a mess of interconnected services where you may have to change 3 or more of them simultaneously in order to implement a change?
  • rtpg 2y ago
    This is how you get to "I wrote to the spec, it's your problem that clicking the button doesn't do the thing". Huge feedback loops. When you run it "locally" enough you can do your integration stuff before you even ask for review.
    • nitwit005 2y ago
      That doesn't seem possible to me. If you have a feature that involves 10 teams and 10 services, nothing will actually work until the 10th change is made (assuming everything was done perfectly).
      • Tainnor 2y ago
        If you have one team per service, yes. In many companies, you may have one team and 10 services though. I wish I was making this up.
    • chipdart 2y ago
      > This is how you get to "I wrote to the spec, it's your problem that clicking the button doesn't do the thing". No, not really. You only find yourself in that spot if you completely failed to do any semblance of integration test, or any acceptance test whatsoever. That's not a microservices problem. That's a you problem. You talk about feedback look. Other than automated tests, what do you believe that is?
  • Neikius 2y ago
    Yes, exactly. I just also like to have an option to run service locally and connect to either cloud instances (test) or local instances depending on what I am troubleshooting/testing. Much better than debugging on prod which may still be required at some point but hopefully not often.
• justinclift 2y ago
  > installing ~50ish microservices on their machine Ouch. Where they using macOS at the time with laptops having not-enough-ram? I've seen that go poorly on macOS with java based microservices. Largely due to java VMs wanting ram pre-assigned for each, which really chews though ram that mostly sits around unused. This was a few years ago though, at the tail end of Intel based mac's where 32GB ram in a mac laptop wasn't really an option.
  • themanmaran 2y ago
    I bet it's less of a RAM issue, and more of an orchestration problem. Making sure you have the latest version of every microservice and it's configuration. "Oh it's not running locally, you need to also run service_18_v2.js, and include the right env variables"
    • skydhash 2y ago
      This + k8s magic that seem to make thing coupled tighter.
      • KronisLV 2y ago
        > This + k8s magic that seem to make thing coupled tighter. I think nobody really talks about this, but unless you have a docker-compose.yml that includes everything you need for local development, it's increasingly more likely that you'll end up coupling things to Kubernetes to such a degree that running without it (and its abstractions) will become more effort than a person can muster. So while people try to create services that are decoupled from one another, they end up instead coupling them to Kubernetes concepts and a service mesh, service discovery, configuration and secret management mechanisms, persistent storage abstractions and so on. Which you can obviously do if you want to, but which might make running things locally in a minimalistic fashion that much more complex. It's the same as with for example using a web server as a reverse proxy for my applications and ending up putting some logic in there (e.g. route rewrites, headers etc.) and then realizing that I must also run a similar web server locally for 1:1 compatibility because something like Vue dev server proxy to the locally running API won't be able to give me all that.
• deleted 2y ago
  [deleted]
• chipdart 2y ago
  > Trying to boot the full service on a single machine required every single developer in the company installing ~50ish microservices on their machine, for things to work correctly. Became totally intractable. This is certainly one of the critical mistakes you did. No developer needs to launch half of the company's services to work on a local deployment. That's crazy, and awfully short-sighted. The only services a developer ever needs to launch locally are the ones that are being changed. Anything else they can consume straight out of a non-prod development environment. That's what non-prod environments are for. You launch your local service locally, you consume whatever you need to consume straight from a cloud environment, you test the contract with a local test set, and you deploy the service. That's it. > I guess one can grumble about bad architecture all day but this had to be solved. Yes, it needs to be solved. You need to launch your service locally while consuming dependencies deployed to any cloud environment. That's not a company problem. That's a problem plaguing that particular service, and one which is trivial to solve. > Both FAANG companies I’ve worked at had remote dev environments that were built in house. All FANG companies I personally know had indeed remote dev environments. They also had their own custom tool sets to deploy services locally, either in isolation or consuming dependencies deployed to the cloud. This is not a FANG cargo cult problem. This is a problem you created for yourself out of short-sightedness and for thinking you're too smart for your own good. Newbies know very well they need to launch one service instance alone because that's what they are changing. Veterans know that too well. Why on earth would anyone believe it's reasonable to launch 50 services to do anything at all? Just launch the one service you're working on. That's it. If you believe something prevents you from doing that, that's the problem you need to fix. Simple. Crazy.
  • athrowaway3z 2y ago
    Ha. It almost looks like either the fan boys or PR department is making up use-cases. Yes what you're saying is correct, but why many words when few do trick: This hypothetical IT department isn't able to host its own development environment, yet suddenly they do have the skills if they switched to gitpod.
  • mattacular 2y ago
    > You launch your local service locally, you consume whatever you need to consume straight from a cloud environment, you test the contract with a local test set, and you deploy the service. That's it. If your services are mostly stateless and/or your development team is very small that can work. If not, you will quickly run into problems sharing the data. Making schema changes to the shared cloud services. Cleaning up dev/test/etc data that has accumulated, etc. Then you are back to thinking of provisioning isolated cloud environment per dev.
• deleted 2y ago
  [deleted]
• gchamonlive 2y ago
  You need an option to enable only the services they need to develop locally and automatically configure them to talk to the other services hosted elsewhere.
  • Kye 2y ago
    What ever happened to make? It seems like so much of what developers do now just duplicates what could be done with a single makefile that pokes around, figures out what you need, and makes a script to set it up.
    • brigandish 2y ago
      Make is horrible though. It works, but it’s horrible. Hardly surprising people have looked for alternatives, it’s just a shame that most of those are horrible too.
    • gchamonlive 2y ago
      "Use make" advices start so sound a lot like fetish and disregard the actual challenges. Make is just a framework for you to do your builds. Sure you can cram anything into it, but that is exactly the kind of area that other tools like Ansible or even Terraform shine. Make isn't a silver bullet. EDIT: Just to make sure, I'm using fetish as something you spend an unreasonable amount of time with it.
      • deleted 2y ago
        [deleted]
• eloisant 2y ago
  In a past job I've had a good experience in this case with docker compose (well, something similar). You would list the services you need (or service groups) in a config file, start a command and all services would start in containers. Sure, you need a lot of RAM with that but on 32Gb it was working fine.
  • nottorp 2y ago
    > but on 32Gb it was working fine. So 64 G to include the ram for the ide and web browsing / javascript apps like slack?
• wlll 2y ago
  That's ~12 people per microservice. I understand that it might be hard to boot and coordinate all that on each developers machine, and that local docker images/vms/whatever might not suit you, but is it possible that this is a problem of your companies own creation?
  • 0xCMP 2y ago
    It's very likely not their decisions that lead to this, but their responsibility to improve velocity. Imagine the goal is to fix the problems (e.g. make it possible to run less of the services or something like that): How do you do that without first running all the services, making the proper changes, and then testing those changes? You need to be able to run all the services in that interim period. So, wouldn't it be nice if there were a solution for this in-general? And, maybe, it would lead to better conditions later on. But in the meantime there is really no way around the existing design/decisions/etc. You simply have to deal with that reality and engineer around it.
    • wlll 2y ago
      > It's very likely not their decisions that lead to this Yeah, I get that, I was deliberate about the phrasing of "your company" rather than just "your". Obviously we don't know anything about the parent commenters company and situation, perhaps 12 people per microservice genuinely is the right solution for them, but it seems like it would be better not to get into this situation in the first place, though once there you obviously have to tackle the problem as it presents itself.
      • 0xCMP 2y ago
        Agreed that it may be the right solution, but it smells bad. If I was there I would be trying to reduce the complexity where I could.
    • chipdart 2y ago
      > How do you do that without first running all the services (...) Why do you need to run all services in isolation to be able to troubleshoot and isolate a problem?
• jen20 2y ago
  > Both FAANG companies I’ve worked at had remote dev environments that were built in house. This is certainly not universal among FAANGs though. Requiring 50 services to be up is absolutely nuts, but it’s actually pretty trivial using something like Nomad locally.