4 ms·

You can be asked to dig holes and close them up again - a completely useless work and you can be given an ever increasing quota to do it with the threat that on

by vaidhy 4y ago

You can be asked to dig holes and close them up again - a completely useless work and you can be given an ever increasing quota to do it with the threat that one of the holes you dig might have your body in it (extremely stressful and ruthless). The challenge most of the people outside of Amazon do not realize is that micro-service architecture imposes a heavy penalty on team co-ordination, esp. as the software matures. A large part of core services have been built over several years (and decades) and they have been architected to optimize for some operations at the cost of other operations. You can think of this as how you would design a DB schema. It is a trade-off between write speed, query speed and availability. Now, think about it happening at massive scale. As an engineer, you would spend a lot of time working/waiting on partner teams to prioritize your changes while under pressure to deliver. A lot of teams ended up building the pieces on their own into their own services leading to a larger maintenance and operational load. From the outside, you are very likely to think that the team is bloated and slow-moving because you are assuming green field development. Internally, Amazon is a 25-year old legacy platform with almost no single vision across teams, tons of semi-duplicate functionality and heavy operational load. Keeping the lights on (KTLO) was a big ticket item on every annual plan I created when I was there.

jsharf 4y ago
I think it's actually normal for teams to have duplicate functionality. The requirements are usually never actually identical and when you try to merge them it results in services with complicated configuration that have N^2 edge cases which leads to more bugs or more complicated code. I suspect what you're describing is somewhat unavoidable, and not even a bad thing. That being said, Google for comparison does have good code re-use for certain core resources, like spanner, cloud, tensorflow, borg, etc. If you're talking about bedrock infrastructure like that, it's quite a different picture.
- vaidhy 4y ago
  I did not say it is bad. I just wanted to point out that N^2 communications are a big hit (of course we know them for long time) and people under-estimate the co-ordination effort and the fact that already running systems impose constraints that are not true for green-field efforts. I think I am regurgitating mythical man-month here and will shut up.
  - VBprogrammer 4y ago
    > the fact that already running systems impose constraints that are not true for green-field efforts Maybe I'm doing it wrong but I often find that doing a new greenfield project sounds easier but by the time you deal with all of the non-functional stuff (connectivity, resilience, logging, build tools, documentation etc) you might have been quicker holding your nose and bodging it in somewhere vaguely appropriate in an existing system.
    - vaidhy 4y ago
      There is shit-ton of work to be done :)
runlevel1 4y ago
> micro-service architecture imposes a heavy penalty on team co-ordination This is, by far, the hardest scaling problem in SaaS: How do we divide responsibilities amongst our (code|services|teams) efficiently? You can only grow each so much before they become unwieldy to manage. Divide scope too much and communication becomes complex. Divide along the wrong abstraction and you get duplication, overlap, and confusion about what's responsible for what. And as time passes, decisions of the past constrain and complicate your options to redesign/rearchitect/reorg today.
- matai_kolila 4y ago
  So it's "bad micro-service architecture imposes a heavy penalty on team co-ordination", then. ...which is a fairly obvious observation. "Bad <x> imposes a heavy penalty on team co-ordination" is generalizable.
  - vaidhy 4y ago
    There is no bad vs good micro-service architecture. It is just that micro-service architecture imposes a different kind of penalty on co-ordination and debugging. Instead of intra-component dependency, it moves it to inter-service dependency. Each service might look simple, but the complexity is moved to service communications. It is not that obvious, imho.
    - bobthepanda 4y ago
      Also, what is good and bad can change over time with the business. Few judgement calls in software will last forever.
    - deleted 4y ago
      [deleted]
    - matai_kolila 4y ago
      There absolutely are bad microservice architectures, that's a weird claim to make. Good microservice architectures minimize or entirely eliminate the friction around the coordination and debugging. I'm working on splitting out a monorepo as we speak and I'm determining tradeoffs we'll need to make. If I do this wrong, there will be a lot of pain around navigating between the services, but there are choices I can make now that minimize (or eliminate in some cases) that pain. I recommend you find a way to spend time with someone who knows how to design a quality microservices architecture. It will completely change your opinion about how microservices interact and the supposed "pain" therein.
P5fRxh5kUvp2th 4y ago
I really wish more people understood this. I'm working for a company that's in the middle of a move to the cloud and it feels like no one is acknowledging the sheer level of complexity being added.
throwaway0asd 4y ago
This sounds like all but two of my software jobs. The greatest challenge of being an individual contributor is being beholden to the constraints of the environment. If they tell you to dig a hole then that’s what you do. If they tell you to waste all day digging into some unnecessary framework to make a single CSS change that should take 30 seconds plus a few minutes for deployment then that's what you do. In front end development people NEED the world's largest application frameworks (sometimes plural) because they cannot figure out state management. State management is stupid simple and a solved problem, but whatever. It will still, at the job, take you all day (maybe a week) to make that 30 second change.
- throwaway23597 4y ago
  This is why I hate frontend so much. Frontend sits right at the uncomfortable intersection between overly complicated tech that's tough to understand, and demanding PMs and UXs who need the feature to look or behave a certain way. I initially enjoyed frontend because of the hands-on nature of it, being able to make something and then play around with it in my browser. But after the 100th time wrestling around with CSS and JS to make some minuscule style change, I just am done with it.
  - throwaway0asd 4y ago
    I super enjoy pushing innovative new front end technologies in personal projects where a 30 second fix might take as much as 5 minutes after test automation and all necessary due diligence, but at past jobs I just dig that hole they tell me to dig. If you try to bring up that the days work could be solved in less than hour if only we didn’t make all the wrong decisions and fear the platform they will look at you like you’re retarded. I have heard this mentality expressed perfectly in the movie Edge of Tomorrow: Don’t tell anybody. Are we clear? Best case scenario you’re gonna end up in a psych ward. Worse case you’ll get dissected for study
    - Andys 4y ago
      Best argument around for doing your own startup.

by vaidhy 4y ago

jsharf 4y ago

I think it's actually normal for teams to have duplicate functionality. The requirements are usually never actually identical and when you try to merge them it results in services with complicated configuration that have N^2 edge cases which leads to more bugs or more complicated code. I suspect what you're describing is somewhat unavoidable, and not even a bad thing. That being said, Google for comparison does have good code re-use for certain core resources, like spanner, cloud, tensorflow, borg, etc. If you're talking about bedrock infrastructure like that, it's quite a different picture.

vaidhy 4y ago
I did not say it is bad. I just wanted to point out that N^2 communications are a big hit (of course we know them for long time) and people under-estimate the co-ordination effort and the fact that already running systems impose constraints that are not true for green-field efforts. I think I am regurgitating mythical man-month here and will shut up.
- VBprogrammer 4y ago
  > the fact that already running systems impose constraints that are not true for green-field efforts Maybe I'm doing it wrong but I often find that doing a new greenfield project sounds easier but by the time you deal with all of the non-functional stuff (connectivity, resilience, logging, build tools, documentation etc) you might have been quicker holding your nose and bodging it in somewhere vaguely appropriate in an existing system.
  - vaidhy 4y ago
    There is shit-ton of work to be done :)

runlevel1 4y ago

> micro-service architecture imposes a heavy penalty on team co-ordination This is, by far, the hardest scaling problem in SaaS: How do we divide responsibilities amongst our (code|services|teams) efficiently? You can only grow each so much before they become unwieldy to manage. Divide scope too much and communication becomes complex. Divide along the wrong abstraction and you get duplication, overlap, and confusion about what's responsible for what. And as time passes, decisions of the past constrain and complicate your options to redesign/rearchitect/reorg today.

matai_kolila 4y ago
So it's "bad micro-service architecture imposes a heavy penalty on team co-ordination", then. ...which is a fairly obvious observation. "Bad <x> imposes a heavy penalty on team co-ordination" is generalizable.
- vaidhy 4y ago
  There is no bad vs good micro-service architecture. It is just that micro-service architecture imposes a different kind of penalty on co-ordination and debugging. Instead of intra-component dependency, it moves it to inter-service dependency. Each service might look simple, but the complexity is moved to service communications. It is not that obvious, imho.
  - bobthepanda 4y ago
    Also, what is good and bad can change over time with the business. Few judgement calls in software will last forever.
  - deleted 4y ago
    [deleted]
  - matai_kolila 4y ago
    There absolutely are bad microservice architectures, that's a weird claim to make. Good microservice architectures minimize or entirely eliminate the friction around the coordination and debugging. I'm working on splitting out a monorepo as we speak and I'm determining tradeoffs we'll need to make. If I do this wrong, there will be a lot of pain around navigating between the services, but there are choices I can make now that minimize (or eliminate in some cases) that pain. I recommend you find a way to spend time with someone who knows how to design a quality microservices architecture. It will completely change your opinion about how microservices interact and the supposed "pain" therein.

P5fRxh5kUvp2th 4y ago

I really wish more people understood this. I'm working for a company that's in the middle of a move to the cloud and it feels like no one is acknowledging the sheer level of complexity being added.

throwaway0asd 4y ago

This sounds like all but two of my software jobs. The greatest challenge of being an individual contributor is being beholden to the constraints of the environment. If they tell you to dig a hole then that’s what you do. If they tell you to waste all day digging into some unnecessary framework to make a single CSS change that should take 30 seconds plus a few minutes for deployment then that's what you do. In front end development people NEED the world's largest application frameworks (sometimes plural) because they cannot figure out state management. State management is stupid simple and a solved problem, but whatever. It will still, at the job, take you all day (maybe a week) to make that 30 second change.

throwaway23597 4y ago
This is why I hate frontend so much. Frontend sits right at the uncomfortable intersection between overly complicated tech that's tough to understand, and demanding PMs and UXs who need the feature to look or behave a certain way. I initially enjoyed frontend because of the hands-on nature of it, being able to make something and then play around with it in my browser. But after the 100th time wrestling around with CSS and JS to make some minuscule style change, I just am done with it.
- throwaway0asd 4y ago
  I super enjoy pushing innovative new front end technologies in personal projects where a 30 second fix might take as much as 5 minutes after test automation and all necessary due diligence, but at past jobs I just dig that hole they tell me to dig. If you try to bring up that the days work could be solved in less than hour if only we didn’t make all the wrong decisions and fear the platform they will look at you like you’re retarded. I have heard this mentality expressed perfectly in the movie Edge of Tomorrow: Don’t tell anybody. Are we clear? Best case scenario you’re gonna end up in a psych ward. Worse case you’ll get dissected for study
  - Andys 4y ago
    Best argument around for doing your own startup.