10 ms·

This is a classic case of not understanding micro services and trying to fit a problem around a tool. At work, we have close to ~50 services(no one calls them

This is a classic case of not understanding micro services and trying to fit a problem around a tool. At work, we have close to ~50 services(no one calls them microservices), but they do not suffer from this brittleness. We segregate our services based on languages. So, all C services go under coco/ , all Java services go under jumanji/ , all go services go under goat/ , all JS services go under js/. This means, everytime you touch something under a repo, it affects everyone. You are forced to use existing code or improve it, or you risk breaking code for everyone else. What does this solve? This solves the fundamental problem a lot of leetcode/hackerrank monkeys miss, programming is a Social activity it is not a go into a cave and come out with a perfect solution in a month activity. More interaction among developers means Engineers are forced to account for trade offs. Software Engineering in its entirety is all about trade offs, unlike theoretical Comp Science. Anyway, this helps because as Engineers we must respect and account for other Engineers decisions. This methods helps tremendously to do this. No one complains, everyone who wants 1000 more microservices usually turns out to be a code monkey entangled in new fad, or who doesn't want to work with other Engineers. You want to use rust? There is a repo named fe2O3/, go on. Accountability and responsibility is on your shoulders now. If you think about it, an Engineer is tied to his tools, why not segregate repos at language level instead of some arbitrary boundary no one knows about in a dynamic ecosystem?

jiveturkey 8y ago
> This means, everytime you touch something under a repo, it affects everyone. This is horrible at scale. > This is a classic case of not understanding micro services and trying to fit a problem around a tool. That much I agree with. TFA even acknowledges that, in the conclusion. Not in so many words, but they basically admit they did it wrong.
- stickfigure 8y ago
  >> This means, everytime you touch something under a repo, it affects everyone. >This is horrible at scale. I just want to reiterate this. In the early 2000s I worked in the online platform group at EA. The list of things done poorly there was long, but picture: * 40+ engineers * Monorepo with hundreds of thousands of classes; all code deployed to all servers. * Hundreds of different services running across thousands of servers. * Communication based on Java serialization, so all code had to be deployed to all servers at the same time. * Deployments (and thus downtime) sometimes lasted up to a hour. Worldwide audience; it was always in the middle of someone's day. * Rational Clearcase for version control. It took nearly an hour to sync to tip. Pretty much every morning you'd come in, spend an hour syncing, find that someone broke the build, hunt them down, and resync for another hour. Generally speaking the first few hours of every day were wasted for 40 engineers. This was a very poor platform. Sometimes I wonder how it's going there these days.
  - alttab 8y ago
    The Java serialization downtime was sad, but I died when you said clear case. When they rolled out clear case in my team at IBM, I quit a few weeks later.
  - eitally 8y ago
    Funny that it was this bad at an actual tech company. What you described is pretty common in normal big enterprise, though.
- jdoliner 8y ago
  > This is horrible at scale. This can work fine at scale, Google does it with however many tens of thousands of engineers they have these days. Having everything in a monorepo doesn't solve the communication problem, it doesn't prevent solving it either.
  - eitally 8y ago
    Sort of. Google is also well known for having a dozen chat apps. What most people don't know is that there are also likely a dozen different libraries/utilities/interfaces performing near identical services just because an engineer or product team weren't 100% satisfied with what already existed. So yes, it's a monorep, but holy heck there's a lot of duplicative cruft.
  - seabrookmx 8y ago
    > all code deployed to all servers This is the main point of the post. Google most certainly doesn't do this, even if they have all their code (or at least, all their private code) in one repo (piper).
ljm 8y ago
I'd wager that microservices, a lot of the time, are basically used as a management structure rather than for their benefits as pure tech, so less mature teams can silo themselves off and avoid communication (e.g. "I can work just on my backend image processing bit without dealing with the React guys now", "now the CTO won't be on my back so much," or whatever). The irony being that anything approaching SOA (or microservices) requires exactly the same amount of communication. More likely they require more since it's almost certain that such a decision introduced chaos.
- stcredzero 8y ago
  I'd wager that microservices, a lot of the time, are basically used as a management structure rather than for their benefits as pure tech A lot of software is used to control or impose someone's will on others organizationally.
  - johndubchak 8y ago
    Very well said.
  - flukus 8y ago
    This is why things like ticketing systems and other workflow tools almost universally suck. Manager needs x field to run some report so x field becomes mandatory and the software is a pain to use because know one apart from that one manager knows what the field even means. Currently our JIRA kanban board is crippled because someone just had to take a simple system and impose a workflow that can't be deviated from.
- organsnyder 8y ago
  I agree, though I'm not so cynical about it. Well-defined API contracts are themselves a communication mechanism. If I provide an API, I am declaring that if you interact with me in a given way, I will behave in a certain way. Given that one of the hardest parts of scaling an organization is the boundaries between individuals and teams, providing a structured mechanism to define system behavior is incredibly valuable. Well-written APIs are SLAs on steroids.
thelastidiot 8y ago
Amen to that. Just by curiosity, what is the barrier of entry for adding a new language in your ecosystem?
- acroback 8y ago
  Totally technical considerations which engineers must present and justify. e.g When we added Go, we had to present what it brings? A good testing framework, light weight(subjective), goroutines(which fit our use case), in built benchmark support, mature support of third party packages etc etc.
  - YawningAngel 8y ago
    Aren't literally all of the above present in Java?
    - acroback 8y ago
      No, we do not want to pay extra to run Java services bloating our measly VMs. Also, we found that it is far easier to get consistent latency with Go code than with Java code. Again, YMMV benchmark for your use case. That said, Java doesn't have go routines, doesn't provide low level access to memory, benchmarking framework is easy to work with in Go. Go gives access to sync Pool which when use correctly gives extremely good gains. One of our measly dual core VMs, can do more than 40k QPS with predictable latency at 99 percentile. With JAVA oom panics were common and latency percentile was all over the place. Sure, you can perhaps tweak the Java code, but it was far easier to do with Go.
    - tetha 8y ago
      They are. Add around 1.5 gig of necessary heap per feature you use.
    - anxiouspete 8y ago
      Like the others said, our JVM apps would require at least 1GB for the JVM and then 1-4GB for heap, depending on the service. This made the average JVM service require about 4GB of RAM in the container scheduler. Meanwhile our Go services almost never had a working set larger than 256MB and most didn't even need that. We could schedule on average 16 Go services for each JVM service.
merinowool 8y ago
I wonder why those fancy names for all except JS? I would have called it jizz/
- acroback 8y ago
  Haha, I made that up to gain free internet points ;)
ljw1001 8y ago
what happens when the java code breaks the c code? Or does everyone roll-their-own everything in each language?
- acroback 8y ago
  This means either your contract agreement between services is not language agonistic or someone did not do their job when Interface schema was changed during reviews. Just like any other software, interfaces live in separate repos e.g protoplasm/ for proto-buf definitions. avrobber/ for avro and so on so forth. So, any change to protoplasm/ triggers automated tests on all other services irrespective of boundaries.
  - ljw1001 8y ago
    ok. But _much_ easier to have compile time checking between projects in the same language/repo. Not saying this approach is wrong, but it's a little like riding a unicycle when a perfectly good bicycle is sitting right there.
    - otterlicious 8y ago
      It's much easier to drive a car than an 18 wheel truck too, but the latter have their uses.
    - acroback 8y ago
      I think I did not present this correctly, I apologize for that. What I mean this this. Any IDL interfaces live at their functional boundaries. e.g all proto definitions shared internally by java services live under jumanji/proto/, all proto definitions shared internally bu Go service live under goat/proto/ and so on so forth. But anything which is shared in a public manner e.g any proto definitions between Java and C live outside either of these repos in top level common_proto/ repo. The build system is configured to take care of pulling the correct proto version from common_proto repo. We use maven for this, but nothing much we can do about that. Build systems are ugly and we have to live with maven in near future. EDIT: Oh that said, stop using required field in protobufs if you are stuck using proto2 like us, whoever decided to add required field to proto2 grammar made a mess. Things change and deprecate repeatedly, gazillions of required fields are annoying and recipe for disaster.
r00tanon 8y ago
Sharing schemas and contracts in a distributed architecture is a hard problem. The interesting thing is these are central to communication, so we should definitely talk as a team about them - if anything, that's probably the one thing if you had to choose what to talk about. Then the INTPs can scurry back to their holes and continue creating stuff while the ENFPs form a committee for the next pub walk. Full disclosure: INTP here.
- humbleMouse 8y ago
  Sharing schemas and contract's isnt a hard problem. Toss them in a shared repo and push it to artifactory. Anyone who updates it increments the repo. Done.
- hyperdimension 8y ago
  Are you sure that this dichotomy is a good way to approach the situation? It just seems like it just creates an us vs. them ideology.
spraak 8y ago
> all JS services go under js/ Aww, no clever name or the JS services?
- asdsa5325 8y ago
  badlanguage/ is too long
  - leoc 8y ago
    goodbits/ is smaller though ;)
  - spraak 8y ago
    Such a tired view/joke. Python is just as bad but doesn't receive nearly as much hate
    - JamesBarney 8y ago
      JavaScript is a great language because you can run it on pretty much any machine. Buts it's especially terrible because it was designed by a guy without much language expertise in a handful of weeks.
      - tylerhou 8y ago
        > Buts it's especially terrible because it was designed by a guy without much language expertise in a handful of weeks. The core isn't too bad; I'd argue that the main reason why it's hard to work with is because you don't have tight control over your execution environment, mostly since the language wasn't really standardized until pretty late. It also solves a much different problem than most languages since it has to optimize for better UX (not crashing the whole program on exceptions, for example).
    - asdsa5325 8y ago
      I'd like to hear why you think Python is as bad
LovelyWaste 8y ago
The grandiose tone of your message suggests you believe you are giving some insteful advice but all I see is a single suggestion grouped with a lot of that weird marketingspeak every single programmer, for whatever reason, feels they need to emulate on their blogs.
HumanDrivenDev 8y ago
This is a classic case of not understanding micro services and trying to fit a problem around a tool. This sentence needs to be repeated for everything on the 'wrong end' of the hype curve. This is a classic case of not understanding NoSQL and trying to fit a problem around a tool. This is a classic case of not understanding OOP and trying to fit a problem around a tool. This is a classic case of not understanding dynamic typing and trying to fit a problem around a tool.
- onderdonken 8y ago
  So what's the 'respectable end' of the veneration curve? Does it have an inverse sentence? This is a classic case of generally understanding Turing completeness and eventually, inadvertently implementing a half of Common Lisp in C anyway. Something like that?
  - HumanDrivenDev 8y ago
    Our industry is too immature to have veneration for things that are old, I think.
imajes 8y ago
What is the value proposition of tieing an ingester, parser, transactional service (e.g. all written with go), and then tieing together your api, client app, mobile app (all in js), just because they share a common language? arguably they aren't close to each other in tiers of the stack, and don't really overlap. The sorts of libraries you might choose to use to do something could differ, (e.g. json processing - in the front end, you'd pick usability and security over a lower level more optimized transcoder for the api). That said, I agree with your core premise- not understanding, and chasing a nail with a hammer, but geekily named repos per language just sounds like someone who doesn't understand how something git works...
magic_beans 8y ago
Can you be the Jack Doneghy to my Liz Lemon? Please?
pweissbrod 8y ago
Ok so if I want to build a fledgling project all I need to do is leverage a language nobody uses yet. Lolcats ftw. Seriously though constraining dev's to use a monolithic repository in the name of discouraging cowboys is like tying people's legs together to ensure they walk in an aligned direction. Sure it works but seriously unnecessary pain. Your company should invest in integration tests
danielravina 8y ago
why js/ didn't get a cool nickname?

jiveturkey 8y ago

> This means, everytime you touch something under a repo, it affects everyone. This is horrible at scale. > This is a classic case of not understanding micro services and trying to fit a problem around a tool. That much I agree with. TFA even acknowledges that, in the conclusion. Not in so many words, but they basically admit they did it wrong.

stickfigure 8y ago
>> This means, everytime you touch something under a repo, it affects everyone. >This is horrible at scale. I just want to reiterate this. In the early 2000s I worked in the online platform group at EA. The list of things done poorly there was long, but picture: * 40+ engineers * Monorepo with hundreds of thousands of classes; all code deployed to all servers. * Hundreds of different services running across thousands of servers. * Communication based on Java serialization, so all code had to be deployed to all servers at the same time. * Deployments (and thus downtime) sometimes lasted up to a hour. Worldwide audience; it was always in the middle of someone's day. * Rational Clearcase for version control. It took nearly an hour to sync to tip. Pretty much every morning you'd come in, spend an hour syncing, find that someone broke the build, hunt them down, and resync for another hour. Generally speaking the first few hours of every day were wasted for 40 engineers. This was a very poor platform. Sometimes I wonder how it's going there these days.
- alttab 8y ago
  The Java serialization downtime was sad, but I died when you said clear case. When they rolled out clear case in my team at IBM, I quit a few weeks later.
- eitally 8y ago
  Funny that it was this bad at an actual tech company. What you described is pretty common in normal big enterprise, though.
jdoliner 8y ago
> This is horrible at scale. This can work fine at scale, Google does it with however many tens of thousands of engineers they have these days. Having everything in a monorepo doesn't solve the communication problem, it doesn't prevent solving it either.
- eitally 8y ago
  Sort of. Google is also well known for having a dozen chat apps. What most people don't know is that there are also likely a dozen different libraries/utilities/interfaces performing near identical services just because an engineer or product team weren't 100% satisfied with what already existed. So yes, it's a monorep, but holy heck there's a lot of duplicative cruft.
- seabrookmx 8y ago
  > all code deployed to all servers This is the main point of the post. Google most certainly doesn't do this, even if they have all their code (or at least, all their private code) in one repo (piper).

ljm 8y ago

I'd wager that microservices, a lot of the time, are basically used as a management structure rather than for their benefits as pure tech, so less mature teams can silo themselves off and avoid communication (e.g. "I can work just on my backend image processing bit without dealing with the React guys now", "now the CTO won't be on my back so much," or whatever). The irony being that anything approaching SOA (or microservices) requires exactly the same amount of communication. More likely they require more since it's almost certain that such a decision introduced chaos.

stcredzero 8y ago
I'd wager that microservices, a lot of the time, are basically used as a management structure rather than for their benefits as pure tech A lot of software is used to control or impose someone's will on others organizationally.
- johndubchak 8y ago
  Very well said.
- flukus 8y ago
  This is why things like ticketing systems and other workflow tools almost universally suck. Manager needs x field to run some report so x field becomes mandatory and the software is a pain to use because know one apart from that one manager knows what the field even means. Currently our JIRA kanban board is crippled because someone just had to take a simple system and impose a workflow that can't be deviated from.
organsnyder 8y ago
I agree, though I'm not so cynical about it. Well-defined API contracts are themselves a communication mechanism. If I provide an API, I am declaring that if you interact with me in a given way, I will behave in a certain way. Given that one of the hardest parts of scaling an organization is the boundaries between individuals and teams, providing a structured mechanism to define system behavior is incredibly valuable. Well-written APIs are SLAs on steroids.

thelastidiot 8y ago

Amen to that. Just by curiosity, what is the barrier of entry for adding a new language in your ecosystem?

acroback 8y ago
Totally technical considerations which engineers must present and justify. e.g When we added Go, we had to present what it brings? A good testing framework, light weight(subjective), goroutines(which fit our use case), in built benchmark support, mature support of third party packages etc etc.
- YawningAngel 8y ago
  Aren't literally all of the above present in Java?
  - acroback 8y ago
    No, we do not want to pay extra to run Java services bloating our measly VMs. Also, we found that it is far easier to get consistent latency with Go code than with Java code. Again, YMMV benchmark for your use case. That said, Java doesn't have go routines, doesn't provide low level access to memory, benchmarking framework is easy to work with in Go. Go gives access to sync Pool which when use correctly gives extremely good gains. One of our measly dual core VMs, can do more than 40k QPS with predictable latency at 99 percentile. With JAVA oom panics were common and latency percentile was all over the place. Sure, you can perhaps tweak the Java code, but it was far easier to do with Go.
  - tetha 8y ago
    They are. Add around 1.5 gig of necessary heap per feature you use.
  - anxiouspete 8y ago
    Like the others said, our JVM apps would require at least 1GB for the JVM and then 1-4GB for heap, depending on the service. This made the average JVM service require about 4GB of RAM in the container scheduler. Meanwhile our Go services almost never had a working set larger than 256MB and most didn't even need that. We could schedule on average 16 Go services for each JVM service.

merinowool 8y ago

I wonder why those fancy names for all except JS? I would have called it jizz/

acroback 8y ago
Haha, I made that up to gain free internet points ;)

ljw1001 8y ago

what happens when the java code breaks the c code? Or does everyone roll-their-own everything in each language?

acroback 8y ago
This means either your contract agreement between services is not language agonistic or someone did not do their job when Interface schema was changed during reviews. Just like any other software, interfaces live in separate repos e.g protoplasm/ for proto-buf definitions. avrobber/ for avro and so on so forth. So, any change to protoplasm/ triggers automated tests on all other services irrespective of boundaries.
- ljw1001 8y ago
  ok. But _much_ easier to have compile time checking between projects in the same language/repo. Not saying this approach is wrong, but it's a little like riding a unicycle when a perfectly good bicycle is sitting right there.
  - otterlicious 8y ago
    It's much easier to drive a car than an 18 wheel truck too, but the latter have their uses.
  - acroback 8y ago
    I think I did not present this correctly, I apologize for that. What I mean this this. Any IDL interfaces live at their functional boundaries. e.g all proto definitions shared internally by java services live under jumanji/proto/, all proto definitions shared internally bu Go service live under goat/proto/ and so on so forth. But anything which is shared in a public manner e.g any proto definitions between Java and C live outside either of these repos in top level common_proto/ repo. The build system is configured to take care of pulling the correct proto version from common_proto repo. We use maven for this, but nothing much we can do about that. Build systems are ugly and we have to live with maven in near future. EDIT: Oh that said, stop using required field in protobufs if you are stuck using proto2 like us, whoever decided to add required field to proto2 grammar made a mess. Things change and deprecate repeatedly, gazillions of required fields are annoying and recipe for disaster.

r00tanon 8y ago

Sharing schemas and contracts in a distributed architecture is a hard problem. The interesting thing is these are central to communication, so we should definitely talk as a team about them - if anything, that's probably the one thing if you had to choose what to talk about. Then the INTPs can scurry back to their holes and continue creating stuff while the ENFPs form a committee for the next pub walk. Full disclosure: INTP here.

humbleMouse 8y ago
Sharing schemas and contract's isnt a hard problem. Toss them in a shared repo and push it to artifactory. Anyone who updates it increments the repo. Done.
hyperdimension 8y ago
Are you sure that this dichotomy is a good way to approach the situation? It just seems like it just creates an us vs. them ideology.

spraak 8y ago

> all JS services go under js/ Aww, no clever name or the JS services?

asdsa5325 8y ago
badlanguage/ is too long
- leoc 8y ago
  goodbits/ is smaller though ;)
- spraak 8y ago
  Such a tired view/joke. Python is just as bad but doesn't receive nearly as much hate
  - JamesBarney 8y ago
    JavaScript is a great language because you can run it on pretty much any machine. Buts it's especially terrible because it was designed by a guy without much language expertise in a handful of weeks.
    - tylerhou 8y ago
      > Buts it's especially terrible because it was designed by a guy without much language expertise in a handful of weeks. The core isn't too bad; I'd argue that the main reason why it's hard to work with is because you don't have tight control over your execution environment, mostly since the language wasn't really standardized until pretty late. It also solves a much different problem than most languages since it has to optimize for better UX (not crashing the whole program on exceptions, for example).
  - asdsa5325 8y ago
    I'd like to hear why you think Python is as bad

LovelyWaste 8y ago

The grandiose tone of your message suggests you believe you are giving some insteful advice but all I see is a single suggestion grouped with a lot of that weird marketingspeak every single programmer, for whatever reason, feels they need to emulate on their blogs.

HumanDrivenDev 8y ago

This is a classic case of not understanding micro services and trying to fit a problem around a tool. This sentence needs to be repeated for everything on the 'wrong end' of the hype curve. This is a classic case of not understanding NoSQL and trying to fit a problem around a tool. This is a classic case of not understanding OOP and trying to fit a problem around a tool. This is a classic case of not understanding dynamic typing and trying to fit a problem around a tool.

onderdonken 8y ago
So what's the 'respectable end' of the veneration curve? Does it have an inverse sentence? This is a classic case of generally understanding Turing completeness and eventually, inadvertently implementing a half of Common Lisp in C anyway. Something like that?
- HumanDrivenDev 8y ago
  Our industry is too immature to have veneration for things that are old, I think.

imajes 8y ago

What is the value proposition of tieing an ingester, parser, transactional service (e.g. all written with go), and then tieing together your api, client app, mobile app (all in js), just because they share a common language? arguably they aren't close to each other in tiers of the stack, and don't really overlap. The sorts of libraries you might choose to use to do something could differ, (e.g. json processing - in the front end, you'd pick usability and security over a lower level more optimized transcoder for the api). That said, I agree with your core premise- not understanding, and chasing a nail with a hammer, but geekily named repos per language just sounds like someone who doesn't understand how something git works...

magic_beans 8y ago

Can you be the Jack Doneghy to my Liz Lemon? Please?

pweissbrod 8y ago

Ok so if I want to build a fledgling project all I need to do is leverage a language nobody uses yet. Lolcats ftw. Seriously though constraining dev's to use a monolithic repository in the name of discouraging cowboys is like tying people's legs together to ensure they walk in an aligned direction. Sure it works but seriously unnecessary pain. Your company should invest in integration tests

danielravina 8y ago

why js/ didn't get a cool nickname?