10 ms·

"-No, look into microservices. It’s the future. It’s how we do everything now. You take your monolithic app and you split it into like 12 services. One for each

"-No, look into microservices. It’s the future. It’s how we do everything now. You take your monolithic app and you split it into like 12 services. One for each job you do. That seems excessive" A 100 times yes. We tried to split our monolithic Rails app into micro-services built in Go. 2 years and many fires later, we decided to abandon the project. It was mostly because the monitoring and alerting were now split into many different pieces. Also, the team spent too much time debating standards etc. I think micro-services can be valuable, but we definitely didn't do it right, and I think a lot of companies get it wrong. Any positive experiences with micro-services here?

forgottenacc56 10y ago
What was the benefit you envisioned?
- mrhektor 10y ago
  We were looking to break up our monolithic rails app into micro-services so devs could iterate and develop faster. We also thought that the application as a whole would become more failure resistant. Unfortunately, inter-dependancies among the services themselves meant that the failure-resistance didn't pan out as we thought.
nicolas_t 10y ago
I think that splitting into micro services is valuable if and only if you reach a scale where it makes sense to split into micro services. By scale, I mean the number of people on the team (if you have a lot of people, it can make sense to split into micro-services to limit communication bottlenecks between developers) or in term of traffic, in which case microservices can be very useful to better optimize the system piece by piece. A small team starting a new project should not waste a single second considering microservices unless there's something that is so completely obviously decoupled in a way that not splitting it into a microservice will lead to extra work. It's also way easier to split into microservices after the fact than when you're developing a new app and you don't have a clue how it will look like or what the overall structure of the app will be in a year (most common case for startups).
- vmateixeira 10y ago
  I think it's more to do with a need rather than going straight just because you have enough people on a team. For instance, if you find that some of your processing/specific request handling can outperform better by using a different framework, programming language than the ones it's currently developed on, then you should definitely consider a microservice approach by decoupling that specific service/functionality from your current stack.
  - RyanZAG 10y ago
    Careful with this one too. Usually adding new features to adapt to a changing marketplace can have new requirements across many of your services that need to be finished quickly. If those services are each in a different language, it can slow everything down by weeks or months. Multiple platforms is not a problem and generally a good thing as long as it's not excessive. You don't want to be in a case where you have the same number of different platforms as developers or anything like that. I'm guessing there is a rule of thumb here, but I'm not sure what it would be. Max 1 different platform per 5 developers? Something like that.
  - crdoconnor 10y ago
    >decoupling that specific service/functionality from your current stack. I do wish people would stop conflating "running in a different service" and "loose coupling". They are completely orthogonal. I've worked on some horrendously tightly coupled microservices.
    - jcadam 10y ago
      OSGi makes it easy to end up with a cornucopia of tightly coupled nanoservices all running in the same JVM. Unless you can coax dOSGi into working (which is tons of fun), then you can have services tightly coupled to other services running on entirely different machines causing frequent (and hilarious) cascades of bundle failures whenever the network hiccups. OSGi is a trigger word for me now. I've worked on two large OSGi projects (previous job and current job) and it's always the same. Sh*t is always broken (and my lead still insists that OSGi is the one true way to modular bliss). And the OSGi fanboys always say "Your team is using it wrong!" Which very well might be true, but I no longer care. Apparently it's just too damn hard to get a team of code monkeys to respect service boundaries when OSGi makes it so damn easy to ignore them. If I'm ever in a position of getting to design a new software architecture (hasn't happened in 10 years, but hey I can dream), I'll punch anyone who suggests "OSGi" to me right in the face.
LoSboccacc 10y ago
saw done properly once, with good separation of services etc, but it still required a truckload of commitment to make it work, because one thing is interface changes when the code just don't work when you merge, another is figuring out the publish and restart order of each service when you have to add an operation so you don't have to knock out the whole system at every upgrade. I don't really like that model applied to everything, but eh now you are kind of forced in a hybrid approach - say, your macro vertical plus whatever payment gateway service, intercom or equivalent customer interaction services, metrics services, retargeting services, there are a lot of heterogeneous pieces going into your average startup. but back on topic, what Docker really needs now is a whack on the head of whoever thought swarms/overlays and a proper, sane way to handle discovery and fail-over - instead we got a key-value service deployment to handle, which cannot be in docker and highly available unless you like infinite recursion.
mrweasel 10y ago
>Any positive experiences with micro-services here? It makes sense for some thing. We run a webshop, but have a separate service that handles everything regarding payments. It has worked out really well, because it allows us to fiddle around with pretty much everything else and not worry about breaking the payment part. It helps that it's system where we can have just one test deployment and everyone just uses that during testing of other systems. I've also work at a company where we had to run 12 different systems in their own VMs to have a full development environment. That sucked beyond belief. The idea of micro-service are is enticing, but if you need to spin up and configure more than a couple to do your work, it starts hurting productivity.
- buro9 10y ago
  > have a separate service that handles everything regarding payments. It has worked out really well, because it allows us to fiddle around with pretty much everything else and not worry about breaking the payment part. Is the payments service a single service that manages the whole transaction, or have you go for multiple services handling each part and, if so, how did you manage failure with a distributed transaction?
  - notimetorelax 10y ago
    Not sure if it's the case here. But what works really well for us is queues with at-least-once guarantee. (For payment services you might need an additional check to guarantee exactly one execution.) I think you can find this queue offered by most providers.
  - mrweasel 10y ago
    It's a single service. It just sites between us and our PSPs. That way no other system needs to worry about integrating directly with the PSPs.
- drdaeman 10y ago
  Totally agree. We had almost the same story with payments. Except for we've jumped to a payment-processing SaaS but got dissatisfied (all those SaaSes I saw don't work with PayPal EC without so-called "reference transactions" enabled) and decided that wasn't a good idea and we have to jump back to in-house implementation. I didn't want to re-integrate the payments code back to the monolith - thought it would take me more time and make code messier. So I wrote a service (it's small but to heck with "micro" prefix) that resembled that SaaS' API (the parts we've used). It had surely evolved and isn't compatible anymore, but it doesn't matter as we're not going back anyway. Works nicely and now I'm feel more relaxed - touching the monolith won't break payments. On the other hand, I see how too many services may easily lead to fatigue. Automated management tooling (stuff like docker-compose) may remedy this, but also may bring their own headaches.
humanrebar 10y ago
> Any positive experiences with micro-services here? Yep. We already had a feature flag system, a minimal monitoring system, and a robust alerting system in place. Microservices make our deployments much more granular. No longer do we have to roll back perfectly good changes because of bugs in unrelated parts of the codebase. Before, we had to have involved conversations about deployments, and there were many things we just didn't do because the change was too big. We can now incrementally upgrade library versions, upgrade language versions, and even change languages now, which is a huge win from the cleaning up technical debt perspective.
- aidos 10y ago
  How granular are your services? I've heard a lot of talk about microservices without much talk about how micro they are. As someone who's happy with the approach, would you mind giving a little context in terms of what sort of degree you've split the system down?
  - humanrebar 10y ago
    They are of varying sizes. We have maybe 3-4 per developer. And they vary in size. The smallest are maybe 5-6 python classes. To be honest, we still have a monolithic application at the heart of our system that we've been slow to decompose, though we're working on it. We deploy it on a regular cadence and use feature flags heavily to make it play nice with everything else.
- moron4hire 10y ago
  That sounds more like your team doesn't know how to use git beyond nothing more than an SVN replacement.
  - vertex-four 10y ago
    "We rolled out this update with 220 changes. There's a breaking bug. Where is it? We need to find out in the next 5 minutes, revert, and deploy. Otherwise we have to revert the whole thing- we're losing money." Git doesn't really help with that. More granular deployments do, and if microservices help with more granular deployments, go for it.
firasd 10y ago
I'm building a podcast discovery app and I find myself being de-facto pulled towards modularity. It's because my feed checker is in Elixir, my site is WordPress-based, and I communicate between them using the WP API, and I'm using Google Cloud SQL, and Elasticsearch on its own virtual machine... The thing is though, the Elixir feed checker has its own database table that tracks whether it's seen an episode in a feed. And when there's a new episode it sends an API call to WP to insert the new post. The problem is that sometimes the API calls fail! Now what? I'll need to build logging, re-try etc. So I'm thinking of making the feed checker 'stateless' and only using WP with a lot of query caching as the holder of 'state' information about whether an episode has been seen before. To sum up my experience so far, there's something nice about being able to use the right tech for each task, and separating resources for each service, but the complexity--keeping track of whether a task completed properly--definitely increases.
- iagooar 10y ago
  Care sharing your progress with the podcast discovery app? I'm a cofounder of Podigee which is a podcast hosting service. Maybe we can exchange know-how, find some synergies or even join forces on certain topics. Feel free to drop me a line at mati@podigee.com
  - firasd 10y ago
    Sure, emailed. Also added a screenshot/twitter to my profile in case anyone else is interested.
- asimuvPR 10y ago
  I think your problem might be not expanding wordpress. PHP will gladly do the above through wordpress plugins and cron jobs. I built something similar and ended up going that way. The system is still running to this day. Sure, its not hip but gets the job done with miniml fuss and makes money.
  - firasd 10y ago
    You are right that PHP can do the feed checking part but I wanted to use something with easy async/concurrency out of the box (and wanted to learn Elixir instead of using Node.) One hard tech limit is that with 50k podcasts, 4million+ episodes, search definitely doesn't work well. Not just WP, but SQL itself. Hence Elasticsearch. I also plan to work on recommendations, etc. so will need probably to be exporting SQL data into other systems anyway for making the "people who liked this also liked this" kinda things. Also I kinda lied about using the WP API--that's how I built the system initially (and will switch to it moving forward), but to import the first few million posts from the content of the feeds, I just used wp_insert_post against the DB of new entries that Elixir fetched (I posted the code I used here: http://wordpress.stackexchange.com/a/233786/30906 http://wordpress.stackexchange.com/a/233786/30906). I also plan to write the whole front-end in React (including server side rendering) so will have to figure out how to get that done. Would probably use the WP-API with a Node.js app in front of it, will look into hypernova from AirBNB. So probably more usage of WP API accessed by another service...
pmelendez 10y ago
I do have a positive micro-service experience, and although we are still in that process of breaking down our monolith SOA based app, we have seen the benefits already. The more dramatic effect was on a particular set of endpoints that have a relative high traffic (it peaks at 1000 req/s) that was killing the app, making upset our relational database (with frequent deadlocks) and driving our Elasticsearch cluster crazy. We did more than just split the endpoints into microservices. We also designed the new system to be more resilient. We changed our persistence strategy to make it more sensible to our traffic using a distributed key-value database and designed documents accordingly. The result was very dramatic, like entering into a loud club and suddenly everything goes silent. No more outages, very consistent response times, the instances scaled with traffic increase very smoothly and in overall a more robust system. The moral of this experience (at least for me) is that breaking a monolith app into pieces has to have a purpose and implies more than just move the code to several services keeping the same strategy (that's actually slower, time consuming and harder to monitor)
- abarringer 10y ago
  Which one of the four major improvements do you attribute the success to though? Could you have done the work on making it more resilient, persistence, sensible, redesign the docs without breaking into micro-services and still have seen the positive results?
  - pmelendez 10y ago
    I don't think the level success comes from one dimension, but I don't think either that we could have achieved the resiliency without breaking it in micro-services (or just services that happened to be small if you will). One key factor was decoupling the high volume updates from the users requests so one didn't affect the other one.
- mirekrusin 10y ago
  Do you think the result could also be a dramatic improvement if you kept old system and do those other things except splitting into microservices? I can't get my head around how people introduce changes to their system if they have to update 12 different microservices at once? It must be horrible. Often you hear stories how people are converting monolithic app to microservices - but this is easy. Rewriting code is easy and it's fair to say it always yields better code (with or without splitting into microservices - it doesn't matter). What I'd like to hear is something about companies doing active development in microservice world. How do they handle things like schema changes in postgres where 7 microservices are backed by the same db? What are the benefits compared to monolithic app in those cases? It seems to me that microservices can easily violate DRY because they "materialise" communication interfaces and changes need to be propagated at every api "barrier", no?
zer0gravity 10y ago
"Message passing as the fundamental operation of the OS is just an excercise in computer science masturbation. It may feel good, but you don't actually get anything DONE." -- Linus Torvalds He said that about micro-kernels but it may apply to micro-services as well. Breaking things apart too much will only cause you grief.
ptero 10y ago
Microservices is a technology. IMO, like any other tech, it should be used when there are clear benefits expected in the near future, not as a blanket "always microservices" policies. Although I personally had to deal with some monolithic monsters that I wished were split into smaller services.
stepanhruda 10y ago
Wait a minute, this sounds familiar. looks at username Oh.
- jessaustin 10y ago
  'mrhektor is a green account.
  - stepanhruda 10y ago
    I should've been more clear. The story sounded familiar, because I worked with them and recognized them based on the username.
    - mrhektor 10y ago
      Hi Stepan!
amelius 10y ago
The problem with microservices is that your state is spread over multiple systems. You completely lose the concept of transactional integrity, so you will have to work around that from the start. The advantage though is that APIs (system boundaries) are usually better defined. Perhaps one should use the best of both worlds, and run microservices on a common database, and somehow allow to pass transactions between services (so multiple services can act within the same transaction).
- alexhornbake 10y ago
  One of the big advantages of microservices is scaling/migrating the databases behind each service independently. If you need transactions across multiple services then one could argue that either your API endpoint is doing too much, or your services are doing too little. It's not perfect, and certainly not always convenient, but it's a balance. Microservices with a common DB is asking for trouble. The monolith is a better option in that case IMO.
- nbevans 10y ago
  Idempotency, event sourcing and sagas with compensations are ways to solve your problem. A shared database is an anti-pattern in distributed systems. Similarly, distributed transactions (ala. DTC) is an anti-pattern. Distributed systems aren't hard. They're just different.
  - prodigal_erik 10y ago
    Say you sell a widget. You want to update both your cash account and your inventory, and never one without the other. Which is easier to understand and more reliable: doing them atomically, or making sure you have designed in 2^n intermediate states and all the code required to complete work that should happen but hasn't yet?
- kafkaesq 10y ago
  The problem with microservices is that your state is spread over multiple systems. Then again, sometimes it's advantageous to identify parts of your system where aspects of state can be safely decoupled. And in which having them reside in disparate systems (and yes, sometimes be inconsistent or differently available) might actually be a better overall fit. You completely lose the concept of transactional integrity, so you will have to work around that from the start. Then again, sometimes your state changes not only don't need to be transactional; it can be disadvantageous to think of them that way. Depends, depends, depends.
augustl 10y ago
> Any positive experiences with micro-services here? In my experience, any monolith that can be broken up into a queue based system will benefit enormously. This cleans up the pipelines, and adds monitoring and scaling points (the queues). Queues removes run-time dependencies to the other services. It requires that these services are _actually_ independent, of course. I do, however, avoid RPC based micro-services like the plague. RPC adds run-time dependencies to services. If possible, I limit RPC to other (micro) services to launch/startup/initialization/bootstrap, not run-time. In many cases, though, the RPC can be avoided entirely.
FuNe 10y ago
<sarcastic mode on> Maybe so. But didn't the move create a cool Software Architect job position out of nowhere -wink-wink- ? <sarcastic mode off>
nailer 10y ago
> You take your monolithic app and you split it into like 12 services. The non-web world has been doing this with message queueing for about 15 years. Maybe more.
- sangnoir 10y ago
  You do know the author is a taking the piss out of the practice, right?
  - nailer 10y ago
    You know nothing in the comment you're replying to indicates I wouldn't, right?
- drdaeman 10y ago
  Probably more. I'd say, like, at least 30-40 years. I mean, the infamous "UNIX way" of "do one thing, do it well" (something we nearly lost with popularity of "do everything in a manner incompatible with how others do it" approach in too many modern systems), when complex behavior was frequently achieved through the modularity of smaller programs communicating through well-defined interfaces. Heck, microkernels are all about this, and their ideas haven't grew out of nowhere. And HURD (even though it was never finished) is quarter a century old already.
- flukus 10y ago
  Don't be so quick to assume micros services involve message queuing. For most it seems to just be an elaborate RPC mechanism (unfortunately).
  - nailer 10y ago
    Oh yeah, there's certainly other ways of doing it. My own experience is just that message queuing seems to be the default loosely-coupled RPC mechanism for larger orgs (from before the term 'micro services' was popular).
deleted 10y ago
[deleted]
LeanderK 10y ago
was it a complete rewrite? I don't think thats the right way to transition. Why didn't you try to separate the features one-by-one from the monolith? That would give more immediate feedback and real problems to work on instead of the possibility to get stuck on the holy architecture debate.
- mrhektor 10y ago
  No it wasn't a complete rewrite. We started by separating out the most mission critical components. Maybe that's where we went wrong; the most mission critical components were quite large and unwieldy to split out all at once. There was also the overhead of keeping the newly separated out component and the monolithic app in sync.
asimuvPR 10y ago
I only use a microservice if its something that can operate by itself. Things like a file data store, reports generation, etc. But all business logic goes in the monolith.
metaphorm 10y ago
positive experience with microservices: identify discrete functions that can considered "stateless" (i.e. no side effects, deterministic output for given input) and factor those out into stand-alone microservices. a good example of this that I've used in production at my current $dayjob: dynamic PDF generation. user makes request from our website, request data is used to fill out a pdf template context which is then sent over to our PDFgen microservice which does its thing and streams a response back to the user.
- mirekrusin 10y ago
  Does it connect to the database to fill in some values in the template? Does it keep connection pool of let's say 5 connection always open (as libraries like to do)? Does it have authentication? Is it public or private API? Who is managing security? Is it running behind it's own nginx or other proxy? Does it have DoS protection (PDF generation can be CPU intense)? What about the schema for request? How do you manage changes to the schema? They need to be deployed together with changes in other services, right? What about changes to database schema - you need to remember to update that service as well and redeploy it at the right time as well - just after successful db migrations - which live in another project. All of that and much more needs to be replicated for each microservice, right? Why not just have a module in your monolithic app that does it. The logic will still be separate. In most languages/frameworks you can spawn pdf generation task. Any changes are easier to introduce as well. There's no artificially materialised interface. Updates are naturally introduced. All auth logic is there already, you don't need to worry about deploying yet another service, same with logging etc.
  - metaphorm 10y ago
    > Does it connect to the database to fill in some values in the template? the template has values that are related to database models. the main app (still mostly monolithic) fills out the template context. the context itself is what's passed to the microservice. the microservice does not connect to a database at all. > Does it keep connection pool of let's say 5 connection always open (as libraries like to do)? no. the service probably handles a few hundred requests per day, it is not in constant use. communication is over HTTPS. it opens a new connection on each request. this does impact throughput, but its a low throughput use case, and pdf rendering itself is much slower and that time totally dominates the overhead of opening and closing connections anyway. > Does it have authentication? yes, it auths with a bearer token that is borne only by our own internal server. this is backend technology so we don't have to auth an arbitrary user. we know in advance which users are authorized. > Is it public or private API? private > Who is managing security? we are, with a lot of assistance from the built-in security model of AWS. > Is it running behind it's own nginx or other proxy? the main app is behind nginx. the microservice is running in a docker container that exposes itself over a dedicated port. there's no proxy for the microservice, again, because of the low throughput/low load on the service. no need to have a load balancer for this so the most obvious benefit of a proxy wasn't applicable. > Does it have DoS protection (PDF generation can be CPU intense)? yes, it's an internal service and our entire infrastructure is deployed behind a gatekeeper server and firewall. the service is inaccessible by outside requests. the internal requests are queue'd up and processed 1 at a time. > What about the schema for request? request payload validation handled on both ends. the user input is validated by the main app to form a valid template context. the pdf generator validates the template context before attempting to generate one also. its possible to have a valid schema that has data that can't be handled correctly though. errors are just returned as a 500 response though. happens infrequently. > They need to be deployed together with changes in other services, right? nope. the microservice is fully stand alone. > What about changes to database schema - you need to remember to update that service as well and redeploy it at the right time as well - just after successful db migrations - which live in another project. the microservice doesn't interact with a database at all. schema changes in the main app database could potentially influence the pdf template context generation, but there are unit tests for that, so if it does happen we'll get visibility in a test failure and update the template context generation code as needed. none of this impacts the microservice itself though. it is fully stand alone. that's the point. > All of that and much more needs to be replicated for each microservice, right? in principle yes, and these are good guidelines for determining what is or is not suitable to be a microservice. if it would need to auth an arbitrary user, or have direct database access, or be exposed to public requests, it might not be a good candidate for a microservice. things that can stand alone and have limited functional dependencies are much better candidates. > Why not just have a module in your monolithic app that does it. because the monolithic app is Python/django and the PDF generation tool is Java. one of the main advantages of microservices architecture is much greater flexibility in technology selection. A previous solution used Python subprocesses to call out to PDF generation software. It's actually easier and cleaner for us to use a microservice instead.
shados 10y ago
Yeah, we do micro services, the "real" kind. Not the "SOA with a new name kind", but the "some services are literally a few douzan lines of code and we have 100x the amount of services as we do devs" kind. The thing is, you need a massive investment in infrastructure to make it happen. But once you do, its great. You can create and deploy a new service in a few seconds. You can rewrite any individual service to be latest and greatest in an afternoon. Different teams don't have to agree on coding standards (so you don't argue about it). But, the infrastructure cost is really high, a big chunk of what you save in development you pay in devops, and its harder to be "eventually consistant" (eg: an upgrade of your stack across the board can take 10x longer, because there's no big push that HAS to happen for a tiny piece to get the benefits). Monolithic apps have their advantages too, and many forget it: less devops cost, easier to refactor (especially in statically typed languages: a right click -> rename will propagate through the entire app) and while its harder to upgrade the stack, once its done, your entire stack is up to date, not just parts of it being all over. Code reuse is significantly easier, too.
- hobs 10y ago
  Yeah, add in things like MORE THAN ONE PRODUCTION ENVIRONMENT and LETTING YOUR CUSTOMER HOST AN INSTANCE OF YOUR MICROSERVICES and you have guaranteed your own suffering.
  - neeleshs 10y ago
    It is a matter of tooling. One data center or ten, it does not matter much with proper tooling. We deploy to seven data centers with a click of a button, with rollback, staggered deployment etc. Centralized logging using ELK gives us great visibility in to each DC, without worrying about individual microservice instances.
    - bpicolo 10y ago
      Easy until you realize you need to somehow manage + configure hundreds of services to run your dev environment...
themartorana 10y ago
Yes. A lot of success. And with only one person on the backend full time. That said, in places where it doesn't make sense we didn't try to force it. Our main game API is somewhat monolithic, but behind it we have almost 10 other services. Here's a quick breakdown: - Turn based API service (largest, "monolithic") - Real-time API service (about 50% the size of turn-based) - config service (serves configuration settings to clients for game balancing) - ad waterfall service (dynamic waterfall, no actual ads) - push notification service - analytics collection service (mostly a fast collector that dumps into Big Query) - Open graph service (for rich sharing) - push maintenance service (executes token management based on GCM/APNS feedback) - help desk form service (simple front-end to help desk) - service update service (monitors CI for new binaries, updates services on the fly - made easy by Go binary deployment from CI to S3) - service ping service (monitors all service health, responds to ELB pings) - Facebook web front-end service (just serves WebGL version of our game binary for play on Facebook) - NATS.io for all IPC between services ...and a few more in the works. Some of these might push the line of "micro" in that they almost all do more than a single function's worth of work, but that level of granularity isn't practical. But don't get too caught up on the "micro" part. Split services where domain lines naturally form, and don't constrain service size by arbitrary definitions. You know, right tool for the job and whatnot.
wtbob 10y ago
> Also, the team spent too much time debating standards etc. Ah yes, the 'let's have decentralised microservices with centralised standards!' anti-pattern. It results in lots of full-fledged, heavyweight, slow-to-update services, which also have all the problems of a distributed system. It's the worst of both worlds.
clifanatic 10y ago
> It was mostly because the monitoring and alerting were now split into many different pieces Well, there's your problem - you need a monitoring microservice and an alerting microservice! Well, those may be too coarse by themselves, but once you break them down into 5 or 6 microservices each, you'll be ready for production.
huherto 10y ago
> Also, the team spent too much time debating standards etc. IMHO. You need a lead with a clear vision that drives the effort. Too many leads will create chaos.
- mrhektor 10y ago
  We did have a lead with a vision, and part of the vision was standards for each service (for example, file structure in Go). I can see the rationale behind it; a new dev can onboard very quickly on to a new service. But in hindsight, maybe it wasn't thought out enough.
- deleted 10y ago
  [deleted]
vonmoltke 10y ago
> Any positive experiences with micro-services here? I'm currently working on a large refactoring effort along these lines. The end goal is to create a modular, potentially distributed system that can be deployed in a variety of configurations, updated piecemeal for different customers, and integrated by our customers with the third-party or in-house code of their choice using defined APIs. We aren't typical of the other examples, though, in that we do literally ship our software to our customers and they run it on their own clusters.
mr_tristan 10y ago
Oh yes. We're splitting up a large monolith into a bunch of different services. Completely amazing, though there's a ton of tools (like Netflix's Hysterix, etc) that make it much, much easier to do. I wouldn't, however, just "do microservices" from day one on a young app. But usually that young app has no idea what the true business value is, i.e., you have no idea what down time of certain parts of your services really means to the business. That's the #1 pain point we're solving: having mission critical things up 100%, and then rapidly iterating on new, less stable feature designs in separate services. You should, however, keep an eye on how "splittable" everything is, i.e., does everything need to be in the same DB schema? Most languages have package concepts, which typically align (somehow) with "service" concepts. Do you know their dependencies? That sort of thing. Then, the later process of "refactor -> split out service" is pretty straightforward and easy to plan.
Mc_Big_G 10y ago
Heh, since the day I heard of "microservices" the only thing I could think was "have fun maintaining that".
Alex63 10y ago
I don't know if the community will consider this example to be microservices as currently defined, but many years ago I wrote client-server systems using a PC-based fat client application, and transactions running on a CICS server. I think this was pretty similar to what people currently think of as a microservices architecture, although we didn't have to worry about running/monitoring multiple servers (all the transactions/services ran on a single mainframe server), and the transaction monitor managed things like start-up and shutdown pretty simply. This approach worked really well for us, and we built several robust, scalable applications using this approach. To be clear, we numbered our users in hundreds, not thousands or millions. I can well understand how scaling this approach across many servers could be very challenging.
nickbauman 10y ago
You split off specific parts of your app into microservices because you want to scale those parts independently from the rest. It's not just a blind decomposition of a monolith for the sake of decomposition.

forgottenacc56 10y ago

What was the benefit you envisioned?

mrhektor 10y ago
We were looking to break up our monolithic rails app into micro-services so devs could iterate and develop faster. We also thought that the application as a whole would become more failure resistant. Unfortunately, inter-dependancies among the services themselves meant that the failure-resistance didn't pan out as we thought.

nicolas_t 10y ago

I think that splitting into micro services is valuable if and only if you reach a scale where it makes sense to split into micro services. By scale, I mean the number of people on the team (if you have a lot of people, it can make sense to split into micro-services to limit communication bottlenecks between developers) or in term of traffic, in which case microservices can be very useful to better optimize the system piece by piece. A small team starting a new project should not waste a single second considering microservices unless there's something that is so completely obviously decoupled in a way that not splitting it into a microservice will lead to extra work. It's also way easier to split into microservices after the fact than when you're developing a new app and you don't have a clue how it will look like or what the overall structure of the app will be in a year (most common case for startups).

vmateixeira 10y ago
I think it's more to do with a need rather than going straight just because you have enough people on a team. For instance, if you find that some of your processing/specific request handling can outperform better by using a different framework, programming language than the ones it's currently developed on, then you should definitely consider a microservice approach by decoupling that specific service/functionality from your current stack.
- RyanZAG 10y ago
  Careful with this one too. Usually adding new features to adapt to a changing marketplace can have new requirements across many of your services that need to be finished quickly. If those services are each in a different language, it can slow everything down by weeks or months. Multiple platforms is not a problem and generally a good thing as long as it's not excessive. You don't want to be in a case where you have the same number of different platforms as developers or anything like that. I'm guessing there is a rule of thumb here, but I'm not sure what it would be. Max 1 different platform per 5 developers? Something like that.
- crdoconnor 10y ago
  >decoupling that specific service/functionality from your current stack. I do wish people would stop conflating "running in a different service" and "loose coupling". They are completely orthogonal. I've worked on some horrendously tightly coupled microservices.
  - jcadam 10y ago
    OSGi makes it easy to end up with a cornucopia of tightly coupled nanoservices all running in the same JVM. Unless you can coax dOSGi into working (which is tons of fun), then you can have services tightly coupled to other services running on entirely different machines causing frequent (and hilarious) cascades of bundle failures whenever the network hiccups. OSGi is a trigger word for me now. I've worked on two large OSGi projects (previous job and current job) and it's always the same. Sh*t is always broken (and my lead still insists that OSGi is the one true way to modular bliss). And the OSGi fanboys always say "Your team is using it wrong!" Which very well might be true, but I no longer care. Apparently it's just too damn hard to get a team of code monkeys to respect service boundaries when OSGi makes it so damn easy to ignore them. If I'm ever in a position of getting to design a new software architecture (hasn't happened in 10 years, but hey I can dream), I'll punch anyone who suggests "OSGi" to me right in the face.

LoSboccacc 10y ago

saw done properly once, with good separation of services etc, but it still required a truckload of commitment to make it work, because one thing is interface changes when the code just don't work when you merge, another is figuring out the publish and restart order of each service when you have to add an operation so you don't have to knock out the whole system at every upgrade. I don't really like that model applied to everything, but eh now you are kind of forced in a hybrid approach - say, your macro vertical plus whatever payment gateway service, intercom or equivalent customer interaction services, metrics services, retargeting services, there are a lot of heterogeneous pieces going into your average startup. but back on topic, what Docker really needs now is a whack on the head of whoever thought swarms/overlays and a proper, sane way to handle discovery and fail-over - instead we got a key-value service deployment to handle, which cannot be in docker and highly available unless you like infinite recursion.

mrweasel 10y ago

>Any positive experiences with micro-services here? It makes sense for some thing. We run a webshop, but have a separate service that handles everything regarding payments. It has worked out really well, because it allows us to fiddle around with pretty much everything else and not worry about breaking the payment part. It helps that it's system where we can have just one test deployment and everyone just uses that during testing of other systems. I've also work at a company where we had to run 12 different systems in their own VMs to have a full development environment. That sucked beyond belief. The idea of micro-service are is enticing, but if you need to spin up and configure more than a couple to do your work, it starts hurting productivity.

buro9 10y ago
> have a separate service that handles everything regarding payments. It has worked out really well, because it allows us to fiddle around with pretty much everything else and not worry about breaking the payment part. Is the payments service a single service that manages the whole transaction, or have you go for multiple services handling each part and, if so, how did you manage failure with a distributed transaction?
- notimetorelax 10y ago
  Not sure if it's the case here. But what works really well for us is queues with at-least-once guarantee. (For payment services you might need an additional check to guarantee exactly one execution.) I think you can find this queue offered by most providers.
- mrweasel 10y ago
  It's a single service. It just sites between us and our PSPs. That way no other system needs to worry about integrating directly with the PSPs.
drdaeman 10y ago
Totally agree. We had almost the same story with payments. Except for we've jumped to a payment-processing SaaS but got dissatisfied (all those SaaSes I saw don't work with PayPal EC without so-called "reference transactions" enabled) and decided that wasn't a good idea and we have to jump back to in-house implementation. I didn't want to re-integrate the payments code back to the monolith - thought it would take me more time and make code messier. So I wrote a service (it's small but to heck with "micro" prefix) that resembled that SaaS' API (the parts we've used). It had surely evolved and isn't compatible anymore, but it doesn't matter as we're not going back anyway. Works nicely and now I'm feel more relaxed - touching the monolith won't break payments. On the other hand, I see how too many services may easily lead to fatigue. Automated management tooling (stuff like docker-compose) may remedy this, but also may bring their own headaches.

humanrebar 10y ago

> Any positive experiences with micro-services here? Yep. We already had a feature flag system, a minimal monitoring system, and a robust alerting system in place. Microservices make our deployments much more granular. No longer do we have to roll back perfectly good changes because of bugs in unrelated parts of the codebase. Before, we had to have involved conversations about deployments, and there were many things we just didn't do because the change was too big. We can now incrementally upgrade library versions, upgrade language versions, and even change languages now, which is a huge win from the cleaning up technical debt perspective.

aidos 10y ago
How granular are your services? I've heard a lot of talk about microservices without much talk about how micro they are. As someone who's happy with the approach, would you mind giving a little context in terms of what sort of degree you've split the system down?
- humanrebar 10y ago
  They are of varying sizes. We have maybe 3-4 per developer. And they vary in size. The smallest are maybe 5-6 python classes. To be honest, we still have a monolithic application at the heart of our system that we've been slow to decompose, though we're working on it. We deploy it on a regular cadence and use feature flags heavily to make it play nice with everything else.
moron4hire 10y ago
That sounds more like your team doesn't know how to use git beyond nothing more than an SVN replacement.
- vertex-four 10y ago
  "We rolled out this update with 220 changes. There's a breaking bug. Where is it? We need to find out in the next 5 minutes, revert, and deploy. Otherwise we have to revert the whole thing- we're losing money." Git doesn't really help with that. More granular deployments do, and if microservices help with more granular deployments, go for it.

firasd 10y ago

I'm building a podcast discovery app and I find myself being de-facto pulled towards modularity. It's because my feed checker is in Elixir, my site is WordPress-based, and I communicate between them using the WP API, and I'm using Google Cloud SQL, and Elasticsearch on its own virtual machine... The thing is though, the Elixir feed checker has its own database table that tracks whether it's seen an episode in a feed. And when there's a new episode it sends an API call to WP to insert the new post. The problem is that sometimes the API calls fail! Now what? I'll need to build logging, re-try etc. So I'm thinking of making the feed checker 'stateless' and only using WP with a lot of query caching as the holder of 'state' information about whether an episode has been seen before. To sum up my experience so far, there's something nice about being able to use the right tech for each task, and separating resources for each service, but the complexity--keeping track of whether a task completed properly--definitely increases.

iagooar 10y ago
Care sharing your progress with the podcast discovery app? I'm a cofounder of Podigee which is a podcast hosting service. Maybe we can exchange know-how, find some synergies or even join forces on certain topics. Feel free to drop me a line at mati@podigee.com
- firasd 10y ago
  Sure, emailed. Also added a screenshot/twitter to my profile in case anyone else is interested.
asimuvPR 10y ago
I think your problem might be not expanding wordpress. PHP will gladly do the above through wordpress plugins and cron jobs. I built something similar and ended up going that way. The system is still running to this day. Sure, its not hip but gets the job done with miniml fuss and makes money.
- firasd 10y ago
  You are right that PHP can do the feed checking part but I wanted to use something with easy async/concurrency out of the box (and wanted to learn Elixir instead of using Node.) One hard tech limit is that with 50k podcasts, 4million+ episodes, search definitely doesn't work well. Not just WP, but SQL itself. Hence Elasticsearch. I also plan to work on recommendations, etc. so will need probably to be exporting SQL data into other systems anyway for making the "people who liked this also liked this" kinda things. Also I kinda lied about using the WP API--that's how I built the system initially (and will switch to it moving forward), but to import the first few million posts from the content of the feeds, I just used wp_insert_post against the DB of new entries that Elixir fetched (I posted the code I used here: http://wordpress.stackexchange.com/a/233786/30906 http://wordpress.stackexchange.com/a/233786/30906). I also plan to write the whole front-end in React (including server side rendering) so will have to figure out how to get that done. Would probably use the WP-API with a Node.js app in front of it, will look into hypernova from AirBNB. So probably more usage of WP API accessed by another service...

pmelendez 10y ago

I do have a positive micro-service experience, and although we are still in that process of breaking down our monolith SOA based app, we have seen the benefits already. The more dramatic effect was on a particular set of endpoints that have a relative high traffic (it peaks at 1000 req/s) that was killing the app, making upset our relational database (with frequent deadlocks) and driving our Elasticsearch cluster crazy. We did more than just split the endpoints into microservices. We also designed the new system to be more resilient. We changed our persistence strategy to make it more sensible to our traffic using a distributed key-value database and designed documents accordingly. The result was very dramatic, like entering into a loud club and suddenly everything goes silent. No more outages, very consistent response times, the instances scaled with traffic increase very smoothly and in overall a more robust system. The moral of this experience (at least for me) is that breaking a monolith app into pieces has to have a purpose and implies more than just move the code to several services keeping the same strategy (that's actually slower, time consuming and harder to monitor)

abarringer 10y ago
Which one of the four major improvements do you attribute the success to though? Could you have done the work on making it more resilient, persistence, sensible, redesign the docs without breaking into micro-services and still have seen the positive results?
- pmelendez 10y ago
  I don't think the level success comes from one dimension, but I don't think either that we could have achieved the resiliency without breaking it in micro-services (or just services that happened to be small if you will). One key factor was decoupling the high volume updates from the users requests so one didn't affect the other one.
mirekrusin 10y ago
Do you think the result could also be a dramatic improvement if you kept old system and do those other things except splitting into microservices? I can't get my head around how people introduce changes to their system if they have to update 12 different microservices at once? It must be horrible. Often you hear stories how people are converting monolithic app to microservices - but this is easy. Rewriting code is easy and it's fair to say it always yields better code (with or without splitting into microservices - it doesn't matter). What I'd like to hear is something about companies doing active development in microservice world. How do they handle things like schema changes in postgres where 7 microservices are backed by the same db? What are the benefits compared to monolithic app in those cases? It seems to me that microservices can easily violate DRY because they "materialise" communication interfaces and changes need to be propagated at every api "barrier", no?

zer0gravity 10y ago

"Message passing as the fundamental operation of the OS is just an excercise in computer science masturbation. It may feel good, but you don't actually get anything DONE." -- Linus Torvalds He said that about micro-kernels but it may apply to micro-services as well. Breaking things apart too much will only cause you grief.

ptero 10y ago

Microservices is a technology. IMO, like any other tech, it should be used when there are clear benefits expected in the near future, not as a blanket "always microservices" policies. Although I personally had to deal with some monolithic monsters that I wished were split into smaller services.

stepanhruda 10y ago

Wait a minute, this sounds familiar. looks at username Oh.

jessaustin 10y ago
'mrhektor is a green account.
- stepanhruda 10y ago
  I should've been more clear. The story sounded familiar, because I worked with them and recognized them based on the username.
  - mrhektor 10y ago
    Hi Stepan!

amelius 10y ago

The problem with microservices is that your state is spread over multiple systems. You completely lose the concept of transactional integrity, so you will have to work around that from the start. The advantage though is that APIs (system boundaries) are usually better defined. Perhaps one should use the best of both worlds, and run microservices on a common database, and somehow allow to pass transactions between services (so multiple services can act within the same transaction).

alexhornbake 10y ago
One of the big advantages of microservices is scaling/migrating the databases behind each service independently. If you need transactions across multiple services then one could argue that either your API endpoint is doing too much, or your services are doing too little. It's not perfect, and certainly not always convenient, but it's a balance. Microservices with a common DB is asking for trouble. The monolith is a better option in that case IMO.
nbevans 10y ago
Idempotency, event sourcing and sagas with compensations are ways to solve your problem. A shared database is an anti-pattern in distributed systems. Similarly, distributed transactions (ala. DTC) is an anti-pattern. Distributed systems aren't hard. They're just different.
- prodigal_erik 10y ago
  Say you sell a widget. You want to update both your cash account and your inventory, and never one without the other. Which is easier to understand and more reliable: doing them atomically, or making sure you have designed in 2^n intermediate states and all the code required to complete work that should happen but hasn't yet?
kafkaesq 10y ago
The problem with microservices is that your state is spread over multiple systems. Then again, sometimes it's advantageous to identify parts of your system where aspects of state can be safely decoupled. And in which having them reside in disparate systems (and yes, sometimes be inconsistent or differently available) might actually be a better overall fit. You completely lose the concept of transactional integrity, so you will have to work around that from the start. Then again, sometimes your state changes not only don't need to be transactional; it can be disadvantageous to think of them that way. Depends, depends, depends.

augustl 10y ago

> Any positive experiences with micro-services here? In my experience, any monolith that can be broken up into a queue based system will benefit enormously. This cleans up the pipelines, and adds monitoring and scaling points (the queues). Queues removes run-time dependencies to the other services. It requires that these services are _actually_ independent, of course. I do, however, avoid RPC based micro-services like the plague. RPC adds run-time dependencies to services. If possible, I limit RPC to other (micro) services to launch/startup/initialization/bootstrap, not run-time. In many cases, though, the RPC can be avoided entirely.

FuNe 10y ago

<sarcastic mode on> Maybe so. But didn't the move create a cool Software Architect job position out of nowhere -wink-wink- ? <sarcastic mode off>

nailer 10y ago

> You take your monolithic app and you split it into like 12 services. The non-web world has been doing this with message queueing for about 15 years. Maybe more.

sangnoir 10y ago
You do know the author is a taking the piss out of the practice, right?
- nailer 10y ago
  You know nothing in the comment you're replying to indicates I wouldn't, right?
drdaeman 10y ago
Probably more. I'd say, like, at least 30-40 years. I mean, the infamous "UNIX way" of "do one thing, do it well" (something we nearly lost with popularity of "do everything in a manner incompatible with how others do it" approach in too many modern systems), when complex behavior was frequently achieved through the modularity of smaller programs communicating through well-defined interfaces. Heck, microkernels are all about this, and their ideas haven't grew out of nowhere. And HURD (even though it was never finished) is quarter a century old already.
flukus 10y ago
Don't be so quick to assume micros services involve message queuing. For most it seems to just be an elaborate RPC mechanism (unfortunately).
- nailer 10y ago
  Oh yeah, there's certainly other ways of doing it. My own experience is just that message queuing seems to be the default loosely-coupled RPC mechanism for larger orgs (from before the term 'micro services' was popular).

deleted 10y ago

[deleted]

LeanderK 10y ago

was it a complete rewrite? I don't think thats the right way to transition. Why didn't you try to separate the features one-by-one from the monolith? That would give more immediate feedback and real problems to work on instead of the possibility to get stuck on the holy architecture debate.

mrhektor 10y ago
No it wasn't a complete rewrite. We started by separating out the most mission critical components. Maybe that's where we went wrong; the most mission critical components were quite large and unwieldy to split out all at once. There was also the overhead of keeping the newly separated out component and the monolithic app in sync.

asimuvPR 10y ago

I only use a microservice if its something that can operate by itself. Things like a file data store, reports generation, etc. But all business logic goes in the monolith.

metaphorm 10y ago

positive experience with microservices: identify discrete functions that can considered "stateless" (i.e. no side effects, deterministic output for given input) and factor those out into stand-alone microservices. a good example of this that I've used in production at my current $dayjob: dynamic PDF generation. user makes request from our website, request data is used to fill out a pdf template context which is then sent over to our PDFgen microservice which does its thing and streams a response back to the user.

mirekrusin 10y ago
Does it connect to the database to fill in some values in the template? Does it keep connection pool of let's say 5 connection always open (as libraries like to do)? Does it have authentication? Is it public or private API? Who is managing security? Is it running behind it's own nginx or other proxy? Does it have DoS protection (PDF generation can be CPU intense)? What about the schema for request? How do you manage changes to the schema? They need to be deployed together with changes in other services, right? What about changes to database schema - you need to remember to update that service as well and redeploy it at the right time as well - just after successful db migrations - which live in another project. All of that and much more needs to be replicated for each microservice, right? Why not just have a module in your monolithic app that does it. The logic will still be separate. In most languages/frameworks you can spawn pdf generation task. Any changes are easier to introduce as well. There's no artificially materialised interface. Updates are naturally introduced. All auth logic is there already, you don't need to worry about deploying yet another service, same with logging etc.
- metaphorm 10y ago
  > Does it connect to the database to fill in some values in the template? the template has values that are related to database models. the main app (still mostly monolithic) fills out the template context. the context itself is what's passed to the microservice. the microservice does not connect to a database at all. > Does it keep connection pool of let's say 5 connection always open (as libraries like to do)? no. the service probably handles a few hundred requests per day, it is not in constant use. communication is over HTTPS. it opens a new connection on each request. this does impact throughput, but its a low throughput use case, and pdf rendering itself is much slower and that time totally dominates the overhead of opening and closing connections anyway. > Does it have authentication? yes, it auths with a bearer token that is borne only by our own internal server. this is backend technology so we don't have to auth an arbitrary user. we know in advance which users are authorized. > Is it public or private API? private > Who is managing security? we are, with a lot of assistance from the built-in security model of AWS. > Is it running behind it's own nginx or other proxy? the main app is behind nginx. the microservice is running in a docker container that exposes itself over a dedicated port. there's no proxy for the microservice, again, because of the low throughput/low load on the service. no need to have a load balancer for this so the most obvious benefit of a proxy wasn't applicable. > Does it have DoS protection (PDF generation can be CPU intense)? yes, it's an internal service and our entire infrastructure is deployed behind a gatekeeper server and firewall. the service is inaccessible by outside requests. the internal requests are queue'd up and processed 1 at a time. > What about the schema for request? request payload validation handled on both ends. the user input is validated by the main app to form a valid template context. the pdf generator validates the template context before attempting to generate one also. its possible to have a valid schema that has data that can't be handled correctly though. errors are just returned as a 500 response though. happens infrequently. > They need to be deployed together with changes in other services, right? nope. the microservice is fully stand alone. > What about changes to database schema - you need to remember to update that service as well and redeploy it at the right time as well - just after successful db migrations - which live in another project. the microservice doesn't interact with a database at all. schema changes in the main app database could potentially influence the pdf template context generation, but there are unit tests for that, so if it does happen we'll get visibility in a test failure and update the template context generation code as needed. none of this impacts the microservice itself though. it is fully stand alone. that's the point. > All of that and much more needs to be replicated for each microservice, right? in principle yes, and these are good guidelines for determining what is or is not suitable to be a microservice. if it would need to auth an arbitrary user, or have direct database access, or be exposed to public requests, it might not be a good candidate for a microservice. things that can stand alone and have limited functional dependencies are much better candidates. > Why not just have a module in your monolithic app that does it. because the monolithic app is Python/django and the PDF generation tool is Java. one of the main advantages of microservices architecture is much greater flexibility in technology selection. A previous solution used Python subprocesses to call out to PDF generation software. It's actually easier and cleaner for us to use a microservice instead.

shados 10y ago

Yeah, we do micro services, the "real" kind. Not the "SOA with a new name kind", but the "some services are literally a few douzan lines of code and we have 100x the amount of services as we do devs" kind. The thing is, you need a massive investment in infrastructure to make it happen. But once you do, its great. You can create and deploy a new service in a few seconds. You can rewrite any individual service to be latest and greatest in an afternoon. Different teams don't have to agree on coding standards (so you don't argue about it). But, the infrastructure cost is really high, a big chunk of what you save in development you pay in devops, and its harder to be "eventually consistant" (eg: an upgrade of your stack across the board can take 10x longer, because there's no big push that HAS to happen for a tiny piece to get the benefits). Monolithic apps have their advantages too, and many forget it: less devops cost, easier to refactor (especially in statically typed languages: a right click -> rename will propagate through the entire app) and while its harder to upgrade the stack, once its done, your entire stack is up to date, not just parts of it being all over. Code reuse is significantly easier, too.

hobs 10y ago
Yeah, add in things like MORE THAN ONE PRODUCTION ENVIRONMENT and LETTING YOUR CUSTOMER HOST AN INSTANCE OF YOUR MICROSERVICES and you have guaranteed your own suffering.
- neeleshs 10y ago
  It is a matter of tooling. One data center or ten, it does not matter much with proper tooling. We deploy to seven data centers with a click of a button, with rollback, staggered deployment etc. Centralized logging using ELK gives us great visibility in to each DC, without worrying about individual microservice instances.
  - bpicolo 10y ago
    Easy until you realize you need to somehow manage + configure hundreds of services to run your dev environment...

themartorana 10y ago

Yes. A lot of success. And with only one person on the backend full time. That said, in places where it doesn't make sense we didn't try to force it. Our main game API is somewhat monolithic, but behind it we have almost 10 other services. Here's a quick breakdown: - Turn based API service (largest, "monolithic") - Real-time API service (about 50% the size of turn-based) - config service (serves configuration settings to clients for game balancing) - ad waterfall service (dynamic waterfall, no actual ads) - push notification service - analytics collection service (mostly a fast collector that dumps into Big Query) - Open graph service (for rich sharing) - push maintenance service (executes token management based on GCM/APNS feedback) - help desk form service (simple front-end to help desk) - service update service (monitors CI for new binaries, updates services on the fly - made easy by Go binary deployment from CI to S3) - service ping service (monitors all service health, responds to ELB pings) - Facebook web front-end service (just serves WebGL version of our game binary for play on Facebook) - NATS.io for all IPC between services ...and a few more in the works. Some of these might push the line of "micro" in that they almost all do more than a single function's worth of work, but that level of granularity isn't practical. But don't get too caught up on the "micro" part. Split services where domain lines naturally form, and don't constrain service size by arbitrary definitions. You know, right tool for the job and whatnot.

wtbob 10y ago

> Also, the team spent too much time debating standards etc. Ah yes, the 'let's have decentralised microservices with centralised standards!' anti-pattern. It results in lots of full-fledged, heavyweight, slow-to-update services, which also have all the problems of a distributed system. It's the worst of both worlds.

clifanatic 10y ago

> It was mostly because the monitoring and alerting were now split into many different pieces Well, there's your problem - you need a monitoring microservice and an alerting microservice! Well, those may be too coarse by themselves, but once you break them down into 5 or 6 microservices each, you'll be ready for production.

huherto 10y ago

> Also, the team spent too much time debating standards etc. IMHO. You need a lead with a clear vision that drives the effort. Too many leads will create chaos.

mrhektor 10y ago
We did have a lead with a vision, and part of the vision was standards for each service (for example, file structure in Go). I can see the rationale behind it; a new dev can onboard very quickly on to a new service. But in hindsight, maybe it wasn't thought out enough.
deleted 10y ago
[deleted]

vonmoltke 10y ago

> Any positive experiences with micro-services here? I'm currently working on a large refactoring effort along these lines. The end goal is to create a modular, potentially distributed system that can be deployed in a variety of configurations, updated piecemeal for different customers, and integrated by our customers with the third-party or in-house code of their choice using defined APIs. We aren't typical of the other examples, though, in that we do literally ship our software to our customers and they run it on their own clusters.

mr_tristan 10y ago

Oh yes. We're splitting up a large monolith into a bunch of different services. Completely amazing, though there's a ton of tools (like Netflix's Hysterix, etc) that make it much, much easier to do. I wouldn't, however, just "do microservices" from day one on a young app. But usually that young app has no idea what the true business value is, i.e., you have no idea what down time of certain parts of your services really means to the business. That's the #1 pain point we're solving: having mission critical things up 100%, and then rapidly iterating on new, less stable feature designs in separate services. You should, however, keep an eye on how "splittable" everything is, i.e., does everything need to be in the same DB schema? Most languages have package concepts, which typically align (somehow) with "service" concepts. Do you know their dependencies? That sort of thing. Then, the later process of "refactor -> split out service" is pretty straightforward and easy to plan.

Mc_Big_G 10y ago

Heh, since the day I heard of "microservices" the only thing I could think was "have fun maintaining that".

Alex63 10y ago

I don't know if the community will consider this example to be microservices as currently defined, but many years ago I wrote client-server systems using a PC-based fat client application, and transactions running on a CICS server. I think this was pretty similar to what people currently think of as a microservices architecture, although we didn't have to worry about running/monitoring multiple servers (all the transactions/services ran on a single mainframe server), and the transaction monitor managed things like start-up and shutdown pretty simply. This approach worked really well for us, and we built several robust, scalable applications using this approach. To be clear, we numbered our users in hundreds, not thousands or millions. I can well understand how scaling this approach across many servers could be very challenging.

nickbauman 10y ago

You split off specific parts of your app into microservices because you want to scale those parts independently from the rest. It's not just a blind decomposition of a monolith for the sake of decomposition.