8 ms·

> Some data belongs in Postgres, some in DynamoDB, some in JSON files. Now, how do we do reporting? One of the key concepts in microservice architecture is dat

> Some data belongs in Postgres, some in DynamoDB, some in JSON files. Now, how do we do reporting? One of the key concepts in microservice architecture is data sovereignity. It doesn't matter how/where the data is stored. The only thing that cares about the details of the data storage is the service itself. If you need some data the service operates on for reporting purposes, make an API that gets you this data and make it part of the service. You can architect layers around it, maybe write a separate service that aggregates data from multiple other services into a central analytics database and then reporting can be done from there or keep requests in real time, but introduce a caching layer or whatever. But you do not simply go and poke your reporting fingers into individual service databases. In a good microservice architecture you should not even be able to do that.

echelon 7y ago
> But you do not simply go and poke your reporting fingers into individual service databases. This is why I distrust all of the monolith folks. Yes, it's easier to get your data, but in the long run you create unmaintainable spaghetti that can't ever change without breaking things you can't easily surface. Monoliths are undisciplined and encourage unhealthy and unsustainable engineering. Microservices enforce separation of concerns and data ownership. It can be done wrong, but when executed correctly results in something you can easily make sense of.
- adamc 7y ago
  I do not claim expertise here, but it would seem like microservices would add significant performance costs. Stitching together a bunch of results from different microservices is going to be a LOT more expensive than running a query with joins.
  - ddek 7y ago
    That wouldn't be a microservice then. There's going to be a relationship between data in your services, but it shouldn't be directly referential.
  - echelon 7y ago
    Humans are the most expensive part of the system. You have to make it easy for humans to understand and change the system, and at the end of the day that's the number one thing to optimize for. This is why microservices are compelling. But to speak directly to your concern, you have to think about service boundaries and granularity correctly. Nobody is saying make a microservice out of every conceivable table. Think about the bigger picture, at a systems level. Wherever you can draw boxes you might have a service boundary. Why would you need to join payment data to session and login data? Do you need to compare employee roles and ACLs against product shipping data? These things belong in different systems. If you keep them in the same monolith, there's the danger that people will write code that intertwines the model in ways it shouldn't. Deploying and ownership become hard problems. The goal is to keep things that are highly functionally related together in a microservice and expose an API where the different microservices in your ecosystem are required to interact. (Eg, your employees will login.) When the data analytics folks want to do advanced reporting on the joins of these systems (typically offline behavior), you can expose a feed that exports your data. But don't expose an internal view of it to them or they'll find ways of turning you into a monolith.
    - mewpmewp2 7y ago
      In my experience it is a lot more difficult to navigate around all the different microservices to understand what needs to be done compared to being in a monolith where you can jump from file to file. Also then what also happens is microservices are created using different languages which in turn adds so much complexity to understand what is going on on the whole big picture level. And code gets repeated a lot more. If there is change in a microservices or update everyone will need to figure out what services depend on and how they will have to adapt. With monolith you can just use your IDE to see what will break if you make a change. So much repeated business logic. Creating a new feature involves having to have many meetings to figure out what services in which way have to be updated. It is crazy mess in my opinion. I have been with a company that had monolith application which they split up to more than 15 services (some python, some js, Scala, Java, etc...). Monolith still is used for some parts that are not migrated. I was working on single service having no idea how the whole system worked together. Then I had to do something in the old parts and I very quickly got an understanding how everything works together.
  - pas 7y ago
    If you already can't serve your requests from one DB, and you already want to factor out the analytics stuff, the long running background queries, modularize the spaghetti, scale the maintenance load, CI build + testing time, etc... That's what SOA and microservices is supposed to solve. At that scale you do reporting from a purpose-built service. Allegedly.
    - adamc 7y ago
      We do a lot of reporting that when. Then the users are unhappy that the data is slightly "stale". It serves some purposes, but not all purposes.
- smoe 7y ago
  > It can be done wrong, but when executed correctly [...] Quite the self-fulfilling prophecy there. > Yes, it's easier to get your data, but in the long run [...] Systems can and should be evolved and adapted over time. E.g. deploying components of the monolith as separate services. You can't easily predict what the requirements for your software going to be in say 10 years. And depending on the stage a company is, easy access to data for business decisions outweighs engineering idealism.
  - pg_is_a_butt 7y ago
    > easy access to data for business decisions outweighs engineering idealism. GET HIM!
  - echelon 7y ago
    > Quite the self-fulfilling prophecy there. Microservices require your organization to have an engineering culture. I would be afraid of introducing them at, say, Home Depot where (I've heard) your average programmer doesn't even write tests. If you have engineering talent within a small multiplicative factor of Google (say 0.5), then you can pull off Microservices at your org. Edit: I'm being downvoted, but I don't think it's a dangerous assumption or point to make that it takes a certain amount of discipline and experience to implement microservices correctly. When you have that technical capacity and the project calls for it, the benefit is tremendous.
    - redact207 7y ago
      I think you're being downvoted because you're implying monoliths don't require an engineering culture and that microservices are a silver bullet in getting systems built correctly. I've seen good and bad in each approach. It's certainly possible to enforce good SOCs and proper boundaries in monorepos, and also possible to plough a system into the ground with microservices. They're all just tools in your toolbox and both have a part to play in modern development.
      - deleted 7y ago
        [deleted]
- twh270 7y ago
  My client is doing a lot of it wrong. To be fair, they got sold a lot of really horrible and ridiculous advice from IBM consultants (is there another kind?), but they also have people in charge (organizationally and technically) who aren't great decision-makers. As the article says though, you can't fix a people problem (bad engineering practices and discipline) by going from one technology to another (monolith to microservices).
- NicoJuicy 7y ago
  DDD enforces seperation also. It's about code quality, microservices are easy replaceable. Modules are too. With both systems, the core part ( eg. mesh, Infrastructure, ... ) Is crucial. I think experienced developers can see this, the ones that actually delivered products and had big code changes. The ones that handled their "legacy" code. Microservices are just a way to enforce it, there are others. None are perfect or bad, both have their use-case.
  - deleted 7y ago
    [deleted]
- StavrosK 7y ago
  You're saying "monoliths encourage unhealthy engineering" and then in the next sentence say "when executed correctly" for microservices. That sounds like a having/eating cake type situation.
  - rriepe 7y ago
    To be fair, this is how I've seen tech decisions presented at most big tech companies.
    - geggam 7y ago
      Quit talking about what is behind the curtain
  - jayd16 7y ago
    Not exactly. It's hard to tell from the outside if a monolith was architecture well or is about the fall over. In a microservice architecture it's harder to pretend you're doing it right.
    - StavrosK 7y ago
      I disagree, "doing it wrong" just looks different there.
      - throwaway894345 7y ago
        The argument (which I sort of buy) is that microservices provide rails that keep people from doing certain stupid things like N clients depending on the data schema (making the schema a de-facto public interface). The trick with microservices is that the ecosystem is maturing and there are still lots of ways to screw up other things that are harder to screw up with monoliths. In time 95% of those will go away (my specific prediction is that one day we will write programs that express concurrency and the compiler/toolchain will work out distributing the program across Cloud services--although "Cloud" will be an antiquated term by then--including stitching together the relevant logs, etc and possibly even a coherent distributed debugger experience).
- wtetzner 7y ago
  > you create unmaintainable spaghetti that can't ever change without breaking things you can't easily surface. How does creating a tangle of microservices (effectively distributed objects) really solve the problem?
  - NicoJuicy 7y ago
    Constructed good, it's Ravioli and not spaghetti.
    - wtetzner 7y ago
      But that's true of a well-constructed monolith, too. And it has far fewer failure modes and less complexity in general.
  - MadWombat 7y ago
    Microservices provide an abstraction. That is kind of the point. If you feel like the data yours service operates on would be better off stored in a redis database instead of an RDBMS, you can rewrite your persistence layer, test and roll out the new version of the service. As long as your APIs do not change, nobody cares how you produce responses to requests. In a monolith, this would be a nightmare. You don't have a single persistence layer to change, you have to go through every module, find all the places where this specific table or tables are being accessed and change retrieval and storage functionality everywhere.
    - geggam 7y ago
      Seems to me you are talking about a database access layer instead of microservices. My understanding of microservices is a bunch of loosely connected services that can be changed with minimal impact to the others Problem with the ideal is in reality this never works as complexity grows the spaghetti code moves to spaghetti infrastructure ( Done a network map of a large k8s / istio deployment lately ? )
      - hopia 7y ago
- lucidone 7y ago
  > enforce separation of concerns and data ownership You can enforce separation of concerns and data ownership in a monolith just as much as you can not enforce these two characteristics in a micro service architecture. Microservices and monoliths are a discussion about deployment artifacts, full stop.
- deleted 7y ago
  [deleted]
- pjmlp 7y ago
  Only when done by folks that never learned how to write modular code and package libraries. The same folks aren't going to magically learn how to do distributed computing properly, rather they will implement unmaintainable spaghetti network calls with all the distributed computing issues on top.
  - wtetzner 7y ago
    And untangling a monolith tends to be much less problematic that untangling a bunch of microservices. For one thing, you can refactor/untangle it all offline, do your testing, and do a single release with the updated version, as opposed to trying to coordinate releases of a bunch of services whose interfaces/boundaries were poorly defined.
- no_wizard 7y ago
  In practice, splitting your code into consumable libraries/modules works equally as well. Then your monolith is just all the modules glued together.
  - flukus 7y ago
    Splitting code into libraries works better, it's simpler and faster. The only thing micro services bring to the table is being to deploy updates independently (although this is also possible with libraries). If you don't need to deploy independently then micro services are useless complexity, if you can't deploy independently then you've got a distributed monolith.
- jffhn 7y ago
  >Microservices enforce separation of concerns Depending on where you work, it can be a problem, because the separation is not always appropriate, and can for political reasons be much harder to revert when visible at the service level (for example because the architect doesn't understand consistency, or because your manager tells you that the distributed architecture documentation has been sent to the client so it cannot be modified). In case of undue separation, reworking the internals of the enclosing monolith should have less chance to cause frictions.
BrentOzar 7y ago
> But you do not simply go and poke your reporting fingers into individual service databases. In a good microservice architecture you should not even be able to do that. I agree. In a monolith architecture, though, you CAN do that (and many shops do.) That's where their pains come from when they migrate from monolith to microservice: development is easier, but reports are way, way harder.
- allover 7y ago
  > when they migrate from monolith to microservice: development is easier [...] Not even that -- that idea is still highly debatable. I would argue that it absolutely isn't easier, and the stepping-back-in-time of developer experience is one of the biggest problems with microservices. Microservices in general, are way, way harder.
sequoia 7y ago
> you do not simply go and poke your reporting fingers into individual service databases Side point: This is a needlessly hostile and unprofessional way to refer to a colleague. Remember that you and the reporting/analytics people at your company are working towards the same goals (the company's business goals). You are collaborators, not combatants. You can express your same point by saying something like "The habit of directly accessing database resources and building out reporting code on this is likely to lead to some very serious problems when the schemas change. This is tantamount to relying upon a private API." etc. We can all achieve much more when we endeavor to treat one another with respect and assume good intentions.
- cheese4242 7y ago
  This is an incredible overreaction to an entirely innocuous comment.
  - frockington1 7y ago
    I've noticed reporting/analytics people going extinct around my workplace as micro services make monitoring easier. There might be some pent up hostility towards the technology side
  - sequoia 7y ago
    If you think telling colleagues not to "simply go and poke your reporting fingers into" things won't insult them or put them on a defensive footing, I encourage you to try it and closely note the reception you receive. In my experience, people do not appreciate being spoken to like that.
    - p1necone 7y ago
      They didn't tell their colleagues to do that, they made a slightly humorous comment on a hacker news thread.
      - sequoia 7y ago
Swinx43 7y ago
I came here simply to echo this statement! Design a reporting solution that is responsible for ingesting data from these micro services' persistence layers. Analysts should only ever be querying this reporting solution and should not be allowed to connect directly to any micro service persistence layer or API. We have a whole industry around Analytics and Data and the tools and processes to build this reporting layer is well established and proven. Nothing will give you as many nightmares as letting your analysts loose on your micro service persistence layers!
- cogman10 7y ago
  This is seriously why my company still has monoliths. Our databases are open to way too many people. What's worse, they are multi tenant making refactoring really hard.
  - oneplane 7y ago
    Having more than one schema owner is practically a death sentence for development and engineering... We used to have a few of those, especially on exadata clusters. Finally carted them out of the local dc after moving to RDS Aurora databases with strict policies. Might have caused 3 or 4 people to quit, but totally worth it for the 500+ people that stayed who now can own their data, schema and development (and be held responsible for it! -- another issue of multi-db-access, it's always someone else's fault). Went from deploying once a day with a 'heads up' message to no-message deploying multiple times per hour.
  - ozim 7y ago
    Why monoliths? Everyone still wants to to have OLAP and OLTP systems where analytics are done on OLAP. Where having this separation you can get data from multiple sources to put into your analytics. I cannot imagine people not doing that and having need to have stats in real time. For most shopping/banking stuff you can get away with once in 24 hours dumps and then analytics can be done on that.
mrits 7y ago
What you are describing certainly isn't unique to microservices.
jonahbenton 7y ago
Sorry, but "making an API that gets you this data" is the wrong answer. Most APIs are glorified wrappers around individual record-level operations like- get me this user- or constrained searches that return a portion of the data, maybe paginated. Reporting needs to see all the data. This is a completely different query and service delivery pattern. What happens to your API service written in a memory managed/garbage-collected language when you ask it to pull all the data from its bespoke database, pass it through its memory space, then send it back down the caller? It goes into GC hell, is what. What happens when your API service when it issues queries for a consistent view of all the data and winds up forcing the database to lock tables? It stops working for users, is what. There are so many ways to fail when your microservice starts pretending it is a database. It is not. Databases are dedicated services, not libraries, for a reason. It is also true that analysts should not be given access to service databases, because the schema and semantics are likely to change out from under them. The least bad solution? The engineering team is responsible for delivering either semantic events or doing the batch transformation themselves into a model that the data team can consume. It's a data delivery format, not an API.
- abhishekjha 7y ago
  >It goes into GC hell Can you exapand on this a little? Or a paper that I can read?
  - solarengineer 7y ago
    The parent comment said “is asked for all records..GC hell “. Since a micro service deals with only its own data and reporting is then across services, we’d need to query across services to get data and make sense of it. If we’d ever need to query all records, then such records would become domain objects in the micro services first before being passed along. A large number of domain objects would require a large amount of memory. Processing and releasing domain objects will result in GC on the released objects.
  - simonh 7y ago
    The service will need to read all its data and put it into objects, then extract the data from the objects to report it, then garbage collect all of that. For every single record in its entire data set. You could say but oh, why not just return the underlying data without making objects? Well now you are exposing the underlying data format, which is what we’re trying to avoid by giving this job to the service.
simonw 7y ago
A co-worker had a smart solution for this: your service's representation in a reporting system (a data warehouse for example) is part of its API. Your team should document it, and should ensure that when that representation changes information about the changes is available to the people who need to know it. This really makes sense to me. I love the idea that part of a microservice team's responsibility is ensuring that a sensible subset of the data is copied over to the reporting systems in such a way that it can be used for analysis without risk of other teams writing queries that depend on undocumented internal details.
- edoloughlin 7y ago
  your service's representation in a reporting system At what point in time?
  - dabeeeenster 7y ago
    At the beginning: https://docs.pact.io/ https://docs.pact.io/

echelon 7y ago

> But you do not simply go and poke your reporting fingers into individual service databases. This is why I distrust all of the monolith folks. Yes, it's easier to get your data, but in the long run you create unmaintainable spaghetti that can't ever change without breaking things you can't easily surface. Monoliths are undisciplined and encourage unhealthy and unsustainable engineering. Microservices enforce separation of concerns and data ownership. It can be done wrong, but when executed correctly results in something you can easily make sense of.

adamc 7y ago
I do not claim expertise here, but it would seem like microservices would add significant performance costs. Stitching together a bunch of results from different microservices is going to be a LOT more expensive than running a query with joins.
- ddek 7y ago
  That wouldn't be a microservice then. There's going to be a relationship between data in your services, but it shouldn't be directly referential.
- echelon 7y ago
  Humans are the most expensive part of the system. You have to make it easy for humans to understand and change the system, and at the end of the day that's the number one thing to optimize for. This is why microservices are compelling. But to speak directly to your concern, you have to think about service boundaries and granularity correctly. Nobody is saying make a microservice out of every conceivable table. Think about the bigger picture, at a systems level. Wherever you can draw boxes you might have a service boundary. Why would you need to join payment data to session and login data? Do you need to compare employee roles and ACLs against product shipping data? These things belong in different systems. If you keep them in the same monolith, there's the danger that people will write code that intertwines the model in ways it shouldn't. Deploying and ownership become hard problems. The goal is to keep things that are highly functionally related together in a microservice and expose an API where the different microservices in your ecosystem are required to interact. (Eg, your employees will login.) When the data analytics folks want to do advanced reporting on the joins of these systems (typically offline behavior), you can expose a feed that exports your data. But don't expose an internal view of it to them or they'll find ways of turning you into a monolith.
  - mewpmewp2 7y ago
    In my experience it is a lot more difficult to navigate around all the different microservices to understand what needs to be done compared to being in a monolith where you can jump from file to file. Also then what also happens is microservices are created using different languages which in turn adds so much complexity to understand what is going on on the whole big picture level. And code gets repeated a lot more. If there is change in a microservices or update everyone will need to figure out what services depend on and how they will have to adapt. With monolith you can just use your IDE to see what will break if you make a change. So much repeated business logic. Creating a new feature involves having to have many meetings to figure out what services in which way have to be updated. It is crazy mess in my opinion. I have been with a company that had monolith application which they split up to more than 15 services (some python, some js, Scala, Java, etc...). Monolith still is used for some parts that are not migrated. I was working on single service having no idea how the whole system worked together. Then I had to do something in the old parts and I very quickly got an understanding how everything works together.
- pas 7y ago
  If you already can't serve your requests from one DB, and you already want to factor out the analytics stuff, the long running background queries, modularize the spaghetti, scale the maintenance load, CI build + testing time, etc... That's what SOA and microservices is supposed to solve. At that scale you do reporting from a purpose-built service. Allegedly.
  - adamc 7y ago
    We do a lot of reporting that when. Then the users are unhappy that the data is slightly "stale". It serves some purposes, but not all purposes.
smoe 7y ago
> It can be done wrong, but when executed correctly [...] Quite the self-fulfilling prophecy there. > Yes, it's easier to get your data, but in the long run [...] Systems can and should be evolved and adapted over time. E.g. deploying components of the monolith as separate services. You can't easily predict what the requirements for your software going to be in say 10 years. And depending on the stage a company is, easy access to data for business decisions outweighs engineering idealism.
- pg_is_a_butt 7y ago
  > easy access to data for business decisions outweighs engineering idealism. GET HIM!
- echelon 7y ago
  > Quite the self-fulfilling prophecy there. Microservices require your organization to have an engineering culture. I would be afraid of introducing them at, say, Home Depot where (I've heard) your average programmer doesn't even write tests. If you have engineering talent within a small multiplicative factor of Google (say 0.5), then you can pull off Microservices at your org. Edit: I'm being downvoted, but I don't think it's a dangerous assumption or point to make that it takes a certain amount of discipline and experience to implement microservices correctly. When you have that technical capacity and the project calls for it, the benefit is tremendous.
  - redact207 7y ago
    I think you're being downvoted because you're implying monoliths don't require an engineering culture and that microservices are a silver bullet in getting systems built correctly. I've seen good and bad in each approach. It's certainly possible to enforce good SOCs and proper boundaries in monorepos, and also possible to plough a system into the ground with microservices. They're all just tools in your toolbox and both have a part to play in modern development.
    - deleted 7y ago
      [deleted]
twh270 7y ago
My client is doing a lot of it wrong. To be fair, they got sold a lot of really horrible and ridiculous advice from IBM consultants (is there another kind?), but they also have people in charge (organizationally and technically) who aren't great decision-makers. As the article says though, you can't fix a people problem (bad engineering practices and discipline) by going from one technology to another (monolith to microservices).
NicoJuicy 7y ago
DDD enforces seperation also. It's about code quality, microservices are easy replaceable. Modules are too. With both systems, the core part ( eg. mesh, Infrastructure, ... ) Is crucial. I think experienced developers can see this, the ones that actually delivered products and had big code changes. The ones that handled their "legacy" code. Microservices are just a way to enforce it, there are others. None are perfect or bad, both have their use-case.
- deleted 7y ago
  [deleted]
StavrosK 7y ago
You're saying "monoliths encourage unhealthy engineering" and then in the next sentence say "when executed correctly" for microservices. That sounds like a having/eating cake type situation.
- rriepe 7y ago
  To be fair, this is how I've seen tech decisions presented at most big tech companies.
  - geggam 7y ago
    Quit talking about what is behind the curtain
- jayd16 7y ago
  Not exactly. It's hard to tell from the outside if a monolith was architecture well or is about the fall over. In a microservice architecture it's harder to pretend you're doing it right.
  - StavrosK 7y ago
    I disagree, "doing it wrong" just looks different there.
    - throwaway894345 7y ago
      The argument (which I sort of buy) is that microservices provide rails that keep people from doing certain stupid things like N clients depending on the data schema (making the schema a de-facto public interface). The trick with microservices is that the ecosystem is maturing and there are still lots of ways to screw up other things that are harder to screw up with monoliths. In time 95% of those will go away (my specific prediction is that one day we will write programs that express concurrency and the compiler/toolchain will work out distributing the program across Cloud services--although "Cloud" will be an antiquated term by then--including stitching together the relevant logs, etc and possibly even a coherent distributed debugger experience).
wtetzner 7y ago
> you create unmaintainable spaghetti that can't ever change without breaking things you can't easily surface. How does creating a tangle of microservices (effectively distributed objects) really solve the problem?
- NicoJuicy 7y ago
  Constructed good, it's Ravioli and not spaghetti.
  - wtetzner 7y ago
    But that's true of a well-constructed monolith, too. And it has far fewer failure modes and less complexity in general.
- MadWombat 7y ago
  Microservices provide an abstraction. That is kind of the point. If you feel like the data yours service operates on would be better off stored in a redis database instead of an RDBMS, you can rewrite your persistence layer, test and roll out the new version of the service. As long as your APIs do not change, nobody cares how you produce responses to requests. In a monolith, this would be a nightmare. You don't have a single persistence layer to change, you have to go through every module, find all the places where this specific table or tables are being accessed and change retrieval and storage functionality everywhere.
  - geggam 7y ago
    Seems to me you are talking about a database access layer instead of microservices. My understanding of microservices is a bunch of loosely connected services that can be changed with minimal impact to the others Problem with the ideal is in reality this never works as complexity grows the spaghetti code moves to spaghetti infrastructure ( Done a network map of a large k8s / istio deployment lately ? )
    - hopia 7y ago
lucidone 7y ago
> enforce separation of concerns and data ownership You can enforce separation of concerns and data ownership in a monolith just as much as you can not enforce these two characteristics in a micro service architecture. Microservices and monoliths are a discussion about deployment artifacts, full stop.
deleted 7y ago
[deleted]
pjmlp 7y ago
Only when done by folks that never learned how to write modular code and package libraries. The same folks aren't going to magically learn how to do distributed computing properly, rather they will implement unmaintainable spaghetti network calls with all the distributed computing issues on top.
- wtetzner 7y ago
  And untangling a monolith tends to be much less problematic that untangling a bunch of microservices. For one thing, you can refactor/untangle it all offline, do your testing, and do a single release with the updated version, as opposed to trying to coordinate releases of a bunch of services whose interfaces/boundaries were poorly defined.
no_wizard 7y ago
In practice, splitting your code into consumable libraries/modules works equally as well. Then your monolith is just all the modules glued together.
- flukus 7y ago
  Splitting code into libraries works better, it's simpler and faster. The only thing micro services bring to the table is being to deploy updates independently (although this is also possible with libraries). If you don't need to deploy independently then micro services are useless complexity, if you can't deploy independently then you've got a distributed monolith.
jffhn 7y ago
>Microservices enforce separation of concerns Depending on where you work, it can be a problem, because the separation is not always appropriate, and can for political reasons be much harder to revert when visible at the service level (for example because the architect doesn't understand consistency, or because your manager tells you that the distributed architecture documentation has been sent to the client so it cannot be modified). In case of undue separation, reworking the internals of the enclosing monolith should have less chance to cause frictions.

BrentOzar 7y ago

> But you do not simply go and poke your reporting fingers into individual service databases. In a good microservice architecture you should not even be able to do that. I agree. In a monolith architecture, though, you CAN do that (and many shops do.) That's where their pains come from when they migrate from monolith to microservice: development is easier, but reports are way, way harder.

allover 7y ago
> when they migrate from monolith to microservice: development is easier [...] Not even that -- that idea is still highly debatable. I would argue that it absolutely isn't easier, and the stepping-back-in-time of developer experience is one of the biggest problems with microservices. Microservices in general, are way, way harder.

sequoia 7y ago

> you do not simply go and poke your reporting fingers into individual service databases Side point: This is a needlessly hostile and unprofessional way to refer to a colleague. Remember that you and the reporting/analytics people at your company are working towards the same goals (the company's business goals). You are collaborators, not combatants. You can express your same point by saying something like "The habit of directly accessing database resources and building out reporting code on this is likely to lead to some very serious problems when the schemas change. This is tantamount to relying upon a private API." etc. We can all achieve much more when we endeavor to treat one another with respect and assume good intentions.

cheese4242 7y ago
This is an incredible overreaction to an entirely innocuous comment.
- frockington1 7y ago
  I've noticed reporting/analytics people going extinct around my workplace as micro services make monitoring easier. There might be some pent up hostility towards the technology side
- sequoia 7y ago
  If you think telling colleagues not to "simply go and poke your reporting fingers into" things won't insult them or put them on a defensive footing, I encourage you to try it and closely note the reception you receive. In my experience, people do not appreciate being spoken to like that.
  - p1necone 7y ago
    They didn't tell their colleagues to do that, they made a slightly humorous comment on a hacker news thread.
    - sequoia 7y ago

Swinx43 7y ago

I came here simply to echo this statement! Design a reporting solution that is responsible for ingesting data from these micro services' persistence layers. Analysts should only ever be querying this reporting solution and should not be allowed to connect directly to any micro service persistence layer or API. We have a whole industry around Analytics and Data and the tools and processes to build this reporting layer is well established and proven. Nothing will give you as many nightmares as letting your analysts loose on your micro service persistence layers!

cogman10 7y ago
This is seriously why my company still has monoliths. Our databases are open to way too many people. What's worse, they are multi tenant making refactoring really hard.
- oneplane 7y ago
  Having more than one schema owner is practically a death sentence for development and engineering... We used to have a few of those, especially on exadata clusters. Finally carted them out of the local dc after moving to RDS Aurora databases with strict policies. Might have caused 3 or 4 people to quit, but totally worth it for the 500+ people that stayed who now can own their data, schema and development (and be held responsible for it! -- another issue of multi-db-access, it's always someone else's fault). Went from deploying once a day with a 'heads up' message to no-message deploying multiple times per hour.
- ozim 7y ago
  Why monoliths? Everyone still wants to to have OLAP and OLTP systems where analytics are done on OLAP. Where having this separation you can get data from multiple sources to put into your analytics. I cannot imagine people not doing that and having need to have stats in real time. For most shopping/banking stuff you can get away with once in 24 hours dumps and then analytics can be done on that.

mrits 7y ago

What you are describing certainly isn't unique to microservices.

jonahbenton 7y ago

Sorry, but "making an API that gets you this data" is the wrong answer. Most APIs are glorified wrappers around individual record-level operations like- get me this user- or constrained searches that return a portion of the data, maybe paginated. Reporting needs to see all the data. This is a completely different query and service delivery pattern. What happens to your API service written in a memory managed/garbage-collected language when you ask it to pull all the data from its bespoke database, pass it through its memory space, then send it back down the caller? It goes into GC hell, is what. What happens when your API service when it issues queries for a consistent view of all the data and winds up forcing the database to lock tables? It stops working for users, is what. There are so many ways to fail when your microservice starts pretending it is a database. It is not. Databases are dedicated services, not libraries, for a reason. It is also true that analysts should not be given access to service databases, because the schema and semantics are likely to change out from under them. The least bad solution? The engineering team is responsible for delivering either semantic events or doing the batch transformation themselves into a model that the data team can consume. It's a data delivery format, not an API.

abhishekjha 7y ago
>It goes into GC hell Can you exapand on this a little? Or a paper that I can read?
- solarengineer 7y ago
  The parent comment said “is asked for all records..GC hell “. Since a micro service deals with only its own data and reporting is then across services, we’d need to query across services to get data and make sense of it. If we’d ever need to query all records, then such records would become domain objects in the micro services first before being passed along. A large number of domain objects would require a large amount of memory. Processing and releasing domain objects will result in GC on the released objects.
- simonh 7y ago
  The service will need to read all its data and put it into objects, then extract the data from the objects to report it, then garbage collect all of that. For every single record in its entire data set. You could say but oh, why not just return the underlying data without making objects? Well now you are exposing the underlying data format, which is what we’re trying to avoid by giving this job to the service.

simonw 7y ago

A co-worker had a smart solution for this: your service's representation in a reporting system (a data warehouse for example) is part of its API. Your team should document it, and should ensure that when that representation changes information about the changes is available to the people who need to know it. This really makes sense to me. I love the idea that part of a microservice team's responsibility is ensuring that a sensible subset of the data is copied over to the reporting systems in such a way that it can be used for analysis without risk of other teams writing queries that depend on undocumented internal details.

edoloughlin 7y ago
your service's representation in a reporting system At what point in time?
- dabeeeenster 7y ago
  At the beginning: https://docs.pact.io/ https://docs.pact.io/