3 ms·

Maybe it is just me, but I always understood that properly designed microservices have their own specific datastore, which is not shared with other microservice

by Pamar 5y ago

Maybe it is just me, but I always understood that properly designed microservices have their own specific datastore, which is not shared with other microservices even if these all collaborate to the same process. If this is actually (still) true, that means that "the way you organize your code" is a bit simplistic. Your example of an "http api that converts pdfs to ..." is surely a valid example of a microservice, but most business products have to handle much more "state" than those, and this will create further complications which go far beyond "how to organize your code" (and make monoliths more appealing).

pibi 5y ago
Well, it's all about the data responsibility: who is the owner of the data, how others can access the data. Once you have defined these, you see that you can "share the access" with other microservices (for example read only mode on a view), as long as the ownership and the access rules are preserved.
- Pamar 5y ago
  Yes, a view would be exactly how I would address the problem I described in my other answer above ("What happens if I need to change zip code representation in a data source that is read by multiple microservices?"). But this also means that we are now back into "YesSQL" territory, and specifically that we have to use a RDBMS which allows us to create Views. Goodbye NoSQL, goodbye Key+Valus datastore. (Or maybe you will just create an extra "newZipCode" and mantain it in parallel with "ZipCode" allowing every other consumer to adapt at their leisure...?). So it is another step back to "more traditional ways" to design a system... or a recipe for a disaster as soon as you start dealing with significant quantities of "state".
VBprogrammer 5y ago
IMHO Microservices done well should actually cut a whole vertical through your applications feature space. So not only should it be responsible completely for its own storage of data but it should be responsible for how that data is shown on the front end (or as close to that as you can realistically achieve). A microservice should ideally be reviews or left navigation not customer authentication or order processing.
jhoelzel 5y ago
Well I dont believe that i need a seperate datastore, but yes i need to communicate my state much more. The PDF Example for instance: I have to provide all assets that will be converted to startdust or at least links for them I have to define where the finished stardust would be send to (e.g http post) but its function is nonetheless independent. the epepheral disk on your container is a datastore too. If you need what is stored actually for longer than the request.... that is another story.
preseinger 5y ago
Yes, services should definitely encapsulate their state layer.
barrkel 5y ago
Single writer multiple readers, ideally with permissions to enforce, is a useful hybrid. Enforce going through the front door if you want side-effects. Reporting and ad-hoc data enrichment can be painful to materialize otherwise. When you have multiple bits of code responsible for writing the same columns, maintaining global invariants becomes much harder. I can still see rationale for exceptions to the rule, e.g. status fields for rows which logically represent some ongoing process or job, where a UI controller may set something to "processing" and the background job sets it to "processed" when it's complete. But there are ways to close the loop here. For example, implement the background job by having the job processing system invoke the same system which owns the field and sets it in response to the UI, like this: UI start ---> Back End ---> status := processing Back End ---> add job to job queue Job Queue ---> Back End does work Back End ---> status := processed
- jhoelzel 5y ago
  Thank you for this as this is exactly how its implemented in a recent project of mine. I didn't want to put unneeded information in the first post but here i have some space ;) What I ended up doing was: - two sqs queues, one for processing one for failures after 3 tries - the go backend reads from its queue and the message contains a json object with all the information needed for pdf generation - the backend posts the file back to the "internal api" of the application, which saves it in s3 and also sets processing to true which means the user can go and download it. This results in: - my pdf generated the way i want it (sorry i dont actually make stardust) - the queues are only reachable through the assosiated iam roles - if a job failes it will be added to the failed queue, which sends me an email - if i recieve an email by the failed queue, 90% of the time i have found another edge case - since the message already contains everything that i need to build the pdf, my edge case has just defined itself - for the postback i supply an api token that is only valid for that one file All this means my client has total flexibility in deployment.
- Pamar 5y ago
  Caveat: I am really not qualified to discuss the nuances (because I have never used microservices so the little I know is based on reading a bit on those here and on other online forums). "Single writer multiple readers", yes, this is what I would probably use, but yet again, wasn't the "promise" of Microservices being able to work in total isolation? If I have one table (e.g. "Customer") which is written by one specific microservice and read by a dozen or more... what happens when I decide that I have to change the schema because the current representation of "Zip Code" is not adequate anymore because, I dunno, we started dealing with UK customers now? Lo and behold, I have to change code in 13 Microservices - the one actually writing to it, and the 12 more that only need to get the data to show or print or convert to JSON or whatever... ¯\_(ツ)_/¯

by Pamar 5y ago

pibi 5y ago

Well, it's all about the data responsibility: who is the owner of the data, how others can access the data. Once you have defined these, you see that you can "share the access" with other microservices (for example read only mode on a view), as long as the ownership and the access rules are preserved.

Pamar 5y ago
Yes, a view would be exactly how I would address the problem I described in my other answer above ("What happens if I need to change zip code representation in a data source that is read by multiple microservices?"). But this also means that we are now back into "YesSQL" territory, and specifically that we have to use a RDBMS which allows us to create Views. Goodbye NoSQL, goodbye Key+Valus datastore. (Or maybe you will just create an extra "newZipCode" and mantain it in parallel with "ZipCode" allowing every other consumer to adapt at their leisure...?). So it is another step back to "more traditional ways" to design a system... or a recipe for a disaster as soon as you start dealing with significant quantities of "state".

VBprogrammer 5y ago

IMHO Microservices done well should actually cut a whole vertical through your applications feature space. So not only should it be responsible completely for its own storage of data but it should be responsible for how that data is shown on the front end (or as close to that as you can realistically achieve). A microservice should ideally be reviews or left navigation not customer authentication or order processing.

jhoelzel 5y ago

Well I dont believe that i need a seperate datastore, but yes i need to communicate my state much more. The PDF Example for instance: I have to provide all assets that will be converted to startdust or at least links for them I have to define where the finished stardust would be send to (e.g http post) but its function is nonetheless independent. the epepheral disk on your container is a datastore too. If you need what is stored actually for longer than the request.... that is another story.

preseinger 5y ago

Yes, services should definitely encapsulate their state layer.

barrkel 5y ago

Single writer multiple readers, ideally with permissions to enforce, is a useful hybrid. Enforce going through the front door if you want side-effects. Reporting and ad-hoc data enrichment can be painful to materialize otherwise. When you have multiple bits of code responsible for writing the same columns, maintaining global invariants becomes much harder. I can still see rationale for exceptions to the rule, e.g. status fields for rows which logically represent some ongoing process or job, where a UI controller may set something to "processing" and the background job sets it to "processed" when it's complete. But there are ways to close the loop here. For example, implement the background job by having the job processing system invoke the same system which owns the field and sets it in response to the UI, like this: UI start ---> Back End ---> status := processing Back End ---> add job to job queue Job Queue ---> Back End does work Back End ---> status := processed

jhoelzel 5y ago
Thank you for this as this is exactly how its implemented in a recent project of mine. I didn't want to put unneeded information in the first post but here i have some space ;) What I ended up doing was: - two sqs queues, one for processing one for failures after 3 tries - the go backend reads from its queue and the message contains a json object with all the information needed for pdf generation - the backend posts the file back to the "internal api" of the application, which saves it in s3 and also sets processing to true which means the user can go and download it. This results in: - my pdf generated the way i want it (sorry i dont actually make stardust) - the queues are only reachable through the assosiated iam roles - if a job failes it will be added to the failed queue, which sends me an email - if i recieve an email by the failed queue, 90% of the time i have found another edge case - since the message already contains everything that i need to build the pdf, my edge case has just defined itself - for the postback i supply an api token that is only valid for that one file All this means my client has total flexibility in deployment.
Pamar 5y ago
Caveat: I am really not qualified to discuss the nuances (because I have never used microservices so the little I know is based on reading a bit on those here and on other online forums). "Single writer multiple readers", yes, this is what I would probably use, but yet again, wasn't the "promise" of Microservices being able to work in total isolation? If I have one table (e.g. "Customer") which is written by one specific microservice and read by a dozen or more... what happens when I decide that I have to change the schema because the current representation of "Zip Code" is not adequate anymore because, I dunno, we started dealing with UK customers now? Lo and behold, I have to change code in 13 Microservices - the one actually writing to it, and the 12 more that only need to get the data to show or print or convert to JSON or whatever... ¯\_(ツ)_/¯