3 ms·

Single writer multiple readers, ideally with permissions to enforce, is a useful hybrid. Enforce going through the front door if you want side-effects. Reportin

Single writer multiple readers, ideally with permissions to enforce, is a useful hybrid. Enforce going through the front door if you want side-effects. Reporting and ad-hoc data enrichment can be painful to materialize otherwise. When you have multiple bits of code responsible for writing the same columns, maintaining global invariants becomes much harder. I can still see rationale for exceptions to the rule, e.g. status fields for rows which logically represent some ongoing process or job, where a UI controller may set something to "processing" and the background job sets it to "processed" when it's complete. But there are ways to close the loop here. For example, implement the background job by having the job processing system invoke the same system which owns the field and sets it in response to the UI, like this: UI start ---> Back End ---> status := processing Back End ---> add job to job queue Job Queue ---> Back End does work Back End ---> status := processed

jhoelzel 5y ago
Thank you for this as this is exactly how its implemented in a recent project of mine. I didn't want to put unneeded information in the first post but here i have some space ;) What I ended up doing was: - two sqs queues, one for processing one for failures after 3 tries - the go backend reads from its queue and the message contains a json object with all the information needed for pdf generation - the backend posts the file back to the "internal api" of the application, which saves it in s3 and also sets processing to true which means the user can go and download it. This results in: - my pdf generated the way i want it (sorry i dont actually make stardust) - the queues are only reachable through the assosiated iam roles - if a job failes it will be added to the failed queue, which sends me an email - if i recieve an email by the failed queue, 90% of the time i have found another edge case - since the message already contains everything that i need to build the pdf, my edge case has just defined itself - for the postback i supply an api token that is only valid for that one file All this means my client has total flexibility in deployment.
Pamar 5y ago
Caveat: I am really not qualified to discuss the nuances (because I have never used microservices so the little I know is based on reading a bit on those here and on other online forums). "Single writer multiple readers", yes, this is what I would probably use, but yet again, wasn't the "promise" of Microservices being able to work in total isolation? If I have one table (e.g. "Customer") which is written by one specific microservice and read by a dozen or more... what happens when I decide that I have to change the schema because the current representation of "Zip Code" is not adequate anymore because, I dunno, we started dealing with UK customers now? Lo and behold, I have to change code in 13 Microservices - the one actually writing to it, and the 12 more that only need to get the data to show or print or convert to JSON or whatever... ¯\_(ツ)_/¯
- jhoelzel 5y ago
  actually you would only have to change the one reader and the one writer. The services themselves do not differ, but are fanned out. E.G. so 100 workers watch your pdf queue to generate pdfs. If your zip code is really used by many services, the question is how you want to communicate those objects and there are really a lot of choices out there. GRPC / Soap / Shared model package / and the list goes one. On the other hand i have obsverved that people push full blown models around just to end up reading one field of it in their monolith. I believe the PHP frameworks of the past where the biggest offenders like that. When i was working with sugarcrm for instance, it was not uncommon to pass huuuuge objects between functions because you needed 1 field. It's PHP so by default, function arguments are passed by value, so you can already see where the fun begins.
- barrkel 5y ago
  Database schemas are generally isomorphic with RPC data structures - usually a subset one way or the other. If you had a service which handed out Zip Codes and the structure of Zip Codes changed (maybe you used numbers and now you realize you need to use strings, which I infer from your UK example) you'd have an isomorphic problem: updating 12 consumers and 1 producer for the new type in the RPC message. Having an RPC to migrate means you can put code on the read path to support backward compatibility while migrating. But you can do the same thing with the database - support both fields until all readers have ported to the new field - and since big database migrations are expensive, in practice large systems do migrations that way anyhow. Introduce the new field, start writing to it, back-fill it, port readers across individually (there may be many read locations even within a single service), drop the old field. If a system evolved from a monolith, one way to partition services is to provide views to limit set of visible columns. That's another point at which you can put shims for compatibility while porting schemas.
- HelloNurse 5y ago
  First you publish the next version of your service with the new data, then the disparate teams in charge of 12 clients (not you) update the rest of the application, then the old version is retired and every team has coordinated properly. Microservices allow lean coordination, basically just asking every team when they think they'll be ready without anyone messing with someone else's code.

jhoelzel 5y ago

Thank you for this as this is exactly how its implemented in a recent project of mine. I didn't want to put unneeded information in the first post but here i have some space ;) What I ended up doing was: - two sqs queues, one for processing one for failures after 3 tries - the go backend reads from its queue and the message contains a json object with all the information needed for pdf generation - the backend posts the file back to the "internal api" of the application, which saves it in s3 and also sets processing to true which means the user can go and download it. This results in: - my pdf generated the way i want it (sorry i dont actually make stardust) - the queues are only reachable through the assosiated iam roles - if a job failes it will be added to the failed queue, which sends me an email - if i recieve an email by the failed queue, 90% of the time i have found another edge case - since the message already contains everything that i need to build the pdf, my edge case has just defined itself - for the postback i supply an api token that is only valid for that one file All this means my client has total flexibility in deployment.

Pamar 5y ago

Caveat: I am really not qualified to discuss the nuances (because I have never used microservices so the little I know is based on reading a bit on those here and on other online forums). "Single writer multiple readers", yes, this is what I would probably use, but yet again, wasn't the "promise" of Microservices being able to work in total isolation? If I have one table (e.g. "Customer") which is written by one specific microservice and read by a dozen or more... what happens when I decide that I have to change the schema because the current representation of "Zip Code" is not adequate anymore because, I dunno, we started dealing with UK customers now? Lo and behold, I have to change code in 13 Microservices - the one actually writing to it, and the 12 more that only need to get the data to show or print or convert to JSON or whatever... ¯\_(ツ)_/¯

jhoelzel 5y ago
actually you would only have to change the one reader and the one writer. The services themselves do not differ, but are fanned out. E.G. so 100 workers watch your pdf queue to generate pdfs. If your zip code is really used by many services, the question is how you want to communicate those objects and there are really a lot of choices out there. GRPC / Soap / Shared model package / and the list goes one. On the other hand i have obsverved that people push full blown models around just to end up reading one field of it in their monolith. I believe the PHP frameworks of the past where the biggest offenders like that. When i was working with sugarcrm for instance, it was not uncommon to pass huuuuge objects between functions because you needed 1 field. It's PHP so by default, function arguments are passed by value, so you can already see where the fun begins.
barrkel 5y ago
Database schemas are generally isomorphic with RPC data structures - usually a subset one way or the other. If you had a service which handed out Zip Codes and the structure of Zip Codes changed (maybe you used numbers and now you realize you need to use strings, which I infer from your UK example) you'd have an isomorphic problem: updating 12 consumers and 1 producer for the new type in the RPC message. Having an RPC to migrate means you can put code on the read path to support backward compatibility while migrating. But you can do the same thing with the database - support both fields until all readers have ported to the new field - and since big database migrations are expensive, in practice large systems do migrations that way anyhow. Introduce the new field, start writing to it, back-fill it, port readers across individually (there may be many read locations even within a single service), drop the old field. If a system evolved from a monolith, one way to partition services is to provide views to limit set of visible columns. That's another point at which you can put shims for compatibility while porting schemas.
HelloNurse 5y ago
First you publish the next version of your service with the new data, then the disparate teams in charge of 12 clients (not you) update the rest of the application, then the old version is retired and every team has coordinated properly. Microservices allow lean coordination, basically just asking every team when they think they'll be ready without anyone messing with someone else's code.