3 ms·
There is no one-size-fits-all approach to atomicity in microservices. Microservices typically want to store the most minimal amount of state possible, and push
by jrockway 7y ago
There is no one-size-fits-all approach to atomicity in microservices. Microservices typically want to store the most minimal amount of state possible, and push the coordination up a level; i.e., the calling service will send all the relevant information to the service that is going to handle this tiny piece of the work. Eventually there is going to be one service that owns some sort of workflow, and pushes the rest of the world in the direction of its desired outcome.
A year or so ago I needed to write a service to rotate accounts on network equipment every day or so. The control channel and the network hardware tended to be flaky, so I designed a state machine to make as much progress as possible, and be able to pick up where it left off. Each newly-created account was a row in a transactional database, and the successful completion of the operation was noted by changing a column from NULL to the current time. The flow was; if a new account is needed, generate the credential (generated_at). Find the old credentials and log into the device. Add the new account (applied_at). Try logging in with the new account (verified_at). Wait until account expiration (2 days later). Delete the account (deleted_at). Verify that we can't log in with the old account (delete_verified_at).
From this data, we could see what state every account was in, and query for a list of operations that needed to be performed. (Or if nothing needed to be done, how long to sleep for so that we would re-run the loop at the exact instant when work needed to be done, not that it was critical.)
I believe that your account creation and account deletion should follow a similar workflow. Accounts that fail to be created after retrying for X length of time should just move into the deletion workflow.
The user creation service and the day-to-day user operation service should probably be the same thing. The "user" row in your database should be what the state machine uses to figure out what operations need to be performed.
A user record would probably look something like:
username, password, email, ... = <stuff your user gives you>
created_at = <now>
stripe_account_id = <what the stripe account creation rpc returns>
stripe_account_verified_at = <time when you verified that the stripe account works>
auth0_account_id = ...
auth0_account_verified_at = ...
delete_requested_at = <when the user clicked "delete account" or you decided you don't want to retry the above steps any more>
stripe_account_delete_verified_at = <time that you ensured the account is gone>
auth0_account_delete_verified_at = <ditto>
Now you can do queries to figure out what work needs to be done. "select username from users where delete_requested_at is null and stripe_account_verified_at is null" -> create stripe accounts for those users. "select username from users where delete_requested_at is not null and auth0_account_delete_verified_at is null" -> delete auth0 accounts for those users.
The last bit of complexity is to prevent two copies of your user service from running the query at the exact instant and deciding to create a stripe account twice. I would personally just run one replica of that job. This makes deployments slightly more irritating, since no user creation can occur during a rollout (when the number of replicas is 0), but it sure is simple. I worked on a user-facing feature at Google that did that, and although I hated it, it worked fine. It is also possible to add a column to lock the row; check that stripe_account_creation_started_at is null; change it to the current time; commit. Only one replica will be able to successfully commit that and progress to the next step, but then you need a way of finding deadlocks (the app could crash right after the commit) and breaking them.
It is a little complicated but I personally would rather have exact accounting of what is happening and why, rather than guessing when some user logs in and doesn't have a stripe account.
Edit to add: one last bit... I like running this all in a background loop that wakes up periodically to see what needs to be done, but your createAccount RPC should also be able to immediately wake this loop up. That way if everything goes well, you don't add any latency by introducing this state machine / workflow manager. If something happens like Stripe being down... progress will resume when the loop wakes up again. For that reason, I think you should be explicit and provide the end user with a system that lets them request an account and lets them check the status of the request. (Maybe not the user directly, but your signup UI will be consuming this stuff and providing them with appropriate feedback. You don't want the createAccount RPC to just hang while you're talking to Stripe and Auth0 in the background. Probably. The happy case might take a second, but the worst case could take much longer. Design your UI to be good for the worst case, and it will be good for the happy case too.)
- c89X 7y agoI like this approach: it closely matches my initial instincts of having a 'transaction' table for each type of event, with columns for each of the steps. The reason I'm weary of this approach, is because I have done an implementation of this in the past but it turned out to become a nightmare - this is most likely due to a combination of the complexity of that particular domain, my inexperience with this type of problem and external pressures. I did find some pretty good resources on Sagas - do you have any thought on comparing Sagas pro/cons vis-a-vis this approach?
- jrockway 7y agoI do not know much about sagas. I like this approach because you need this user table anyway (probably!) and no data is duplicated in your system. There is one authoritative place that maps username to stripe id, and the logic to work on computing that (making stripe api calls, retrying) lives inside the system that is responsible for telling other parts of your infrastructure that ID. So you do get a consistent view from everywhere; no system will ever have the "wrong answer". The complexity here comes from handling every possible transition. With an ad-hoc approach, some transitions are hidden and can be ignored until they actually happen. With this approach, you have to think about it all up front and write the necessary unit tests. The first version of my password changer thing detected states I hadn't thought about and just sent an alert for someone to manually do something. There was a control plane outage and everything broke one day, and that forced me to implement some of the automatic cleanup. Some other comments suggest "don't write this, just send your ops team an email when something breaks" and that is fine for the initial version. All you really want is a detailed record of what's happened and what went wrong, from there you can either fix it one-off, or finally write the code to fix it automatically. You don't need to implement retries and rollbacks on version 0... just flag the account and go fix it yourself until you decide you'd rather have the computer do it.