4 ms·
Wouldn't this only work when your client is the one that last updated the record? Because how else would you know the correct min lsn? Seems like you would need
by foota 9y ago
Wouldn't this only work when your client is the one that last updated the record? Because how else would you know the correct min lsn? Seems like you would need to store the map from user to lsn in something other than pg to get any benefit here.
- dustingetz 9y agoI think there's a stateful load balancer polling every replica "By routing read operations only to replicas that are caught up enough to run them accurately" "To save every api process from having to reach out and check on the replication status of every replica for itself, we’ll have a process called an observer that periodically refreshes the state of every replica and stores it to a Postgres table. ... Keep in mind that this status information could really go anywhere. If we have Redis available, we could put it in there for fast access" "The observer runs in a loop, and executes something like this on every iteration" Also, "The technique is inspired by GitLab’s article on scaling their database, where they refer to it as 'sticky connections'"
- brandur 9y ago(I'm the author.) I'm storing `min_lsn` directly to the database, so anything connecting to it and loading a user record should be able to take advantage of the information. I have a note in there about this as well, but although I'm showing how this information can be persisted to Postgres so that I'm only using one data store, it could really live in any kind of shared persistence layer. For example, replication statuses and a user's `min_lsn` might be put in Redis instead, and be treated as ephemeral data such that we don't worry about losing them because they can be repopulated on every restart. It's also probably a good idea to have individual API workers caching replication statuses directly in memory so they can select a read candidate without even having to go to a database.
- foota 9y agoThanks for writing the article, it was interesting. To see if I'm missing something here though, in order for this to offer benefit the way it is now, I would need to already have a correct min_lsn for a user? But the issue is that if I got this from a previous request it could be out of date, isn't that accurate?