5 ms·

I'm probably misunderstanding something here, but it seems like this whole article ignores race conditions. For instance, if you 'update on write', how do guara

by wryun 6y ago

I'm probably misunderstanding something here, but it seems like this whole article ignores race conditions. For instance, if you 'update on write', how do guarantee that the cache writes don't occur in the wrong order? e.g. write B write A update cache with A update cache with B You're just replicated all the usual single machine multi-threading issues when you abandon the ACID happy place by adding a cache... EDIT: and this is why I would use TTLs even with update on write, folks. And this is why I mostly happily use volatile-lru on multi-purpose redis caches, since everything I'd want to evict has a TTL.

calpaterson 6y ago
I don't disagree this is a problem but it is so general that it applies everywhere you use multiple backing services. A classic is sending an email: do you send the notification email before or after committing to the database? If you send after you run the risk of committing to the database and then failing to send the notification email. If you send before you might send the email and then fail to commit. I would love to have cross database/mailserver/cache/etc transactions but it's implemented nowhere as far as I can see, thought there seems to have been work towards it to the 90s I think they ultimately gave up.
- wryun 6y ago
  If you agree it's a problem, it'd be great if you'd at least add this onto the tips and pitfalls section. The main claim of the article is get rid of TTLs and "don't give up correctness to gain speed", which seems incorrect to me. One of the main reasons that TTLs are good is exactly this; if you remove TTLs as you advise, strategy 2 and 3 can have permanently wrong data in the cache, right? This is ... not good? EDIT: I don't think the email one is a good comparison, either. (a) you have to do this (it's not an optional bonus, as the cache is), and (b) it's perfectly reasonable for critical emails to record in the db whether it's successful or report it elsewhere - you can have multiple states! If we're just talking notifications in general, I would prefer sending them after commit (i.e. don't tell people until we're sure the action has happened, and wear the possible not telling), though depending on your priorities you may switch it. But I think of this as an external action, NOT something where you're messing with your system's version of the truth.
  - calpaterson 6y ago
    Fair dos - I will try to add it when I get some time later today. The email thing is just an example, you might have a preference on how to handle that particular case but hopefully you can recognise the principle I'm getting at. I don't necessarily agree that this problem is a cache specific issue and I definitely wouldn't start putting random TTLs on things in an attempt to try to hack around it. If anything, that is likely only to make the issue even more baffling for the user if it does occur. And of course you are now back to TTL numberism - how long a TTL should you set? If it really is a realistic concern (and I would suggest it's not for most!) then time to look at having locks somewhere.
    - wryun 6y ago
      Thanks very much for thinking about adding this little caveat - part of the reason I'm keen is that this really is a good list of 'things to think about' re caching, and I'd like to add this one to the pile. With the TTL advice, why do you think this would make it _more_ baffling for the user? If the choices are 'it's wrong "forever"' or 'it's wrong for 5 minutes', isn't the latter the better one? I'd have to have a compelling reason to cache indefinitely if I knew it might be wrong... admittedly, if there are a lot of writes (where we're most likely to mess the cache up) you're likely to get a new cached value quickly. In terms of whether this issue is cache specific, I honestly think it is _in this context_ because of the way you've phrased the article: that you can add an external cache to your db accesses and everything is faster without any cost in correctness. It's almost better to think of it as 'I'm adding an entirely new kind of database', and realise that keeping those databases reliably in sync is non-trivial without relaxing your guarantees.
- oftenwrong 6y ago
  A bunch of databases support XA transactions [1], or otherwise have generic support for 2PC that can be hooked up to an external transaction manager [2]. In theory, we could build support into most systems. It does seem like this idea has lost momentum, though. I am not sure why. [1] https://en.wikipedia.org/wiki/X/Open_XA https://en.wikipedia.org/wiki/X/Open_XA [2] for example, https://www.postgresql.org/docs/13/sql-prepare-transaction.html https://www.postgresql.org/docs/13/sql-prepare-transaction.h...
  - cratermoon 6y ago
    Two-phase commit is out of style because everything is distributed and the Fallacies of Distributed Systems presents several reasons why the 2PC protocols spend most of their effort on error handling and recovery. Simply put, at scale, something is always failing, so a 2PC setup is always doing recovery.
- namibj 6y ago
  You write to the database, and have a separate notification-sender that feeds on the database change stream, sends out notifications, and keeps track of which notifications it has sent. If that process crashes, it will re-send one or maybe a few notifications.

by wryun 6y ago

calpaterson 6y ago

I don't disagree this is a problem but it is so general that it applies everywhere you use multiple backing services. A classic is sending an email: do you send the notification email before or after committing to the database? If you send after you run the risk of committing to the database and then failing to send the notification email. If you send before you might send the email and then fail to commit. I would love to have cross database/mailserver/cache/etc transactions but it's implemented nowhere as far as I can see, thought there seems to have been work towards it to the 90s I think they ultimately gave up.

wryun 6y ago
If you agree it's a problem, it'd be great if you'd at least add this onto the tips and pitfalls section. The main claim of the article is get rid of TTLs and "don't give up correctness to gain speed", which seems incorrect to me. One of the main reasons that TTLs are good is exactly this; if you remove TTLs as you advise, strategy 2 and 3 can have permanently wrong data in the cache, right? This is ... not good? EDIT: I don't think the email one is a good comparison, either. (a) you have to do this (it's not an optional bonus, as the cache is), and (b) it's perfectly reasonable for critical emails to record in the db whether it's successful or report it elsewhere - you can have multiple states! If we're just talking notifications in general, I would prefer sending them after commit (i.e. don't tell people until we're sure the action has happened, and wear the possible not telling), though depending on your priorities you may switch it. But I think of this as an external action, NOT something where you're messing with your system's version of the truth.
- calpaterson 6y ago
  Fair dos - I will try to add it when I get some time later today. The email thing is just an example, you might have a preference on how to handle that particular case but hopefully you can recognise the principle I'm getting at. I don't necessarily agree that this problem is a cache specific issue and I definitely wouldn't start putting random TTLs on things in an attempt to try to hack around it. If anything, that is likely only to make the issue even more baffling for the user if it does occur. And of course you are now back to TTL numberism - how long a TTL should you set? If it really is a realistic concern (and I would suggest it's not for most!) then time to look at having locks somewhere.
  - wryun 6y ago
    Thanks very much for thinking about adding this little caveat - part of the reason I'm keen is that this really is a good list of 'things to think about' re caching, and I'd like to add this one to the pile. With the TTL advice, why do you think this would make it _more_ baffling for the user? If the choices are 'it's wrong "forever"' or 'it's wrong for 5 minutes', isn't the latter the better one? I'd have to have a compelling reason to cache indefinitely if I knew it might be wrong... admittedly, if there are a lot of writes (where we're most likely to mess the cache up) you're likely to get a new cached value quickly. In terms of whether this issue is cache specific, I honestly think it is _in this context_ because of the way you've phrased the article: that you can add an external cache to your db accesses and everything is faster without any cost in correctness. It's almost better to think of it as 'I'm adding an entirely new kind of database', and realise that keeping those databases reliably in sync is non-trivial without relaxing your guarantees.
oftenwrong 6y ago
A bunch of databases support XA transactions [1], or otherwise have generic support for 2PC that can be hooked up to an external transaction manager [2]. In theory, we could build support into most systems. It does seem like this idea has lost momentum, though. I am not sure why. [1] https://en.wikipedia.org/wiki/X/Open_XA https://en.wikipedia.org/wiki/X/Open_XA [2] for example, https://www.postgresql.org/docs/13/sql-prepare-transaction.html https://www.postgresql.org/docs/13/sql-prepare-transaction.h...
- cratermoon 6y ago
  Two-phase commit is out of style because everything is distributed and the Fallacies of Distributed Systems presents several reasons why the 2PC protocols spend most of their effort on error handling and recovery. Simply put, at scale, something is always failing, so a 2PC setup is always doing recovery.
namibj 6y ago
You write to the database, and have a separate notification-sender that feeds on the database change stream, sends out notifications, and keeps track of which notifications it has sent. If that process crashes, it will re-send one or maybe a few notifications.