4 ms·

Close, IIRC we cached the fact you had just done a write, and a subsequent read request that arrived on the replica region was then proxied to the primary regio

Close, IIRC we cached the fact you had just done a write, and a subsequent read request that arrived on the replica region was then proxied to the primary region instead of serviced locally.

IgorPartola 6y ago
Pretty clever. Is that still how it works?
- glittershark 6y ago
  Pretty sure this paper describes what they're doing now: https://research.fb.com/publications/flighttracker-consistency-across-read-optimized-online-stores-at-facebook/ https://research.fb.com/publications/flighttracker-consisten...
  - mackman 6y ago
    I'm not sure if FlightTracker completely replaced the need for the internal consistency inside Tao. You can read about that here: https://www.usenix.org/system/files/conference/atc13/atc13-bronson.pdf https://www.usenix.org/system/files/conference/atc13/atc13-b...
eismcc 6y ago
Dirty bits at scale
- gogopuppygogo 6y ago
  I’m guessing this is ecc memory so likely correcting for bad data.
  - Hackbraten 6y ago
    I think they meant dirty bit as in “a flag that means update needed,” not as in “bit flipped due to glitch.”
deleted 6y ago
[deleted]
cranekam 6y ago
My memory is fuzzy now but this dates back to when there were only two datacenter regions and one of them held all the primary DBs (2011 or so). All write endpoints were served in that region, so if a user routed to the secondary region did a write the request was proxied to the primary region. After doing a write a cookie was set for the user in question which caused any future reads to be proxied to the primary region for a few seconds while the DB replication stream (upon which cache invalidation was piggybacked) caught up, because if they went to the secondary region memcached was now stale. It hasn’t been this way since around 2013 but again I am fuzzy on how. I think that’s when most such data was switched to TAO, which has local read what you wrote consistency. As long as users landed in the same cluster (and thus TAO cluster) what they wrote was visible to them, even if the DB write hadn’t yet replicated to their region. FlightTracker postdates my time at FB (ended 2018ish) so I’m not sure how that is used. These systems evolved a lot over time as requirements changed. I don’t remember anything about writes being batched in memcached and merged in on page load.

Close, IIRC we cached the fact you had just done a write, and a subsequent read request that arrived on the replica region was then proxied to the primary region instead of serviced locally.

IgorPartola 6y ago

Pretty clever. Is that still how it works?

glittershark 6y ago
Pretty sure this paper describes what they're doing now: https://research.fb.com/publications/flighttracker-consistency-across-read-optimized-online-stores-at-facebook/ https://research.fb.com/publications/flighttracker-consisten...
- mackman 6y ago
  I'm not sure if FlightTracker completely replaced the need for the internal consistency inside Tao. You can read about that here: https://www.usenix.org/system/files/conference/atc13/atc13-bronson.pdf https://www.usenix.org/system/files/conference/atc13/atc13-b...

eismcc 6y ago

Dirty bits at scale

gogopuppygogo 6y ago
I’m guessing this is ecc memory so likely correcting for bad data.
- Hackbraten 6y ago
  I think they meant dirty bit as in “a flag that means update needed,” not as in “bit flipped due to glitch.”

deleted 6y ago

[deleted]

cranekam 6y ago

My memory is fuzzy now but this dates back to when there were only two datacenter regions and one of them held all the primary DBs (2011 or so). All write endpoints were served in that region, so if a user routed to the secondary region did a write the request was proxied to the primary region. After doing a write a cookie was set for the user in question which caused any future reads to be proxied to the primary region for a few seconds while the DB replication stream (upon which cache invalidation was piggybacked) caught up, because if they went to the secondary region memcached was now stale. It hasn’t been this way since around 2013 but again I am fuzzy on how. I think that’s when most such data was switched to TAO, which has local read what you wrote consistency. As long as users landed in the same cluster (and thus TAO cluster) what they wrote was visible to them, even if the DB write hadn’t yet replicated to their region. FlightTracker postdates my time at FB (ended 2018ish) so I’m not sure how that is used. These systems evolved a lot over time as requirements changed. I don’t remember anything about writes being batched in memcached and merged in on page load.