5 ms·

> I worked with a very high-scale feature flag system a while ago - thousands of flag checks a second. May I ask why the flags are checked that frequently? Cou

> I worked with a very high-scale feature flag system a while ago - thousands of flag checks a second. May I ask why the flags are checked that frequently? Couldn't they be cached for at least a minute? > It does it by acting as a PostgreSQL proxy. [...] and any writes are forwarded on to "real" PostgreSQL. What happens if there's a multi-statement transaction with a bunch of writes sent-off to the mothership - which then get returned to the client via logical replication, but then there's a ROLLBACK - how would that situation be handled such that both the SQLite edge DBs and the mothership DB are able to rollback okay - would this impact other clients?

vasco 3y ago
> May I ask why the flags are checked that frequently? Couldn't they be cached for at least a minute? Not in that project but feature flags don't have to be all or nothing. You can apply flags to specific cohorts of your users for example, so if you have a large user base, even if you cache them per-user, it still translates into many checks a second for large systems.
- mikepurvis 3y ago
  I guess the cost of doing it precisely isn't terribly high, but if the majority of the flags were "off" (eg, subsets of users being opted into a beta or something), I wonder if you cut a bunch of the queries by sending down a bloom filter. So basically you check the filter first, and if that says the feature is enabled, only then do you actually ask the real DB.
  - kiitos 3y ago
    Checking a feature flag for a given cohort/user/demographic/etc. should always be a memory read, not a syscall. (You can't have syscalls in a hot loop!) That means feature flags will always be cached in each application instance from their source of truth. And that means you would never query the source of truth directly from the application, regardless of any property of the flag or its users.
runeks 3y ago
My limited understanding of logical replication is that writes only happen at COMMIT. Ie. nothing is replicated until it's committed.
zknill 3y ago
The logical replication protocol sends a series of messages that essentially follow the flow that a database transaction would. i.e. a stream of messages like: "BEGIN", "[the data]", ["COMMIT" or "ROLLBACK"]. So any application that listens to the Postgres replication protocol can handle the transaction in the same way that Postgres does. Concretely you might choose to open a SQLite transaction on BEGIN, apply the statements, and then COMMIT or ROLLBACK based on the next messages received on the stream replication protocol. The data sent on the replication protocol includes the state of the row after the write query has completed. This means you don't need to worry about getting out of sync on queries like "UPDATE field = field + 1" because you have access to the exact resulting value as stored by Postgres. TL;DR - you can follow the same begin/change/commit flow that the original transaction did on the upstream Postgres server, and you have access to the exact underlying data after the write was committed. It's also true (as other commenters have pointed out) that for not-huge transactions (i.e. not streaming transactions, new feature in Postgres 15) the BEGIN message will only be sent if the transaction was committed. It's pretty unlikely that you will ever process a ROLLBACK message from the protocol (although possible).
- deleted 3y ago
  [deleted]
- paulddraper 3y ago
  Logical replication never includes uncommitted data, unless you have written an custom output plugin. EDIT: https://stackoverflow.com/questions/52202534/postgresql-does-logical-replication-include-rollback-transaction https://stackoverflow.com/questions/52202534/postgresql-does...
  - zknill 3y ago
    It's true for protocol version 1 that only the committed data is sent on the replication connection. In protocol version 2 (introduced in postgres 14) large in-progress transactions can appear in the new "Stream" messages sent on the replication connection: StreamStart, StreamStop, StreamAbort, StreamCommit, etc. In the case of large in-progress transactions uncommitted data might end up in the replication connection after a StreamStart message. But you would also receive a StreamCommit or StreamAbort message to tell you what happened to that transaction. I've not worked out what qualifies as a "large" transaction though. But it is _possible_ to get uncommitted data in the replication connection, although unlikely. https://www.postgresql.org/docs/15/protocol-logicalrep-message-formats.html https://www.postgresql.org/docs/15/protocol-logicalrep-messa...
jmull 3y ago
> May I ask why the flags are checked that frequently? Couldn't they be cached for at least a minute? Not the previous poster, but it appears in the scenario, the SQLite database is the cache.
paulddraper 3y ago
> Couldn't they be cached for at least a minute? Only per feature+per user. (Though 1000s per second does seem high unless your scale is gigantic.) > What happens if there's a multi-statement transaction with a bunch of writes sent-off to the mothership - which then get returned to the client via logical replication, but then there's a ROLLBACK Nothing makes it into the replication stream until it is committed.
rockostrich 3y ago
Feature flag systems are usually based on a set of rules that could be serialized and evaluated locally (this is how pretty much every open source feature flag system and feature flag SaaS works). Usually it's based on some kind of UUID being hashed with a per-flag seed and bucketed after some set of targeting rules are applied to other properties passed in for that user. There are added features where you can stores large cohorts to do specific targeting and usually there's some kind of local cache added to make that look-up faster for recent users. I'm not sure what the original commenter was doing but it sounds like they had some kind of targeting that was almost entirely based on cohorts or maybe they needed to have stability over time which would require a database. We did something similar recently except we just store a "session ID" with a blob for look-up and the evaluation only happens on the first request for a given session ID.
simonw 3y ago
They were being cached for at least a minute (maybe even more than that, I can't remember the details) - that's what the local memcached instance was for. This was problematic though because changing a feature flag and then waiting for a minute plus to see if the change actually worked can be frustrating, especially if it relates to an outage of some sort.
- kiitos 3y ago
  If the cache is out-of-process, access is going to be subject to faults and latencies that are categorically different than anything that would be in-process. You'd never want to include a request to an out-of-process service in any kind of hot loop, even if that service is on the same machine.

vasco 3y ago

> May I ask why the flags are checked that frequently? Couldn't they be cached for at least a minute? Not in that project but feature flags don't have to be all or nothing. You can apply flags to specific cohorts of your users for example, so if you have a large user base, even if you cache them per-user, it still translates into many checks a second for large systems.

mikepurvis 3y ago
I guess the cost of doing it precisely isn't terribly high, but if the majority of the flags were "off" (eg, subsets of users being opted into a beta or something), I wonder if you cut a bunch of the queries by sending down a bloom filter. So basically you check the filter first, and if that says the feature is enabled, only then do you actually ask the real DB.
- kiitos 3y ago
  Checking a feature flag for a given cohort/user/demographic/etc. should always be a memory read, not a syscall. (You can't have syscalls in a hot loop!) That means feature flags will always be cached in each application instance from their source of truth. And that means you would never query the source of truth directly from the application, regardless of any property of the flag or its users.

runeks 3y ago

My limited understanding of logical replication is that writes only happen at COMMIT. Ie. nothing is replicated until it's committed.

zknill 3y ago

The logical replication protocol sends a series of messages that essentially follow the flow that a database transaction would. i.e. a stream of messages like: "BEGIN", "[the data]", ["COMMIT" or "ROLLBACK"]. So any application that listens to the Postgres replication protocol can handle the transaction in the same way that Postgres does. Concretely you might choose to open a SQLite transaction on BEGIN, apply the statements, and then COMMIT or ROLLBACK based on the next messages received on the stream replication protocol. The data sent on the replication protocol includes the state of the row after the write query has completed. This means you don't need to worry about getting out of sync on queries like "UPDATE field = field + 1" because you have access to the exact resulting value as stored by Postgres. TL;DR - you can follow the same begin/change/commit flow that the original transaction did on the upstream Postgres server, and you have access to the exact underlying data after the write was committed. It's also true (as other commenters have pointed out) that for not-huge transactions (i.e. not streaming transactions, new feature in Postgres 15) the BEGIN message will only be sent if the transaction was committed. It's pretty unlikely that you will ever process a ROLLBACK message from the protocol (although possible).

deleted 3y ago
[deleted]
paulddraper 3y ago
Logical replication never includes uncommitted data, unless you have written an custom output plugin. EDIT: https://stackoverflow.com/questions/52202534/postgresql-does-logical-replication-include-rollback-transaction https://stackoverflow.com/questions/52202534/postgresql-does...
- zknill 3y ago
  It's true for protocol version 1 that only the committed data is sent on the replication connection. In protocol version 2 (introduced in postgres 14) large in-progress transactions can appear in the new "Stream" messages sent on the replication connection: StreamStart, StreamStop, StreamAbort, StreamCommit, etc. In the case of large in-progress transactions uncommitted data might end up in the replication connection after a StreamStart message. But you would also receive a StreamCommit or StreamAbort message to tell you what happened to that transaction. I've not worked out what qualifies as a "large" transaction though. But it is _possible_ to get uncommitted data in the replication connection, although unlikely. https://www.postgresql.org/docs/15/protocol-logicalrep-message-formats.html https://www.postgresql.org/docs/15/protocol-logicalrep-messa...

jmull 3y ago

> May I ask why the flags are checked that frequently? Couldn't they be cached for at least a minute? Not the previous poster, but it appears in the scenario, the SQLite database is the cache.

paulddraper 3y ago

> Couldn't they be cached for at least a minute? Only per feature+per user. (Though 1000s per second does seem high unless your scale is gigantic.) > What happens if there's a multi-statement transaction with a bunch of writes sent-off to the mothership - which then get returned to the client via logical replication, but then there's a ROLLBACK Nothing makes it into the replication stream until it is committed.

rockostrich 3y ago

Feature flag systems are usually based on a set of rules that could be serialized and evaluated locally (this is how pretty much every open source feature flag system and feature flag SaaS works). Usually it's based on some kind of UUID being hashed with a per-flag seed and bucketed after some set of targeting rules are applied to other properties passed in for that user. There are added features where you can stores large cohorts to do specific targeting and usually there's some kind of local cache added to make that look-up faster for recent users. I'm not sure what the original commenter was doing but it sounds like they had some kind of targeting that was almost entirely based on cohorts or maybe they needed to have stability over time which would require a database. We did something similar recently except we just store a "session ID" with a blob for look-up and the evaluation only happens on the first request for a given session ID.

simonw 3y ago

They were being cached for at least a minute (maybe even more than that, I can't remember the details) - that's what the local memcached instance was for. This was problematic though because changing a feature flag and then waiting for a minute plus to see if the change actually worked can be frustrating, especially if it relates to an outage of some sort.

kiitos 3y ago
If the cache is out-of-process, access is going to be subject to faults and latencies that are categorically different than anything that would be in-process. You'd never want to include a request to an out-of-process service in any kind of hot loop, even if that service is on the same machine.