3 ms·
This really is a great explanation and the strategy I have taken with several similar types of systems (not ES, but various specialized indexes that are "slaved
by stickfigure 12y ago
This really is a great explanation and the strategy I have taken with several similar types of systems (not ES, but various specialized indexes that are "slaved" to a master db). In fact, it's very similar to the replication model of MySQL.
One thing you didn't mention but should be pointed out is that you have to manage clock skew carefully. If you have a single database instance, you can rely on timestamps inserted by the db server ("select now()") but if your master database is clustered, even this is not trustworthy. And of course same goes for timestamps generated by your application. Clocks are the bane of distributed systems.
My usual strategy (aside from ensuring everything is ntp synced) is to pick a plausible time period that represents an amount of clock skew that the system should never experience and subtract that from the cursor at every poll. It means you'll always reindex the last few seconds (or whatever you pick for N) worth of data, but you won't miss out on updates.
- lobster_johnson 12y agoYeah, you should generate an always-increasing sequence number in the source database. Timestamps cannot be used to "select ... where T > ?", since at the time of polling another record may come in with the same timestamp as the newest one, and this will skip records; "where T >= ?" won't work either unless you can de-duplicate, and that requires local state.
- stickfigure 12y agoUnfortunately you don't get monotonic sequence numbers in distributed databases. Timestamps actually work very well for most applications, at the cost of extra reindexing proportional to how closely you can manage clock skew. Keep in mind that you are always subtracting a reasonable max skew value and getting extra records. All operations on the secondary store must be upserts.