3 ms·
> Our schema is fully denormalized, which is particularly important for performance And yet appending a single event (you dedupe for each event, right?) takes
by twic 13y ago
> Our schema is fully denormalized, which is particularly important for performance
And yet appending a single event (you dedupe for each event, right?) takes half a second. That's an eternity!
- drob 13y agoQuery performance is critical. Insert perf is a comparatively minor concern. Even so, there are a few factors to consider: - Half second dedupes are for users with 100k+ events, which is <<1% of them. - We batch events for ~5s before adding them to the cluster, so we aren't deduping for every event -- only once per ~5s of events per user. If this becomes an issue, we can remove the deduping from normal operation and only call it when we're backfilling / updating events. Even so, we still need this function to exist, and the 100x performance improvement is very helpful.