Query performance is critical. Insert perf is a comparatively minor concern. Even so, there are a few factors to consider: - Half second dedupes are for users

Query performance is critical. Insert perf is a comparatively minor concern. Even so, there are a few factors to consider: - Half second dedupes are for users with 100k+ events, which is <<1% of them. - We batch events for ~5s before adding them to the cluster, so we aren't deduping for every event -- only once per ~5s of events per user. If this becomes an issue, we can remove the deduping from normal operation and only call it when we're backfilling / updating events. Even so, we still need this function to exist, and the 100x performance improvement is very helpful.