3 ms·

Looks a bit overengineered due to Netflix's own microservices nonsense. I would be more interested in how a higher traffic video company like Pornhub handles t

Looks a bit overengineered due to Netflix's own microservices nonsense. I would be more interested in how a higher traffic video company like Pornhub handles things like this.

oreoftw 2y ago
How would you design it to support mentioned use cases?
- klaussilveira 2y ago
  HyperLogLog and PostgreSQL: https://github.com/citusdata/postgresql-hll https://github.com/citusdata/postgresql-hll Or even simpler, Roaring Bitmaps: https://pncnmnp.github.io/blogs/roaring-bitmaps.html https://pncnmnp.github.io/blogs/roaring-bitmaps.html https://blog.quastor.org/p/grab-rate-limiting https://blog.quastor.org/p/grab-rate-limiting https://github.com/RoaringBitmap/CRoaring https://github.com/RoaringBitmap/CRoaring
  - philjohn 2y ago
    HyperLogLog doesn't support exact counters though, does it? That seems to be one of the core requirements of the queue-based solution.
    - rshrin 2y ago
      HyperLogLog (or even Count-Min Sketch) will not support some of the requirements of even the Best-Effort counter (clearing counts for specific keys, decrementing counts by any arbitrary number, having a TTL on counts etc.). For Accurate counters, we are trying to solve for multi-region read/write availability at low single-digit millisecond latency at cheap costs, using the infrastructure we already operate and deploy. There are also other requirements such as tracking the provenance of increments, which play a part.
philjohn 2y ago
Have you worked with distributed counters before? It's a hard problem to solve. Typical tradeoffs are lower cardinality for exact counters. The queue solution is pretty elegant.
- ElevenLathe 2y ago
  The queuing reminds me of old "tote board" technology. I can't find a reference easily but these were machines used to automatically figure and display parimutuel payouts at horse tracks. One particular kind of them would have the cashiers' terminals emit ball bearings on a track back at the central (digital but electromechanical) counter for each betting pool. This arrangement allowed the cashiers to sell as fast as they liked without jamming the works, and then allowed the calculation/display of odds to be "eventually consistent".
Alupis 2y ago
> Netflix's own microservices nonsense How many times has Netflix been entirely down over the years? Seems it's not "nonesense".
- ilrwbwrkhv 2y ago
  Netflix's technology is so nonsense that right now for the boxing match they are dropping the quality to 720 pixels.

Looks a bit overengineered due to Netflix's own microservices nonsense. I would be more interested in how a higher traffic video company like Pornhub handles things like this.

oreoftw 2y ago

How would you design it to support mentioned use cases?

klaussilveira 2y ago
HyperLogLog and PostgreSQL: https://github.com/citusdata/postgresql-hll https://github.com/citusdata/postgresql-hll Or even simpler, Roaring Bitmaps: https://pncnmnp.github.io/blogs/roaring-bitmaps.html https://pncnmnp.github.io/blogs/roaring-bitmaps.html https://blog.quastor.org/p/grab-rate-limiting https://blog.quastor.org/p/grab-rate-limiting https://github.com/RoaringBitmap/CRoaring https://github.com/RoaringBitmap/CRoaring
- philjohn 2y ago
  HyperLogLog doesn't support exact counters though, does it? That seems to be one of the core requirements of the queue-based solution.
  - rshrin 2y ago
    HyperLogLog (or even Count-Min Sketch) will not support some of the requirements of even the Best-Effort counter (clearing counts for specific keys, decrementing counts by any arbitrary number, having a TTL on counts etc.). For Accurate counters, we are trying to solve for multi-region read/write availability at low single-digit millisecond latency at cheap costs, using the infrastructure we already operate and deploy. There are also other requirements such as tracking the provenance of increments, which play a part.

philjohn 2y ago

Have you worked with distributed counters before? It's a hard problem to solve. Typical tradeoffs are lower cardinality for exact counters. The queue solution is pretty elegant.

ElevenLathe 2y ago
The queuing reminds me of old "tote board" technology. I can't find a reference easily but these were machines used to automatically figure and display parimutuel payouts at horse tracks. One particular kind of them would have the cashiers' terminals emit ball bearings on a track back at the central (digital but electromechanical) counter for each betting pool. This arrangement allowed the cashiers to sell as fast as they liked without jamming the works, and then allowed the calculation/display of odds to be "eventually consistent".

Alupis 2y ago

> Netflix's own microservices nonsense How many times has Netflix been entirely down over the years? Seems it's not "nonesense".

ilrwbwrkhv 2y ago
Netflix's technology is so nonsense that right now for the boxing match they are dropping the quality to 720 pixels.