6 ms·

For others who were left scratching their heads at what exactly this pop-sci-explained PoolCounter mechanism actually is: https://wikitech.wikimedia.org/wiki/P

For others who were left scratching their heads at what exactly this pop-sci-explained PoolCounter mechanism actually is: https://wikitech.wikimedia.org/wiki/PoolCounter https://wikitech.wikimedia.org/wiki/PoolCounter TL;DR: It's a limiter on how many workers start rendering the new page version when the old page version in cache has been invalidated.

zatkin 11y ago
+1 for whoever named it the "Michael Jackson problem". I'm going to start using that if I ever encounter the same issue myself.
- encoderer 11y ago
  Is this not the thundering herd problem?
  - gerry_shaw 11y ago
    As I understand it yes, but I think the "Micheal Jackson problem" is a better name.
    - techdragon 11y ago
      It's a cute name but there is existing Research that relates to this issue which calls it "the thundering herd problem" so I would recommend the one you can actually search for without problems.
    - yrro 11y ago
      Who?
  - bubuga 11y ago
    > Is this not the thundering herd problem? There's a wikipedia article on the thundering herd problem. https://en.wikipedia.org/wiki/Thundering_herd_problem https://en.wikipedia.org/wiki/Thundering_herd_problem The article needs a little love, though.
- The_ed17 11y ago
  That was Tim Starling: https://wikitech.wikimedia.org/w/index.php?title=PoolCounter&oldid=48734 https://wikitech.wikimedia.org/w/index.php?title=PoolCounter... https://wikimediafoundation.org/wiki/User:Tim_Starling_(WMF) https://wikimediafoundation.org/wiki/User:Tim_Starling_(WMF)
- RyJones 11y ago
  American Idol caused widespread rearchitecting of SMSGW/SMSC. Fun times; voting was a very distributed DoS.
atdt 11y ago
Yeah, I am the engineer mentioned in the article, and I agree the explanation doesn't really work. The pieces of the explanation that ended up in the article itself don't add up to a coherent explanation. The fault for that is mostly mine. In hindsight, my original explanation was too long and too elaborate to be helpful. It's a good reminder that it is easy to go to far with an analogy and end up complicating the thing you were trying to simplify. Oh well, live and learn :) There are more (coherent and to-the-point) details about PoolCounter in the prologue to PoolCounter.php in MediaWiki's source tree: https://github.com/wikimedia/mediawiki/blob/1617e7822eaf74261687d2d2c3148ce10ab5da69/includes/poolcounter/PoolCounter.php#L3-L43 https://github.com/wikimedia/mediawiki/blob/1617e7822eaf7426... And in a short blog post by Domas Mituzas, who is the original author of PoolCounter: https://dom.as/2009/06/26/embarrassment/ https://dom.as/2009/06/26/embarrassment/
- homero 11y ago
  Basically a page view checks the cache and rebuilds if necessary. Thousands of page hits in the same second before the build is over starts thousands of parallel rebuilds.
  - taf2 11y ago
    Also known as the thundering herd...
    - homero 11y ago
      Yeah I never really solved it while I was using memcached but I'm not Wikipedia
      - andrewingram 11y ago
        The way i've seen it done is something like this: * Cache never automatically expires, but you do have some notion of staleness * Whenever you request data, you get it from the cache (if it exists), and check for staleness, so you cached data needs to know when it was cached. * Return the data as usual, but if the data was also stale, you fire off a worker to update the cached data. * If you have lots of requests happening at the same time, you have a system for seeing if a worker already exists, to ensure that you only create one (for each piece of cached data). * For the time it takes for the worker to complete, you have to be okay with serving stale data, in most cases this is okay. There's an edge case missed here, which is what to do when the cache is empty (either because it's one of the first requests, or because the cached data has been evicted). That's up to you, depending on your use case. You can basically either return a default value, you can pre-warm your cache, or you can let the requests hang until the data is ready.

zatkin 11y ago

+1 for whoever named it the "Michael Jackson problem". I'm going to start using that if I ever encounter the same issue myself.

encoderer 11y ago
Is this not the thundering herd problem?
- gerry_shaw 11y ago
  As I understand it yes, but I think the "Micheal Jackson problem" is a better name.
  - techdragon 11y ago
    It's a cute name but there is existing Research that relates to this issue which calls it "the thundering herd problem" so I would recommend the one you can actually search for without problems.
  - yrro 11y ago
    Who?
- bubuga 11y ago
  > Is this not the thundering herd problem? There's a wikipedia article on the thundering herd problem. https://en.wikipedia.org/wiki/Thundering_herd_problem https://en.wikipedia.org/wiki/Thundering_herd_problem The article needs a little love, though.
The_ed17 11y ago
That was Tim Starling: https://wikitech.wikimedia.org/w/index.php?title=PoolCounter&oldid=48734 https://wikitech.wikimedia.org/w/index.php?title=PoolCounter... https://wikimediafoundation.org/wiki/User:Tim_Starling_(WMF) https://wikimediafoundation.org/wiki/User:Tim_Starling_(WMF)
RyJones 11y ago
American Idol caused widespread rearchitecting of SMSGW/SMSC. Fun times; voting was a very distributed DoS.

atdt 11y ago

Yeah, I am the engineer mentioned in the article, and I agree the explanation doesn't really work. The pieces of the explanation that ended up in the article itself don't add up to a coherent explanation. The fault for that is mostly mine. In hindsight, my original explanation was too long and too elaborate to be helpful. It's a good reminder that it is easy to go to far with an analogy and end up complicating the thing you were trying to simplify. Oh well, live and learn :) There are more (coherent and to-the-point) details about PoolCounter in the prologue to PoolCounter.php in MediaWiki's source tree: https://github.com/wikimedia/mediawiki/blob/1617e7822eaf74261687d2d2c3148ce10ab5da69/includes/poolcounter/PoolCounter.php#L3-L43 https://github.com/wikimedia/mediawiki/blob/1617e7822eaf7426... And in a short blog post by Domas Mituzas, who is the original author of PoolCounter: https://dom.as/2009/06/26/embarrassment/ https://dom.as/2009/06/26/embarrassment/

homero 11y ago
Basically a page view checks the cache and rebuilds if necessary. Thousands of page hits in the same second before the build is over starts thousands of parallel rebuilds.
- taf2 11y ago
  Also known as the thundering herd...
  - homero 11y ago
    Yeah I never really solved it while I was using memcached but I'm not Wikipedia
    - andrewingram 11y ago
      The way i've seen it done is something like this: * Cache never automatically expires, but you do have some notion of staleness * Whenever you request data, you get it from the cache (if it exists), and check for staleness, so you cached data needs to know when it was cached. * Return the data as usual, but if the data was also stale, you fire off a worker to update the cached data. * If you have lots of requests happening at the same time, you have a system for seeing if a worker already exists, to ensure that you only create one (for each piece of cached data). * For the time it takes for the worker to complete, you have to be okay with serving stale data, in most cases this is okay. There's an edge case missed here, which is what to do when the cache is empty (either because it's one of the first requests, or because the cached data has been evicted). That's up to you, depending on your use case. You can basically either return a default value, you can pre-warm your cache, or you can let the requests hang until the data is ready.