7 ms·

Wikimedia's infrastructure is radically different than most FAANG. In large part because 99% (+/-) of their traffic is read only. While Facebook and Google hav

Wikimedia's infrastructure is radically different than most FAANG. In large part because 99% (+/-) of their traffic is read only. While Facebook and Google have to do heavy workloads for every click and action taken on their services, Wikimedia can cache basically everything. Allowing them to operate on a tiny fraction of the number of machines (and infrastructure) that the rest of the players do.

Retric 6y ago
You can do quite a bit of processing per page load without issue. Facebook and Google just take it rather past that point into near absurdity, while still being highly profitable.
- ianlevesque 6y ago
  Yeah why do they keep spending billions to build new datacenters when they could just stop being absurd instead? The contempt on here is crazy sometimes.
  - civilized 6y ago
    Or maybe you're just reading too much into "absurd" which can just be a colorful word for "an extremely huge amount"
  - MereInterest 6y ago
    I don't think that Facebook/Google developers are foolish or incompetent. That would be contempt. Instead, I think that Facebook and Google as conglomerate entities are fundamentally opposed to my right to privacy. That they make decisions to rationally follow their self-interest does not excuse the absurd lengths to which they go to stalk the general population's activities.
    - ryanianian 6y ago
      > I don't think that Facebook/Google developers are foolish or incompetent. Nobody in this thread is saying that. Parent to you said: > they could just stop being absurd instead [of building more DCs] implying FB could build fewer DCs by scaling down some of their per-page complexity/"absurdity". Basically saying their needs are artificial or borne of requirements that aren't. > conglomerate entities are fundamentally opposed to my right to privacy That's a common view, but it's not on topic to this thread. This thread is mostly about the tech itself and how WikiMedia scales versus how the bigger techs scale. It has an interesting diversion into some of the reasons why their scaling needs are different. You could instead continue the thread stating that they could save a lot of money and complexity while also tearing down some of their reputation for being slow and privacy-hostile by removing some of the very features these DCs support (perhaps) without ruining the net bottom line. This continues the thread and allows the conversation to continue to what the ROI actually is on the sort of complexity that benefits the company but not the user.
      - Retric 6y ago
        I was the one saying absurdity and I think you’re missing the context. Work out how much processing power is worth even just another 1 cent per thousand page loads and perfectly rational behavior starts to look crazy to the little guys. Let’s suppose the Facebook cluster spends the equivalent of 1 full second of 1 full CPU core per request. That’s a lot of processing power and for most small scale architectures likely adding wildly unacceptable latency per page load. Further, as small scale traffic is very spiky even low traffic sites would be expensive to host making it a ludicrous amount of processing power. However, Google has enough traffic to smooth things out, it’s splitting that across multiple of computers and much of it is after the request so latency isn’t an issue, and it isn’t paying retail so processing power is little more than just hardware costs and electricity. Estimate the rough order of magnitude their paying for 1 second of 1 core per request and it’s cheap enough to be a rounding error.
  - bo1024 6y ago
    The idea of marginal value/marginal cost is that companies will generally continue spending one billion dollars to add size and complexity, as long as they get back a bit more than a billion dollars in revenue. So it wouldn't necessarily be contradictory if most of their core functionality could be replicated very simply, yet the actual product is immensely complicated. I forget where I first read this point, but probably on HN.
- throwaway3699 6y ago
  To be fair, there's a bit of a combinatoric effect of scale * features going on there. I'm sure you could build most of a Facebook equiv. 100x-1000x cheaper if it only served one city instead of the whole planet.
  - klodolph 6y ago
    The effects of scale are less combinatoric than you might think. Most people on my Facebook feed are from the same city anyway, even though Facebook is global.
    - erichurkman 6y ago
      The effects and scale of sales (ads) are very combinatoric, though.
- cmckn 6y ago
  Every request at FB is handled in a new container. This isn’t absurd, it’s actually pretty neat :) Edit: I don’t know what I’m talking about. Happy Monday!
  - robmurrer 6y ago
    is not neat... is freakish
  - ROARosen 6y ago
    Wow that sounds interesting, does anyone know if this is true?
    - wilsonthewhale 6y ago
      I'm not on the team that handles this, but I highly doubt that this is the case.
  - rachelbythebay 6y ago
    What? Are you calling the context of a HHVM request a container just to confuse people? Also, there's way more than just the web tier out there.
    - cmckn 6y ago
      Wasn’t my intention to confuse, just repeating something I’ve been told by FB folks. Everyone, please listen to Rachel and never ever me.
ashtonkem 6y ago
They also have looser latency SLAs. The only hard requirement is that a user can read back their own writes, but it’s okay if other users are served stale data for a few seconds or minutes even. This makes cache invalidation, one of the most notoriously difficult and expensive operations at large scale, much much easier.
- nostrademons 6y ago
  Facebook also has a similar SLA. I've heard that at one point in their architecture (~2010), they literally stored the user's own writes in memcached and then merged them back into the page when rendered. You would see a page consistent with your actions, but if you logged into Facebook as any of your friends your updates might not show up until replication lag passed.
  - mackman 6y ago
    Close, IIRC we cached the fact you had just done a write, and a subsequent read request that arrived on the replica region was then proxied to the primary region instead of serviced locally.
    - IgorPartola 6y ago
      Pretty clever. Is that still how it works?
      - glittershark 6y ago
        Pretty sure this paper describes what they're doing now: https://research.fb.com/publications/flighttracker-consistency-across-read-optimized-online-stores-at-facebook/ https://research.fb.com/publications/flighttracker-consisten...
        mackman 6y ago
        I'm not sure if FlightTracker completely replaced the need for the internal consistency inside Tao. You can read about that here: https://www.usenix.org/system/files/conference/atc13/atc13-bronson.pdf https://www.usenix.org/system/files/conference/atc13/atc13-b...
nostrademons 6y ago
Search is also largely read-only. The advantage Wikipedia has is that its traffic overwhelming goes to the head of the page distribution, so simple caching solutions work very well. Google has a pretty extreme long-tail distribution (~15% of daily queries have never been seen before), and so needs to do a lot of computation per query.
- zelon88 6y ago
  > Google has a pretty extreme long-tail distribution (~15% of daily queries have never been seen before) Do you have a source for this? I'd be willing to bet that the ONLY reason why 15% of their daily queries "haven't been seen before" is because they add un-needed complexity like fingerprinting. You're making it seem like they've never seen a query for "cute animals" before when obviously they have. They choose to do a lot of extra leg work because of who you are. So your claim that 15% of their queries have "never been seen before" is probably inaccurate. I'd be willing to bet that "15% of their queries are unique because of the user, location, or other external factor separate from the query itself." They've seen your query before. They've just never seen you make this query from this device on this side of town before.
  - tylerhou 6y ago
    If you took into account user, location, etc. 15% seems too low. I almost never search for the exact same thing twice in the same location. 15% of the queries themselves are unique. https://blog.google/products/search/our-latest-quality-improvements-search/ https://blog.google/products/search/our-latest-quality-impro... https://www.google.com/search/howsearchworks/responses/ https://www.google.com/search/howsearchworks/responses/ I work for Google (and used to work on Search).
    - ma2rten 6y ago
      I think you mean "15% seems too high". Any easy way to think about this is the following: even if search the entire internet you will almost never see the same sentence twice, assuming it's has a certain number of words. There is a combinatorial explosion in possible sentences to write. Search queries are essentially just sentences without stopwords.
anang 6y ago
Wikimedia also has less incentive/drive to meticulously track every interaction on their pages. The level of tracking present on Facebook and Google has to be extremely computationally intensive.
_joe 6y ago
This is indeed correct. Wikimedia overall uses less than 2000 bare-metal servers, so yes the infrastructure is tiny compared to those. What can be interesting, I think, is that you have a completely open infrastructure that has to solve problems on a global traffic scale. If people are interested in knowing more, I suggest you also take a peek at the wikimedia techblog, specifically to the SRE category https://techblog.wikimedia.org/category/site-reliability-engineering/ https://techblog.wikimedia.org/category/site-reliability-eng... and the performance one https://techblog.wikimedia.org/category/performance/ https://techblog.wikimedia.org/category/performance/
dalbasal 6y ago
I agree. Another (no contradiction) way of looking at this is that Wikimedia infrastructure is radically different because Wikimedia is radically different. They need it to be a certain way in order to operate. The limitations and advantages of how software gets made. Why it gets made. The way the software works. How and why product decisions were made over the last 2 decades. What resources they have/had available. It's all a totally different game. Not surprising that different soil and a different climate grow different plants. One of Google's early coup d'etats, when they were a strategic step ahead of the boomers, was bankrolling gmail, youtube and such. Gmail offered free giant inboxes. They got all the customers. This cost billions (maybe 100s of millions), but storage costs go down every year while the value of ads/data/lock-in and such go up every year. Similar logic for youtube. (1) Buy a leading video-sharing site; (2)bankroll HD streaming because you have the deepest pockets (3) Own online free TV entirely. That's who Google is, good or bad. How funding works. What products get built. What infrastructure is necessary, possible, affordable. All interlinked. Wikipedia & Google were founded at the same time. Within 5 years (circa 2006) Google was buying charters and fiefdoms. Wikimedia, meanwhile, was starting to take flak for raising 3 or 4 million in donations. It's kinda crazy that Wikipedia is comparable in scale to FAANGs when you consider these disparities.
MaxBarraclough 6y ago
Does it hurt their caching if you browse Wikipedia when signed in? I recall reading HackerNews used to have that problem, unsure if it still does.
- tim333 6y ago
  Looking at the source of a Wikipedia page it has my username appearing 6 times so I guess it must reduce caching a bit. Though I guess they could cache the user info bits and the rest of the page and just splice them together.

Retric 6y ago

You can do quite a bit of processing per page load without issue. Facebook and Google just take it rather past that point into near absurdity, while still being highly profitable.

ianlevesque 6y ago
Yeah why do they keep spending billions to build new datacenters when they could just stop being absurd instead? The contempt on here is crazy sometimes.
- civilized 6y ago
  Or maybe you're just reading too much into "absurd" which can just be a colorful word for "an extremely huge amount"
- MereInterest 6y ago
  I don't think that Facebook/Google developers are foolish or incompetent. That would be contempt. Instead, I think that Facebook and Google as conglomerate entities are fundamentally opposed to my right to privacy. That they make decisions to rationally follow their self-interest does not excuse the absurd lengths to which they go to stalk the general population's activities.
  - ryanianian 6y ago
    > I don't think that Facebook/Google developers are foolish or incompetent. Nobody in this thread is saying that. Parent to you said: > they could just stop being absurd instead [of building more DCs] implying FB could build fewer DCs by scaling down some of their per-page complexity/"absurdity". Basically saying their needs are artificial or borne of requirements that aren't. > conglomerate entities are fundamentally opposed to my right to privacy That's a common view, but it's not on topic to this thread. This thread is mostly about the tech itself and how WikiMedia scales versus how the bigger techs scale. It has an interesting diversion into some of the reasons why their scaling needs are different. You could instead continue the thread stating that they could save a lot of money and complexity while also tearing down some of their reputation for being slow and privacy-hostile by removing some of the very features these DCs support (perhaps) without ruining the net bottom line. This continues the thread and allows the conversation to continue to what the ROI actually is on the sort of complexity that benefits the company but not the user.
    - Retric 6y ago
      I was the one saying absurdity and I think you’re missing the context. Work out how much processing power is worth even just another 1 cent per thousand page loads and perfectly rational behavior starts to look crazy to the little guys. Let’s suppose the Facebook cluster spends the equivalent of 1 full second of 1 full CPU core per request. That’s a lot of processing power and for most small scale architectures likely adding wildly unacceptable latency per page load. Further, as small scale traffic is very spiky even low traffic sites would be expensive to host making it a ludicrous amount of processing power. However, Google has enough traffic to smooth things out, it’s splitting that across multiple of computers and much of it is after the request so latency isn’t an issue, and it isn’t paying retail so processing power is little more than just hardware costs and electricity. Estimate the rough order of magnitude their paying for 1 second of 1 core per request and it’s cheap enough to be a rounding error.
- bo1024 6y ago
  The idea of marginal value/marginal cost is that companies will generally continue spending one billion dollars to add size and complexity, as long as they get back a bit more than a billion dollars in revenue. So it wouldn't necessarily be contradictory if most of their core functionality could be replicated very simply, yet the actual product is immensely complicated. I forget where I first read this point, but probably on HN.
throwaway3699 6y ago
To be fair, there's a bit of a combinatoric effect of scale * features going on there. I'm sure you could build most of a Facebook equiv. 100x-1000x cheaper if it only served one city instead of the whole planet.
- klodolph 6y ago
  The effects of scale are less combinatoric than you might think. Most people on my Facebook feed are from the same city anyway, even though Facebook is global.
  - erichurkman 6y ago
    The effects and scale of sales (ads) are very combinatoric, though.
cmckn 6y ago
Every request at FB is handled in a new container. This isn’t absurd, it’s actually pretty neat :) Edit: I don’t know what I’m talking about. Happy Monday!
- robmurrer 6y ago
  is not neat... is freakish
- ROARosen 6y ago
  Wow that sounds interesting, does anyone know if this is true?
  - wilsonthewhale 6y ago
    I'm not on the team that handles this, but I highly doubt that this is the case.
- rachelbythebay 6y ago
  What? Are you calling the context of a HHVM request a container just to confuse people? Also, there's way more than just the web tier out there.
  - cmckn 6y ago
    Wasn’t my intention to confuse, just repeating something I’ve been told by FB folks. Everyone, please listen to Rachel and never ever me.

ashtonkem 6y ago

They also have looser latency SLAs. The only hard requirement is that a user can read back their own writes, but it’s okay if other users are served stale data for a few seconds or minutes even. This makes cache invalidation, one of the most notoriously difficult and expensive operations at large scale, much much easier.

nostrademons 6y ago
Facebook also has a similar SLA. I've heard that at one point in their architecture (~2010), they literally stored the user's own writes in memcached and then merged them back into the page when rendered. You would see a page consistent with your actions, but if you logged into Facebook as any of your friends your updates might not show up until replication lag passed.
- mackman 6y ago
  Close, IIRC we cached the fact you had just done a write, and a subsequent read request that arrived on the replica region was then proxied to the primary region instead of serviced locally.
  - IgorPartola 6y ago
    Pretty clever. Is that still how it works?
    - glittershark 6y ago
      Pretty sure this paper describes what they're doing now: https://research.fb.com/publications/flighttracker-consistency-across-read-optimized-online-stores-at-facebook/ https://research.fb.com/publications/flighttracker-consisten...
      - mackman 6y ago
        I'm not sure if FlightTracker completely replaced the need for the internal consistency inside Tao. You can read about that here: https://www.usenix.org/system/files/conference/atc13/atc13-bronson.pdf https://www.usenix.org/system/files/conference/atc13/atc13-b...

nostrademons 6y ago

Search is also largely read-only. The advantage Wikipedia has is that its traffic overwhelming goes to the head of the page distribution, so simple caching solutions work very well. Google has a pretty extreme long-tail distribution (~15% of daily queries have never been seen before), and so needs to do a lot of computation per query.

zelon88 6y ago
> Google has a pretty extreme long-tail distribution (~15% of daily queries have never been seen before) Do you have a source for this? I'd be willing to bet that the ONLY reason why 15% of their daily queries "haven't been seen before" is because they add un-needed complexity like fingerprinting. You're making it seem like they've never seen a query for "cute animals" before when obviously they have. They choose to do a lot of extra leg work because of who you are. So your claim that 15% of their queries have "never been seen before" is probably inaccurate. I'd be willing to bet that "15% of their queries are unique because of the user, location, or other external factor separate from the query itself." They've seen your query before. They've just never seen you make this query from this device on this side of town before.
- tylerhou 6y ago
  If you took into account user, location, etc. 15% seems too low. I almost never search for the exact same thing twice in the same location. 15% of the queries themselves are unique. https://blog.google/products/search/our-latest-quality-improvements-search/ https://blog.google/products/search/our-latest-quality-impro... https://www.google.com/search/howsearchworks/responses/ https://www.google.com/search/howsearchworks/responses/ I work for Google (and used to work on Search).
  - ma2rten 6y ago
    I think you mean "15% seems too high". Any easy way to think about this is the following: even if search the entire internet you will almost never see the same sentence twice, assuming it's has a certain number of words. There is a combinatorial explosion in possible sentences to write. Search queries are essentially just sentences without stopwords.

anang 6y ago

Wikimedia also has less incentive/drive to meticulously track every interaction on their pages. The level of tracking present on Facebook and Google has to be extremely computationally intensive.

_joe 6y ago

This is indeed correct. Wikimedia overall uses less than 2000 bare-metal servers, so yes the infrastructure is tiny compared to those. What can be interesting, I think, is that you have a completely open infrastructure that has to solve problems on a global traffic scale. If people are interested in knowing more, I suggest you also take a peek at the wikimedia techblog, specifically to the SRE category https://techblog.wikimedia.org/category/site-reliability-engineering/ https://techblog.wikimedia.org/category/site-reliability-eng... and the performance one https://techblog.wikimedia.org/category/performance/ https://techblog.wikimedia.org/category/performance/

dalbasal 6y ago

I agree. Another (no contradiction) way of looking at this is that Wikimedia infrastructure is radically different because Wikimedia is radically different. They need it to be a certain way in order to operate. The limitations and advantages of how software gets made. Why it gets made. The way the software works. How and why product decisions were made over the last 2 decades. What resources they have/had available. It's all a totally different game. Not surprising that different soil and a different climate grow different plants. One of Google's early coup d'etats, when they were a strategic step ahead of the boomers, was bankrolling gmail, youtube and such. Gmail offered free giant inboxes. They got all the customers. This cost billions (maybe 100s of millions), but storage costs go down every year while the value of ads/data/lock-in and such go up every year. Similar logic for youtube. (1) Buy a leading video-sharing site; (2)bankroll HD streaming because you have the deepest pockets (3) Own online free TV entirely. That's who Google is, good or bad. How funding works. What products get built. What infrastructure is necessary, possible, affordable. All interlinked. Wikipedia & Google were founded at the same time. Within 5 years (circa 2006) Google was buying charters and fiefdoms. Wikimedia, meanwhile, was starting to take flak for raising 3 or 4 million in donations. It's kinda crazy that Wikipedia is comparable in scale to FAANGs when you consider these disparities.

MaxBarraclough 6y ago

Does it hurt their caching if you browse Wikipedia when signed in? I recall reading HackerNews used to have that problem, unsure if it still does.

tim333 6y ago
Looking at the source of a Wikipedia page it has my username appearing 6 times so I guess it must reduce caching a bit. Though I guess they could cache the user info bits and the rest of the page and just splice them together.