3 ms·

One thing to keep in mind that their work-load is very ready-heavy which eases things a lot when scaling the system. The same is true for Wikipedia. Scaling a w

One thing to keep in mind that their work-load is very ready-heavy which eases things a lot when scaling the system. The same is true for Wikipedia. Scaling a write-heavy workload is way more complex than scaling a read-heavy workload.

hliyan 7y ago
There is a significant amount of question submission, commenting and voting going on on SO.
- yawn 7y ago
  A slim subset of humans typing things is not what I would label "write heavy". Write heavy is more like 100k+ devices out in the field sending their current position every 10s. That's still very manageable, but requires some thoughtful design.
  - Retric 7y ago
    10k simple updates per second is hardly high bandwidth. I knew people handling similar workloads on a 1U server 10 years ago.
    - pushpop 7y ago
      Indeed and I literally was. And the back end was written in mod_perl (which isn’t exactly known for performance). Even back then 10k/s was our smallest service - both on income as well as traffic - so there wasn’t the incentive to rewrite it in a faster language.
  - stefan_ 7y ago
    The Raspberry Pi 3 in my cupboard without a heat sink on can do 80000 Redis write ops/sec.
manigandham 7y ago
SO content changes all the time. Votes, comments, moderation, edits, tagging, search and recommendations, etc. There are also real-time community features. It's not as simple as it seems.
- RhodesianHunter 7y ago
  They said read heavy, not read only. I guarantee you the bulk of traffic to SO is hitting a page and performing zero writes.
  - manigandham 7y ago
    They log every single pageview to SQL Server. There are plenty of writes to various other counters, recommendation system, message inboxes, analytics, etc. Also the ads system, although I'm not sure how much of that is still 3rd-party. Yes it's read-heavy but there's still plenty of work done in assembling a page. It's definitely not as simple as caching at the CDN edge for every hit.
    - dvirsky 7y ago
      IIRC most of the "real time" stuff is done on Redis, for which this QPS load is usually a joke (depending on requests of course, but counters etc are easy)
  - fock 7y ago
    so, would you mind showing any non-FANG, but write heavy site? Besides that, even the operation of most social networks should be very easy to be split into town-sized instances with batch-updates in-between instances (see mastodon specs https://github.com/tootsuite/documentation/blob/master/Running-Mastodon/Resources-needed.md https://github.com/tootsuite/documentation/blob/master/Runni...). What actually costs a lot is constantly surveilling all user interactions, data-warehousing that and playing out ads. None of which is necessary, and a 20ct/per user/per month should cover the actual service nicely...
    - 7y ago

hliyan 7y ago

There is a significant amount of question submission, commenting and voting going on on SO.

yawn 7y ago
A slim subset of humans typing things is not what I would label "write heavy". Write heavy is more like 100k+ devices out in the field sending their current position every 10s. That's still very manageable, but requires some thoughtful design.
- Retric 7y ago
  10k simple updates per second is hardly high bandwidth. I knew people handling similar workloads on a 1U server 10 years ago.
  - pushpop 7y ago
    Indeed and I literally was. And the back end was written in mod_perl (which isn’t exactly known for performance). Even back then 10k/s was our smallest service - both on income as well as traffic - so there wasn’t the incentive to rewrite it in a faster language.
- stefan_ 7y ago
  The Raspberry Pi 3 in my cupboard without a heat sink on can do 80000 Redis write ops/sec.

manigandham 7y ago

SO content changes all the time. Votes, comments, moderation, edits, tagging, search and recommendations, etc. There are also real-time community features. It's not as simple as it seems.

RhodesianHunter 7y ago
They said read heavy, not read only. I guarantee you the bulk of traffic to SO is hitting a page and performing zero writes.
- manigandham 7y ago
  They log every single pageview to SQL Server. There are plenty of writes to various other counters, recommendation system, message inboxes, analytics, etc. Also the ads system, although I'm not sure how much of that is still 3rd-party. Yes it's read-heavy but there's still plenty of work done in assembling a page. It's definitely not as simple as caching at the CDN edge for every hit.
  - dvirsky 7y ago
    IIRC most of the "real time" stuff is done on Redis, for which this QPS load is usually a joke (depending on requests of course, but counters etc are easy)
- fock 7y ago
  so, would you mind showing any non-FANG, but write heavy site? Besides that, even the operation of most social networks should be very easy to be split into town-sized instances with batch-updates in-between instances (see mastodon specs https://github.com/tootsuite/documentation/blob/master/Running-Mastodon/Resources-needed.md https://github.com/tootsuite/documentation/blob/master/Runni...). What actually costs a lot is constantly surveilling all user interactions, data-warehousing that and playing out ads. None of which is necessary, and a 20ct/per user/per month should cover the actual service nicely...
  - 7y ago