4 ms·

Handling five billion sessions a day in real time

98 points by knes 12y ago

domoarevil 12y ago
twitter: @copypastaa and then appending ?hn is at least consistent :)
buster 12y ago
What i never see is how much servers/hardware/resources they are spending. It's _always_ like "we process a quadrillion messages per second" but it never mentions that you'll need a quadrillion expensive servers to do so. Same for many many database benchmarks, sadly...
- frostmatthew 12y ago
  It's a couple years old but What it takes to run Stack Overflow[1] lists the number of DB servers, Web servers, etc [1] http://nickcraver.com/blog/2013/11/22/what-it-takes-to-run-stack-overflow/ http://nickcraver.com/blog/2013/11/22/what-it-takes-to-run-s...
  - jackweirdy 12y ago
    This update is more recent: http://highscalability.com/blog/2014/7/21/stackoverflow-update-560m-pageviews-a-month-25-servers-and-i.html http://highscalability.com/blog/2014/7/21/stackoverflow-upda...
- jared314 12y ago
  StackExchange is pretty open with their hardware metrics. The StackExchange performance site: https://stackexchange.com/performance https://stackexchange.com/performance
  - Someone1234 12y ago
    > RAM: 48GB Per server or in total..? Either one is believable.
    - zuck9 12y ago
      I don't think a single server can do its task in just 5 GiB (48 GiB/9)
    - Kurtz79 12y ago
      It must be per server, adding RAM is really the quickest and cheapest way to improve performance.
    - Moter8 12y ago
      See the tagline "statistics by server" Image: http://i.imgur.com/q1oDamy.png http://i.imgur.com/q1oDamy.png
joosters 12y ago
"How we handle spying on your mobile phone activity on a global scale"
- bpg_92 12y ago
  hahaha but it is a good architecture you have to admint :D
toblender 12y ago
First thing I noticed was Amazon in there Stack. Amazon really owns the internet...
- philippnagel 12y ago
  In what way does Amazon own the Internet? Yes, they own some of the computers that make up the Internet, but by far not all of them.
  - metaphorm 12y ago
    they are the dominant vendor of hardware/hosting to the largest and most complex web applications on the internet. AWS is the pillar that some of the most high volume sites on the internet (like Twitter and Netflix) rely on. Its safe to say that without AWS (or some other similar vendor) the internet as we know it wouldn't exist.
    - latch 12y ago
      That's ridiculous. It's like you're saying: without Honda, the car industry, as we know it, wouldn't exist. The hosting industry is huge. There's a ton options, many of them better, larger and older than AWS. There are so many sites that don't use AWS. For example, I believe that, pre-Google, youtube used ServerBeach.
      - sanderjd 12y ago
        > There's a ton options, many of them better, larger and older than AWS. Some examples would be more interesting than a general "you're wrong".
        latch 12y ago
        For every site that's reliant on AWS, there's probably more than 10 000 that aren't. The giants, for example: Google. Facebook. Wikipedia. Microsoft. It's hard to get actual numbers, but Equinix might be the largest. I believe their Singapore data center is actually what AWS, Digital Ocean and Azure use. (there's a lot more of these, such as Internap and Level 3 (which provided (and might still) services for Netflix and iTunes.)) AT&T is quite large. World of Warcraft used to be hosted on AT&T's infrastructure (might still be). The above tend to be for large setups. Going smaller, you have the Rackspaces and Softlayers (now IBM). And, to a lesser extent, providers like OVH and Hetzner. And, you can't forget the likes of Digital Ocean and Linode. It's hard to come up with examples because more things are examples than aren't. AWS' market share is growing, but that's to be expected, in a lot of ways, despite EC2 being ~10 years old , it's still a new player. AFAIC, what's more interesting about AWS is: 1 - How dominating it is over Microsoft and Google 2 - How popular the less invasive offerings are. A lot of people are sticking to their colocated, dedicated or VPSs, while using some part of AWS for DNS, backups or email. I hope they keep this up.
meesterdude 12y ago
> What happens if our speed (real-time) processing layer goes down? Our on-call engineers will get paged and address the problem. Innovative! But for real, that's some architecture, and very insightful to see what they've got. Totally nothing I'd have guessed.
MrBuddyCasino 12y ago
I had a hunch I'd see the data structures du jour in there: Bloom Filters and HyperLogLog. Since they are pretty accurate by themselves already but run much faster than the brute force approach, why keep the latter? The error margin, if used properly, should be small enough so people shouldn't really care.
- ropiku 12y ago
  They are not for accuracy. One usage for instance is to recompute stats if there are bugs in the stats processing code. Or if you want to add more metrics. (I work at Twitter but not on Answers).

98 points by knes 12y ago

twitter: @copypastaa and then appending ?hn is at least consistent :)

What i never see is how much servers/hardware/resources they are spending. It's _always_ like "we process a quadrillion messages per second" but it never mentions that you'll need a quadrillion expensive servers to do so. Same for many many database benchmarks, sadly...

frostmatthew 12y ago
It's a couple years old but What it takes to run Stack Overflow[1] lists the number of DB servers, Web servers, etc [1] http://nickcraver.com/blog/2013/11/22/what-it-takes-to-run-stack-overflow/ http://nickcraver.com/blog/2013/11/22/what-it-takes-to-run-s...
- jackweirdy 12y ago
  This update is more recent: http://highscalability.com/blog/2014/7/21/stackoverflow-update-560m-pageviews-a-month-25-servers-and-i.html http://highscalability.com/blog/2014/7/21/stackoverflow-upda...
jared314 12y ago
StackExchange is pretty open with their hardware metrics. The StackExchange performance site: https://stackexchange.com/performance https://stackexchange.com/performance
- Someone1234 12y ago
  > RAM: 48GB Per server or in total..? Either one is believable.
  - zuck9 12y ago
    I don't think a single server can do its task in just 5 GiB (48 GiB/9)
  - Kurtz79 12y ago
    It must be per server, adding RAM is really the quickest and cheapest way to improve performance.
  - Moter8 12y ago
    See the tagline "statistics by server" Image: http://i.imgur.com/q1oDamy.png http://i.imgur.com/q1oDamy.png

joosters 12y ago

"How we handle spying on your mobile phone activity on a global scale"

bpg_92 12y ago
hahaha but it is a good architecture you have to admint :D

toblender 12y ago

First thing I noticed was Amazon in there Stack. Amazon really owns the internet...

philippnagel 12y ago
In what way does Amazon own the Internet? Yes, they own some of the computers that make up the Internet, but by far not all of them.
- metaphorm 12y ago
  they are the dominant vendor of hardware/hosting to the largest and most complex web applications on the internet. AWS is the pillar that some of the most high volume sites on the internet (like Twitter and Netflix) rely on. Its safe to say that without AWS (or some other similar vendor) the internet as we know it wouldn't exist.
  - latch 12y ago
    That's ridiculous. It's like you're saying: without Honda, the car industry, as we know it, wouldn't exist. The hosting industry is huge. There's a ton options, many of them better, larger and older than AWS. There are so many sites that don't use AWS. For example, I believe that, pre-Google, youtube used ServerBeach.
    - sanderjd 12y ago
      > There's a ton options, many of them better, larger and older than AWS. Some examples would be more interesting than a general "you're wrong".
      - latch 12y ago
        For every site that's reliant on AWS, there's probably more than 10 000 that aren't. The giants, for example: Google. Facebook. Wikipedia. Microsoft. It's hard to get actual numbers, but Equinix might be the largest. I believe their Singapore data center is actually what AWS, Digital Ocean and Azure use. (there's a lot more of these, such as Internap and Level 3 (which provided (and might still) services for Netflix and iTunes.)) AT&T is quite large. World of Warcraft used to be hosted on AT&T's infrastructure (might still be). The above tend to be for large setups. Going smaller, you have the Rackspaces and Softlayers (now IBM). And, to a lesser extent, providers like OVH and Hetzner. And, you can't forget the likes of Digital Ocean and Linode. It's hard to come up with examples because more things are examples than aren't. AWS' market share is growing, but that's to be expected, in a lot of ways, despite EC2 being ~10 years old , it's still a new player. AFAIC, what's more interesting about AWS is: 1 - How dominating it is over Microsoft and Google 2 - How popular the less invasive offerings are. A lot of people are sticking to their colocated, dedicated or VPSs, while using some part of AWS for DNS, backups or email. I hope they keep this up.

meesterdude 12y ago

> What happens if our speed (real-time) processing layer goes down? Our on-call engineers will get paged and address the problem. Innovative! But for real, that's some architecture, and very insightful to see what they've got. Totally nothing I'd have guessed.

MrBuddyCasino 12y ago

I had a hunch I'd see the data structures du jour in there: Bloom Filters and HyperLogLog. Since they are pretty accurate by themselves already but run much faster than the brute force approach, why keep the latter? The error margin, if used properly, should be small enough so people shouldn't really care.

ropiku 12y ago
They are not for accuracy. One usage for instance is to recompute stats if there are bugs in the stats processing code. Or if you want to add more metrics. (I work at Twitter but not on Answers).