5 ms·

I love reading about how companies scale their BigHuge data but it bothers me that we still haven't reached the point where scalability is a commodity instead o

I love reading about how companies scale their BigHuge data but it bothers me that we still haven't reached the point where scalability is a commodity instead of a patchwork of technology that everyone actor solves in their own way.

joering2 14y ago
That's because 99% of internet websites will never see 200 million requests.
- jeremyjh 14y ago
  Much fewer than 1% will.
NelsonMinar 14y ago
Scalability can only be a commodity if all big sites were built the same way. But they never are. A scalable read-only system is entirely different from a system with a mix of read/write which is different again from a message routing system like Twitter or Facebook.
- batista 14y ago
  That doesn't mean they cant take advantage of ready-made "functional architectural components" for each of their needs. So they can pick and match from the "scalable read only" commodity system, the scalable "message routing system" etc.
nostrademons 14y ago
It's because "how to make a website scale" depends heavily upon which website, what it does, and how big it needs to be. Making a messaging queue like Twitter or G+ scale is very different from making Google Search scale. Hell, making the indexing system of Google search scale is a very different problem from making the serving system scale. You can't really avoid having a patchwork of technology, because it's a patchwork of problems. Instead, there're a bunch of tools at your disposal, a few "best practices" which are highly contextual, and you have to use your judgment and knowledge of the problem domain to put them together.
- batista 14y ago
  >It's because "how to make a website scale" depends heavily upon which website, what it does, and how big it needs to be. Making a messaging queue like Twitter or G+ scale is very different from making Google Search scale. Hell, making the indexing system of Google search scale is a very different problem from making the serving system scale. For most websites is not THAT different. Actually, most have pretty similar needs, and you can sum those up in 3-5 different website architectural styles anyway. There is far more duplication of work and ad-hoc solutions to the SAME problems than are "heavily different" needs.
  - nostrademons 14y ago
    What would be those 3-5 different website architectural styles?
    - batista 14y ago
      News/Magazine/Portal like (read heavy), Game site (evented, concurrent users, game engine computations), Social Platform (read-write heavy), etc. Most needs are bog standard. If you really look at most successful sites they use might same-ish architectures, only with different components/languages/libs each. Basically all high volume sites use something like the notions behind the Google App Engine, and the services offered. The various AWS tools are also similar (S3, the table they offer, etc).
      - nostrademons 14y ago
        I think you're missing a lot of complexity of the considerations that actually go into implementing any of the above. I can think of 3 subsystems within Reddit alone (reading, voting, and messages) that all have different usage patterns and (if they're doing it right) require different approaches to scaling. Where's e-commerce on your list? The approaches for scaling eBay are completely different than for scaling Reddit or YouTube, because eBay can't afford eventual consistency. You can't rely on caching to show a buyer a page whose price is an hour out-of-date. Here's something else to think about: why do (the now-defunct) Google real-time search and GMail Chat have completely different architectures, despite both of them having the same basic structure of "a message comes in, and gets displayed in a scrolling window on the screen"? The answer is latency. With real-time search, a latency of 30 seconds is acceptable, since you aren't going to know when the tweet was posted in the first place. With GChat, it has to be immediate, because it's frequently used in the context of someone verbally saying "I'll ping you this link" and it's kinda embarrassing if the link doesn't arrive for 30 seconds. Real-time search also has to run much more computationally-intensive algorithms to determine relevance & ranking than GChat does. I've personally worked on Google Search, Google+, and Google Fiber. I can tell you that they do not all use something like the notions behind Google AppEngine. There's no way you could build Google Search on AppEngine, and G+ would be a stretch.
ksec 14y ago
Well because every site are made different, and uses different tech, with different bottleneck. But with SSD, RAM becoming a commodity, ( both prices has been dropping shapely ) I/O are much easier to deal with. And with every release of PostgreSQL / MySQL, Easier Database Replication and more common practice of scaling we are much much better at it then say 2 - 3 years ago.

joering2 14y ago

That's because 99% of internet websites will never see 200 million requests.

jeremyjh 14y ago
Much fewer than 1% will.

NelsonMinar 14y ago

Scalability can only be a commodity if all big sites were built the same way. But they never are. A scalable read-only system is entirely different from a system with a mix of read/write which is different again from a message routing system like Twitter or Facebook.

batista 14y ago
That doesn't mean they cant take advantage of ready-made "functional architectural components" for each of their needs. So they can pick and match from the "scalable read only" commodity system, the scalable "message routing system" etc.

nostrademons 14y ago

It's because "how to make a website scale" depends heavily upon which website, what it does, and how big it needs to be. Making a messaging queue like Twitter or G+ scale is very different from making Google Search scale. Hell, making the indexing system of Google search scale is a very different problem from making the serving system scale. You can't really avoid having a patchwork of technology, because it's a patchwork of problems. Instead, there're a bunch of tools at your disposal, a few "best practices" which are highly contextual, and you have to use your judgment and knowledge of the problem domain to put them together.

batista 14y ago
>It's because "how to make a website scale" depends heavily upon which website, what it does, and how big it needs to be. Making a messaging queue like Twitter or G+ scale is very different from making Google Search scale. Hell, making the indexing system of Google search scale is a very different problem from making the serving system scale. For most websites is not THAT different. Actually, most have pretty similar needs, and you can sum those up in 3-5 different website architectural styles anyway. There is far more duplication of work and ad-hoc solutions to the SAME problems than are "heavily different" needs.
- nostrademons 14y ago
  What would be those 3-5 different website architectural styles?
  - batista 14y ago
    News/Magazine/Portal like (read heavy), Game site (evented, concurrent users, game engine computations), Social Platform (read-write heavy), etc. Most needs are bog standard. If you really look at most successful sites they use might same-ish architectures, only with different components/languages/libs each. Basically all high volume sites use something like the notions behind the Google App Engine, and the services offered. The various AWS tools are also similar (S3, the table they offer, etc).
    - nostrademons 14y ago
      I think you're missing a lot of complexity of the considerations that actually go into implementing any of the above. I can think of 3 subsystems within Reddit alone (reading, voting, and messages) that all have different usage patterns and (if they're doing it right) require different approaches to scaling. Where's e-commerce on your list? The approaches for scaling eBay are completely different than for scaling Reddit or YouTube, because eBay can't afford eventual consistency. You can't rely on caching to show a buyer a page whose price is an hour out-of-date. Here's something else to think about: why do (the now-defunct) Google real-time search and GMail Chat have completely different architectures, despite both of them having the same basic structure of "a message comes in, and gets displayed in a scrolling window on the screen"? The answer is latency. With real-time search, a latency of 30 seconds is acceptable, since you aren't going to know when the tweet was posted in the first place. With GChat, it has to be immediate, because it's frequently used in the context of someone verbally saying "I'll ping you this link" and it's kinda embarrassing if the link doesn't arrive for 30 seconds. Real-time search also has to run much more computationally-intensive algorithms to determine relevance & ranking than GChat does. I've personally worked on Google Search, Google+, and Google Fiber. I can tell you that they do not all use something like the notions behind Google AppEngine. There's no way you could build Google Search on AppEngine, and G+ would be a stretch.

ksec 14y ago

Well because every site are made different, and uses different tech, with different bottleneck. But with SSD, RAM becoming a commodity, ( both prices has been dropping shapely ) I/O are much easier to deal with. And with every release of PostgreSQL / MySQL, Easier Database Replication and more common practice of scaling we are much much better at it then say 2 - 3 years ago.