4 ms·

seriously dude... write that book... Cause i can't find it anywhere...

patio11 16y ago
This is one of those books that the market is basically incapable of publishing, because its like "Hey why don't you do 6 months of hard work and then we'll pay you a $10,000 ~ $20,000 advance which you will never earn out" when Plan B looks like "Save some large company whose system being down is costing $X00,000 a day, make client look like hero, get compensated accordingly."
- buro9 16y ago
  And as I was trying to emphasise in my post, there's not a "right way" to do things, some problems just make some approaches more right than others in those instances. Just with Bentley's Programming Pearls book, he underlines again and again that knowing your problem is more important than knowing the best algorithms, the one that will work for you depends on your problem and only you know that. Highscalability.com and shared slidedecks act as a community generated set of architectural patterns, but no-one should implement them without knowing what their problems will be.
  - grovulent 16y ago
    What I don't understand - and perhaps will be incapable of understanding until I face these issues myself (and hopefully I will in the future) - is why this view that scaling is in a sense un-documentable... What is it about scaling an application that can't be reduced to an algorithm - at some level of abstraction at least, so newbies can at least get an idea of how they should start thinking about it. To put the question in another way - could a framework like django ever come to provide scaling tools out of the box? Or is it just something that fundamentally can't be reduced. Might it be that there just haven't been enough people who have faced scaling problems that repeatable patterns haven't yet become obvious?
    - Twirrim 16y ago
      It's undocumentable because as strange as it seems, no two problems are the same. There is no one-size-fits-all approach, not even a one-size-fits-many. At best we have a series of one-size-may-fit-you-if-you're-lucky options. Disqus isn't using NoSQL or Nginx, something a good number of scaled web applications have switched to. Why? It doesn't solve their problem. Why is their problem different from others? That's a long and complex answer that revolves around almost every aspect of how their applications run, access data, what types of data they're accessing, and so on, and so forth. Is the problem their algorithm? Does it spend a long amount of CPU time working away? Could it be written a different way? Is it data access, is the lag caused by queries taking to long? Is that down to badly formatted queries, inefficient schema, server problems or something else? Is it even a single problem or a combination of a multitude of minor little niggles that combine into a big headache? Do you do thousands of little queries, or smaller numbers of big ones? Are your tables narrow or wide? What size and types of data do you store in the fields? A number of the things that Disqus have done to scale out aren't appropriate for other environments, by nature of the fundamentals of the app. All they can advise on is how to scale your python/Django/MySQL based commenting system, but even then your approach to writing one might be different to theirs. Quite simply, no one can tell you how to scale your application and its infrastructure, because every application and infrastructure is unique by the very nature of every problem being unique, and every solution more so. That's not to say there is no value in the information that Disqus has provided. Quite the contrary, there is every bit of value there, and I greatly appreciate them posting it. There is a good chance that whilst some of what they've done won't be of use to you, some of it may be. It may even be of use for other reasons that are entirely different from those that benefit Disqus. Quick example. You want to load balance web traffic, what do you chose for it? Is software of hardware best? Do you do lots of SSL (would an SSL Accelerator be of use?) Do you want the servers to directly respond to the client, or respond through the load balancers? Apache httpd+mod_proxy Nginx HAProxy ldirectord lighttpd You want to add in caching: Varnish Apache Traffic Server Squid Polipo and so on! Each software package has its own particular strengths and weaknesses, and its as more a matter of gut instinct and intimate knowledge of the way your code and site works, than anything else that can help you find the right way to scale.
sciurus 16y ago
Can anyone speak to how close these books come to this? Or recommend other books? Building Scalable Web Sites http://oreilly.com/catalog/9780596102357 http://oreilly.com/catalog/9780596102357 The Art of Capacity Planning http://oreilly.com/catalog/9780596518585 http://oreilly.com/catalog/9780596518585 Web Operations http://oreilly.com/catalog/0636920000136 http://oreilly.com/catalog/0636920000136
- buro9 16y ago
  Both of John Allspaw's books (the latter two on your list) look good from their table of contents. And if you're in doubt, John is now VP of Ops at Etsy and came from Flickr before that: http://www.kitchensoap.com/about-me/ http://www.kitchensoap.com/about-me/ His blog is interesting too: http://www.kitchensoap.com/ http://www.kitchensoap.com/ So without having read the books, I would shoot for the latter 2 if I wanted to have hard copies around to introduce me to this kind of stuff.
  - ruckusing 16y ago
    I've found "Scalable Internet Architetures" by Theo Schlossnagle to also be quite valuable. It contains general advice on how to approach problem solving when it comes to building uh, scalable architectures. http://www.amazon.com/Scalable-Internet-Architectures-Theo-Schlossnagle/dp/067232699X http://www.amazon.com/Scalable-Internet-Architectures-Theo-S...
- simonw 16y ago
  I'm a big fan of Building Scalable Web Sites - it's a bit old now (it predates cloud-computing-for-everything) but still very relevant if you're just starting to learn about this stuff. It's basically everything the author learned scaling Flickr from a tiny site to several hundred million photos.
- jqp 16y ago
  John Allspaw's books are great. I'd recommend these also: The Art of Scalability http://www.amazon.com/Art-Scalability-Architecture-Organizations-Enterprise/dp/0137030428/ http://www.amazon.com/Art-Scalability-Architecture-Organizat... Scalable Internet Architectures http://www.amazon.com/Scalable-Internet-Architectures-Theo-Schlossnagle/dp/067232699X/ http://www.amazon.com/Scalable-Internet-Architectures-Theo-S... Enterprise Cloud Computing http://www.amazon.com/Enterprise-Cloud-Computing-Architecture-Applications/dp/0521137357/ http://www.amazon.com/Enterprise-Cloud-Computing-Architectur...

This is one of those books that the market is basically incapable of publishing, because its like "Hey why don't you do 6 months of hard work and then we'll pay you a $10,000 ~ $20,000 advance which you will never earn out" when Plan B looks like "Save some large company whose system being down is costing $X00,000 a day, make client look like hero, get compensated accordingly."

buro9 16y ago
And as I was trying to emphasise in my post, there's not a "right way" to do things, some problems just make some approaches more right than others in those instances. Just with Bentley's Programming Pearls book, he underlines again and again that knowing your problem is more important than knowing the best algorithms, the one that will work for you depends on your problem and only you know that. Highscalability.com and shared slidedecks act as a community generated set of architectural patterns, but no-one should implement them without knowing what their problems will be.
- grovulent 16y ago
  What I don't understand - and perhaps will be incapable of understanding until I face these issues myself (and hopefully I will in the future) - is why this view that scaling is in a sense un-documentable... What is it about scaling an application that can't be reduced to an algorithm - at some level of abstraction at least, so newbies can at least get an idea of how they should start thinking about it. To put the question in another way - could a framework like django ever come to provide scaling tools out of the box? Or is it just something that fundamentally can't be reduced. Might it be that there just haven't been enough people who have faced scaling problems that repeatable patterns haven't yet become obvious?
  - Twirrim 16y ago
    It's undocumentable because as strange as it seems, no two problems are the same. There is no one-size-fits-all approach, not even a one-size-fits-many. At best we have a series of one-size-may-fit-you-if-you're-lucky options. Disqus isn't using NoSQL or Nginx, something a good number of scaled web applications have switched to. Why? It doesn't solve their problem. Why is their problem different from others? That's a long and complex answer that revolves around almost every aspect of how their applications run, access data, what types of data they're accessing, and so on, and so forth. Is the problem their algorithm? Does it spend a long amount of CPU time working away? Could it be written a different way? Is it data access, is the lag caused by queries taking to long? Is that down to badly formatted queries, inefficient schema, server problems or something else? Is it even a single problem or a combination of a multitude of minor little niggles that combine into a big headache? Do you do thousands of little queries, or smaller numbers of big ones? Are your tables narrow or wide? What size and types of data do you store in the fields? A number of the things that Disqus have done to scale out aren't appropriate for other environments, by nature of the fundamentals of the app. All they can advise on is how to scale your python/Django/MySQL based commenting system, but even then your approach to writing one might be different to theirs. Quite simply, no one can tell you how to scale your application and its infrastructure, because every application and infrastructure is unique by the very nature of every problem being unique, and every solution more so. That's not to say there is no value in the information that Disqus has provided. Quite the contrary, there is every bit of value there, and I greatly appreciate them posting it. There is a good chance that whilst some of what they've done won't be of use to you, some of it may be. It may even be of use for other reasons that are entirely different from those that benefit Disqus. Quick example. You want to load balance web traffic, what do you chose for it? Is software of hardware best? Do you do lots of SSL (would an SSL Accelerator be of use?) Do you want the servers to directly respond to the client, or respond through the load balancers? Apache httpd+mod_proxy Nginx HAProxy ldirectord lighttpd You want to add in caching: Varnish Apache Traffic Server Squid Polipo and so on! Each software package has its own particular strengths and weaknesses, and its as more a matter of gut instinct and intimate knowledge of the way your code and site works, than anything else that can help you find the right way to scale.

sciurus 16y ago

Can anyone speak to how close these books come to this? Or recommend other books? Building Scalable Web Sites http://oreilly.com/catalog/9780596102357 http://oreilly.com/catalog/9780596102357 The Art of Capacity Planning http://oreilly.com/catalog/9780596518585 http://oreilly.com/catalog/9780596518585 Web Operations http://oreilly.com/catalog/0636920000136 http://oreilly.com/catalog/0636920000136

buro9 16y ago
Both of John Allspaw's books (the latter two on your list) look good from their table of contents. And if you're in doubt, John is now VP of Ops at Etsy and came from Flickr before that: http://www.kitchensoap.com/about-me/ http://www.kitchensoap.com/about-me/ His blog is interesting too: http://www.kitchensoap.com/ http://www.kitchensoap.com/ So without having read the books, I would shoot for the latter 2 if I wanted to have hard copies around to introduce me to this kind of stuff.
- ruckusing 16y ago
  I've found "Scalable Internet Architetures" by Theo Schlossnagle to also be quite valuable. It contains general advice on how to approach problem solving when it comes to building uh, scalable architectures. http://www.amazon.com/Scalable-Internet-Architectures-Theo-Schlossnagle/dp/067232699X http://www.amazon.com/Scalable-Internet-Architectures-Theo-S...
simonw 16y ago
I'm a big fan of Building Scalable Web Sites - it's a bit old now (it predates cloud-computing-for-everything) but still very relevant if you're just starting to learn about this stuff. It's basically everything the author learned scaling Flickr from a tiny site to several hundred million photos.
jqp 16y ago
John Allspaw's books are great. I'd recommend these also: The Art of Scalability http://www.amazon.com/Art-Scalability-Architecture-Organizations-Enterprise/dp/0137030428/ http://www.amazon.com/Art-Scalability-Architecture-Organizat... Scalable Internet Architectures http://www.amazon.com/Scalable-Internet-Architectures-Theo-Schlossnagle/dp/067232699X/ http://www.amazon.com/Scalable-Internet-Architectures-Theo-S... Enterprise Cloud Computing http://www.amazon.com/Enterprise-Cloud-Computing-Architecture-Applications/dp/0521137357/ http://www.amazon.com/Enterprise-Cloud-Computing-Architectur...