3 ms·

You can handle a much larger number of connections provided they don't do anything useful. However if you expect to do anything useful, you are likely to need s

by pkl 15y ago

You can handle a much larger number of connections provided they don't do anything useful. However if you expect to do anything useful, you are likely to need some horizontal scalability once you have thousands of connections. AdWords charge about $1 per click. Lets say your hardware budget is only $1 per connection. With 10K connections you still have enough money for 10 servers if you wanted.

jshen 15y ago
"You can handle a much larger number of connections provided they don't do anything useful." Define useful? I think you mean, computationally complex, but that isn't the same thing as useful. At my last job I worked on a web crawler, and fetching a page, then persisting it in our grid, was useful work. It was useful to have more than 10k connections in that context. One of my pet peeves is this idea of throwing servers at problems. We currently have enormous problems with power and heat dissipation in data centers. At my last job, where I did the web crawler, we couldn't put any more servers in our data center because we didn't have enough power. Doing more, with fewer servers can be very important. I also wonder how much larger the carbon footprint of computing is because of such philosophies.
- deleted 15y ago
  [deleted]
- peter_lawrey 15y ago
  You make a valid point. There are many organisations which are struggling to manage their data centre costs and the cost of the server is a small piece of the puzzle. They are often unable to upgrade their hardware to minimise costs. Throwing servers at the problem is not a good idea. However, you have to be in a position trade off software vs hardware and many organisation don't make good choices as they are managed separately. Organisation inefficiency often results in having more servers than required. I suspect having more than 10K connections improved how much of your organisations bandwidth you could get. I suspect increasing your available bandwidth would have resulted in less connections required. However if you want to start worrying about the planet, you could ask; does the world really need yet another web crawler. ;) The most effect way to do anything is to avoid doing it at all.

by pkl 15y ago

jshen 15y ago

"You can handle a much larger number of connections provided they don't do anything useful." Define useful? I think you mean, computationally complex, but that isn't the same thing as useful. At my last job I worked on a web crawler, and fetching a page, then persisting it in our grid, was useful work. It was useful to have more than 10k connections in that context. One of my pet peeves is this idea of throwing servers at problems. We currently have enormous problems with power and heat dissipation in data centers. At my last job, where I did the web crawler, we couldn't put any more servers in our data center because we didn't have enough power. Doing more, with fewer servers can be very important. I also wonder how much larger the carbon footprint of computing is because of such philosophies.

deleted 15y ago
[deleted]
peter_lawrey 15y ago
You make a valid point. There are many organisations which are struggling to manage their data centre costs and the cost of the server is a small piece of the puzzle. They are often unable to upgrade their hardware to minimise costs. Throwing servers at the problem is not a good idea. However, you have to be in a position trade off software vs hardware and many organisation don't make good choices as they are managed separately. Organisation inefficiency often results in having more servers than required. I suspect having more than 10K connections improved how much of your organisations bandwidth you could get. I suspect increasing your available bandwidth would have resulted in less connections required. However if you want to start worrying about the planet, you could ask; does the world really need yet another web crawler. ;) The most effect way to do anything is to avoid doing it at all.