5 ms·

This seems too expensive and I'd like to try and help, so I'm going to give you an idea of my hosting bill and why it's low and then suggest something for you:

This seems too expensive and I'd like to try and help, so I'm going to give you an idea of my hosting bill and why it's low and then suggest something for you: I pay around $3k per month. I own my own servers and lease a full rack and I serve roughly 1 billion page impressions per month. My bandwidth consumption is measured in Mbps rather than amount of data transferred because I get billed using 95th percentile billing. I average around 130 megabits per second of transfer - constantly, peaking at 150mbps I'm transferring roughly 40 terabytes of data per month. 95th percentile billing and owning your own servers is they key here. To give you an idea, for one month of your hosting bill you can buy 1, possibly 2 servers from Dell and put them in a half rack that will cost you around $800/month including power, secure access, bandwidth, etc. Those servers will last around 5 years with a possible drive replacement or two during that time for a few bucks. But I think you have another problem that's making things worse. With 15,000 active users you should be able to support them on one or two small Linode servers using round robin DNS. That's a relatively small user base and the number of requests per second can't be anything more than 10 per second? So I'm guessing something about your basic app architecture is off. It could be that you're not using Nginx to reverse proxy to Apache and you think you need more apache children/processes, and therefore more memory, than you actually do. You could have a DB that doesn't have indexes in the right places and so you're IO bound. I would suggest first looking at your app and seeing where the bottlenecks are in performance. Fix that first, then look at hosting. Questions: -How many servers do you currently have and give us a rough idea of config. -How many app requests per second do you get at peak? -What's your peak bandwidth throughput in Mbps? -On your servers, is lack of CPU or lack of disk throughput the bottleneck? -Have you had problems running out of memory that caused you to buy more servers? Which app ran out of memory? -Give us an idea of your server config. e.g. nginx -> Apache -> MySQL & Redis. Do the servers talk to each other and if so what do they do?

timsally 15y ago
Take note. This is actually a useful response that people benefit from reading. As opposed to the several one liners already posted that all say variants of "your hosting is expensive". Thanks for taking the time to write this response up.
thaumaturgy 15y ago
Just for another data point, a while back one of my customers' sites got really popular overnight, and peaked at around 25Mbit/s sustained traffic, with Apache handling around 800+ requests/second, on a Linode 768. No RAM issues, the server wasn't stumbling at all, and the site was still loading nice and quick. No nginx magic in that configuration, either. Just straight Apache, with a few tweaks.
- mmaunder 15y ago
  Could you share your apache config please! I find this incredible. Just very curious how you managed this and hoping to learn something. The reason I ask is because: Apache has two stable modes of operation, threaded or one process per child. Both of them require one process or thread per connection busy being served, which requires memory. Apache consumes around 20M per thread or process if PHP or mod_perl are loaded. Roughly 5M if neither are loaded. If you have keepalive enabled and you're doing 800 req/s and keepalive timeout is say 30 seconds (the default is 300) then you're going to need roughly 800 * 30 seconds apache children to keep up. So that's 24000 apache children or 120 gigs of memory in your server. A common hack is to disable keepalive which causes clients to disconnect quicker, but they still tend to stay connected for 1 to 2 seconds per request - especially folks geographically far away thanks to latency. So at 800 r/s you're going to need 1600 apache children to keep up, or 8 gigabytes of memory in your server. Now you could have used an experimental MPM and gotten these results, but I haven't heard of that being used much. So I'm really curious how you managed to get 800 r/s from Apache on a server with 768 Megs of memory. For the uninitiated: Most people put Nginx or Lighttpd in front of apache which can serve 10,000+ concurrent connections with a single thread. That then talks to apache and connects/disconnects very fast which keeps the threads or processes free and often 10 processes is enough for thousands of concurrent connections with keepalive enabled on the front end.
  - thaumaturgy 15y ago
    Well, you were right in your math -- I made my previous comment before going back and checking the actual numbers, which was really stupid on my part. From the "Congratulations!" email I sent the client: > ...sustained 100+ requests per second for the last couple of hours; peaks of over 40Mbits of traffic; thousands of simultaneous connections. So, 1/8th of what you were calculating. Sorry about that. I use mpm-worker and have keepalives turned off altogether. I also use suexec and fcgid (not fastcgi, unfortunately). I was using mem_cache at the time, but that's off now because it breaks the newest version of Wordpress. Also, I should have mentioned that this was a static site, so PHP wasn't a factor. If anybody's still interested in the server config, I'll be happy to share it.
anonym 15y ago
Don't you think web crawling (for users with archival accounts, where the full text + associated images/resources for each bookmark is stored and indexed on Pinboard's servers) is probably using up more of those resources than actually serving up pages? I don't think 10 pages per second or whatever is really the relevant metric for this app.
patio11 15y ago
This is a generous offering of expertise, but supposing you were to help him out and he were to take on implementation costs, wouldn't "Here's what I'd do to get you to a billion impressions a month of customer demand for your product" be a higher-priority task than saving a few hundred bucks? The most common failure mode in scaling for startups is to have no scaling problem at all.
- wheels 15y ago
  Costs that are that out of whack with expectations would usually indicate some very low hanging fruit. If you can save $800/month by adding a couple of database indexes and tweaking your config file, then yeah, it's worth it. That said, from what Maciej has written in the past, he sounds competent in these things (i.e. has his databases set up for remote replication and failover), so it would seem that the culprit is more likely over-engineering than under.
- deleted 15y ago
  [deleted]
- rednaught 15y ago
  I think his questions aren't meant to signify an offer expertise but rather say this base level of knowledge and data should be available whether you're on a shared hosting account or run your own Tier IV center. He's not suggesting a rearchitecture, he's just suggesting know the environment. Decisions are wishy-washy without evidence.
- deleted 15y ago
  [deleted]
- code_duck 15y ago
  Well, if we scaled this expense level straight up to a billion impressions, pinboard would be bankrupt instantly.
joelhaasnoot 15y ago
While I pay nowhere near this much, and we're not nearly as popular, I still like his solutions. Why? I like to sleep. I'd rather pay 2x as much for a reliable host for a solution that works than that I'm up at 3am fixing stuff.
biturd 15y ago
I don't know anything about your setup and architecture but do remember, pinboard was one of the sites that was swooped up in the RAID that got instapaper and a few others. Instapaper stayed up because the server taken was only a slave. Pinboard had some of their main hardware taken, and while things did slow down, they too stayed up. Half, maybe more of those hosting fees could be servers that do nothing at all but sit around and wait for a RAID. Would you still be able to build out a hypothetical system as you mention and be able to handle one data center effectively disappearing? Or does that mean taking your information and essentially doubling it, to have a second backup source?
sbierwagen 15y ago
jambo is not Maciej Cegłowski, (username "idlewords") who runs pinboard. So asking him for technical details won't be very helpful. Incidentally, Maciej is banned from HN,[1] due to not being being nice enough to someone. 1: https://twitter.com/#!/pinboard/status/111332316458135553 https://twitter.com/#!/pinboard/status/111332316458135553
- jambo 15y ago
  Thanks. I didn't know idlewords was banned & was assuming he'd be by to answer.
- deleted 15y ago
  [deleted]
- tptacek 15y ago
  Maciej is not currently banned from HN (he was, briefly, after an impolitic comment about a post, but Paul Graham unbanned him within moments of me questioning the ban). Maciej seems pretty annoyed by what happened, and I don't blame him. But there you go. Shit happens! He's basically one of the friendliest people I've met on the Internet, though, so if you want to help him out you can just ping him on Twitter.
  - staunch 15y ago
    You use the word "friendliest" to describe a guy who three comments ago called someone a "douchebag" because his post was too long? He's got a history of similar behavior. He really is a bit of a troll. He certainly doesn't follow HN guideline of "Don't say things you wouldn't say in a face to face conversation.". My guess is PG only unbanned him so he wouldn't face the criticism of seeming to censor one of his critics.
    - joshu 15y ago
      How do you know he wouldn't say that in person?
      - 15y ago
nphase 15y ago
I can second this. I hosted CentSports (a recently acquired side project) (~1mm userbase, ~500gb dataset) in a full cabinet for under $1100/mo. At it's peak we did about 80 million pageviews per month on a heavily IO-constrained, heavily active OLTP workload (...not nearly as impressive as mmaunder's!) By the time the doors closed, we had about 16 RU's filled, IIRC. The beefiest DB box had 64gb of RAM in it, which as it turns out is a $1200 one-time upgrade for colo vs $1500/mo extra on your dedicated (quick price checks on newegg and softlayer, I'm sure both numbers can come down). Yes, making the leap into colo can seem like a big up front cost, but over time these costs do work in your favor. Yes, there is a support cost involved (either with remote hands or you getting up at 3am and replacing that disk yourself). But If your workload and dataset are such that you don't necessarily require a new box spun up in less than 24 hours, the savings could be quite great for you.
mapgrep 15y ago
Maciej has said repeatedly (including on HN) that disk is the bottleneck keeping him off Linode and VPS in general. Not all web apps use resources in the same way; you're quite likely making an apples and oranges comparison here (though you don't say what your own web app actually does). It makes sense that Pinboard hits the disk a lot because there's not a ton of shared data -- each user tends to have his own trove of data. Yes, certain URLs will be bookmarked a lot but each bookmark can have different access restrictions, summary data, tags and so on. And no one bookmark is going to make up a big percentage of hits -- you're talking about a ton of bookmarks, each accessed rarely. It sounds a lot more like webmail than, say, a blog in terms of access patterns. You have questions, and it's nice that you (eventually) asked them and admitted all that you DON'T know about Pinboard. But you should probably ask them before you dole out unsolicited advice, not after.
- tptacek 15y ago
  I would be thrilled to get unsolicited advice like this about anything I built, even if the advice was wrong. You made a good point, but if you had just worded it a little differently, it wouldn't come across like you were trying to take someone down a peg for writing something thoughtful on HN.

timsally 15y ago

Take note. This is actually a useful response that people benefit from reading. As opposed to the several one liners already posted that all say variants of "your hosting is expensive". Thanks for taking the time to write this response up.

thaumaturgy 15y ago

Just for another data point, a while back one of my customers' sites got really popular overnight, and peaked at around 25Mbit/s sustained traffic, with Apache handling around 800+ requests/second, on a Linode 768. No RAM issues, the server wasn't stumbling at all, and the site was still loading nice and quick. No nginx magic in that configuration, either. Just straight Apache, with a few tweaks.

mmaunder 15y ago
Could you share your apache config please! I find this incredible. Just very curious how you managed this and hoping to learn something. The reason I ask is because: Apache has two stable modes of operation, threaded or one process per child. Both of them require one process or thread per connection busy being served, which requires memory. Apache consumes around 20M per thread or process if PHP or mod_perl are loaded. Roughly 5M if neither are loaded. If you have keepalive enabled and you're doing 800 req/s and keepalive timeout is say 30 seconds (the default is 300) then you're going to need roughly 800 * 30 seconds apache children to keep up. So that's 24000 apache children or 120 gigs of memory in your server. A common hack is to disable keepalive which causes clients to disconnect quicker, but they still tend to stay connected for 1 to 2 seconds per request - especially folks geographically far away thanks to latency. So at 800 r/s you're going to need 1600 apache children to keep up, or 8 gigabytes of memory in your server. Now you could have used an experimental MPM and gotten these results, but I haven't heard of that being used much. So I'm really curious how you managed to get 800 r/s from Apache on a server with 768 Megs of memory. For the uninitiated: Most people put Nginx or Lighttpd in front of apache which can serve 10,000+ concurrent connections with a single thread. That then talks to apache and connects/disconnects very fast which keeps the threads or processes free and often 10 processes is enough for thousands of concurrent connections with keepalive enabled on the front end.
- thaumaturgy 15y ago
  Well, you were right in your math -- I made my previous comment before going back and checking the actual numbers, which was really stupid on my part. From the "Congratulations!" email I sent the client: > ...sustained 100+ requests per second for the last couple of hours; peaks of over 40Mbits of traffic; thousands of simultaneous connections. So, 1/8th of what you were calculating. Sorry about that. I use mpm-worker and have keepalives turned off altogether. I also use suexec and fcgid (not fastcgi, unfortunately). I was using mem_cache at the time, but that's off now because it breaks the newest version of Wordpress. Also, I should have mentioned that this was a static site, so PHP wasn't a factor. If anybody's still interested in the server config, I'll be happy to share it.

anonym 15y ago

Don't you think web crawling (for users with archival accounts, where the full text + associated images/resources for each bookmark is stored and indexed on Pinboard's servers) is probably using up more of those resources than actually serving up pages? I don't think 10 pages per second or whatever is really the relevant metric for this app.

patio11 15y ago

This is a generous offering of expertise, but supposing you were to help him out and he were to take on implementation costs, wouldn't "Here's what I'd do to get you to a billion impressions a month of customer demand for your product" be a higher-priority task than saving a few hundred bucks? The most common failure mode in scaling for startups is to have no scaling problem at all.

wheels 15y ago
Costs that are that out of whack with expectations would usually indicate some very low hanging fruit. If you can save $800/month by adding a couple of database indexes and tweaking your config file, then yeah, it's worth it. That said, from what Maciej has written in the past, he sounds competent in these things (i.e. has his databases set up for remote replication and failover), so it would seem that the culprit is more likely over-engineering than under.
deleted 15y ago
[deleted]
rednaught 15y ago
I think his questions aren't meant to signify an offer expertise but rather say this base level of knowledge and data should be available whether you're on a shared hosting account or run your own Tier IV center. He's not suggesting a rearchitecture, he's just suggesting know the environment. Decisions are wishy-washy without evidence.
deleted 15y ago
[deleted]
code_duck 15y ago
Well, if we scaled this expense level straight up to a billion impressions, pinboard would be bankrupt instantly.

joelhaasnoot 15y ago

While I pay nowhere near this much, and we're not nearly as popular, I still like his solutions. Why? I like to sleep. I'd rather pay 2x as much for a reliable host for a solution that works than that I'm up at 3am fixing stuff.

biturd 15y ago

I don't know anything about your setup and architecture but do remember, pinboard was one of the sites that was swooped up in the RAID that got instapaper and a few others. Instapaper stayed up because the server taken was only a slave. Pinboard had some of their main hardware taken, and while things did slow down, they too stayed up. Half, maybe more of those hosting fees could be servers that do nothing at all but sit around and wait for a RAID. Would you still be able to build out a hypothetical system as you mention and be able to handle one data center effectively disappearing? Or does that mean taking your information and essentially doubling it, to have a second backup source?

sbierwagen 15y ago

jambo is not Maciej Cegłowski, (username "idlewords") who runs pinboard. So asking him for technical details won't be very helpful. Incidentally, Maciej is banned from HN,[1] due to not being being nice enough to someone. 1: https://twitter.com/#!/pinboard/status/111332316458135553 https://twitter.com/#!/pinboard/status/111332316458135553

jambo 15y ago
Thanks. I didn't know idlewords was banned & was assuming he'd be by to answer.
deleted 15y ago
[deleted]
tptacek 15y ago
Maciej is not currently banned from HN (he was, briefly, after an impolitic comment about a post, but Paul Graham unbanned him within moments of me questioning the ban). Maciej seems pretty annoyed by what happened, and I don't blame him. But there you go. Shit happens! He's basically one of the friendliest people I've met on the Internet, though, so if you want to help him out you can just ping him on Twitter.
- staunch 15y ago
  You use the word "friendliest" to describe a guy who three comments ago called someone a "douchebag" because his post was too long? He's got a history of similar behavior. He really is a bit of a troll. He certainly doesn't follow HN guideline of "Don't say things you wouldn't say in a face to face conversation.". My guess is PG only unbanned him so he wouldn't face the criticism of seeming to censor one of his critics.
  - joshu 15y ago
    How do you know he wouldn't say that in person?
    - 15y ago

nphase 15y ago

I can second this. I hosted CentSports (a recently acquired side project) (~1mm userbase, ~500gb dataset) in a full cabinet for under $1100/mo. At it's peak we did about 80 million pageviews per month on a heavily IO-constrained, heavily active OLTP workload (...not nearly as impressive as mmaunder's!) By the time the doors closed, we had about 16 RU's filled, IIRC. The beefiest DB box had 64gb of RAM in it, which as it turns out is a $1200 one-time upgrade for colo vs $1500/mo extra on your dedicated (quick price checks on newegg and softlayer, I'm sure both numbers can come down). Yes, making the leap into colo can seem like a big up front cost, but over time these costs do work in your favor. Yes, there is a support cost involved (either with remote hands or you getting up at 3am and replacing that disk yourself). But If your workload and dataset are such that you don't necessarily require a new box spun up in less than 24 hours, the savings could be quite great for you.

mapgrep 15y ago

Maciej has said repeatedly (including on HN) that disk is the bottleneck keeping him off Linode and VPS in general. Not all web apps use resources in the same way; you're quite likely making an apples and oranges comparison here (though you don't say what your own web app actually does). It makes sense that Pinboard hits the disk a lot because there's not a ton of shared data -- each user tends to have his own trove of data. Yes, certain URLs will be bookmarked a lot but each bookmark can have different access restrictions, summary data, tags and so on. And no one bookmark is going to make up a big percentage of hits -- you're talking about a ton of bookmarks, each accessed rarely. It sounds a lot more like webmail than, say, a blog in terms of access patterns. You have questions, and it's nice that you (eventually) asked them and admitted all that you DON'T know about Pinboard. But you should probably ask them before you dole out unsolicited advice, not after.

tptacek 15y ago
I would be thrilled to get unsolicited advice like this about anything I built, even if the advice was wrong. You made a good point, but if you had just worded it a little differently, it wouldn't come across like you were trying to take someone down a peg for writing something thoughtful on HN.