6 ms·

The Google App Engine Hoax

41 points by jjx 18y ago

timf 18y ago
This is overblown. Estimate 10 objects per page (if you don't have good cache headers)... 43 million GETs a day (1.3 trillion requests a month) at 10/page is around 4 million pages a day and 130 billion a month. Adjust your estimate as necessary but it's quite a lot of page views... Plus, you can request a quota increase here: http://code.google.com/support/bin/request.py?contact_type=AppEngineCPURequest http://code.google.com/support/bin/request.py?contact_type=A...
- damienkatz 18y ago
  It's about fast scale-out. If you are getting that kind of sustained traffic, then you can likely pay for the conventional hosting. The problem is what happens when you have normal trafic but get spikes? The NY Times links to your startup one morning and suddenly you have hundreds of thousands potential new customers kicking the tires at the same time. You'd like to be able to handle that easily, so you can convert some small % into paying customers and take advantage of the free publicity. But if your site goes down in flames you give a bad first impression to all those potential sales, and they will likely never give you another chance.
  - timf 18y ago
    I agree but still think that's possible with the quota (before you request an increase). In an eight hour period (the work day), 500 requests per second is enough to serve 1 million people 17 GETs. And the vast majority of people will bounce away after one page view. So unless you cured cancer or something, I wouldn't worry.
ajkirwin 18y ago
Talk about much ado about nothing. If Google's limits are, somehow, a problem for you.. go get yourself some EC2 servers, or co-locate or.. a hundred other possible things.
- Tichy 18y ago
  But there is a problem in that the App Engine is not Open Source, or is it? So to move one's site to EC2, a lot of reprogramming would be necessary.
  - voidpointer 18y ago
    That is what you get for building your project on a proprietary platform. You are giving control to a third party and you better have a plan how to deal with the associated risk. Along these lines, it strikes me as odd how people that would want to run open-source software on their own computers are completely content with running closed web applications where they have no freedom and no control at all. It may be the same reason people are willing to pay for bottled tap-water: convenience - it seems to trump other factors (including ethics and morality) by far.
- tjogin 18y ago
  What are the odds that this blogger guy actually has a site that's already pushing 500 requests per second?
  - redrobot5050 18y ago
    Bottom barrel shared hosting companies like www.myhosting.com and www.dreamhost.com top out at about 50/requests/second. Compared to that, 30,000 reqs/sec seems reasonable enough to me.
- jimbokun 18y ago
  Isn't that what he said? "Azure here I come." (...I believe qualifies as one of "a hundred other possible things.")
- mseebach 18y ago
  The promise of the cloud was a goodbye to limits. Deploy to the cloud, and you'll never run out of anything. Ever. Amazon (, promise of): We add resources faster than you can use them, kthxbye. Google apps: Hit the once-in-a-lifetime really, really big one, and we'll be serving up 503's to your customers instead of writing invoices to you. Oh, and if it's sustained we might need you to spend critical time re-deploying to a different architecture.
pstatho 18y ago
Gotta agree that 500 req/s for a scale on demand service. I'm building my service on the Azure platform, though nothing has been officially announced, it doesn't look like they'll have similar limits.
- vegai 18y ago
  But doesn't Azure have a much worse limitation: Microsoft?
TweedHeads 18y ago
Fake blog, disperse posts, incendiary article against google. Smells like propaganda, follow the money trail to redmond.
- josefresco 18y ago
  What you didn't like his Religion = Slavery post from 2008? http://brokenpromisms.blogspot.com/2008/10/religion-slavery.html http://brokenpromisms.blogspot.com/2008/10/religion-slavery....
- jjx 18y ago
  jjx here. This is not propaganda, nor is it incendiary. It's the truth. I have been promoting App Engine to everyone I talk to, because of the ease of deployment and the promise of scalability. I have been waiting for months for the beta quotas to be lifted. My wait has been in vain. It seems that scalability to Google means it will scale on one little machine in their infrastructure. If 500 requests per second is the best I can get from "10 years of knowledge Google has in running massively scalable, performance driven systems", then the hype of Google is a fiction.
  - mcantelon 18y ago
    All you have to do is ask them to raise your quota. Why are you omitting this fact? http://code.google.com/support/bin/request.py?contact_type=AppEngineCPURequest http://code.google.com/support/bin/request.py?contact_type=A...
patio11 18y ago
Umm, not to be picky, but can't a single server handle 500 requests a second if (as is likely) most of them are for static content? I just hit my site's homepage with ab (Apache Benchmark) just for giggles. Its on a 256 MB Slicehost VPS ($20 a month), with a bog-standard install of Nginx serving static content. Nginx has a maximum of two processes. Result of smashing it with 25 concurrent requests through AB: served 80 requests a second. I'm guessing I could set that number of child processes higher and watch as it dutifully saturated my outbound link. 500 requests a second is only a factor of six and change, that is probably easily doable. 90% of the workload going into an app engine app is going to be the same stuff, right? Hitting small bits of static content, repeatedly. Remind me why you need to rearchitecture your entire application if the limits to what The Cloooooooooooud can scale to are in any way comparable to what you can get from a single $20 VPS?
- timf 18y ago
  If you are hosting predominantly static content there, then it doesn't make a lot of sense for this level of traffic because your bandwidth bill is going to be sky high (put your static content elsewhere). Also there are uptime properties with the app and its datastore you are not going to get at slicehost without work and other instances. And besides, this is not a limit on Google's architecture, this is an initial limit they put in place for you to bump up against... which is why I find the guy's argument inflammatory, there is no "hoax" here. EC2 has an initial speedbump too (20 nodes) and they'll bump it up if you ask, our max is at 200 right now. (btw I'm using slicehost, EC2 etc. myself, and I don't see appengine as some silver bullet. I just don't think this article's argument holds any water)
  - irrelative 18y ago
    Even for dynamic content, VPS can push through stuff. I can hit a page that's generated dynamically at about 170 requests per second. On my $30 a month VPS. For a page requiring database access, I can run about 40 per second... with the DB running on the same VPS. Frankly, this is pretty embarrassing for google.
    - 18y ago
kg 18y ago
I agree it seems weird the guy has been blogging since 1994 (per his Blogger about page) with a whopping 2 posts, yet somehow ends up here on HN? Kinda fishy... also, isn't Azure the polar opposite of the technology spectrum from AppEngine? I would of expected "EC2 here I come" instead.
- peregrine 18y ago
  There are people here who hate Google AppEngine and Google in general.
- TimothyFitz 18y ago
  Azure and AppEngine are much more closely related than either to EC2. Both Azure and AppEngine are platform-aware and do the scale-out magic for you; you write your code and they make sure it scales (more or less, I'm certain there's still some application-efficiency/scalability rewriting, but you never worry about individual machines, in theory) EC2 means machine level administration and custom scale-out/scale-up techniques. Not to mention there are load-balancing concerns when you're in the more-requests-than-a-single-machine-can-handle range, although that's above 500/sec for many applications.
lacker 18y ago
The article is wrong. You can get more than 500 requests per second, you just have to contact Google first. http://code.google.com/support/bin/request.py?contact_type=AppEngineCPURequest http://code.google.com/support/bin/request.py?contact_type=A... Also, the limit isn't 500 requests per second, it's 30,000 requests per minute. So this doesn't kick in until you get 30,000 requests. The App Engine help says this should be sufficient for a Digg or Slashdot link... seems reasonable to me.
- DenisM 18y ago
  I can see it now - I got New York Times coverage (once in a lifetime thing), the new users start hammering the site and I am sending an email to Google begging them to increase the quota. Which they may or may not get by the next week or something. You know, this just does not look like a good plan for success. Whatever happened to worry-less scale that they promised?
  - litewulf 18y ago
    I'm going to assume since they mention getting dugg on that form that they will respond to that form quickly. But you know what they say about assumptions.
- sam_in_nyc 18y ago
  Does anybody know what actual figures are for getting Dugg, Reddited, Slashdotted, NYTimed, etc? 500 requests/second seems pretty high to me. But I'm admittedly new to the big leagues.
  - deleted 18y ago
    [deleted]
  - timf 18y ago
    It varies with the emphasis (front page or not) etc., here is some of the latest information I could find on super surges (some people reported over a million viewers): http://www.datacenterknowledge.com/archives/2008/06/12/yahoo-buzz-traffic-the-slashdot-effect-squared/ http://www.datacenterknowledge.com/archives/2008/06/12/yahoo... Here's someone saying that dramatic swings can happen inside 60 seconds: http://omniti.com/seeds/dissecting-todays-internet-traffic-spikes http://omniti.com/seeds/dissecting-todays-internet-traffic-s...
livando 18y ago
my web apps only need to handle about 5 hits a minute, so I'm good.
jjx 18y ago
jjx here (The original poster). My blog post is accurate. Read the description of scalability that Google provides. It implies that you can scale to any degree. 500 request a second is an absolute joke. I can use any web development environment (like Rails), and get 500 requests per second on a crappy little machine. So App Engine doesn't scale, at all. The main reason I wrote this post is out of utter frustration with Google. They should get out of the developer tools business. For months I have been coding around their onerous and ridiculous quotas, waiting for the nirvana of paid quotas, only to be horrified by the new quotas. If these quotas are not lifted, then I've completely wasted my time. As it stands now, I'm abandoning App Engine.
- timf 18y ago
  "if these quotas are not lifted" Did you ask them to increase your quota? http://code.google.com/support/bin/request.py?contact_type=AppEngineCPURequest http://code.google.com/support/bin/request.py?contact_type=A...
  - timf 18y ago
    On your blog, you put this in comments: "For all those saying that "all I need to do is ask (beg) Google for more quota, read this:" http://www.reddit.com/r/programming/comments/81f1k/the_google_app_engine_hoax http://www.reddit.com/r/programming/comments/81f1k/the_googl... I read that and it seems like the person is talking about not getting a special free quota he had and that he was in negotiations for (but had not heard back yet)? That seems like a completely different situation to me than a paying customer who wants to increase their quota. Note that this translates to "give Google more money" so I don't know why you think this is something like begging. On Amazon, they also want to make money and they happily increase quotas. They just want to know what's up, if you're a legitimate customer who's really going to pay them and not do a chargeback on the credit card the first month, etc.
DenisM 18y ago
@jjx, you should have known better than investing your resources into a platform that does not suit your needs. When buying a product buy it for it is today, no what it may or may not become tomorrow.

41 points by jjx 18y ago

timf 18y ago

This is overblown. Estimate 10 objects per page (if you don't have good cache headers)... 43 million GETs a day (1.3 trillion requests a month) at 10/page is around 4 million pages a day and 130 billion a month. Adjust your estimate as necessary but it's quite a lot of page views... Plus, you can request a quota increase here: http://code.google.com/support/bin/request.py?contact_type=AppEngineCPURequest http://code.google.com/support/bin/request.py?contact_type=A...

damienkatz 18y ago
It's about fast scale-out. If you are getting that kind of sustained traffic, then you can likely pay for the conventional hosting. The problem is what happens when you have normal trafic but get spikes? The NY Times links to your startup one morning and suddenly you have hundreds of thousands potential new customers kicking the tires at the same time. You'd like to be able to handle that easily, so you can convert some small % into paying customers and take advantage of the free publicity. But if your site goes down in flames you give a bad first impression to all those potential sales, and they will likely never give you another chance.
- timf 18y ago
  I agree but still think that's possible with the quota (before you request an increase). In an eight hour period (the work day), 500 requests per second is enough to serve 1 million people 17 GETs. And the vast majority of people will bounce away after one page view. So unless you cured cancer or something, I wouldn't worry.

ajkirwin 18y ago

Talk about much ado about nothing. If Google's limits are, somehow, a problem for you.. go get yourself some EC2 servers, or co-locate or.. a hundred other possible things.

Tichy 18y ago
But there is a problem in that the App Engine is not Open Source, or is it? So to move one's site to EC2, a lot of reprogramming would be necessary.
- voidpointer 18y ago
  That is what you get for building your project on a proprietary platform. You are giving control to a third party and you better have a plan how to deal with the associated risk. Along these lines, it strikes me as odd how people that would want to run open-source software on their own computers are completely content with running closed web applications where they have no freedom and no control at all. It may be the same reason people are willing to pay for bottled tap-water: convenience - it seems to trump other factors (including ethics and morality) by far.
tjogin 18y ago
What are the odds that this blogger guy actually has a site that's already pushing 500 requests per second?
- redrobot5050 18y ago
  Bottom barrel shared hosting companies like www.myhosting.com and www.dreamhost.com top out at about 50/requests/second. Compared to that, 30,000 reqs/sec seems reasonable enough to me.
jimbokun 18y ago
Isn't that what he said? "Azure here I come." (...I believe qualifies as one of "a hundred other possible things.")
mseebach 18y ago
The promise of the cloud was a goodbye to limits. Deploy to the cloud, and you'll never run out of anything. Ever. Amazon (, promise of): We add resources faster than you can use them, kthxbye. Google apps: Hit the once-in-a-lifetime really, really big one, and we'll be serving up 503's to your customers instead of writing invoices to you. Oh, and if it's sustained we might need you to spend critical time re-deploying to a different architecture.

pstatho 18y ago

Gotta agree that 500 req/s for a scale on demand service. I'm building my service on the Azure platform, though nothing has been officially announced, it doesn't look like they'll have similar limits.

vegai 18y ago
But doesn't Azure have a much worse limitation: Microsoft?

TweedHeads 18y ago

Fake blog, disperse posts, incendiary article against google. Smells like propaganda, follow the money trail to redmond.

josefresco 18y ago
What you didn't like his Religion = Slavery post from 2008? http://brokenpromisms.blogspot.com/2008/10/religion-slavery.html http://brokenpromisms.blogspot.com/2008/10/religion-slavery....
jjx 18y ago
jjx here. This is not propaganda, nor is it incendiary. It's the truth. I have been promoting App Engine to everyone I talk to, because of the ease of deployment and the promise of scalability. I have been waiting for months for the beta quotas to be lifted. My wait has been in vain. It seems that scalability to Google means it will scale on one little machine in their infrastructure. If 500 requests per second is the best I can get from "10 years of knowledge Google has in running massively scalable, performance driven systems", then the hype of Google is a fiction.
- mcantelon 18y ago
  All you have to do is ask them to raise your quota. Why are you omitting this fact? http://code.google.com/support/bin/request.py?contact_type=AppEngineCPURequest http://code.google.com/support/bin/request.py?contact_type=A...

patio11 18y ago

Umm, not to be picky, but can't a single server handle 500 requests a second if (as is likely) most of them are for static content? I just hit my site's homepage with ab (Apache Benchmark) just for giggles. Its on a 256 MB Slicehost VPS ($20 a month), with a bog-standard install of Nginx serving static content. Nginx has a maximum of two processes. Result of smashing it with 25 concurrent requests through AB: served 80 requests a second. I'm guessing I could set that number of child processes higher and watch as it dutifully saturated my outbound link. 500 requests a second is only a factor of six and change, that is probably easily doable. 90% of the workload going into an app engine app is going to be the same stuff, right? Hitting small bits of static content, repeatedly. Remind me why you need to rearchitecture your entire application if the limits to what The Cloooooooooooud can scale to are in any way comparable to what you can get from a single $20 VPS?

timf 18y ago
If you are hosting predominantly static content there, then it doesn't make a lot of sense for this level of traffic because your bandwidth bill is going to be sky high (put your static content elsewhere). Also there are uptime properties with the app and its datastore you are not going to get at slicehost without work and other instances. And besides, this is not a limit on Google's architecture, this is an initial limit they put in place for you to bump up against... which is why I find the guy's argument inflammatory, there is no "hoax" here. EC2 has an initial speedbump too (20 nodes) and they'll bump it up if you ask, our max is at 200 right now. (btw I'm using slicehost, EC2 etc. myself, and I don't see appengine as some silver bullet. I just don't think this article's argument holds any water)
- irrelative 18y ago
  Even for dynamic content, VPS can push through stuff. I can hit a page that's generated dynamically at about 170 requests per second. On my $30 a month VPS. For a page requiring database access, I can run about 40 per second... with the DB running on the same VPS. Frankly, this is pretty embarrassing for google.
  - 18y ago

kg 18y ago

I agree it seems weird the guy has been blogging since 1994 (per his Blogger about page) with a whopping 2 posts, yet somehow ends up here on HN? Kinda fishy... also, isn't Azure the polar opposite of the technology spectrum from AppEngine? I would of expected "EC2 here I come" instead.

peregrine 18y ago
There are people here who hate Google AppEngine and Google in general.
TimothyFitz 18y ago
Azure and AppEngine are much more closely related than either to EC2. Both Azure and AppEngine are platform-aware and do the scale-out magic for you; you write your code and they make sure it scales (more or less, I'm certain there's still some application-efficiency/scalability rewriting, but you never worry about individual machines, in theory) EC2 means machine level administration and custom scale-out/scale-up techniques. Not to mention there are load-balancing concerns when you're in the more-requests-than-a-single-machine-can-handle range, although that's above 500/sec for many applications.

lacker 18y ago

The article is wrong. You can get more than 500 requests per second, you just have to contact Google first. http://code.google.com/support/bin/request.py?contact_type=AppEngineCPURequest http://code.google.com/support/bin/request.py?contact_type=A... Also, the limit isn't 500 requests per second, it's 30,000 requests per minute. So this doesn't kick in until you get 30,000 requests. The App Engine help says this should be sufficient for a Digg or Slashdot link... seems reasonable to me.

DenisM 18y ago
I can see it now - I got New York Times coverage (once in a lifetime thing), the new users start hammering the site and I am sending an email to Google begging them to increase the quota. Which they may or may not get by the next week or something. You know, this just does not look like a good plan for success. Whatever happened to worry-less scale that they promised?
- litewulf 18y ago
  I'm going to assume since they mention getting dugg on that form that they will respond to that form quickly. But you know what they say about assumptions.
sam_in_nyc 18y ago
Does anybody know what actual figures are for getting Dugg, Reddited, Slashdotted, NYTimed, etc? 500 requests/second seems pretty high to me. But I'm admittedly new to the big leagues.
- deleted 18y ago
  [deleted]
- timf 18y ago
  It varies with the emphasis (front page or not) etc., here is some of the latest information I could find on super surges (some people reported over a million viewers): http://www.datacenterknowledge.com/archives/2008/06/12/yahoo-buzz-traffic-the-slashdot-effect-squared/ http://www.datacenterknowledge.com/archives/2008/06/12/yahoo... Here's someone saying that dramatic swings can happen inside 60 seconds: http://omniti.com/seeds/dissecting-todays-internet-traffic-spikes http://omniti.com/seeds/dissecting-todays-internet-traffic-s...

livando 18y ago

my web apps only need to handle about 5 hits a minute, so I'm good.

jjx 18y ago

jjx here (The original poster). My blog post is accurate. Read the description of scalability that Google provides. It implies that you can scale to any degree. 500 request a second is an absolute joke. I can use any web development environment (like Rails), and get 500 requests per second on a crappy little machine. So App Engine doesn't scale, at all. The main reason I wrote this post is out of utter frustration with Google. They should get out of the developer tools business. For months I have been coding around their onerous and ridiculous quotas, waiting for the nirvana of paid quotas, only to be horrified by the new quotas. If these quotas are not lifted, then I've completely wasted my time. As it stands now, I'm abandoning App Engine.

timf 18y ago
"if these quotas are not lifted" Did you ask them to increase your quota? http://code.google.com/support/bin/request.py?contact_type=AppEngineCPURequest http://code.google.com/support/bin/request.py?contact_type=A...
- timf 18y ago
  On your blog, you put this in comments: "For all those saying that "all I need to do is ask (beg) Google for more quota, read this:" http://www.reddit.com/r/programming/comments/81f1k/the_google_app_engine_hoax http://www.reddit.com/r/programming/comments/81f1k/the_googl... I read that and it seems like the person is talking about not getting a special free quota he had and that he was in negotiations for (but had not heard back yet)? That seems like a completely different situation to me than a paying customer who wants to increase their quota. Note that this translates to "give Google more money" so I don't know why you think this is something like begging. On Amazon, they also want to make money and they happily increase quotas. They just want to know what's up, if you're a legitimate customer who's really going to pay them and not do a chargeback on the credit card the first month, etc.

DenisM 18y ago

@jjx, you should have known better than investing your resources into a platform that does not suit your needs. When buying a product buy it for it is today, no what it may or may not become tomorrow.