11 ms·

Moving The New York Times Games Platform to Google App Engine

297 points by spyspy 9y ago

vs2 9y ago
"Due to the inelastic architecture of our AWS system, we needed to have the systems scaled up to handle our peak traffic at 10PM when the daily puzzle is published." WT... I had to reread this to make sure I didnt misunderstand... why not work on making the current arhictecture elastic?! #cloudPorn
- bpicolo 9y ago
  Imo, one really killer bit is the first piece they mentioned: "Google provides an SDK that enables users to run a suite of services along with an admin interface, database and caching layer with a single command." I really wish AWS had a decent local dev story, rather than relying on 10 separate half-baked OSS solutions
  - rstupek 9y ago
    Have you tried the documentation for google's sdk? I stopped in disgust trying to track down what I needed and then trying to manage through multiple different admin interfaces.
    - bpicolo 9y ago
      I've been using google bits recently. I agree that the docs for e.g. all the python sdks need a lot of work. That said, boto3 has thorough docs, but I wouldn't consider them particularly well organized. I can only really navigate AWS sdk docs because I already know what I want to do, and can google the specific terminology
      - CobrastanJorji 9y ago
        Are you talking about this Python SDK doc? https://googlecloudplatform.github.io/google-cloud-python/latest/ https://googlecloudplatform.github.io/google-cloud-python/la...
        bpicolo 9y ago
        Yeah. In particular, I was really looking for something for "here's how to do most of the basics for Cloud Storage". If you look at https://googlecloudplatform.github.io/google-cloud-python/latest/storage/buckets.html https://googlecloudplatform.github.io/google-cloud-python/la..., for example, there's not even a top level navigation index that I can read through to guess what function I might need by name.
        funkjunky 9y ago
        (Google cloud support here) The pages you linked to are supposed to serve as client libraries reference only. If you want higher level instructions and examples, always start with the main Google Cloud docs first. On any page that offers instructions, the top of the code windows offers a selection of client library languages/CLI tools/REST API available to do whatever that task is. For cloud storage, start here: https://cloud.google.com/storage/docs/how-to https://cloud.google.com/storage/docs/how-to Choose a topic, and select "Python" at the top. It should provide instructions and examples using the Python libraries. Also, we have a repo of demo projects and examples for nearly every GCP product/service, and then some. Great examples to be found here (some might be out of date though): https://github.com/GoogleCloudPlatform/python-docs-samples https://github.com/GoogleCloudPlatform/python-docs-samples
        bpicolo 9y ago
        Heya, Thanks for the link! You're right that this is what I was looking for. Unfortunately, that hadn't shown up in a convenient place while I was googling around. Would be good to add direct links to those from the client lib references, because those pop up for e.g. "google storage python" first. (Unless they're already there and I didn't see them). Links from the READMEs of the repo might be useful too to help people get there? https://github.com/GoogleCloudPlatform/google-cloud-python https://github.com/GoogleCloudPlatform/google-cloud-python
- tyingq 9y ago
  The "inelastic" might have been a shot at AWS. When pressed, the AWS people do use phrases like "pre-warming", "over provisioning" and "advance notice" around their ELB/ALB setup and ECS. Google's cloud salespeople pitch that they don't require any of that.
  - g09980 9y ago
    Curious if the need for pre-warming ELB/ALB still applies. Last time this came up, an AWS employee mentioned it is no longer necessary (https://news.ycombinator.com/item?id=14052079 https://news.ycombinator.com/item?id=14052079), but would be nice if this was documented.
    - tyingq 9y ago
      Possible. It is still in the documentation: https://aws.amazon.com/articles/1636185810492479#pre-warming https://aws.amazon.com/articles/1636185810492479#pre-warming The "advance notice" and "over provision" advice is still being given for things that could scale up fairly large. (where fairly large isn't anything that exciting, really)
    - callalex 9y ago
      I don't want to dox myself but about a year ago when my employer forgot to notify AWS about switching our production traffic (about 5K rps at that time) from one ELB to another we failed requests for several minutes before we decided to just switch back to the old ELB then ask them to do a prewarming before we switched again.
    - gtaylor 9y ago
      It definitely still applies.
    - PaulRobinson 9y ago
      The poster was talking about ALBs. I'm not experienced enough with them to know whether the claim is true. I do know on ELBs though, pre-warming is essential for high throughput.
  - 9y ago
- tziki 9y ago
  The main story here is probably this line: "The system is generally at that peak traffic for only a few minutes a day". AWS charges by hour. Google by minute. If your peak traffic only last a few minutes, hourly billing is inflexible, simple as that.
  - random023987 9y ago
    AWS lambda charges by the 1/10th of a GB/CPU/second. I'd bet they could have seen significant cost savings on AWS by migrating to lambda, and gotten continuous scaling to boot.
    - tziki 9y ago
      Serverless is probably far too immature for companies like NYT.
      - bastawhiz 9y ago
        I don't disagree with you, but it's also true that you don't have to go all-in. I run services that run server-ful-ly on EC2, while requests to the "front-end" are handled by Lambda. If you've got one or two very hot endpoints and the majority of your server load spike is thanks to the processing for those endpoints, moving just those endpoints to Lambda could give you reassurance that you don't have all of your eggs in one basket (not having to go full-serverless) while also getting the benefit of being able to scale up and down essentially effortlessly. Of course, there are lots of other consideration that you should be making when going even partially serverless, and it could be that the NYT chose not to experiment with Lambda for other reasons. For instance, you're effectively tied to CloudWatch for logging and monitoring, which could be a deal-breaker. Much of the processing could be happening in the DB, which would make Lambda moot. It may simply be that their estimated usage of Lambda was too costly.
        sdenton4 9y ago
        Three devs seems like a bit of a bottleneck... Why maintain two separate architectures for a single problem if there's another option?
- mattbillenstein 9y ago
  I'm not sure why people are so enamored with ELB -- just terminate SSL at your web boxes using nginx and publish the public ips of all of these machines in your DNS records. You remove a bunch of ELB per-request costs doing it this way and you can scale it however you see fit.
  - bastawhiz 9y ago
    That works great until a machine starts failing health checks and you need to take it out of rotation ASAP. It's also the case that DNS gets cached and not all users create the same load: one user making many requests will burden one server instead of having the load evenly distributed.
    - mattbillenstein 9y ago
      Clients will try another ip if they can't connect -- partial failures may be a problem, but in my experience as long as nginx is alive, you can load balance to a different web backend if the app processes on that machine are wedged. I've deployed this solution in 50k req/s environments and not seen a single user be a problem like you mention -- any motivated bad actor could cause problems in either scenario I expect.
      - bastawhiz 9y ago
        Clients might not fail to connect. That's even worse. They connect and the server hangs and returns no response (perhaps due to bad configuration changes). Now you're stuck, and your server is too oversaturated to SSH in and fix it. It out depends on your application and users. Building a website? Probably not much of an issue. Building a low-latency API? YMMV. Keeping your load evenly balanced across your front end cluster also can keep your cost low, since you are able to distribute load more evenly. That's another point: if you scale your cluster size up and down frequently to accommodate load, doing that with DNS is a nightmare.
    - awj 9y ago
      > It's also the case that DNS gets cached and not all users create the same load It's also, also the case that DNS gets cached and propagating the removal of a broken server could take ages.
mbesto 9y ago
This reads eerily like a press release for GCP...
- deleted 9y ago
  [deleted]
- colordrops 9y ago
  A third of the posts to HN are disguised press releases.
- paulcole 9y ago
  well duh
  - dang 9y ago
    Would you please stop posting unsubstantive comments to HN?
- s17n 9y ago
  But check out OPs posting history.
  - lallysingh 9y ago
    4 posts about NYT crossword. So they're talking about their work, and their work uses GCP.
    - spyspy 9y ago
      Quite
pgrote 9y ago
Interesting they are using Medium instead of in house publishing tools. It is the first time I've noticed then open.nytimes.com articles.
- devrandomguy 9y ago
  Didn't you hear? Medium pays per clap now, so the NYT authors are all jumping ship /s
revelation 9y ago
I was sure this was about some multiplayer game thing, but no, it's a crossword. Not entirely sure what they are even scaling here, I was expecting an article about a CDN..
- jprob 9y ago
  The game allows you to sync your game progress across multiple devices and it's subscription based so a CDN wouldn't do much help there. Also, realtime multiplayer crosswords are coming! I'll be speaking at GothamGo this year about that exact topic.
- jonknee 9y ago
  It’s a paid service that hundreds of thousands of people use. Not world changing stuff, but considering competitors are still using Java applets and Flash, I for one applaud their efforts.
bsaul 9y ago
Has anybody had successful experience deploying docker containers on appengine ? Last time i tried, i had such a bad experience in terms of deployment speed ( time to build the image, then upload it, then waiting for the stuff to deploy) that i reverted to managing my own gce instance. But maybe i had bad luck..
- tlarkworthy 9y ago
  Yes uploads are fast and you can spin up a VM directly from an image now
  - ledgerdev 9y ago
    How many minutes/seconds do you mean by fast? Can you point to guidance on how to make that happen because I am seeing 8+ minute re-deploy times.
- tarr11 9y ago
  I just tried to move a rails app to appengine. It uses the flexible environment (eg, docker). It took like 10 or 15 minutes to deploy, each time. Heroku does my deploy in about 5 mins. https://groups.google.com/forum/#!topic/google-appengine/hZMEkmmObDU https://groups.google.com/forum/#!topic/google-appengine/hZM... GCloud is cheaper though. Also, VMs spin up in GCloud amazingly fast. Like 5 seconds. Feels like somebody at Gcloud just needs to go and fix this. No reason this is so bad...
  - brianwawok 9y ago
    Deploy is 15 seconds on GKE. Wonder why app engine is so bad.
    - lftl 9y ago
      With GKE you don't necessarily update your load balancer rules with a deploy correct? The linked thread points the blame for app engine deploys on waiting for GCLB to update.
obulpathi 9y ago
Two things keep coming up while comparing GCP and AWS: * This accomplishment would not have been possible for our three-person team of engineers with out Google Cloud (AWS is too low level, hard to work with and does not scale well). * We’ve also managed to cut our infrastructure costs in half during this time period (Per minute billing, seamless autoscaling, performance, sustained usage discounts, ... )
- fha 9y ago
  > AWS . . . does not scale well GCP may have some advantages over AWS, but the reverse is also true and it's hard to take what you say seriously when you say something like that.
  - CharlesW 9y ago
    Totally. Also… > AWS is too low level It seems very strange to paint AWS with such a broad brush, considering that AWS has tons of services at various levels of abstraction (including high-level abstractions like Elastic Beanstalk and AWS Lambda).
    - obulpathi 9y ago
      Sorry to not add context. I was referring to the use case the author of article was talking about: running a website: You need to stitch: ELB / EC2 / Database / Caching / Service Splitting / Auth / Scaling / ... Where as on Google Cloud, App Engine covers most of the points.
      - deleted 9y ago
        [deleted]
      - deleted 9y ago
        [deleted]
      - tomcam 9y ago
        Jeez, you're taking a lot of hits. Thanks for the patient answers
Jedi72 9y ago
Getting pro GCP articles to the top of HN must no-doubt be a high priority for the Google marketing team. This is the nature of modern advertising, sneakily trying to subvert your thinking by masquerading as something else.
- ocdtrekkie 9y ago
  Google likely offers significant discounts to companies which write these sorts of pieces for them. And obviously a lot of Googlers participate actively here, so getting upvotes naturally is likely not a challenge.
- theDoug 9y ago
  We actually spent the entire day at the Giants game today. ¯\(ツ)/¯ There's no incentive for high ranking HN posts, or any HN posts, actually. If there were, you wouldn't see others continually submit our news here before we do. This was a nice and unprompted post for everyone in GCP to read, as well. (Disclosure: I work on GCP as a product marketer.)
  - ben_jones 9y ago
    A GCP product marketer responding to negative commentary on HN within ~20 minutes of it being posted strikes me as automated. EDIT: It is a double standard though, HN readers want access and responses from people on the GCP team but at the same time tinfoil hat subliminal marketing etc.
    - jacquesm 9y ago
      > A GCP product marketer responding to negative commentary on HN within ~20 minutes of it being posted strikes me as automated. Nonsense, there are many Googlers on HN and all it would take is a colleague poking another with a relevant thread.
    - sb8244 9y ago
      I would be reading this every 10 minutes for new comments if it was my product up for discussion.
nrjames 9y ago
I migrated a big data stack to GCP from AWS. Reasons: GCP has better documentation, the AWS console and various services confuse the heck out of me (I guess I'm getting too old), and the security integration between GCP services saves a huge amount of time. It's super easy and very fast to used the Google Compute Engine VMs. Given that the company I work for uses G Suite, it's a piece of cake to implement SSO and other integration pieces. It's also cheaper for us than AWS and more performant.
- jsudhams 9y ago
  Wait until GCP becomes older with bigquery and other versions. Also GCP does not have regions ready and it takes forever to get a new region. Waiting for Mumbai region for more than 3 to 4 months now. But one thing i like about GCP is that it allows to the limit setting in terms of cost and ensure you wont cross it. In case of AWS it can give alerts but for some reason say all you team in in one location and there is emergency like flood etc and you dont check email then you are done. I stopped using AWS after i learned that there is simply no way to set limit. Waiting for GCP to open their Mumbai region. sigh. Also AWS is very deceiving with free tier , there is simply no way to understand which products get free and worst case is after free tier you will get charged.
  - yeukhon 9y ago
    AWS doesn't add a new region every 3-4 months either. Adding a new region is very complicated. Normally vendor does not actually build the DC themselves. They would source from other data centers in the region whenever possible. Building a new DC is not something can be taken lightly. Then finally local laws.
foxylad 9y ago
OT. How nice not see a single "avoid all Google services because reader" comment. Maybe we are finally moving on.
- CaveTech 9y ago
  Yet here you are.
zitterbewegung 9y ago
Nice advertisement that Google bought from the NyTimes.
NightlyDev 9y ago
This doesn't really make much sense to me. How many peak users are there? What's the number of requests per second? I can't imagine that the load would be so high that it wouldn't be possible to do it without GCP with three developers. It would be way more interesting with performance details. :)
ciguy 9y ago
As a DevOps consultant I've actually worked with clients migrating stacks to and from GCE/AWS (Yeah, both ways, not the same client). What I've found in aggregate is that GCE is a bit easier to use at first as AWS has a LOT of features and terminology to learn. When it comes down to it though, many GCE services felt really immature, particularly their CloudSQL offering. One client recently moved from GCE to AWS simply because their CloudSQL (Fully replicated with fail-over setup according to GCE recommendations) kept randomly dying for several minutes at a time. After a LOT of back and forth Google finally admitted that they had updated the replica and the master at the same time, so when it failed over the replica was also down. There were other instances of unexplained downtime that were never adequately explained, but overall that experience was enough for me (And the client) to totally lose faith in the GCE teams competence. Even getting a serious investigation into intermittent downtime and an explanation took over a month. By that time our migration to AWS was in progress. GCE never did explain why they would choose to apply updates to replica + master SQL at the same time and as far as I know they are still doing this. I asked if we could at least be notified of update events, was told that's not possible. There were other issues as well that taken together just made GCE seem amateurish. I'm sure as they mature a bit things will get better, and it is cheaper which is why I wouldn't necessarily recommend against them for startups just getting going today. By the time you are really scaling it's like they'll have more of the kinks worked out.
- ringaroundthetx 9y ago
  Yeah Azure is like that too. Stuff is just really immature and you'll get blocked by lots of things that make no sense. If you have to get vendor locked in your quest to not have to manage servers while learning unintuitively misnamed parts of a computer, choose Amazon.
  - wutwutwutwut 9y ago
    I use Azure all the time (app services, storage, cloud services,VMs, SQL, CDN etc) and almost never run into this issue. Can you share some example on what you mean?
neya 9y ago
Hey community, let me share my experience with AppEngine. I work in a small firm, where we've developed a massive Software Application comprising of 12 medium-sized apps. I went with Phoenix 1.3 w/ the new umbrella architecture. With AppEngine, the beauty is that you can have many custom named microservices under one AppEngine project and each microservices can have many versions. You can even decide how much percentage of traffic should be split between each of these microservices. What's awesome is, in addition to the standard runtimes (Ruby, Python, Go, Java, etc.) Google also provides something called custom VMs for AppEngine, meaning you can push docker based setups into your AppEngine service, with basically any stack you want. This alone is a HUGE incentive to move to AppEngine because usually custom stack will require you to maintain the server side of things, but with Docker + AppEngine, zero devops. Their network panel is also very intuitive to add/delete rules to keep your app secured. I've been using AppEngine for over 4 years now and every time I tried a competitive offering (such as AWS Beanstalk, for example) I've only been disappointed. AppEngine is great for startups. For example, a lesser known feature within AppEngine is their real-time image processing service API. This allows you to scale/crop/resize images in real time and the service is offered free of charge (except for storage). Works really well for web applications with basic image manipulation requirements. https://cloud.google.com/appengine/docs/standard/python/images/ https://cloud.google.com/appengine/docs/standard/python/imag... The best part is, you call your image with specific parameters that'll do transformations on the fly. For example, <image url>/image.jpg?s=120 will return a 120px image. Appending -c will give you a cropped version, etc. I really hope to see AppEngine get more love from startups as it's a brilliant platform, much more performant than it's competitors' offerings. For example, I was previously a huge proponent of Heroku and upon comparing numbers, I realized AppEngine is way more performant (in my use case). I'm so glad we made the switch. If you're looking/considering to move to AppEngine, let me know here and I'll try my best to answer your questions.
- smn1234 9y ago
  The way you've delivered this "experience" makes you sound like you either work for Google or were asked to make a sponsored statement - for credits or $.
  - 9y ago
kennethh 9y ago
Anyone know how much it cost to add a custom domain and SSL to AppEngine(Standard og Flexible)? I have been looking and not able to find out how much it cost.
- iamgopal 9y ago
  SSL is free. Custom domain also free.
  - kennethh 9y ago
    That explains it, would be nice if they could write it down somewhere as a feature:)
    - rhodysurf 9y ago
      Its free for custom but only to use existing certs. You still have to set up the cert and such with your domain as you normally would. You also get the *.appengine.com domain with automatic SSL>
      - iamgopal 9y ago
        SSL cert provided by Google is free and is in private alpha stage. It's coming soon with GA.
evangeline04 9y ago
"Luckily, we at The Times recently decided to move all product development to the Google Cloud Platform" Great decision. Google Cloud is doing all the right things on their PaaS.
zachary_zemlak 9y ago
We use automatic scaling with warmup requests enabled and haven’t run into any issues when migrating traffic to newer versions. We also still use the “original” SDK which allows you to deploy and migrate traffic in two separate steps.
jody39 9y ago
I’ve noticed challenges with the time taken for the load balancing around AppEngine to reconfigure when pushing new releases. Did you have any trouble with that?
merb 9y ago
Uh thanks to this article I've seen that AppEngine know supports Java8, this is really really cool.

297 points by spyspy 9y ago

vs2 9y ago

"Due to the inelastic architecture of our AWS system, we needed to have the systems scaled up to handle our peak traffic at 10PM when the daily puzzle is published." WT... I had to reread this to make sure I didnt misunderstand... why not work on making the current arhictecture elastic?! #cloudPorn

bpicolo 9y ago
Imo, one really killer bit is the first piece they mentioned: "Google provides an SDK that enables users to run a suite of services along with an admin interface, database and caching layer with a single command." I really wish AWS had a decent local dev story, rather than relying on 10 separate half-baked OSS solutions
- rstupek 9y ago
  Have you tried the documentation for google's sdk? I stopped in disgust trying to track down what I needed and then trying to manage through multiple different admin interfaces.
  - bpicolo 9y ago
    I've been using google bits recently. I agree that the docs for e.g. all the python sdks need a lot of work. That said, boto3 has thorough docs, but I wouldn't consider them particularly well organized. I can only really navigate AWS sdk docs because I already know what I want to do, and can google the specific terminology
    - CobrastanJorji 9y ago
      Are you talking about this Python SDK doc? https://googlecloudplatform.github.io/google-cloud-python/latest/ https://googlecloudplatform.github.io/google-cloud-python/la...
      - bpicolo 9y ago
        Yeah. In particular, I was really looking for something for "here's how to do most of the basics for Cloud Storage". If you look at https://googlecloudplatform.github.io/google-cloud-python/latest/storage/buckets.html https://googlecloudplatform.github.io/google-cloud-python/la..., for example, there's not even a top level navigation index that I can read through to guess what function I might need by name.
        funkjunky 9y ago
        (Google cloud support here) The pages you linked to are supposed to serve as client libraries reference only. If you want higher level instructions and examples, always start with the main Google Cloud docs first. On any page that offers instructions, the top of the code windows offers a selection of client library languages/CLI tools/REST API available to do whatever that task is. For cloud storage, start here: https://cloud.google.com/storage/docs/how-to https://cloud.google.com/storage/docs/how-to Choose a topic, and select "Python" at the top. It should provide instructions and examples using the Python libraries. Also, we have a repo of demo projects and examples for nearly every GCP product/service, and then some. Great examples to be found here (some might be out of date though): https://github.com/GoogleCloudPlatform/python-docs-samples https://github.com/GoogleCloudPlatform/python-docs-samples
        bpicolo 9y ago
        Heya, Thanks for the link! You're right that this is what I was looking for. Unfortunately, that hadn't shown up in a convenient place while I was googling around. Would be good to add direct links to those from the client lib references, because those pop up for e.g. "google storage python" first. (Unless they're already there and I didn't see them). Links from the READMEs of the repo might be useful too to help people get there? https://github.com/GoogleCloudPlatform/google-cloud-python https://github.com/GoogleCloudPlatform/google-cloud-python
tyingq 9y ago
The "inelastic" might have been a shot at AWS. When pressed, the AWS people do use phrases like "pre-warming", "over provisioning" and "advance notice" around their ELB/ALB setup and ECS. Google's cloud salespeople pitch that they don't require any of that.
- g09980 9y ago
  Curious if the need for pre-warming ELB/ALB still applies. Last time this came up, an AWS employee mentioned it is no longer necessary (https://news.ycombinator.com/item?id=14052079 https://news.ycombinator.com/item?id=14052079), but would be nice if this was documented.
  - tyingq 9y ago
    Possible. It is still in the documentation: https://aws.amazon.com/articles/1636185810492479#pre-warming https://aws.amazon.com/articles/1636185810492479#pre-warming The "advance notice" and "over provision" advice is still being given for things that could scale up fairly large. (where fairly large isn't anything that exciting, really)
  - callalex 9y ago
    I don't want to dox myself but about a year ago when my employer forgot to notify AWS about switching our production traffic (about 5K rps at that time) from one ELB to another we failed requests for several minutes before we decided to just switch back to the old ELB then ask them to do a prewarming before we switched again.
  - gtaylor 9y ago
    It definitely still applies.
  - PaulRobinson 9y ago
    The poster was talking about ALBs. I'm not experienced enough with them to know whether the claim is true. I do know on ELBs though, pre-warming is essential for high throughput.
- 9y ago
tziki 9y ago
The main story here is probably this line: "The system is generally at that peak traffic for only a few minutes a day". AWS charges by hour. Google by minute. If your peak traffic only last a few minutes, hourly billing is inflexible, simple as that.
- random023987 9y ago
  AWS lambda charges by the 1/10th of a GB/CPU/second. I'd bet they could have seen significant cost savings on AWS by migrating to lambda, and gotten continuous scaling to boot.
  - tziki 9y ago
    Serverless is probably far too immature for companies like NYT.
    - bastawhiz 9y ago
      I don't disagree with you, but it's also true that you don't have to go all-in. I run services that run server-ful-ly on EC2, while requests to the "front-end" are handled by Lambda. If you've got one or two very hot endpoints and the majority of your server load spike is thanks to the processing for those endpoints, moving just those endpoints to Lambda could give you reassurance that you don't have all of your eggs in one basket (not having to go full-serverless) while also getting the benefit of being able to scale up and down essentially effortlessly. Of course, there are lots of other consideration that you should be making when going even partially serverless, and it could be that the NYT chose not to experiment with Lambda for other reasons. For instance, you're effectively tied to CloudWatch for logging and monitoring, which could be a deal-breaker. Much of the processing could be happening in the DB, which would make Lambda moot. It may simply be that their estimated usage of Lambda was too costly.
      - sdenton4 9y ago
        Three devs seems like a bit of a bottleneck... Why maintain two separate architectures for a single problem if there's another option?
mattbillenstein 9y ago
I'm not sure why people are so enamored with ELB -- just terminate SSL at your web boxes using nginx and publish the public ips of all of these machines in your DNS records. You remove a bunch of ELB per-request costs doing it this way and you can scale it however you see fit.
- bastawhiz 9y ago
  That works great until a machine starts failing health checks and you need to take it out of rotation ASAP. It's also the case that DNS gets cached and not all users create the same load: one user making many requests will burden one server instead of having the load evenly distributed.
  - mattbillenstein 9y ago
    Clients will try another ip if they can't connect -- partial failures may be a problem, but in my experience as long as nginx is alive, you can load balance to a different web backend if the app processes on that machine are wedged. I've deployed this solution in 50k req/s environments and not seen a single user be a problem like you mention -- any motivated bad actor could cause problems in either scenario I expect.
    - bastawhiz 9y ago
      Clients might not fail to connect. That's even worse. They connect and the server hangs and returns no response (perhaps due to bad configuration changes). Now you're stuck, and your server is too oversaturated to SSH in and fix it. It out depends on your application and users. Building a website? Probably not much of an issue. Building a low-latency API? YMMV. Keeping your load evenly balanced across your front end cluster also can keep your cost low, since you are able to distribute load more evenly. That's another point: if you scale your cluster size up and down frequently to accommodate load, doing that with DNS is a nightmare.
  - awj 9y ago
    > It's also the case that DNS gets cached and not all users create the same load It's also, also the case that DNS gets cached and propagating the removal of a broken server could take ages.

mbesto 9y ago

This reads eerily like a press release for GCP...

deleted 9y ago
[deleted]
colordrops 9y ago
A third of the posts to HN are disguised press releases.
paulcole 9y ago
well duh
- dang 9y ago
  Would you please stop posting unsubstantive comments to HN?
s17n 9y ago
But check out OPs posting history.
- lallysingh 9y ago
  4 posts about NYT crossword. So they're talking about their work, and their work uses GCP.
  - spyspy 9y ago
    Quite

pgrote 9y ago

Interesting they are using Medium instead of in house publishing tools. It is the first time I've noticed then open.nytimes.com articles.

devrandomguy 9y ago
Didn't you hear? Medium pays per clap now, so the NYT authors are all jumping ship /s

revelation 9y ago

I was sure this was about some multiplayer game thing, but no, it's a crossword. Not entirely sure what they are even scaling here, I was expecting an article about a CDN..

jprob 9y ago
The game allows you to sync your game progress across multiple devices and it's subscription based so a CDN wouldn't do much help there. Also, realtime multiplayer crosswords are coming! I'll be speaking at GothamGo this year about that exact topic.
jonknee 9y ago
It’s a paid service that hundreds of thousands of people use. Not world changing stuff, but considering competitors are still using Java applets and Flash, I for one applaud their efforts.

bsaul 9y ago

Has anybody had successful experience deploying docker containers on appengine ? Last time i tried, i had such a bad experience in terms of deployment speed ( time to build the image, then upload it, then waiting for the stuff to deploy) that i reverted to managing my own gce instance. But maybe i had bad luck..

tlarkworthy 9y ago
Yes uploads are fast and you can spin up a VM directly from an image now
- ledgerdev 9y ago
  How many minutes/seconds do you mean by fast? Can you point to guidance on how to make that happen because I am seeing 8+ minute re-deploy times.
tarr11 9y ago
I just tried to move a rails app to appengine. It uses the flexible environment (eg, docker). It took like 10 or 15 minutes to deploy, each time. Heroku does my deploy in about 5 mins. https://groups.google.com/forum/#!topic/google-appengine/hZMEkmmObDU https://groups.google.com/forum/#!topic/google-appengine/hZM... GCloud is cheaper though. Also, VMs spin up in GCloud amazingly fast. Like 5 seconds. Feels like somebody at Gcloud just needs to go and fix this. No reason this is so bad...
- brianwawok 9y ago
  Deploy is 15 seconds on GKE. Wonder why app engine is so bad.
  - lftl 9y ago
    With GKE you don't necessarily update your load balancer rules with a deploy correct? The linked thread points the blame for app engine deploys on waiting for GCLB to update.

obulpathi 9y ago

Two things keep coming up while comparing GCP and AWS: * This accomplishment would not have been possible for our three-person team of engineers with out Google Cloud (AWS is too low level, hard to work with and does not scale well). * We’ve also managed to cut our infrastructure costs in half during this time period (Per minute billing, seamless autoscaling, performance, sustained usage discounts, ... )

fha 9y ago
> AWS . . . does not scale well GCP may have some advantages over AWS, but the reverse is also true and it's hard to take what you say seriously when you say something like that.
- CharlesW 9y ago
  Totally. Also… > AWS is too low level It seems very strange to paint AWS with such a broad brush, considering that AWS has tons of services at various levels of abstraction (including high-level abstractions like Elastic Beanstalk and AWS Lambda).
  - obulpathi 9y ago
    Sorry to not add context. I was referring to the use case the author of article was talking about: running a website: You need to stitch: ELB / EC2 / Database / Caching / Service Splitting / Auth / Scaling / ... Where as on Google Cloud, App Engine covers most of the points.
    - deleted 9y ago
      [deleted]
    - deleted 9y ago
      [deleted]
    - tomcam 9y ago
      Jeez, you're taking a lot of hits. Thanks for the patient answers

Jedi72 9y ago

Getting pro GCP articles to the top of HN must no-doubt be a high priority for the Google marketing team. This is the nature of modern advertising, sneakily trying to subvert your thinking by masquerading as something else.

ocdtrekkie 9y ago
Google likely offers significant discounts to companies which write these sorts of pieces for them. And obviously a lot of Googlers participate actively here, so getting upvotes naturally is likely not a challenge.
theDoug 9y ago
We actually spent the entire day at the Giants game today. ¯\(ツ)/¯ There's no incentive for high ranking HN posts, or any HN posts, actually. If there were, you wouldn't see others continually submit our news here before we do. This was a nice and unprompted post for everyone in GCP to read, as well. (Disclosure: I work on GCP as a product marketer.)
- ben_jones 9y ago
  A GCP product marketer responding to negative commentary on HN within ~20 minutes of it being posted strikes me as automated. EDIT: It is a double standard though, HN readers want access and responses from people on the GCP team but at the same time tinfoil hat subliminal marketing etc.
  - jacquesm 9y ago
    > A GCP product marketer responding to negative commentary on HN within ~20 minutes of it being posted strikes me as automated. Nonsense, there are many Googlers on HN and all it would take is a colleague poking another with a relevant thread.
  - sb8244 9y ago
    I would be reading this every 10 minutes for new comments if it was my product up for discussion.

nrjames 9y ago

I migrated a big data stack to GCP from AWS. Reasons: GCP has better documentation, the AWS console and various services confuse the heck out of me (I guess I'm getting too old), and the security integration between GCP services saves a huge amount of time. It's super easy and very fast to used the Google Compute Engine VMs. Given that the company I work for uses G Suite, it's a piece of cake to implement SSO and other integration pieces. It's also cheaper for us than AWS and more performant.

jsudhams 9y ago
Wait until GCP becomes older with bigquery and other versions. Also GCP does not have regions ready and it takes forever to get a new region. Waiting for Mumbai region for more than 3 to 4 months now. But one thing i like about GCP is that it allows to the limit setting in terms of cost and ensure you wont cross it. In case of AWS it can give alerts but for some reason say all you team in in one location and there is emergency like flood etc and you dont check email then you are done. I stopped using AWS after i learned that there is simply no way to set limit. Waiting for GCP to open their Mumbai region. sigh. Also AWS is very deceiving with free tier , there is simply no way to understand which products get free and worst case is after free tier you will get charged.
- yeukhon 9y ago
  AWS doesn't add a new region every 3-4 months either. Adding a new region is very complicated. Normally vendor does not actually build the DC themselves. They would source from other data centers in the region whenever possible. Building a new DC is not something can be taken lightly. Then finally local laws.

foxylad 9y ago

OT. How nice not see a single "avoid all Google services because reader" comment. Maybe we are finally moving on.

CaveTech 9y ago
Yet here you are.

zitterbewegung 9y ago

Nice advertisement that Google bought from the NyTimes.

NightlyDev 9y ago

This doesn't really make much sense to me. How many peak users are there? What's the number of requests per second? I can't imagine that the load would be so high that it wouldn't be possible to do it without GCP with three developers. It would be way more interesting with performance details. :)

ciguy 9y ago

As a DevOps consultant I've actually worked with clients migrating stacks to and from GCE/AWS (Yeah, both ways, not the same client). What I've found in aggregate is that GCE is a bit easier to use at first as AWS has a LOT of features and terminology to learn. When it comes down to it though, many GCE services felt really immature, particularly their CloudSQL offering. One client recently moved from GCE to AWS simply because their CloudSQL (Fully replicated with fail-over setup according to GCE recommendations) kept randomly dying for several minutes at a time. After a LOT of back and forth Google finally admitted that they had updated the replica and the master at the same time, so when it failed over the replica was also down. There were other instances of unexplained downtime that were never adequately explained, but overall that experience was enough for me (And the client) to totally lose faith in the GCE teams competence. Even getting a serious investigation into intermittent downtime and an explanation took over a month. By that time our migration to AWS was in progress. GCE never did explain why they would choose to apply updates to replica + master SQL at the same time and as far as I know they are still doing this. I asked if we could at least be notified of update events, was told that's not possible. There were other issues as well that taken together just made GCE seem amateurish. I'm sure as they mature a bit things will get better, and it is cheaper which is why I wouldn't necessarily recommend against them for startups just getting going today. By the time you are really scaling it's like they'll have more of the kinks worked out.

ringaroundthetx 9y ago
Yeah Azure is like that too. Stuff is just really immature and you'll get blocked by lots of things that make no sense. If you have to get vendor locked in your quest to not have to manage servers while learning unintuitively misnamed parts of a computer, choose Amazon.
- wutwutwutwut 9y ago
  I use Azure all the time (app services, storage, cloud services,VMs, SQL, CDN etc) and almost never run into this issue. Can you share some example on what you mean?

neya 9y ago

Hey community, let me share my experience with AppEngine. I work in a small firm, where we've developed a massive Software Application comprising of 12 medium-sized apps. I went with Phoenix 1.3 w/ the new umbrella architecture. With AppEngine, the beauty is that you can have many custom named microservices under one AppEngine project and each microservices can have many versions. You can even decide how much percentage of traffic should be split between each of these microservices. What's awesome is, in addition to the standard runtimes (Ruby, Python, Go, Java, etc.) Google also provides something called custom VMs for AppEngine, meaning you can push docker based setups into your AppEngine service, with basically any stack you want. This alone is a HUGE incentive to move to AppEngine because usually custom stack will require you to maintain the server side of things, but with Docker + AppEngine, zero devops. Their network panel is also very intuitive to add/delete rules to keep your app secured. I've been using AppEngine for over 4 years now and every time I tried a competitive offering (such as AWS Beanstalk, for example) I've only been disappointed. AppEngine is great for startups. For example, a lesser known feature within AppEngine is their real-time image processing service API. This allows you to scale/crop/resize images in real time and the service is offered free of charge (except for storage). Works really well for web applications with basic image manipulation requirements. https://cloud.google.com/appengine/docs/standard/python/images/ https://cloud.google.com/appengine/docs/standard/python/imag... The best part is, you call your image with specific parameters that'll do transformations on the fly. For example, <image url>/image.jpg?s=120 will return a 120px image. Appending -c will give you a cropped version, etc. I really hope to see AppEngine get more love from startups as it's a brilliant platform, much more performant than it's competitors' offerings. For example, I was previously a huge proponent of Heroku and upon comparing numbers, I realized AppEngine is way more performant (in my use case). I'm so glad we made the switch. If you're looking/considering to move to AppEngine, let me know here and I'll try my best to answer your questions.

smn1234 9y ago
The way you've delivered this "experience" makes you sound like you either work for Google or were asked to make a sponsored statement - for credits or $.
- 9y ago

kennethh 9y ago

Anyone know how much it cost to add a custom domain and SSL to AppEngine(Standard og Flexible)? I have been looking and not able to find out how much it cost.

iamgopal 9y ago
SSL is free. Custom domain also free.
- kennethh 9y ago
  That explains it, would be nice if they could write it down somewhere as a feature:)
  - rhodysurf 9y ago
    Its free for custom but only to use existing certs. You still have to set up the cert and such with your domain as you normally would. You also get the *.appengine.com domain with automatic SSL>
    - iamgopal 9y ago
      SSL cert provided by Google is free and is in private alpha stage. It's coming soon with GA.

evangeline04 9y ago

"Luckily, we at The Times recently decided to move all product development to the Google Cloud Platform" Great decision. Google Cloud is doing all the right things on their PaaS.

zachary_zemlak 9y ago

We use automatic scaling with warmup requests enabled and haven’t run into any issues when migrating traffic to newer versions. We also still use the “original” SDK which allows you to deploy and migrate traffic in two separate steps.

jody39 9y ago

I’ve noticed challenges with the time taken for the load balancing around AppEngine to reconfigure when pushing new releases. Did you have any trouble with that?

merb 9y ago

Uh thanks to this article I've seen that AppEngine know supports Java8, this is really really cool.