5 ms·

"Due to the inelastic architecture of our AWS system, we needed to have the systems scaled up to handle our peak traffic at 10PM when the daily puzzle is publis

by vs2 9y ago

"Due to the inelastic architecture of our AWS system, we needed to have the systems scaled up to handle our peak traffic at 10PM when the daily puzzle is published." WT... I had to reread this to make sure I didnt misunderstand... why not work on making the current arhictecture elastic?! #cloudPorn

bpicolo 9y ago
Imo, one really killer bit is the first piece they mentioned: "Google provides an SDK that enables users to run a suite of services along with an admin interface, database and caching layer with a single command." I really wish AWS had a decent local dev story, rather than relying on 10 separate half-baked OSS solutions
- rstupek 9y ago
  Have you tried the documentation for google's sdk? I stopped in disgust trying to track down what I needed and then trying to manage through multiple different admin interfaces.
  - bpicolo 9y ago
    I've been using google bits recently. I agree that the docs for e.g. all the python sdks need a lot of work. That said, boto3 has thorough docs, but I wouldn't consider them particularly well organized. I can only really navigate AWS sdk docs because I already know what I want to do, and can google the specific terminology
    - CobrastanJorji 9y ago
      Are you talking about this Python SDK doc? https://googlecloudplatform.github.io/google-cloud-python/latest/ https://googlecloudplatform.github.io/google-cloud-python/la...
      - bpicolo 9y ago
        Yeah. In particular, I was really looking for something for "here's how to do most of the basics for Cloud Storage". If you look at https://googlecloudplatform.github.io/google-cloud-python/latest/storage/buckets.html https://googlecloudplatform.github.io/google-cloud-python/la..., for example, there's not even a top level navigation index that I can read through to guess what function I might need by name.
        funkjunky 9y ago
        (Google cloud support here) The pages you linked to are supposed to serve as client libraries reference only. If you want higher level instructions and examples, always start with the main Google Cloud docs first. On any page that offers instructions, the top of the code windows offers a selection of client library languages/CLI tools/REST API available to do whatever that task is. For cloud storage, start here: https://cloud.google.com/storage/docs/how-to https://cloud.google.com/storage/docs/how-to Choose a topic, and select "Python" at the top. It should provide instructions and examples using the Python libraries. Also, we have a repo of demo projects and examples for nearly every GCP product/service, and then some. Great examples to be found here (some might be out of date though): https://github.com/GoogleCloudPlatform/python-docs-samples https://github.com/GoogleCloudPlatform/python-docs-samples
        bpicolo 9y ago
        Heya, Thanks for the link! You're right that this is what I was looking for. Unfortunately, that hadn't shown up in a convenient place while I was googling around. Would be good to add direct links to those from the client lib references, because those pop up for e.g. "google storage python" first. (Unless they're already there and I didn't see them). Links from the READMEs of the repo might be useful too to help people get there? https://github.com/GoogleCloudPlatform/google-cloud-python https://github.com/GoogleCloudPlatform/google-cloud-python
        funkjunky 9y ago
        That's not a bad idea, I'll bring it up with our client library maintainers today.
tyingq 9y ago
The "inelastic" might have been a shot at AWS. When pressed, the AWS people do use phrases like "pre-warming", "over provisioning" and "advance notice" around their ELB/ALB setup and ECS. Google's cloud salespeople pitch that they don't require any of that.
- g09980 9y ago
  Curious if the need for pre-warming ELB/ALB still applies. Last time this came up, an AWS employee mentioned it is no longer necessary (https://news.ycombinator.com/item?id=14052079 https://news.ycombinator.com/item?id=14052079), but would be nice if this was documented.
  - tyingq 9y ago
    Possible. It is still in the documentation: https://aws.amazon.com/articles/1636185810492479#pre-warming https://aws.amazon.com/articles/1636185810492479#pre-warming The "advance notice" and "over provision" advice is still being given for things that could scale up fairly large. (where fairly large isn't anything that exciting, really)
  - callalex 9y ago
    I don't want to dox myself but about a year ago when my employer forgot to notify AWS about switching our production traffic (about 5K rps at that time) from one ELB to another we failed requests for several minutes before we decided to just switch back to the old ELB then ask them to do a prewarming before we switched again.
  - gtaylor 9y ago
    It definitely still applies.
  - PaulRobinson 9y ago
    The poster was talking about ALBs. I'm not experienced enough with them to know whether the claim is true. I do know on ELBs though, pre-warming is essential for high throughput.
- 9y ago
tziki 9y ago
The main story here is probably this line: "The system is generally at that peak traffic for only a few minutes a day". AWS charges by hour. Google by minute. If your peak traffic only last a few minutes, hourly billing is inflexible, simple as that.
- random023987 9y ago
  AWS lambda charges by the 1/10th of a GB/CPU/second. I'd bet they could have seen significant cost savings on AWS by migrating to lambda, and gotten continuous scaling to boot.
  - tziki 9y ago
    Serverless is probably far too immature for companies like NYT.
    - bastawhiz 9y ago
      I don't disagree with you, but it's also true that you don't have to go all-in. I run services that run server-ful-ly on EC2, while requests to the "front-end" are handled by Lambda. If you've got one or two very hot endpoints and the majority of your server load spike is thanks to the processing for those endpoints, moving just those endpoints to Lambda could give you reassurance that you don't have all of your eggs in one basket (not having to go full-serverless) while also getting the benefit of being able to scale up and down essentially effortlessly. Of course, there are lots of other consideration that you should be making when going even partially serverless, and it could be that the NYT chose not to experiment with Lambda for other reasons. For instance, you're effectively tied to CloudWatch for logging and monitoring, which could be a deal-breaker. Much of the processing could be happening in the DB, which would make Lambda moot. It may simply be that their estimated usage of Lambda was too costly.
      - sdenton4 9y ago
        Three devs seems like a bit of a bottleneck... Why maintain two separate architectures for a single problem if there's another option?
mattbillenstein 9y ago
I'm not sure why people are so enamored with ELB -- just terminate SSL at your web boxes using nginx and publish the public ips of all of these machines in your DNS records. You remove a bunch of ELB per-request costs doing it this way and you can scale it however you see fit.
- bastawhiz 9y ago
  That works great until a machine starts failing health checks and you need to take it out of rotation ASAP. It's also the case that DNS gets cached and not all users create the same load: one user making many requests will burden one server instead of having the load evenly distributed.
  - mattbillenstein 9y ago
    Clients will try another ip if they can't connect -- partial failures may be a problem, but in my experience as long as nginx is alive, you can load balance to a different web backend if the app processes on that machine are wedged. I've deployed this solution in 50k req/s environments and not seen a single user be a problem like you mention -- any motivated bad actor could cause problems in either scenario I expect.
    - bastawhiz 9y ago
      Clients might not fail to connect. That's even worse. They connect and the server hangs and returns no response (perhaps due to bad configuration changes). Now you're stuck, and your server is too oversaturated to SSH in and fix it. It out depends on your application and users. Building a website? Probably not much of an issue. Building a low-latency API? YMMV. Keeping your load evenly balanced across your front end cluster also can keep your cost low, since you are able to distribute load more evenly. That's another point: if you scale your cluster size up and down frequently to accommodate load, doing that with DNS is a nightmare.
  - awj 9y ago
    > It's also the case that DNS gets cached and not all users create the same load It's also, also the case that DNS gets cached and propagating the removal of a broken server could take ages.

by vs2 9y ago

bpicolo 9y ago

Imo, one really killer bit is the first piece they mentioned: "Google provides an SDK that enables users to run a suite of services along with an admin interface, database and caching layer with a single command." I really wish AWS had a decent local dev story, rather than relying on 10 separate half-baked OSS solutions

rstupek 9y ago
Have you tried the documentation for google's sdk? I stopped in disgust trying to track down what I needed and then trying to manage through multiple different admin interfaces.
- bpicolo 9y ago
  I've been using google bits recently. I agree that the docs for e.g. all the python sdks need a lot of work. That said, boto3 has thorough docs, but I wouldn't consider them particularly well organized. I can only really navigate AWS sdk docs because I already know what I want to do, and can google the specific terminology
  - CobrastanJorji 9y ago
    Are you talking about this Python SDK doc? https://googlecloudplatform.github.io/google-cloud-python/latest/ https://googlecloudplatform.github.io/google-cloud-python/la...
    - bpicolo 9y ago
      Yeah. In particular, I was really looking for something for "here's how to do most of the basics for Cloud Storage". If you look at https://googlecloudplatform.github.io/google-cloud-python/latest/storage/buckets.html https://googlecloudplatform.github.io/google-cloud-python/la..., for example, there's not even a top level navigation index that I can read through to guess what function I might need by name.
      - funkjunky 9y ago
        (Google cloud support here) The pages you linked to are supposed to serve as client libraries reference only. If you want higher level instructions and examples, always start with the main Google Cloud docs first. On any page that offers instructions, the top of the code windows offers a selection of client library languages/CLI tools/REST API available to do whatever that task is. For cloud storage, start here: https://cloud.google.com/storage/docs/how-to https://cloud.google.com/storage/docs/how-to Choose a topic, and select "Python" at the top. It should provide instructions and examples using the Python libraries. Also, we have a repo of demo projects and examples for nearly every GCP product/service, and then some. Great examples to be found here (some might be out of date though): https://github.com/GoogleCloudPlatform/python-docs-samples https://github.com/GoogleCloudPlatform/python-docs-samples
        bpicolo 9y ago
        Heya, Thanks for the link! You're right that this is what I was looking for. Unfortunately, that hadn't shown up in a convenient place while I was googling around. Would be good to add direct links to those from the client lib references, because those pop up for e.g. "google storage python" first. (Unless they're already there and I didn't see them). Links from the READMEs of the repo might be useful too to help people get there? https://github.com/GoogleCloudPlatform/google-cloud-python https://github.com/GoogleCloudPlatform/google-cloud-python
        funkjunky 9y ago
        That's not a bad idea, I'll bring it up with our client library maintainers today.

tyingq 9y ago

The "inelastic" might have been a shot at AWS. When pressed, the AWS people do use phrases like "pre-warming", "over provisioning" and "advance notice" around their ELB/ALB setup and ECS. Google's cloud salespeople pitch that they don't require any of that.

g09980 9y ago
Curious if the need for pre-warming ELB/ALB still applies. Last time this came up, an AWS employee mentioned it is no longer necessary (https://news.ycombinator.com/item?id=14052079 https://news.ycombinator.com/item?id=14052079), but would be nice if this was documented.
- tyingq 9y ago
  Possible. It is still in the documentation: https://aws.amazon.com/articles/1636185810492479#pre-warming https://aws.amazon.com/articles/1636185810492479#pre-warming The "advance notice" and "over provision" advice is still being given for things that could scale up fairly large. (where fairly large isn't anything that exciting, really)
- callalex 9y ago
  I don't want to dox myself but about a year ago when my employer forgot to notify AWS about switching our production traffic (about 5K rps at that time) from one ELB to another we failed requests for several minutes before we decided to just switch back to the old ELB then ask them to do a prewarming before we switched again.
- gtaylor 9y ago
  It definitely still applies.
- PaulRobinson 9y ago
  The poster was talking about ALBs. I'm not experienced enough with them to know whether the claim is true. I do know on ELBs though, pre-warming is essential for high throughput.
9y ago

tziki 9y ago

The main story here is probably this line: "The system is generally at that peak traffic for only a few minutes a day". AWS charges by hour. Google by minute. If your peak traffic only last a few minutes, hourly billing is inflexible, simple as that.

random023987 9y ago
AWS lambda charges by the 1/10th of a GB/CPU/second. I'd bet they could have seen significant cost savings on AWS by migrating to lambda, and gotten continuous scaling to boot.
- tziki 9y ago
  Serverless is probably far too immature for companies like NYT.
  - bastawhiz 9y ago
    I don't disagree with you, but it's also true that you don't have to go all-in. I run services that run server-ful-ly on EC2, while requests to the "front-end" are handled by Lambda. If you've got one or two very hot endpoints and the majority of your server load spike is thanks to the processing for those endpoints, moving just those endpoints to Lambda could give you reassurance that you don't have all of your eggs in one basket (not having to go full-serverless) while also getting the benefit of being able to scale up and down essentially effortlessly. Of course, there are lots of other consideration that you should be making when going even partially serverless, and it could be that the NYT chose not to experiment with Lambda for other reasons. For instance, you're effectively tied to CloudWatch for logging and monitoring, which could be a deal-breaker. Much of the processing could be happening in the DB, which would make Lambda moot. It may simply be that their estimated usage of Lambda was too costly.
    - sdenton4 9y ago
      Three devs seems like a bit of a bottleneck... Why maintain two separate architectures for a single problem if there's another option?

mattbillenstein 9y ago

I'm not sure why people are so enamored with ELB -- just terminate SSL at your web boxes using nginx and publish the public ips of all of these machines in your DNS records. You remove a bunch of ELB per-request costs doing it this way and you can scale it however you see fit.

bastawhiz 9y ago
That works great until a machine starts failing health checks and you need to take it out of rotation ASAP. It's also the case that DNS gets cached and not all users create the same load: one user making many requests will burden one server instead of having the load evenly distributed.
- mattbillenstein 9y ago
  Clients will try another ip if they can't connect -- partial failures may be a problem, but in my experience as long as nginx is alive, you can load balance to a different web backend if the app processes on that machine are wedged. I've deployed this solution in 50k req/s environments and not seen a single user be a problem like you mention -- any motivated bad actor could cause problems in either scenario I expect.
  - bastawhiz 9y ago
    Clients might not fail to connect. That's even worse. They connect and the server hangs and returns no response (perhaps due to bad configuration changes). Now you're stuck, and your server is too oversaturated to SSH in and fix it. It out depends on your application and users. Building a website? Probably not much of an issue. Building a low-latency API? YMMV. Keeping your load evenly balanced across your front end cluster also can keep your cost low, since you are able to distribute load more evenly. That's another point: if you scale your cluster size up and down frequently to accommodate load, doing that with DNS is a nightmare.
- awj 9y ago
  > It's also the case that DNS gets cached and not all users create the same load It's also, also the case that DNS gets cached and propagating the removal of a broken server could take ages.