5 ms·

More of a rant. There's time you need more flexibility. I'm serving 1.2MM requests per second from 3 GCP regions, managing instances and GKE clusters with terr

More of a rant. There's time you need more flexibility. I'm serving 1.2MM requests per second from 3 GCP regions, managing instances and GKE clusters with terraform, and I cannot see how could I possibly set that up in a resilient fashion with DigitalOcean. I think DO is perfect for certain scale apps. You usually care about UI things mostly when you spin up couple servers; but when you operate hundreds of machines you need automation. GCP has it's quirks, e.g. * 130k connections/core limit due to conntrack, * lower networking throughput compared to AWS (16Gbps on GCP vs 25Gbps on AWS), * no support for enhanced networking (haven't tested recent Andromeda 2.1 yet, though) * no way to attach more than 8 local SSDs (arguably a good thing) * etc So does AWS, so does DO and you have to pick what's best for your project. One thing I like in general here is competition that makes all of those services better. EDIT: Fix conntrack typo

mluggy 9y ago
You do get additional 2-3TB per instance which alone is a couple hundred $$ in bandwidth savings. Will it cover the price of these extra instances? Obviously if you need 8 SSDs DO isn't for you. But the real question is - could GCP be more lean back and still provide the same level of service? I think so.
- user5994461 9y ago
  Anyone who count in a unit less than kilo dollar is a waste of time for AWS/Google. A big business doesn't care about "hundreds of dollars" of savings, it's like cents.
throwawayReply 9y ago
What do you do that serves 1.2mm requests / second? I don't disbelieve you, I'm just wondering what type of site that is since English Wikipedia is several orders of magnitude lower than that.
- FBISurveillance 9y ago
  Ads, one of the most shitshow industries in the world.
  - FBISurveillance 9y ago
    Forgot to add a joke: you can judge from my nickname! :-)
  - piva00 9y ago
    Can add to that, I worked for a few months in a small mobile ad network and the throughput of that was insane for the size of the company. Also a shitshow of a company on many levels.
  - ericnyamu 9y ago
    1.2m request per sec of ads looks like adblock isnt working .
    - jjeaff 9y ago
      Good news to me. Let everyone else keep finding all the free services. Everyone can't use adblock or nothing is free anymore.
  - dhimes 9y ago
    So Google built a platform that is excellent for running ad networks. It seems so obvious now.
    - user5994461 9y ago
      Ads are a super intensive workload with super low margin. It'd be more correct to say that google built a very efficient platform that can run anything. Well, anything easier than ad, so almost anything.
therealmarv 9y ago
so it seems in general you like GCP more than AWS? What is your opinion?
- FBISurveillance 9y ago
  Personally, not professionally, I like GCP more. Feedback cycle is (used to be) faster and it feels (used to feel) less enterprisey. I also liked the idea of a new player on the market, and wanted to make sure I vote for this diversity with dollars. I think these days GCP and AWS are more or less on par. One thing I learned the hard way: invest time into calculating your expected cloud spend. Your use case, partners you integrate with, your audience. Those impact significantly on cloud pricing.
  - mluggy 9y ago
    "used to" - correct. my thoughts exactly 1 year ago: https://news.ycombinator.com/item?id=12235003 https://news.ycombinator.com/item?id=12235003
  - FBISurveillance 9y ago
    Also, some GCP folks are hanging out here and it's nice to have some insights into platform internals.
    - sitepodmatt 9y ago
      Yep, pretty regularly I see "Disclaimer: I work for Google Cloud" on HN, with AWS and Azure I see a lot less, no solid metrics here. I also like I can get staff on a Google Groups for many of their products like StackDriver to share feedback and get a PM to respond eventually.
      - therealmarv 9y ago
        I'm sure there is some internal rule (contract/NDS) at AWS that you are not allowed to talk in any way outside of Amazon about AWS without supervision ;)
comboy 9y ago
It's interesting that 130k connections/core is limiting to you. Care to share some more details? Are those websocket connections? If yes what do you use as your backend? Something custom?
- FBISurveillance 9y ago
  Those mostly are slow user connections. Most of the direct user connections to our backends performed using slow mobile connectivity so request taking 4 seconds considered normal and that exhausts connection limit. Ended up serving most of the traffic from n1-standard-16 or lower.
bogomipz 9y ago
>"GCP has it's quirks, e.g. * 130k connections/core limit due to conntrack, * lower networking throughput compared to AWS (16Gbps on GCP vs 25Gbps on AWS)," Isn't the 130K limit only for core count < 8? From: https://cloud.google.com/vpc/docs/firewalls https://cloud.google.com/vpc/docs/firewalls 130000 per instance for instances with shared-core machine types 130000 per CPU for instances with 1 to 8 CPUs 130000 *8 (1040000) per instance for instances with > 8 CPUs

mluggy 9y ago

You do get additional 2-3TB per instance which alone is a couple hundred $$ in bandwidth savings. Will it cover the price of these extra instances? Obviously if you need 8 SSDs DO isn't for you. But the real question is - could GCP be more lean back and still provide the same level of service? I think so.

user5994461 9y ago
Anyone who count in a unit less than kilo dollar is a waste of time for AWS/Google. A big business doesn't care about "hundreds of dollars" of savings, it's like cents.

throwawayReply 9y ago

What do you do that serves 1.2mm requests / second? I don't disbelieve you, I'm just wondering what type of site that is since English Wikipedia is several orders of magnitude lower than that.

FBISurveillance 9y ago
Ads, one of the most shitshow industries in the world.
- FBISurveillance 9y ago
  Forgot to add a joke: you can judge from my nickname! :-)
- piva00 9y ago
  Can add to that, I worked for a few months in a small mobile ad network and the throughput of that was insane for the size of the company. Also a shitshow of a company on many levels.
- ericnyamu 9y ago
  1.2m request per sec of ads looks like adblock isnt working .
  - jjeaff 9y ago
    Good news to me. Let everyone else keep finding all the free services. Everyone can't use adblock or nothing is free anymore.
- dhimes 9y ago
  So Google built a platform that is excellent for running ad networks. It seems so obvious now.
  - user5994461 9y ago
    Ads are a super intensive workload with super low margin. It'd be more correct to say that google built a very efficient platform that can run anything. Well, anything easier than ad, so almost anything.

therealmarv 9y ago

so it seems in general you like GCP more than AWS? What is your opinion?

FBISurveillance 9y ago
Personally, not professionally, I like GCP more. Feedback cycle is (used to be) faster and it feels (used to feel) less enterprisey. I also liked the idea of a new player on the market, and wanted to make sure I vote for this diversity with dollars. I think these days GCP and AWS are more or less on par. One thing I learned the hard way: invest time into calculating your expected cloud spend. Your use case, partners you integrate with, your audience. Those impact significantly on cloud pricing.
- mluggy 9y ago
  "used to" - correct. my thoughts exactly 1 year ago: https://news.ycombinator.com/item?id=12235003 https://news.ycombinator.com/item?id=12235003
- FBISurveillance 9y ago
  Also, some GCP folks are hanging out here and it's nice to have some insights into platform internals.
  - sitepodmatt 9y ago
    Yep, pretty regularly I see "Disclaimer: I work for Google Cloud" on HN, with AWS and Azure I see a lot less, no solid metrics here. I also like I can get staff on a Google Groups for many of their products like StackDriver to share feedback and get a PM to respond eventually.
    - therealmarv 9y ago
      I'm sure there is some internal rule (contract/NDS) at AWS that you are not allowed to talk in any way outside of Amazon about AWS without supervision ;)

comboy 9y ago

It's interesting that 130k connections/core is limiting to you. Care to share some more details? Are those websocket connections? If yes what do you use as your backend? Something custom?

FBISurveillance 9y ago
Those mostly are slow user connections. Most of the direct user connections to our backends performed using slow mobile connectivity so request taking 4 seconds considered normal and that exhausts connection limit. Ended up serving most of the traffic from n1-standard-16 or lower.

bogomipz 9y ago

>"GCP has it's quirks, e.g. * 130k connections/core limit due to conntrack, * lower networking throughput compared to AWS (16Gbps on GCP vs 25Gbps on AWS)," Isn't the 130K limit only for core count < 8? From: https://cloud.google.com/vpc/docs/firewalls https://cloud.google.com/vpc/docs/firewalls 130000 per instance for instances with shared-core machine types 130000 per CPU for instances with 1 to 8 CPUs 130000 *8 (1040000) per instance for instances with > 8 CPUs