9 ms·

So piggybacking on this, I have a similar story to tell. We had a nice young startup, infra entirely built out on Google Cloud. Nicely, resiliently built, good

by aleph- 8y ago

So piggybacking on this, I have a similar story to tell. We had a nice young startup, infra entirely built out on Google Cloud. Nicely, resiliently built, good solid stuff. Because of a keyword monitor picked up by their auto-moderation bot our entire project was shut down immediately, wasn't able to bring it up for several hours, thank god we hadn't gone live yet as we were then told by support that because of the grey area of our tech, they couldn't guarantee this wouldn't keep happening. And in fact told us straight out that it would and we should move. So maybe think about which hosting provider to go with, don't get me wrong I like their tech. But their moderation does need a more human element, to be frank all their products do. Simply ceding control to algorithmic judgement just won't work in the short term if ever at all.

dewey 8y ago
> Because of a keyword monitor picked up by their auto-moderation bot Can you elaborate on that? What do they monitor with the moderation bot?
- asah 8y ago
  +1 - I'm curious as well.
- harryh 8y ago
  "because of the grey area of our tech" "told us straight out that it would and we should move" Sounds shady. I bet this would make more sense if OP explained what his company actually does.
  - golfer 8y ago
    Exactly -- there is a lot the OP isn't telling us. Maybe Google was right to shut them down.
    - __david__ 8y ago
      I'd rather my cloud provider err on the permissive side. Preemptively shutting suspicious things down without an external complaint seems a bit much…
    - nodesocket 8y ago
      Yup, agree. OP was probably doing something against the terms. Care to provide details?
    - justinclift 8y ago
      Well, there are all kinds of grey area stuff. One fairly obvious example is various security services, which have a wide variety. Not everything is outright "likely to get banned" (eg pron things). ;)
  - path411 8y ago
    Well, "grey" can mean a lot of things when you are talking about the same company that moderates Youtube.
- purerandomness 8y ago
  The point is that anyone could fall into that category when laws change. Imagine you're running a cosplay community, and all of a sudden all your content is being deleted because the SESTA/FOSTA bill gets passed in a country where your "cloud" happens to reside in: https://hardware.slashdot.org/story/18/03/25/0614209/sex-workers-say-porn-on-google-drive-is-suddenly-disappearing https://hardware.slashdot.org/story/18/03/25/0614209/sex-wor...
- pests 8y ago
  I know they specifically ban cryptocurrency mining on their free credit / tier. Even called out on their public product pages. I assumed they could tell that via CPU usage with they already monitor for quotas.
ma2rten 8y ago
What gray area are you in?
zAy0LfpBZLC8mAC 8y ago
> Nicely, resiliently built, good solid stuff. Erm ... no, evidently not?
- halbritt 8y ago
  Projects in the context of GCP can encompass all the necessary infrastructure to build a highly available service using standard practices. There's no indication anywhere from GCP themselves that a project could be a domain of failure. If asked, I doubt they would consider it as such. A prudent person might consider a cloud provider to be a domain of failure and choose a multi-cloud option, which would probably be the correct way to address this resiliency issue. However, that's not really an appropriate approach for an early stage startup, where availability is generally not that much of a concern.
  - zAy0LfpBZLC8mAC 8y ago
    In other words: It wasn't resiliently built stuff. Is an exploding car safe because it is built by an early stage startup? Just because you decide that implementing resiliency isn't a good business decision for some early stage startup, doesn't magically make the product resilient, it just isn't and that may be OK. There are many options to choose from for implementing resiliency, it could be having multiple providers concurrently, it could be having a plan for restoring service with a different provider in case one provider fails, it could be by setting up a contract with a sufficiently solvent provider that they pay for your damages if they fail to implement the resiliency that you need, whatever. But if you fail to consider an obvious failure mode of a central component of your system in your planning, then you are obviously not building a resilient system. Edit: One more thing: > There's no indication anywhere from GCP themselves that a project could be a domain of failure. If asked, I doubt they would consider it as such. Then you are asking wrong, which still is your failure if you are responsible for designing a resilient system. If you ask them "Is a complete project expected to fail at once?", of course they will say "no". That's why you ask them "Will you pay me 10 million bucks if my complete project goes offline with less than one month advance warning?", and you can be sure you will get the response to the problem that you are actually trying to solve.
  - 8y ago
halbritt 8y ago
This occasionally happens with gsuite users as well. Businesses lose access to all their email and documents. Good times.
AlexCoventry 8y ago
> the grey area of our tech Cryptocurrency?
- IloveHN84 8y ago
  Blockchain for sure
hartator 8y ago
I am really curious, what was the business?
setquk 8y ago
I’m starting to favour buying physical rack space again and running everything 2005 style with a light weight ansible layer. As long as your workload is predictable, the lock in, unpredictability, navigation through the maze of billing, weird rules and what-the-fuckism you have to deal with on a daily basis is merely trading one vendor specific hell for another. Your knowledge isn’t transferable between cloud vendors either so I’d rather have a hell I'm totally in control of and of which the knowledge has some retention value and will move around vendors no problems. You can also span vendors then thus avoiding the whole all eggs in one basket problem.
- TAForObvReasons 8y ago
  More importantly, there is a wealth of competent labor in the relatively stable area of maintaining physical servers (both on the hardware and software side). The modern cloud services move fast and break things, leading to a general shortage of resources and competent people. As a business, even if slightly more expensive initially, it makes more sense to start lower and work up to the cloud services as the need presents itself.
- Chris_Chambers 8y ago
  I feel sorry for cloud cucks. I’ve been bare metal only since 2003 and my life gets better with each passing day.
- bitL 8y ago
  Hybrid is what you are looking for. Have a rack or two for your core and rent everything else from multiple cloud vendors, integrated with whatever orchestration you are running on your own racks (K8s? DC/OS? Ansible?).
  - setquk 8y ago
    Or just two DCs in active/active. Still works out cheaper for workloads than AWS does even factoring staff in at this point. AWS always turns into cost and administrative chaos as well unless it is tightly controlled which in itself is costly and difficult the moment you have more than one actor. GCP probably the same but I have no experience with that. Very much more difficult to do this when you have physical constraints. Two man startup, perhaps but I think the transition should go: VPS (linode etc) for MVP, colo half rack, active/active racks two sites then scale out however your workload requires.
bitL 8y ago
I can tell a similar story with Amazon MWS, where even if we had access to "human support", it felt like talking to some bad ML, not understanding what we were saying. Ultimately that start up was disbanded, never violating any rule they had, but flagged because of a false positive, and we couldn't even prove we didn't violate anything because we didn't even go live yet. It felt Kafkaesque, punishing one of a myriad possible intents due to malfunctioning ML, with no recourse. Maybe support just needed to satisfy their quota of kicked out companies for the month, who knows?
- stickdogg 8y ago
  Lol. That's definitely a possibility.
kerng 8y ago
Thanks for sharing, I thought maybe it was a one off - it helps to avoid similar issues and luckily there is plenty of cloud competition.
pvg 8y ago
because of the grey area of our tech The nature of the tech in question seems important in this story.
S3raph 8y ago
Is it only me or does it seem if you are not a "famous" person that has a lot of public visibility and is able to create pressure through a tweet or blog post you are lost, no number to call, no mail to write. Over the years I saw a lot of similar stories, youtube or in general "google accounts" blocked for no clear reason and no way to contact somebody to solve the issue... kinda scary...
nodesocket 8y ago
I've gotta whole heartily disagree. I've never encountered this on GCE. I run a DevOps consulting company and for standard EC2/machines I much prefer GCP. It's not even close. AWS for the most part lacks any or little user experience testing on UI's and developer interfaces. AWS region specific resources are a nightmare, billing on GCP with sustained use and custom machine types is vastly superior. Disks are much easier to grok, no provisioned IOPS, EBS optimized, enhanced networking hoopla. By chance are you located out of the United States? These are not downtime issues, but anti-fraud prevention and finance issues.
- deerpig 8y ago
  I've noticed that over the last few years it's become increasingly difficult to do things with US based services (especially banking) if you are outside of the US. And this goes double if you are a US citizen with no ties to the States other than citizenship. Americans as a general rule have never been terribly adept at anything international; banking, languages, or even base geography. We have offices in Cambodia and Laos and I have been told by more than one US based-service/company that Laos is not a real country. I suppose they think the .la domain stands for Los Angeles :) We are looking to set up an office in Hong Kong or Singapore and use that to deal with Western countries. But we're a small not-for-profit operation and HK and Singapore are EXPENSIVE.
ucaetano 8y ago
Sounds like "we were doing something sketchy, got caught, but somehow it isn't our fault".
- ithkuil 8y ago
  It can happen to you too: you get hacked, hackers run arbitrary code in your account
  - ucaetano 8y ago
    If your cloud services account was hacked, you'd most likely be thanking Google or Amazon for stopping the services.
    - ithkuil 8y ago
      stopped yes, deleted the project if the photo id of the credit card account holder cannot be reached in 3 days might be an over-reaction though. I hope there is a possibility to put a backup contact person / credit card so organisations can deal with people going on vacation or being sick or whatever. IMHO this should be nicely documented as any other technical material you get to learn about the cloud product when you create an account (e.g. important steps to ensure your account remains open even in case of important security breaches, yadda yadda it's possible we'll need a way to prove that you are you yadda yadda, this can happen when yadda yadda, be prepared, do yadda yadda)
      - ucaetano 8y ago
        I agree that it seems like an over-reaction. But on an account with intense usage, a single credit card on file, no backup, and a fraud warning it does seem very suspicious. AFAIK, Google Cloud credit card payments are processed through Google Pay, which supports multiple credit cards, debit cards, bank accounts, etc. Ideally, in this case the company shouldn't be using the CFOs credit card, but entered into a payments agreement with Google, receiving POs, invoices and so on, including a credit line. Never set up a crucial service like you'd set up a consumer service.
rullelito 8y ago
Was it porn or cryptos?

by aleph- 8y ago

dewey 8y ago

> Because of a keyword monitor picked up by their auto-moderation bot Can you elaborate on that? What do they monitor with the moderation bot?

asah 8y ago
+1 - I'm curious as well.
harryh 8y ago
"because of the grey area of our tech" "told us straight out that it would and we should move" Sounds shady. I bet this would make more sense if OP explained what his company actually does.
- golfer 8y ago
  Exactly -- there is a lot the OP isn't telling us. Maybe Google was right to shut them down.
  - __david__ 8y ago
    I'd rather my cloud provider err on the permissive side. Preemptively shutting suspicious things down without an external complaint seems a bit much…
  - nodesocket 8y ago
    Yup, agree. OP was probably doing something against the terms. Care to provide details?
  - justinclift 8y ago
    Well, there are all kinds of grey area stuff. One fairly obvious example is various security services, which have a wide variety. Not everything is outright "likely to get banned" (eg pron things). ;)
- path411 8y ago
  Well, "grey" can mean a lot of things when you are talking about the same company that moderates Youtube.
purerandomness 8y ago
The point is that anyone could fall into that category when laws change. Imagine you're running a cosplay community, and all of a sudden all your content is being deleted because the SESTA/FOSTA bill gets passed in a country where your "cloud" happens to reside in: https://hardware.slashdot.org/story/18/03/25/0614209/sex-workers-say-porn-on-google-drive-is-suddenly-disappearing https://hardware.slashdot.org/story/18/03/25/0614209/sex-wor...
pests 8y ago
I know they specifically ban cryptocurrency mining on their free credit / tier. Even called out on their public product pages. I assumed they could tell that via CPU usage with they already monitor for quotas.

ma2rten 8y ago

What gray area are you in?

zAy0LfpBZLC8mAC 8y ago

> Nicely, resiliently built, good solid stuff. Erm ... no, evidently not?

halbritt 8y ago
Projects in the context of GCP can encompass all the necessary infrastructure to build a highly available service using standard practices. There's no indication anywhere from GCP themselves that a project could be a domain of failure. If asked, I doubt they would consider it as such. A prudent person might consider a cloud provider to be a domain of failure and choose a multi-cloud option, which would probably be the correct way to address this resiliency issue. However, that's not really an appropriate approach for an early stage startup, where availability is generally not that much of a concern.
- zAy0LfpBZLC8mAC 8y ago
  In other words: It wasn't resiliently built stuff. Is an exploding car safe because it is built by an early stage startup? Just because you decide that implementing resiliency isn't a good business decision for some early stage startup, doesn't magically make the product resilient, it just isn't and that may be OK. There are many options to choose from for implementing resiliency, it could be having multiple providers concurrently, it could be having a plan for restoring service with a different provider in case one provider fails, it could be by setting up a contract with a sufficiently solvent provider that they pay for your damages if they fail to implement the resiliency that you need, whatever. But if you fail to consider an obvious failure mode of a central component of your system in your planning, then you are obviously not building a resilient system. Edit: One more thing: > There's no indication anywhere from GCP themselves that a project could be a domain of failure. If asked, I doubt they would consider it as such. Then you are asking wrong, which still is your failure if you are responsible for designing a resilient system. If you ask them "Is a complete project expected to fail at once?", of course they will say "no". That's why you ask them "Will you pay me 10 million bucks if my complete project goes offline with less than one month advance warning?", and you can be sure you will get the response to the problem that you are actually trying to solve.
- 8y ago

halbritt 8y ago

This occasionally happens with gsuite users as well. Businesses lose access to all their email and documents. Good times.

AlexCoventry 8y ago

> the grey area of our tech Cryptocurrency?

IloveHN84 8y ago
Blockchain for sure

hartator 8y ago

I am really curious, what was the business?

setquk 8y ago

I’m starting to favour buying physical rack space again and running everything 2005 style with a light weight ansible layer. As long as your workload is predictable, the lock in, unpredictability, navigation through the maze of billing, weird rules and what-the-fuckism you have to deal with on a daily basis is merely trading one vendor specific hell for another. Your knowledge isn’t transferable between cloud vendors either so I’d rather have a hell I'm totally in control of and of which the knowledge has some retention value and will move around vendors no problems. You can also span vendors then thus avoiding the whole all eggs in one basket problem.

TAForObvReasons 8y ago
More importantly, there is a wealth of competent labor in the relatively stable area of maintaining physical servers (both on the hardware and software side). The modern cloud services move fast and break things, leading to a general shortage of resources and competent people. As a business, even if slightly more expensive initially, it makes more sense to start lower and work up to the cloud services as the need presents itself.
Chris_Chambers 8y ago
I feel sorry for cloud cucks. I’ve been bare metal only since 2003 and my life gets better with each passing day.
bitL 8y ago
Hybrid is what you are looking for. Have a rack or two for your core and rent everything else from multiple cloud vendors, integrated with whatever orchestration you are running on your own racks (K8s? DC/OS? Ansible?).
- setquk 8y ago
  Or just two DCs in active/active. Still works out cheaper for workloads than AWS does even factoring staff in at this point. AWS always turns into cost and administrative chaos as well unless it is tightly controlled which in itself is costly and difficult the moment you have more than one actor. GCP probably the same but I have no experience with that. Very much more difficult to do this when you have physical constraints. Two man startup, perhaps but I think the transition should go: VPS (linode etc) for MVP, colo half rack, active/active racks two sites then scale out however your workload requires.

bitL 8y ago

I can tell a similar story with Amazon MWS, where even if we had access to "human support", it felt like talking to some bad ML, not understanding what we were saying. Ultimately that start up was disbanded, never violating any rule they had, but flagged because of a false positive, and we couldn't even prove we didn't violate anything because we didn't even go live yet. It felt Kafkaesque, punishing one of a myriad possible intents due to malfunctioning ML, with no recourse. Maybe support just needed to satisfy their quota of kicked out companies for the month, who knows?

stickdogg 8y ago
Lol. That's definitely a possibility.

kerng 8y ago

Thanks for sharing, I thought maybe it was a one off - it helps to avoid similar issues and luckily there is plenty of cloud competition.

pvg 8y ago

because of the grey area of our tech The nature of the tech in question seems important in this story.

S3raph 8y ago

Is it only me or does it seem if you are not a "famous" person that has a lot of public visibility and is able to create pressure through a tweet or blog post you are lost, no number to call, no mail to write. Over the years I saw a lot of similar stories, youtube or in general "google accounts" blocked for no clear reason and no way to contact somebody to solve the issue... kinda scary...

nodesocket 8y ago

I've gotta whole heartily disagree. I've never encountered this on GCE. I run a DevOps consulting company and for standard EC2/machines I much prefer GCP. It's not even close. AWS for the most part lacks any or little user experience testing on UI's and developer interfaces. AWS region specific resources are a nightmare, billing on GCP with sustained use and custom machine types is vastly superior. Disks are much easier to grok, no provisioned IOPS, EBS optimized, enhanced networking hoopla. By chance are you located out of the United States? These are not downtime issues, but anti-fraud prevention and finance issues.

deerpig 8y ago
I've noticed that over the last few years it's become increasingly difficult to do things with US based services (especially banking) if you are outside of the US. And this goes double if you are a US citizen with no ties to the States other than citizenship. Americans as a general rule have never been terribly adept at anything international; banking, languages, or even base geography. We have offices in Cambodia and Laos and I have been told by more than one US based-service/company that Laos is not a real country. I suppose they think the .la domain stands for Los Angeles :) We are looking to set up an office in Hong Kong or Singapore and use that to deal with Western countries. But we're a small not-for-profit operation and HK and Singapore are EXPENSIVE.

ucaetano 8y ago

Sounds like "we were doing something sketchy, got caught, but somehow it isn't our fault".

ithkuil 8y ago
It can happen to you too: you get hacked, hackers run arbitrary code in your account
- ucaetano 8y ago
  If your cloud services account was hacked, you'd most likely be thanking Google or Amazon for stopping the services.
  - ithkuil 8y ago
    stopped yes, deleted the project if the photo id of the credit card account holder cannot be reached in 3 days might be an over-reaction though. I hope there is a possibility to put a backup contact person / credit card so organisations can deal with people going on vacation or being sick or whatever. IMHO this should be nicely documented as any other technical material you get to learn about the cloud product when you create an account (e.g. important steps to ensure your account remains open even in case of important security breaches, yadda yadda it's possible we'll need a way to prove that you are you yadda yadda, this can happen when yadda yadda, be prepared, do yadda yadda)
    - ucaetano 8y ago
      I agree that it seems like an over-reaction. But on an account with intense usage, a single credit card on file, no backup, and a fraud warning it does seem very suspicious. AFAIK, Google Cloud credit card payments are processed through Google Pay, which supports multiple credit cards, debit cards, bank accounts, etc. Ideally, in this case the company shouldn't be using the CFOs credit card, but entered into a payments agreement with Google, receiving POs, invoices and so on, including a credit line. Never set up a crucial service like you'd set up a consumer service.

rullelito 8y ago

Was it porn or cryptos?