7 ms·

Disclosure: I work on Google Cloud (but I'm not in SRE, oncall, etc.). As the updates to [1] say, we're working to resolve a networking issue. The Region isn't

by boulos 7y ago

Disclosure: I work on Google Cloud (but I'm not in SRE, oncall, etc.). As the updates to [1] say, we're working to resolve a networking issue. The Region isn't (and wasn't) "down", but obviously network latency spiking up for external connectivity is bad. We are currently experiencing an issue with a subset of the fiber paths that supply the region. We're working on getting that restored. In the meantime, we've removed almost all Google.com traffic out of the Region to prefer GCP customers. That's why the latency increase is subsiding, as we're freeing up the fiber paths by shedding our traffic. Edit: (since it came up) that also means that if you’re using GCLB and have other healthy Regions, it will rebalance to avoid this congestion/slowdown automatically. That seemed the better trade off given the reduced network capacity during this outage. [1] https://status.cloud.google.com/incident/cloud-networking/19016 https://status.cloud.google.com/incident/cloud-networking/19...

deleted 7y ago
[deleted]
ricardobeat 7y ago
Tangential question: does Google allow employees, not directly tasked with it, to represent the company online as they wish? Most companies I know of have a strict ‘do not speak for the company’ policy.
- jauer 7y ago
  It's probably less "as they wish" and more "here's an approved statement" or "your role involves engaging with external parties, here are some guidelines"
- kyrra 7y ago
  It's a fine line. We are not allowed to represent Google in any kind of public discussion. But we can talk about some things we do, as long as we state it's our own opinion and we don't represent Google's views.
  - IX-103 7y ago
    And don't disclose material nonpublic information (since that would run afoul of insider trading laws). It's probably okay to say that we know the problem and here are the steps we're taking to mitigate it. It would not be okay to say something with large scale stock price implications for Google it another publicly traded corporation. For instance a Google employee shouldn't say something like "faulty solar panels fried Google's 10 largest data centers and twelve others have been lost to rebel drone strikes", even if false, since it could have a drastic impact on the earnings and future value of Google, Google's customers, and Google's competitors. Even less obvious things like Google's plans for adding privacy features to the Chromium open source project can have a serious impact (see https://www.barrons.com/articles/google-chrome-privacy-questions-51553871811 https://www.barrons.com/articles/google-chrome-privacy-quest...).
    - titanomachy 7y ago
      I'm not a lawyer, but if the information is false I don't think you could get dinged for insider trading. The legal approach that's used to prosecute insider trading is basically "theft of secrets".
- boulos 7y ago
  As kyrra says below, you're in the clear if you state that this is just your opinion. Naturally, prefacing something terrible as "just your opinion" doesn't make it fine. In my case, Cloud PR knows me, but I also knowingly risk my job (I clearly believe I have good enough judgment in what I post). If Urs and Ben think I should be fired, I'm okay with that, as it would represent a significant enough difference in opinion, that I wouldn't want to continue working here anyway. Finally, for what it's worth, I have been reported before for "leaking internal secrets" here on HN! It turned out to be a totally hilarious discussion with the person tasked with questioning me. Still not fired, gotta try harder :).
  - kyrra 7y ago
    To add my own story. I have made comments about other teams services on hacker news before. I've been contacted by the SRE responsible for the service I commented on asking me to correct what I said. Luckily no reports for leaking info. :) Whenever I talk about the inner workings of Google I try to reference to external talks, books, or white papers to go along with my comments. Luckily a lot has already been set externally about how Google works.
  - bardworx 7y ago
    That’s...that’s some petty fucking shit. I didn’t go through your comments but considering your email is in your profile, someone really had to have a hard-on to report you for leaks. I would love to understand the though process of someone going out of their way to remove someone’s livelihood from them because of a comment on HN (when applied in a normal circumstance of adding additional information or correcting a misconception — I’m clearly not saying that bonehead comments shouldn’t have consequences.)
    - deleted 7y ago
      [deleted]
    - praneshp 7y ago
      > I would love to understand the though process of someone going out of their way to remove someone’s livelihood from them because of a comment on HN This is not so difficult though. You just need to adjust your starting point to someone who doesn't like boulos' first. That's not so difficult IMO, it's a large org and boulos' seems to be a fairly prolific commenter here.
- user5994461 7y ago
  Google employees are commenting publicly and on Hacker News all the time. If there is a policy of not speaking publicly about the company, this has been the most blatantly ignored policy ever.
  - akhilcacharya 7y ago
    I’m 90% sure it’s just to flex, honestly. I work at another FANG with a roughly equal engineering community and I don’t see my kind commenting as much at all!
    - laluser 7y ago
      Another FANG = Amazon? If so, Amazon is pretty restrictive in how it wants employees to communicate about internal activities. Most people err on the side of caution and don't comment publicly.
      - akhilcacharya 7y ago
        It is - but all companies I’ve ever worked for are. I’m not convinced the letter of the policy is much different.
        9nGQluzmnq3M 7y ago
        There are definitely major differences between the FAANGs: what was the last time you saw an Apple employee commenting on anything on HN?
        zaphirplane 7y ago
        Definitely not all FADANGs are the same. never seen a Disney employee comment ;) Or oracle in FADANGOs ;) just kidding
        saagarjha 7y ago
        Apple employees comment on Hacker News all the time: they just don’t identify themselves as speaking for the company and make sure to only talk about publicly available information.
- munificent 7y ago
  I work at Google on an open source project and comment on it frequently. One of the things I really like about working at Google is that they place a lot of trust in the judgement of the individual employees. I generally make it clear when I'm stating my personal opinion versus the "official" (for whatever that means given how informal the project is) one, but I don't have to carefully go through an approved list of talking points, run my HN by the legal department, etc. Obviously, in certain situations, things get more official and formal. For example, when I went to Google IO to give a talk, we did have some documentation and coaching beforehand about how to handle various questions we might get about non-public stuff, other projects related to ours, etc. We are also expected to run any slides by legal before being publicly shown in a venue with a wide audience like IO. But, even then, the legal folks I've worked with have been a pleasure to talk to. The company's culture is basically "We hired you because you're smart. We trust you to use your brain." It would be squandering resources to not let their employees use their own intelligence and judgement.
  - cameronbrown 7y ago
    Off-topic, but I noticed in your bio you wrote Game Programming Patterns. Was a great read!
    - danso 7y ago
      Also off-topic: am looking forward to the finishing of craftinginterpreters.com, which has been a fantastic read so far
    - munificent 7y ago
      Thank you!
edwintorok 7y ago
You seem to have 3 status messages on the dashboard at 14:31, 14:44 and 14:48 with exactly the same contents. Were those messages really posted 3 times, or did something go wrong and they got duplicated?
- FrankPetrilli 7y ago
  We're aware this happened - that posting is the responsibility of an adjacent team to my own, specifically the person right next to me. :)
walshemj 7y ago
Sounds like back hoe fade (from the write up) and it sounds like multiple cables sharing the same physical route got taken out.
deleted 7y ago
[deleted]
mrweasel 7y ago
>The Region isn't (and wasn't) "down", but obviously network latency spiking up for external connectivity is bad. As one of my old bosses said: I don't care that the site/service is technically running, if the customers can't reach it, then IT'S DOWN.
- jodrellblank 7y ago
  Your boss picked a ridiculous time to nitpick over wording, to shout and add stress to an already difficult situation, and giving up accuracy and precise understanding at a time those are most important.
  - Voxoff 7y ago
    "to shout and add stress to an already difficult situation" now that's accuracy
  - mrweasel 7y ago
    I don’t think he’s the one nitpicking. From a business perspective the site was down. Nitpicking is telling him: No it is in fact up, the customer just can’t use it.
    - jodrellblank 7y ago
      "Customers are complaining they can't access a thing" "OK, we have databases up, load balancers responding, DNS records check out, last change/deployment was at this time, all these services are up, and the latest test suite is running all green, this narrows down the places where a failure might be with some useful differential diagnosis, now we can move attention to.." "I DON'T CARE THAT YOU THINK THINGS ARE WORKING, IF THE CUSTOMER CANNOT GET TO IT, IT'S DOWN" "Thanks for that helpful input, let's divert troubleshooting attention from this P1 incident, and have a discussion about what "DOWN" means. You want me to treat the working databases as down because the customer can't get to them? Even though they're working? It's like the hatred for "works on my machine". "WELL I'M NOT RUNNING ON YOUR MACHINE". No you aren't, but this demonstrates the current build works, the commands you're using are coherent and sensible, excludes many possible causes of failure, and adds useful information to the situation.

by boulos 7y ago

deleted 7y ago

[deleted]

ricardobeat 7y ago

Tangential question: does Google allow employees, not directly tasked with it, to represent the company online as they wish? Most companies I know of have a strict ‘do not speak for the company’ policy.

jauer 7y ago
It's probably less "as they wish" and more "here's an approved statement" or "your role involves engaging with external parties, here are some guidelines"
kyrra 7y ago
It's a fine line. We are not allowed to represent Google in any kind of public discussion. But we can talk about some things we do, as long as we state it's our own opinion and we don't represent Google's views.
- IX-103 7y ago
  And don't disclose material nonpublic information (since that would run afoul of insider trading laws). It's probably okay to say that we know the problem and here are the steps we're taking to mitigate it. It would not be okay to say something with large scale stock price implications for Google it another publicly traded corporation. For instance a Google employee shouldn't say something like "faulty solar panels fried Google's 10 largest data centers and twelve others have been lost to rebel drone strikes", even if false, since it could have a drastic impact on the earnings and future value of Google, Google's customers, and Google's competitors. Even less obvious things like Google's plans for adding privacy features to the Chromium open source project can have a serious impact (see https://www.barrons.com/articles/google-chrome-privacy-questions-51553871811 https://www.barrons.com/articles/google-chrome-privacy-quest...).
  - titanomachy 7y ago
    I'm not a lawyer, but if the information is false I don't think you could get dinged for insider trading. The legal approach that's used to prosecute insider trading is basically "theft of secrets".
boulos 7y ago
As kyrra says below, you're in the clear if you state that this is just your opinion. Naturally, prefacing something terrible as "just your opinion" doesn't make it fine. In my case, Cloud PR knows me, but I also knowingly risk my job (I clearly believe I have good enough judgment in what I post). If Urs and Ben think I should be fired, I'm okay with that, as it would represent a significant enough difference in opinion, that I wouldn't want to continue working here anyway. Finally, for what it's worth, I have been reported before for "leaking internal secrets" here on HN! It turned out to be a totally hilarious discussion with the person tasked with questioning me. Still not fired, gotta try harder :).
- kyrra 7y ago
  To add my own story. I have made comments about other teams services on hacker news before. I've been contacted by the SRE responsible for the service I commented on asking me to correct what I said. Luckily no reports for leaking info. :) Whenever I talk about the inner workings of Google I try to reference to external talks, books, or white papers to go along with my comments. Luckily a lot has already been set externally about how Google works.
- bardworx 7y ago
  That’s...that’s some petty fucking shit. I didn’t go through your comments but considering your email is in your profile, someone really had to have a hard-on to report you for leaks. I would love to understand the though process of someone going out of their way to remove someone’s livelihood from them because of a comment on HN (when applied in a normal circumstance of adding additional information or correcting a misconception — I’m clearly not saying that bonehead comments shouldn’t have consequences.)
  - deleted 7y ago
    [deleted]
  - praneshp 7y ago
    > I would love to understand the though process of someone going out of their way to remove someone’s livelihood from them because of a comment on HN This is not so difficult though. You just need to adjust your starting point to someone who doesn't like boulos' first. That's not so difficult IMO, it's a large org and boulos' seems to be a fairly prolific commenter here.
user5994461 7y ago
Google employees are commenting publicly and on Hacker News all the time. If there is a policy of not speaking publicly about the company, this has been the most blatantly ignored policy ever.
- akhilcacharya 7y ago
  I’m 90% sure it’s just to flex, honestly. I work at another FANG with a roughly equal engineering community and I don’t see my kind commenting as much at all!
  - laluser 7y ago
    Another FANG = Amazon? If so, Amazon is pretty restrictive in how it wants employees to communicate about internal activities. Most people err on the side of caution and don't comment publicly.
    - akhilcacharya 7y ago
      It is - but all companies I’ve ever worked for are. I’m not convinced the letter of the policy is much different.
      - 9nGQluzmnq3M 7y ago
        There are definitely major differences between the FAANGs: what was the last time you saw an Apple employee commenting on anything on HN?
        zaphirplane 7y ago
        Definitely not all FADANGs are the same. never seen a Disney employee comment ;) Or oracle in FADANGOs ;) just kidding
        saagarjha 7y ago
        Apple employees comment on Hacker News all the time: they just don’t identify themselves as speaking for the company and make sure to only talk about publicly available information.
munificent 7y ago
I work at Google on an open source project and comment on it frequently. One of the things I really like about working at Google is that they place a lot of trust in the judgement of the individual employees. I generally make it clear when I'm stating my personal opinion versus the "official" (for whatever that means given how informal the project is) one, but I don't have to carefully go through an approved list of talking points, run my HN by the legal department, etc. Obviously, in certain situations, things get more official and formal. For example, when I went to Google IO to give a talk, we did have some documentation and coaching beforehand about how to handle various questions we might get about non-public stuff, other projects related to ours, etc. We are also expected to run any slides by legal before being publicly shown in a venue with a wide audience like IO. But, even then, the legal folks I've worked with have been a pleasure to talk to. The company's culture is basically "We hired you because you're smart. We trust you to use your brain." It would be squandering resources to not let their employees use their own intelligence and judgement.
- cameronbrown 7y ago
  Off-topic, but I noticed in your bio you wrote Game Programming Patterns. Was a great read!
  - danso 7y ago
    Also off-topic: am looking forward to the finishing of craftinginterpreters.com, which has been a fantastic read so far
  - munificent 7y ago
    Thank you!

edwintorok 7y ago

You seem to have 3 status messages on the dashboard at 14:31, 14:44 and 14:48 with exactly the same contents. Were those messages really posted 3 times, or did something go wrong and they got duplicated?

FrankPetrilli 7y ago
We're aware this happened - that posting is the responsibility of an adjacent team to my own, specifically the person right next to me. :)

walshemj 7y ago

Sounds like back hoe fade (from the write up) and it sounds like multiple cables sharing the same physical route got taken out.

deleted 7y ago

[deleted]

mrweasel 7y ago

>The Region isn't (and wasn't) "down", but obviously network latency spiking up for external connectivity is bad. As one of my old bosses said: I don't care that the site/service is technically running, if the customers can't reach it, then IT'S DOWN.

jodrellblank 7y ago
Your boss picked a ridiculous time to nitpick over wording, to shout and add stress to an already difficult situation, and giving up accuracy and precise understanding at a time those are most important.
- Voxoff 7y ago
  "to shout and add stress to an already difficult situation" now that's accuracy
- mrweasel 7y ago
  I don’t think he’s the one nitpicking. From a business perspective the site was down. Nitpicking is telling him: No it is in fact up, the customer just can’t use it.
  - jodrellblank 7y ago
    "Customers are complaining they can't access a thing" "OK, we have databases up, load balancers responding, DNS records check out, last change/deployment was at this time, all these services are up, and the latest test suite is running all green, this narrows down the places where a failure might be with some useful differential diagnosis, now we can move attention to.." "I DON'T CARE THAT YOU THINK THINGS ARE WORKING, IF THE CUSTOMER CANNOT GET TO IT, IT'S DOWN" "Thanks for that helpful input, let's divert troubleshooting attention from this P1 incident, and have a discussion about what "DOWN" means. You want me to treat the working databases as down because the customer can't get to them? Even though they're working? It's like the hatred for "works on my machine". "WELL I'M NOT RUNNING ON YOUR MACHINE". No you aren't, but this demonstrates the current build works, the commands you're using are coherent and sensible, excludes many possible causes of failure, and adds useful information to the situation.