5 ms·

CorpIT was one of a handful of reasons why I left Alphabet. Maybe it's all rainbows and unicorns if you're a software engineer and you do all your work on a Chr

CorpIT was one of a handful of reasons why I left Alphabet. Maybe it's all rainbows and unicorns if you're a software engineer and you do all your work on a Chromebook. As a hardware engineer it was pretty miserable. You end up with 4-5 computers, only the most-recently-used of which works. Every time you need to switch you end up re-imaging. All infosec policies are designed with no regard for non-SWE use-cases. Want to move files to or from test equipment via flash drive? Infosec policy violation. Want to use Drive to move things to your untrusted machine instead? Sorry, you can't log in to your corp gAccount from a non-corp computer. Want to have two NICs so you can use one to talk to Ethernet-connected hardware directly? Explicitly an anti-supported use case and your machine will often fail to boot, can't be remotely boot-unlocked, and will frequently drop certs. As far as I could tell it was a huge group of bored people chasing Impact and not solving real problems or even bothering to read tickets. Most of those problems were the product of previous generations of Impact-chasers.

uudecoded 5y ago
Could you elaborate more on the control mechanisms that supported the outcome of "your machine will often fail to boot, can't be remotely boot-unlocked" for said anti-supported use cases?
- zbrozek 5y ago
  As far as I was able to tell, the system didn't know which NIC it should be routing through. So it would sit at the disk encryption screen and you wouldn't be able to remote unlock it. And you don't have access to the building to go in and do it in person. Or if you successfully cleared that, the OS also rolled the dice to let you authenticate. Rebooting was painful, typically taking 5-15 minutes. And if you tried to edit configurations to try and load the dice, puppet would overwrite them. Tickets seeking help were mostly ignored, citing that dual NIC was explicitly not supported.
  - exikyut 5y ago
    That's interesting to hear that you can remote-unlock machines that are waiting for a disk encryption key. I presume the remote-unlock case is the primary one. I've long been curious how early boot works on Google servers (and TIL workstations too, although it makes perfect sense) - primarily because I want to copy the techniques myself! :D How is key storage and device attestation actually done?
    - qmarchi 5y ago
      That's a good question, now isn't it? In reality, what we can expose, is that during the early boot process, our systems reach out to another system that register's its interest, and as a user (from another trusted device, phone, laptop, etc.) you can visit the web service and click a button. How and where keys are stored is a great big "?" that's up to the implementer to solve..
      - exikyut 5y ago
        So basically you have 2FA for workstation boot. That's really quite cool. Of course I want to know how and where the keys are stored :D so I can do the same thing myself! Arguably security systems in this class demonstrate their integrity *because* their architecture is fully open, documented and straightforwardly reproducible. It isn't science if it isn't reproducible, right? Something something computer science... That's the idealistic view, of course. In the <insert cartoon punching fight cloud here> real world, we have The Legacy PC Problem™, where secure boot isn't, TPMs can be bus sniffed, SGX doesn't really support the hacker/tinkerer exploration necessary to power defense in depth, ME is a ginormous black box that eats authentication headers like they might as well be glue... and it doesn't matter that I am a dog residing on Mars because the MDM my 2FA device is signed into has decided I'm legit. Hence my interest in real security. It's a giant debacle, surely there are some genuinely cool wins to be had out there that truly make a dent ._.
        zbrozek 5y ago
        Like most Alphabet implementations it's 80% awesome and 20% undone. Unfortunately that means it's a bad time.
vinay_ys 5y ago
Google's internal security was one of the best I have experienced. Everywhere else, just the broken VPN drove me nuts. Also, one discipline I have maintained across many companies is clear separation between work devices and personal devices. I don't do personal stuff on work devices. I had multiple high-privilege access enabled work devices and all of them worked just fine. Of course authentication tokens/certs had short expiry (rightly so) and I would have to re-authenticate (which was quick and painless and worked 100% reliably). I wasn't a hardware engineer, but I suppose for non-trivial test hardware setups I would be required to work out of a on-premise device lab. For flashing android mobile devices, I could easily do that from my work laptop.
- deleted 5y ago
  [deleted]
exikyut 5y ago
I've slowly come to appreciate Google's emphasis on process/police over engineering coherence (and real security :P), so maybe I'm describing the infuriatingly futile here, but... - Box with Ethernet port that connects straight to CorpIT (something something service account) and client/gadget USB port that shows up as mass storage; you talk to the box via a remote server using a userspace tool that speaks FUSE. Everything staged/uploaded gets archived. You could go one step further and tie this in with goma and have the box receive securely built code, so you'd only need to archive git metadata instead of giant blobs. - Engineering-specific Drive frontend could give you token/TOTP/straightforward based access to files, with download audit logging/data archiving (...this would be so simple for CorpIT to provide...) - PCI card with two Ethernet ports, one special one that plugs straight into CorpIT and a data port for single devices or even a local n-port switch; Linux on the card signs the special port into CorpIT via a service account and then uses custom RPC/whatever to mirror all traffic flowing over the data port. Chances are this setup would likely only be able to viably do 10Mbps given that all traffic would be mirrored. The card would require a wakeup (and/or firmware upload) event before the PHY appears, preventing EFI boot confusion. - "Secure bringup services" branch of CorpIT implements processes and makes services available that focus on the fundamentally open-ended requirements of hardware design, which would ideally provide more cohesive solutions that the workarounds described above. IMHO I reckon all those bored engineers would have an absolute field day doing stuff like the above (chasing Impact in a hardware context); it's a great pity they aren't adequately enabled to demonstrate the security implications and priority of doing so.
- numpad0 5y ago
  It sounds to me like the problem he encountered stems from a flawed understanding that IP:MAC:Machine has 1:1:1 relationship and that subnet:router:default-route:external-ip:machine has a 1:1:1:1:n relationship, which are only the cases for media consumption devices and cloud servers. And so when he tried to work with a machine without exactly one(1) IP and one(1) active NIC going up the NAT router to the Internet it immediately failed. This is surprising to me that of MANGA, Google seems to be on the weak side with respect to IP networking.
sleepydog 5y ago
This has not been my experience, but I'm not a hardware engineer, and haven't tried any of the things you were frustrated with, so we could both be right. I use a linux laptop at home and a linux desktop in the office. For almost 2 years of mostly remote working during the pandemic, beyondcorp has caused me trouble maybe 4 times, and 2 of those times were when my office re-opened and I had to revive my security keys that hadn't been used in over a year. There is a VPN, but I've never had to use it. They kept my corp desktop running in the office which I could use as a jump host when I needed it. Tunneling X11 over SSH worked well, and the browser-based remote desktop worked ~ok.

uudecoded 5y ago

Could you elaborate more on the control mechanisms that supported the outcome of "your machine will often fail to boot, can't be remotely boot-unlocked" for said anti-supported use cases?

zbrozek 5y ago
As far as I was able to tell, the system didn't know which NIC it should be routing through. So it would sit at the disk encryption screen and you wouldn't be able to remote unlock it. And you don't have access to the building to go in and do it in person. Or if you successfully cleared that, the OS also rolled the dice to let you authenticate. Rebooting was painful, typically taking 5-15 minutes. And if you tried to edit configurations to try and load the dice, puppet would overwrite them. Tickets seeking help were mostly ignored, citing that dual NIC was explicitly not supported.
- exikyut 5y ago
  That's interesting to hear that you can remote-unlock machines that are waiting for a disk encryption key. I presume the remote-unlock case is the primary one. I've long been curious how early boot works on Google servers (and TIL workstations too, although it makes perfect sense) - primarily because I want to copy the techniques myself! :D How is key storage and device attestation actually done?
  - qmarchi 5y ago
    That's a good question, now isn't it? In reality, what we can expose, is that during the early boot process, our systems reach out to another system that register's its interest, and as a user (from another trusted device, phone, laptop, etc.) you can visit the web service and click a button. How and where keys are stored is a great big "?" that's up to the implementer to solve..
    - exikyut 5y ago
      So basically you have 2FA for workstation boot. That's really quite cool. Of course I want to know how and where the keys are stored :D so I can do the same thing myself! Arguably security systems in this class demonstrate their integrity *because* their architecture is fully open, documented and straightforwardly reproducible. It isn't science if it isn't reproducible, right? Something something computer science... That's the idealistic view, of course. In the <insert cartoon punching fight cloud here> real world, we have The Legacy PC Problem™, where secure boot isn't, TPMs can be bus sniffed, SGX doesn't really support the hacker/tinkerer exploration necessary to power defense in depth, ME is a ginormous black box that eats authentication headers like they might as well be glue... and it doesn't matter that I am a dog residing on Mars because the MDM my 2FA device is signed into has decided I'm legit. Hence my interest in real security. It's a giant debacle, surely there are some genuinely cool wins to be had out there that truly make a dent ._.
      - zbrozek 5y ago
        Like most Alphabet implementations it's 80% awesome and 20% undone. Unfortunately that means it's a bad time.

vinay_ys 5y ago

Google's internal security was one of the best I have experienced. Everywhere else, just the broken VPN drove me nuts. Also, one discipline I have maintained across many companies is clear separation between work devices and personal devices. I don't do personal stuff on work devices. I had multiple high-privilege access enabled work devices and all of them worked just fine. Of course authentication tokens/certs had short expiry (rightly so) and I would have to re-authenticate (which was quick and painless and worked 100% reliably). I wasn't a hardware engineer, but I suppose for non-trivial test hardware setups I would be required to work out of a on-premise device lab. For flashing android mobile devices, I could easily do that from my work laptop.

deleted 5y ago
[deleted]

exikyut 5y ago

I've slowly come to appreciate Google's emphasis on process/police over engineering coherence (and real security :P), so maybe I'm describing the infuriatingly futile here, but... - Box with Ethernet port that connects straight to CorpIT (something something service account) and client/gadget USB port that shows up as mass storage; you talk to the box via a remote server using a userspace tool that speaks FUSE. Everything staged/uploaded gets archived. You could go one step further and tie this in with goma and have the box receive securely built code, so you'd only need to archive git metadata instead of giant blobs. - Engineering-specific Drive frontend could give you token/TOTP/straightforward based access to files, with download audit logging/data archiving (...this would be so simple for CorpIT to provide...) - PCI card with two Ethernet ports, one special one that plugs straight into CorpIT and a data port for single devices or even a local n-port switch; Linux on the card signs the special port into CorpIT via a service account and then uses custom RPC/whatever to mirror all traffic flowing over the data port. Chances are this setup would likely only be able to viably do 10Mbps given that all traffic would be mirrored. The card would require a wakeup (and/or firmware upload) event before the PHY appears, preventing EFI boot confusion. - "Secure bringup services" branch of CorpIT implements processes and makes services available that focus on the fundamentally open-ended requirements of hardware design, which would ideally provide more cohesive solutions that the workarounds described above. IMHO I reckon all those bored engineers would have an absolute field day doing stuff like the above (chasing Impact in a hardware context); it's a great pity they aren't adequately enabled to demonstrate the security implications and priority of doing so.

numpad0 5y ago
It sounds to me like the problem he encountered stems from a flawed understanding that IP:MAC:Machine has 1:1:1 relationship and that subnet:router:default-route:external-ip:machine has a 1:1:1:1:n relationship, which are only the cases for media consumption devices and cloud servers. And so when he tried to work with a machine without exactly one(1) IP and one(1) active NIC going up the NAT router to the Internet it immediately failed. This is surprising to me that of MANGA, Google seems to be on the weak side with respect to IP networking.

sleepydog 5y ago

This has not been my experience, but I'm not a hardware engineer, and haven't tried any of the things you were frustrated with, so we could both be right. I use a linux laptop at home and a linux desktop in the office. For almost 2 years of mostly remote working during the pandemic, beyondcorp has caused me trouble maybe 4 times, and 2 of those times were when my office re-opened and I had to revive my security keys that hadn't been used in over a year. There is a VPN, but I've never had to use it. They kept my corp desktop running in the office which I could use as a jump host when I needed it. Tunneling X11 over SSH worked well, and the browser-based remote desktop worked ~ok.