5 ms·

Outside of the case, the most striking thing to me about this is the level of detail that Google has over the logs and actions of their laptop. They were able t

Outside of the case, the most striking thing to me about this is the level of detail that Google has over the logs and actions of their laptop. They were able to tell that a memory card was plugged in for 8 hours a year ago? I have trouble finding exactly how a customer encountered a 500 10 minutes ago.

ryandrake 10y ago
You're talking about a company whose mission is "to organize the world's information". If you're an employee thinking about wrong-doing, you've gotta have that in the back if your mind.
cvwright 10y ago
They have some pretty cool stuff for monitoring their internal systems. For example, this distributed live forensics system: GRR Rapid Response https://github.com/google/grr https://github.com/google/grr http://grr-response.blogspot.com/ http://grr-response.blogspot.com/
- zeptomu 10y ago
  Funny note: these screenshots show an AWS instance :) https://github.com/google/grr/blob/gh-pages/screenshots/Screenshot%20from%202013-11-18%2018-36-13.png https://github.com/google/grr/blob/gh-pages/screenshots/Scre...
  - navek 10y ago
    maybe simply to keep keep all forensic operations as separated from any google operations as possible?
d--b 10y ago
Mountable device tracking is very common to log. My company started implementing when we went from 20 to 30 employees. When things like this are at stakes, it's a small price to pay. In fact, it is so well known that this kind of tracking exists that very few people actually do try and actually do insert mountable devices in their computers, which makes the tracking even easier.
dantiberian 10y ago
Yeah I really enjoyed the technical details in Gary Brown's deposition: https://www.documentcloud.org/documents/3515476-Declaration-of-Gary-Brown-PUBLIC.html https://www.documentcloud.org/documents/3515476-Declaration-.... It looks like they could see: * When the device made check-ins to their management server, and appeared on the Google network * That the Windows OS was re-installed * MoMA search queries (some kind of internal system?) for "chauffeur svn login" * Downloading and installing TortoiseSVN * Transferring files from SVN * When a card reader was plugged in * When the OS was reformatted to Linux * File downloads from Google drive
- jsmthrowaway 10y ago
  Just think of "MoMA" as "the intranet," which is about the best way to explain it. Not super interesting. Weird that he'd allegedly type "I've never logged into Subversion before, but suddenly I need access" into MoMA, but hey. The whole saga just reminds me that people are really bad at opsec. The Google Drive/GAIA linkage is the most interesting technical revelation from a quick read. I'm surprised they didn't redact that little trick, because that seems to be the best way to catch the very people doing this. Most people would be forgiven for assuming that a Google consumer product might be a little more obfuscated in terms of forensics (logs access, PII, etc), and I could see that train of thought even though it's dumb. I guess they're willing to burn the technique, because the next person who attempts this will definitely avoid Google Drive.
- chx 10y ago
  I now read that deposition in full, this guy is dead and buried six feet under.
deleted 10y ago
[deleted]
haldean 10y ago
If you send your kernel logs to a syslog server you could get that pretty easily. From plugging a USB stick into my computer just now, then running dmesg: [14863.014481] usb-storage 2-4:1.0: USB Mass Storage device detected ... [15045.488440] usb 2-4: USB disconnect, device number 34 They do, though, have quite a bit of pretty impressive logging!
- chx 10y ago
  If I read this correctly at this point the machine was running Windows. I have zero doubts, however, that Windows has equivalent impressive logging, apparently the USB vendor and device ID pair was recorded as well.
  - NamTaf 10y ago
    Absolutely. The Windows Event Viewer logs heaps of stuff, and has heaps more you can enable. When I'm particularly lazy and forget to do my timesheet, I use it to look when I booted/shut/locked/unlocked down my work PC to get an idea of how long I worked. A cursory google search shows this for USB events: https://dfstream.blogspot.com.au/2014/01/the-windows-7-event-log-and-usb-device.html https://dfstream.blogspot.com.au/2014/01/the-windows-7-event...
nopriorarrests 10y ago
How many thousands of hours of video is uploaded to youtube _daily_ and stored on google servers forever? And you are talking about logging actions of your own employees. Let's say 100kb per day per employee (these logs are pretty detailed). 100kb * 50 000 employees = 4.7Gb per day. 1740Gb per year. Probably much, much less then 1 day of youtube videos.
- robotresearcher 10y ago
  It's obviously not the volume of data that is surprising. It's that they retained system message logs from a random laptop for years. They must regard this as potentially important business information. Does your company do this? Mine sure doesn't.
  - acchow 10y ago
    Data is useful for large companies - especially when you want to make data-driven decisions. - Can we stop dispensing USB cables in the vending machines? - Should we deploy the new Macbook Pro without classic USB ports as the new standard? - Do our conference rooms really still need USB presentation remotes? Does anyone actually use them?
  - ghughes 10y ago
    Turns out it was important business information after all.
  - x0x0 10y ago
    For a company at Google's scale, figuring out how to cull even a couple TB a year may cost much more in salary than the money saved. Also, it's occasionally really really useful.
  - syntheticnature 10y ago
    What's interesting is that I can see these logs as being a liability as well as an asset; there's a reason most business records retention policies are about getting rid of records, not keeping them. From that perspective it's surprisingly they'd reveal the level of logging in play.
WatchDog 10y ago
Does anyone know if this information was collected as part of a monitoring service or if it was collected by forensically recovering the data from the laptops hard disk?
- londons_explore 10y ago
  If the laptop was re-imaged to Windows, I doubt much of the syslogs from linux were left. Especially since windows is much bigger than a default linux install these days, and this was a rarely used laptop, it seems unlikley any linux was left, even in free sectors.
  - chx 10y ago
    No, it was running Windows and was reimaged to Linux. It is very strongly implied this was done to cover tracks as it was never switched on again.
kirykl 10y ago
This level of logging is easy to come by forensically, even on a stock windows install - except the year time window, probably need to archive logs for that
NelsonMinar 10y ago
Google has been the target of infiltration by staet actors. At least once we know for certain, with the incident in China, and presumably other times.
nunez 10y ago
I was a Windows engineer for Google's Corp Eng PaaS SRE team and have familiarity with this topic. It's not hard to get this data or set this data collection up on a Windows domain or machine. Anyone can set this up as long as you have the TBs to collect the data and the presentation/searching layer to find it. Windows has the ability to log everything, including device installs/uninstalls, file opens/closes/creation/deletion, logons/logoffs, you name it. The Windows Auditing library is really, really, really extensive. Additionally, setting this up in Active Directory is really easy, both manually and magically with Powershell. It takes about ten minutes or so. As you would iamgine, most of these policies were enabled on the domain to which most users authenticated and the data it collected was siphoned off to essentially a giant cluster of syslog servers. In fact, just about any domain will audit device plugs/unplugs. Had Anthony known about this (it's easy to find out even if you're not an admin), he would've not plugged in that memory stick :) In general, it's pretty hard to do stealthy stuff on Google's network. Everything is logged eight ways to Sunday, especially with GAIA and key-based, two factor auth to EVERYTHING. And unlike most other networks, I wouldn't put it past their security engineers to find shady behavior in a moment's notice.
- obstinate 10y ago
  I don't think GAIA has been publicly commented on by the company, in case that matters to you at all. Could be wrong, though, as it seems the term is present in the deposition and un-redacted.
  - dmoy 10y ago
    I mean it's in some public docs, e.g. here: https://www.google.com/support/enterprise/static/gsa/docs/admin/70/admin_console_help/cloud_google_apps.html https://www.google.com/support/enterprise/static/gsa/docs/ad...
  - MertsA 10y ago
    FWIW Google seems to refer to the GAIA ID in stuff like random error messages, documentation for Google Apps, and in data downloaded from Google Takeout. If it's not supposed to be public information then Google isn't doing a very good job.

ryandrake 10y ago

You're talking about a company whose mission is "to organize the world's information". If you're an employee thinking about wrong-doing, you've gotta have that in the back if your mind.

cvwright 10y ago

They have some pretty cool stuff for monitoring their internal systems. For example, this distributed live forensics system: GRR Rapid Response https://github.com/google/grr https://github.com/google/grr http://grr-response.blogspot.com/ http://grr-response.blogspot.com/

zeptomu 10y ago
Funny note: these screenshots show an AWS instance :) https://github.com/google/grr/blob/gh-pages/screenshots/Screenshot%20from%202013-11-18%2018-36-13.png https://github.com/google/grr/blob/gh-pages/screenshots/Scre...
- navek 10y ago
  maybe simply to keep keep all forensic operations as separated from any google operations as possible?

d--b 10y ago

Mountable device tracking is very common to log. My company started implementing when we went from 20 to 30 employees. When things like this are at stakes, it's a small price to pay. In fact, it is so well known that this kind of tracking exists that very few people actually do try and actually do insert mountable devices in their computers, which makes the tracking even easier.

dantiberian 10y ago

Yeah I really enjoyed the technical details in Gary Brown's deposition: https://www.documentcloud.org/documents/3515476-Declaration-of-Gary-Brown-PUBLIC.html https://www.documentcloud.org/documents/3515476-Declaration-.... It looks like they could see: * When the device made check-ins to their management server, and appeared on the Google network * That the Windows OS was re-installed * MoMA search queries (some kind of internal system?) for "chauffeur svn login" * Downloading and installing TortoiseSVN * Transferring files from SVN * When a card reader was plugged in * When the OS was reformatted to Linux * File downloads from Google drive

jsmthrowaway 10y ago
Just think of "MoMA" as "the intranet," which is about the best way to explain it. Not super interesting. Weird that he'd allegedly type "I've never logged into Subversion before, but suddenly I need access" into MoMA, but hey. The whole saga just reminds me that people are really bad at opsec. The Google Drive/GAIA linkage is the most interesting technical revelation from a quick read. I'm surprised they didn't redact that little trick, because that seems to be the best way to catch the very people doing this. Most people would be forgiven for assuming that a Google consumer product might be a little more obfuscated in terms of forensics (logs access, PII, etc), and I could see that train of thought even though it's dumb. I guess they're willing to burn the technique, because the next person who attempts this will definitely avoid Google Drive.
chx 10y ago
I now read that deposition in full, this guy is dead and buried six feet under.

deleted 10y ago

[deleted]

haldean 10y ago

If you send your kernel logs to a syslog server you could get that pretty easily. From plugging a USB stick into my computer just now, then running dmesg: [14863.014481] usb-storage 2-4:1.0: USB Mass Storage device detected ... [15045.488440] usb 2-4: USB disconnect, device number 34 They do, though, have quite a bit of pretty impressive logging!

chx 10y ago
If I read this correctly at this point the machine was running Windows. I have zero doubts, however, that Windows has equivalent impressive logging, apparently the USB vendor and device ID pair was recorded as well.
- NamTaf 10y ago
  Absolutely. The Windows Event Viewer logs heaps of stuff, and has heaps more you can enable. When I'm particularly lazy and forget to do my timesheet, I use it to look when I booted/shut/locked/unlocked down my work PC to get an idea of how long I worked. A cursory google search shows this for USB events: https://dfstream.blogspot.com.au/2014/01/the-windows-7-event-log-and-usb-device.html https://dfstream.blogspot.com.au/2014/01/the-windows-7-event...

nopriorarrests 10y ago

How many thousands of hours of video is uploaded to youtube _daily_ and stored on google servers forever? And you are talking about logging actions of your own employees. Let's say 100kb per day per employee (these logs are pretty detailed). 100kb * 50 000 employees = 4.7Gb per day. 1740Gb per year. Probably much, much less then 1 day of youtube videos.

robotresearcher 10y ago
It's obviously not the volume of data that is surprising. It's that they retained system message logs from a random laptop for years. They must regard this as potentially important business information. Does your company do this? Mine sure doesn't.
- acchow 10y ago
  Data is useful for large companies - especially when you want to make data-driven decisions. - Can we stop dispensing USB cables in the vending machines? - Should we deploy the new Macbook Pro without classic USB ports as the new standard? - Do our conference rooms really still need USB presentation remotes? Does anyone actually use them?
- ghughes 10y ago
  Turns out it was important business information after all.
- x0x0 10y ago
  For a company at Google's scale, figuring out how to cull even a couple TB a year may cost much more in salary than the money saved. Also, it's occasionally really really useful.
- syntheticnature 10y ago
  What's interesting is that I can see these logs as being a liability as well as an asset; there's a reason most business records retention policies are about getting rid of records, not keeping them. From that perspective it's surprisingly they'd reveal the level of logging in play.

WatchDog 10y ago

Does anyone know if this information was collected as part of a monitoring service or if it was collected by forensically recovering the data from the laptops hard disk?

londons_explore 10y ago
If the laptop was re-imaged to Windows, I doubt much of the syslogs from linux were left. Especially since windows is much bigger than a default linux install these days, and this was a rarely used laptop, it seems unlikley any linux was left, even in free sectors.
- chx 10y ago
  No, it was running Windows and was reimaged to Linux. It is very strongly implied this was done to cover tracks as it was never switched on again.

kirykl 10y ago

This level of logging is easy to come by forensically, even on a stock windows install - except the year time window, probably need to archive logs for that

NelsonMinar 10y ago

Google has been the target of infiltration by staet actors. At least once we know for certain, with the incident in China, and presumably other times.

nunez 10y ago

I was a Windows engineer for Google's Corp Eng PaaS SRE team and have familiarity with this topic. It's not hard to get this data or set this data collection up on a Windows domain or machine. Anyone can set this up as long as you have the TBs to collect the data and the presentation/searching layer to find it. Windows has the ability to log everything, including device installs/uninstalls, file opens/closes/creation/deletion, logons/logoffs, you name it. The Windows Auditing library is really, really, really extensive. Additionally, setting this up in Active Directory is really easy, both manually and magically with Powershell. It takes about ten minutes or so. As you would iamgine, most of these policies were enabled on the domain to which most users authenticated and the data it collected was siphoned off to essentially a giant cluster of syslog servers. In fact, just about any domain will audit device plugs/unplugs. Had Anthony known about this (it's easy to find out even if you're not an admin), he would've not plugged in that memory stick :) In general, it's pretty hard to do stealthy stuff on Google's network. Everything is logged eight ways to Sunday, especially with GAIA and key-based, two factor auth to EVERYTHING. And unlike most other networks, I wouldn't put it past their security engineers to find shady behavior in a moment's notice.

obstinate 10y ago
I don't think GAIA has been publicly commented on by the company, in case that matters to you at all. Could be wrong, though, as it seems the term is present in the deposition and un-redacted.
- dmoy 10y ago
  I mean it's in some public docs, e.g. here: https://www.google.com/support/enterprise/static/gsa/docs/admin/70/admin_console_help/cloud_google_apps.html https://www.google.com/support/enterprise/static/gsa/docs/ad...
- MertsA 10y ago
  FWIW Google seems to refer to the GAIA ID in stuff like random error messages, documentation for Google Apps, and in data downloaded from Google Takeout. If it's not supposed to be public information then Google isn't doing a very good job.