8 ms·

It something I would expect security hardware to have automatically stopped. Even an employee shouldn't be able to download 125GB of stuff without flipping a sa

by leros 5y ago

It something I would expect security hardware to have automatically stopped. Even an employee shouldn't be able to download 125GB of stuff without flipping a safety switch somewhere.

com2kid 5y ago
> Even an employee shouldn't be able to download 125GB of stuff without flipping a safety switch somewhere. I am trying to recall, but I am pretty sure when I worked in Microsoft Office that a build would pull down many tens of gigabytes of data. 125GB in one day from the build system wouldn't be uncommon!
- Raidion 5y ago
  That's ingress though. Companies should be monitoring and worrying about egress. Edit: This won't help against a thumbdrive, but that type of thing should be also tracked.
  - AustinDev 5y ago
    I'm working on a project and just had to repull my workspace after some local corruption. I pulled 1.2TB out of the office and never got an email. I think it's pretty common for places not to monitor egress that closely.
yawaworht1978 5y ago
Indeed , how could this happen, really curious. So let's say someone with access to all GitHub repos gave the password to someone else, maybe then it was downloaded from another machine? Or someone stole the credentials and downloaded from another machine? Or someone got access to such a machine? It's it not possible to prevent these cases? How long does such a download take?
stefan_ 5y ago
Cue monorepo discussion
- munk-a 5y ago
  Cue "Don't check payment receipts into git" discussion - although I strongly suspect this hack wasn't just about acquiring appropriate credentials and then running `git clone`. It sounds to me like a backup service was compromised.
toomuchtodo 5y ago
That's 6.25GB/day over a 20 day working month. More time, less data per work day, harder to detect.
- jandrese 5y ago
  And it might be disguised as a video stream coming out of the video streaming servers. But it could also be a 128GB thumb drive plugged into the system somewhere.
  - vineyardmike 5y ago
    > And it might be disguised as a video stream coming out of the video streaming servers. Just log in to FB messenger or Discord and egress it as small data chunks that way. Lots of people have private chats on work computer for practical purposes. Discord allows for bots, so you could easily write a script to chunk data and egress, and another to re-assemble.
AshamedCaptain 5y ago
Trying to protect against leaking developers/employees is like trying to protect against lone gunman terrorists: useless. And, if you try anyway, it is likely to cause more annoyance to everyone involved than actual protection (think TSA).
- yupper32 5y ago
  I disagree. Locking down and logging access to raw data like password hashes or payout information to only those who absolutely need it doesn't cause much annoyance and is very useful. It protects the company against rogue employees (not even strictly malicious, but also curious employees who want to see more than they should). It limits exposure if an employee's account gets hacked (my pet theory for this Twitch hack). And if something does go wrong, logs help track down the issue/leak. And at the end of the day, there should be a lightweight way to request access. Many times I've seen people request access that they didn't actually need. And most other times they have access pretty quickly.
  - xmprt 5y ago
    Adding to your pet theory I think that WFH has led to a lot of people being casual about their workplace security. For example, leaving a laptop unattended at a Starbucks. This is just a guess but I wouldn't be surprised if companies have to start taking stricter precautions with their security in a WFH world.
  - AshamedCaptain 5y ago
    Note that it was code that was leaked. Preventing developers from leaking the codebase they are working with is outright impossible. Now combine that with a "monorepo" and even the most junior developer has access to practically the entire company codebase and version control history. And you can try to prevent them from accessing live/real customer data, but the cost is that they will never be able to debug issues in production. Most companies, even very large ones, are just not able to pay that cost. Not to mention that once you have access to the codebase there are a million ways to leak customer data anyway -- it is a lost battle.
deleted 5y ago
[deleted]
CobrastanJorji 5y ago
If the bulk of it is a git repo, it's probably expected that every engineer will download it regularly.
- manojlds 5y ago
  Case against monorepos?
  - crdrost 5y ago
    There are much better cases than this; in this case a monorepo makes it slightly more likely to be caught rather than less. (A monorepo can get to Google size and then you can't check it all out at once and it needs bespoke tooling, which can make it harder to pull this off.) On the flip side while many smaller repos _can_ have independent ACLs, you are very unlikely to set those up until you reach a certain scale -- and then when you reach that scale it gets hard to implement ACLs across everything at once. So your engineers probably all have access to all your repos until you reach a very large size anyway. So the question becomes just "can someone write a for-loop over all of the repo names and check them all out," and it's like, yeah, that's not terribly hard, I as a programmer can do that pretty easily in bash. Ideal repo size should not in my view be directed at "how do I prevent compromise to the external world," because VCS is not designed to give you the superpower of being resilient around being compromised. Rather VCS is trying to give you the superpower of time travel. So you should probably scope your repo to "what is the unit that makes sense to time travel with?" -- in other words if you are adamant that you have these independent services which operate decoupled and running this one backwards by a year should not affect that one, then those services should be in separate repos. If on the other hand they have some moderate coupling and rewinding this service by 1 year would break the APIs that that service uses to communicate... then those should ideally be in the same repo so that you can coordinate changes between them to their shared protocol.
    - vineyardmike 5y ago
      > So your engineers probably all have access to all your repos until you reach a very large size anyway. Happens at my company. We have rudimentary ACL but not sure how its implemented because you can find things via explicit searching, or via "organic finding" via links from repo->repo but it won't be surfaced if you just search for code.
munk-a 5y ago
Gosh - I've worked at shops where we handled multi-terabyte images and we'd regularly stream large chunks of that while debugging tools. I've also worked at places where data was king and 125GB of stuff might be a reasonable dispatch of data to help someone debug. The volume of data is irrelevant - source code is usually teensy tiny and of far more value to companies than, say, three months of livestream chat logs. I'm not certain what security hardware you're thinking of - but I'm pretty sure I hate it already since it doesn't effectively guard anything while making everyone's lives difficult. For effective corporate security you need 1) data use policies and 2) access control lists - both of those are generally more effectively implemented at an entirely software level.
- retbull 5y ago
  Yeah volume is a terrible metric to go by. I work as a data engineer and a lot of the time if I am working between environments or when migrating between data centers will have a copy of the data locally that I can write tests against or move to somewhere I can compare it to a running output. This would be possible to do entirely remotely I guess but not nearly as easy. (note I never do this with anything that contains PII)
  - manquer 5y ago
    It is still fraught with problems, while you (knowingly) wouldn't do it with PII, is not all that reassuring, others could, or compromised system could be used to exfiltrate this data, if the only control is just trust on the users behaving well with their access That fact in general industry the controls on how PII data is accessed internally is so lightly managed should worry everyone
tptacek 5y ago
There was a fad for tools that accomplished this in enterprise networks, with much clearer rules for who needs to access what (it was called "data loss prevention", or DLP) and those tools for the most part don't work. This is a harder problem than it looks like.
- cpach 5y ago
  Thank you for mentioning this. I always had a gut feeling that it seems like an extremely hard problem to solve in a sensible way.
- tfigment 5y ago
  My company has this. It encrypts any file touched on USB. And other software logs every app run. Prevents casual copying but easily circumvented. But somewhere logs may have enough info to trace the source of leak I guess.
- realitylabs 5y ago
  These tools (DLP) have gotten better with app migration to K8s, since traffic can be watched prior to encryption in a standardized way. Just an FYI….
  - tptacek 5y ago
    The enterprise DLP tools were deployed fleetwide as agents and at network choke points; getting access to the raw data wasn't the problem.
- unethical_ban 5y ago
  DLP products tend to be more about scanning the contents of data for sensitive patterns, at least in my observation of the market. There are other products (typically built into SIEM) that do correlation on login events, network traffic and whatnot to detect anomalous behavior.
  - AmericanChopper 5y ago
    I’ve worked on a lot of DLP projects in big enterprise, and I have a very dim view of the entire category of product. A lot of their functionality is just magic black boxes, that unsurprisingly achieve very little. The primary motive for deploying them is not that they’re particularly effective, it’s so that you can tell auditors and other scrutineers that you’ve got a “DLP solution”. The idea that you can grant people access to huge quantities information, but then very strictly control what they do with it is fundamentally flawed. Especially on networks that require large amounts of in and outflow for BAU. Even the most tightly controlled data in the world cannot be protected from an inside leaker (or adversary who has taken control of an insiders access), because it runs into the same “analog hole” issue that DRM products have.
cheeze 5y ago
I feel like once you have it pulled downm, it would be as simple as an upload to s3 (which wouldn't trigger any flags), then making the bucket public whenever you want. Hell, S3 used to (still does?) support being part of a torrent swarm...
ABeeSea 5y ago
ML engineers / data scientists are regularly moving terabytes of data around at Amazon.
ljm 5y ago
Why would that help? They just have to accumulate work over a period of time and then 'lose' their laptop.
outworlder 5y ago
> It something I would expect security hardware to have automatically stopped. Even an employee shouldn't be able to download 125GB of stuff without flipping a safety switch somewhere. Remember that Twitch handles streams. Good luck implementing this without having all sorts of false alarms everywhere. Plus, you don't have to exfiltrate 125GB in one go.

by leros 5y ago

It something I would expect security hardware to have automatically stopped. Even an employee shouldn't be able to download 125GB of stuff without flipping a safety switch somewhere.

com2kid 5y ago

> Even an employee shouldn't be able to download 125GB of stuff without flipping a safety switch somewhere. I am trying to recall, but I am pretty sure when I worked in Microsoft Office that a build would pull down many tens of gigabytes of data. 125GB in one day from the build system wouldn't be uncommon!

Raidion 5y ago
That's ingress though. Companies should be monitoring and worrying about egress. Edit: This won't help against a thumbdrive, but that type of thing should be also tracked.
- AustinDev 5y ago
  I'm working on a project and just had to repull my workspace after some local corruption. I pulled 1.2TB out of the office and never got an email. I think it's pretty common for places not to monitor egress that closely.

yawaworht1978 5y ago

Indeed , how could this happen, really curious. So let's say someone with access to all GitHub repos gave the password to someone else, maybe then it was downloaded from another machine? Or someone stole the credentials and downloaded from another machine? Or someone got access to such a machine? It's it not possible to prevent these cases? How long does such a download take?

stefan_ 5y ago

Cue monorepo discussion

munk-a 5y ago
Cue "Don't check payment receipts into git" discussion - although I strongly suspect this hack wasn't just about acquiring appropriate credentials and then running `git clone`. It sounds to me like a backup service was compromised.

toomuchtodo 5y ago

That's 6.25GB/day over a 20 day working month. More time, less data per work day, harder to detect.

jandrese 5y ago
And it might be disguised as a video stream coming out of the video streaming servers. But it could also be a 128GB thumb drive plugged into the system somewhere.
- vineyardmike 5y ago
  > And it might be disguised as a video stream coming out of the video streaming servers. Just log in to FB messenger or Discord and egress it as small data chunks that way. Lots of people have private chats on work computer for practical purposes. Discord allows for bots, so you could easily write a script to chunk data and egress, and another to re-assemble.

AshamedCaptain 5y ago

Trying to protect against leaking developers/employees is like trying to protect against lone gunman terrorists: useless. And, if you try anyway, it is likely to cause more annoyance to everyone involved than actual protection (think TSA).

yupper32 5y ago
I disagree. Locking down and logging access to raw data like password hashes or payout information to only those who absolutely need it doesn't cause much annoyance and is very useful. It protects the company against rogue employees (not even strictly malicious, but also curious employees who want to see more than they should). It limits exposure if an employee's account gets hacked (my pet theory for this Twitch hack). And if something does go wrong, logs help track down the issue/leak. And at the end of the day, there should be a lightweight way to request access. Many times I've seen people request access that they didn't actually need. And most other times they have access pretty quickly.
- xmprt 5y ago
  Adding to your pet theory I think that WFH has led to a lot of people being casual about their workplace security. For example, leaving a laptop unattended at a Starbucks. This is just a guess but I wouldn't be surprised if companies have to start taking stricter precautions with their security in a WFH world.
- AshamedCaptain 5y ago
  Note that it was code that was leaked. Preventing developers from leaking the codebase they are working with is outright impossible. Now combine that with a "monorepo" and even the most junior developer has access to practically the entire company codebase and version control history. And you can try to prevent them from accessing live/real customer data, but the cost is that they will never be able to debug issues in production. Most companies, even very large ones, are just not able to pay that cost. Not to mention that once you have access to the codebase there are a million ways to leak customer data anyway -- it is a lost battle.

deleted 5y ago

[deleted]

CobrastanJorji 5y ago

If the bulk of it is a git repo, it's probably expected that every engineer will download it regularly.

manojlds 5y ago
Case against monorepos?
- crdrost 5y ago
  There are much better cases than this; in this case a monorepo makes it slightly more likely to be caught rather than less. (A monorepo can get to Google size and then you can't check it all out at once and it needs bespoke tooling, which can make it harder to pull this off.) On the flip side while many smaller repos _can_ have independent ACLs, you are very unlikely to set those up until you reach a certain scale -- and then when you reach that scale it gets hard to implement ACLs across everything at once. So your engineers probably all have access to all your repos until you reach a very large size anyway. So the question becomes just "can someone write a for-loop over all of the repo names and check them all out," and it's like, yeah, that's not terribly hard, I as a programmer can do that pretty easily in bash. Ideal repo size should not in my view be directed at "how do I prevent compromise to the external world," because VCS is not designed to give you the superpower of being resilient around being compromised. Rather VCS is trying to give you the superpower of time travel. So you should probably scope your repo to "what is the unit that makes sense to time travel with?" -- in other words if you are adamant that you have these independent services which operate decoupled and running this one backwards by a year should not affect that one, then those services should be in separate repos. If on the other hand they have some moderate coupling and rewinding this service by 1 year would break the APIs that that service uses to communicate... then those should ideally be in the same repo so that you can coordinate changes between them to their shared protocol.
  - vineyardmike 5y ago
    > So your engineers probably all have access to all your repos until you reach a very large size anyway. Happens at my company. We have rudimentary ACL but not sure how its implemented because you can find things via explicit searching, or via "organic finding" via links from repo->repo but it won't be surfaced if you just search for code.

munk-a 5y ago

Gosh - I've worked at shops where we handled multi-terabyte images and we'd regularly stream large chunks of that while debugging tools. I've also worked at places where data was king and 125GB of stuff might be a reasonable dispatch of data to help someone debug. The volume of data is irrelevant - source code is usually teensy tiny and of far more value to companies than, say, three months of livestream chat logs. I'm not certain what security hardware you're thinking of - but I'm pretty sure I hate it already since it doesn't effectively guard anything while making everyone's lives difficult. For effective corporate security you need 1) data use policies and 2) access control lists - both of those are generally more effectively implemented at an entirely software level.

retbull 5y ago
Yeah volume is a terrible metric to go by. I work as a data engineer and a lot of the time if I am working between environments or when migrating between data centers will have a copy of the data locally that I can write tests against or move to somewhere I can compare it to a running output. This would be possible to do entirely remotely I guess but not nearly as easy. (note I never do this with anything that contains PII)
- manquer 5y ago
  It is still fraught with problems, while you (knowingly) wouldn't do it with PII, is not all that reassuring, others could, or compromised system could be used to exfiltrate this data, if the only control is just trust on the users behaving well with their access That fact in general industry the controls on how PII data is accessed internally is so lightly managed should worry everyone

tptacek 5y ago

There was a fad for tools that accomplished this in enterprise networks, with much clearer rules for who needs to access what (it was called "data loss prevention", or DLP) and those tools for the most part don't work. This is a harder problem than it looks like.

cpach 5y ago
Thank you for mentioning this. I always had a gut feeling that it seems like an extremely hard problem to solve in a sensible way.
tfigment 5y ago
My company has this. It encrypts any file touched on USB. And other software logs every app run. Prevents casual copying but easily circumvented. But somewhere logs may have enough info to trace the source of leak I guess.
realitylabs 5y ago
These tools (DLP) have gotten better with app migration to K8s, since traffic can be watched prior to encryption in a standardized way. Just an FYI….
- tptacek 5y ago
  The enterprise DLP tools were deployed fleetwide as agents and at network choke points; getting access to the raw data wasn't the problem.
unethical_ban 5y ago
DLP products tend to be more about scanning the contents of data for sensitive patterns, at least in my observation of the market. There are other products (typically built into SIEM) that do correlation on login events, network traffic and whatnot to detect anomalous behavior.
- AmericanChopper 5y ago
  I’ve worked on a lot of DLP projects in big enterprise, and I have a very dim view of the entire category of product. A lot of their functionality is just magic black boxes, that unsurprisingly achieve very little. The primary motive for deploying them is not that they’re particularly effective, it’s so that you can tell auditors and other scrutineers that you’ve got a “DLP solution”. The idea that you can grant people access to huge quantities information, but then very strictly control what they do with it is fundamentally flawed. Especially on networks that require large amounts of in and outflow for BAU. Even the most tightly controlled data in the world cannot be protected from an inside leaker (or adversary who has taken control of an insiders access), because it runs into the same “analog hole” issue that DRM products have.

cheeze 5y ago

I feel like once you have it pulled downm, it would be as simple as an upload to s3 (which wouldn't trigger any flags), then making the bucket public whenever you want. Hell, S3 used to (still does?) support being part of a torrent swarm...

ABeeSea 5y ago

ML engineers / data scientists are regularly moving terabytes of data around at Amazon.

ljm 5y ago

Why would that help? They just have to accumulate work over a period of time and then 'lose' their laptop.

outworlder 5y ago

> It something I would expect security hardware to have automatically stopped. Even an employee shouldn't be able to download 125GB of stuff without flipping a safety switch somewhere. Remember that Twitch handles streams. Good luck implementing this without having all sorts of false alarms everywhere. Plus, you don't have to exfiltrate 125GB in one go.