4 ms·

Unsealed court documents reveal data anarchy at Meta

42 points by donohoe 4y ago

MichaelZuo 4y ago
There's the possibility that nobody at Meta genuinely knows how the '149 different data systems' interact and interoperate. I wonder if the EU has the legal authority to force them to understand via some means? Otherwise it seems like a stretch of a demand.
- falcolas 4y ago
  I think it’s fair (if biased) to say that Meta’s incompetence is no excuse to break the law.
- jsnell 4y ago
  No individual needs to know all of it. But any team owning a system that reads data should know what they're using it for. Likewise anyone providing data to other teams should at least know which other systems are reading that data; even if they don't know for certain what the data is used for they know who to ask about it. So the process of understanding can easily be sharded, and the results merged in a fairly mechanical manner. The timeline here seems to be that they were asked for this in December and required to produce it in January. It probably is unreasonable to get this done from scratch in that time. But if they haven't started a project to do it by now, then it does feel a bit like they prefer being able to say "we don't know".
  - MichaelZuo 4y ago
    Do you think a single person or team could up a coherent flowchart/diagram of all data processes at least, given enough time?
jl6 4y ago
> The documents describe data anarchy within the company, where people responsible for data systems are unaware of how other people in the company use their system. In some cases even the engineers using a system may not be able to understand what is happening because, according to a Meta engineer, ‘it is not possible for humans to understand’. Honestly, I think a lot of big orgs with deep legacy tech and a bunch of shadow IT are in a similar position.
- delusional 4y ago
  I work in an old school bank. Can confirm. You're lucky if people even know what the data they themselves use actually means. Usually there's all sorts of edge cases that make them go "ohh no, it can't do that, that would make it completely wrong", but it does do that and it is indeed wrong.
doingtheiroming 4y ago
This means that most European banks, big companies and government departments will soon be fined by the EU... The following could be said of almost all large corporations and institutions, including the year long project that failed to catalog data sources effectively: "When ordered to produce information about what 149 different data systems within Meta do, and what parts of Meta’s business use them, the company was unable to respond. This was despite having conducted a year-long investigation of those systems." There is no point having a law which essentially no company can be compliant with. It means that the law will almost certainly be applied only by exception and hence capriciously. The likes of Facebook may be able to use that as a defense despite their breaches being egregious. As a side-note, the idea that Meta only has 149 in-scope data systems sounds ridiculous.
- Manfred 4y ago
  Lots of organizations and government agencies have already been fined because of these laws and lots of organizations comply with these laws. I work for a company that makes healthcare software solutions and we're fully compliant with the local GDPR laws.
rainingmonkey 4y ago
> people responsible for data systems are unaware of how other people in the company use their system. In some cases even the engineers using a system may not be able to understand what is happening because, according to a Meta engineer, ‘it is not possible for humans to understand’. Sounds like a good way to hide a "Room 641A" situation from your own engineers... https://en.wikipedia.org/wiki/Room_641A https://en.wikipedia.org/wiki/Room_641A
- MertsA 4y ago
  It sounds like someone asked a vague question about a neural network during a deposition and didn't understand the response.
rwhaling 4y ago
Speaking as a data practitioner - data lineage is a profoundly hard and still unsolved problem. In the last few years, I feel like we've begun to crack it - dbt's data lineage graph is a huge advance over anything I've ever used before: https://docs.getdbt.com/terms/data-lineage https://docs.getdbt.com/terms/data-lineage But in general - as an industry, we aren't there yet. All judgments about Meta's practices aside - no large organization likely has the ability to comply with a law like this without years or decades of engineering work, and the law seems pretty clearly intended for selective enforcement.
cvccvroomvroom 4y ago
Uh, horse poo. Prod data is guarded like Fort Knox. No one can get access to anything on the graph without explicit approvals for a business purpose. I tried it. I can't even see who are my coworkers from my own FBID in internal apps.
hobbitstan 4y ago
This is clearly data chaos by design as a way to skirt privacy laws that may impact business performance and profits. The EU won’t fall for it!
JKCalhoun 4y ago
> When ordered to produce information about what 149 different data systems within Meta do, and what parts of Meta’s business use them, the company was unable to respond. This feels to me more like a "tantrum-compliance" (I made it up) with a court order. But what do I know about it (nothing), maybe there is data anarchy at Meta.

42 points by donohoe 4y ago

There's the possibility that nobody at Meta genuinely knows how the '149 different data systems' interact and interoperate. I wonder if the EU has the legal authority to force them to understand via some means? Otherwise it seems like a stretch of a demand.

falcolas 4y ago
I think it’s fair (if biased) to say that Meta’s incompetence is no excuse to break the law.
jsnell 4y ago
No individual needs to know all of it. But any team owning a system that reads data should know what they're using it for. Likewise anyone providing data to other teams should at least know which other systems are reading that data; even if they don't know for certain what the data is used for they know who to ask about it. So the process of understanding can easily be sharded, and the results merged in a fairly mechanical manner. The timeline here seems to be that they were asked for this in December and required to produce it in January. It probably is unreasonable to get this done from scratch in that time. But if they haven't started a project to do it by now, then it does feel a bit like they prefer being able to say "we don't know".
- MichaelZuo 4y ago
  Do you think a single person or team could up a coherent flowchart/diagram of all data processes at least, given enough time?

jl6 4y ago

> The documents describe data anarchy within the company, where people responsible for data systems are unaware of how other people in the company use their system. In some cases even the engineers using a system may not be able to understand what is happening because, according to a Meta engineer, ‘it is not possible for humans to understand’. Honestly, I think a lot of big orgs with deep legacy tech and a bunch of shadow IT are in a similar position.

delusional 4y ago
I work in an old school bank. Can confirm. You're lucky if people even know what the data they themselves use actually means. Usually there's all sorts of edge cases that make them go "ohh no, it can't do that, that would make it completely wrong", but it does do that and it is indeed wrong.

doingtheiroming 4y ago

This means that most European banks, big companies and government departments will soon be fined by the EU... The following could be said of almost all large corporations and institutions, including the year long project that failed to catalog data sources effectively: "When ordered to produce information about what 149 different data systems within Meta do, and what parts of Meta’s business use them, the company was unable to respond. This was despite having conducted a year-long investigation of those systems." There is no point having a law which essentially no company can be compliant with. It means that the law will almost certainly be applied only by exception and hence capriciously. The likes of Facebook may be able to use that as a defense despite their breaches being egregious. As a side-note, the idea that Meta only has 149 in-scope data systems sounds ridiculous.

Manfred 4y ago
Lots of organizations and government agencies have already been fined because of these laws and lots of organizations comply with these laws. I work for a company that makes healthcare software solutions and we're fully compliant with the local GDPR laws.

rainingmonkey 4y ago

> people responsible for data systems are unaware of how other people in the company use their system. In some cases even the engineers using a system may not be able to understand what is happening because, according to a Meta engineer, ‘it is not possible for humans to understand’. Sounds like a good way to hide a "Room 641A" situation from your own engineers... https://en.wikipedia.org/wiki/Room_641A https://en.wikipedia.org/wiki/Room_641A

MertsA 4y ago
It sounds like someone asked a vague question about a neural network during a deposition and didn't understand the response.

rwhaling 4y ago

Speaking as a data practitioner - data lineage is a profoundly hard and still unsolved problem. In the last few years, I feel like we've begun to crack it - dbt's data lineage graph is a huge advance over anything I've ever used before: https://docs.getdbt.com/terms/data-lineage https://docs.getdbt.com/terms/data-lineage But in general - as an industry, we aren't there yet. All judgments about Meta's practices aside - no large organization likely has the ability to comply with a law like this without years or decades of engineering work, and the law seems pretty clearly intended for selective enforcement.

cvccvroomvroom 4y ago

Uh, horse poo. Prod data is guarded like Fort Knox. No one can get access to anything on the graph without explicit approvals for a business purpose. I tried it. I can't even see who are my coworkers from my own FBID in internal apps.

hobbitstan 4y ago

This is clearly data chaos by design as a way to skirt privacy laws that may impact business performance and profits. The EU won’t fall for it!

JKCalhoun 4y ago

> When ordered to produce information about what 149 different data systems within Meta do, and what parts of Meta’s business use them, the company was unable to respond. This feels to me more like a "tantrum-compliance" (I made it up) with a court order. But what do I know about it (nothing), maybe there is data anarchy at Meta.