11 ms·

Facebook engineers: we have no idea where we keep all your personal data

216 points by mikece 4y ago

personjerry 4y ago
Well that's because the question is meaningless - What does it mean "all your personal data"? What does "where" mean? Physically? Which tables? Which data is relevant? Friends? Friends of friends? Ad data? Behavioural data? ANY inferences, models built on user datasets that might include that user? They don't even know how to ask the right questions to get to what they want to know. Context: I worked at Facebook
- piva00 4y ago
  All your personal data in the article references data generated by inference from personal data, it was pretty clear in some paragraphs and quotes. The where part is also alluded to as to mean "systems" from what I gathered.
- seanhunter 4y ago
  Companies which deal with personal data do so in the context of regulatory frameworks (CCPA, GDPR etc) which define specifically what the definition is and what’s covered. They also have to record what they collect and process and for what purpose. If you’re Facebook you have to know this in order to process that data. You can’t pretend not to understand the question.
  - lupire 4y ago
    The people who write the question don't understand the question. They are "not even wrong". They are telling the businesses to figure it out. Maybe that's desirable, or maybe it's pointless. In prod strokes you can say data is either fully controlled by the users, shared with other user data for product model building, or also used for ad targeting. But it's unclear if being more specific is useful to anyone. You can point to an example like equal housing opportunity enforcement, as it interacts with ML-targeted or attribute -targeted ads. It's equivalent to detecting discrimination anywhere a human makes an opinionated choice. It's a problem no one knows how to solve. It' not a law like "don't shoot or stab people" (which is also complex in some ways).
- gthrone 4y ago
  Ya the way he responds saying a full team would be needed to uncover this sounds like he was treating "where" holistically. Every bit of information in every server, application, etc. Feels like a trained response.
  - PeterisP 4y ago
    The way I read this, his response saying that a full team would be needed to uncover this is not a flaw in the question - because IMHO that's exactly what was required - but rather that Facebook has not done the work required to meet their legal requirements. Yes, it's quite plausible that Facebook might need a team to do extensive work to produce the analysis and documentation about where private data is flowing - that's not a valid excuse though, Facebook simply needs to make that team and do that work, no matter if they want to or not, until they can properly answer these questions. They need to have an exhaustive list of how they're using private data, and they need to have a process ensuring that their engineers are not adding new sources of private data or not using existing private data without the company approving and updating that list. Yes, their current processes aren't fit for that - as Meta documents quoted in TFA say "We do not have an adequate level of control and explainability over how our systems use data, and thus we can’t confidently make controlled policy changes or external commitments such as ‘we will not use X data for Y purpose,’" - so these processes must be changed.
- RichardCNormos 4y ago
  [flagged]
  - Hauthorn 4y ago
    Or maybe just implying that Meta isn't up for the task of complying with GDPR, and lack the necessary ability to comply with regulations.
  - lolc 4y ago
    The way I understood it, the questions determined that Facebook didn't even try to gather the evidence they were required to. Saying "we didn't want to do that because it is hard" may fly when they know how hard it is. But they don't even know that! Institutionally incapable because nobody pays for it, they argued.
  - AlwaysRock 4y ago
    So why would Meta send someone to explain who doesnt have the capacity to answer those questions? At best they sent the wrong person on accident or they couldnt send the right people for some unknown reason. More likely, they send people with limited knowledge and tell them to only answer questions about areas they are 100% certain about. If Meta actually wanted to answer they could have 25 people in this room with all the answers. But they dont want to actually answer this question.
- zo1 4y ago
  I just started reading the transcript. They asked a lot of questions trying to figure out details about various products. It's 850 pages long, and the few spots I jumped to the guy kept answering with "I don't know", "I'm not familiar with this", etc. Even the parts he says "I'm familiar with this service", the guy goes on to answer "I don't know more than what's written there". The guy couldn't even answer if certain databases were accessible by third parties (not implying they were). This must have been painful to be a part of. 100 variations of "I can neither confirm nor deny".
- alexhill 4y ago
  > They don't even know how to ask the right questions to get to what they want to know. You're right. If the questions were being asked of someone who was genuinely motivated to answer them, this wouldn't be an obstacle. The phrase "all your personal data" (or the quote from the article, "every bit of data associated with a given user account") feels vague and imprecise only if you have an internal understanding that for practical purposes, it really is impossible. The Facebook engineers know that and everyone here knows that. It's not literally impossible - an internal team of digital archaeologists with unlimited resources and no deadline might be able to do it for a single user. But practically impossible across the whole userbase. If you were genuinely interested in providing insight to the hearing, you could explain all that, and then explain how you could go about creating a rough sketch of an answer, and how you'd go about adding some detail to that sketch. The Facebook engineers have no real motivation to provide insight off their own bat. They can take the question literally, and answer honestly that they don't know, that nobody else knows, and that it's probably impossible TO know. I'm not even sure if this is acting in bad faith. I can easily imagine feeling cagey in this situation, and responding as such. Experiencing an emotion isn't acting in bad faith. All that said...these questions were asked by a court-appointed subject matter expert. Either the subject matter expert is not the right person for the job, or...and now I realise that of course I haven't read the transcript and there's every chance that what's quoted in the article is not the most insight the court managed to prise out of the engineers. posting anyway in 3 2 1
- PeterisP 4y ago
  Yes. All of that. That is the right question, the very first question as a starting point. A basic requirement from a company is to provide an exhaustive, true, up-to-date list of all of these - and if they can't, they should not be permitted to handle private data.
  - nathanaldensr 4y ago
    Agreed. I'm tired of these companies assuming that it's some kind of natural order or divine right to collect this data to begin with.
- AlwaysRock 4y ago
  >Well that's because the question is meaningless No. >What does it mean "all your personal data"? It means all your personal data. >What does "where" mean? Where facebook stores it. This is only a complicated question if you want to start saying, "Well this isnt really their personal data, we just use it for x..." If its data related to my account or me as a user it's my personal data. Maybe its difficult to track all that down but certainly that is a challenge worth tackling in a world where GDPR exists.
- kwertyoowiyop 4y ago
  Sure, and what does “is” mean exactly? And what really is “data”? Their answers seem like passive-aggressive BS. If their lives or careers depended on it, they’d come up with a lot of answers PDQ.
motbus 4y ago
deleted 4y ago
[deleted]
jtbayly 4y ago
> The special master at times seemed in disbelief, as when he questioned the engineers over whether any documentation existed for a particular Facebook subsystem. “Someone must have a diagram that says this is where this data is stored,” he said, according to the transcript. Zarashaw responded: “We have a somewhat strange engineering culture compared to most where we don’t generate a lot of artifacts during the engineering process. Effectively the code is its own design document often.” He quickly added, “For what it’s worth, this is terrifying to me when I first joined as well.” The cynic in me wants to say that not documenting certain decisions is a common method of hiding immoral decisions.
- namelessoracle 4y ago
  The cynic in me says it's building your own job security. Documentation only really gets created if leadership forces the issue in my experience. You also get to sell the refactor/rewrite because "nobody understand the existing code".
  - 988747 4y ago
    I've been a programmer for 15 years and I've never seen it done for "job security". I think this is mostly about low tolerance for boredom of most programmers: writing code is exciting, documenting it is boring, so you do not do it unless forced to.
    - namelessoracle 4y ago
      Ive seen people "neglect" to offer training peers or document things before to protect their own position. Anecdotes and all that. I mean people "usually" wont say that out loud of course.
- roland35 4y ago
  Trust me, at (most) big tech companies there isn't anything that nefarious going on at the engineer level. In fact everyone basically agrees that improving documentation is important, but it constantly gets swept aside in the constant churn of new features, fires to put out, and frequent meetings. Information gets spread out across various docs, wikis, readmes, etc. What ends up happening is each part of the system has a "point of contact" who is the go to for that piece. Lots of decision history is in the heads of team elders. Hopefully they are still around!
mrweasel 4y ago
Besides the legal issues, isn’t there also a technical problem? Facebook is potentially storing the same information multiple times, or missing opportunities, because a engineer did know that some data was available. I guess storage is cheap enough, but even for Facebook, being able to save say 10% on storage must mean something.
jerf 4y ago
Kind of serves as a good example of "single central conspiracy" versus "many actors given common cause to act in a certain way" ways of thought. It's easy to imagine Zuckerberg going to work every day and laughing maniacally as he personally shepherds your stolen conversation with friends yesterday about your experiences with laundry detergents into The Facebook Info Vault and then personally uses that information to send you ads about Tide laundry detergent, but what Facebook really is by sheer necessity is a whole bunch of agents operating with their own goals. The end result is a massive machine that turns privacy violations into money, but there isn't necessarily a single place where the bad thing happens. In fact you could conceivably be taken on a tour of the system and agree that every individual component is acceptable, or that the vast bulk of them are OK and there's only a single-digit number out of thousands that are problematic. Nobody could possibly manage Facebook as a centralized single entity, but it's hard to imagine it any other way from the outside.
- jkingsbery 4y ago
  While I appreciate this distinction and is an interesting way to look at things, I don't think it's inevitable. I've never worked at Facebook, but I work for another large tech company. Every project at our company, particularly the ones that deal with personal data, must produce a threat model [1] before going into production. In another thread, someone claims "Well that's because the question is meaningless," but from a threat modeling standpoint an engineer needs to understand all aspects of "where" data is for the system under design. Where physically is it stored? What database is it stored in? And knowing that, one must look at what threats are available. Who has access to the data? What other systems have access to the data? How are those users/systems authenticated? Is that access logged? Now, sometimes systems diverge from the design, and sometimes threat models are incomplete. But the exercise of generating a threat model makes understanding who manages data more manageable. [1]: https://en.wikipedia.org/wiki/Threat_model https://en.wikipedia.org/wiki/Threat_model
  - lupire 4y ago
    > understand all aspects of "where" data is for the system under design. There easy! But also useless! The data is everywhere. That's why every "where" was created. Everything is a potential threat vector and needs to be protected by layers of fallible security.
ctur 4y ago
In many ways you can think of large, long-living tech companies not unlike old cities like, say, London or Paris. The buildings and roads you see are built on top of older buildings and ruins. The streets are weirdly shaped and intersect at odd angles because they were made hundreds of years before and adapted over time as needs evolved. There are catacombs underneath sidewalks and no one genuinely understands it all nor does a single reference exist that explains it. Literally everything predates everyone who lives there. Generations and generations of original designers, architects, and laborers have arrived, plied their trade, and moved away. There are people who are experts in certain parts, and who can build a new skyscraper at any given spot, but it is just layering and organic growth. The emergent complexity of centuries of being lived in and adapted belies easy understanding. Large tech companies are similar. You just can't understand how "it" all works. If you were to build it from scratch, perhaps you could, because it would be simpler and clearer, but nothing was made with the current state in mind. It evolved and adapted over time. So reading this, I am not surprised. I think you'd get the same answer about many other aspects of data, code, system history, etc at any other 10+ yr old tech giant.
- TomSwirly 4y ago
  What you appear to be saying is that large technology companies are designed from the ground up around flouting the laws about data retention. Why is this OK?
- borbulon 4y ago
  yeah "favela" is usually the metaphor I use, but it's the same general gist, if a little messier.
- CaptainTaboo 4y ago
  It's not just the tech giants. I work for a company that is fairly dominant in its particular industry (If you're in the US, you've probably done business with one of its customers). Much of our growth has come through the acquisition of other companies, each of which gathers its own user and customer data. From my perspective in IT operations, I don't have any visibility into how we capture user and customer data, but I know we're big enough to track a user across the various products and websites, but still siloed enough that internally we still refer to the various business units as if they were separate organizations, each holding its own subset of the data. To sum up, this description of Facebook's internals doesn't surprise me at all.
LinuxBender 4y ago
This article reminds me of past audit experiences. It is important to know who to send into the conference room to talk to the auditor. The wrong people can answer too many questions or volunteer the wrong information, leading to more questions and going deeper down the rabbit holes. I suppose the same could apply in this case. It's just a gut feeling.
handity 4y ago
After reading some more of the transcript, I think the article does such a bad job of describing what was being asked for that it makes the court seem incompetent and Facebook actually rather reasonable. My original comment is below: Government contracted construction workers: We Have No Idea Where Your Tax Money Goes When tasked with answering the simple question "Which specific bricks did my tax money buy this year", the two veteran construction workers looked confused and tried to explain that that's not really how taxes work. The special master at times seemed in disbelief, as when he questioned the engineers over whether any invoices existed for a particular road building contract. “Someone must have a receipt that says this is who the money came from that bought these bricks” I'm not sure the analogy fits 100%, but it's the closest I could think of. This reads like the author thinks the Facebook algorithm is a human-readable decision tree that only takes into account the data of a single user at a time. As usual, the problem is not data "collection" or "retention" or "privacy", but "creation". Regulation will always remain woefully inadequate to control such organizations, and the only solution is to adopt systems that don't spew user data everywhere in the first place.
- twanvl 4y ago
  Money is fungible, personal data is not.
  - jtbayly 4y ago
    Ha! “Fungible” is the word I settled on, too.
  - guywithahat 4y ago
    You'd think that but personal data piles up quickly and the individuals who make it up dispensary. It's a little like saying any particular dollar is unique; I suppose it is but if you get enough of them nobody cares. I think expecting Facebook to treat all of their data as if it's ITAR is unfortunately unrealistic, we're not all that unique or special and if you use the internet your data has already been leaked or is otherwise public
- jtbayly 4y ago
mathgladiator 4y ago
I believe we need a privacy centered view of technology. Even a traditional database creates a spiderweb of 'where is your personal data spread across so many tables', and I hope there is a reckoning against this approach. I have considered pivoting my service into a privacy respecting data store ( https://www.adama-platform.com/ https://www.adama-platform.com/ ), but I've yet to meet anyone that actually cares about user privacy to rethink their data layer.
- fsociety 4y ago
  Actually FB has some wicked privacy tech around their central data store - and are building more. Nuance gets lost in these articles, for example even with proper documentation there are simply too many things for one person to know where data flows in that company. Or data flowing through RPC calls can make it hard to know what will fail downstream if you remove a certain piece of data.
  - mathgladiator 4y ago
    Alternatively, how that privacy logic is built and the infra is an albatross. And worse, a checkbox without any guarantees.
hk1337 4y ago
I hear /dev/null is a good place for large amounts of personal data.
pavlov 4y ago
> The systemic fogginess of Facebook’s data storage made answering even the most basic question futile. At another point, the special master asked how one could find out which systems actually contain user data that was created through machine inference. “I don’t know,” answered Zarashaw. “It’s a rather difficult conundrum.” This is not a basic question because of how ML works. Can we say a system contains my data if my data was used as a training input to a neural network at some point in the past? In that case, can I sue anyone using Stable Diffusion for stealing my data because the billions of images in its training set included something I created?
- dannyw 4y ago
  > In that case, can I sue anyone using Stable Diffusion for stealing my data because the billions of images in its training set included something I created? No, because of the landmark Authors Guild, Inc. v. Google, Inc. case which found that ML training can be considered fair use in some circumstances.
  - pavlov 4y ago
    So why do we expect Meta to internally track this kind of data usage? Seems like it’s both fair use and covered by the terms of service that FB users have agreed to.
    - kmeisthax 4y ago
      Copyright and privacy law are two different animals. Copyright law covers things that are expected to be published - i.e. creative works. We decide that authors get to control how and when they are published. Privacy law is different: the subject is information that is held in confidence and should not be published under any circumstance. Fair use would circumvent the "copyright my nudes so I can DMCA them off Facebook" kind of action, but not, say, the GDPR. GDPR does not care about fair use, and its skeptical of TOS agreements.
tru3_power 4y ago
There seems to be a pretty concerted effort to paint Meta as worse than the other big ad-tech companies. Maybe it’s just the contrarian in me, but I’ve noticed a way big uptick in these stories over the year.
snowwrestler 4y ago
There is a difference between what is known and what is knowable. For example if you asked me now if I know where all the receipts are for my tax-deductible donations, I would say no, I don’t know. Why? There is no reason for me to know right now. But if the IRS told me I was being audited and must produce those records or go to jail, I would find them. And I have confidence that I could, because I know that in general, those are the sorts of records I keep (somewhere). Most Facebook employees do not need to know “where personal data is” to operate their business, so they don’t know that. But at the same time, Facebook could not operate if the data was “nowhere” (destroyed), or if it was randomly distributed with no queryable structure. So the question is not really what they know now, it is: what incentive do they have to go find out.
IncRnd 4y ago
> Zarashaw responded: “We have a somewhat strange engineering culture compared to most where we don’t generate a lot of artifacts during the engineering process. Effectively the code is its own design document often.”
kornhole 4y ago
Because I suspected this mess, I took a less expedient path to deleting my account. I found a picture of somebody with my same name and made it my picture. I unfriended everybody I actually knew and only kept randos. I liked a bunch of random stuff. I did not actually delete my account until a year later to let the chaotic data propagate into all the subsystems, backups, and partner networks.
Delphiza 4y ago
Many here on HN are saying that such a record of what tech companies do with data is a) impossible because complex systems have evolved over time and the (historical) flows of data are unknown and b) unnecessary because the need to know what happens to data is not relevant to the user or the questioner. The old models of processing data at scale to extract value, as FAANG has been doing, are eventually going to come to an end. Maybe not where there is a power imbalance, such as between Facebook and their users, but at least where users of services have more clout and are paying for the product. I see this in B2B IoT solutions where big customers are very picky about how telemetry is collected by product vendors and, if not pushing back hard, are at least choosing not to use services that are not clear on how data is processed and handled. The amount of data that we all see, every day, that is grossly mishandled may signal the end of the ML and AI goldrush. You can only build models, and run data through models, with the consent of the owner of that data. Large producers of data (think vehicle fleet operators) are beginning to take ownership of their data, and are only _licensing_ it to processors for very specific purposes. In the example of vehicle fleet operators, they may only want route planning, and not have their data used to sell them tyres based on mileage. Also, while governments may be busy with other stuff currently, at some point they may decide to turn on the regulatory screws.
canoebuilder 4y ago
What are we talking about here? Data associated with specific users? Well in order for your grandma to log into Facebook her user account must have a primary key associated with it so she sees her info when she logs in and not someone else’s. We are talking about computers and databases. When did using a computer to search a database become a difficult, nigh impossible thing to do? Even if design documents and flow charts or whatever don’t exist could they not fairly straightforwardly be reverse engineered by taking a sample of users and searching all databases for associated information? This seems like a transparent ploy on the part of Facebook to avoid regulation by casting the perfectly doable, searching a database, into some sort of incomprehensible impossible task. The credulous author of the article and many comments here seem to be strangely buying into it. FB investors don’t seem to have lost faith in the company’s ability to search databases. When it comes to making the company money from those searches, no problem at all. But regarding lawsuits or potential regulation, we can only throw our hands into the air and simply wonder and gasp at this incomprehensible mess we have made.
- ctide 4y ago
  It became difficult when your 'database' turned into exabytes of random files in s3.
ridgered4 4y ago
I recall when Windows 10 came out some company had demanded to know all of the things it sent back to Microsoft over the wire. Microsoft had some guy basically run wireshark and a bunch of network scans while using it and put that in a report. It blew my mind because it implied nobody at Microsoft actually knew what it was collecting and when anymore than your average security researcher, possibly because there are so many sub-entities in the company hoovering it up for different purposes.
- kornhole 4y ago
  When I worked in Redmond, they pressed us heavily to collect telemetry where possible. Each line of business basically operated as a separate company with many subsystems and no effective way of connecting customers or users between them. I have been at many big companies and seen similar. If these big companies with expertise and money can't manage their data well, it should be no surprise to anybody how poorly it is handled at smaller companies.
0xbadcafebee 4y ago
"The special master at times seemed in disbelief, as when he questioned the engineers over whether any documentation existed for a particular Facebook subsystem. “Someone must have a diagram that says this is where this data is stored,” he said, according to the transcript." Oh you sweet summer child.
citizenpaul 4y ago
This is misdirection plain and simple. In the best most convincing way. Get tunnel vision (sorry) nerds to make an honest display of their ignorance of the bigger picture at Meta. Aww shucks we don't know where that data comes from.... Of course senior engineers don't know where the data comes from exactly. Does your mechanic know where each tire comes from or care? Does your fav restaurant chef know each field that a produce came from? The suppliers know that info. I know beyond any doubt that there are most definitively some people that know at least roughly where most data about most subjects are at Meta. I'm sure the data is a vastly beyond what a human can process. However some people there know where to go looking. The truth is in the Piles of money they generate by selling that data. Awww gee wizz your honor. We have no idea how we do it. People just keep throwing piles of money at us so we keep taking it. I don't know why they do it. We definitely don't have mountains of highly specialized data on our users that we tell them we can get exactly what they want then provide that info to them. Somehow though when the court asks without piles of money we just can't make that magic happen...........
- notacoward 4y ago
  > there are most definitively some people that know at least roughly "At least roughly" is doing a lot of work here. I think what you're getting at is that there are people who know which storage systems are likely to hold that data, but knowing exactly which objects do is far more challenging. Copies get made while data pipelines are tweaked, and then forgotten. Copies get made internally on storage systems as disks/machines fail and get repaired, as rebalancing occurs, and in other cases that typically only one or two members of the team understand at all. (I worked on one of Facebook's largest storage systems for two years BTW.) Copies get made for AI training, experiments, shadow-traffic generation, etc. Could they do better? Should they do better? Yes and yes. But it's not a lie to say nobody knows how to find every last orphan and fragment. They can pretty reliably delete everything they could find to sell, but that's a very different thing and you're conflating the two.
  - citizenpaul 4y ago
    I'm saying they have vast mountains of data on people. They make enough money from it to be one of the most profitable companies on the planet. This is not happening by accident. I'm quite certain that if I showed up to FB with a $50 million dollar ad campaign to run they would have quite the opposite story about their data knowledge. Marching dozens of people into the room promising they can slice and dice every bit of info down to the atom for my needs. Yet in court they want to claim they live by the seat of their pants. Its a deception tactic. Even though I'm sure its as you describe partly true in technical concept. I do believe that FB is the least privacy invasive of all the big tech giants for what its worth. Nothing against them. I actually think this case is an intentional aggravation to the courts to have regulation created on data collection. That way FAANGM can cement themselves as the offical national data brokers with costly startup regulation for any compeditors.
y42 4y ago
I assume it's the same in every company after a particular amount of time. They start clean and then everything grows organically because you spend more money in innovation than cleaning up and documentation. It's not even a thing of competition or capitalism, but limited resources.
ElijahLynn 4y ago
The special master at times seemed in disbelief, as when he questioned the engineers over whether any documentation existed for a particular Facebook subsystem. “Someone must have a diagram that says this is where this data is stored,” he said, according to the transcript. Zarashaw responded: “We have a somewhat strange engineering culture compared to most where we don’t generate a lot of artifacts during the engineering process. Effectively the code is its own design document often.” He quickly added, “For what it’s worth, this is terrifying to me when I first joined as well.” “We do not have an adequate level of control and explainability over how our systems use data, and thus we can’t confidently make controlled policy changes or external commitments such as ‘we will not use X data for Y purpose,’” the 2021 document read.
deleted 4y ago
[deleted]
notacoward 4y ago
The basic problem is one of tracking (or not tracking) copies of data. Your information at Facebook is stored primarily on several storage systems, but extra copies might exist in backups, data-analysis pipelines, logging systems, etc. At the time I left (two years ago), there was a cutely-named project well under way to ensure deletion of those primary copies and (IIRC) backups. Fragments still in data-analysis systems might still exist, but they're also less personally identifying. I don't recall the state of anonymization and provenance tracking that would allow even these remnants to be found and purged for certain. So we're basically talking about two different questions. (1) Is it possible to be sure that the primary copies and backups are gone, so that finding anything that's left would require some very specialized knowledge and/or an infeasibly massive scan? I believe the answer to this is probably yes at this point. (2) Is it possible to be sure that absolutely every last vestige of the person's time on Facebook is gone? I believe the answer to this is still probably no, and likely to remain so for some time. At the very last, some artifacts will remain in those opaque AI models. I suspect the same two answers exist at many companies. What is HN's data retention policy? Oh, oops, none of my data would be deleted if I left. I suspect that Google and/or Apple, maybe Microsoft as well, are a lot closer to "yes" on that second question, but even then I suspect gaps appear from time to time. I say this not to condemn or defend anyone, and I know companies under stricter regulatory regimes can give more definite answers. It's just the state of the art at Big Tech companies as I understand it.
riddleronroof 4y ago
Isn’t this a good thing? Every Facebook employee should not know where all PII is. Further, it doesn’t say what data they were looking for. Eg Around the time of the scandal, lots of Facebook quizzes were asking for PII. but it was not stored at facebook. I’m not a Facebook fan, but I’m not a fan of unfounded scary stories either.
salawat 4y ago
I would not be surprised at all. I can count on one hand the number of Engineering candidates that given a system description involving three visible networked laptops in view of the camera have answered "where is the data, right now?" , or can at least give a decent stab at "trace the datapath through the machine". I can't explain somehow this is the average candidate, yet somehow... Life goes on.
egberts1 4y ago
Not surprising that the same can be said for a certain US government department having hosted a personal mail server in a bathroom closet and trafficking classified materials and caught much later by OIG. Or that one time I found a dialup modem under the raised floor and attached to the Equifax (then TRW) mainframe. Or catching someone inserting a USB stick into a PC inside a secured white-lab area. Absolutely crazy times for large entities.

216 points by mikece 4y ago

Well that's because the question is meaningless - What does it mean "all your personal data"? What does "where" mean? Physically? Which tables? Which data is relevant? Friends? Friends of friends? Ad data? Behavioural data? ANY inferences, models built on user datasets that might include that user? They don't even know how to ask the right questions to get to what they want to know. Context: I worked at Facebook

piva00 4y ago
All your personal data in the article references data generated by inference from personal data, it was pretty clear in some paragraphs and quotes. The where part is also alluded to as to mean "systems" from what I gathered.
seanhunter 4y ago
Companies which deal with personal data do so in the context of regulatory frameworks (CCPA, GDPR etc) which define specifically what the definition is and what’s covered. They also have to record what they collect and process and for what purpose. If you’re Facebook you have to know this in order to process that data. You can’t pretend not to understand the question.
- lupire 4y ago
  The people who write the question don't understand the question. They are "not even wrong". They are telling the businesses to figure it out. Maybe that's desirable, or maybe it's pointless. In prod strokes you can say data is either fully controlled by the users, shared with other user data for product model building, or also used for ad targeting. But it's unclear if being more specific is useful to anyone. You can point to an example like equal housing opportunity enforcement, as it interacts with ML-targeted or attribute -targeted ads. It's equivalent to detecting discrimination anywhere a human makes an opinionated choice. It's a problem no one knows how to solve. It' not a law like "don't shoot or stab people" (which is also complex in some ways).
gthrone 4y ago
Ya the way he responds saying a full team would be needed to uncover this sounds like he was treating "where" holistically. Every bit of information in every server, application, etc. Feels like a trained response.
- PeterisP 4y ago
  The way I read this, his response saying that a full team would be needed to uncover this is not a flaw in the question - because IMHO that's exactly what was required - but rather that Facebook has not done the work required to meet their legal requirements. Yes, it's quite plausible that Facebook might need a team to do extensive work to produce the analysis and documentation about where private data is flowing - that's not a valid excuse though, Facebook simply needs to make that team and do that work, no matter if they want to or not, until they can properly answer these questions. They need to have an exhaustive list of how they're using private data, and they need to have a process ensuring that their engineers are not adding new sources of private data or not using existing private data without the company approving and updating that list. Yes, their current processes aren't fit for that - as Meta documents quoted in TFA say "We do not have an adequate level of control and explainability over how our systems use data, and thus we can’t confidently make controlled policy changes or external commitments such as ‘we will not use X data for Y purpose,’" - so these processes must be changed.
RichardCNormos 4y ago
[flagged]
- Hauthorn 4y ago
  Or maybe just implying that Meta isn't up for the task of complying with GDPR, and lack the necessary ability to comply with regulations.
- lolc 4y ago
  The way I understood it, the questions determined that Facebook didn't even try to gather the evidence they were required to. Saying "we didn't want to do that because it is hard" may fly when they know how hard it is. But they don't even know that! Institutionally incapable because nobody pays for it, they argued.
- AlwaysRock 4y ago
  So why would Meta send someone to explain who doesnt have the capacity to answer those questions? At best they sent the wrong person on accident or they couldnt send the right people for some unknown reason. More likely, they send people with limited knowledge and tell them to only answer questions about areas they are 100% certain about. If Meta actually wanted to answer they could have 25 people in this room with all the answers. But they dont want to actually answer this question.
zo1 4y ago
I just started reading the transcript. They asked a lot of questions trying to figure out details about various products. It's 850 pages long, and the few spots I jumped to the guy kept answering with "I don't know", "I'm not familiar with this", etc. Even the parts he says "I'm familiar with this service", the guy goes on to answer "I don't know more than what's written there". The guy couldn't even answer if certain databases were accessible by third parties (not implying they were). This must have been painful to be a part of. 100 variations of "I can neither confirm nor deny".
alexhill 4y ago
> They don't even know how to ask the right questions to get to what they want to know. You're right. If the questions were being asked of someone who was genuinely motivated to answer them, this wouldn't be an obstacle. The phrase "all your personal data" (or the quote from the article, "every bit of data associated with a given user account") feels vague and imprecise only if you have an internal understanding that for practical purposes, it really is impossible. The Facebook engineers know that and everyone here knows that. It's not literally impossible - an internal team of digital archaeologists with unlimited resources and no deadline might be able to do it for a single user. But practically impossible across the whole userbase. If you were genuinely interested in providing insight to the hearing, you could explain all that, and then explain how you could go about creating a rough sketch of an answer, and how you'd go about adding some detail to that sketch. The Facebook engineers have no real motivation to provide insight off their own bat. They can take the question literally, and answer honestly that they don't know, that nobody else knows, and that it's probably impossible TO know. I'm not even sure if this is acting in bad faith. I can easily imagine feeling cagey in this situation, and responding as such. Experiencing an emotion isn't acting in bad faith. All that said...these questions were asked by a court-appointed subject matter expert. Either the subject matter expert is not the right person for the job, or...and now I realise that of course I haven't read the transcript and there's every chance that what's quoted in the article is not the most insight the court managed to prise out of the engineers. posting anyway in 3 2 1
PeterisP 4y ago
Yes. All of that. That is the right question, the very first question as a starting point. A basic requirement from a company is to provide an exhaustive, true, up-to-date list of all of these - and if they can't, they should not be permitted to handle private data.
- nathanaldensr 4y ago
  Agreed. I'm tired of these companies assuming that it's some kind of natural order or divine right to collect this data to begin with.
AlwaysRock 4y ago
>Well that's because the question is meaningless No. >What does it mean "all your personal data"? It means all your personal data. >What does "where" mean? Where facebook stores it. This is only a complicated question if you want to start saying, "Well this isnt really their personal data, we just use it for x..." If its data related to my account or me as a user it's my personal data. Maybe its difficult to track all that down but certainly that is a challenge worth tackling in a world where GDPR exists.
kwertyoowiyop 4y ago
Sure, and what does “is” mean exactly? And what really is “data”? Their answers seem like passive-aggressive BS. If their lives or careers depended on it, they’d come up with a lot of answers PDQ.

motbus 4y ago

deleted 4y ago

[deleted]

jtbayly 4y ago

> The special master at times seemed in disbelief, as when he questioned the engineers over whether any documentation existed for a particular Facebook subsystem. “Someone must have a diagram that says this is where this data is stored,” he said, according to the transcript. Zarashaw responded: “We have a somewhat strange engineering culture compared to most where we don’t generate a lot of artifacts during the engineering process. Effectively the code is its own design document often.” He quickly added, “For what it’s worth, this is terrifying to me when I first joined as well.” The cynic in me wants to say that not documenting certain decisions is a common method of hiding immoral decisions.

namelessoracle 4y ago
The cynic in me says it's building your own job security. Documentation only really gets created if leadership forces the issue in my experience. You also get to sell the refactor/rewrite because "nobody understand the existing code".
- 988747 4y ago
  I've been a programmer for 15 years and I've never seen it done for "job security". I think this is mostly about low tolerance for boredom of most programmers: writing code is exciting, documenting it is boring, so you do not do it unless forced to.
  - namelessoracle 4y ago
    Ive seen people "neglect" to offer training peers or document things before to protect their own position. Anecdotes and all that. I mean people "usually" wont say that out loud of course.
roland35 4y ago
Trust me, at (most) big tech companies there isn't anything that nefarious going on at the engineer level. In fact everyone basically agrees that improving documentation is important, but it constantly gets swept aside in the constant churn of new features, fires to put out, and frequent meetings. Information gets spread out across various docs, wikis, readmes, etc. What ends up happening is each part of the system has a "point of contact" who is the go to for that piece. Lots of decision history is in the heads of team elders. Hopefully they are still around!

mrweasel 4y ago

Besides the legal issues, isn’t there also a technical problem? Facebook is potentially storing the same information multiple times, or missing opportunities, because a engineer did know that some data was available. I guess storage is cheap enough, but even for Facebook, being able to save say 10% on storage must mean something.

jerf 4y ago

Kind of serves as a good example of "single central conspiracy" versus "many actors given common cause to act in a certain way" ways of thought. It's easy to imagine Zuckerberg going to work every day and laughing maniacally as he personally shepherds your stolen conversation with friends yesterday about your experiences with laundry detergents into The Facebook Info Vault and then personally uses that information to send you ads about Tide laundry detergent, but what Facebook really is by sheer necessity is a whole bunch of agents operating with their own goals. The end result is a massive machine that turns privacy violations into money, but there isn't necessarily a single place where the bad thing happens. In fact you could conceivably be taken on a tour of the system and agree that every individual component is acceptable, or that the vast bulk of them are OK and there's only a single-digit number out of thousands that are problematic. Nobody could possibly manage Facebook as a centralized single entity, but it's hard to imagine it any other way from the outside.

jkingsbery 4y ago
While I appreciate this distinction and is an interesting way to look at things, I don't think it's inevitable. I've never worked at Facebook, but I work for another large tech company. Every project at our company, particularly the ones that deal with personal data, must produce a threat model [1] before going into production. In another thread, someone claims "Well that's because the question is meaningless," but from a threat modeling standpoint an engineer needs to understand all aspects of "where" data is for the system under design. Where physically is it stored? What database is it stored in? And knowing that, one must look at what threats are available. Who has access to the data? What other systems have access to the data? How are those users/systems authenticated? Is that access logged? Now, sometimes systems diverge from the design, and sometimes threat models are incomplete. But the exercise of generating a threat model makes understanding who manages data more manageable. [1]: https://en.wikipedia.org/wiki/Threat_model https://en.wikipedia.org/wiki/Threat_model
- lupire 4y ago
  > understand all aspects of "where" data is for the system under design. There easy! But also useless! The data is everywhere. That's why every "where" was created. Everything is a potential threat vector and needs to be protected by layers of fallible security.

ctur 4y ago

In many ways you can think of large, long-living tech companies not unlike old cities like, say, London or Paris. The buildings and roads you see are built on top of older buildings and ruins. The streets are weirdly shaped and intersect at odd angles because they were made hundreds of years before and adapted over time as needs evolved. There are catacombs underneath sidewalks and no one genuinely understands it all nor does a single reference exist that explains it. Literally everything predates everyone who lives there. Generations and generations of original designers, architects, and laborers have arrived, plied their trade, and moved away. There are people who are experts in certain parts, and who can build a new skyscraper at any given spot, but it is just layering and organic growth. The emergent complexity of centuries of being lived in and adapted belies easy understanding. Large tech companies are similar. You just can't understand how "it" all works. If you were to build it from scratch, perhaps you could, because it would be simpler and clearer, but nothing was made with the current state in mind. It evolved and adapted over time. So reading this, I am not surprised. I think you'd get the same answer about many other aspects of data, code, system history, etc at any other 10+ yr old tech giant.

TomSwirly 4y ago
What you appear to be saying is that large technology companies are designed from the ground up around flouting the laws about data retention. Why is this OK?
borbulon 4y ago
yeah "favela" is usually the metaphor I use, but it's the same general gist, if a little messier.
CaptainTaboo 4y ago
It's not just the tech giants. I work for a company that is fairly dominant in its particular industry (If you're in the US, you've probably done business with one of its customers). Much of our growth has come through the acquisition of other companies, each of which gathers its own user and customer data. From my perspective in IT operations, I don't have any visibility into how we capture user and customer data, but I know we're big enough to track a user across the various products and websites, but still siloed enough that internally we still refer to the various business units as if they were separate organizations, each holding its own subset of the data. To sum up, this description of Facebook's internals doesn't surprise me at all.

LinuxBender 4y ago

This article reminds me of past audit experiences. It is important to know who to send into the conference room to talk to the auditor. The wrong people can answer too many questions or volunteer the wrong information, leading to more questions and going deeper down the rabbit holes. I suppose the same could apply in this case. It's just a gut feeling.

handity 4y ago

After reading some more of the transcript, I think the article does such a bad job of describing what was being asked for that it makes the court seem incompetent and Facebook actually rather reasonable. My original comment is below: Government contracted construction workers: We Have No Idea Where Your Tax Money Goes When tasked with answering the simple question "Which specific bricks did my tax money buy this year", the two veteran construction workers looked confused and tried to explain that that's not really how taxes work. The special master at times seemed in disbelief, as when he questioned the engineers over whether any invoices existed for a particular road building contract. “Someone must have a receipt that says this is who the money came from that bought these bricks” I'm not sure the analogy fits 100%, but it's the closest I could think of. This reads like the author thinks the Facebook algorithm is a human-readable decision tree that only takes into account the data of a single user at a time. As usual, the problem is not data "collection" or "retention" or "privacy", but "creation". Regulation will always remain woefully inadequate to control such organizations, and the only solution is to adopt systems that don't spew user data everywhere in the first place.

twanvl 4y ago
Money is fungible, personal data is not.
- jtbayly 4y ago
  Ha! “Fungible” is the word I settled on, too.
- guywithahat 4y ago
  You'd think that but personal data piles up quickly and the individuals who make it up dispensary. It's a little like saying any particular dollar is unique; I suppose it is but if you get enough of them nobody cares. I think expecting Facebook to treat all of their data as if it's ITAR is unfortunately unrealistic, we're not all that unique or special and if you use the internet your data has already been leaked or is otherwise public
jtbayly 4y ago

mathgladiator 4y ago

I believe we need a privacy centered view of technology. Even a traditional database creates a spiderweb of 'where is your personal data spread across so many tables', and I hope there is a reckoning against this approach. I have considered pivoting my service into a privacy respecting data store ( https://www.adama-platform.com/ https://www.adama-platform.com/ ), but I've yet to meet anyone that actually cares about user privacy to rethink their data layer.

fsociety 4y ago
Actually FB has some wicked privacy tech around their central data store - and are building more. Nuance gets lost in these articles, for example even with proper documentation there are simply too many things for one person to know where data flows in that company. Or data flowing through RPC calls can make it hard to know what will fail downstream if you remove a certain piece of data.
- mathgladiator 4y ago
  Alternatively, how that privacy logic is built and the infra is an albatross. And worse, a checkbox without any guarantees.

hk1337 4y ago

I hear /dev/null is a good place for large amounts of personal data.

pavlov 4y ago

> The systemic fogginess of Facebook’s data storage made answering even the most basic question futile. At another point, the special master asked how one could find out which systems actually contain user data that was created through machine inference. “I don’t know,” answered Zarashaw. “It’s a rather difficult conundrum.” This is not a basic question because of how ML works. Can we say a system contains my data if my data was used as a training input to a neural network at some point in the past? In that case, can I sue anyone using Stable Diffusion for stealing my data because the billions of images in its training set included something I created?

dannyw 4y ago
> In that case, can I sue anyone using Stable Diffusion for stealing my data because the billions of images in its training set included something I created? No, because of the landmark Authors Guild, Inc. v. Google, Inc. case which found that ML training can be considered fair use in some circumstances.
- pavlov 4y ago
  So why do we expect Meta to internally track this kind of data usage? Seems like it’s both fair use and covered by the terms of service that FB users have agreed to.
  - kmeisthax 4y ago
    Copyright and privacy law are two different animals. Copyright law covers things that are expected to be published - i.e. creative works. We decide that authors get to control how and when they are published. Privacy law is different: the subject is information that is held in confidence and should not be published under any circumstance. Fair use would circumvent the "copyright my nudes so I can DMCA them off Facebook" kind of action, but not, say, the GDPR. GDPR does not care about fair use, and its skeptical of TOS agreements.

tru3_power 4y ago

There seems to be a pretty concerted effort to paint Meta as worse than the other big ad-tech companies. Maybe it’s just the contrarian in me, but I’ve noticed a way big uptick in these stories over the year.

snowwrestler 4y ago

There is a difference between what is known and what is knowable. For example if you asked me now if I know where all the receipts are for my tax-deductible donations, I would say no, I don’t know. Why? There is no reason for me to know right now. But if the IRS told me I was being audited and must produce those records or go to jail, I would find them. And I have confidence that I could, because I know that in general, those are the sorts of records I keep (somewhere). Most Facebook employees do not need to know “where personal data is” to operate their business, so they don’t know that. But at the same time, Facebook could not operate if the data was “nowhere” (destroyed), or if it was randomly distributed with no queryable structure. So the question is not really what they know now, it is: what incentive do they have to go find out.

IncRnd 4y ago

> Zarashaw responded: “We have a somewhat strange engineering culture compared to most where we don’t generate a lot of artifacts during the engineering process. Effectively the code is its own design document often.”

kornhole 4y ago

Because I suspected this mess, I took a less expedient path to deleting my account. I found a picture of somebody with my same name and made it my picture. I unfriended everybody I actually knew and only kept randos. I liked a bunch of random stuff. I did not actually delete my account until a year later to let the chaotic data propagate into all the subsystems, backups, and partner networks.

Delphiza 4y ago

Many here on HN are saying that such a record of what tech companies do with data is a) impossible because complex systems have evolved over time and the (historical) flows of data are unknown and b) unnecessary because the need to know what happens to data is not relevant to the user or the questioner. The old models of processing data at scale to extract value, as FAANG has been doing, are eventually going to come to an end. Maybe not where there is a power imbalance, such as between Facebook and their users, but at least where users of services have more clout and are paying for the product. I see this in B2B IoT solutions where big customers are very picky about how telemetry is collected by product vendors and, if not pushing back hard, are at least choosing not to use services that are not clear on how data is processed and handled. The amount of data that we all see, every day, that is grossly mishandled may signal the end of the ML and AI goldrush. You can only build models, and run data through models, with the consent of the owner of that data. Large producers of data (think vehicle fleet operators) are beginning to take ownership of their data, and are only _licensing_ it to processors for very specific purposes. In the example of vehicle fleet operators, they may only want route planning, and not have their data used to sell them tyres based on mileage. Also, while governments may be busy with other stuff currently, at some point they may decide to turn on the regulatory screws.

canoebuilder 4y ago

What are we talking about here? Data associated with specific users? Well in order for your grandma to log into Facebook her user account must have a primary key associated with it so she sees her info when she logs in and not someone else’s. We are talking about computers and databases. When did using a computer to search a database become a difficult, nigh impossible thing to do? Even if design documents and flow charts or whatever don’t exist could they not fairly straightforwardly be reverse engineered by taking a sample of users and searching all databases for associated information? This seems like a transparent ploy on the part of Facebook to avoid regulation by casting the perfectly doable, searching a database, into some sort of incomprehensible impossible task. The credulous author of the article and many comments here seem to be strangely buying into it. FB investors don’t seem to have lost faith in the company’s ability to search databases. When it comes to making the company money from those searches, no problem at all. But regarding lawsuits or potential regulation, we can only throw our hands into the air and simply wonder and gasp at this incomprehensible mess we have made.

ctide 4y ago
It became difficult when your 'database' turned into exabytes of random files in s3.

ridgered4 4y ago

I recall when Windows 10 came out some company had demanded to know all of the things it sent back to Microsoft over the wire. Microsoft had some guy basically run wireshark and a bunch of network scans while using it and put that in a report. It blew my mind because it implied nobody at Microsoft actually knew what it was collecting and when anymore than your average security researcher, possibly because there are so many sub-entities in the company hoovering it up for different purposes.

kornhole 4y ago
When I worked in Redmond, they pressed us heavily to collect telemetry where possible. Each line of business basically operated as a separate company with many subsystems and no effective way of connecting customers or users between them. I have been at many big companies and seen similar. If these big companies with expertise and money can't manage their data well, it should be no surprise to anybody how poorly it is handled at smaller companies.

0xbadcafebee 4y ago

"The special master at times seemed in disbelief, as when he questioned the engineers over whether any documentation existed for a particular Facebook subsystem. “Someone must have a diagram that says this is where this data is stored,” he said, according to the transcript." Oh you sweet summer child.

citizenpaul 4y ago

This is misdirection plain and simple. In the best most convincing way. Get tunnel vision (sorry) nerds to make an honest display of their ignorance of the bigger picture at Meta. Aww shucks we don't know where that data comes from.... Of course senior engineers don't know where the data comes from exactly. Does your mechanic know where each tire comes from or care? Does your fav restaurant chef know each field that a produce came from? The suppliers know that info. I know beyond any doubt that there are most definitively some people that know at least roughly where most data about most subjects are at Meta. I'm sure the data is a vastly beyond what a human can process. However some people there know where to go looking. The truth is in the Piles of money they generate by selling that data. Awww gee wizz your honor. We have no idea how we do it. People just keep throwing piles of money at us so we keep taking it. I don't know why they do it. We definitely don't have mountains of highly specialized data on our users that we tell them we can get exactly what they want then provide that info to them. Somehow though when the court asks without piles of money we just can't make that magic happen...........

notacoward 4y ago
> there are most definitively some people that know at least roughly "At least roughly" is doing a lot of work here. I think what you're getting at is that there are people who know which storage systems are likely to hold that data, but knowing exactly which objects do is far more challenging. Copies get made while data pipelines are tweaked, and then forgotten. Copies get made internally on storage systems as disks/machines fail and get repaired, as rebalancing occurs, and in other cases that typically only one or two members of the team understand at all. (I worked on one of Facebook's largest storage systems for two years BTW.) Copies get made for AI training, experiments, shadow-traffic generation, etc. Could they do better? Should they do better? Yes and yes. But it's not a lie to say nobody knows how to find every last orphan and fragment. They can pretty reliably delete everything they could find to sell, but that's a very different thing and you're conflating the two.
- citizenpaul 4y ago
  I'm saying they have vast mountains of data on people. They make enough money from it to be one of the most profitable companies on the planet. This is not happening by accident. I'm quite certain that if I showed up to FB with a $50 million dollar ad campaign to run they would have quite the opposite story about their data knowledge. Marching dozens of people into the room promising they can slice and dice every bit of info down to the atom for my needs. Yet in court they want to claim they live by the seat of their pants. Its a deception tactic. Even though I'm sure its as you describe partly true in technical concept. I do believe that FB is the least privacy invasive of all the big tech giants for what its worth. Nothing against them. I actually think this case is an intentional aggravation to the courts to have regulation created on data collection. That way FAANGM can cement themselves as the offical national data brokers with costly startup regulation for any compeditors.

y42 4y ago

I assume it's the same in every company after a particular amount of time. They start clean and then everything grows organically because you spend more money in innovation than cleaning up and documentation. It's not even a thing of competition or capitalism, but limited resources.

ElijahLynn 4y ago

The special master at times seemed in disbelief, as when he questioned the engineers over whether any documentation existed for a particular Facebook subsystem. “Someone must have a diagram that says this is where this data is stored,” he said, according to the transcript. Zarashaw responded: “We have a somewhat strange engineering culture compared to most where we don’t generate a lot of artifacts during the engineering process. Effectively the code is its own design document often.” He quickly added, “For what it’s worth, this is terrifying to me when I first joined as well.” “We do not have an adequate level of control and explainability over how our systems use data, and thus we can’t confidently make controlled policy changes or external commitments such as ‘we will not use X data for Y purpose,’” the 2021 document read.

deleted 4y ago

[deleted]

notacoward 4y ago

The basic problem is one of tracking (or not tracking) copies of data. Your information at Facebook is stored primarily on several storage systems, but extra copies might exist in backups, data-analysis pipelines, logging systems, etc. At the time I left (two years ago), there was a cutely-named project well under way to ensure deletion of those primary copies and (IIRC) backups. Fragments still in data-analysis systems might still exist, but they're also less personally identifying. I don't recall the state of anonymization and provenance tracking that would allow even these remnants to be found and purged for certain. So we're basically talking about two different questions. (1) Is it possible to be sure that the primary copies and backups are gone, so that finding anything that's left would require some very specialized knowledge and/or an infeasibly massive scan? I believe the answer to this is probably yes at this point. (2) Is it possible to be sure that absolutely every last vestige of the person's time on Facebook is gone? I believe the answer to this is still probably no, and likely to remain so for some time. At the very last, some artifacts will remain in those opaque AI models. I suspect the same two answers exist at many companies. What is HN's data retention policy? Oh, oops, none of my data would be deleted if I left. I suspect that Google and/or Apple, maybe Microsoft as well, are a lot closer to "yes" on that second question, but even then I suspect gaps appear from time to time. I say this not to condemn or defend anyone, and I know companies under stricter regulatory regimes can give more definite answers. It's just the state of the art at Big Tech companies as I understand it.

riddleronroof 4y ago

Isn’t this a good thing? Every Facebook employee should not know where all PII is. Further, it doesn’t say what data they were looking for. Eg Around the time of the scandal, lots of Facebook quizzes were asking for PII. but it was not stored at facebook. I’m not a Facebook fan, but I’m not a fan of unfounded scary stories either.

salawat 4y ago

I would not be surprised at all. I can count on one hand the number of Engineering candidates that given a system description involving three visible networked laptops in view of the camera have answered "where is the data, right now?" , or can at least give a decent stab at "trace the datapath through the machine". I can't explain somehow this is the average candidate, yet somehow... Life goes on.

egberts1 4y ago

Not surprising that the same can be said for a certain US government department having hosted a personal mail server in a bathroom closet and trafficking classified materials and caught much later by OIG. Or that one time I found a dialup modem under the raised floor and attached to the Equifax (then TRW) mainframe. Or catching someone inserting a USB stick into a PC inside a secured white-lab area. Absolutely crazy times for large entities.