5 ms·

This is a seductive notion, but it's indeed possible for a tech company to understand how a single piece of data goes through its systems. It might take a whil

This is a seductive notion, but it's indeed possible for a tech company to understand how a single piece of data goes through its systems. It might take a while, and it might involve weeks of code spelunking and dozens of conversations with engineers — but it is indeed possible. A codebase is not encumbered with the physical constaints of an old city. It's not like trying to figure out the physical state of a certain area of London 30 meters deep — which might be impossible without digging and therefore disrupting other structures in the area. A codebase is simply instructions, which are readable and understandable (with the exception of black-box machine-learning models, but those at least have defined inputs and outputs).

leros 4y ago
The thing with tech though is that it's easy to add new connections, connecting anything to anything. You might have two independent datasets that can't be stored together due to privacy or security regulations. The teams who work on that know and understand that. But some other team working on something completely unrelated might join those datasets for some non-nefarious purpose, which then creates a new dataset with those data combined. Now other teams who also don't know about the regulations might build on top of that combined dataset. They might even generate new data fields from the data that's not supposed to be joined. This stuff can propogate through layers of systems, queries, service calls, analytical engines, offline spreadsheets, etc. It's very difficult to keep track while keeping your engineering teams independent and nimble.
rufus_foreman 4y ago
>> It might take a while, and it might involve weeks of code spelunking and dozens of conversations with engineers — but it is indeed possible If it takes a while it will be out of date by the time it is complete. A single request might directly impact a dozen separate systems, which in turn call other services. You won't be able to find anyone who knows much past the layer they directly interact with. All of these services are being worked on. Some older ones are in the process of being deprecated in favor of newer ones. Someone is already working on the design of the replacements for the newer ones. Changes are being pushed weekly if not daily. If you send a request and then send it again a minute later, it might go through a completely different set of services based on a feature toggle. You can't step into the same river twice.
notacoward 4y ago
> it might involve weeks of code spelunking and dozens of conversations with engineers It's worse than you think. First, a little background: I worked on one of Facebook's largest storage systems for two years. So let's talk about some concrete examples. I knew the rebalancing code better than anyone by the time I left. This is a pretty essential bit of functionality, running regularly on any cluster that has been up for any amount of time and moving quite a bit of data each time. After a chunk was copied, my code would go to delete the old one. I know for a fact that the deletion could fail without my code even getting a useful error from lower layers. I found and fixed many such cases, but I'm sure more remained. That could leave an "orphan" chunk where nobody would know to look for it, and chunks were self-identifying enough that if enough such orphans existed they could be reconstructed into a whole block possibly containing user data. I knew the data-repair code almost as well. Same problem. I knew the system-repair code less well. Similar there too. In fact I was involved in pulling back "repaired" systems and using them to recover data that would have been lost, more than once. I watched other engineers get rewarded for cleaning up petabytes' worth of no-longer-reachable data (because of cases like I'd mentioned, or bugs, or whatever) that was still taking up space on our millions of disks - again, multiple times. And that's all just one storage system. I'll bet others had similar issues. Also, the problems with truly erasing disks and particularly SSDs are pretty well known. If a machine was taken out of our system entirely and repurposed for another one, or vice versa, there could still be data on its platters/chips that could be recovered with sufficient forensic effort. (These things were physically destroyed before leaving FB, and I've even seen the impressive machines that do it, but not between "lifetimes" within the company.) So knowing the current code is not enough. You'd have to know every past state of the code during a relevant timeframe, including what bugs it had, which is challenging to say the least. Every configuration detail, too. You'd have to know every rebalancing, reconstruction, or repair event that might have affected each disk. It really is like figuring out the physical state of an area in London. Nothing short of scanning every sector, even as more hardware enters and leaves the system almost every minute, could produce an absolute guarantee. I'm not saying it can't be done in any system of similar size and complexity. Just that it couldn't in Facebook's, even without malice, largely because so much code was written without that in mind and it's a really hard thing to bolt on afterward. I know there are people working toward it. It's just going to take longer than most people - even technical people - think, and saying it's done would be a lie.
deleted 4y ago
[deleted]

leros 4y ago

The thing with tech though is that it's easy to add new connections, connecting anything to anything. You might have two independent datasets that can't be stored together due to privacy or security regulations. The teams who work on that know and understand that. But some other team working on something completely unrelated might join those datasets for some non-nefarious purpose, which then creates a new dataset with those data combined. Now other teams who also don't know about the regulations might build on top of that combined dataset. They might even generate new data fields from the data that's not supposed to be joined. This stuff can propogate through layers of systems, queries, service calls, analytical engines, offline spreadsheets, etc. It's very difficult to keep track while keeping your engineering teams independent and nimble.

rufus_foreman 4y ago

>> It might take a while, and it might involve weeks of code spelunking and dozens of conversations with engineers — but it is indeed possible If it takes a while it will be out of date by the time it is complete. A single request might directly impact a dozen separate systems, which in turn call other services. You won't be able to find anyone who knows much past the layer they directly interact with. All of these services are being worked on. Some older ones are in the process of being deprecated in favor of newer ones. Someone is already working on the design of the replacements for the newer ones. Changes are being pushed weekly if not daily. If you send a request and then send it again a minute later, it might go through a completely different set of services based on a feature toggle. You can't step into the same river twice.

notacoward 4y ago

> it might involve weeks of code spelunking and dozens of conversations with engineers It's worse than you think. First, a little background: I worked on one of Facebook's largest storage systems for two years. So let's talk about some concrete examples. I knew the rebalancing code better than anyone by the time I left. This is a pretty essential bit of functionality, running regularly on any cluster that has been up for any amount of time and moving quite a bit of data each time. After a chunk was copied, my code would go to delete the old one. I know for a fact that the deletion could fail without my code even getting a useful error from lower layers. I found and fixed many such cases, but I'm sure more remained. That could leave an "orphan" chunk where nobody would know to look for it, and chunks were self-identifying enough that if enough such orphans existed they could be reconstructed into a whole block possibly containing user data. I knew the data-repair code almost as well. Same problem. I knew the system-repair code less well. Similar there too. In fact I was involved in pulling back "repaired" systems and using them to recover data that would have been lost, more than once. I watched other engineers get rewarded for cleaning up petabytes' worth of no-longer-reachable data (because of cases like I'd mentioned, or bugs, or whatever) that was still taking up space on our millions of disks - again, multiple times. And that's all just one storage system. I'll bet others had similar issues. Also, the problems with truly erasing disks and particularly SSDs are pretty well known. If a machine was taken out of our system entirely and repurposed for another one, or vice versa, there could still be data on its platters/chips that could be recovered with sufficient forensic effort. (These things were physically destroyed before leaving FB, and I've even seen the impressive machines that do it, but not between "lifetimes" within the company.) So knowing the current code is not enough. You'd have to know every past state of the code during a relevant timeframe, including what bugs it had, which is challenging to say the least. Every configuration detail, too. You'd have to know every rebalancing, reconstruction, or repair event that might have affected each disk. It really is like figuring out the physical state of an area in London. Nothing short of scanning every sector, even as more hardware enters and leaves the system almost every minute, could produce an absolute guarantee. I'm not saying it can't be done in any system of similar size and complexity. Just that it couldn't in Facebook's, even without malice, largely because so much code was written without that in mind and it's a really hard thing to bolt on afterward. I know there are people working toward it. It's just going to take longer than most people - even technical people - think, and saying it's done would be a lie.

deleted 4y ago

[deleted]