7 ms·

> "Facebook currently stores browsing data by date and time, not by which user it belongs to" I call bull on this. I seriously doubt Facebook is collecting thi

> "Facebook currently stores browsing data by date and time, not by which user it belongs to" I call bull on this. I seriously doubt Facebook is collecting this browsing history and not associating it with users.

code_duck 8y ago
‘Stored by’ is a bit of a trick or a joke as far as a databases. What would that even mean, to a technical person? The primary key? One could say it is ‘stored by’ any field that you could query for. Obviously they have the records associated with individuals, considering that’s the focus of their business.
- chillacy 8y ago
  It’s probably not a database if I had to guess, given the volume of event stream data. At my current company we store event streams as time organized flat files in HDFS for analytics. Our site isn’t as big as facebooks but it measures into the low petabytes across all time. So how you store the data is important in a post-sql world.
  - code_duck 8y ago
    I’m sure they’re not but using a standard relational database too. They obviously extract the information and work with it, so whether it’s ‘a database’ is basically an implementation detail. For the purposes of this discussion, a flat file is essentially a database. A CSV has fields and is easily made searchable. The overall point is that of course they can access the data by individual and not just by time of access.
    - code_duck 8y ago
      (Minus the extraneous ‘but’ in the first sentence)
      - chillacy 8y ago
        Thanks for clarifying. Maybe I'm misunderstanding the situation. I work with "big data". I'm not sure what facebook is at but something's not adding up here because the statement > For the purposes of this discussion, a flat file is essentially a database. A CSV has fields and is easily made searchable absolutely doesn't apply to the type of data I'm thinking about. (Data in the petabytes+ is not easily made searchable, indexing isn't common, joins are basically impossible at scale. Instead the data is partitioned on a specific field which makes it fast to access by that field but slow to access on others. Usually for this type of data that field would be a timestamp, but it could also technically be partitioned by a userid which makes it easier to split on user at the cost of splitting by time, since you now need to search all partitions in order to find a time range. ) So either I'm mistaken about the nature of facebook's data (I assumed it was clickstream data, like "User X clicked Box Y at Time Z with Metadata A"), or I'm misunderstanding what you mean by that statement. Also > The overall point is that of course they can access the data by individual and not just by time of access. I can access clickstream data by a specific individual user with a 10 minute Mapreduce job, but re-partitioning the data is going to be a production. Apologize if you already know all of this, but I saw a lot of people making references to SQL in this extended thread which strikes me as strange because even sql-frontends to analytics backends like HIVE get compiled down to 10 minute mapreduce jobs, it's an entirely different world.
        code_duck 8y ago
        The overall question is, is that statement by Facebook misleading? In my opinion, yes. They are tracking individual behavior, not producing overall website analytics. There is no question that they can and do have that data in some way associated with a specific individual, not concentrating on time of access as if that’s the important part of the data. Are you saying you believe that Facebook just leave that data in difficult to access data-stores and doesn’t use it for anything? Considering it’s practically the core of their business? The implementation details are fairly unimportant. It’s what they do with the data, and the satement that it is “stored by” time of access is intended to be misleading to non-experts, in my opinion. Apparently they are even managing to distract experts and technical people.
marricks 8y ago
“[Stores] by date and time” doesn’t preclude having user information in there either! It’s the perfect non-answer.
- stochastic_monk 8y ago
  It’s like saying the contents of a dictionary don’t matter because they’re sorted lexicographically.
adonnjohn 8y ago
Imagine the legal liability if they found legitimately dangerous trends! They probably have simple ways to reference the IP address of where that data came from; they probably have a user id correlated to each data record. Their policy is one of those 'technically the truth' statements that is gravely overshadowed by other events that have transpired.
isostatic 8y ago
Yup, it's in a big spreadsheet. Column A is the default sort, and that's Date/Time. Column B is the user, and they don't sort by column B, that would be crazy!
- dictum 8y ago
  AUDITOR: So that's how you store these visits... wait, why are the values next to the URLs so... strange? ALICE: Column A is the URL. Column B is just a token we use to ensure it's a valid entry, and it's the default sort. As you can see, it's gibberish. No personally identifiable information. JANE: We deriv... [JAKE hits JANE with his elbow, discreetly] JAKE: We derive great pleasure from helping people connect with each other, is what Jane's saying. And that's something we truly believe in.
- jlarocco 8y ago
  You would think they'd know how to access the data by now given all of their experience using it to target ads to specific users...
  - jhenkens 8y ago
    The amount of data they collect surely they stream much of it directly to log files. Those log files are not by user. You try reading and rewriting petabytes of data out of S3 to remove lines matching a specified ID.
aiiane 8y ago
Reading between the lines, what this statement actually means is likely "we have a lot of this user data in flat logs, which makes it hard to delete a single individual's data out of the logs, so we need to move from flat user-agnostic logs to per-user structured data".
- jandrese 8y ago
  Given their business model of sifting through and selling your data this doesn't seem all that likely that they've left it in gigantic combined flatfiles. It seems more likely to me that yes, they could do a DELETE FROM BROWSING_HISTORY WHERE USER = "%S", but that's not their business model. Instead they have to add a column to the database that flags data that will be shown to the user or not, and then go through every part of the site where the data might be visible and add a check for that flag. It's a lot more work, and worse they have to get it right the first time because people will be angry if there is a leak somewhere and their supposedly deleted data is visible.
  - tylerhou 8y ago
    Or, maybe, making a product which has to deal with legacy code for billions of users is actually difficult? Like, we know Facebook is evil, but they can’t be that evil, right?
    - deleted 8y ago
      [deleted]
    - jandrese 8y ago
      > Like, we know Facebook is evil, but they can’t be that evil, right? People keep saying this and they keep being surprised.
    - emilsedgh 8y ago
      I don't know man when they wanted to generate a highlight video of new year's resolution for each user and they went from 0 to executed in a few days and blogged about it they seemed very capable people.

code_duck 8y ago

‘Stored by’ is a bit of a trick or a joke as far as a databases. What would that even mean, to a technical person? The primary key? One could say it is ‘stored by’ any field that you could query for. Obviously they have the records associated with individuals, considering that’s the focus of their business.

chillacy 8y ago
It’s probably not a database if I had to guess, given the volume of event stream data. At my current company we store event streams as time organized flat files in HDFS for analytics. Our site isn’t as big as facebooks but it measures into the low petabytes across all time. So how you store the data is important in a post-sql world.
- code_duck 8y ago
  I’m sure they’re not but using a standard relational database too. They obviously extract the information and work with it, so whether it’s ‘a database’ is basically an implementation detail. For the purposes of this discussion, a flat file is essentially a database. A CSV has fields and is easily made searchable. The overall point is that of course they can access the data by individual and not just by time of access.
  - code_duck 8y ago
    (Minus the extraneous ‘but’ in the first sentence)
    - chillacy 8y ago
      Thanks for clarifying. Maybe I'm misunderstanding the situation. I work with "big data". I'm not sure what facebook is at but something's not adding up here because the statement > For the purposes of this discussion, a flat file is essentially a database. A CSV has fields and is easily made searchable absolutely doesn't apply to the type of data I'm thinking about. (Data in the petabytes+ is not easily made searchable, indexing isn't common, joins are basically impossible at scale. Instead the data is partitioned on a specific field which makes it fast to access by that field but slow to access on others. Usually for this type of data that field would be a timestamp, but it could also technically be partitioned by a userid which makes it easier to split on user at the cost of splitting by time, since you now need to search all partitions in order to find a time range. ) So either I'm mistaken about the nature of facebook's data (I assumed it was clickstream data, like "User X clicked Box Y at Time Z with Metadata A"), or I'm misunderstanding what you mean by that statement. Also > The overall point is that of course they can access the data by individual and not just by time of access. I can access clickstream data by a specific individual user with a 10 minute Mapreduce job, but re-partitioning the data is going to be a production. Apologize if you already know all of this, but I saw a lot of people making references to SQL in this extended thread which strikes me as strange because even sql-frontends to analytics backends like HIVE get compiled down to 10 minute mapreduce jobs, it's an entirely different world.
      - code_duck 8y ago
        The overall question is, is that statement by Facebook misleading? In my opinion, yes. They are tracking individual behavior, not producing overall website analytics. There is no question that they can and do have that data in some way associated with a specific individual, not concentrating on time of access as if that’s the important part of the data. Are you saying you believe that Facebook just leave that data in difficult to access data-stores and doesn’t use it for anything? Considering it’s practically the core of their business? The implementation details are fairly unimportant. It’s what they do with the data, and the satement that it is “stored by” time of access is intended to be misleading to non-experts, in my opinion. Apparently they are even managing to distract experts and technical people.

marricks 8y ago

“[Stores] by date and time” doesn’t preclude having user information in there either! It’s the perfect non-answer.

stochastic_monk 8y ago
It’s like saying the contents of a dictionary don’t matter because they’re sorted lexicographically.

adonnjohn 8y ago

Imagine the legal liability if they found legitimately dangerous trends! They probably have simple ways to reference the IP address of where that data came from; they probably have a user id correlated to each data record. Their policy is one of those 'technically the truth' statements that is gravely overshadowed by other events that have transpired.

isostatic 8y ago

Yup, it's in a big spreadsheet. Column A is the default sort, and that's Date/Time. Column B is the user, and they don't sort by column B, that would be crazy!

dictum 8y ago
AUDITOR: So that's how you store these visits... wait, why are the values next to the URLs so... strange? ALICE: Column A is the URL. Column B is just a token we use to ensure it's a valid entry, and it's the default sort. As you can see, it's gibberish. No personally identifiable information. JANE: We deriv... [JAKE hits JANE with his elbow, discreetly] JAKE: We derive great pleasure from helping people connect with each other, is what Jane's saying. And that's something we truly believe in.
jlarocco 8y ago
You would think they'd know how to access the data by now given all of their experience using it to target ads to specific users...
- jhenkens 8y ago
  The amount of data they collect surely they stream much of it directly to log files. Those log files are not by user. You try reading and rewriting petabytes of data out of S3 to remove lines matching a specified ID.

aiiane 8y ago

Reading between the lines, what this statement actually means is likely "we have a lot of this user data in flat logs, which makes it hard to delete a single individual's data out of the logs, so we need to move from flat user-agnostic logs to per-user structured data".

jandrese 8y ago
Given their business model of sifting through and selling your data this doesn't seem all that likely that they've left it in gigantic combined flatfiles. It seems more likely to me that yes, they could do a DELETE FROM BROWSING_HISTORY WHERE USER = "%S", but that's not their business model. Instead they have to add a column to the database that flags data that will be shown to the user or not, and then go through every part of the site where the data might be visible and add a check for that flag. It's a lot more work, and worse they have to get it right the first time because people will be angry if there is a leak somewhere and their supposedly deleted data is visible.
- tylerhou 8y ago
  Or, maybe, making a product which has to deal with legacy code for billions of users is actually difficult? Like, we know Facebook is evil, but they can’t be that evil, right?
  - deleted 8y ago
    [deleted]
  - jandrese 8y ago
    > Like, we know Facebook is evil, but they can’t be that evil, right? People keep saying this and they keep being surprised.
  - emilsedgh 8y ago
    I don't know man when they wanted to generate a highlight video of new year's resolution for each user and they went from 0 to executed in a few days and blogged about it they seemed very capable people.