4 ms·

(Minus the extraneous ‘but’ in the first sentence)

chillacy 8y ago
Thanks for clarifying. Maybe I'm misunderstanding the situation. I work with "big data". I'm not sure what facebook is at but something's not adding up here because the statement > For the purposes of this discussion, a flat file is essentially a database. A CSV has fields and is easily made searchable absolutely doesn't apply to the type of data I'm thinking about. (Data in the petabytes+ is not easily made searchable, indexing isn't common, joins are basically impossible at scale. Instead the data is partitioned on a specific field which makes it fast to access by that field but slow to access on others. Usually for this type of data that field would be a timestamp, but it could also technically be partitioned by a userid which makes it easier to split on user at the cost of splitting by time, since you now need to search all partitions in order to find a time range. ) So either I'm mistaken about the nature of facebook's data (I assumed it was clickstream data, like "User X clicked Box Y at Time Z with Metadata A"), or I'm misunderstanding what you mean by that statement. Also > The overall point is that of course they can access the data by individual and not just by time of access. I can access clickstream data by a specific individual user with a 10 minute Mapreduce job, but re-partitioning the data is going to be a production. Apologize if you already know all of this, but I saw a lot of people making references to SQL in this extended thread which strikes me as strange because even sql-frontends to analytics backends like HIVE get compiled down to 10 minute mapreduce jobs, it's an entirely different world.
- code_duck 8y ago
  The overall question is, is that statement by Facebook misleading? In my opinion, yes. They are tracking individual behavior, not producing overall website analytics. There is no question that they can and do have that data in some way associated with a specific individual, not concentrating on time of access as if that’s the important part of the data. Are you saying you believe that Facebook just leave that data in difficult to access data-stores and doesn’t use it for anything? Considering it’s practically the core of their business? The implementation details are fairly unimportant. It’s what they do with the data, and the satement that it is “stored by” time of access is intended to be misleading to non-experts, in my opinion. Apparently they are even managing to distract experts and technical people.
  - chillacy 8y ago
    I’m not sure they are “tracking individual user behavior”, what do you mean by that? Usually when you train a model you extract features for each event and then you apply your model at prediction time. There is no step in which you have the user’s entire history set aside in an easy to query fashion. Instead the model learns aggregate user behavior. Out of curiosity do you work with analytics scale data? Nothing in their statements seemed odd to me, it would easily take a few months at my current company for those reasons. For analogy it would be like if fb said “oh we support IE 7 to 11 but it’s taking us longer to support IE6”, and people said “it’s just one lower than 7 how hard can it be? Facebook is clearly hiding something, I write for IE7 all the time”. And I’m not quite sure how to convey the fact that IE6 is a different beast altogether.
    - code_duck 8y ago
      Okay, so your position is that Facebook is unable to query for a list of activity by user, and does not have it divided by user in any other way? That seems nonsensical to me, but if you say so.
      - chillacy 8y ago
        Let's take Facebook out of the picture, it's too hot a topic. Let's talk about the datawarehouse of some other big tech company like amazon or adcrunch. The DW can be queried by user but each query might take 10 mins to evaluate, so it can't be used for realtime UIs. Instead you would have to denormalize the data to display to a user. That's a project, it might take a few months.
    - code_duck 8y ago
      To add to this, your questions are bizarre. What do I mean by ‘Facebook tracking individual user behavior’? Um... I don’t know what to say to that. What a strange question. It’s pretty clear what is meant by that. It’s the primary focus of their business. Next, “usually when you train a model you extract features for each event and then you apply your model at prediction time”. That’s nice, but when did we start discussing ‘training models’? Facebook has a list of internet activity. It’s stored by user. They’re in the business of selling personal data. Nothing you said has anything to do with that. I have no idea what we are even discussing. Whether someone can make an ad hoc a query by user is completely not the point and has nothing to do with anything. I would assume that they have extracted the information into other reports or data stores, obviously, not process trillions of lines of entries whenever they want to access it by individual.

Thanks for clarifying. Maybe I'm misunderstanding the situation. I work with "big data". I'm not sure what facebook is at but something's not adding up here because the statement > For the purposes of this discussion, a flat file is essentially a database. A CSV has fields and is easily made searchable absolutely doesn't apply to the type of data I'm thinking about. (Data in the petabytes+ is not easily made searchable, indexing isn't common, joins are basically impossible at scale. Instead the data is partitioned on a specific field which makes it fast to access by that field but slow to access on others. Usually for this type of data that field would be a timestamp, but it could also technically be partitioned by a userid which makes it easier to split on user at the cost of splitting by time, since you now need to search all partitions in order to find a time range. ) So either I'm mistaken about the nature of facebook's data (I assumed it was clickstream data, like "User X clicked Box Y at Time Z with Metadata A"), or I'm misunderstanding what you mean by that statement. Also > The overall point is that of course they can access the data by individual and not just by time of access. I can access clickstream data by a specific individual user with a 10 minute Mapreduce job, but re-partitioning the data is going to be a production. Apologize if you already know all of this, but I saw a lot of people making references to SQL in this extended thread which strikes me as strange because even sql-frontends to analytics backends like HIVE get compiled down to 10 minute mapreduce jobs, it's an entirely different world.

code_duck 8y ago
The overall question is, is that statement by Facebook misleading? In my opinion, yes. They are tracking individual behavior, not producing overall website analytics. There is no question that they can and do have that data in some way associated with a specific individual, not concentrating on time of access as if that’s the important part of the data. Are you saying you believe that Facebook just leave that data in difficult to access data-stores and doesn’t use it for anything? Considering it’s practically the core of their business? The implementation details are fairly unimportant. It’s what they do with the data, and the satement that it is “stored by” time of access is intended to be misleading to non-experts, in my opinion. Apparently they are even managing to distract experts and technical people.
- chillacy 8y ago
  I’m not sure they are “tracking individual user behavior”, what do you mean by that? Usually when you train a model you extract features for each event and then you apply your model at prediction time. There is no step in which you have the user’s entire history set aside in an easy to query fashion. Instead the model learns aggregate user behavior. Out of curiosity do you work with analytics scale data? Nothing in their statements seemed odd to me, it would easily take a few months at my current company for those reasons. For analogy it would be like if fb said “oh we support IE 7 to 11 but it’s taking us longer to support IE6”, and people said “it’s just one lower than 7 how hard can it be? Facebook is clearly hiding something, I write for IE7 all the time”. And I’m not quite sure how to convey the fact that IE6 is a different beast altogether.
  - code_duck 8y ago
    Okay, so your position is that Facebook is unable to query for a list of activity by user, and does not have it divided by user in any other way? That seems nonsensical to me, but if you say so.
    - chillacy 8y ago
      Let's take Facebook out of the picture, it's too hot a topic. Let's talk about the datawarehouse of some other big tech company like amazon or adcrunch. The DW can be queried by user but each query might take 10 mins to evaluate, so it can't be used for realtime UIs. Instead you would have to denormalize the data to display to a user. That's a project, it might take a few months.
  - code_duck 8y ago
    To add to this, your questions are bizarre. What do I mean by ‘Facebook tracking individual user behavior’? Um... I don’t know what to say to that. What a strange question. It’s pretty clear what is meant by that. It’s the primary focus of their business. Next, “usually when you train a model you extract features for each event and then you apply your model at prediction time”. That’s nice, but when did we start discussing ‘training models’? Facebook has a list of internet activity. It’s stored by user. They’re in the business of selling personal data. Nothing you said has anything to do with that. I have no idea what we are even discussing. Whether someone can make an ad hoc a query by user is completely not the point and has nothing to do with anything. I would assume that they have extracted the information into other reports or data stores, obviously, not process trillions of lines of entries whenever they want to access it by individual.