6 ms·

I was about to dismiss this but have thought about it a bit more, and there's something interesting here. There are already APIs built on top of you. There ar

I was about to dismiss this but have thought about it a bit more, and there's something interesting here. There are already APIs built on top of you. There are the ones you know about, like Facebook and Twitter. There are the ones we know exist but have no direct access to, such as Amazon's backend or your police record. And there are the ones we may never know about, such as the NSA bot scanning this post right now and correlating it to my recent overseas cell phone calls. I'm intrigued by the concept of aggregating this data. Imagine if the wealth of data collected about you all went to one place, and, most importantly, was controlled by you. Imagine if Company X had to pay you to know about your Amazon spending habits instead of paying Amazon. Imagine if scientists could perform long-term health studies in an instant because the data they need is already collected and just needs to be crunched. I think the concept that's missing from your blog post is the idea of discoverability.

jckt 13y ago
+1, but do you mind expanding on "discoverability"?
- abraininavat 13y ago
  I'm no expert, but it occurs to me that a mountain of data coming in from disparate sources needs the concept of discoverability to be adequately used. That is, there has to be a way for a machine to find occurrences of data that it cares about without specifically knowing the exact attributes of the data. A simple example off the top of my head. Let's say there's a popular app that collects your pulse from your cellphone somehow, and that data is stored in your "Personal API" in some format. Some scientists write a routine that scans people's pulse patterns (with their permission, say) and warns them of possible risk factors. Later on hospitals start feeding their pulse data into the same personal API, but in a different format and using different units (bpm instead of bph, say), because they've never heard of the cellphone app. I take "discoverability" as a general concept addressing how the scientists' routine might discover that the hospital data exists in a person's "Personal API", discover that the thing being measured by the hospital is the same as the thing being measured by the app, and discover how to map both formats into a common workable format and units.
  - fosk 13y ago
    If I understand correctly, isn't this the purpose of a good documentation?
    - abraininavat 13y ago
      I don't think so. Documentation is for people. Discoverability is for machines. Notice I suggested it's the scientists' routine which does the discovery, not the scientists themselves.
    - klenwell 13y ago
      It's more about money. This data is worth a lot of money to marketers, retailers, pharmaceutical companies, etc. They have no incentive to share it freely or, I suspect, at a price reasonable to most individual consumers. (Maybe I'm wrong here?) There was that interesting article a while back about how Target was able to figure out a teenage girl was pregnant before her own father did: http://www.nytimes.com/2012/02/19/magazine/shopping-habits.html?pagewanted=all&_r=0 http://www.nytimes.com/2012/02/19/magazine/shopping-habits.h... It glosses this point: Target can buy data about your ethnicity, job history, the magazines you read, if you’ve ever declared bankruptcy or got divorced, the year you bought (or lost) your house, where you went to college, what kinds of topics you talk about online, whether you prefer certain brands of coffee, paper towels, cereal or applesauce, your political leanings, reading habits, charitable giving and the number of cars you own. (In a statement, Target declined to identify what demographic information it collects or purchases.) I'd be very curious, and a little scared, to see all this kind of information collected on me.
  - naveenium 13y ago
    couldn't one also keep track of the 'source' from where a data point comes (i do this, but don't reveal it in the API - yet)? this way, when it comes to discoverability via some algorithm, you could always separate the 'verified' bucket from the 'unknown' bucket?
- klenwell 13y ago
  A trivial example from my own recent experience: getting my highlights and notes out of the Amazon Kindle app on my phone. Amazon could make these available through a simple RSS feed or export feature, but they don't. The only way I can get my hands on them is to scrape their website, which depends on some fragile scripting and is subject to their terms of service. I really like this idea, but without some kind of positive government regulation, I don't see it happening. At least not in the sort of meaningful ways the OP envisions.
rasur 13y ago
Have you read Jaron Laniers new book "Who Owns the Future"? You have similar ideas ("Imagine if the wealth of data collected about you all went to one place, and, most importantly, was controlled by you. Imagine if Company X had to pay you to know about your Amazon spending habits instead of paying Amazon.") It's a thought provoking book.. and slightly worrisome to boot (about the state of things at the moment, with regard to the economics of "sharing data").
- naveenium 13y ago
  yes, i do think of it that way: as we leave all this data exhaust behind us, i keep wanting a system to have it all in one place – or at least, an "interface" that shows it to me in one place, even if that's not how the backend really is. thanks for the book recommendation - i will check it out this weekend!
  - amirmc 13y ago
    Academia has been thinking about this for a while. I'm involved with several projects that aim to put users in control of their 'digital footprint' and what you've described are two aspects of the work. First is how to collate and access all the data and second is how to allow third parties to query/process that data in a way that respects a users' privacy. [1] I'm keen to commercialize the work when it's ready but it's a research project precisely because it's a tough problem. Of course, one approach is to just make that personal info public but I don't think that will really benefit users in the long term (companies would certainly benefit, though) [1] http://perscon.net/overview/dataware.html http://perscon.net/overview/dataware.html
- msvan 13y ago
  Jaron Lanier looks and behaves like a total eccentric, and his ideas can seem pretty out there, but I found his latest book to be a really interesting read. This idea is not the same thing though, because the data ends up being more or less public and demonetized. Companies are already collecting lots of data about you. This, in theory, just gives them even more of it, free of charge.
icebraining 13y ago
This is very much the goal of Project VRM: 1. https://cyber.law.harvard.edu/projectvrm/Main_Page https://cyber.law.harvard.edu/projectvrm/Main_Page 2. http://blogs.law.harvard.edu/vrm/ http://blogs.law.harvard.edu/vrm/
- j_s 13y ago
  Neat project! Took me about 5 clicks to find out that VRM is defined as 'Vendor Relationship Management'.
  - icebraining 13y ago
    Hmm, it's the first phrase of the first link. About VRM VRM stands for Vendor Relationship Management. (...)
wslh 13y ago
In my spare time I am doing something like it. It's called Egont (No Ego OR Ego New Technology). If you liked this article, you might like: i) Egont, a web orchestration language: http://blog.databigbang.com/ideas-egont-a-web-orchestration-language/ http://blog.databigbang.com/ideas-egont-a-web-orchestration-... ii) Egont part II: http://blog.databigbang.com/egont-part-ii/ http://blog.databigbang.com/egont-part-ii/ iii) Parsing s-expressions in C# using OMeta: http://blog.databigbang.com/parsing-s-expressions-in-c-using-ometa/ http://blog.databigbang.com/parsing-s-expressions-in-c-using... Because your stuff is defined as an s-expression and then optimized to a DAG.
- quantumpotato_ 13y ago
  [1] http://en.wikipedia.org/wiki/Directed_acyclic_graph http://en.wikipedia.org/wiki/Directed_acyclic_graph ? Can you explain more for us layfolk?
  - wslh 13y ago
    Yes, An s-expression is a tree. Imagine that you want to know the average score of a list of films like: (average listOfFilms) and someone else like the same average for the same list. You will end up with two trees that indeed are the same and can be calculated only once. A DAG is a way to represent dependencies, so if two cells are linked to the same cell (average listOfFilm).
    - wslh 13y ago
      One last clarification: not all trees can be converted to a DAG. There cannot be circular references.
  - deleted 13y ago
    [deleted]
  - deleted 13y ago
    [deleted]
  - deleted 13y ago
    [deleted]
iusable 13y ago
Agree 100%! The point that's important to remember about discoverability is that it is usually mission-centric. e.g. - you want to discover the 'right' kind of people to follow, as to have better or more entertaining or more useful or _____________ information. What is 'right' for you may not be 'right' for me, since our goals most likely aren't the same. I love Naveen's idea (and more importantly his implemention) & will be playing with it a bit more this weekend. But, the value may not be apparent right away. Aggregating and mining one type of data across a relationship may be more valuable. It may be that I have to track a similar set of data for myself and my significant other to make sense of the aggregates.
skybrian 13y ago
I think this is a very libertarian notion that misses out on the social aspects of reputation. What you claim about yourself is often less valuable to a third party than what someone else claims about you. Perhaps you could store signed endorsements made by other people. But then you need support for revocation, and what about negative claims? Free speech includes the ability to say negative things about others. Social software is very complicated because of issues like this. Something like a personal dropbox is straightforward - single customer, no community. You only have to worry about people hacking you, not what they do to each other. Anyone who gets into social software inevitably sets themselves up for making rules and judging disputes, and that often requires hard decisions. Or compare email with social messaging; the rights and responsibilities are different and it's due to the software. Personal servers only support setting ground rules in a certain way (for example, the inability to revoke or prevent copies). Centralized services can support communities with different rules, for better or worse.
graue 13y ago
> Imagine if the wealth of data collected about you all went to one place, and, most importantly, was controlled by you. Imagine if Company X had to pay you to know about your Amazon spending habits instead of paying Amazon. Even more than Project VRM linked above, this is precisely the eventual goal of Tent: https://tent.io/ https://tent.io/ That said, I'm skeptical of these projects', or anyone's, ability to achieve this utopia. Why would Amazon or Google adopt such a system? Your shopping/searching history is valuable data that gives them a business advantage. Also, while a personal data store that you control may be a geek's wet dream, for an average nontechnical person, it would likely be difficult to use, insecure and offer no clear benefits. And if it ends up just a toy for geeks, the lack of scale would make it even harder to do anything interesting with it or get any companies of significant size to play ball. I do hope ideas like this surface in some form, but I'm skeptical of the personal data store ever really becoming a thing.
- abraininavat 13y ago
  Why would Amazon or Google adopt such a system I could imagine laws requiring that personal data be controlled by the subjects. it would likely be difficult to use, insecure... Who says? How can you assume this? and offer no clear benefits Some would consider the ability to keep your information private at will a benefit. Others would consider the ability to sell said information a benefit. the lack of scale would make it even harder to do anything interesting Lack of scale? An individual's data would be controlled by that individual, but the system would scale across all of humanity. I don't think you're thinking big enough -- I'm not talking about each person running his personal EC2.
  - graue 13y ago
    Look, I'm not saying don't go for this idea. I'd love to see you try it. My doubts shouldn't get in your way. Code it up. As for why it would be difficult to use and insecure? I'm guessing most of the people you hang out with are probably geeks. For perspective, take a look at this video: https://www.youtube.com/watch?v=o4MwTvtyrUQ https://www.youtube.com/watch?v=o4MwTvtyrUQ Imagine each one of those people in the video has a personal data store that contains their complete medical history, right alongside all their Google searches, Amazon purchases and tweets. How secure of a password do you think they picked? How hard would it be to trick them with a phishing attack? How much do you think they even know how to do with it in the first place?

jckt 13y ago

+1, but do you mind expanding on "discoverability"?

abraininavat 13y ago
I'm no expert, but it occurs to me that a mountain of data coming in from disparate sources needs the concept of discoverability to be adequately used. That is, there has to be a way for a machine to find occurrences of data that it cares about without specifically knowing the exact attributes of the data. A simple example off the top of my head. Let's say there's a popular app that collects your pulse from your cellphone somehow, and that data is stored in your "Personal API" in some format. Some scientists write a routine that scans people's pulse patterns (with their permission, say) and warns them of possible risk factors. Later on hospitals start feeding their pulse data into the same personal API, but in a different format and using different units (bpm instead of bph, say), because they've never heard of the cellphone app. I take "discoverability" as a general concept addressing how the scientists' routine might discover that the hospital data exists in a person's "Personal API", discover that the thing being measured by the hospital is the same as the thing being measured by the app, and discover how to map both formats into a common workable format and units.
- fosk 13y ago
  If I understand correctly, isn't this the purpose of a good documentation?
  - abraininavat 13y ago
    I don't think so. Documentation is for people. Discoverability is for machines. Notice I suggested it's the scientists' routine which does the discovery, not the scientists themselves.
  - klenwell 13y ago
    It's more about money. This data is worth a lot of money to marketers, retailers, pharmaceutical companies, etc. They have no incentive to share it freely or, I suspect, at a price reasonable to most individual consumers. (Maybe I'm wrong here?) There was that interesting article a while back about how Target was able to figure out a teenage girl was pregnant before her own father did: http://www.nytimes.com/2012/02/19/magazine/shopping-habits.html?pagewanted=all&_r=0 http://www.nytimes.com/2012/02/19/magazine/shopping-habits.h... It glosses this point: Target can buy data about your ethnicity, job history, the magazines you read, if you’ve ever declared bankruptcy or got divorced, the year you bought (or lost) your house, where you went to college, what kinds of topics you talk about online, whether you prefer certain brands of coffee, paper towels, cereal or applesauce, your political leanings, reading habits, charitable giving and the number of cars you own. (In a statement, Target declined to identify what demographic information it collects or purchases.) I'd be very curious, and a little scared, to see all this kind of information collected on me.
- naveenium 13y ago
  couldn't one also keep track of the 'source' from where a data point comes (i do this, but don't reveal it in the API - yet)? this way, when it comes to discoverability via some algorithm, you could always separate the 'verified' bucket from the 'unknown' bucket?
klenwell 13y ago
A trivial example from my own recent experience: getting my highlights and notes out of the Amazon Kindle app on my phone. Amazon could make these available through a simple RSS feed or export feature, but they don't. The only way I can get my hands on them is to scrape their website, which depends on some fragile scripting and is subject to their terms of service. I really like this idea, but without some kind of positive government regulation, I don't see it happening. At least not in the sort of meaningful ways the OP envisions.

rasur 13y ago

Have you read Jaron Laniers new book "Who Owns the Future"? You have similar ideas ("Imagine if the wealth of data collected about you all went to one place, and, most importantly, was controlled by you. Imagine if Company X had to pay you to know about your Amazon spending habits instead of paying Amazon.") It's a thought provoking book.. and slightly worrisome to boot (about the state of things at the moment, with regard to the economics of "sharing data").

naveenium 13y ago
yes, i do think of it that way: as we leave all this data exhaust behind us, i keep wanting a system to have it all in one place – or at least, an "interface" that shows it to me in one place, even if that's not how the backend really is. thanks for the book recommendation - i will check it out this weekend!
- amirmc 13y ago
  Academia has been thinking about this for a while. I'm involved with several projects that aim to put users in control of their 'digital footprint' and what you've described are two aspects of the work. First is how to collate and access all the data and second is how to allow third parties to query/process that data in a way that respects a users' privacy. [1] I'm keen to commercialize the work when it's ready but it's a research project precisely because it's a tough problem. Of course, one approach is to just make that personal info public but I don't think that will really benefit users in the long term (companies would certainly benefit, though) [1] http://perscon.net/overview/dataware.html http://perscon.net/overview/dataware.html
msvan 13y ago
Jaron Lanier looks and behaves like a total eccentric, and his ideas can seem pretty out there, but I found his latest book to be a really interesting read. This idea is not the same thing though, because the data ends up being more or less public and demonetized. Companies are already collecting lots of data about you. This, in theory, just gives them even more of it, free of charge.

icebraining 13y ago

This is very much the goal of Project VRM: 1. https://cyber.law.harvard.edu/projectvrm/Main_Page https://cyber.law.harvard.edu/projectvrm/Main_Page 2. http://blogs.law.harvard.edu/vrm/ http://blogs.law.harvard.edu/vrm/

j_s 13y ago
Neat project! Took me about 5 clicks to find out that VRM is defined as 'Vendor Relationship Management'.
- icebraining 13y ago
  Hmm, it's the first phrase of the first link. About VRM VRM stands for Vendor Relationship Management. (...)

wslh 13y ago

In my spare time I am doing something like it. It's called Egont (No Ego OR Ego New Technology). If you liked this article, you might like: i) Egont, a web orchestration language: http://blog.databigbang.com/ideas-egont-a-web-orchestration-language/ http://blog.databigbang.com/ideas-egont-a-web-orchestration-... ii) Egont part II: http://blog.databigbang.com/egont-part-ii/ http://blog.databigbang.com/egont-part-ii/ iii) Parsing s-expressions in C# using OMeta: http://blog.databigbang.com/parsing-s-expressions-in-c-using-ometa/ http://blog.databigbang.com/parsing-s-expressions-in-c-using... Because your stuff is defined as an s-expression and then optimized to a DAG.

quantumpotato_ 13y ago
[1] http://en.wikipedia.org/wiki/Directed_acyclic_graph http://en.wikipedia.org/wiki/Directed_acyclic_graph ? Can you explain more for us layfolk?
- wslh 13y ago
  Yes, An s-expression is a tree. Imagine that you want to know the average score of a list of films like: (average listOfFilms) and someone else like the same average for the same list. You will end up with two trees that indeed are the same and can be calculated only once. A DAG is a way to represent dependencies, so if two cells are linked to the same cell (average listOfFilm).
  - wslh 13y ago
    One last clarification: not all trees can be converted to a DAG. There cannot be circular references.
- deleted 13y ago
  [deleted]
- deleted 13y ago
  [deleted]
- deleted 13y ago
  [deleted]

iusable 13y ago

Agree 100%! The point that's important to remember about discoverability is that it is usually mission-centric. e.g. - you want to discover the 'right' kind of people to follow, as to have better or more entertaining or more useful or _____________ information. What is 'right' for you may not be 'right' for me, since our goals most likely aren't the same. I love Naveen's idea (and more importantly his implemention) & will be playing with it a bit more this weekend. But, the value may not be apparent right away. Aggregating and mining one type of data across a relationship may be more valuable. It may be that I have to track a similar set of data for myself and my significant other to make sense of the aggregates.

skybrian 13y ago

I think this is a very libertarian notion that misses out on the social aspects of reputation. What you claim about yourself is often less valuable to a third party than what someone else claims about you. Perhaps you could store signed endorsements made by other people. But then you need support for revocation, and what about negative claims? Free speech includes the ability to say negative things about others. Social software is very complicated because of issues like this. Something like a personal dropbox is straightforward - single customer, no community. You only have to worry about people hacking you, not what they do to each other. Anyone who gets into social software inevitably sets themselves up for making rules and judging disputes, and that often requires hard decisions. Or compare email with social messaging; the rights and responsibilities are different and it's due to the software. Personal servers only support setting ground rules in a certain way (for example, the inability to revoke or prevent copies). Centralized services can support communities with different rules, for better or worse.

graue 13y ago

> Imagine if the wealth of data collected about you all went to one place, and, most importantly, was controlled by you. Imagine if Company X had to pay you to know about your Amazon spending habits instead of paying Amazon. Even more than Project VRM linked above, this is precisely the eventual goal of Tent: https://tent.io/ https://tent.io/ That said, I'm skeptical of these projects', or anyone's, ability to achieve this utopia. Why would Amazon or Google adopt such a system? Your shopping/searching history is valuable data that gives them a business advantage. Also, while a personal data store that you control may be a geek's wet dream, for an average nontechnical person, it would likely be difficult to use, insecure and offer no clear benefits. And if it ends up just a toy for geeks, the lack of scale would make it even harder to do anything interesting with it or get any companies of significant size to play ball. I do hope ideas like this surface in some form, but I'm skeptical of the personal data store ever really becoming a thing.

abraininavat 13y ago
Why would Amazon or Google adopt such a system I could imagine laws requiring that personal data be controlled by the subjects. it would likely be difficult to use, insecure... Who says? How can you assume this? and offer no clear benefits Some would consider the ability to keep your information private at will a benefit. Others would consider the ability to sell said information a benefit. the lack of scale would make it even harder to do anything interesting Lack of scale? An individual's data would be controlled by that individual, but the system would scale across all of humanity. I don't think you're thinking big enough -- I'm not talking about each person running his personal EC2.
- graue 13y ago
  Look, I'm not saying don't go for this idea. I'd love to see you try it. My doubts shouldn't get in your way. Code it up. As for why it would be difficult to use and insecure? I'm guessing most of the people you hang out with are probably geeks. For perspective, take a look at this video: https://www.youtube.com/watch?v=o4MwTvtyrUQ https://www.youtube.com/watch?v=o4MwTvtyrUQ Imagine each one of those people in the video has a personal data store that contains their complete medical history, right alongside all their Google searches, Amazon purchases and tweets. How secure of a password do you think they picked? How hard would it be to trick them with a phishing attack? How much do you think they even know how to do with it in the first place?