3 ms·

I'm working on some software to record as much data as possible, from as many sources as possible, into one unified form so that you can actually analyze it. I

by thejash 13y ago

I'm working on some software to record as much data as possible, from as many sources as possible, into one unified form so that you can actually analyze it. I want the data sources to include far more than just health data--GPS, title of active computer window, sms history, file modification times, device and sensor readings, snapshots from your webcam, etc--really anything that you can record. If anyone is interested, feel free to email me!

icebraining 13y ago
What's the unified form?
- thejash 13y ago
  I thought about it for a while, and this is the best I could come up with: - All data is just a stream of "events" from different "sources" for any given "data type" - Sources are things like my desktop, my laptop, my phone, my camera, etc - Data Types are things like global position, cpu usage, webcam snapshots, webpage text, etc. Anything really. - Events are just a timestamp + zero or more other "lightly typed" "fields". - By "lightly typed" I mean the field is either: binary data, searchable text, a float, or an integer - Fields give the actual data for the event. If there are no fields, it's a countable event (ex: imagine recording every heartbeat, all you need is a timestamp). For something like global position, you would have latitude and longitude fields. For another datatype called "semantic location" you have just a searchable text string with things like "car", "work", "home" or whatever other labels you have. Fields can also be binary data (like photos, videos, etc). I'll probably try to make lots of data types, each with very few fields, to make it easier to work with the data (ie, it's nice if there is only one numeric field in any given data type, then it's easy to graph over time) I'm currently collecting all of the events via some scripts that are specific to each data type, and then emailing those events (serialized as json) to my own server, which just inserts the events into a sqlite database specific to that source/data type pair. The nice thing about this approach is that anyone can easily send an email from any language, and if you don't want to use my backend, you can just send the events to your own email server and then do whatever you want with them once they're there. Another benefit is that you and I don't have to agree about exactly what fields and data types there are. If you call your gps data "gps" and I call mine "global position", well, who cares? It's our own personal data. If this takes off, I'll make guidelines about what to call what information and how to format it for easier interoperability.
antjanus 13y ago
Sounds like you're working for the NSA. =D
- thejash 13y ago
  Every time I hear about that stuff, I get annoyed--if they'd just open up their API, I wouldn't have to build this ;) Bet seriously, security is definitely a goal of this project.

by thejash 13y ago

icebraining 13y ago

What's the unified form?

thejash 13y ago
I thought about it for a while, and this is the best I could come up with: - All data is just a stream of "events" from different "sources" for any given "data type" - Sources are things like my desktop, my laptop, my phone, my camera, etc - Data Types are things like global position, cpu usage, webcam snapshots, webpage text, etc. Anything really. - Events are just a timestamp + zero or more other "lightly typed" "fields". - By "lightly typed" I mean the field is either: binary data, searchable text, a float, or an integer - Fields give the actual data for the event. If there are no fields, it's a countable event (ex: imagine recording every heartbeat, all you need is a timestamp). For something like global position, you would have latitude and longitude fields. For another datatype called "semantic location" you have just a searchable text string with things like "car", "work", "home" or whatever other labels you have. Fields can also be binary data (like photos, videos, etc). I'll probably try to make lots of data types, each with very few fields, to make it easier to work with the data (ie, it's nice if there is only one numeric field in any given data type, then it's easy to graph over time) I'm currently collecting all of the events via some scripts that are specific to each data type, and then emailing those events (serialized as json) to my own server, which just inserts the events into a sqlite database specific to that source/data type pair. The nice thing about this approach is that anyone can easily send an email from any language, and if you don't want to use my backend, you can just send the events to your own email server and then do whatever you want with them once they're there. Another benefit is that you and I don't have to agree about exactly what fields and data types there are. If you call your gps data "gps" and I call mine "global position", well, who cares? It's our own personal data. If this takes off, I'll make guidelines about what to call what information and how to format it for easier interoperability.

antjanus 13y ago

Sounds like you're working for the NSA. =D

thejash 13y ago
Every time I hear about that stuff, I get annoyed--if they'd just open up their API, I wouldn't have to build this ;) Bet seriously, security is definitely a goal of this project.